Skip to content

Add C# comparison and align benchmark reports and CI - #17

Merged
konard merged 7 commits into
mainfrom
issue-16-e3181c5664cc
Oct 7, 2026
Merged

konard merged 7 commits into
mainfrom
issue-16-e3181c5664cc

Conversation

@konard

@konard konard commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Adds the C# comparison and makes both languages report all seven operations against SpacetimeDB and four Doublets layouts. Results now show three significant digits with units and ≈ same when medians differ by less than 5% or their ±1 standard deviation ranges overlap. For example, the historical Query All comparison changes from 21580 (1.0x faster) to 21.6 µs (≈ same).

Fixes #16.

  • C# uses .NET 10, the official SpacetimeDB SDK 2.10.1 and Platform.Data.Doublets 0.18.1, with locked dependencies and generated bindings for the shared WASM module. Workloads, setup/reset and Criterion sampling settings match Rust, including automatic flat sampling for slow iterations.
  • Differential tests exercise all five backends in each language, including repeated distributed queries, updates and resets. They exposed two existing Rust bugs: create_by discarded source/target values, and Split updates classified self-references after detaching them from the index. Both are corrected. Query datasets now use unique pairs referencing actual background IDs, and query results are consumed to preserve the measured work.
  • CI runs each language/backend on a separate VM, pins the CLI/server to 2.10.1, fails on Clippy warnings and cancels superseded runs. Recorded metadata includes CPU, server/module, SDK and Doublets versions alongside date, run and sizes. The report verifies complete measurements and consistent sizes, updates separate README sections and generates both languages' charts.
  • Removes the unrelated root template crate, tests/example and package release workflow; updates contributor commands and adds a changelog fragment.

Validation completed locally:

  • Rust formatting and cargo clippy --locked --all-targets -- -D warnings.
  • All 16 Rust tests, explicitly including the live-server equivalence test.
  • Locked C# restore, formatting, Release build with warnings treated as errors, and all 5 tests against the live server.
  • Regenerated C# bindings match the committed files.
  • All 39 report tests, repository file-size checks, shell syntax checks and actionlint.
  • Installed the pinned CLI through the new installer and verified recorded version/CPU metadata.

Reproduction and regression coverage: create_preserves_endpoints fails before the adapter correction; experiments/reproduce-split-index.sh runs a bounded differential reproduction of the Split index error; rust/tests/same_behavior.rs and the C# same-behavior tests compare actual rows, normalized identities and repeated benchmark datasets. Report tests cover the original Query All noise case, deviation overlap, unit boundaries, independent backend artifacts, provenance and mismatched sizes.

The README explicitly identifies existing Rust numbers as historical and does not invent missing CPU information. C# measurements and both updated language reports are generated by PR CI as review artifacts; the first full default-branch run publishes current tables and charts. PR validation uses N=10/B=30, while full runs use N=1000/B=3000.

CI on 79b4726: all nine jobs passed, including Rust tests on Linux/macOS, live C# tests, all four isolated benchmark jobs and report generation. Reviewed the benchmark-report artifact: both complete 7-operation/5-backend tables, four valid charts, raw measurements and recorded metadata.

C# PR validation report (N=10, B=30)

Generated 2026-10-06 23:46 UTC by GitHub Actions run 37548080915 — 10 benchmarked links, 30 background links. SpacetimeDB.ClientSDK 2.10.1; Platform.Data.Doublets 0.18.1; SpacetimeDB module 2.10.1; SpacetimeDB server/CLI 2.10.1; CPU — spacetimedb: AMD EPYC 7763 64-Core Processor; doublets: AMD EPYC 7763 64-Core Processor.

Operation Doublets United Volatile Doublets United NonVolatile Doublets Split Volatile Doublets Split NonVolatile SpacetimeDB
Create 4.04 µs (6,363× faster) 4.04 µs (6,353× faster) 2.20 µs (11,683× faster) 2.27 µs (11,293× faster) 25.7 ms
Update 5.10 µs (4,891× faster) 5.14 µs (4,860× faster) 1.77 µs (14,111× faster) 1.76 µs (14,151× faster) 25.0 ms
Delete 2.54 µs (5,095× faster) 2.69 µs (4,816× faster) 2.15 µs (6,029× faster) 2.26 µs (5,725× faster) 13.0 ms
Query All 998 ns (23.5× faster) 1.31 µs (18× faster) 1.04 µs (22.6× faster) 1.01 µs (23.1× faster) 23.5 µs
Query by Id 1.37 µs (5.49× faster) 1.43 µs (5.28× faster) 1.52 µs (4.98× faster) 1.48 µs (5.11× faster) 7.55 µs
Query by Source 1.52 µs (9.78× faster) 1.52 µs (9.77× faster) 1.43 µs (10.4× faster) 1.38 µs (10.7× faster) 14.8 µs
Query by Target 2.06 µs (7.13× faster) 2.06 µs (7.13× faster) 1.47 µs (10× faster) 1.45 µs (10.1× faster) 14.7 µs

Changes

  • 54 file(s) modified
  • 3106 line(s) added
  • 1404 line(s) removed
  • Files:
    • .github/workflows/release.yml
    • .github/workflows/rust-benchmark.yml
    • .gitignore
    • .pre-commit-config.yaml
    • CONTRIBUTING.md
    • Cargo.lock
    • Cargo.toml
    • README.md
    • changelog.d/20261006_comparison_parity.md
    • csharp/SpacetimeDBVSDoublets.Tests/HarnessTests.cs
    • csharp/SpacetimeDBVSDoublets.Tests/SameBehaviorTests.cs
    • csharp/SpacetimeDBVSDoublets.Tests/SpacetimeDBVSDoublets.Tests.csproj
    • csharp/SpacetimeDBVSDoublets.Tests/packages.lock.json
    • csharp/SpacetimeDBVSDoublets.slnx
    • csharp/SpacetimeDBVSDoublets/Benchmarks.cs
    • csharp/SpacetimeDBVSDoublets/DoubletsLinks.cs
    • csharp/SpacetimeDBVSDoublets/Harness.cs
    • csharp/SpacetimeDBVSDoublets/ILinks.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/Reducers/CreateLink.g.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/Reducers/DeleteAllLinks.g.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/Reducers/DeleteLink.g.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/Reducers/UpdateLink.g.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/SpacetimeDBClient.g.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/Tables/Links.g.cs
    • csharp/SpacetimeDBVSDoublets/ModuleBindings/Types/Link.g.cs
    • csharp/SpacetimeDBVSDoublets/Program.cs
    • csharp/SpacetimeDBVSDoublets/SpacetimeDBVSDoublets.csproj
    • csharp/SpacetimeDBVSDoublets/SpacetimeLinks.cs
    • csharp/SpacetimeDBVSDoublets/packages.lock.json
    • csharp/global.json
    • examples/basic_usage.rs
    • experiments/reproduce-split-index.sh
    • rust/Cargo.toml
    • rust/benches/bench.rs
    • rust/doublets-patched/PATCHES.md
    • rust/doublets-patched/doublets/src/mem/split/store.rs
    • rust/out.py
    • rust/results.md
    • rust/spacetime-module/Cargo.toml
    • rust/src/benched/doublets_benched.rs
    • rust/src/benched/mod.rs
    • rust/src/benched/spacetimedb_benched.rs
    • rust/src/doublets_impl.rs
    • rust/src/lib.rs
    • rust/src/workload.rs
    • rust/test_out.py
    • rust/tests/same_behavior.rs
    • scripts/benchmark-metadata.py
    • scripts/install-spacetimedb.sh
    • scripts/spacetimedb-version
    • …and 4 more

Adding .gitkeep for PR creation (default mode).
This file will be removed when the task is complete.

Issue: #16
@konard konard self-assigned this Oct 6, 2026
@konard konard changed the title [WIP] Add a C# version and sync the report and CI with the other comparisons Add C# comparison and align benchmark reports and CI Oct 6, 2026
@konard
konard marked this pull request as ready for review October 6, 2026 23:52
@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Working session summary

PR #17 is implemented, pushed, and ready for review.

Added C# benchmarks, consistent reports and provenance, backend equivalence tests, isolated CI, and template cleanup. Tests also uncovered and fixed two Rust correctness bugs.

All local checks and all nine CI jobs pass. Working tree is clean. Full-size README results will publish after merge.


This summary was automatically extracted from the AI working session output.

@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Model: GPT-6.1 Sol
  • Provider: OpenAI
  • Public pricing estimate: $4.028427

📊 Context and tokens usage:

  • 388.1K input tokens across requests (cumulative, larger than the 200K context window, so not one request's context), 68.1K / 128K (53%) output tokens

Total: (388.1K + 7.3M cached) input tokens, 68.1K output tokens, $4.028427 cost

🤖 Models used:

  • Tool: OpenAI Codex
  • Requested: gpt-6.1-sol
  • Thinking level: xhigh (~31999 tokens)
  • Model: GPT-6.1 Sol (gpt-6.1-sol)

📎 Log file uploaded as Gist (5934KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard
konard merged commit 80fe811 into main Oct 7, 2026
9 checks passed
@konard

konard commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

🎉 Auto-merged

This pull request has been automatically merged by hive-mind.

  • All CI checks have passed

Auto-merged by hive-mind with --auto-merge flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a C# version and sync the report and CI with the other comparisons

1 participant