Skip to content

Add C# benchmarks and synchronize reports, correctness tests and CI - #43

Merged
konard merged 6 commits into
mainfrom
issue-42-5c34a9a6fc98
Oct 6, 2026
Merged

konard merged 6 commits into
mainfrom
issue-42-5c34a9a6fc98

Conversation

@konard

@konard konard commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Fixes #42.

Rust and C# now benchmark the same eight point-link operations across four Doublets stores and PostgreSQL with and without an explicit transaction. Reports use the fastest PostgreSQL implementation as the baseline and distinguish × faster, × slower, and ≈ same (medians within 5% or overlapping ± one standard deviation ranges).

Changes:

  • Add .NET 10 benchmarks using Npgsql 10.0.0 and Platform.Data.Doublets 0.18.1, following the sibling Neo4j comparison's store and harness patterns. Commit NuGet dependency locks.
  • Generate the entire README results section between results:start/results:end markers. Every table records PostgreSQL, driver and Doublets versions, CPU, UTC date and an Actions run link. Preserve linear and logarithmic charts for both languages and every size.
  • Add tested parsing and validation for missing, malformed, duplicate or mismatched results and provenance. Keep rust/out.py as a compatibility entry point to the shared report script.
  • Run rustfmt, clippy with warnings as errors, Rust tests, C# formatting/build/tests, and report tests in CI. Each benchmark job runs one language, backend and size on its own runner. PostgreSQL runs only in the jobs that need it.
  • Pin PostgreSQL to 17.6; add workflow path filters, concurrency and job timeouts. PRs use 100 background / 10 active links; default-branch runs also measure 1,000 / 100. PR reports are artifacts; default-branch reports are committed automatically.
  • Read Rust's connection from POSTGRES_CONNECTION and C#'s from NPGSQL_CONNECTION. Document both connection formats, sizes, measured boundaries and local commands. Reject invalid benchmark sizes.
  • Remove the dormant C++ implementation and workflow, as permitted by the issue. Replace the old hand-written size and speedup claims with workload and provenance explanations.

The new correctness tests also exposed existing Rust adapter defects: identity enumeration used %1 instead of $1, one count query joined AND to the next column without a space, and missing-link lookups indexed empty result sets. These are fixed. SQL clients now use mutexes instead of the unsafe shared-mutable Exclusive wrapper. Delete benchmarks previously deleted N + 1 links, including one background point; both languages now delete exactly the N prepared points.

Reproduction and verification:

  • python3 experiments/reproduce_report_ratio.py previously failed with 200 (0.5x faster) for Doublets taking twice the PostgreSQL time; it now prints 2× slower. This reproducer needs no plotting dependencies.
  • python3 -m unittest discover -s scripts -v: 10 tests cover direction, the noise boundary, overlapping deviations, fastest-baseline selection, parsing, incomplete results, mismatched provenance and marker replacement.
  • cargo fmt --manifest-path rust/Cargo.toml -- --check and cargo clippy --locked --manifest-path rust/Cargo.toml --all-targets --all-features -- -D warnings pass.
  • cargo test --locked --manifest-path rust/Cargo.toml -- --include-ignored: both integration tests pass, exercising all six stores, all query shapes, update/restore, delete counts, absent links and repeated fork cleanup. A local run used a custom database and host to exercise the connection override.
  • C# formatting and the release build with warnings as errors pass. dotnet test csharp/PostgreSQLVSDoublets.slnx -c Release --no-build --blame-hang-timeout 2m: 7 passed, none skipped with PostgreSQL configured. Tests exercise all six stores, every benchmark operation, cleanup, sampling, statistics and output formatting.
  • actionlint .github/workflows/rust.yml passes. experiments/run_benchmarks.sh completed all 96 measurements and generated both languages’ reports and all four charts locally. See the README for setup and complete commands.

Measurement scope: preparation and cleanup are excluded from both harnesses. PostgreSQL transaction begin/commit is outside the timed operations, matching the existing Rust workload; memory-mapped Doublets stores are not explicitly flushed to disk. Indexed read iterations query every background point, while writes use the active count. The parity tests cover this point-link benchmark workload. Local PostgreSQL validation uses version 16; CI uses the pinned 17.6 image.

The isolated CI run passed all test, benchmark and report jobs on commit 2396e66. The README tables and charts below use this run's PostgreSQL 17.6 measurements. Separate runner CPUs are identified in each table's provenance; these ratios apply to this workload and those machines.

Final-commit CI also passed all eight jobs on 0142cb9, including both languages’ PostgreSQL/Doublets measurements and report generation. CodeFactor passed. The branch includes current main and the working tree is clean.

Rust results, logarithmic scale

C# results, logarithmic scale

Changes

  • 70 file(s) modified
  • 2190 line(s) added
  • 2140 line(s) removed
  • Files:
    • .github/workflows/cpp.yml
    • .github/workflows/rust.yml
    • .gitignore
    • Docs/bench_csharp_100.png
    • Docs/bench_csharp_log_scale_100.png
    • Docs/bench_rust_100.png
    • Docs/bench_rust_log_scale_100.png
    • README.md
    • cpp/Benchmarks/AllBenchmarks.cpp
    • cpp/Benchmarks/Create.cpp
    • cpp/Benchmarks/Delete.cpp
    • cpp/Benchmarks/Each/All.cpp
    • cpp/Benchmarks/Each/Concrete.cpp
    • cpp/Benchmarks/Each/Identity.cpp
    • cpp/Benchmarks/Each/Incoming.cpp
    • cpp/Benchmarks/Each/Outgoing.cpp
    • cpp/Benchmarks/SetupTeardown.cpp
    • cpp/Benchmarks/Update.cpp
    • cpp/CMakeLists.txt
    • cpp/LinksPSQL/Client.h
    • cpp/LinksPSQL/ISQL.h
    • cpp/LinksPSQL/PSQLBase.h
    • cpp/LinksPSQL/Transaction.h
    • cpp/conanfile.txt
    • cpp/out.py
    • csharp/PostgreSQLVSDoublets.Tests/PostgreSQLVSDoublets.Tests.csproj
    • csharp/PostgreSQLVSDoublets.Tests/SameBehaviorTests.cs
    • csharp/PostgreSQLVSDoublets.Tests/packages.lock.json
    • csharp/PostgreSQLVSDoublets.slnx
    • csharp/PostgreSQLVSDoublets/Benchmarks.cs
    • csharp/PostgreSQLVSDoublets/DoubletsLinks.cs
    • csharp/PostgreSQLVSDoublets/Harness.cs
    • csharp/PostgreSQLVSDoublets/IBenchedLinks.cs
    • csharp/PostgreSQLVSDoublets/PostgreSQLVSDoublets.csproj
    • csharp/PostgreSQLVSDoublets/PostgresLinks.cs
    • csharp/PostgreSQLVSDoublets/Program.cs
    • csharp/PostgreSQLVSDoublets/packages.lock.json
    • csharp/global.json
    • experiments/reproduce_report_ratio.py
    • experiments/run_benchmarks.sh
    • rust/benches/bench.rs
    • rust/benches/benchmarks/doublets/create.rs
    • rust/benches/benchmarks/doublets/delete.rs
    • rust/benches/benchmarks/doublets/each/all.rs
    • rust/benches/benchmarks/doublets/each/concrete.rs
    • rust/benches/benchmarks/doublets/each/identity.rs
    • rust/benches/benchmarks/doublets/each/incoming.rs
    • rust/benches/benchmarks/doublets/each/outgoing.rs
    • rust/benches/benchmarks/doublets/update.rs
    • rust/benches/benchmarks/psql/create.rs
    • …and 20 more

Adding .gitkeep for PR creation (default mode).
This file will be removed when the task is complete.

Issue: #42
@konard konard self-assigned this Oct 6, 2026
@konard konard changed the title [WIP] Add a C# version and sync the report, tests and CI with the other comparisons Add C# benchmarks and synchronize reports, correctness tests and CI Oct 6, 2026
@konard
konard marked this pull request as ready for review October 6, 2026 23:53
@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Working session summary

Implemented issue #42 in PR #43, now ready for review.

Added C# benchmarks, corrected report comparisons, generated tables and charts with provenance, added parity tests, isolated CI jobs, and removed dormant C++ code.

All 19 tests, 96 local benchmark measurements, and final-commit CI passed. The branch includes current main; the working tree is clean.


This summary was automatically extracted from the AI working session output.

@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Model: GPT-6.1 Sol
  • Provider: OpenAI
  • Public pricing estimate: $2.651629

📊 Context and tokens usage:

  • 235.1K input tokens across requests (cumulative, larger than the 200K context window, so not one request's context), 48.7K / 128K (38%) output tokens

Total: (235.1K + 4.9M cached) input tokens, 48.7K output tokens, $2.651629 cost

🤖 Models used:

  • Tool: OpenAI Codex
  • Requested: gpt-6.1-sol
  • Thinking level: xhigh (~31999 tokens)
  • Model: GPT-6.1 Sol (gpt-6.1-sol)

📎 Log file uploaded as Gist (3836KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard
konard merged commit d905b41 into main Oct 6, 2026
9 checks passed
@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

🎉 Auto-merged

This pull request has been automatically merged by hive-mind.

  • All CI checks have passed

Auto-merged by hive-mind with --auto-merge flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a C# version and sync the report, tests and CI with the other comparisons

1 participant