Repository navigation

bench: MCP-transport sibling of the code-graph track - #684

Draft

DvirDukhan wants to merge 28 commits into

dvirdukhan/benchmark-planfrom

dvirdukhan/bench-mcp-track

Draft

bench: MCP-transport sibling of the code-graph track#684
DvirDukhan wants to merge 28 commits into
dvirdukhan/benchmark-planfrom
dvirdukhan/bench-mcp-track

Commits on May 27, 2026
- bench: add MCP-transport sibling of the code_graph track
  
  DvirDukhan
  and
  Copilot
  committed
  60e2bd1
- bench: wire code_graph_mcp into mini_runner dispatch
  
  DvirDukhan
  and
  Copilot
  committed
  f17d437
- bench: fail loudly on indexing errors + bump analyze_folder timeout
  
  DvirDukhan
  and
  Copilot
  committed
  b14432b
- fix(analyzer): resolve LSP CALLS edges on repos without a venv
  
  dvirdukhan
  and
  Copilot
  committed
  532d849
- bench: add resume support + ignore sympy rubi rules
  
  dvirdukhan
  and
  Copilot
  committed
  476bc73
Commits on May 28, 2026
- fix(analyzer): defensive skip when second_pass references untracked file
  
  DvirDukhan
  and
  Copilot
  committed
  d23ef79
- bench: add start-api.sh helper enabling tree-sitter fast resolver
  
  DvirDukhan
  and
  Copilot
  committed
  ec7fac6
Commits on May 30, 2026
- feat(bench): LocAgent-style localization benchmark runner
  
  DvirDukhan
  and
  Copilot
  committed
  2d9c630
- fix(bench): reap process tree on timeout in localization runner
  
  DvirDukhan
  and
  Copilot
  committed
  c75dfd6
Commits on May 31, 2026
- fix(bench): bound localization API calls with timeout + retries
  
  DvirDukhan
  and
  Copilot
  committed
  515ffa2
- fix(bench): SIGALRM per-call timeout+retry for stalled localization API calls
  
  DvirDukhan
  and
  Copilot
  committed
  6f3b374
- feat(bench): forced-workflow ablation for localization benchmark
  
  DvirDukhan
  and
  Copilot
  committed
  6e62143
- fix(bench): retry localization worktree clone on transient exit-128
  
  DvirDukhan
  and
  Copilot
  committed
  a9db403
- feat(bench): support SWE-bench-Live corpus for localization
  
  DvirDukhan
  and
  Copilot
  committed
  91720fe
- bench: add deterministic structural-query token-compression benchmark
  
  DvirDukhan
  and
  Copilot
  committed
  327c665
- bench: add --instances-file to reproduce an exact instance set
  
  DvirDukhan
  and
  Copilot
  committed
  0d14d85
- bench: add Opus-vs-Sonnet model comparison report (accuracy/tokens/USD)
  
  DvirDukhan
  and
  Copilot
  committed
  9700831
- bench: record 'ungraded' when inline verify can't run pytest
  
  DvirDukhan
  and
  Copilot
  committed
  b1eb0c2
- fix(bench): resolve Copilot cache/log paths to absolute
  
  DvirDukhan
  and
  Copilot
  committed
  550c60d
- bench(copilot): TCO accounting + runner tests; ignore harness logs/
  
  DvirDukhan
  and
  Copilot
  committed
  9b63f8f
Commits on Jun 1, 2026
- feat(bench): add nudge + localize modes to copilot_runner
  
  DvirDukhan
  and
  Copilot
  committed
  15155cc
- bench(copilot): harden localization harness + add --no-leak structural selection
  
  DvirDukhan
  and
  Copilot
  committed
  e357a2f
- bench: offline retrieval probe for Phase A localization gap
  
  DvirDukhan
  and
  Copilot
  committed
  be01f47
Commits on Jun 4, 2026
- bench: add Lane 1 adoption-calibration prompt arms (CTRL/SEM/RAT)
  
  DvirDukhan
  and
  Copilot
  committed
  331d708
- bench: validate adopt_arm value in build_prompt/run_one
  
  DvirDukhan
  and
  Copilot
  committed
  d5bdf13
- bench: add per-arm adoption diagnostics (adopt_diag)
  
  DvirDukhan
  and
  Copilot
  committed
  b44b1e6
- bench: thread NOISY distractor injection env into code_graph localize runner
  
  DvirDukhan
  and
  Copilot
  committed
  e1ebe59
Commits on Jun 6, 2026
- bench: multi-hop nav harness, trace/adoption analysis, relocated MCP controls
  
  DvirDukhan
  and
  Copilot
  committed
  a19b65e

Commits

Commits on May 27, 2026

Commits on May 28, 2026

Commits on May 30, 2026

Commits on May 31, 2026

Commits on Jun 1, 2026

Commits on Jun 4, 2026

Commits on Jun 6, 2026