Repository navigation

feat(evals): add doc-sync quality evals and hillclimb loop - #2036

Draft

soheimam wants to merge 41 commits into

masterfrom

docs/improve-agent-doc-writing-skills

Commits on Sep 30, 2026

docs(evals): add doc-sync quality eval build plan
soheimam
committed
1f03b17
feat(evals): add keccak-256 implementation for selector grading
soheimam
committed
d8b3a3a
feat(evals): add line-diff helper for grading added lines
soheimam
committed
f12eebe
feat(evals): add GitHub REST client, stats, and taxonomy modules for metrics lane
soheimam
committed
13471a0
feat(evals): add merge-rate, mine-feedback, and report scripts plus the report workflow
soheimam
committed
4c6a675
wip(evals): checkpoint replay harness and cases (timed-out worker)
soheimam
committed
e418fe7
wip(evals): checkpoint stats exclusions (timed-out worker)
soheimam
committed
1118b73
feat(evals): add page-role helper and scope code check
soheimam
committed
1270646
feat(evals): add grounding code check
soheimam
committed
4be53f4
feat(evals): add selector code check
soheimam
committed
1ee2f46
fix(evals): cap reconstructed changed_paths and mine PR issue comments
soheimam
committed
d4aa4e3
feat(evals): add lint and housekeeping code checks
soheimam
committed
1ebfac5
fix(evals): treat nav-file writes as touched pages in log parsing
soheimam
committed
5344ac2
feat(evals): add changelog.shape and changelog.fidelity code checks
soheimam
committed
46d5023
test(evals): cover worktree create/remove lifecycle including failure cleanup
soheimam
committed
f9175f3
feat(evals): add noop code check and code-checks aggregator
soheimam
committed
dd3ca2d
docs(evals): add Lane A replay harness README with decisions log
soheimam
committed
755ddab
feat(evals): add LLM judge (prompt/parse/run split)
soheimam
committed
6789831
feat(evals): add pairwise comparison grader
soheimam
committed
cdd87af
fix(evals): fetch per-PR totals for the human rewrite ratio
soheimam
committed
7c97d7b
feat(evals): add grade.mjs CLI and gradeRep() core
soheimam
committed
7107910
Merge evals/harness: replay harness and eval cases
soheimam
committed
8fa6e7a
Merge evals/metrics: merge-rate, feedback mining, weekly report
soheimam
committed
90329ef
chore(evals): run doc-evals tests in npm test, ignore mined candidates, mirror sync env in replay
soheimam
committed
82dc2ab
feat(evals): add calibrate.mjs human-calibration tool
soheimam
committed
3408582
fix(evals): exclude legacy-layout cases, drop drafted scope pages absent at base, lazy-load sync code for CI
soheimam
committed
032230a
Merge integration branch into evals/graders (updated PLAN contracts, cases, harness)
soheimam
committed
3bf25d5
fix(evals): drop static index.mjs import from graders pageRole so CI tests run without the sdk
soheimam
committed
5afa4f9
fix(evals): lazy-load LLM client in judge runner, retry with backoff per model, cap source diff in prompts
soheimam
committed
cc550a5
fix(evals): report drafted scope checks as unscored (pass null) and assert graders receive the payload diff
soheimam
committed
d1626f7
feat(evals): pairwise judges both orders with source diff and reviewer rubric; win/loss only when orders agree
soheimam
committed
ba91c5c
fix(evals): changelog checks ignore pre-existing shape gaps and edit-only source diffs; report variance token cost
soheimam
committed
b1f8ae7
Merge evals/graders: code checks, LLM judge, pairwise, grade CLI, calibration
soheimam
committed
cd73045
feat(evals): apply owner decisions: confirmed scope, directory scope rules, grading errors excluded, no pairwise on changelog entries
soheimam
committed
4d11d89
feat(evals): hillclimb decision rule, noise estimate, and budget guard
soheimam
committed
c22dd96
feat(evals): hillclimb proposer prompts and patch validation/scratch application
soheimam
committed
6b80fdf
feat(evals): hillclimb loop, evaluation, report, and CLI
soheimam
committed
0e9a303
docs(evals): document the hillclimb loop and add CLI arg tests
soheimam
committed
4da55e8
Merge evals/hillclimb: hillclimb loop with train/test decisions, budget guard, reflection
soheimam
committed
bf5ffc0
fix(evals): make scope its own weighted term so scope creep dominates the overall score
soheimam
committed
698889e
feat(evals): Build on Base off-limits for every case, 16k output tokens for evals, no judge in first hillclimb runs
soheimam
committed
a4e0fe2