feat(evals): add doc-sync quality evals and hillclimb loop

.github/workflows/doc-quality-report.yml

@@ -0,0 +1,61 @@
+# Weekly, non-blocking pipeline metrics for the base-std docs sync bot
+# (scripts/sync-from-base-std/, see scripts/doc-evals/PLAN.md Lane C).
+#
+# Scores nothing and blocks nothing: no pull_request trigger, and the only
+# GitHub write this workflow makes is create-or-update on a single tracking
+# issue titled "Docs sync quality report" (found by exact title match, so a
+# re-run edits the same issue instead of piling up duplicates).
+#
+# Hardening follows this repo's existing pattern exactly (see
+# docs-style-conformance.yml and base-std-docs-sync.yml): pinned
+# step-security/harden-runner, pinned checkout/setup-node SHAs, top-level
+# `permissions: {}` with the job requesting only what it needs, and no
+# `${{ github.event.* }}` interpolated into a `run:` block — the run step
+# below only reads plain env vars it set itself.
+
+name: Docs sync quality report
+
+on:
+  schedule:
+    - cron: "0 13 * * 1" # Monday 13:00 UTC
+  workflow_dispatch: {}
+
+permissions: {}
+
+concurrency:
+  group: doc-quality-report
+  cancel-in-progress: false
+
+jobs:
+  report:
+    name: Docs sync quality report
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: read
+    steps:
+      - name: Harden the runner
+        uses: step-security/harden-runner@002fdce3c6a235733a90a27c80493a3241e56863 # v2.12.1
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-node@1e60f620b9541d16bece96c5465dc8ee9832be0b # v4.0.3
+        with:
+          node-version: "22"
+
+      - name: Install script dependencies
+        run: npm ci --prefix scripts --no-audit --no-fund
+
+      - name: Generate and post the quality report
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          REPORT_OWNER: ${{ github.repository_owner }}
+        run: |
+          set -euo pipefail
+          repo_name="${GITHUB_REPOSITORY#*/}"
+          node scripts/doc-evals/metrics/report.mjs --owner "$REPORT_OWNER" --repo "$repo_name"

package.json

@@ -3,6 +3,6 @@
     "mintlify": "^4.2.823"
   },
   "scripts": {
-    "test": "node --test .github/scripts/__tests__/*.test.mjs scripts/__tests__/*.test.mjs"
+    "test": "node --test .github/scripts/__tests__/*.test.mjs scripts/__tests__/*.test.mjs scripts/doc-evals/__tests__/*.test.mjs"
   }
 }

scripts/doc-evals/.gitignore

@@ -0,0 +1,7 @@
+# Replay output — see the "Replay output" contract in PLAN.md. Every run
+# writes here; none of it is committed.
+/runs/
+
+# Candidate cases from metrics/mine-feedback.mjs. A human reviews each one,
+# runs build-cases.mjs, and promotes it into cases/; raw candidates stay local.
+/cases/_candidates/

scripts/doc-evals/README.md

@@ -0,0 +1,232 @@
+# Doc-sync quality evals — replay harness (Lane A)
+
+This is the "replay" half of the doc-sync quality eval project described in
+[`PLAN.md`](./PLAN.md). It answers one question offline, without touching
+GitHub: *if we run the base-std docs sync's current code and prompts against
+a real historical source change, what does it produce?*
+
+It does **not** score anything — that's Lane B's graders
+(`scripts/doc-evals/graders/**`, not yet built in this worktree). This lane
+only builds frozen test cases and reruns the sync against them.
+
+## What's here
+
+| Path | What |
+|---|---|
+| `cases/<id>.json` | Frozen replay cases — see the "Case file" contract in `PLAN.md`. |
+| `build-cases.mjs` | Builds `cases/*.json` from the hand-curated seed list inside it. |
+| `replay/run.mjs` | CLI: replays one or more cases through a candidate sync checkout. |
+| `replay/worktree.mjs` | Throwaway git worktree lifecycle (create, overlay, remove). |
+| `replay/log-parser.mjs` | Parses touched/rejected/unchanged pages out of the sync's log. |
+| `runs/` | Replay output (gitignored — see the "Replay output" contract in `PLAN.md`). |
+| `__tests__/harness-*.test.mjs` | Offline unit tests (`node --test`). |
+
+## Building cases
+
+```sh
+node scripts/doc-evals/build-cases.mjs            # rebuild all 9 seed cases
+node scripts/doc-evals/build-cases.mjs --only 253bb15-transfer-executor,64bd955-inverted-seize-holder
+```
+
+This is idempotent and network-only (GitHub REST API against the public
+`base/docs` and `base/base-std` repos — no LLM calls). It authenticates with
+`gh auth token` when available and falls back to unauthenticated (rate-limited)
+requests otherwise. Never edit a file under `cases/` by hand; fix the seed
+list or the reconstruction logic in `build-cases.mjs` and regenerate.
+
+To add a case: add an entry to the `SEED` array in `build-cases.mjs` (bot PR
+number, source sha, split, heavy flag, and — when a human-merged answer
+exists — the reference commit/PR/pages), then run with `--only <id>`.
+
+## Running a replay
+
+```sh
+node scripts/doc-evals/replay/run.mjs --cases 868d513-seize-integrator-guidance --reps 1
+node scripts/doc-evals/replay/run.mjs --split train --reps 2 --concurrency 3
+node scripts/doc-evals/replay/run.mjs --split all --include-heavy
+```
+
+Flags: `--cases id,id` | `--split train|test|all`, `--include-heavy` (heavy
+cases are excluded by default), `--reps N` (default 1), `--candidate <dir>`
+(defaults to this checkout's `scripts/sync-from-base-std/`; point it at a
+scratch copy to replay a candidate patch), `--run-id X` (defaults to a
+timestamp), `--concurrency N` (default 1).
+
+Per case/rep, this:
+
+1. Checks out `docs_base_commit` detached into a fresh `git worktree` under
+   `os.tmpdir()` — historical docs content.
+2. Overlays the candidate's `scripts/sync-from-base-std/` directory (and
+   symlinks its `scripts/node_modules`) — candidate sync code + prompts.
+3. Runs `node scripts/sync-from-base-std/index.mjs --payload <file>` inside
+   the worktree with `RUNNER_TEMP` set, `GITHUB_OUTPUT` unset, and
+   `SOURCE_REPO_TOKEN` from `gh auth token` when available, under a
+   30-minute timeout (mirrors the real workflow's job timeout).
+4. Parses `touched`/`rejected`/`unchanged` from the captured log (see
+   `replay/log-parser.mjs`'s doc comment for the exact patterns), snapshots
+   before/after content of every touched page, `git diff`s the worktree, and
+   copies `.sync-bench/*.jsonl`.
+5. Always removes the worktree (`finally`, so a thrown error or a timeout
+   still cleans up) and runs `git worktree prune`.
+
+Output lands at `runs/<run-id>/<case-id>/rep-<n>/` with the six files in the
+"Replay output" contract (`meta.json`, `diff.patch`, `before/`, `after/`,
+`bench.jsonl`, `sync.log`) — see `PLAN.md`. `replayCase(caseDef, opts)` is
+also exported from `replay/run.mjs` for the Phase 2 hillclimb loop to call
+programmatically.
+
+Every worktree lives under `os.tmpdir()`, created and removed only through
+`git worktree` subcommands run with `cwd: REPO_ROOT` — this checkout, not any
+other lane's worktree — so a bug elsewhere can't point cleanup at the wrong
+directory. `scripts/doc-evals/__tests__/harness-worktree.test.mjs` covers the
+create/remove lifecycle, including that cleanup still runs when the work
+between create and remove throws.
+
+## Cost notes
+
+The sync makes one LLM call per touched page. `bench.jsonl` (copied from the
+candidate's `.sync-bench/`) has the per-call token counts; sum
+`inputTokens`/`outputTokens` across a run's `bench.jsonl` files for its total.
+A single small case (1–2 touched pages) costs a few cents; a heavy case
+(`be6d045-b20-restructure`, 63 pages in scope) is the one case worth budgeting
+for — run it deliberately with `--include-heavy`, not by default. `run.mjs`
+excludes `heavy: true` cases unless asked, for exactly this reason.
+
+## Decisions on ambiguities
+
+The plan (`PLAN.md`, "Lane A") leaves a few things to judgment. Recorded here
+so a reviewer can push back on any of them:
+
+- **Reference PR attribution.** `#1939` and `#2025` are human-written PRs
+  that each squash several base-std source commits into one docs PR. There
+  is no mechanical way to attribute which page in a squashed PR answers which
+  single upstream commit, so `reference.pages` for `be6d045-b20-restructure`,
+  `253bb15-transfer-executor`, `91427ab-policy-not-invert`, and
+  `1505323-reject-self-recipient` is hand-curated in the `SEED` list, cross-checked
+  against each reference PR's own commit message (`#1939`'s names `be6d045`
+  directly; `#2025`'s describes the three Denim changelog entries it adds —
+  token receiver, transfer executor enforcement, NOT/invert policies — which
+  map 1:1 to `1505323`/`253bb15`/`91427ab`'s own topics) and against the
+  actual page list of the reference commit,
+  verified by diffing `git show --name-only <reference commit>` against the
+  claimed `reference.pages` for every one of the four cases. Three pages
+  (`docs/AGENTS.md`, `docs/llms.txt`, `docs/llms-full.txt`) are excluded
+  everywhere as generated indices, per the schema note in `PLAN.md`.
+- **`review_findings` includes general PR conversation comments, not just
+  review bodies and inline diff comments.** The plan's contract text says
+  "pull reviews, review comments, and inline comments"; GitHub's own API
+  separates those from a PR's general conversation thread
+  (`issues/{pr}/comments`), which is where a *closing* comment lands — e.g.
+  `#1928`'s "Closing unmerged... 13 'source file removed' banners" comment,
+  which is the plan's own motivating example for the housekeeping failure
+  mode (see `PLAN.md`'s "Why" table). Excluding it would leave that case's
+  `review_findings` empty despite being the flagship failure example, so
+  `fetchReviewFindings()` pulls all three (reviews, review comments, issue
+  comments), still skipping bot accounts (`mintlify[bot]`, `cb-heimdall`).
+- **`payload.diff`'s cap is the 12 MiB artifact cap, not the 65536-byte inline
+  cap.** The real workflow enforces two different diff caps depending on
+  delivery path: `MAX_INLINE_DIFF_BYTES` (65536) for a diff embedded directly
+  in the dispatch payload, and the much larger `MAX_DIFF_BYTES` (12 MiB, in
+  the "Fetch diff artifact from source repo" step) for a diff too big to
+  inline, uploaded as a workflow artifact and spliced in server-side. Four of
+  the nine seed diffs (`be6d045` 178 KB, `04d645a` 114 KB, `db537f3` 71 KB,
+  `253bb15` 70 KB) exceed the inline cap — since these bot PRs did run, their
+  real dispatches must have used artifact delivery, so the artifact cap is
+  the correct one to mirror when reconstructing what `index.mjs` actually
+  saw. `build-cases.mjs` applies only `MAX_DIFF_BYTES`, per the plan's own
+  citation of the "Fetch diff artifact" step (not "Validate payload schema",
+  where the inline cap lives).
+- **`changed_paths`/`removed_paths` are reconstructed from the GitHub commit
+  API, not from the original dispatcher payload.** The plan says to derive
+  both this way; the real workflow only ever *re-derives* `removed_paths`
+  this way (`removed_paths` is always overwritten from the trusted commit
+  API — see the "Derive trusted removed paths" step) and takes `changed_paths`
+  from the dispatcher's own client_payload for a code-change dispatch, whose
+  original content isn't recoverable after the fact. Recomputing both from
+  the commit API is the best available faithful substitute, and both caps the
+  workflow enforces (200 entries, 512 bytes/entry) are applied — as a hard
+  failure (matching the workflow's `workflow_fail`, not a silent truncation),
+  since a source commit that violates them would never have reached
+  `index.mjs` for real. None of the nine seed cases are anywhere near either
+  cap.
+- **`pr_title`/`pr_body`/`pr_number` are approximated, not recovered.** The
+  original *dispatcher* run (in base-std's own Actions) knew these; after the
+  fact they're derived from the base-std commit's own message (`pr_title`
+  from the commit headline, `pr_number` parsed from its trailing `(#N)`) and
+  that PR's body fetched from base-std. Close enough for the sync's prompts
+  (which only reference them for context), but not byte-identical to what the
+  original dispatch actually sent.
+- **A nav-file write counts as a touched page.** When the sync creates a new
+  changelog entry page, it also writes an `[nav] added ...` line adding that
+  page to `docs/docs.json`'s sidebar — a second, real file write that the
+  original log-parsing missed (it only matched `[write]`/`[create]` lines for
+  the *page itself*). `replay/log-parser.mjs` now also matches `[nav]` lines,
+  so `docs/docs.json` correctly shows up in `meta.json`'s `touched` array and
+  gets a `before/`/`after/` snapshot — which matters because every reference
+  case's `scope.in` includes `docs/docs.json`.
+
+## Live smoke test
+
+Acceptance requires one live replay of a small (`heavy: false`) case,
+producing every contract file, with reported token usage. Use the smallest
+seed diff:
+
+```sh
+npm ci --prefix scripts --no-audit --no-fund   # once, if not already installed
+node scripts/doc-evals/replay/run.mjs --cases 868d513-seize-integrator-guidance --reps 1
+```
+
+Requires `LLM_GATEWAY_API_KEY` in the environment. See the top-level report
+for this lane's actual smoke output and token counts.
+
+## Hillclimb
+
+`hillclimb/run.mjs` proposes one root-cause change at a time to the sync's
+prompts and/or route table, replays and grades it on the train and test cases,
+and keeps it only when the scores say it helped. See `PLAN.md` "Phase 2".
+
+```sh
+node scripts/doc-evals/hillclimb/run.mjs --rounds 5 --reps 2 --max-usd 25
+node scripts/doc-evals/hillclimb/run.mjs --rounds 1 --reps 1 --max-usd 8 \
+  --cases-train 64bd955-inverted-seize-holder --cases-test 868d513-seize-integrator-guidance \
+  --baseline-run scripts/doc-evals/runs/<existing-replay-run>
+```
+
+Flags: `--rounds`, `--reps`, `--max-usd`, `--surface prompts|route-table|both`,
+`--no-judge`, `--baseline-run <dir>`, `--cases-train ids`, `--cases-test ids`,
+`--concurrency`. Default sets are the case files' train/test splits minus
+`heavy` and `legacy_layout`. `HILLCLIMB_MODEL` picks the proposer (default
+`claude-opus-4-6`); `HILLCLIMB_PRICES` overrides the price table in
+`hillclimb/budget.mjs`.
+
+- **Baseline.** `--baseline-run` reuses the rep dirs of an existing replay run
+  (graded or not; `grade.json` is written in place if missing). Reps or cases
+  missing from it are replayed. Without the flag the baseline is replayed and
+  graded. Existing `grade.json` files are used as-is.
+- **Noise** = max over splits of (mean over the split's cases of the sample
+  stdev of that case's `overall` across reps), measured on the baseline. With
+  `--reps 1` it is 0 and the report says so.
+- **Proposer** sees only train failures (check details, judge reasons, review
+  findings, capped diffs), the current files for `--surface`, the failure
+  taxonomy and the owner decisions. Test-case content never enters a prompt.
+  It returns `{rationale, root_cause, files:[{path, content}]}` (whole-file
+  replacement); only `llm/prompts.mjs` and `route-table.json` are accepted, the
+  route table must parse, and `prompts.mjs` must keep its exports.
+- **Verification.** The patch is applied to a scratch tree under the run dir
+  (never the real `scripts/sync-from-base-std/`), then the sync's own unit
+  tests run there. Tests already failing before any patch are tolerated; any
+  new failure rejects the patch.
+- **Decision.** Keep iff trainΔ > noise and testΔ > 0. Train up but test flat
+  or down is reverted as "possible overfit". A rep with grading errors is
+  re-graded once; if errors remain the round is skipped (neither keep nor
+  revert). Two consecutive non-keeps trigger one reflection call and stop.
+- **Judge in decisions** only when `calibration/labels.json` exists and clears
+  80% agreement; otherwise decisions use code + pairwise (the judge still runs
+  so the proposer sees its reasons, unless `--no-judge`).
+- **Budget.** Sync cost from each rep's `bench.jsonl`, judge/pairwise from the
+  grade summary tokens, proposer from its usage, priced by a conservative table
+  (`budget.mjs`). A round is not started if spend plus the projected round cost
+  (1.2 x (one full evaluation + proposer)) would exceed `--max-usd`.
+- **Output** in `runs/hillclimb-<ts>/`: `report.md`, `round-<n>.patch` per kept
+  round (`git diff --no-index`), `final-candidate/sync-from-base-std/`, plus
+  the scratch candidates and replays. No git commits are made.

scripts/doc-evals/__tests__/fixtures/calibration-runs/case-x/rep-1/grade.json

@@ -0,0 +1,11 @@
+{
+  "caseId": "case-x",
+  "rep": 1,
+  "checks": [
+    { "id": "scope.precision", "layer": "code", "page": null, "pass": true, "score": 1, "detail": "ok" },
+    { "id": "judge.J1", "layer": "judge", "page": "docs/a.mdx", "pass": true, "score": 1, "detail": "reflected" },
+    { "id": "judge.J2", "layer": "judge", "page": "docs/a.mdx", "pass": false, "score": 0, "detail": "unsupported claim" },
+    { "id": "judge.J1", "layer": "judge", "page": "docs/b.mdx", "pass": true, "score": 1, "detail": "reflected" }
+  ],
+  "summary": { "code": 1, "judge": 0.67, "pairwise": null, "overall": 0.8, "cost": { "inputTokens": 100, "outputTokens": 40 } }
+}

scripts/doc-evals/__tests__/fixtures/calibration-runs/case-y/rep-1/grade.json

@@ -0,0 +1,9 @@
+{
+  "caseId": "case-y",
+  "rep": 1,
+  "checks": [
+    { "id": "judge.J1", "layer": "judge", "page": "docs/c.mdx", "pass": true, "score": 1, "detail": "reflected" },
+    { "id": "judge.J2", "layer": "judge", "page": "docs/c.mdx", "pass": true, "score": 1, "detail": "grounded" }
+  ],
+  "summary": { "code": 1, "judge": 1, "pairwise": null, "overall": 0.9, "cost": { "inputTokens": 50, "outputTokens": 20 } }
+}

scripts/doc-evals/__tests__/fixtures/cases/case-a.json

@@ -0,0 +1,22 @@
+{
+  "id": "case-a",
+  "source_repo": "base/base-std",
+  "source_sha": "0000000000000000000000000000000000000000",
+  "bot_pr": null,
+  "docs_base_commit": "0000000000000000000000000000000000000000",
+  "payload": {
+    "kind": "code-change",
+    "sha": "0000000000000000000000000000000000000000",
+    "diff": "diff --git a/src/interfaces/IB20.sol b/src/interfaces/IB20.sol\n+function transfer(address,uint256) external;\n"
+  },
+  "reference": null,
+  "scope": {
+    "in": ["docs/base-chain/specs/reference/a.mdx"],
+    "out": [],
+    "label_source": "drafted"
+  },
+  "review_findings": [],
+  "split": "train",
+  "heavy": false,
+  "notes": "Fixture case for offline graders-grade tests. Not a real case; do not promote."
+}

scripts/doc-evals/__tests__/fixtures/commit-renames.json

@@ -0,0 +1,13 @@
+{
+  "sha": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+  "commit": {
+    "message": "feat(BOP-495): ERC-8056 interface-review follow-ups (renames + Conversion extension) (#192)"
+  },
+  "files": [
+    { "filename": "docs/B20/Asset.md", "status": "modified" },
+    { "filename": "src/interfaces/IERC8056.sol", "status": "renamed", "previous_filename": "src/interfaces/IScaledUIAmount.sol" },
+    { "filename": "test/unit/B20Asset/multiplier/toUIAmount.t.sol", "status": "added" },
+    { "filename": "test/unit/B20Asset/multiplier/toScaledBalance.t.sol", "status": "removed" },
+    { "filename": "test/unit/B20Asset/multiplier/toScaledBalance.t.sol", "status": "removed" }
+  ]
+}

scripts/doc-evals/__tests__/fixtures/commit-small.json

@@ -0,0 +1,10 @@
+{
+  "sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+  "commit": {
+    "message": "docs(changelog): clarify seize integrator guidance (#205)"
+  },
+  "files": [
+    { "filename": "changelog/02_Cobalt_B20_seize.md", "status": "modified" },
+    { "filename": "src/interfaces/IB20.sol", "status": "modified" }
+  ]
+}