llm-judge: record every judge stage, and report jobs that never reach the person

[EdbertChan]

Summary

Background review checks now record every step: skipped (and why), queued, judged, and delivered.

The old per-run table only saw the Stop hook, which never prints. So "silent" could not tell a broken check from a working one.

A new per-hook summary lines those steps up and flags jobs that were lost on the way. A job queued without a transcript now warns the person right away.

Review Claim

Every background review job leaves one row per step, joined by job id, and the per-hook summary command exits 1 when a job is stuck or never reaches the person.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

A job that cannot be delivered is reported, never silent: a missing transcript prints catstack-hook-error, and a finished verdict with no delivery row counts as a leak. Recording failures print on stderr and do not change what the hook decides.

Slice Rationale

The writer, the three places that call it, and the report that reads the rows are one feature. Any one alone records rows nobody reads, or reads rows nobody writes.

Non-goals

Does not fix the two leaks the report shows (jobs with no transcript, and verdicts from subagent transcripts that are never drained). Does not change any verdict or message.

Test Plan

Test Plan

python3 engine/hooks/_sdk/tests/test_events.py            Ran 8 tests   OK
python3 engine/hooks/llm-judge/tests/test_judge.py        Ran 36 tests  OK
python3 engine/hooks/wrong-check-reflect/tests/test_hooks.py  Ran 29 tests  OK
python3 engine/hooks/_runner/tests/test_report.py         Ran 9 tests   OK

The 7 new stage and report tests fail against the parent commit's code.

Real path, scratch metrics dir, real runner, real Stop and PostToolUse scripts, real transcript:

hook skipped queued finished delivered no_transcript stuck undelivered undelivered_hits
wrong-check-reflect already_prompted=1 2 clean=1,hit=0,unchecked=1 clean=1,hit=0,unchecked=0 1 0 1 0
LEAK: 2 judge job(s) queued with no transcript, stuck, or never delivered after 0s
check exit=1

Revert Plan

Revert Plan

Revert this commit. Stage rows already written have no rule_id, so the rule table keeps ignoring them.

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds metrics and reporting only; hook verdict behavior is unchanged, but new stage writes and --check exit codes could affect monitoring or CI that runs the report CLI.

Overview
Background LLM judge work is now observable end-to-end: skipped, queued, finished, and delivered verdicts are written as metrics events (linked by job id), instead of disappearing into silent Stop-hook paths.

A new write_stage_event helper records lifecycle rows with mode_source: stage. wrong-check-reflect logs every skip with a reason (gate off, already prompted, etc.) and passes harness (claude / codex / cursor) into queued jobs. llm-judge emits judge_queued / judge_finished on enqueue and completion, and stderr-warns when a job is queued without a transcript path.

report.py --judge aggregates per-hook counts and detects leaks (no transcript, stuck past grace, finished but never delivered). --check exits 1 when leaks exist. Stage rows stay out of the existing rule/event table (empty rule_id).

^{Reviewed by Cursor Bugbot for commit 96ff887. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2ca0013. Configure here.}

[EdbertChan]

Mergify repair stopped: GitHub reports merge conflict. The retry cap was reached for current head 2ca0013.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c3fec6fe-2fef-43c9-a261-148c741264e9)

[mergify]

Queued — the merge queue status continues in this comment ↓.

[EdbertChan]

@Mergifyio queue

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-09-24 03:39 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
• ✅ Checks passed · on draft merge queue: checking #818 on main (25b1257), stacked on #821 #896
• ✅ Merged — 2026-09-24 04:06 UTC · at ab34e20cd539c15416a8fc31a47b01f9a0d94845 · squash

This pull request spent 27 minutes 37 seconds in the queue, including 26 minutes 46 seconds running CI.

Required conditions to merge

• check-success = lint
• check-success = test
• check-success = validate