From fcd263a7a4c5d1434965912e4c926d6baf62fe7c Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Thu, 24 Sep 2026 12:55:26 +0800 Subject: [PATCH 1/9] llm-judge: let codex use the account's configured model The codex runner pinned gpt-5.3-codex-spark, which a ChatGPT-account Codex login refuses, so every codex judge call exited 1. Drop -m so codex runs the model from ~/.codex/config.toml, like the investigate runner already does. Co-Authored-By: Claude Opus 5.5 (1M context) Change-Id: I6265302008c925a00186cf22a7b7e5f74426494b --- engine/hooks/llm-judge/README.md | 5 ++++- engine/hooks/llm-judge/judge.py | 2 +- engine/hooks/llm-judge/tests/test_judge.py | 7 +++++++ engine/hooks/wrong-check-reflect/README.md | 2 +- 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/engine/hooks/llm-judge/README.md b/engine/hooks/llm-judge/README.md index fe93a2e75..e4be571a1 100644 --- a/engine/hooks/llm-judge/README.md +++ b/engine/hooks/llm-judge/README.md @@ -76,7 +76,10 @@ the dictionary's `on_hit` text. `ask(prompt)` tries these in order and stops at the first one that answers: -1. **codex**: `codex exec --skip-git-repo-check -m gpt-5.3-codex-spark --sandbox read-only -c notify=[] PROMPT` +1. **codex**: `codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT` + (no `-m`: codex runs the model set in `~/.codex/config.toml`, so the judge + uses a model the account can already call; a ChatGPT-account login refuses + API-only models) 2. **claude**: `claude -p --model haiku --settings '{"disableAllHooks": true}' PROMPT` 3. **cursor**: `cursor-agent -p --output-format text PROMPT` diff --git a/engine/hooks/llm-judge/judge.py b/engine/hooks/llm-judge/judge.py index 1a50cc74e..3fb4b1018 100644 --- a/engine/hooks/llm-judge/judge.py +++ b/engine/hooks/llm-judge/judge.py @@ -32,7 +32,7 @@ RUNNERS_ENV = "CATSTACK_LLM_JUDGE_RUNNERS" STATE_ENV = "CATSTACK_LLM_JUDGE_STATE_DIR" DEFAULT_RUNNERS = ( - ("codex", ["codex", "exec", "--skip-git-repo-check", "-m", "gpt-5.3-codex-spark", "--sandbox", "read-only", "-c", "notify=[]", PROMPT_SLOT]), + ("codex", ["codex", "exec", "--skip-git-repo-check", "--sandbox", "read-only", "-c", "notify=[]", PROMPT_SLOT]), ("claude", ["claude", "-p", "--model", "haiku", "--settings", '{"disableAllHooks": true}', PROMPT_SLOT]), ("cursor", ["cursor-agent", "-p", "--output-format", "text", PROMPT_SLOT]), ) diff --git a/engine/hooks/llm-judge/tests/test_judge.py b/engine/hooks/llm-judge/tests/test_judge.py index 5d083d3d8..ae164766f 100644 --- a/engine/hooks/llm-judge/tests/test_judge.py +++ b/engine/hooks/llm-judge/tests/test_judge.py @@ -179,6 +179,13 @@ def test_default_runner_order_is_codex_then_claude_then_cursor(self): os.environ.pop(judge.RUNNERS_ENV) self.assertEqual([name for name, _ in judge.runners()], ["codex", "claude", "cursor"]) + def test_default_codex_runner_uses_the_account_model_not_a_pinned_one(self): + with patch.dict(os.environ): + os.environ.pop(judge.RUNNERS_ENV) + codex_argv = dict(judge.runners())["codex"] + self.assertNotIn("-m", codex_argv) + self.assertNotIn("--model", codex_argv) + def test_investigate_runner_argv_is_read_only_and_excludes_cursor(self): os.environ.pop(judge.RUNNERS_ENV) self.assertEqual( diff --git a/engine/hooks/wrong-check-reflect/README.md b/engine/hooks/wrong-check-reflect/README.md index 7d8e92b9d..5fa53e40a 100644 --- a/engine/hooks/wrong-check-reflect/README.md +++ b/engine/hooks/wrong-check-reflect/README.md @@ -56,7 +56,7 @@ window left the `/reflect` typed beside it outside. Inside a judge run (`CATSTAC refuses the job. The model call runs in a detached background process, so the reply is never -held up. Runners are tried in `llm-judge` order: `codex` (gpt-5.3-codex-spark), +held up. Runners are tried in `llm-judge` order: `codex` (the account's configured model), then `claude` (haiku, hooks off), then `cursor-agent`, first answer wins. The verdict reports one turn later. On the next prompt the `llm-judge` inbox From 8b07f68731d4197bba5e1877f09663cce2935320 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Thu, 24 Sep 2026 15:59:48 +0800 Subject: [PATCH 2/9] llm-judge: an unchecked verdict says so, names its fail direction, and reaches the user When no judge runner answered, the inbox printed 'could not judge' as agent context only, which read like a note and could be dropped. The line now starts 'llm-judge UNCHECKED', says the reply is unchecked, not clean, says the check fails open, and asks the agent to tell the user. On Claude the hook also sets systemMessage, which the user sees directly. Co-Authored-By: Claude Opus 5.5 (1M context) Change-Id: Ia54236dbfbbbb81a0ebb4368b251186d4269f0a8 --- engine/hooks/llm-judge/README.md | 19 +++++-- .../hooks/llm-judge/claude_post_tool_use.py | 11 ++-- .../hooks/llm-judge/claude_prompt_submit.py | 11 ++-- engine/hooks/llm-judge/inbox.py | 35 ++++++++++--- engine/hooks/llm-judge/tests/test_inbox.py | 50 ++++++++++++++++--- .../llm-judge/tests/test_post_tool_use.py | 12 +++++ 6 files changed, 115 insertions(+), 23 deletions(-) diff --git a/engine/hooks/llm-judge/README.md b/engine/hooks/llm-judge/README.md index e4be571a1..136034b87 100644 --- a/engine/hooks/llm-judge/README.md +++ b/engine/hooks/llm-judge/README.md @@ -180,14 +180,25 @@ one into a line of text: - **hit**: the job's `on_hit` text, word for word, followed by a space and the answer's `report` string when the answer has a non-blank one, clipped to 600 characters. -- **unchecked**: `llm-judge: could not judge the last reply: ` then - `: ` for each try, joined by `; `. If there were no tries - (the judge broke, or the verdict file was unreadable), the verdict's own - `reason` is used instead. +- **unchecked**: `llm-judge UNCHECKED: could not judge the last + reply, so that reply is unchecked, not clean.` It then says the check fails + open, tells the agent to tell the user the check did not run, and ends with + `Tried: ` and `: ` for each try, joined by `; `. If there + were no tries (the judge broke, or the verdict file was unreadable), the + verdict's own `reason` is used instead. - **clean**: nothing. Each verdict is delivered once. Draining deletes it. +An unchecked verdict fails open: the judge runs after the reply is sent, so it +never holds that reply up, and a judge that cannot run cannot hold it up +either. What it must not do is read as clean. So on Claude, when any drained +verdict is unchecked, the hook also sets `systemMessage` to +`llm-judge: check(s) did not run and failed open, so the replies they cover +are unchecked, not clean: `. That line is shown to the user directly, so +it reaches them even if the agent drops the context line. The verdict event is +also written with action `unchecked`. + One small script per harness calls it: | Harness | Script | Event | How the text reaches the agent | diff --git a/engine/hooks/llm-judge/claude_post_tool_use.py b/engine/hooks/llm-judge/claude_post_tool_use.py index 0cd72284d..36b51c6ae 100644 --- a/engine/hooks/llm-judge/claude_post_tool_use.py +++ b/engine/hooks/llm-judge/claude_post_tool_use.py @@ -20,12 +20,17 @@ def main() -> None: print(inbox.NO_TRANSCRIPT.format(harness=HARNESS), file=sys.stderr) return try: - found = inbox.messages(transcript) + found, unchecked = inbox.report(transcript) except Exception as exc: print(f"llm-judge: {HARNESS} could not drain verdicts for {transcript}: {type(exc).__name__}: {exc}", file=sys.stderr) return - if found: - print(json.dumps({"hookSpecificOutput": {"hookEventName": "PostToolUse", "additionalContext": "\n\n".join(found)}})) + if not found: + return + output = {"hookSpecificOutput": {"hookEventName": "PostToolUse", "additionalContext": "\n\n".join(found)}} + notice = inbox.user_notice(unchecked) + if notice: + output["systemMessage"] = notice + print(json.dumps(output)) if __name__ == "__main__": diff --git a/engine/hooks/llm-judge/claude_prompt_submit.py b/engine/hooks/llm-judge/claude_prompt_submit.py index 2beeb3568..9c94a1740 100644 --- a/engine/hooks/llm-judge/claude_prompt_submit.py +++ b/engine/hooks/llm-judge/claude_prompt_submit.py @@ -18,12 +18,17 @@ def main() -> None: print(inbox.NO_TRANSCRIPT.format(harness="Claude UserPromptSubmit"), file=sys.stderr) return try: - found = inbox.messages(transcript) + found, unchecked = inbox.report(transcript) except Exception as exc: print(f"llm-judge: could not drain verdicts for {transcript}: {type(exc).__name__}: {exc}", file=sys.stderr) return - if found: - print(json.dumps({"hookSpecificOutput": {"hookEventName": "UserPromptSubmit", "additionalContext": "\n\n".join(found)}})) + if not found: + return + output = {"hookSpecificOutput": {"hookEventName": "UserPromptSubmit", "additionalContext": "\n\n".join(found)}} + notice = inbox.user_notice(unchecked) + if notice: + output["systemMessage"] = notice + print(json.dumps(output)) if __name__ == "__main__": diff --git a/engine/hooks/llm-judge/inbox.py b/engine/hooks/llm-judge/inbox.py index 262739830..ea18ac3c9 100644 --- a/engine/hooks/llm-judge/inbox.py +++ b/engine/hooks/llm-judge/inbox.py @@ -159,15 +159,33 @@ def enqueue_judge(job_fields: dict, payload: dict) -> str | None: return judge.enqueue(job) +UNCHECKED = ( + "llm-judge UNCHECKED: {hook} could not judge the last reply, so that reply is unchecked, " + "not clean. This check fails open: the reply was already sent and was not held. " + "Tell the user this check did not run before relying on that reply. Tried: {tried}" +) +USER_NOTICE = ( + "llm-judge: {count} check(s) did not run and failed open, so the replies they cover " + "are unchecked, not clean: {hooks}" +) + + def unchecked_message(item: dict) -> str: hook = item.get("hook") or "unknown hook" attempts = item.get("attempts") or [] tried = "; ".join(f"{a.get('runner')}: {a.get('reason')}" for a in attempts if isinstance(a, dict)) - return f"llm-judge: {hook} could not judge the last reply: {tried or item.get('reason') or 'no reason recorded'}" + return UNCHECKED.format(hook=hook, tried=tried or item.get("reason") or "no reason recorded") -def messages(transcript: str) -> list[str]: +def user_notice(unchecked_hooks: list[str]) -> str | None: + if not unchecked_hooks: + return None + return USER_NOTICE.format(count=len(unchecked_hooks), hooks=", ".join(sorted(set(unchecked_hooks)))) + + +def report(transcript: str) -> tuple[list[str], list[str]]: out = [] + unchecked = [] drained = [] for path in [transcript, *subagent_transcripts(transcript)]: drained.extend(judge.drain(path)) @@ -181,10 +199,15 @@ def messages(transcript: str) -> list[str]: text = f"llm-judge: {item.get('hook') or 'unknown hook'} flagged the last reply: {item.get('reason')}" answer = item.get("answer") if isinstance(answer, dict): - report = answer.get("report") - if isinstance(report, str) and report.strip(): - text = f"{text} {report.strip()[:REPORT_LIMIT]}" + detail = answer.get("report") + if isinstance(detail, str) and detail.strip(): + text = f"{text} {detail.strip()[:REPORT_LIMIT]}" out.append(text) continue out.append(unchecked_message(item)) - return out + unchecked.append(str(item.get("hook") or "unknown hook")) + return out, unchecked + + +def messages(transcript: str) -> list[str]: + return report(transcript)[0] diff --git a/engine/hooks/llm-judge/tests/test_inbox.py b/engine/hooks/llm-judge/tests/test_inbox.py index 8787882f7..5b67fd8be 100644 --- a/engine/hooks/llm-judge/tests/test_inbox.py +++ b/engine/hooks/llm-judge/tests/test_inbox.py @@ -236,9 +236,30 @@ def test_clean_with_report_yields_nothing(self): def test_unchecked_yields_one_reason_per_runner(self): self.assertEqual(self.seed(MISSING, CRASHES)["outcome"], "unchecked") + found = inbox.messages(self.transcript) + self.assertEqual(len(found), 1) + self.assertTrue(found[0].endswith("Tried: ghost: not installed; crashes: exit 3: model quota exhausted"), found) + + def test_unchecked_says_unchecked_not_clean_and_names_the_fail_direction(self): + self.seed(MISSING, CRASHES) + found = inbox.messages(self.transcript) + self.assertTrue(found[0].startswith("llm-judge UNCHECKED: demo-hook could not judge the last reply"), found) + self.assertIn("unchecked, not clean", found[0]) + self.assertIn("fails open", found[0]) + self.assertIn("Tell the user this check did not run", found[0]) + + def test_report_lists_unchecked_hooks_apart_from_hits(self): + self.seed(ANSWERS_TRUE, job_id="a") + self.seed(MISSING, job_id="b") + found, unchecked = inbox.report(self.transcript) + self.assertEqual(len(found), 2) + self.assertEqual(unchecked, ["demo-hook"]) + + def test_user_notice_is_none_when_nothing_was_unchecked(self): + self.assertIsNone(inbox.user_notice([])) self.assertEqual( - inbox.messages(self.transcript), - ["llm-judge: demo-hook could not judge the last reply: ghost: not installed; crashes: exit 3: model quota exhausted"], + inbox.user_notice(["b-hook", "a-hook", "b-hook"]), + "llm-judge: 3 check(s) did not run and failed open, so the replies they cover are unchecked, not clean: a-hook, b-hook", ) def test_clean_yields_nothing(self): @@ -256,7 +277,8 @@ def test_unreadable_verdict_file_is_unchecked_not_clean(self): handle.write("{not json") found = inbox.messages(self.transcript) self.assertEqual(len(found), 1) - self.assertTrue(found[0].startswith("llm-judge: unknown hook could not judge the last reply: unreadable verdict file"), found) + self.assertTrue(found[0].startswith("llm-judge UNCHECKED: unknown hook could not judge the last reply"), found) + self.assertIn("Tried: unreadable verdict file", found[0]) def test_verdicts_for_another_transcript_are_not_delivered(self): self.seed(ANSWERS_TRUE) @@ -270,12 +292,26 @@ def test_hit_is_delivered_as_additional_context_once(self): self.seed(MISSING, job_id="b") out, err = self.run_claude(self.claude_payload()) self.assertEqual(err, "") - self.assertEqual(json.loads(out), {"hookSpecificOutput": { - "hookEventName": "UserPromptSubmit", - "additionalContext": ON_HIT + "\n\nllm-judge: demo-hook could not judge the last reply: ghost: not installed", - }}) + data = json.loads(out) + self.assertEqual(data["hookSpecificOutput"]["hookEventName"], "UserPromptSubmit") + context = data["hookSpecificOutput"]["additionalContext"] + self.assertTrue(context.startswith(ON_HIT + "\n\nllm-judge UNCHECKED: demo-hook"), context) + self.assertTrue(context.endswith("Tried: ghost: not installed"), context) self.assertEqual(self.run_claude(self.claude_payload()), ("", "")) + def test_unchecked_is_also_shown_to_the_user_as_a_system_message(self): + self.seed(MISSING) + out, err = self.run_claude(self.claude_payload()) + self.assertEqual(err, "") + data = json.loads(out) + self.assertIn("llm-judge UNCHECKED: demo-hook", data["hookSpecificOutput"]["additionalContext"]) + self.assertEqual(data["systemMessage"], inbox.user_notice(["demo-hook"])) + + def test_hit_alone_adds_no_system_message(self): + self.seed(ANSWERS_TRUE) + data = json.loads(self.run_claude(self.claude_payload())[0]) + self.assertNotIn("systemMessage", data) + def test_clean_prints_nothing(self): self.seed(ANSWERS_FALSE) self.assertEqual(self.run_claude(self.claude_payload()), ("", "")) diff --git a/engine/hooks/llm-judge/tests/test_post_tool_use.py b/engine/hooks/llm-judge/tests/test_post_tool_use.py index f02ed9e7f..fe6a59670 100644 --- a/engine/hooks/llm-judge/tests/test_post_tool_use.py +++ b/engine/hooks/llm-judge/tests/test_post_tool_use.py @@ -93,6 +93,18 @@ def test_malformed_stdin_exits_zero_with_a_stderr_line(self): self.assert_bad_json(self.script, self.harness) + def test_unchecked_verdict_adds_a_system_message_for_the_user(self): + judge.write_json_atomic( + os.path.join(judge.verdict_dir(self.transcript), "job-u.json"), + {"id": "job-u", "hook": "demo-hook", "outcome": "unchecked", "reason": "no runner answered", "attempts": []}, + ) + result = self.run_script(self.script, self.payload()) + self.assert_success(result) + data = json.loads(result.stdout) + self.assertIn("llm-judge UNCHECKED: demo-hook", data["hookSpecificOutput"]["additionalContext"]) + self.assertIn("failed open", data["systemMessage"]) + + class TestCodexPostToolUse(PostToolUseTestCase): script = "codex_post_tool_use.py" harness = "Codex PostToolUse" From 38cdc20907578f91fac2a52455fb8393092236cc Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Tue, 22 Sep 2026 01:10:13 -0700 Subject: [PATCH 3/9] cat-mode: show the escape-hatch tag in the form the gate accepts The sentence that tells the reader to use the tag showed it as a bare token naming no blocker. That is the shape diu-stop rejects, so quoting cat-mode's own rule tripped the gate enforcing it. It now shows the same template engine/CLAUDE.core.md already uses. tests/test_cat_mode.py gains the structural catch: no tag anywhere in the skill may name no blocker, and the skill must still show the tag at all, so the fix cannot be "delete the example". Co-Authored-By: Claude Opus 5 (1M context) Change-Id: If47021e83bdf3d8771ccbdb879001f6effc65a3c --- corpus/skills/cat-mode/SKILL.md | 2 +- tests/test_cat_mode.py | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/corpus/skills/cat-mode/SKILL.md b/corpus/skills/cat-mode/SKILL.md index e056db7c2..351c8278d 100644 --- a/corpus/skills/cat-mode/SKILL.md +++ b/corpus/skills/cat-mode/SKILL.md @@ -288,7 +288,7 @@ agent switch, or resubmit is a fix, and none comes before the repro. **A factual or technical claim gets a real repro script, not a history search.** Judging an old comment or a "probably confabulated" suspicion needs an actual attempt under the claimed conditions, not a `git log` sweep. No citation means "never verified," not "false." -**Unhedged root-cause or fix claims about live system behavior need instrument-level proof in the same message, or a `{{CAT-UNVERIFIED}}` tag naming the blocker.** The gate is the claim type, not a hedge word. Invoking `/prove-it` once does not arm it for later claims. Any hedge auto-runs prove-it in the same turn — a hedge is a trigger to verify, never a place to stop. +**Unhedged root-cause or fix claims about live system behavior need instrument-level proof in the same message, or a `{{CAT-UNVERIFIED: -- cannot verify: }}` tag naming the blocker.** The gate is the claim type, not a hedge word. Invoking `/prove-it` once does not arm it for later claims. Any hedge auto-runs prove-it in the same turn — a hedge is a trigger to verify, never a place to stop. Outputs carry failures explicitly (a status column, an error row), never dropped — [[principle-explicit-errors]]. diff --git a/tests/test_cat_mode.py b/tests/test_cat_mode.py index 6a040ca5b..9e62cf391 100644 --- a/tests/test_cat_mode.py +++ b/tests/test_cat_mode.py @@ -1691,5 +1691,34 @@ def test_hook_blocks_the_same_estimate_once_the_job_already_exited(self): self.assertIsNotNone(detect.decide_stop_from_lines(ESTIMATE_REPLY, lines)) +class TestEscapeHatchTemplateIsWellFormed(unittest.TestCase): + """Every tag this skill shows a reader must be one the gate accepts. + + cat-mode carried a bare `{{CAT-UNVERIFIED}}` in the sentence that tells + the reader to use the tag, so quoting the rule tripped the gate that + enforces it. The template has to name a blocker, the same one + engine/CLAUDE.core.md already shows. + """ + + def markers(self): + import importlib.util as util + path = os.path.join(REPO_ROOT, "engine", "hooks", "_markers", "markers.py") + spec = util.spec_from_file_location("markers_for_test", path) + module = util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + def test_no_tag_in_the_skill_names_no_blocker(self): + with open(SKILL_PATH, encoding="utf-8") as handle: + text = handle.read() + malformed = self.markers().malformed_tags(text) + self.assertEqual(malformed, [], f"tags naming no blocker: {malformed}") + + def test_the_skill_still_shows_the_tag_at_all(self): + with open(SKILL_PATH, encoding="utf-8") as handle: + text = handle.read() + self.assertTrue(self.markers().well_formed_tags(text)) + + if __name__ == "__main__": unittest.main() From 9c71daf8d8ab461f214b3aa12834635d8d1990ea Mon Sep 17 00:00:00 2001 From: CI Bot Date: Wed, 23 Sep 2026 16:59:53 +0000 Subject: [PATCH 4/9] cat-mode: the tag scan reads the reference too, not just SKILL.md The bare `{{CAT-UNVERIFIED}}` was fixed in SKILL.md, but the same sentence lives twice on purpose: SKILL.md keeps the one-line form and references/ verify.md holds the full text the evidence gate leans on. verify.md still showed the tag naming no blocker, so a reader sent to the reference copied the shape diu-stop rejects. verify.md now shows the same template. The structural catch walks every markdown file in the skill instead of SKILL.md alone, and asserts the two files that carry the rule are in the swept list, so an empty or mis-rooted scan cannot report clean. The "still shows the tag at all" assertion now covers both copies, so the fix cannot be deleting the example in either. Fail-before / pass-after on tests.test_cat_mode.TestEscapeHatchTemplateIsWellFormed: the extended scan fails on verify.md's `['{{CAT-UNVERIFIED}}']` before the edit, and the full 91-test tests/test_cat_mode.py suite passes after. Co-Authored-By: Claude Opus 5 (1M context) --- corpus/skills/cat-mode/references/verify.md | 6 ++- tests/test_cat_mode.py | 53 +++++++++++++++++---- 2 files changed, 47 insertions(+), 12 deletions(-) diff --git a/corpus/skills/cat-mode/references/verify.md b/corpus/skills/cat-mode/references/verify.md index 8f384542b..09ab952a2 100644 --- a/corpus/skills/cat-mode/references/verify.md +++ b/corpus/skills/cat-mode/references/verify.md @@ -67,8 +67,10 @@ place immediately, not left in chat until asked again. ## Unhedged causal claims about live system behavior Unhedged root-cause or fix claims about live system behavior need -instrument-level proof in the same message, or a `{{CAT-UNVERIFIED}}` tag naming the blocker. The gate is the -claim type ("this is why it's slow," "this is the bug"), not a hedge word. +instrument-level proof in the same message, or a +`{{CAT-UNVERIFIED: -- cannot verify: }}` tag naming the +blocker. The gate is the claim type ("this is why it's slow," "this is the +bug"), not a hedge word. Log-reading and code-reading aren't enough: attach with `strace`/a debugger, or query live state (raw SQLite `PRAGMA`). Take a second sample before calling a hang. Invoking `/prove-it` once does not arm it for later claims — each new diff --git a/tests/test_cat_mode.py b/tests/test_cat_mode.py index 9e62cf391..283ac35ee 100644 --- a/tests/test_cat_mode.py +++ b/tests/test_cat_mode.py @@ -1691,6 +1691,22 @@ def test_hook_blocks_the_same_estimate_once_the_job_already_exited(self): self.assertIsNotNone(detect.decide_stop_from_lines(ESTIMATE_REPLY, lines)) +VERIFY_REF = os.path.join(REFERENCE_DIR, "verify.md") + + +def cat_mode_markdown_paths(): + """Every markdown file a cat-mode reader loads: SKILL.md and all of + references/. Listing the directory rather than naming files keeps a new + reference inside the scan the day it lands.""" + paths = [SKILL_PATH] + paths.extend(sorted( + os.path.join(REFERENCE_DIR, name) + for name in os.listdir(REFERENCE_DIR) + if name.endswith(".md") + )) + return paths + + class TestEscapeHatchTemplateIsWellFormed(unittest.TestCase): """Every tag this skill shows a reader must be one the gate accepts. @@ -1698,6 +1714,12 @@ class TestEscapeHatchTemplateIsWellFormed(unittest.TestCase): the reader to use the tag, so quoting the rule tripped the gate that enforces it. The template has to name a blocker, the same one engine/CLAUDE.core.md already shows. + + The scan covers references/, not just SKILL.md. That sentence is stored + twice on purpose -- SKILL.md keeps the one-line form and verify.md holds + the full text (TestCatModeReferencePackage pins that split) -- so a + SKILL.md-only scan reports clean while the copy a reader is pointed at + still shows the bare tag. """ def markers(self): @@ -1708,16 +1730,27 @@ def markers(self): spec.loader.exec_module(module) return module - def test_no_tag_in_the_skill_names_no_blocker(self): - with open(SKILL_PATH, encoding="utf-8") as handle: - text = handle.read() - malformed = self.markers().malformed_tags(text) - self.assertEqual(malformed, [], f"tags naming no blocker: {malformed}") - - def test_the_skill_still_shows_the_tag_at_all(self): - with open(SKILL_PATH, encoding="utf-8") as handle: - text = handle.read() - self.assertTrue(self.markers().well_formed_tags(text)) + def read(self, path): + with open(path, encoding="utf-8") as handle: + return handle.read() + + def test_no_tag_anywhere_in_the_skill_names_no_blocker(self): + paths = cat_mode_markdown_paths() + # A scan over an empty or mis-rooted list reports clean, which reads + # the same as a pass. Name the two files that carry the rule so the + # sweep cannot silently cover nothing. + self.assertIn(SKILL_PATH, paths) + self.assertIn(VERIFY_REF, paths) + for path in paths: + with self.subTest(path=os.path.relpath(path, REPO_ROOT)): + malformed = self.markers().malformed_tags(self.read(path)) + self.assertEqual(malformed, [], f"tags naming no blocker: {malformed}") + + def test_both_copies_of_the_rule_still_show_the_tag_at_all(self): + """So the fix cannot be "delete the example" in either file.""" + for path in (SKILL_PATH, VERIFY_REF): + with self.subTest(path=os.path.relpath(path, REPO_ROOT)): + self.assertTrue(self.markers().well_formed_tags(self.read(path))) if __name__ == "__main__": From aac5c8be09c842bec9f69e2835552f0da58e170f Mon Sep 17 00:00:00 2001 From: CI Bot Date: Wed, 23 Sep 2026 17:00:07 +0000 Subject: [PATCH 5/9] =?UTF-8?q?invoker:=20wf-1790182339789-300/repair=20?= =?UTF-8?q?=E2=80=94=20Resolve=20bot=20review=20thread=20on=20PR=20#786?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exit code: 0 From 327975eb6c44db8a3c73dbd74d727c816843e926 Mon Sep 17 00:00:00 2001 From: CI Bot Date: Wed, 23 Sep 2026 17:35:54 +0000 Subject: [PATCH 6/9] cat-mode: carry the empty-sweep rationale in a docstring, not a comment The escape-hatch test explained why it pins SKILL.md and verify.md by name in three `#` comment lines. Comments are banned in code here, and CI's no-comments twin (scripts/ci/check_no_new_comments.py, the gate for engine/hooks/no-comments) failed the PR on exactly those three lines. The reason is worth keeping: a scan over an empty or mis-rooted list reports clean and reads the same as a pass, so the two files that carry the rule are asserted into the list before the sweep runs. Moving that text into the method docstring keeps it where a reader and `unittest -v` both see it, and the detector only looks for `#` and `//`. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_cat_mode.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_cat_mode.py b/tests/test_cat_mode.py index 283ac35ee..9e4e50141 100644 --- a/tests/test_cat_mode.py +++ b/tests/test_cat_mode.py @@ -1735,10 +1735,10 @@ def read(self, path): return handle.read() def test_no_tag_anywhere_in_the_skill_names_no_blocker(self): + """A scan over an empty or mis-rooted list reports clean, which reads + the same as a pass. Naming the two files that carry the rule keeps the + sweep from silently covering nothing.""" paths = cat_mode_markdown_paths() - # A scan over an empty or mis-rooted list reports clean, which reads - # the same as a pass. Name the two files that carry the rule so the - # sweep cannot silently cover nothing. self.assertIn(SKILL_PATH, paths) self.assertIn(VERIFY_REF, paths) for path in paths: From 2e69d29ffb877729511d352386c2a14a7b37f6e4 Mon Sep 17 00:00:00 2001 From: CI Bot Date: Wed, 23 Sep 2026 17:58:00 +0000 Subject: [PATCH 7/9] =?UTF-8?q?invoker:=20wf-1790184153761-310/repair=20?= =?UTF-8?q?=E2=80=94=20Repair=20PR=20#786=20(failed=20check=20test)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exit code: 0 From d7173e7b0ef652ff74cfd83ce777dd5385961949 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Tue, 22 Sep 2026 01:13:45 -0700 Subject: [PATCH 8/9] principle-subagent-inherits-scope: a brief carries decisions, not just facts and a question The parent's side of the skill offered two slots: facts and the question. A decision the user already made is neither, so it gets filed under the question -- and the "don't prime the answer" rule then rewards re-opening it, because that reads as rigour. Third slot added: decisions travel as constraint sentences marked settled. "Don't prime" is scoped to findings, never decisions; leaving a decision out is dropping a constraint, not staying neutral. The skill says plainly that repeating the user's words is not enough: a delegation can contain the requirement verbatim, pass a containment check, and still lose it by appending one open question beside it. No mechanical catch is claimed, and the fires fixture records why -- the containment check returns PASS on the exact call that drifted. Prior art and its caveats are in the skill: Gotel & Finkelstein on pre-requirements-specification traceability, marked single-source because Crossref's issued field is null and its author rendering differs, with no publisher page read. Co-Authored-By: Claude Opus 5 (1M context) Change-Id: I370384357ee26e01c4b4a2d8ad1d65808c496ab3 --- .../SKILL.md | 27 ++++++++++++++++++- .../tests/fires_example.md | 15 +++++++++++ .../tests/stays_silent_example.md | 8 ++++++ 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/corpus/skills/principle-subagent-inherits-scope/SKILL.md b/corpus/skills/principle-subagent-inherits-scope/SKILL.md index f3ee37c0b..fde4521c3 100644 --- a/corpus/skills/principle-subagent-inherits-scope/SKILL.md +++ b/corpus/skills/principle-subagent-inherits-scope/SKILL.md @@ -62,8 +62,33 @@ the work in a single round trip. until verified — `engine/hooks/agent-relay-attribution` flags the shape. - **State the scope in the prompt, not in your head.** An unstated boundary is not inherited. Name the files, the write authority, and the question. +- **A brief carries three things, not two: facts, the question, and + decisions.** A decision the user already made is neither a fact to weigh + nor a question to answer. Relay it as a constraint sentence that says it is + settled — "the toggle gates the re-injection, not the emission; that is + decided, do not re-open it." Filed under the question, it reads as + something to work out, and a subagent that re-opens it looks rigorous while + discarding the only part the user owned outright. Repeating the user's + words is not enough on its own: a brief can carry the requirement verbatim + and still lose it by appending one open question beside it. - **Don't prime the answer.** Hand over the facts and the question. A - subagent told what you expect finds roughly that. + subagent told what you expect finds roughly that. This governs findings, + never decisions. Leaving out a decision the user already made is not + neutrality — it is dropping a constraint, and the anti-priming rule then + rewards re-opening it. + +Prior art for the third slot: Orlena Gotel and Anthony Finkelstein, "An +analysis of the requirements traceability problem", Proc. IEEE International +Conference on Requirements Engineering, https://doi.org/10.1109/ICRE.1994.292398 +— pre-requirements-specification traceability exists so a requirement keeps +its link to the stakeholder who set it; without that link it gets +renegotiated by people who do not own it. Read this citation as +single-source: Crossref's `issued` field for the record is null, so the 1994 +date is inferred from the DOI string and the conference rather than confirmed +by metadata, and Crossref renders the second author as "C.W. Finkelstein" +while the paper is normally cited as Anthony Finkelstein. IEEE Xplore +returned an empty body and ACM DL returned 403, so no publisher page was +read. ## Related diff --git a/corpus/skills/principle-subagent-inherits-scope/tests/fires_example.md b/corpus/skills/principle-subagent-inherits-scope/tests/fires_example.md index 0414e076e..e1c0c9972 100644 --- a/corpus/skills/principle-subagent-inherits-scope/tests/fires_example.md +++ b/corpus/skills/principle-subagent-inherits-scope/tests/fires_example.md @@ -23,3 +23,18 @@ and conventions are not concurrency control, so a read-only brief is not filesystem isolation. A subagent told to write files runs in its own worktree even when the rest of the prompt defaults to read-only, and the parent that omitted one has not stated the boundary at all. + +A third shape, the one the decisions slot exists for. The user has already +decided which stage a new toggle gates. The parent relays that requirement +word for word and then appends "work out what a toggle would actually gate." +A mechanical containment check on the delegation passes — the requirement is +verbatim-contained and every content word is present — and the subagent still +ranks the user's own requirement fourth of six and argues its premise away. +The skill fires here because the decision was relayed as part of the +question instead of as a settled constraint, and because the anti-priming +rule reads re-opening it as rigour. + +No mechanical catch is claimed for that third shape, and the obvious one is +known not to work: the containment check returns PASS on the exact +delegation that drifted, because the words were all there. The gate is the +parent's wording, so this stays an `unchecked` case pinned by prose. diff --git a/corpus/skills/principle-subagent-inherits-scope/tests/stays_silent_example.md b/corpus/skills/principle-subagent-inherits-scope/tests/stays_silent_example.md index 4dcd585df..0dbd60906 100644 --- a/corpus/skills/principle-subagent-inherits-scope/tests/stays_silent_example.md +++ b/corpus/skills/principle-subagent-inherits-scope/tests/stays_silent_example.md @@ -8,3 +8,11 @@ whose authority could exceed the parent's. The principle has no target: its four limits all describe what a delegate may do, and the contradiction contract describes how a delegate reports back. A single agent editing one file on its own behalf is the case this principle is not about. + +A second silent shape, against the decisions slot. A parent hands an explorer +a read-only brief that names the files, the question, and one settled +decision relayed as a constraint the explorer may not re-open. The explorer +reads those files, answers the question, and reports one contradiction with a +`file:line` and the ref it was read at, leaving the decision alone. Nothing +here is a widening: the boundary was stated, the constraint travelled as a +constraint, and the contradiction went back up rather than being acted on. From 9e7ef1adc998e0425e416d5e2d6dc4cfff8ccedf Mon Sep 17 00:00:00 2001 From: CI Bot Date: Thu, 24 Sep 2026 11:28:08 +0000 Subject: [PATCH 9/9] =?UTF-8?q?invoker:=20wf-1790249130384-219/repair=20?= =?UTF-8?q?=E2=80=94=20Repair=20PR=20#923=20(failed=20check=20validate)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exit code: 0