Skip to content

fix(evals): resolve the four remaining eval failures - #1476

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix/remaining-eval-failures
Sep 29, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
potiuk:fix/remaining-eval-failures

Conversation

@potiuk

@potiuk potiuk commented Sep 29, 2026

Copy link
Copy Markdown
Member

Fixes the four eval failures left over from the security-family optimization (#1344).

Summary

  • issue-deduplicate step-0 case-3. With a single argument there is no duplicate, so the expected value is null, and the output spec allows null.
  • release-audit-report step-0 case-3. The blocker assertion also accepts "Planning issue not found", which is what a correct run prints.
  • security-issue-triage step-2a case-1. The two DAG-author rows in the cheat sheet overlapped. The skill now says which one applies. When the DAG author controls the value, it is the code execution row. The routes user input row applies only when someone else's input reaches the operator. The eval's system prompt carries the same note.
  • security-issue-triage step-2.6 case-2. The prompt's match rules required an identical code pointer for STRONG. The skill says a positive precedent of the same shape supports VALID. The rules now follow the skill, and the budget counts as exhausted when the input says all the calls were used.

Test plan

  • prek hooks on commit
  • The four cases pass, run outside the sandbox
  • Full triage step-2.6 (3/3) and step-2a (3/3) suites pass

Part of #1344.

🤖 Generated with Claude Code

- issue-deduplicate step-0 case-3: a single argument leaves no
  duplicate, so the expected value is null, and the output spec
  allows null.
- release-audit-report step-0 case-3: the blocker assertion also
  accepts "Planning issue not found", which is what a correct run
  says.
- security-issue-triage step-2a case-1: the cheat sheet now says
  which DAG-author row applies. When the DAG author controls the
  value, it is the code-execution row. The routes-user-input row
  applies only when someone else's input reaches the operator. The
  same note is in the eval's system prompt.
- security-issue-triage step-2.6 case-2: the prompt's match rules
  now follow the skill's "same shape" wording for positive
  precedents. The budget is exhausted when the input says the
  calls were all used.

Generated-by: Claude Opus 5
@potiuk

potiuk commented Sep 29, 2026

Copy link
Copy Markdown
Member Author

I just noticed that was something my agents have not pushed yet. Completing it as the last thing before signing of entirely - till Sunday :)

@potiuk
potiuk merged commit 02b84bb into apache:main Sep 29, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant