fix(evals): resolve the four remaining eval failures - #1476
Merged
Merged
Conversation
- issue-deduplicate step-0 case-3: a single argument leaves no duplicate, so the expected value is null, and the output spec allows null. - release-audit-report step-0 case-3: the blocker assertion also accepts "Planning issue not found", which is what a correct run says. - security-issue-triage step-2a case-1: the cheat sheet now says which DAG-author row applies. When the DAG author controls the value, it is the code-execution row. The routes-user-input row applies only when someone else's input reaches the operator. The same note is in the eval's system prompt. - security-issue-triage step-2.6 case-2: the prompt's match rules now follow the skill's "same shape" wording for positive precedents. The budget is exhausted when the input says the calls were all used. Generated-by: Claude Opus 5
Member
Author
|
I just noticed that was something my agents have not pushed yet. Completing it as the last thing before signing of entirely - till Sunday :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the four eval failures left over from the security-family optimization (#1344).
Summary
null, and the output spec allowsnull.Test plan
Part of #1344.
🤖 Generated with Claude Code