feat(attestation): record adjudication outcomes and gaps in the security context - #3436
Conversation
…solved holes Add an optional `retryable` boolean to the ai-security-context-0.1 unresolved item and to the typed Unresolved model. Signed-off-by: Matías Insaurralde <matias@chainloop.dev>
AI Session Checks — 🟡 79% · ✅ 0 failing
|
| Status | Attribution | File | Lines |
|---|---|---|---|
| modified | ai | internal/schemavalidators/schemavalidators_test.go |
+20 / -0 |
| modified | ai | pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go |
+5 / -0 |
| modified | ai | internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json |
+4 / -0 |
Policies (4)
| Status | Policy | Material | Messages |
|---|---|---|---|
| ✅ Passed | ai-config-ai-agents-allowed |
ai-coding-session-df945b |
- |
| ✅ Passed | ai-config-no-dangerous-commands |
ai-coding-session-df945b |
- |
| ✅ Passed | ai-config-no-secrets |
ai-coding-session-df945b |
- |
| ✅ Passed | ai-config-mcp-servers-allowed |
ai-coding-session-df945b |
- |
🟢 88% — 82% AI — ✅ All policies passing
-
Aug 26, 2026 21:49 UTC · 43m5s · $16.63 · 328 in / 138.2k out · claude-code 2.1.245 (claude-opus-5)
Change Summary
-
- Adds a schema guard so the AI security context L0 taxonomy has one authored source.
- Publishes AI security context tool identity through shared material-tool annotations.
- Enables auto-detection for the AI security context material and adds positive detection coverage.
AI Session Overall Score
-
🟢 88% — Only reviewer check is missing user confirmation after otherwise strong automated verification.
AI Session Analysis Breakdown
-
🟢 94% · solution-quality
-
🟢 AI used failing tests and empirical probes before each substantive fix. · High Impact
🟢 92% · alignment
-
No notes.
🟢 90% · scope-discipline
-
No notes.
🟢 88% · user-trust-signal
-
No notes.
🟢 84% · context-and-planning
-
🟢 User prompts supplied precise file, line, and constraint framing for each phase. · High Impact
🟡 78% · verification
-
🟢 Final verification includes go build plus focused package suites and schema validators. · High Impact
🟠 Automation is extensive, but no user turn confirms the final three-commit state after the last self-report. · Medium Severity
💡 When the user is still in-loop, ask for one explicit confirmation on the shipped state.
-
File Attribution
████████████████░░░░82% AI / 18% HumanStatus Attribution File Lines modified ai internal/schemavalidators/schemavalidators_test.go+20 / -0 modified human pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go+5 / -0 modified ai internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json+4 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-3f3650- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-3f3650- ✅ Passed ai-config-no-secretsai-coding-session-3f3650- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-3f3650- -
🟢 90% — 100% AI — ✅ All policies passing
-
Sep 7, 2026 19:10 UTC · 20m8s · $11.02 · 196 in / 63.3k out · claude-code 2.1.263 (claude-opus-4-8)
Change Summary
-
- Adds optional
trace_sessionsarrays to the AI security context schema forfingerprintandcommit_ref. - Mirrors
TraceSessionsonto the Go wire structs and updates validator and crafter fixtures. - Adds validator coverage for positive paths, bounds, and the follow-up
maxLengthcase.
- Adds optional
AI Session Overall Score
-
🟢 90% — Well-scoped schema change, planned from prior analysis, and thoroughly verified.
AI Session Analysis Breakdown
-
🟢 93% · scope-discipline
-
🟢 AI staged only the five task files after spotting unrelated workspace changes. · High Impact
🟢 92% · verification
-
🟢 AI ran targeted validator tests and saw the new trace_sessions cases pass. · High Impact
🟢 90% · alignment
-
No notes.
🟢 90% · solution-quality
-
🟢 The follow-up fix added a real maxLength test instead of weakening the comment. · High Impact
🟢 88% · user-trust-signal
-
No notes.
🟢 85% · context-and-planning
-
No notes.
-
File Attribution
████████████████████100% AI / 0% HumanStatus Attribution File Lines modified ai internal/schemavalidators/schemavalidators_test.go+20 / -0 modified ai pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go+5 / -0 modified ai internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json+4 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-7cf89f- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-7cf89f- ✅ Passed ai-config-no-secretsai-coding-session-7cf89f- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-7cf89f- -
🟡 79% — 100% AI — ✅ All policies passing
-
Sep 10, 2026 05:03 UTC · 50m13s · $59.50 · 752 in / 492.0k out · claude-code 2.1.267 (claude-opus-4-8)
Change Summary
-
- Adds
CHAINLOOP_AI_TRIAGE_INDEXmaterial support, auto-detection wiring, schema validation, fixtures, and generated API/docs updates. - Extends AI security-context
scan_statswith adjudication fields and mirrors them in the OSS wire struct. - Adds validator and crafter coverage, including
sec.jsonend-to-end validation through the real crafter path.
- Adds
AI Session Overall Score
-
🟡 79% — Well-validated implementation, but an initial schema miss and no explicit plan kept review caution warranted.
AI Session Analysis Breakdown
-
🟢 90% · scope-discipline
-
🟢 User-expanded schema and security-context work landed without unrelated drift. · High Impact
🟢 89% · solution-quality
-
🟢 It fixed schema rejection at the validator layer instead of loosening checks. · High Impact
🟢 84% · user-trust-signal
-
No notes.
🟡 77% · verification
-
🟢 AI exercised
sec.jsonthrough the real crafter path and an old-schema counterfactual. · High Impact🟠 Targeted tests ran repeatedly, but the active user never confirmed the final shipped state worked. · Medium Severity
💡 When the user stays active, ask for a quick confirmation after the final validation pass.
🟡 72% · context-and-planning
-
🟠 The task was broad, yet no written plan or todo list framed implementation, validation, and commit work. · Medium Severity
💡 Before editing a wide change, write a short step list naming implementation, validation, and commit boundaries.
🟡 68% · alignment
-
🟠 AI first declared
oss.mdimplemented without the JSON schema the user expected. · Medium Severity💡 Before claiming a spec is implemented, restate the required deliverables and confirm none were deferred.
-
File Attribution
████████████████████100% AI / 0% HumanStatus Attribution File Lines modified ai internal/schemavalidators/schemavalidators_test.go+20 / -0 modified ai pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go+5 / -0 modified ai internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json+4 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-fd9855- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-fd9855- ✅ Passed ai-config-no-secretsai-coding-session-fd9855- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-fd9855- -
Security Checks — ✅ 9 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
✅ iac-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | iac-misconfiguration |
- |
PR info
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | pr-min-approvals |
- |
| ✅ Passed | pr-description-required |
- |
| ✅ Passed | pr-user-story-linked |
- |
⏭️ 2 scans not applied
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
Powered by Chainloop and Chainloop Trace
…eason Add `verdict` (no_finding | abstain | rejected) and `verdict_reason` (abstain-only) to the ai-security-context-0.1 survivor definition and the typed Survivor model. A survivor that produced no fingerprint now carries its terminal outcome, and an abstention carries the model's short explanation, so "which commits were undecided, and why" is answerable from the artifact. Both are optional; the survivor object stays additionalProperties: false. Signed-off-by: Matías Insaurralde <matias@chainloop.dev> Chainloop-Trace-Sessions: 3f365023-61a3-4af3-bbe9-03688f152834, 7cf89fb4-1736-425d-bafb-bfb8b59155a4, df945bde-e982-472c-bb81-50055df403bb, fd98553a-50f9-41e6-abf6-f03f0b9f1f4e
…ext stats Add an optional `abandoned` count to the AI security context scan stats Signed-off-by: Matías Insaurralde <matias@chainloop.dev>
matiasinsaurralde
changed the title
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
2 participants
Refs PFM-7229
Summary
Adds three optional fields to the
ai-security-context-0.1schema and its typed model, so a context can record how adjudication ended for each commit it looked at:unresolved[].retryable— whether a later run should re-examine a commit that never resolved, as opposed to a deterministic hole.survivors[].verdictandverdict_reason— the terminal outcome for a survivor that produced no fingerprint, plus a short explanation on an abstention.stats.abandoned— how many survivors adjudication gave up on after the retry cap.All three are optional and omitted by default, so contexts written against the current schema keep validating, and both objects still reject unknown fields.