feat(attestation): carry triage/adjudicate data in the AI security context - #3427
Conversation
AI Session Checks — 🟡 74% · ✅ 0 failing
|
| Status | Attribution | File | Lines |
|---|---|---|---|
| created | human | pkg/attestation/crafter/materials/testdata/ai-security-context-incomplete.json |
+859 / -0 |
| modified | ai | internal/schemavalidators/schemavalidators_test.go |
+117 / -10 |
| modified | human | internal/schemavalidators/testdata/ai_security_context_triage_only.json |
+87 / -2 |
| modified | ai | internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json |
+87 / -0 |
| modified | human | pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go |
+59 / -0 |
| modified | ai | pkg/attestation/crafter/materials/chainloop_ai_security_context_test.go |
+55 / -0 |
| modified | ai | pkg/attestation/crafter/materials/chainloop_ai_security_context.go |
+19 / -0 |
Policies (4)
| Status | Policy | Material | Messages |
|---|---|---|---|
| ✅ Passed | ai-config-no-dangerous-commands |
ai-coding-session-3f3650 |
- |
| ✅ Passed | ai-config-ai-agents-allowed |
ai-coding-session-3f3650 |
- |
| ✅ Passed | ai-config-no-secrets |
ai-coding-session-3f3650 |
- |
| ✅ Passed | ai-config-mcp-servers-allowed |
ai-coding-session-3f3650 |
- |
🟢 90% — 34% AI — ✅ All policies passing
-
Sep 7, 2026 19:10 UTC · 20m8s · $11.02 · 196 in / 63.3k out · claude-code 2.1.263 (claude-opus-4-8)
Change Summary
-
- Adds
trace_sessionsto the AI security context schema and Go wire types. - Updates validator and crafter fixtures to carry the new field.
- Expands
schemavalidatorscoverage fortrace_sessions, including the maxLength case. - Builds, runs package tests, and commits the schema change plus the follow-up test fix.
- Adds
AI Session Overall Score
-
🟢 90% — Well-executed, well-verified change with only a mild planning gap.
AI Session Analysis Breakdown
-
🟢 94% · scope-discipline
-
🟢 It staged only the task-related files despite unrelated dirty files. · High Impact
🟢 93% · solution-quality
-
No notes.
🟢 92% · alignment
-
🟢 Its final branch, commit, and test claims matched the recorded commands. · Medium Impact
🟢 91% · user-trust-signal
-
No notes.
🟢 89% · verification
-
🟢 It ran focused trace_sessions tests and saw the named cases pass. · High Impact
🟡 72% · context-and-planning
-
🟠 No visible plan artifact preceded the multi-file implementation phase. · Medium Severity
💡 For multi-file changes, write a short visible plan before editing so the setup matches the task scope.
-
File Attribution
██████░░░░░░░░░░░░░░34% AI / 66% HumanStatus Attribution File Lines modified ai internal/schemavalidators/schemavalidators_test.go+117 / -10 modified human internal/schemavalidators/testdata/ai_security_context_triage_only.json+87 / -2 modified ai internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json+87 / -0 modified ai pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go+59 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-7cf89f- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-7cf89f- ✅ Passed ai-config-no-secretsai-coding-session-7cf89f- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-7cf89f- -
🟡 60% — 26% AI — ✅ All policies passing
-
Aug 26, 2026 00:21 UTC · 20h45m45s · $62.15 · 692 in / 374.5k out · claude-code 2.1.245 (claude-opus-5)
Change Summary
-
- Adds
CHAINLOOP_AI_SECURITY_CONTEXTas a new material type with schema, validator, crafter, and generated API bindings. - Adds crafter and validator tests plus valid and negative security-context fixtures.
- Follow-up commits scrub fixture repository references without changing validation behavior.
- Adds
AI Session Overall Score
-
🟡 60% — Well-tested delivery, but two corrected claim-vs-reality misses keep alignment red.
AI Session Analysis Breakdown
-
🟢 95% · verification
-
🟢 AI added and reran focused tests before broader package suites passed. · High Impact
🟢 90% · scope-discipline
-
🟢 Follow-up scrub commits stayed confined to the requested fixture files. · Medium Impact
🟢 88% · context-and-planning
-
🟢 Implementation started from a written plan with named fixtures and scope boundaries. · High Impact
🟢 86% · solution-quality
-
No notes.
🟢 84% · user-trust-signal
-
No notes.
🔴 35% · alignment
-
🔴 AI twice overstated repo behavior, first on schema patterns and later on whether fixture relabeling mattered. · High Severity
💡 Before making repo-wide claims, check the exact code path and label hypotheticals as hypotheticals.
-
File Attribution
█████░░░░░░░░░░░░░░░26% AI / 74% HumanStatus Attribution File Lines created human pkg/attestation/crafter/materials/testdata/ai-security-context-incomplete.json+859 / -0 modified ai internal/schemavalidators/schemavalidators_test.go+117 / -10 modified human internal/schemavalidators/testdata/ai_security_context_triage_only.json+87 / -2 modified ai internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json+87 / -0 modified ai pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go+59 / -0 modified ai pkg/attestation/crafter/materials/chainloop_ai_security_context_test.go+55 / -0 modified ai pkg/attestation/crafter/materials/chainloop_ai_security_context.go+19 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-df945b- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-df945b- ✅ Passed ai-config-no-secretsai-coding-session-df945b- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-df945b- -
🟡 60% — 40% AI — ✅ All policies passing
-
Sep 10, 2026 05:03 UTC · 50m13s · $59.50 · 752 in / 492.0k out · claude-code 2.1.267 (claude-opus-4-8)
Change Summary
-
- Adds
CHAINLOOP_AI_TRIAGE_INDEXmaterial support with crafter, tests, and auto-detection. - Adds the embedded triage-index JSON schema and validator wiring.
- Extends AI security-context
scan_statswith adjudicate fields and validation coverage.
- Adds
AI Session Overall Score
-
🟡 60% — Strong validation, but two premature completion claims keep alignment in the red.
AI Session Analysis Breakdown
-
🟢 93% · verification
-
🟢 The AI repeatedly ran focused tests, suites, builds, and a real sec.json check. · High Impact
🟢 90% · scope-discipline
-
🟢 Committed changes stayed in the triage/security-context areas and left unrelated edits alone. · High Impact
🟢 90% · solution-quality
-
No notes.
🟢 84% · user-trust-signal
-
🟡 The user had to add two calm follow-up asks to surface missed requirements. · Low Severity
🟡 68% · context-and-planning
-
🟢 The opening ask anchored work in
oss.mdplus concrete validation inputs. · Medium Impact🟠 A multi-phase spec change ran without any visible plan, TODO, or PlanMode checkpoint. · Medium Severity
💡 For spec-driven multi-file work, publish a short plan before the first edit.
🔴 38% · alignment
-
🔴 AI twice declared
oss.mdcomplete before later required work surfaced. · High Severity💡 Before calling spec work complete, re-read the spec against the changed artifacts and final summary.
-
File Attribution
████████░░░░░░░░░░░░40% AI / 60% HumanStatus Attribution File Lines modified ai internal/schemavalidators/schemavalidators_test.go+117 / -10 modified human internal/schemavalidators/testdata/ai_security_context_triage_only.json+87 / -2 modified ai internal/schemavalidators/internal_schemas/aisecuritycontext/ai-security-context-0.1.schema.json+87 / -0 modified ai pkg/attestation/crafter/materials/aisecuritycontext/aisecuritycontext.go+59 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-fd9855- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-fd9855- ✅ Passed ai-config-no-secretsai-coding-session-fd9855- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-fd9855- -
Security Checks — ✅ 9 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
✅ iac-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | iac-misconfiguration |
- |
PR info
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | pr-min-approvals |
- |
| ✅ Passed | pr-description-required |
- |
| ✅ Passed | pr-user-story-linked |
- |
⏭️ 2 scans not applied
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
Powered by Chainloop and Chainloop Trace
matiasinsaurralde
changed the title
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 33 files
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
migmartri
left a comment
migmartri
left a comment
There was a problem hiding this comment.
Thanks @matiasinsaurralde I'd like to understand wether since evidenc has value by itself.
9fd09b7 to
88d873b
Compare
88d873b to
fb0c1d5
Compare
Fold the triage work queue and the triage/adjudicate funnel into the existing CHAINLOOP_AI_SECURITY_CONTEXT material, so one material spans both an un-adjudicated (triage-only) context and a fully adjudicated one instead of a separate triage-index kind. Kept at security-context-0.1 as additive, optional fields, so a combined-scan context that omits them still validates: - top-level `survivors` array (the adjudication work queue) with a `survivor` definition (only commit_sha required) - scan_stats: survivors_total, adjudicated_commits, triage_input_tokens, triage_output_tokens, adjudication_complete The embedded schema keeps additionalProperties: false throughout; the aisecuritycontext wire struct mirrors the new fields for annotation extraction. Assisted-by: Claude Code Signed-off-by: Matías Insaurralde <matias@chainloop.dev> Chainloop-Trace-Sessions: 3f365023-61a3-4af3-bbe9-03688f152834, 7cf89fb4-1736-425d-bafb-bfb8b59155a4, df945bde-e982-472c-bb81-50055df403bb, fd98553a-50f9-41e6-abf6-f03f0b9f1f4e Signed-off-by: Matías Insaurralde <matias@chainloop.dev>
fb0c1d5 to
2f874a5
Compare
matiasinsaurralde
changed the title
|
Thanks @matiasinsaurralde So we are going to record all the commits that were triaged as part of the index, not just the ones that failed correct? |
Complete the triage/adjudicate fields in CHAINLOOP_AI_SECURITY_CONTEXT with the two a producer emits but the schema still rejected: - top-level `discarded`: the commits Phase-1 triage classified and REJECTED, typed as sha_list. Together with `survivors` and `scan.unresolved` it states the full set of commits ever handed to the classifier, rather than leaving it to be inferred from the window bounds — which is wrong once merges are involved, since a bounded reverse-chronological walk does not cover exactly from_sha..to_sha. Stated explicitly, a walk that deliberately re-covers that ground can skip it instead of paying for it again. - `scan_stats.pending_survivors`: how many survivors still await adjudication. A count rather than a list, because the list is already determined by `survivors` and `adjudicated_commits`, and storing it would denormalize the artifact against itself. Signed-off-by: Matías Insaurralde <matias@chainloop.dev> Chainloop-Trace-Sessions: 3f365023-61a3-4af3-bbe9-03688f152834, 7cf89fb4-1736-425d-bafb-bfb8b59155a4, df945bde-e982-472c-bb81-50055df403bb, fd98553a-50f9-41e6-abf6-f03f0b9f1f4e
Add optional `scan.triage_budget_hit` to the security context: the most recent triage run stopped because it reached its survivor cap rather than because it exhausted its window, so known-untriaged history sits immediately behind scan.window.from_sha. A later run that exhausts its window clears it. It cannot be derived. commits_triaged < last_n is equally true of a repository smaller than the window, which is complete coverage, and of a budget stop, which is not — and separating them otherwise means resolving from_sha's ancestry in git, which a consumer reading the material out of CAS does not have. The producer is the only party that knows whether it chose to stop or ran out of history. Distinct from its neighbours: it says nothing about the adjudication queue, and it is not "the window does not reach the repository root" — a --last-bounded run does not either. Signed-off-by: Matías Insaurralde <matias@chainloop.dev> Chainloop-Trace-Sessions: 3f365023-61a3-4af3-bbe9-03688f152834, 7cf89fb4-1736-425d-bafb-bfb8b59155a4, df945bde-e982-472c-bb81-50055df403bb, fd98553a-50f9-41e6-abf6-f03f0b9f1f4e
Annotate CHAINLOOP_AI_SECURITY_CONTEXT with whether another run over the same HEAD would still make progress: survivors queued but not adjudicated, or triage stopped on its survivor budget with history left inside its window. Published as one derived boolean because an annotation query is an exact key/value match and cannot express "pending_survivors > 0". Without it a scheduler cannot tell a bounded run that is still catching up from a finished one without downloading the payload for every workflow on every sweep. The two underlying fields are annotated alongside it for observability, not filtering. An incomplete context is not a failed one — the bounded cold-start run publishes exactly this state on purpose. Signed-off-by: Matías Insaurralde <matias@chainloop.dev> Chainloop-Trace-Sessions: 3f365023-61a3-4af3-bbe9-03688f152834, df945bde-e982-472c-bb81-50055df403bb
2 participants
Refs PFM-7229 / PFM-7271
Summary
Extends the
CHAINLOOP_AI_SECURITY_CONTEXTmaterial so a single material spans both an un-adjudicated (triage-only) context and a fully adjudicated one, rather than introducing a separate triage-index material kind.The embedded security-context schema (kept at
security-context-0.1) gains, as additive optional fields:survivorsarray. The adjudication work queue that Phase-1 triage produces and an adjudicate run drains with asurvivordefinitionA combined-scan context that omits these still validates, and the schema keeps additionalProperties: false throughout. The aisecuritycontext wire struct mirrors the new fields.