Skip to content

fix(ci): bind QSR coverage evidence to each run - #2648

Open
seonghobae wants to merge 2 commits into
mainfrom
fix/qsr-src03-bound-evidence
Open

seonghobae wants to merge 2 commits into
mainfrom
fix/qsr-src03-bound-evidence

Conversation

@seonghobae

@seonghobae seonghobae commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Narrow SRC03 evidence-consistency slice for the central QSR reusable workflow. This does not complete hostile-workload isolation.

  • Create exclusive run/attempt/lane/head-bound evidence roots for coverage and branch coverage. Reject stale roots and nonregular, symlink, hardlink, missing or empty outputs.
  • Bind payload name, byte count, SHA-256, producer exit, root device/inode and event identity in a closed manifest. Revalidate before upload without changing the product coverage thresholds.
  • Use portable chunked SHA-256; bound manifest publication/decoding to 4096 UTF-8 bytes and payload reads to at most 65536-byte chunks. Count bytes and verify stable descriptor fields before and after reads; do not impose an arbitrary payload cap.
  • Repair the mechanics oracle: require the retained immutable upload-action SHA, literal reviewed producer/guard script hashes, complete observed step/env schemas and ordering. Retain persistent rejection controls for mutable/alternate action refs, substituted scripts, removed/bypassed guards and changed order.
  • Preserve workflow_call-only events, exact raw-head checks, read-only permissions, dedicated group/labels, pinned compiler/tools, caller tests, positive-LSM one-test witness, cleanup failure propagation and strict coverage enforcement.

Refs ContextualWisdomLab/quarantine-sandbox-runtime#136, #1590, #2083. Earlier producer PR #2601 and caller PR #154 are already merged; this is their separate narrow evidence-consistency follow-up, not a retry of them.

Exact source evidence

  • Base: acb8b34f9e7b8649fa1478139fa21c2217bdfbc7
  • Current head: fbc6db1a22da5026dc5d2352d18d3d62d0e7292f
  • Original repaired producer head: 0b24fabf980882576df1a083a9a7cd4f146ffc7d
  • Current tree: ac4b11cee923749545834a873c248cbbd6dd24c0
  • Original repaired-source reviewed tree: 94dfbdc5854c243d7b9cee63b6f4dba8c4491b25
  • Current cumulative full binary patch SHA-256: b57f256ddf83661ed351ace8f4bf4b0c8cced85c88ccc2d7bb7624b854f8d25b
  • Original repaired-source patch SHA-256: 144813a885e3466ea88ccb1725fc2d91405fd71e4da33144b2ee3cf745a0e6d3
  • Original exact three-file repaired-source re-review: PASS, SHA-256 95730b8eacfb0010cc8e1d8aeb6e15d2097a5becb5faea9b6730055639db3335. B01/B02/R03 resolved; unchanged workflow/doc bytes retain that scope.
  • Independent test-only coverage-controls delta review: SCOPED PASS, SHA-256 37d957cce6df45a134c9cb8179c5c2324d254ee723eb5b83dc9ac4c83e276343. Patch b8562afef3133fc7c0c6257dfa79eef92195542e30574c38d165bbf28b3eb67b; no blocking security or logic findings. This is not full-gate or runtime approval.
  • Parent verified all 826 tracked file hashes/modes against the reviewed candidate. The follow-up changes only the test file; workflow and doc bytes remain unchanged. New commit parent/tree, remote branch/PR head, workflow/test Git blobs and clean canonical checkout were read back.

Actual local verification

  • TDD RED/GREEN retained separately: B01 mutable upload/script substitutions; B02 synthetic missing-file_digest capability seam; R03 manifest serialization/read and payload stream/drift boundaries.
  • Original exact canonical scoped QSR/runner/K-CSAP suite: 170 passed, historical predecessor evidence.
  • Current test-delta parent execution without custom Popen/workflow instrumentation: 212 passed, 0 failed, 0 errors, 0 skipped, direct exit 0. The four-file selection is not the whole repository suite.
  • Added 42 parametrized controls across nine test functions, with 15 distinct role/old/new mutation replacement triples. All 42 scratch mutation controls failed intended assertions. The four growth mutants fail the required production diagnostic after the bounded-read seam rejects an unbounded continuation; they do not directly fail the second-stat-count assertion.
  • Four unchanged extracted producer/guard bodies: each producer 61/61 statements, 30/30 branches; each guard 56/56 statements, 26/26 branches. No exclusions or missing paths. Retained report 72a5d22e6ce38f2e8dee4ab74bcc6493885a9ccce7b9536747e691b2487ee83b; independent raw-data consistency review found no mutation/historical-data contamination.
  • Independent re-review: 91 targeted tests passed, 33 deselected; local Python 3.14.5 and synthetic subprocess fixtures, not runner/minimum-version acceptance.
  • Current test-delta Ruff check, format check and staged diff check: exit 0.
  • Parent exact original head local Bash 5.3 selection: 2,387 passed, 0 failed/error/skip, plus 80 stdout subtests reported separately. 4,206 conservatively screened nodes were NOT_RUN. This is macOS compatibility evidence, not Ubuntu/Actions or complete-suite acceptance.
  • Configured scripts/ci branch coverage with unchanged fail-under 100: FAIL on that admitted selection. Statements 10,758/19,668 (54.698%), branches 3,900/8,064 (48.363%), combined 52.856%. Pytest exit 0, report and JSON exits 2. Omitted mocked/static tests require finer safety classification; this is not proof that the full baseline suite fails. All 110 measured scripts/ci file bytes match the exact base.
  • Interrogate configured minimum 100: actual 100%, exit 0. Broad Ruff remains FAIL: 255 identical inherited findings on base/head, zero new; changed-file success does not clear that gate.
  • Exact canonical actionlint with explicit real ShellCheck and private hash-pinned pyflakes 4.0.3: exit 0 with no scanner disabled. A separate negative canary exited 1 and reported both pyflakes undefined-name and ShellCheck SC2086 diagnostics.
  • Earlier child pyflakes-unavailable and parent wrong scanner-path invocations remain partial lint, not PASS. The corrected scanner receipt is separate.
  • Durable parent evidence: QSR Git administrative history qsr-138/history/20261014-src03-review-repair/.

Remaining gates and limits

  • INCOMPLETE/FAIL: complete central repository acceptance, configured whole-source coverage and inherited broad Ruff findings. The embedded-body coverage pass is separate and does not clear these gates.
  • NOT_RUN/UNOBSERVED: actual Linux runner inventory, Cargo/LLVM generation at moved output paths, real upload action, Podman/SELinux E2E, security/SBOM/provenance, literal-auto inference and current-head author-distinct App approval. New-head Actions and reviews require fresh evidence; predecessor results are not inherited.
  • Exclusive directories, O_NOFOLLOW, device/inode and hashes establish artifact consistency only. Same-UID/ancestor replacement, forged manifests and mutation after admission remain possible; this is not hostile-writer isolation or signed provenance.
  • Source/tool/build/process custody, positive-LSM evidence paths, process settlement, retention/deletion custody and cleanup remain incomplete. Ambiguous roots are not deleted.
  • Existing QSR dedicated runner group was absent in the last actual caller run. No general/control runner fallback, runner/ACL mutation, credential issuance/value read or manual dispatch/retry is introduced.
  • Source publication is not main integration, consumer adoption, operational readiness or release. Do not inherit the predecessor Noema approval or treat skipped checks as passing execution.

Summary by CodeRabbit

  • 품질 개선

    • 커버리지 결과를 업로드하기 전에 실행 정보와 파일의 무결성을 확인합니다. 예상한 커밋에서 생성되지 않았거나 생성 후 변경된 결과는 업로드하지 않습니다.
    • 일반 커버리지와 브랜치 커버리지 결과를 실행별로 구분해 보관합니다.
  • 문서

    • 커버리지 결과 검증 절차와 적용 범위, 남아 있는 제한 사항을 문서화했습니다.
  • 테스트

    • 정상 결과와 변조되거나 유효하지 않은 결과를 확인하는 자동 검증을 추가했습니다.

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 44b967e3-e66f-4313-b051-fbadedd2372c
📥 Commits

Reviewing files that changed from the base of the PR and between acb8b34 and 0b24fab.

📒 Files selected for processing: 3
  • .github/workflows/quarantine-sandbox-runtime-ci.yml
  • docs/doctoring/quarantine-sandbox-runtime-self-hosted-ci.md
  • tests/test_quarantine_sandbox_runtime_ci_contract.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

두 커버리지 작업이 실행 정보와 체크아웃 커밋에 바인딩된 증거 파일 및 매니페스트를 생성합니다. 업로드 전 가드가 증거를 검증하고, 검증된 파일을 실행 정보가 포함된 아티팩트로 업로드한 뒤 검사기에 전달합니다. 계약 테스트와 문서도 이 흐름을 다룹니다.

Changes: 커버리지 증거 생성 및 검증

Layer / File(s) Summary
실행 바인딩 증거 생성
.github/workflows/quarantine-sandbox-runtime-ci.yml, tests/test_quarantine_sandbox_runtime_ci_contract.py, docs/doctoring/quarantine-sandbox-runtime-self-hosted-ci.md
두 작업이 실행 ID, 시도, 작업 종류, 예상 커밋을 확인하고 격리된 증거 디렉터리에 커버리지 JSON과 바인딩 매니페스트를 생성합니다. 계약 테스트는 생성 결과와 잘못된 파일 유형 등을 검증합니다.
업로드 전 증거 검증
.github/workflows/quarantine-sandbox-runtime-ci.yml, tests/test_quarantine_sandbox_runtime_ci_contract.py, docs/doctoring/quarantine-sandbox-runtime-self-hosted-ci.md
가드가 디렉터리 권한과 소유자, 매니페스트 및 증거 파일의 유형·크기·해시·실행 정보, 체크아웃 커밋을 확인합니다. 테스트는 변형된 매니페스트와 파일, 크기 제한, 읽기 중 파일 변경을 검사합니다.
아티팩트 업로드 및 검사기 전달
.github/workflows/quarantine-sandbox-runtime-ci.yml, tests/test_quarantine_sandbox_runtime_ci_contract.py, docs/doctoring/quarantine-sandbox-runtime-self-hosted-ci.md
아티팩트 이름에 커밋과 실행 정보를 포함하고 커버리지 JSON 및 매니페스트를 업로드합니다. 이후 검사기에 증거 파일 경로를 전달하며, 계약 테스트는 가드와 업로드의 순서 및 설정을 확인합니다.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant CoverageProducer
  participant AdmissionGuard
  participant ArtifactUpload
  participant CoverageParser
  CoverageProducer->>AdmissionGuard: 증거 파일과 binding.json 전달
  AdmissionGuard->>ArtifactUpload: 검증된 증거 경로 허용
  ArtifactUpload->>CoverageParser: 업로드된 커버리지 JSON 경로 전달
Loading
Merge Risk: ⚪ Minimal · up to 0b24f

This change binds coverage evidence to each CI run and validates it before upload. Coverage thresholds and permissions are unchanged. No concrete defect was established, so the change appears ready to merge with normal checks.

Pre-merge checks: Passed 5
Passed Passed checks: 5 passed
Check name Status Explanation
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 1 files. (2 skipped: 2…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed 제목은 QSR 커버리지 증거를 각 실행에 바인딩하는 핵심 변경을 정확하고 간결하게 설명합니다.
✨ Finishing Touches: 💡 1
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Add 42 parametrized synthetic controls for identity, checkout drift, counted reads, metadata changes, manifest bounds and lane commands.

Independent test-delta review: 37d957cce6df45a134c9cb8179c5c2324d254ee723eb5b83dc9ac4c83e276343.
Embedded source coverage is 100% per body; central full-gate and runtime acceptance remain separate.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode reviewed the current-head product diff. Coverage is a separate gate.

Changed files

  • .github/workflows/quarantine-sandbox-runtime-ci.yml — GitHub Actions review job
  • docs/doctoring/quarantine-sandbox-runtime-self-hosted-ci.md — operator or user guidance
  • tests/test_quarantine_sandbox_runtime_ci_contract.py — regression suite

Changed behavior

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Workflow: quarantine-sandbox-runtime-ci.yml"]
  S1 --> I1["GitHub Actions review job"]
  I1 --> R1["Review risk: Workflow: quarantine-sandbox-runtime-ci.yml"]
  R1 --> V1["actionlint plus required checks"]
  Evidence --> S2["Docs: quarantine-sandbox-runtime-self-hosted-ci.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: quarantine-sandbox-runtime-self-hosted-ci.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_quarantine_sandbox_runtime_ci_contract.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_quarantine_sandbox_runtime_ci_contract.py"]
  R3 --> V3["targeted test run"]
Loading

Findings

No source-backed product finding is synthesized from the coverage gate. A coverage miss belongs in the status comment.

  • Head SHA: fbc6db1a22da5026dc5d2352d18d3d62d0e7292f
  • Workflow run: 38531446132
  • Workflow attempt: 1
  • Coverage gate: failure

Review outcome

Coverage is a gate, not the review. This body reviews the changed product files.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Workflow: quarantine-sandbox-runtime-ci.yml"]
  S1 --> I1["GitHub Actions review job"]
  I1 --> R1["Review risk: Workflow: quarantine-sandbox-runtime-ci.yml"]
  R1 --> V1["actionlint plus required checks"]
  Evidence --> S2["Docs: quarantine-sandbox-runtime-self-hosted-ci.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: quarantine-sandbox-runtime-self-hosted-ci.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_quarantine_sandbox_runtime_ci_contract.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_quarantine_sandbox_runtime_ci_contract.py"]
  R3 --> V3["targeted test run"]
Loading

@opencode-agent

opencode-agent Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

OpenCode Review Overview

Coverage evidence did not pass, so approval is blocked. The formal pull-request review is the source-backed diff review, not this status comment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant