Skip to content

test(forge): live-oracle legs for the state-transition op (RIG-3331) - #1237

Merged
trunk-io[bot] merged 3 commits into
mainfrom
compass-forge/rig-3331-t7-live-oracle
Sep 16, 2026
Merged

trunk-io[bot] merged 3 commits into
mainfrom
compass-forge/rig-3331-t7-live-oracle

Conversation

@rigel-mintaka

@rigel-mintaka rigel-mintaka commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Closes T7 of the frozen forge state-transition record (docs/designs/server/compass-forge-state-transition/design.md), the last unbuilt task on RIG-3331.

Why this existed

The state-transition op shipped with tier-1 coverage only. 13 transition fixtures sit in go/internal/forge/testdata/, but livegithub_test.go had 15 TestLive* functions and not one of them mentioned a transition — git grep -c -i transition -- livegithub_test.go returned nothing. DL-210 requires both tiers, and the record's own acceptance names //go:build livegithub coverage explicitly.

Found while verifying whether RIG-3331 could be closed after its two children (RIG-3517/RIG-3518) landed. A parent whose children are all Done is not therefore Done: the children covered T0–T6, and T7 had no child to close.

What landed

Ten live legs, one per live-reproducible fixture, plus a cross-op scenario on both providers: create → close → an independent GetIssue asserting the state changed → reopen → assert reopened. The separate read is the whole point; a single-call fixture compare pins one response shape and cannot show the write took effect.

Three judgment calls

Three fixtures stay golden-replay-only, named in the file with their reason — each needs live repo or team state a test cannot create for itself: transition_pull_request_reopen_merged (a merged PR), transition_issue_duplicate_name (two states sharing a name), transition_issue_ambiguous_default (two completed-type states). "Every fixture has a matching leg" therefore resolves to a leg or a stated reason; silently omitting them would leave the next reader to re-discover the gap.

The PR-transition legs assert the state fold directly rather than comparing to the fixture. ghPullDetail populates Changed{Files,Additions,Deletions} (the fixture pins {3,10,2}) and those are not in volatileFields, so a live seeded PR would red the compare on data the transition does not control.

Every Linear by-name leg discovers its target workflow-state name from the live team instead of hardcoding one. This one is load-bearing: a hardcoded name on a team that lacks it fails down the unknown-name path, which would have satisfied the type-contradiction assertion for entirely the wrong reason — a test passing while proving nothing. The helper skips when the team has no state of the needed type, and CI's assert-ran guard turns that skip into a red.

No capture specs for the error fixtures

deriveFixtureHalves derives Want from a successful decode and t.Fatalfs on error, so rejection fixtures are hand-written by construction. Consistent with every existing wantError fixture — none has a spec today.

Verification

go vet -tags livegithub ./internal/forge/... clean; go build -tags livegithub ./... clean; untagged go test ./internal/forge/... passes (golden replay unbroken); all 12 new legs register and skip cleanly without credentials; union moon ci vs f4360703 → 27/27.

The live legs cannot be executed here — no testbed credentials. They run on CI's forge live-contract oracle step.

Ledger-impact: none — T8's DL-342/DL-343 rows already describe this op; T7 adds test coverage only.

Closes T7 of the forge state-transition record: the DL-210 tier-2 live-contract
coverage the op shipped without. The 13 committed transition fixtures had golden
replay only, and no live leg mentioned transition at all.

Ten live legs, one per live-reproducible fixture, plus a cross-op scenario on
both providers: create -> close -> an INDEPENDENT GetIssue asserting the state
changed -> reopen -> assert reopened. The separate read is the point -- a
single-call fixture compare pins one response shape and cannot show the write
took effect.

Three fixtures stay golden-replay-only and are named in the file with their
reason: each needs live repo or team state a test cannot create for itself
(a merged PR; two workflow states sharing a name; two completed-type states).

The PR-transition legs assert the state fold directly instead of comparing to
the fixture, because ghPullDetail populates Changed{Files,Additions,Deletions}
and those are not in volatileFields -- a live seeded PR cannot reproduce the
pinned counts, so a fixture compare would red on data the transition does not
control.

Every Linear by-name leg DISCOVERS its target workflow-state name from the live
team rather than hardcoding one. A hardcoded name on a team that lacks it fails
down the unknown-name path, which would have passed the type-contradiction
assertion for the wrong reason.

No capture specs for the error fixtures: deriveFixtureHalves derives Want from a
successful decode and t.Fatalfs on error, so rejection fixtures stay
hand-written -- consistent with every existing wantError fixture.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@trunk-io

trunk-io Bot commented Sep 16, 2026

Copy link
Copy Markdown

😎 Merged successfully - details.

@linear-code

linear-code Bot commented Sep 16, 2026

Copy link
Copy Markdown

RIG-3331

@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

Compass engineering docs preview: https://compass-forge-rig-3331-t7-li.compass-eng-docs.pages.dev

Deployed from compass-forge/rig-3331-t7-live-oracle at 2aa7dae.

The live oracle T7 added found this on its first run: every Linear transition
400s against the real API.

  http 400: Variable "$team" of type "String!" used in position expecting
  type "ID".

`workflowStatesQuery` declares $team as String! and lands it in an id
comparator (`filter: {team: {id: {eq: $team}}}`), which Linear types as ID.
Every Linear state transition failed before reaching the mutation, so the whole
RIG-3331 Linear write path was dead on arrival in production.

Golden replay could never catch this. A fixture answers OUR request, so a
malformed query replays green forever against its own recorded response -- the
prelude entries pin the response bodies, never the query text. Tier 1 proved the
decode; only tier 2 could prove the request.

The mutation's `$id: String!` is deliberately left alone: Linear types
issueUpdate's id argument as String!, and the live error named only $team. The
fixed legs now reach the mutation, so the next oracle run proves that half
rather than my reading of it.

TestLinearIDFilterVariablesAreDeclaredID pins the declared type hermetically.
Verified red before the fix (mutant vet-clean first, then the assertion fails
naming the String! text) and green after.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@rigel-mintaka

Copy link
Copy Markdown
Contributor Author

The oracle found a production bug on its first run. Second commit fixes it.

What T7 caught

Every Linear transition 400s against the real API:

http 400: Variable "$team" of type "String!" used in position expecting type "ID".

workflowStatesQuery in go/internal/forge/linear.go declared $team as String! and landed it in an id comparator — filter: {team: {id: {eq: $team}}} — which Linear types as ID. The failure hits workflowStatesFor, the first call in TransitionIssueState, so the whole RIG-3331 Linear write path was dead on arrival in production. All six Linear legs failed; all six GitHub legs passed.

Why tier 1 could never catch it

A golden fixture answers our own request. The prelude entries pin response bodies, never the query text — so a malformed query replays green against its own recorded response, forever. Tier 1 proved the decode was right; only tier 2 could prove the request was.

That is DL-210's two-tier rationale demonstrating itself on the first run, which is worth more than the coverage itself: this op merged through three PRs, a design freeze, and a review loop with its Linear half non-functional, and nothing in the hermetic pyramid could have said so.

What I deliberately did not change

The mutation's $id: String! in issueUpdateState. Linear types issueUpdate's id argument as String!, and the live error named only $team — so "fix every String! that looks like an id" would have been a guess dressed as thoroughness. The fixed legs now reach the mutation, so the next oracle run proves that half instead of my reading of it. It passed: TestLiveLinearTransitionIssueClose and both cross-op legs exercise the mutation end to end.

Regression pin

TestLinearIDFilterVariablesAreDeclaredID asserts the declared type hermetically, since no fixture can. Verified red before the fix — mutant go vet-clean first, then the assertion fails naming the String! text — and green after.

Evidence

forge check green at 28f36218, and the legs ran rather than skipped (live durations, no SKIP lines):

--- PASS: TestLiveLinearTransitionIssueClose (1.77s)
--- PASS: TestLiveLinearTransitionIssueCloseByName (1.05s)
--- PASS: TestLiveLinearTransitionIssueReopen (4.12s)
--- PASS: TestLiveLinearTransitionUnknownName (0.91s)
--- PASS: TestLiveLinearTransitionTypeContradiction (0.69s)
--- PASS: TestLiveLinearTransitionCrossOp (3.59s)
--- PASS: TestLiveGitHubTransitionCrossOp (2.95s)

(plus the five GitHub legs). rollup=SUCCESS; union moon ci 32/32.

The PR is now two commits: the T7 coverage, and the fix it found. I kept them separate so the fix is reviewable on its own and the sequence stays legible.

Review finding on the T7 legs: liveStateNameOfType's comment claimed its skip
was caught by CI's assert-ran guard. It was not. The guard greps only the two
CREDENTIAL-unset literals, and a package with a skipped test still reports ok --
so a testbed team missing a canceled- or started-type workflow state would have
silently skipped both by-name legs with the required check staying green. The
exact vacuous-green the surrounding guard exists to prevent, claimed as covered
by a comment a future maintainer would trust.

Fixed by making it true rather than by softening the comment: the skip now
carries liveBoardStateSkipPrefix, a stable one-line literal, and the guard
greps it on the same source-derived discipline as the credential skips (sed the
constant out of the test file, fail loudly if the extraction comes back empty so
the guard cannot drift into checking nothing). Verified the sed reads the new
constant verbatim.

Also from review, all documentation precision:

- the PR-leg bypass rationale named only the diff stats; Draft diverges too
  (a closed draft stays a draft, the fixture pins false), so a narrower
  projection would not have worked either -- the fuller reason strengthens the
  bypass rather than excusing it;
- the by-name leg now states what it does NOT prove: forge.Issue folds every
  closed type to "closed", so the landed column is unobservable live and the
  resolved stateId is pinned by the golden fixture instead;
- the capture table said "one spec per committed fixture", untrue once five
  fixtures were deliberately left specless.

Declined: dropping the inert CloseReason from the reopen capture spec. It is
dropped by the provider on a reopen, and it keeps a regenerated fixture
byte-identical to the committed one; removing it means editing the fixture in
the same breath, which is a separate change from adding coverage.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@rigel-mintaka

Copy link
Copy Markdown
Contributor Author

Review loop closed: 0 high, 1 medium, 4 low. Medium + three lows fixed in 2aa7dae6; one low declined.

The medium was my own comment lying about a safety net

liveStateNameOfType's doc claimed its skip was "loud via CI's assert-ran guard". It was not. The guard greps only the two credential-unset literals, and a package with a skipped test still reports ok — so a testbed team missing a canceled- or started-type workflow state would have silently skipped both by-name legs with the required check staying green. Precisely the vacuous-green the surrounding guard exists to prevent, asserted as covered by a comment the next maintainer would trust.

Fixed by making the claim true rather than softening it: the skip now carries liveBoardStateSkipPrefix, a stable one-line literal, and the guard greps it on the same source-derived discipline as the credential skips — sed the constant out of the test file, fail loudly if the extraction returns empty so the guard cannot drift into checking nothing. Verified the sed reads the new constant verbatim.

Worth naming the shape: I added a test whose failure mode was silence, and wrote a comment claiming infrastructure caught it. Only reading the guard's actual grep list disproves that.

Three lows, all documentation precision

  • The PR-leg bypass cited only the diff stats; Draft diverges too (closing a draft leaves it a draft; the fixture pins false), so a narrower projection would not have worked either — the fuller reason strengthens the bypass.
  • The by-name leg now states what it does not prove: forge.Issue folds every closed type to "closed", so the landed column is unobservable live; the resolved stateId is pinned by the golden fixture instead.
  • The capture table said "one spec per committed fixture", untrue once five fixtures were deliberately left specless.

Declined

Dropping the inert CloseReason from the reopen capture spec. The provider drops it on a reopen, and it keeps a regenerated fixture byte-identical to the committed one; removing it means editing the fixture in the same breath, which is a separate change from adding coverage.

Independently confirmed by the reviewer

Worth recording, since these were the calls most likely to be wrong: no fixture-compare leg is vacuous (every residue is non-empty after volatileFields stripping); all three golden-replay-only exclusions are genuine, not lazy; prelude accounting is right on both providers — and specifically that running discovery on the non-recording setup client is load-bearing, since sharing the recording client would have warmed the cache, dropped the workflowStates call, and made the true prelude 2 while the spec claimed 3, silently corrupting any regenerated fixture.

Gates at 2aa7dae6: union moon ci 35/35; go vet -tags livegithub clean; untagged forge tests pass.

@rigel-mintaka
rigel-mintaka marked this pull request as ready for review September 16, 2026 03:18
@rigel-mintaka

Copy link
Copy Markdown
Contributor Author

Open Questions (parked for Matt — overnight mode)

Ready for review; green; nothing blocking. One fork I took conservatively rather than deciding for you.

The Linear write path was broken in production and nothing outside this PR would have told us. The $team: String! bug meant every Linear state transition 400d. It shipped through three PRs, a design freeze, and a review loop. The hermetic tier cannot catch a malformed request by construction — a fixture answers our own request, so the query text is never asserted. That is now pinned for this one query by TestLinearIDFilterVariablesAreDeclaredID.

The fork: the same blind spot covers every Linear GraphQL query, not just the one that happened to break. resolveTeamID, resolveIssueID, ListIssues, the comment/create mutations and the viewer{app} probe all declare variables no hermetic test reads, and only the ops with a live leg have ever had their request validated against the real schema.

What I did (conservative): fixed only the query the live failure named, and pinned only that one. I deliberately did not sweep every String! that looks like an id — the live error named $team alone, and issueUpdate(id:) genuinely is String! on Linear's schema, so a blanket sweep would have "fixed" a correct declaration and broken a working path. Breadth here needs the schema as the oracle, not pattern-matching.

Your call, not mine: whether to (a) leave it — the live oracle now covers the transition ops and future ops get legs as they land; (b) file a follow-up to extend live-leg coverage to the remaining Linear ops that have none; or (c) validate every query against Linear's published schema in CI, which catches the whole class at build time but adds a schema-fetch dependency to the gate.

I have not filed anything for (b)/(c) — that is a scope decision. Also recorded in my tracker.

Nothing else outstanding: RIG-3331 T7 was the last unbuilt task on the record, and T0–T6/T8 are all verified on main.

@trunk-io
trunk-io Bot merged commit e09bd6a into main Sep 16, 2026
15 checks passed
@trunk-io
trunk-io Bot deleted the compass-forge/rig-3331-t7-live-oracle branch September 16, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants