test(forge): live-oracle legs for the state-transition op (RIG-3331) - #1237
Conversation
Closes T7 of the forge state-transition record: the DL-210 tier-2 live-contract
coverage the op shipped without. The 13 committed transition fixtures had golden
replay only, and no live leg mentioned transition at all.
Ten live legs, one per live-reproducible fixture, plus a cross-op scenario on
both providers: create -> close -> an INDEPENDENT GetIssue asserting the state
changed -> reopen -> assert reopened. The separate read is the point -- a
single-call fixture compare pins one response shape and cannot show the write
took effect.
Three fixtures stay golden-replay-only and are named in the file with their
reason: each needs live repo or team state a test cannot create for itself
(a merged PR; two workflow states sharing a name; two completed-type states).
The PR-transition legs assert the state fold directly instead of comparing to
the fixture, because ghPullDetail populates Changed{Files,Additions,Deletions}
and those are not in volatileFields -- a live seeded PR cannot reproduce the
pinned counts, so a fixture compare would red on data the transition does not
control.
Every Linear by-name leg DISCOVERS its target workflow-state name from the live
team rather than hardcoding one. A hardcoded name on a team that lacks it fails
down the unknown-name path, which would have passed the type-contradiction
assertion for the wrong reason.
No capture specs for the error fixtures: deriveFixtureHalves derives Want from a
successful decode and t.Fatalfs on error, so rejection fixtures stay
hand-written -- consistent with every existing wantError fixture.
Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
😎 Merged successfully - details. |
|
Compass engineering docs preview: https://compass-forge-rig-3331-t7-li.compass-eng-docs.pages.dev Deployed from |
The live oracle T7 added found this on its first run: every Linear transition
400s against the real API.
http 400: Variable "$team" of type "String!" used in position expecting
type "ID".
`workflowStatesQuery` declares $team as String! and lands it in an id
comparator (`filter: {team: {id: {eq: $team}}}`), which Linear types as ID.
Every Linear state transition failed before reaching the mutation, so the whole
RIG-3331 Linear write path was dead on arrival in production.
Golden replay could never catch this. A fixture answers OUR request, so a
malformed query replays green forever against its own recorded response -- the
prelude entries pin the response bodies, never the query text. Tier 1 proved the
decode; only tier 2 could prove the request.
The mutation's `$id: String!` is deliberately left alone: Linear types
issueUpdate's id argument as String!, and the live error named only $team. The
fixed legs now reach the mutation, so the next oracle run proves that half
rather than my reading of it.
TestLinearIDFilterVariablesAreDeclaredID pins the declared type hermetically.
Verified red before the fix (mutant vet-clean first, then the assertion fails
naming the String! text) and green after.
Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
The oracle found a production bug on its first run. Second commit fixes it. What T7 caughtEvery Linear transition 400s against the real API:
Why tier 1 could never catch itA golden fixture answers our own request. The prelude entries pin response bodies, never the query text — so a malformed query replays green against its own recorded response, forever. Tier 1 proved the decode was right; only tier 2 could prove the request was. That is DL-210's two-tier rationale demonstrating itself on the first run, which is worth more than the coverage itself: this op merged through three PRs, a design freeze, and a review loop with its Linear half non-functional, and nothing in the hermetic pyramid could have said so. What I deliberately did not changeThe mutation's Regression pin
Evidence
(plus the five GitHub legs). The PR is now two commits: the T7 coverage, and the fix it found. I kept them separate so the fix is reviewable on its own and the sequence stays legible. |
Review finding on the T7 legs: liveStateNameOfType's comment claimed its skip was caught by CI's assert-ran guard. It was not. The guard greps only the two CREDENTIAL-unset literals, and a package with a skipped test still reports ok -- so a testbed team missing a canceled- or started-type workflow state would have silently skipped both by-name legs with the required check staying green. The exact vacuous-green the surrounding guard exists to prevent, claimed as covered by a comment a future maintainer would trust. Fixed by making it true rather than by softening the comment: the skip now carries liveBoardStateSkipPrefix, a stable one-line literal, and the guard greps it on the same source-derived discipline as the credential skips (sed the constant out of the test file, fail loudly if the extraction comes back empty so the guard cannot drift into checking nothing). Verified the sed reads the new constant verbatim. Also from review, all documentation precision: - the PR-leg bypass rationale named only the diff stats; Draft diverges too (a closed draft stays a draft, the fixture pins false), so a narrower projection would not have worked either -- the fuller reason strengthens the bypass rather than excusing it; - the by-name leg now states what it does NOT prove: forge.Issue folds every closed type to "closed", so the landed column is unobservable live and the resolved stateId is pinned by the golden fixture instead; - the capture table said "one spec per committed fixture", untrue once five fixtures were deliberately left specless. Declined: dropping the inert CloseReason from the reopen capture spec. It is dropped by the provider on a reopen, and it keeps a regenerated fixture byte-identical to the committed one; removing it means editing the fixture in the same breath, which is a separate change from adding coverage. Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
Review loop closed: 0 high, 1 medium, 4 low. Medium + three lows fixed in The medium was my own comment lying about a safety net
Fixed by making the claim true rather than softening it: the skip now carries Worth naming the shape: I added a test whose failure mode was silence, and wrote a comment claiming infrastructure caught it. Only reading the guard's actual Three lows, all documentation precision
DeclinedDropping the inert Independently confirmed by the reviewerWorth recording, since these were the calls most likely to be wrong: no fixture-compare leg is vacuous (every residue is non-empty after Gates at |
Open Questions (parked for Matt — overnight mode)Ready for review; green; nothing blocking. One fork I took conservatively rather than deciding for you. The Linear write path was broken in production and nothing outside this PR would have told us. The The fork: the same blind spot covers every Linear GraphQL query, not just the one that happened to break. What I did (conservative): fixed only the query the live failure named, and pinned only that one. I deliberately did not sweep every Your call, not mine: whether to (a) leave it — the live oracle now covers the transition ops and future ops get legs as they land; (b) file a follow-up to extend live-leg coverage to the remaining Linear ops that have none; or (c) validate every query against Linear's published schema in CI, which catches the whole class at build time but adds a schema-fetch dependency to the gate. I have not filed anything for (b)/(c) — that is a scope decision. Also recorded in my tracker. Nothing else outstanding: RIG-3331 T7 was the last unbuilt task on the record, and T0–T6/T8 are all verified on |
Closes T7 of the frozen forge state-transition record (
docs/designs/server/compass-forge-state-transition/design.md), the last unbuilt task on RIG-3331.Why this existed
The state-transition op shipped with tier-1 coverage only. 13 transition fixtures sit in
go/internal/forge/testdata/, butlivegithub_test.gohad 15TestLive*functions and not one of them mentioned a transition —git grep -c -i transition -- livegithub_test.goreturned nothing. DL-210 requires both tiers, and the record's own acceptance names//go:build livegithubcoverage explicitly.Found while verifying whether RIG-3331 could be closed after its two children (RIG-3517/RIG-3518) landed. A parent whose children are all Done is not therefore Done: the children covered T0–T6, and T7 had no child to close.
What landed
Ten live legs, one per live-reproducible fixture, plus a cross-op scenario on both providers: create → close → an independent
GetIssueasserting the state changed → reopen → assert reopened. The separate read is the whole point; a single-call fixture compare pins one response shape and cannot show the write took effect.Three judgment calls
Three fixtures stay golden-replay-only, named in the file with their reason — each needs live repo or team state a test cannot create for itself:
transition_pull_request_reopen_merged(a merged PR),transition_issue_duplicate_name(two states sharing a name),transition_issue_ambiguous_default(two completed-type states). "Every fixture has a matching leg" therefore resolves to a leg or a stated reason; silently omitting them would leave the next reader to re-discover the gap.The PR-transition legs assert the state fold directly rather than comparing to the fixture.
ghPullDetailpopulatesChanged{Files,Additions,Deletions}(the fixture pins{3,10,2}) and those are not involatileFields, so a live seeded PR would red the compare on data the transition does not control.Every Linear by-name leg discovers its target workflow-state name from the live team instead of hardcoding one. This one is load-bearing: a hardcoded name on a team that lacks it fails down the unknown-name path, which would have satisfied the type-contradiction assertion for entirely the wrong reason — a test passing while proving nothing. The helper skips when the team has no state of the needed type, and CI's assert-ran guard turns that skip into a red.
No capture specs for the error fixtures
deriveFixtureHalvesderivesWantfrom a successful decode andt.Fatalfs on error, so rejection fixtures are hand-written by construction. Consistent with every existingwantErrorfixture — none has a spec today.Verification
go vet -tags livegithub ./internal/forge/...clean;go build -tags livegithub ./...clean; untaggedgo test ./internal/forge/...passes (golden replay unbroken); all 12 new legs register and skip cleanly without credentials; unionmoon civsf4360703→ 27/27.The live legs cannot be executed here — no testbed credentials. They run on CI's forge live-contract oracle step.
Ledger-impact: none — T8's DL-342/DL-343 rows already describe this op; T7 adds test coverage only.