test(autoland): pin step execution behavior with golden scenarios

[micahjsmith]

Stacked PRs:

• docs(autoland): add the autoland plan for landing the step-abstraction stack #76
• feat(autoland): add an x step that runs a command and stops the land if it fails #75
• refactor(autoland): inject services everywhere and test with fakes instead of patches #74
• refactor(autoland): handle Ctrl+C once at the run boundary #73
• refactor(autoland): merge land runs as a stack through a plan optimization policy #72
• refactor(autoland): execute each step through a StepContext with injected services #71
• refactor(autoland): have steps describe themselves with a StepView #70
• refactor(autoland): give each plan step type its own class #69
• ->test(autoland): pin step execution behavior with golden scenarios #68

---

Summary

Phase 0 of the autoland step-abstraction stack: before execute_plan and its step rendering/persistence are refactored (an AutolandStep base class, StepView, injected services, plan optimization, cancellation), this pins what a user can observe of a whole autoland run today, so those refactors can show they change nothing.

Each scenario runs the real stack-pr autoland flow (run_autoland → fresh/resume/replan → engine.execute_plan) against a scripted GitHub and local clone, and compares five artifacts with golden files in tests/golden/autoland/:

• <scenario>.out.txt: the full console output (plain console, rich off), including what was typed at prompts
• <scenario>.checkpoint.json: the last checkpoint the run saved
• <scenario>.status.txt / <scenario>.status.json: autoland --status and --status -o json over that checkpoint
• <scenario>.plan.txt: that plan as the -i editor shows it; the test also checks it parses back (parse_plan) to the same text

Scenarios:

1. one_at_a_time: two PRs landed one at a time, with a workflow and a confirmation between them (checks pending at first, a review that comes in later, one wrong answer at the prompt)
2. merge_as_stack: a run of two l steps merged as a GitHub stack, then a workflow, a bare c, and a lone l
3. stack_merge_falls_through: the stack merge fails after landing only its bottom PR; the rest lands one at a time
4. checks_fail: checks still fail after their rerun, so the run aborts
5. workflow_fails: every deploy run fails, so the workflow checkpoint times out and the run aborts
6. confirm_eof: a confirmation with no terminal (EOF), so the run aborts
7. resume_mid_plan: --resume after a workflow timeout, in a plan with an already-landed prefix (l 101, l 102) and assumed-completed workflow/confirm steps
8. landed_prefix: a fresh run of a plan whose first PRs already landed
9. replan_carry_over: --replan with an edited plan file; finished checkpoints carry over, a new one runs, and a removed one is reported as not carried over

All scenario setup is in tests/autoland_scenarios.py, and every patch is in its _install_seams, so when the next commits move a seam, only that function needs re-pointing. The patches: runtime.console (the plain console, with scripted input()), runtime.HAVE_RICH, display.GLYPHS, runtime.resilient_sleep (advances the fake's clock instead of sleeping), and shell_commands.run_shell_command / cli.get_command_output, which a FakeWorld answers for gh and git. It also stubs cli.get_stack and cli.command_submit, pins git.get_current_branch_name, and makes any real subprocess fail the test. Because the fake sits at the subprocess boundary, the real gh.GitHub client, the retry wrapper, and its $ gh ... echo all run. The golden artifacts come from real output and the real checkpoint file, so they don't depend on how anything is patched. Temp paths become <TMP> and save times <SAVED>; SHAs are deterministic.

To regenerate after an intended behavior change (then review the diff):

UPDATE_GOLDEN=1 uv run --no-sync pytest tests/test_autoland_golden.py

No src/ changes.

Test plan

• New tests/test_autoland_golden.py: 9 golden scenarios, about 0.2s total, no real sleeps or subprocesses
• Regenerating twice gives byte-identical golden files (deterministic)
• Passes with gh unauthenticated (GH_CONFIG_DIR=$(mktemp -d) GH_TOKEN=)
• Editing a golden file makes its scenario fail, with a message pointing at UPDATE_GOLDEN=1
• Reviewed every golden output by hand; it reads like real autoland output

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com