Repository navigation
docs: run recording PRD (design only, no behaviour change) - #469
Merged
Merged
Conversation
Design for recording each test run step by step: the HTTP traffic each step caused, screenshots and widget trees, the target element, a self-contained viewer, a replayable test, native capture through a device proxy with a per-run CA, and credential redaction. Opt-in by design; no product behaviour changes in this PR. Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
The agent returns only finished HTTP exchanges. Advancing the cursor to the highest seq returned could skip a request that finishes later, so the PRD now says the cursor stops at the first unfinished seq. Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
This was referenced Oct 16, 2026
Merged
pbertsch
added a commit
that referenced
this pull request
Oct 17, 2026
… test ## Summary - Adds `probe test --record` and `--record-dir`. Each test attempt gets a bundle under `<reports_folder>/recordings/<run>/<device>/`: its steps with source file and line, screenshots before and after each step, a widget tree after each step, the app's `dart:io` HTTP exchanges with credentials redacted, and an offline HTML viewer. - Adds a blocking per-step recorder hook to the runner, separate from `OnStep`. It is nil by default, and with no recorder configured the runner's behaviour is unchanged. - Adds `internal/record.Recorder`, which writes one bundle per test attempt. Exchange files are numbered per bundle, so an app restart cannot overwrite an earlier exchange. - Recording is an observer. Capture failures become warnings and never change a result or the exit code. Nothing is uploaded. - Docs: the CLI reference, the Probe Cloud preview note, the README, the reports page, the landing feature card, the MCP `run_tests` flags, and the CHANGELOG. Builds on #469 (design) and #470 (bundle writer, HTTP cursor, redaction, viewer, step start times). Both are merged. ## Not recorded yet (stated in the docs) - Native SDK and WebView traffic. Only the app's `dart:io` HTTP is captured. - The element each step acted on. - Screenshot redaction rules. Screenshots are not redacted. - Widget trees are captured after steps, not before them. - The manifest's `platform` and `os_version` fields are empty. The CLI does not pass them yet. ## Review notes - The recorder numbers exchange files per bundle. Using the agent's sequence number would overwrite files after an app restart. A regression test covers this. - Composite runs: the composite runner is rebuilt when recording is on, with the same timeout and verbosity as `buildCompositeRunner`, plus the recorder factory. `buildCompositeRunner` registers no recipes, so nothing else is lost. - Local runs do not set a device name in the run metadata, so the recording directory falls back to the `--device` serial. - `internal/cli/test.go` was run through gofmt, so some untouched alignment lines changed too. Those hunks are whitespace only. - `probe record` (the interaction recorder) is a different command from `probe test --record`. The docs name both. ## Test plan - [x] `go test ./...` passes on the root module - [x] `go build ./...` passes - [x] `gofmt -l` is clean for the changed files, and `go vet` is clean for `internal/cli`, `internal/mcp`, `internal/record` and `internal/runner` - [x] New tests: bundle writing, redaction, retries and name collisions, device errors, the app-restart regression, runner hook ordering and error isolation, and the CLI event conversion and device slug - [x] `probe test --record` run on an iOS simulator and an Android emulator with the project's end-to-end fixture app (5 tests on each). The fixture makes no HTTP calls, so no `http/` folder is written - [ ] Open a bundle in a browser at phone width and in dark mode. Not done yet - [x] CI passes, including the disclosure guard - [ ] Review 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/prd/run-recording.md, the design for recording a test run step by step: the HTTP traffic each step caused, screenshots and widget trees, and the element each step acted on.No code or behaviour change. The design is for review before implementation starts.
Test plan
main(docs/prd/ai-visual-assertions-prd.md,docs/roadmap/gap-analysis-2026-08.md)🤖 Generated with Claude Code