Skip to content

feat(record): run recording groundwork (bundle, HTTP cursor, redaction, viewer, step times) - #470

Merged
pbertsch merged 7 commits into
mainfrom
feat/run-recording-foundations
Oct 17, 2026
Merged

pbertsch merged 7 commits into
mainfrom
feat/run-recording-foundations

Conversation

@pbertsch

Copy link
Copy Markdown
Member

Summary

  • Adds the groundwork for run recording in internal/record:
    • a bundle writer (manifest, timeline, exchanges, widget trees, screenshots) with validated names and atomic 0600 writes;
    • an HTTP log cursor that waits for requests still in flight and counts exchanges the agent's ring evicted;
    • redaction for headers, URLs and bodies, including numeric card numbers and PINs;
    • an offline viewer that renders a bundle without a server. Bundle values go in as text only, and a content security policy blocks network requests.
  • Adds StepEvent.At, the wall-clock time each step event was emitted.
  • No command or agent behaviour changes. Nothing is recorded yet: the runner hook and probe test --record come in a follow-up PR.
  • Screenshot blackout will reuse internal/redact, which this PR does not change.
  • The CHANGELOG links to the design doc in docs: run recording PRD (design only, no behaviour change) #469. Merge docs: run recording PRD (design only, no behaviour change) #469 first so that link resolves.

One existing expectation changed: redact_test.go used to expect "token": 123 to stay unredacted. It now expects "<redacted>", because numeric tokens, PINs and card numbers are credentials.

Test plan

  • go test ./... passes on the root module
  • go build ./... passes
  • gofmt -l is clean for internal/record and the two changed runner files
  • go vet ./internal/record/ ./internal/runner/ is clean
  • Open internal/record/viewer/index.html in a browser against a real bundle: check the layout at phone width and in dark mode, and the screenshot paths
  • CI passes, including the disclosure guard

gofmt -l internal/runner lists seven files that were already unformatted on main (ai_text_test.go, executor_ai_test.go, http_steps.go, http_steps_test.go, reporter.go, runner.go, runner_test.go). This PR does not touch them.

🤖 Generated with Claude Code

Adds the internal/record package foundations for run recording. The cursor reads probe.http_log without skipping requests that are still in flight, detects app restarts and counts exchanges the agent's ring evicted. The bundle writer lays out manifest, timeline, exchanges, widget trees and screenshots with validated names and atomic 0600 writes. Nothing is wired into the runner or the CLI yet.

Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
Step events carry the wall-clock time they were emitted, so traffic from outside the agent can be placed in a step. The redaction helpers remove credential headers, query values and body values, including numeric card numbers and PINs, and keep everything else unchanged. Both pieces are internal; nothing is wired into the CLI yet.

Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
Renders a recording bundle as one HTML page that opens from disk with no server. The manifest and timeline are inlined with script-safe escaping, every bundle value is inserted as text, and a content security policy blocks network requests.

Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
@pbertsch
pbertsch requested a review from a team as a code owner October 16, 2026 19:01
The viewer cannot read files once it is opened from disk, so a page built from the timeline alone would show exchange IDs and no widget trees. WriteViewer now inlines the exchanges and trees a timeline names, and refuses unsafe IDs and missing files. Checked in a browser against a sample bundle: steps, screenshots, redacted bodies, widget trees and failures all render, with no console errors.

Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
@pbertsch

Copy link
Copy Markdown
Member Author

Browser check of the offline viewer, against a generated sample bundle served from loopback, at desktop width in light mode:

  • Steps list with status badges, before and after screenshots, the interaction target, and the widget tree all render.
  • Request and response bodies show <redacted> where credentials were.
  • A failed step shows its error, its 500 response and its widget tree.
  • No console errors on load.

Not verified: phone width and dark mode. The browser tool could not resize the window, and the viewer's Content-Security-Policy blocks framing, so the page cannot be checked inside an iframe.

Found and fixed in c0f4b11: a page built from the timeline alone showed exchange IDs and no widget trees. WriteViewer now inlines them.

Follow-up: an empty request body renders as a blank block rather than a label.

🤖 Generated with Claude Code

The disclosure guard flags complete secret-shaped tokens in added lines. The three Stripe and webhook fixtures are split across string pieces, as internal/hosted/scrub.go already does for its webhook pattern. The tests still exercise the same rules.

Claude-Session: https://claude.ai/code/session_01QGnLBdCp6x94qKGY5kAxHK
@pbertsch
pbertsch merged commit 68b9d46 into main Oct 17, 2026
17 of 18 checks passed
@pbertsch
pbertsch deleted the feat/run-recording-foundations branch October 17, 2026 05:19
pbertsch added a commit that referenced this pull request Oct 17, 2026
… test

## Summary

- Adds `probe test --record` and `--record-dir`. Each test attempt gets a bundle under `<reports_folder>/recordings/<run>/<device>/`: its steps with source file and line, screenshots before and after each step, a widget tree after each step, the app's `dart:io` HTTP exchanges with credentials redacted, and an offline HTML viewer.
- Adds a blocking per-step recorder hook to the runner, separate from `OnStep`. It is nil by default, and with no recorder configured the runner's behaviour is unchanged.
- Adds `internal/record.Recorder`, which writes one bundle per test attempt. Exchange files are numbered per bundle, so an app restart cannot overwrite an earlier exchange.
- Recording is an observer. Capture failures become warnings and never change a result or the exit code. Nothing is uploaded.
- Docs: the CLI reference, the Probe Cloud preview note, the README, the reports page, the landing feature card, the MCP `run_tests` flags, and the CHANGELOG.

Builds on #469 (design) and #470 (bundle writer, HTTP cursor, redaction, viewer, step start times). Both are merged.

## Not recorded yet (stated in the docs)

- Native SDK and WebView traffic. Only the app's `dart:io` HTTP is captured.
- The element each step acted on.
- Screenshot redaction rules. Screenshots are not redacted.
- Widget trees are captured after steps, not before them.
- The manifest's `platform` and `os_version` fields are empty. The CLI does not pass them yet.

## Review notes

- The recorder numbers exchange files per bundle. Using the agent's sequence number would overwrite files after an app restart. A regression test covers this.
- Composite runs: the composite runner is rebuilt when recording is on, with the same timeout and verbosity as `buildCompositeRunner`, plus the recorder factory. `buildCompositeRunner` registers no recipes, so nothing else is lost.
- Local runs do not set a device name in the run metadata, so the recording directory falls back to the `--device` serial.
- `internal/cli/test.go` was run through gofmt, so some untouched alignment lines changed too. Those hunks are whitespace only.
- `probe record` (the interaction recorder) is a different command from `probe test --record`. The docs name both.

## Test plan

- [x] `go test ./...` passes on the root module
- [x] `go build ./...` passes
- [x] `gofmt -l` is clean for the changed files, and `go vet` is clean for `internal/cli`, `internal/mcp`, `internal/record` and `internal/runner`
- [x] New tests: bundle writing, redaction, retries and name collisions, device errors, the app-restart regression, runner hook ordering and error isolation, and the CLI event conversion and device slug
- [x] `probe test --record` run on an iOS simulator and an Android emulator with the project's end-to-end fixture app (5 tests on each). The fixture makes no HTTP calls, so no `http/` folder is written
- [ ] Open a bundle in a browser at phone width and in dark mode. Not done yet
- [x] CI passes, including the disclosure guard
- [ ] Review

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant