Skip to content

feat(ios): drive ASWebAuthenticationSession sign-in sheets in place (#2438) - #2448

Open
thymikee wants to merge 16 commits into
mainfrom
claude/agent-device-issue-2438-b5f950
Open

feat(ios): drive ASWebAuthenticationSession sign-in sheets in place (#2438)#2448
thymikee wants to merge 16 commits into
mainfrom
claude/agent-device-issue-2438-b5f950

Conversation

@thymikee

@thymikee thymikee commented Sep 10, 2026

Copy link
Copy Markdown
Member

Summary

iOS apps that sign in via ASWebAuthenticationSession present the identity provider in com.apple.SafariViewService, out of the app's process. Two facts, both verified live on the iOS 26.2 Simulator, made these flows unautomatable: activating or launching the host cancels the auth session, and the host AX bridge cannot see the sheet because the app stays the AX primaryApp.

This serves and drives the sheet in place:

  • A closed registry names the host (contracts/fixtures/ios-system-surface-hosts.json, mirrored by the TypeScript and Swift registries under a parity test).
  • The runner reads and drives the host without activation and never adopts it as the session target, so the session stays bound to the app.
  • The Simulator route detects a running host with a cheap device-scoped ps probe and takes the runner path; the bridge would serve the occluded app tree as if healthy.
  • open com.apple.SafariViewService is refused (UNSUPPORTED_OPERATION); captures carry a system-surface disclosure.

Presence is foreground state, not tree content — a torn-down host serves a richer tree than a live one — and the never-activate guard is what keeps that predicate sound, which also makes issue #2438's stale-tree bug unrepresentable for this flow. Design rationale in the ADR 0004 amendment.

17 files, Apple platform + contracts + docs only. Closes #2438.

Validation

Tested at 0cde7a4d2e. pnpm check:affected --run passes (40/40; lint, typecheck, layering, fallow, unit). Swift: runner builds + 2 registry parity tests pass.

Live on iPhone 17 / iOS 26.2 (real daemon + runner, an ASWebAuthenticationSession probe app + local IdP): open svs refused with the sheet intact; wait label="LOGIN"press role=textfield label="Email" + typefill role=securetextfield (values confirmed) → press role=button label="LOGIN" fired the auth callback and returned to the app; post-dismiss captures revert to the bridge with no disclosure.

Follow-ups: #2449 (test-app fixture + CI lane), #2450 (verify/settle lineage guard), #2451 (selector-route disclosure parity), #2452 (ADR 0011 offscreen comment).

Caveat: the first snapshot after a cold runner can return 0 nodes (pre-existing mid-bundle flake); use wait stable/wait <element>.

…2438)

iOS apps that sign in via ASWebAuthenticationSession present the identity
provider in com.apple.SafariViewService, out of the app's process. Two facts,
both verified live on the iOS 26.2 Simulator, made these flows unautomatable:
activating or launching the host cancels the auth session, and the host AX
bridge cannot see the sheet because the app stays the AX primaryApp.

Serve and drive the sheet in place. A closed registry names the host (shared by
the TypeScript and Swift sides under a parity test); the runner reads and drives
it without activation and never adopts it as the session target; and the
Simulator route detects a running host with a cheap device-scoped ps probe and
takes the runner path, since the bridge would serve the occluded app tree as if
healthy. open refuses to launch a registered host, and captures carry a
system-surface disclosure.

Presence is foreground state, not tree content: a torn-down host serves a richer
tree than a live one, so content heuristics cannot tell them apart. The
never-activate guard is what keeps the foreground predicate sound, which also
makes the stale-tree failure mode unrepresentable for this flow.

Closes #2438
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.53 MB 4.55 MB +14.8 kB
Package (unpacked) 4.53 MB 4.54 MB +14.8 kB
Package (download) 1.34 MB 1.35 MB +4.0 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 25.8 ms 25.8 ms +0.0 ms
CLI --help 72.4 ms 73.0 ms +0.6 ms

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-2448/

Built to branch gh-pages at 2026-09-11 14:14 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@thymikee

Copy link
Copy Markdown
Member Author

There are still correctness gaps at 0cde7a4.

The presence probe caches absence for one second and treats ps failures as absence. A sheet opened just after an app capture, or while ps fails, therefore falls through to the bridge and returns the occluded app tree. Cache only positive presence, route unknown results to the runner, and test absent-to-present within the TTL plus probe failure.

The explicit-URL branches in openIosApp return before the new host guard. open com.apple.SafariViewService https://example.com can still launch the host and cancel authentication. Apply the refusal before every resolved-host launch or termination, and test the URL variants.

The provenance is reduced to a warning instead of reaching comparison identity. App and sheet captures can therefore enter legacy same-presentation matching, including recorded-tap failure corroboration. #2450 needs to land with this change, or comparison across this boundary must be explicitly refused. Selector responses also drop the disclosure because they only read Android systemSurfaceOnly; include #2451 or narrow the advertised contract explicitly.

Coverage fails on a related import-closure regression: app-lifecycle-facade grows from 120 to 121 modules through the new static import. Fix the import boundary without raising the budget. The iOS failure is an initial fixture-app capture stall, which looks unrelated to the auth route.

Please itemize the +8.1 kB unpacked growth and briefly record the smaller design considered. The reported live sign-in flow is useful evidence, but does not cover the failure paths above.

Presence probe: absence and probe failure are no longer reported as "no
surface". The probe returns present/absent/unknown and the route takes the
runner for anything but a proven absent, so a sheet opened between two captures,
or a probe that cannot answer, can no longer fall through to a bridge capture
that would answer confidently from the occluded app tree. Only a positive
observation is memoized. The probe now matches with pgrep and reads only a
matched pid's environment, which is ~3x cheaper than the previous full
process-environment dump and stops copying every process's environment.

Open guard: the refusal moved to every resolved-host launch and terminate, so
the URL, deep-link and launch-args branches that returned before the old check
can no longer launch the host. Terminating a host is refused too, since that
cancels the presented session just as launching it does.

Comparison: the surface identity now reaches SnapshotState, and tap-failure
corroboration refuses outright when a baseline and a post-action capture
disagree about it, instead of letting app and sheet captures meet in legacy
same-presentation matching. Selector routes disclose an iOS system surface
through the shared disclosure seam rather than reading only the Android field.

The contracts import in the launch path is deferred so the app-lifecycle
facade's eager closure stays flat, and the runner's comment prose is trimmed
because apple/runner ships to npm as uncompiled source.
The probe shelled out with runCmd, so every eligible capture spawned a real
process even in provider-backed tests that stub the Apple tool seam — 17 real
spawns in one scenario file, which is both wasted work and added latency on
timing-sensitive settle paths. It now goes through runAppleToolCommand like the
sibling ps probe, so a stubbed provider answers instead of spawning.
Routing an unprovable probe to the runner is right, but the early return also
skipped runFallback, so the response lost its warning and kept an identity that
could still be compared against a bridge publication. An unknown probe now falls
back through the same disclosed path as a bridge failure, with its own reason.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed at 00e4185. All four gaps were real; two more surfaced while verifying them.

Presence probe. It now returns present/absent/unknown, and the route takes the runner for anything but a proven absent. Only a positive observation is memoized, so a sheet opening between two captures is seen by the very next one, and a probe that cannot answer no longer reads as absence. Tests cover absent→present inside the TTL, unknown→present inside the TTL, and each failure mode (scan failure, unreadable environment, non-"no match" exit). It also stopped dumping every process's environment: pgrep -f first, then a single ps eww on a matched pid only. ~25 ms vs ~170 ms, so re-probing every non-present capture is affordable.

Open guard. Moved to every resolved-host launch and terminate, so the URL, deep-link and launch-args branches that returned before the old check are covered — parameterized tests assert all four plus that no simctl command ever names the host bundle. Terminate is refused too, since that cancels the session just as launching does.

Provenance. #2450 and #2451 are folded in rather than deferred. The surface identity reaches SnapshotState as iosSystemSurfaceBundleId; hasMatchingPresentation refuses outright when baseline and post-action disagree, so app and sheet captures cannot meet in legacy matching; and withSystemSurfaceDisclosure emits the iOS disclosure on the shared selector seam instead of reading only the Android field. Regression tests for both.

Import closure. The contracts import in the launch path is deferred, so app-lifecycle-facade stays at 120. No budget raised.

Two more found while verifying. The probe used runCmd instead of the Apple tool seam, so it spawned real processes even in provider-backed tests (17 in one scenario file); it now goes through runAppleToolCommand like the sibling ps probe. And routing an unprovable probe to the runner had returned early past runFallback, silently dropping the warning and leaving a comparable identity — it now falls back through the disclosed path with its own reason.

iOS smoke. Pre-existing, not this PR. The identical signature (wait_capture_stalled, readableCaptures: 0, one poll consuming the deadline, same step and scenario) occurs on main before the probe existed — runs 34400074702 (main, 09-09, overshoot 7 ms) and 34226333904 (main, 09-08, overshoot 1341 ms) versus 608 ms here. It is already tracked as #2343, whose own measurements account for ~9.2s inside the capture with no probe present, and that job fails on main at ~39% (7/18 recent runs). A 3s-bounded probe cannot produce a 10.6s single-capture stall. Caveat kept honest: the probe does add unconditional latency, so it can nudge a case that already had 7 ms of margin — which is part of why it is now ~7x cheaper.

Size, itemized (measured at the reviewed commit). +4,273 B is Swift: apple/runner/ ships as source into dist/apple/runner/, so RunnerSystemSurfaceHostPolicy.swift (1,448 B post-strip), RunnerTests+CommandExecution.swift (+2,302 B) and RunnerTests+Models.swift (+523 B) all land in the tarball; the #if AGENT_DEVICE_RUNNER_UNIT_TESTS block (1,596 B) was correctly stripped. +1,964 B is TypeScript: the new ios-system-surface.js chunk (808 B, ~570 B of it the two agent-facing strings), the probe (531 B), 476 B of inline hunks, 149 B of cross-chunk imports. Tests, the fixture, the ADR, the subpath export and the layering snapshot add 12.4 kB of source and ship zero bytes; public .d.ts is unchanged. The ~1.9 kB gap to 8.1 kB is diffuse minifier chunk-graph churn, which is why download moved only +1.7 kB.

Smaller design considered. Collapsing the contracts module into an existing one saves only ~200 B, because most of that chunk is the irreducible refusal/disclosure strings — not worth conflating the registry with an unrelated module. The real lever was Swift comment prose, which ships uncompiled: I trimmed the new blocks to pointers at the ADR/TS twin. That also turned up a repo-wide finding — comments are 16.6% of the 446 kB of shipped runner Swift (~74 kB) — filed as #2461 with a PR in flight.

Gates: pnpm check:affected --run green at 00e4185.

@thymikee

Copy link
Copy Markdown
Member Author

Still needs changes at 00e4185:

  • A successful ps read without SIMULATOR_UDID is treated as absence. Missing or ambiguous device scope must remain unknown; otherwise a live sheet can be routed to the occluded app tree. Add a missing-environment regression.
  • Generic post-gesture comparison still uses comparisonKey or the backend, without the new system-surface identity. App and sheet XCTest captures can therefore remain comparable. Carry surface identity through the shared comparison path and cover sheet appearance/dismissal in stabilization, verify and settle.
  • Mutating find drops iosSystemSurfaceBundleId in find-target-capture.ts, so the shared disclosure helper cannot report the sheet. Preserve it and test both found and not-found responses.

The current size report is +10.6 kB unpacked, while the explanation accounts for +8.1 kB. Please reconcile the remaining growth against this head. CI is green, but these code gaps remain.

…cope

A ps read that carries no SIMULATOR_UDID at all was reported as absence, so an
unreadable or truncated environment could route a live sheet to the occluded app
tree. Only a scope naming a different device is a real negative now; a missing
one stays unknown.

The shared post-gesture comparison token used comparisonKey or the backend
alone, so an app capture and a sheet capture — both XCTest — compared equal and
a sheet appearing or dismissing read as a stable surface. The token now carries
the surface, which covers stabilization, verify and settle through the one path
they share.

Mutating find rebuilt its capture without iosSystemSurfaceBundleId, so the
shared disclosure helper could not report the sheet on either outcome. It is
preserved now.

Each fix has a regression that fails without it.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed at 2200150. All three code gaps were real, and each fix has a regression that fails without it (verified by reverting the fix and watching the new test go red).

Probe device scope. A successful ps read carrying no SIMULATOR_UDID at all was reported as absence. Now only a scope naming a different device is a real negative; a missing or unreadable scope stays unknown and takes the runner. Two regressions: an environment with no scope token, and an empty read.

Generic post-gesture comparison. You were right that this was still open — snapshotComparisonKey returned comparisonKey ?? backend, so an app capture and a sheet capture, both XCTest, compared equal on backend alone. The token now carries the surface identity, which covers stabilization, verify and settle through the one path they share. Regressions for both directions: a sheet appearing mid-poll and a sheet dismissing mid-poll, each asserting a re-baseline rather than a verdict.

Mutating find. find-target-capture.ts rebuilt its capture without iosSystemSurfaceBundleId, so the shared helper could not see it. Preserved now, with tests on both the found and the not-found outcome plus a direct helper test.

Size, reconciled against this head. Reproducing CI's own definition locally (tsdown + package-apple-runner-source.mjs + npm pack --json unpackedSize), +10.8 kB is +10,778 B: dist/src/** +7,162 and dist/apple/runner/** +3,616, with every other shipped path at exactly 0.

Of that, +8,908 B is this branch's own code measured against the merge-base — Swift +3,122 (RunnerSystemSurfaceHostPolicy.swift +1,016 new, RunnerTests+CommandExecution.swift +1,798, RunnerTests+Models.swift +308) and dist/src +5,786, the largest items being sdk-selectors.d.ts +1,770, runtime4.js +1,393, the new ios-system-surface.js chunk +1,041, interactor.js +410, sdk-contracts.d.ts +348, app-launch.js +285. The remaining +1,870 B is base drift: main shrank by exactly that between the merge-base and this PR's base sha, so the diff against a smaller base reads larger by the same amount.

The 8.1 → 10.6 → 10.8 walk is the review fixes, per commit: 1d253c4c +3,457 B of dist/src while giving back −1,151 B of Swift comment prose, then −12, +159, +184. 8,141 + 2,637 = 10,778 exactly.

Correction to my earlier comment: the "~1.9 kB of diffuse minifier churn" I claimed does not exist. That figure was a base mismatch — I itemized against the merge-base and compared it to a CI number measured against the PR base sha. Rebuilding the same sha in two independent directories yields byte-identical dist/src totals, so the build is deterministic and the residual after this itemization is 0 B. One genuine item I had also missed: .d.ts keeps doc comments even though the minified .js drops them, so the new contracts module costs +2,118 B of declarations — that, not churn, is where the surprise was.

Gates: pnpm check:affected --run green at 2200150.

@thymikee

Copy link
Copy Markdown
Member Author

The probe and mutating-find fixes are addressed at 2200150, and the size accounting now reconciles. The verify/settle gap remains: the new key only protects deferred post-gesture stabilization. captureVerifyEvidence and settleEvidence still compare node digests, while readSettledOutcome diffs node-only baselines; none compares the captured surface identity. Carry that identity through these actual command paths and test app-to-sheet and sheet-to-app transitions through --verify, --settle, and their combination. The new deferred-stabilization tests do not exercise those routes.

thymikee added a commit that referenced this pull request Sep 10, 2026
An eligible iOS simulator snapshot is served by the host AX bridge
(packages/platform-apple/src/snapshot-route.ts), which never reaches the
runner's prepareActiveCommandContext. The rescue's direct querySelector always
does, so the two requests share that surface policy only when the capture is
runner-routed too - which is the case #2448 forces for the system surface.
Keep the unchanged statement that no surface identity crosses the two requests.
`--verify` compared node digests and `--settle` diffed node-only baselines, so an app
baseline and an in-place system-surface capture (a web sign-in sheet) were treated as one
presentation: a meaningless changed verdict, and a whole-surface replacement presented as an
in-surface diff with refs.

The pre-action baseline now travels with the surface its capture described, from the resolution
and the session frame through to the settled capture, and one module owns the comparison for
both routes. Across a surface change no same-surface claim is made: evidence reports the
transition instead of a digest comparison, the settled diff and its refs are withheld, and both
payloads disclose the transition.
@thymikee

Copy link
Copy Markdown
Member Author

Verify/settle gap addressed at 36fdde3.

Identity now travels with the baseline, not beside it. ResolvedInteractionTarget's pre-action fields are one member type (preActionNodes + preActionSurfaceBundleId), emitted from a single preActionBaselineFields helper in resolution.ts wherever nodes are taken — point evidence baseline, ref resolution (RefResolution carries the surface), selector capture, native-ref preflight, and the find handoff (PreresolvedInteractionTarget.iosSystemSurfaceBundleId, set from find's own target capture). SettleOutcome returns settledCapture: { nodes, surfaceBundleId } instead of bare settledNodes; resolveSettleBaseline keeps the surface on all three baseline sources (authorized ref frame, resolution evidence, session fallback), and the generic scroll/back route passes baselineSurfaceBundleId from the stored snapshot.

One module, src/commands/interaction/runtime/post-action-surface.ts, owns the comparison for both routes, so the refusal cannot hold on one and drop on the other.

What a cross-surface verify/settle now does

  • --verify: no digest comparison is made across the boundary. evidence.changedFromBefore reports the transition itself (the whole observed surface was replaced) and evidence.surfaceChange = { from, to, disclosure } says so — from/to are the host bundle id or app.
  • --settle: no settled diff is attached, so (since diff presence is what issues refs) no refs are issued and no tail either; settle.surfaceChange carries the same triple and hint states the transition plus "take a snapshot to read the current surface". The loop's own verdict (never settled, stalled, sparse, tiny tree) is still appended, not replaced.
  • --settle --verify: the shared final capture feeds settleEvidence, which routes through the same rule, so both payloads report one consistent transition.
  • Disclosure text reuses IOS_SYSTEM_SURFACE_DISCLOSURE when the sheet is now on screen; a new iosSystemSurfaceTransitionDisclosure adds the sheet-has-left sentence, since the standing present-tense one cannot say that.
  • The settled sheet still becomes the stored observation, and the non-hittable "may have had no visible effect" hint is dropped when a surface change is disclosed next to it.

Same-surface behavior is byte-identical: all 55 pre-existing verify/settle tests pass unchanged (two accessor renames in settle-transition-baseline.test.ts for settledCapture.nodes; assertions untouched).

New tests (6, beside the existing verify/settle tests) cover app→sheet and sheet→app through each route: --verify (both directions, in interactions.test.ts), --settle (both directions) and --settle --verify (both directions) in settle.test.ts. Two of them pin the premise that the sheet and app trees digest identically, so the old digest comparison would have reported "nothing changed" across a whole-surface replacement.

Revert-sensitivity — three separate reverts, each confirmed:

  1. comparison ignores surface identity → all 6 new tests fail, 55 existing pass;
  2. diff gating removed (!surfaceChange dropped) → the 4 settle-route tests fail;
  3. surface dropped from the baseline plumbing → exactly the 3 sheet→app tests fail (app→sheet passes by accident when the baseline surface is absent, which is why both directions are covered).

Gates on the pushed commit: pnpm format (oxfmt, clean), pnpm lint, pnpm typecheck, pnpm check:affected --run — all runnable checks passed (full set selected; vitest-related ran 774 files / 5879 tests). No device or simulator use.

@thymikee

thymikee commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

The verify/settle comparison fix looks correct at 36fdde3, including both transition directions and the combined press path. Two validation gaps remain: add an app-to-sheet and sheet-to-app regression through generic scroll/back --settle, where the baseline is passed separately, and exercise the new verify/settle transitions on the live sign-in flow. The earlier live sign-in evidence does not cover these new outputs. The generic-route test should fail if its surface identity is dropped and assert that no cross-surface diff or refs are returned.

Please also update the size breakdown for the current +14.8 kB report; the previous accounting covers +10.8 kB.

Coverage has now failed on a related test-size guard: settle.test.ts grew from 2,359 to 2,528 lines. Split the new surface tests along their owning module, preserving command-path coverage, rather than raising the limit. Native smoke checks are still running. The readiness hold is for the missing evidence, not CI alone.

@thymikee

Copy link
Copy Markdown
Member Author

Challenging the size of this one too — and I ran the spike rather than proposing a redesign on theory. There is a simpler mechanism that is both smaller and strictly more accurate than what this PR currently ships.

The claim. The host-side presence probe (system-surface-presence.ts, 123 lines + 137 of tests, a tri-state contract, a memo, a route branch, and a pgrep+ps fork/exec on every eligible capture) can be replaced by one AX call inside the bridge round trip that already happens.

Why the bridge can answer. +[AXElement primaryApp] disassembles to systemWideElement → systemApplication → currentApplications → firstObject. The bridge already calls that list three times per capture (SnapshotBridgeRuntime.m:319/326/337) and discards everything but index 0.

What I measured (iOS 26.2 simulator, real ASWebAuthenticationSession sheet, standalone probe built the same way the bridge is):

state host process today's probe says +[AXElement applicationAtCoordinate:]
no sheet absent absent → bridge com.callstack.authprobe
sheet presented running present → runner com.apple.SafariViewService
dismissed, host lingering still running present → runner com.callstack.authprobe

Two findings, one negative and one positive:

  • The free option is refuted. currentApplications stays count=1 with the sheet up — SafariViewService never appears in it. So "scan the list we already have" does not work, and I'd have shipped a broken simplification if I'd proposed it from the disassembly alone.
  • The hit test works and is better. applicationAtCoordinate: at the centre of the primary app's frame names the sheet's host exactly while it is presented, and names the app again the moment it is dismissed — while the host process is still running. That third row is the interesting one: it is a false positive the current probe has by construction, because process liveness is not presentation. Today every capture in that lingering window needlessly takes the slow path.

So the trade is not "smaller but rougher" — it is smaller, faster and more accurate:

  • deletes 260 lines, the tri-state, the memo and the processExecutable field (it exists only to feed pgrep -f; the guest works off bundleId, like the Swift registry already does)
  • removes a fork/exec per capture (~25 ms) in favour of one AX call already inside the round trip
  • fixes the lingering-host false positive
  • needs no protocol bump: a new error_code under the existing kind: 'unsupported' passes straight through bridgeFailureFromEnvelope, and editing the .m auto-invalidates the cached binary via the existing source hash

Two things that are genuinely not free, so I'm flagging them rather than hiding them: it needs one new early branch in snapshot-route.ts (routing the typed failure through fallbackAfterFailure would disabledGenerations.add(...) and permanently disable the bridge for that generation over a transient sheet), and it shifts the check to after resolveTarget, so while a sheet is up each capture pays a round trip instead of a pgrep. Still net cheaper, and only while a sheet is up. It would also close a hole the host probe never covered: createLaunchObservationProbe doesn't consult it at all, so open's launch observation can currently report observable off an occluded tree.

Separately, on shape. The surface identity has now spread across six comparison sites over four review rounds — tap corroboration, post-gesture stabilisation, find, the disclosure seam, and now verify/settle. Each round found another site that compared across the boundary. That reads less like a missing check and more like a defect that should be made unrepresentable: the comparison primitives take bare SnapshotNode[], so a cross-surface comparison is always constructible and every new call site has to remember to ask. The last commit moved in the right direction by giving post-action-surface.ts ownership for both routes. The end state I'd argue for is that the comparison primitives take a capture (nodes + surface) rather than a node array, so the boundary can't be crossed silently anywhere.

I haven't changed anything here — this PR works and is tested, and swapping the mechanism mid-review is your call, not mine. Happy to do it in this PR or as a follow-up that deletes the probe. The spike binary and method are reproducible if you want to see it yourself.

… mirror

The #2438 cross-surface cases were appended to `settle.test.ts`, taking it over the
test-file size ratchet (2528 lines, 2359 at the merge-base). They assert the
comparison `post-action-surface.ts` owns, so they move to that module's mirror test
file, and the device double plus the trees both files drive move to a sibling
fixtures module under `__tests__/` rather than being duplicated.

Pure move: every test and every assertion is unchanged, and `settle.test.ts` is back
under its merge-base length.
@thymikee

Copy link
Copy Markdown
Member Author

Fixed the Coverage job's test-file size ratchet in f27156c by taking the split it prescribes — no ratchet or baseline edit.

What moved where:

  • The four #2438 cross-surface settle tests (and their block-local WEB_SIGN_IN_SHEET_BUNDLE_ID, webSignInSheetSnapshot, WEB_SIGN_IN_SHEET_LABELS) now live in src/commands/interaction/runtime/post-action-surface.test.ts, mirroring the post-action-surface.ts module this PR added to own the cross-surface comparison for both the verify and settle routes.
  • The helpers both files need — createSettleDevice, buttonSnapshot, welcomeSnapshot, createFakeClock — moved verbatim to src/commands/interaction/runtime/__tests__/settle-device-fixtures.ts and are imported by both, so nothing is duplicated.

settle.test.ts is 2292 lines, under its 2359-line merge-base length; the new test file is 179.

Pure move: the 35 test names and all 155 assertion lines are byte-identical to 36fdde3ead (verified by diffing the sorted sets). Nothing was re-scoped, weakened, or deleted.

Gates on the pushed head: pnpm vitest run scripts/__tests__/test-file-size-ratchet.test.ts green; settle.test.ts (31) + post-action-surface.test.ts (4) green; pnpm format (oxfmt, no diff), pnpm lint, pnpm typecheck, and pnpm check:affected --run (fail-open full set, 58 checks) all pass.

@thymikee

Copy link
Copy Markdown
Member Author

The test split is clean at f27156c: the four transition tests retain their assertions and still exercise the command path. The previously requested generic-route regression, live verify/settle evidence and current size breakdown remain outstanding.

The reported AX hit-test spike is useful evidence for a smaller design, especially with a lingering host process. It has not been integrated or validated through the production bridge, so it is a proposal rather than a resolved gap. Please keep that design decision separate from the remaining validation.

…oute

`scroll --settle` and `back --settle` plumb the baseline's surface identity
through `baselineSurfaceBundleId`, but nothing asserted it: the generic route
had zero coverage of the #2438 refusal, so a regression there would have been
silent while the element-targeted route stayed green.

Assert the same contract the targeted route guarantees, in both directions and
for both commands: no diff is attached across an app/sheet boundary — therefore
no tail and no `refsGeneration` — the transition is disclosed, and the settle
observation still reports its own verdict alongside that disclosure.

Each direction falsifies a different half of the plumbing, so both are needed:
dropping the baseline's surface identity fails only the sheet-to-app tests (an
app baseline has no surface id to lose), and dropping the settled capture's
fails only the app-to-sheet tests. No production change: the plumbing was
correct, only untested.
@thymikee

Copy link
Copy Markdown
Member Author

Added the outstanding generic-route regression in 38c1d9fd43.

Routes now coveredsrc/daemon/__tests__/generic-settle.test.ts, four tests:

route direction
scroll --settle app → sheet
scroll --settle sheet → app
back --settle app → sheet
back --settle sheet → app

Each asserts the same contract the element-targeted route already guarantees in src/commands/interaction/runtime/post-action-surface.test.ts, through one shared expectCrossSurfaceSettle helper so the two routes cannot drift: surfaceChange is disclosed with the right from/to/disclosure (the departing-sheet sentence is distinct from the standing one), no diff and no tail are attached across the boundary, and therefore no refs — refsGeneration is absent and the frame the mutating leaf expired stays expired. The settle observation still reports its own verdict (settled, captures, quietMs, timeoutMs) alongside the disclosure. The back tests also check the refusal rides alongside back's own closed result shape, and the app→sheet tests check the settled sheet still becomes the stored observation — which is where the next command's baseline surface comes from.

Revert-sensitivity. Each direction falsifies a different half of the plumbing, which is why both are needed:

  1. Baseline surface removed from generic-settle.ts (both the surfaceBundleId on the baseline and baselineSurfaceBundleId on the runtime call) → 2 failed: scroll --settle ... sheet-to-app and back --settle ... sheet-to-app, both with surfaceChange undefined (so a diff was built across the boundary, refs included). The two app→sheet tests pass under this revert, and cannot fail it: an app baseline has no surface id to lose.
  2. Settled capture's surface removed (surfaceScopedNodes(...){ nodes } in settle.ts) → 2 failed: the two app-to-sheet tests. The sheet→app tests pass, since the baseline still disagrees with the after capture.
  3. The refusal itself removed (resolvePostActionSurfaceChangeundefined) → all 4 failed.

So no new test is vacuous, and production files were restored to their pushed state after each probe.

No production change. The regression found no gap — the generic route's plumbing was already correct, just untested. Test file only, 365 → 614 lines (tripwire 1,000).

Gates on the pushed commit: pnpm format (oxfmt), pnpm lint, pnpm typecheck, pnpm check:affected --run (40/40), and the test-file-size ratchet all pass. No device or simulator used.

@thymikee

Copy link
Copy Markdown
Member Author

The generic-route regressions at 38c1d9f address the coverage gap: scroll and back exercise both surface transitions through the real settle path and assert that no diff or refs escape. No findings in this test-only change. The remaining requirements are live verify/settle transition evidence and an updated size breakdown; the earlier sign-in run does not cover those new outputs.

@thymikee

Copy link
Copy Markdown
Member Author

All three outstanding items are addressed. Keeping the AX hit-test design out of this, as asked — it is filed separately below.

Live verify/settle evidence

Run on the shipped path at 38c1d9f (rebuilt dist + iOS runner, real daemon, iPhone 17 Pro / iOS 26.2, a real ASWebAuthenticationSession sheet over a probe app):

case result
app→sheet, press --settle surfaceChange {from: app, to: com.apple.SafariViewService}, no diff, no tail, disclosure in hint, settled: true (12 captures)
sheet→app, press --settle --verify surfaceChange {from: com.apple.SafariViewService, to: app} on both settle and evidence, no diff/tail, changedFromBefore: true
app→sheet, --verify alone evidence.surfaceChange present, no settle payload
control — same-surface --settle --verify surfaceChange absent on both payloads, and diff present

The control is the one that matters for regression risk: a same-surface press still produces a real settled diff with refs, so the refusal is scoped to the boundary rather than suppressing normal settle output. Both transition directions produce their own disclosure wording ("is presented" vs "was presented over the app").

Generic-route regression

Covered at 38c1d9f: scroll --settle and back --settle, both directions, asserting the same contract as the targeted route (disclosed surfaceChange, no diff, no tail, no refsGeneration, ref frame left expired, loop verdict still reported).

The revert probes turned up something worth recording: the two directions falsify different halves of the plumbing. Reverting the baseline surface fails only the two sheet→app tests — app→sheet structurally cannot fail that revert, because an app baseline has no surface id to lose and the change is derived from the settled capture alone. Reverting the settled capture's surface fails only the two app→sheet tests. Removing the refusal fails all four. So neither direction substitutes for the other; that is now documented in the test file header. No production gap was found — the generic route's plumbing was already correct, just untested.

Size breakdown at this head

+14,753 B unpacked against CI's base 3bbeb61917, matching the current report (+14.8 kB): dist/src/** +11,137, dist/apple/runner/** +3,616, every other shipped path exactly 0 — including android/**, where the only changes are byte-identical 0.21.00.21.1 renames.

Per commit from 22001504c4: +10,778 → +14,753 → +14,753. The verify/settle commit 36fdde3e costs +3,975 B, entirely dist/src across seven chunks and zero Swift (sdk-selectors.d.ts +1,587, runtime.js +1,247, command-tools.js +493, ios-system-surface.js +266, generic-settle.js +195, find.js +182, interaction2.js +5). The test split f27156cd costs 0 B, verified rather than asserted: all 507 tarball entries identical in size and dist/src JS byte-identical (3,511,779 raw, 1,162,486 summed gzip); the only movement anywhere is +3 B of gzip-stream noise in the download figure.

Against the merge-base, +12,883 B is this branch and −1,870 B is base drift (main shrank between the merge-base and this PR's base sha, so the diff against a smaller base reads larger by the same amount). 12,883 + 1,870 = 14,753 — residual 0 B. These are CI's own measurements, read from the size-report artifacts of runs 34567881016 / 34522060244 / 34501499488 / 34399427351, not a local approximation. The drift figure was re-derived from a separate run whose base happened to be the merge-base, rather than carried over from my earlier comment.

AX hit-test

Agreed it is a proposal, not a resolved gap — it has not been through the production bridge. Filed as its own issue so it does not ride on this PR's validation.

@thymikee

Copy link
Copy Markdown
Member Author

The reported live runs at 38c1d9f cover both surface transitions and the same-surface control, and the size breakdown accounts for the growth. The outstanding validation is addressed; no code findings remain and checks are green. The AX hit-test simplification can stay in #2485.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 11, 2026
iosSystemSurfaceDisclosure() only mapped provenance-or-nothing onto the shared
constant for one caller, so the caller now reads the constant directly and the
wrapper is gone. Its test becomes a test of the transition disclosure, which is
the function that still earns its place (the "sheet is gone" sentence).

readAppleSnapshotResult also called readSystemSurfaceProvenance twice inside one
spread; it is bound to a local and read once.
prepareActiveCommandContext checks for a presented system surface before it
resolves or activates command.appBundleId, so a command naming a different app is
still served the sheet. That is intended, but the code does not read that way;
the amendment now says it plainly.
@thymikee

Copy link
Copy Markdown
Member Author

Reviewer cleanups pushed in 7ea537a + d84349b (head d84349b21e).

Done

  1. Inlined the thin wrapper. iosSystemSurfaceDisclosure() had one caller and only mapped provenance-or-nothing onto IOS_SYSTEM_SURFACE_DISCLOSURE; runnerSnapshotWarnings now pushes the constant directly and the wrapper is deleted. Its test is repointed at iosSystemSurfaceTransitionDisclosure, which stays — both its arms are checked there, and the "sheet is gone" sentence is the reason it exists. Its other arm is the standing constant, deliberately: that is the single source for the sentence, and collapsing it further would either change an emitted sentence or push the undefined check out to the caller, so it is left as is.
  2. Bound the repeated call. readAppleSnapshotResult reads readSystemSurfaceProvenance(result.systemSurface) once into a local.
  3. ADR. Verified in prepareActiveCommandContext that the presented-surface branch is the third arm of the if/else chain, ahead of the !isRunnerLifecycleCommand arm that resolves and activates command.appBundleId. The iOS: ASWebAuthenticationSession cannot be automated — binding the session to com.apple.SafariViewService destroys the view, and the tree served afterwards is stale #2438 amendment in docs/adr/0004 now states plainly that a presented surface outranks an explicitly requested bundle id, why that is intended, and why it costs nothing afterwards.

Not done — item 3 (payload back to let)

Stopped rather than forced, as instructed. The mutation is the local binding in executeSnapshotPrepared (DataPayload's fields were already var on main, like every other field on that struct). Constructing the payload complete means the surface has to reach every place a returned payload is built:

  • snapshotFast / snapshotRaw (new parameter each)
  • their early return from boundedBlockingSystemAlertSnapshotblockingSystemAlertSnapshot — this path is stamped today, so skipping it would silently drop the disclosure when a SpringBoard alert is captured while a host is foreground
  • runSnapshotCapturePlan (new parameter; 4 internal uses)
  • stampedSnapshotPayload (the DataPayload(...) constructor; 3 production call sites, 7 more under AGENT_DEVICE_RUNNER_UNIT_TESTS)
  • snapshotAccessibilityUnavailablesparseTruncatedSnapshotPayload (the AX-unavailable terminal)

That is five or six signatures and ~10 call sites to remove one var and four lines, and it pushes a response-level fact that only the command context knows ("which surface does this whole capture describe") into capture backends whose job is tree acquisition and quality. Happy to do it if you still want it.

Gates (on the pushed commit): pnpm format (oxfmt, no diff), pnpm lint, pnpm typecheck, pnpm check:affected --run all pass, plus check:agent-guidance for the docs touch. No Swift changed, so no runner rebuild was needed.

…ineage

A capture of an in-place system surface (a web sign-in sheet) describes a
different presentation than a capture of the app, so it must never compare
equal to one. The `present` branch of the iOS snapshot route returned a bare
fallback, so that capture carried no comparison identity at all, and two
comparison sites hand-rolled the distinction from `iosSystemSurfaceBundleId`
instead.

The probe now reports which host it matched, and the `present` branch goes
through `runFallback` like the `unknown` branch beside it, lineaged to
`<device>:<host bundle>`. The comparison key then differs from an app
capture's by construction, so the surface branch in `hasMatchingPresentation`
and the surface concatenation in `snapshotComparisonKey` are gone: both sites
are plain key equality again, and neither knows that system surfaces exist.
Two captures of the same surface still share a lineage, so they stay
comparable with each other.

A presented surface is not a bridge failure, so it gets its own warning
wording: the bridge is inapplicable here, not unavailable.
@thymikee

Copy link
Copy Markdown
Member Author

Applied the second-lineage-channel simplification (5ab9ea41fe, rebased onto d84349b21e).

The cause, fixed at the source. snapshot-route.ts's present branch returned a bare fallback(input), so a system-surface capture carried no comparison identity at all — which is why two comparison sites had to reconstruct the distinction by hand. The probe now reports which host it matched ({ kind: 'present'; host } alongside 'absent' / 'unknown'), and the present branch goes through runFallback like the unknown branch one line below it, lineaged to `${device.id}:${host.bundleId}`. iosSnapshotComparisonIdentityKey stringifies lineage, so the key differs from an app capture's by construction.

"Anything but a proven absent takes the runner" is now also literal in the code: the two runner branches sit under that one condition. Memo and failure behavior are unchanged (only a positive observation is memoized; it now memoizes the host with it).

Deleted

  • The surface branch in hasMatchingPresentation and its ios_tap_failure_corroboration_surface_mismatch diagnostic — interaction-ios-tap-outcome.ts is back to key equality and no longer knows that system surfaces exist.
  • The surface concatenation in snapshotComparisonKey (deferred-interaction-outcome.ts), including the unknown-backend sentinel it needed. It is a one-line read again.

Deliberately not deleted: SnapshotState.iosSystemSurfaceBundleId, still required by the disclosure seam and the verify/settle comparison.

One design point in the new branch. No unknown-generation residue on a present capture, unlike the unknown branch. That residue carries a per-capture UUID, which would make two captures of the same sheet incomparable with each other and so would never let a post-gesture poll settle on a sheet. A surface is not an app generation, so the lineage alone carries it. Pinned by the route test.

Warning wording. "Simulator AX snapshot unavailable" was wrong here — the bridge is healthy and merely cannot see the surface. runFallback now picks the sentence from the typed reason; every other reason keeps the existing one verbatim:

Simulator AX snapshot inapplicable (system-surface-presented); used XCTest to read the system surface presented over the app.

How the guarantee is pinned, and that it is not just assertion-shaped. The chain is two links, and I mutation-checked each:

  1. snapshot-route.test.ts — "a presented system surface captures under the host lineage, never the app lineage": the lineage is the host, the residue carries no per-capture id, and two surface captures stay identity-equal. Pointing the branch at input.options.appBundleId instead fails exactly this test.
  2. Both daemon comparison sites keep their cross-surface regression, adapted to ride the key. They build both keys through the production iosSnapshotComparisonIdentityKey (shared fixture src/daemon/__tests__/ios-comparison-key-fixture.ts), so they fail if the construction ever stops distinguishing app from host — they are not asserting two hand-written strings differ. Collapsing lineage to {} in that fixture fails exactly three tests and nothing else:
    • post-gesture-stabilization.test.ts — surface appears mid-poll, and the mirror case where it dismisses (both still re-baseline instead of concluding).
    • interaction-ios-tap-outcome.test.ts — a system-surface capture still cannot corroborate a tap taken against the app.

One honesty note on scope. The guarantee is structural wherever the route attaches an identity, which is every eligible iOS Simulator capture — and every system-surface capture is one by construction. Captures that bypass the route entirely (preferredBackend pinned, customActions) still carry no identity and still fall through to legacy presentation matching. The deleted hand-rolled check nominally covered that corner, but the rest of #2438 does not work there either: with the route bypassed the bridge serves the occluded app tree and never reports a surface at all, so there was no cross-surface pair to refuse. That corner is the pre-existing "no identity → legacy matching" limit, not surface-specific, and I left it alone.

Line delta: +163 / −47, net +116. Production code is roughly flat (−9 in the two daemon sites, +25 in the route, +14 for the probe carrying its host); the growth is the regression that pins the new structural property plus the shared key fixture. The win here is one lineage channel instead of two, not line count — say the word if you would rather have the route test trimmed further.

Gates (on the pushed commit): pnpm format (oxfmt, no changes), pnpm lint, pnpm typecheck, pnpm check:affected --run → all runnable checks passed, test-file-size ratchet green. Also ran pnpm check:layering since the fixture is a new module — OK. No device or simulator use.

One earlier check:affected run failed in daemon-entrypoint.test.ts / client-metro.test.ts under contention (a 5 s timeout and a stale claims dir); both pass in isolation in 1.6 s and 0.57 s and neither touches this diff. The final run on the pushed commit was clean.

…coped value

The same pre-action tree travelled as a flattened nodes/surface pair at every
boundary, and each boundary rebuilt it with a conditional spread. Carry
SurfaceScopedNodes itself instead:

- ResolvedInteractionTarget gets preAction?: SurfaceScopedNodes, replacing the
  preActionNodes/preActionSurfaceBundleId pair and the PreActionBaselineFields
  intersection on all three arms of the union.
- SettleObservationCommandOptions gets baseline: SurfaceScopedNodes, replacing
  baselineNodes/baselineSurfaceBundleId.
- RefResolution carries tree: SurfaceScopedNodes instead of nodes plus a loose
  surfaceBundleId.

That retires preActionBaselineFields(), preActionBaseline(), evidenceBaseline(),
the local SettleBaseline type, the split-then-reassemble in
settleObservationCommand, and the 'preActionNodes' in resolved narrowing tests.
SurfaceScopedNodes moves to contracts, where ResolvedInteractionTarget can name
it; only two sites now mint one from a SnapshotState.

Behaviour is unchanged: the cross-surface guarantees keep their existing tests.
@thymikee

Copy link
Copy Markdown
Member Author

Applied the "one value, five names" simplification in 6ba5b3da81. Net −72 lines (81 insertions, 153 deletions across 13 files).

The nodes and their surface now travel as one SurfaceScopedNodes value at every boundary, so no boundary can carry a baseline of nodes and drop the surface it came from.

Type changes

  • SurfaceScopedNodes moved to packages/contracts/src/interaction.ts, where ResolvedInteractionTarget can name it (contracts stays types-only).
  • ResolvedInteractionTarget gets preAction?: SurfaceScopedNodes on each arm, replacing the preActionNodes/preActionSurfaceBundleId pair and the PreActionBaselineFields intersection surgery on the three-arm union.
  • SettleObservationCommandOptions gets baseline: SurfaceScopedNodes, replacing baselineNodes/baselineSurfaceBundleId. Confirmed your caller claim: runtime.interactions.settleObservation has exactly one caller, src/daemon/generic-settle.ts, in-process; no test constructs those options.
  • RefResolution (resolution.ts) carries tree: SurfaceScopedNodes instead of nodes plus a loose surfaceBundleId — that killed a fourth rebuild-with-conditional-spread at the describeResolvedInteractionNode call.

Deleted

  • PreActionBaselineFields (contracts)
  • preActionBaselineFields() (resolution.ts) and its three call sites' spreads
  • preActionBaseline() (post-action-surface.ts) and its three call sites — consumers read resolved.preAction directly
  • evidenceBaseline() (settle.ts), which existed only to re-wrap what preActionBaseline() unwrapped
  • the local SettleBaseline type (generic-settle.ts)
  • the split-then-reassemble in settleObservationCommand, now a one-liner passing options straight through
  • both 'preActionNodes' in resolved narrowing tests (post-action-surface.ts, interaction-touch-response.ts)

Conditional spreads that write surfaceBundleId went from 6 to 2, and both survivors are mint points converting a SnapshotState into the carried value (surfaceScopedNodes() for the targeted route, the session-snapshot read for the generic route).

The drift-hazard comments you flagged are gone. Each was warning about something the type now states, so they were deleted rather than reworded:

  • preActionBaselineFields(): "Emitted from one place so the nodes can never travel without their surface identity."
  • SettleObservationCommandOptions.baselineSurfaceBundleId: "Without it the settled diff could be built across a surface replacement."
  • settleEvidence(): "The baseline keeps its own surface identity, so a settled sheet capture is never digest-compared against an app baseline."
  • captureVerifyEvidence(): "Both sides of the comparison carry the surface they describe, so a capture of an in-place system surface is never digest-compared against an app baseline."
  • describeResolvedInteractionNode(): "surface identity included, so the pre-action baseline it publishes can never lose which surface it describes."
  • RefResolution.surfaceBundleId: "The surface nodes describes, carried to the post-action comparison baseline."
  • SettleOutcome.settledCapture: "nodes and the surface they describe; doubles as the --verify evidence source, which needs both to know whether its baseline is comparable."

The one remaining "why" lives at the owning type in contracts, which is where a reader needs it.

Behaviour unchanged. No test assertion was weakened, rescoped or deleted. The cross-surface regressions (post-action-surface.test.ts, generic-settle.test.ts, system-surface-disclosure.test.ts, the verify/settle and contract suites) pass with their assertions as written. Three test edits follow the rename so they keep their exact strength — leaving the old key would have made them pass vacuously:

  • native-ref.contract.test.ts and interactions.test.ts: Array.isArray(result.preActionNodes)Array.isArray(result.preAction?.nodes)
  • native-ref.contract.test.ts and the press/click guards in interaction-target-evidence.test.ts: the "payload must not carry the resolution baseline" guards now name preAction. The get/is/wait guards in the same file still name preActionNodes, because the selector-read family keeps its own flat preActionNodes field — it is a different type and out of scope here.
  • settle-transition-baseline.test.ts: fixture construction only.

Out of scope, flagged: PreresolvedInteractionTarget still carries nodes + iosSystemSurfaceBundleId as a flat pair on the find → interaction-leaf channel. Its field name deliberately mirrors SnapshotState, which withSystemSurfaceDisclosure also reads off FindTargetTree, so converting it would move the spread rather than remove it. Happy to do it as a follow-up if you want that boundary closed too.

Gates (on the pushed commit, rebased onto 5ab9ea41fe): pnpm format clean, pnpm lint clean, pnpm typecheck clean, pnpm check:affected --run all lanes passed — 775 test files / 5884 tests, plus layering, fallow, depgraph, production-exports, build and the test-file-size ratchet.

@thymikee

Copy link
Copy Markdown
Member Author

The new lineage construction at 6ba5b3d introduces a cross-surface comparison bug. The process probe can stay positive after the sheet is dismissed, while XCTest correctly returns app content. The route still stamps that capture with the host lineage, so it compares equal to the preceding sheet capture and post-gesture polling can miss the transition. Derive the identity from the returned capture’s systemSurface, not the process probe. Add a regression where the probe stays positive while the runner changes from sheet to app.

The removed surface check also covered preferredBackend/customActions captures that bypass the route. Those still go through XCTest and can report a system surface, but have no comparison key; they now fall back to legacy presentation matching and can corroborate a tap across app/sheet content. Preserve the distinction at the capture owner for those paths too, with a regression for a route-bypassing capture. The earlier live evidence predates this identity change; repeat the dismissal/lingering-host case after fixing it.

@thymikee thymikee removed the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 11, 2026
@thymikee

Copy link
Copy Markdown
Member Author

All three simplifications are in at 6ba5b3d. Measured effect of the pass: −50 production lines, +84 in tests.

1. Second lineage channel — removed. Your diagnosis was exact. The probe now reports which host matched, and the present branch routes through runFallback with targetId: ${device.id}:${host.bundleId} and reason system-surface-presented, mirroring unknown one line below. Both hand-rolled sites are gone: the surface branch and its diagnostic in hasMatchingPresentation (back to plain key equality) and the concatenation in snapshotComparisonKey (now one line). Warning variant added, since "unavailable" was the wrong word — the bridge is inapplicable, not broken.

This was flat in production lines (−9 at the two daemon sites, +25 route, +14 probe); the win is one channel instead of two, not line count. Coverage was mutation-checked rather than assumed: collapsing lineage in the shared fixture fails exactly three tests — both cross-surface post-gesture regressions and the tap-corroboration one — and nothing else. Those two daemon regressions were adapted, not dropped, and now build both keys through the production iosSnapshotComparisonIdentityKey, so they fail if the construction stops distinguishing rather than comparing two hand-written strings.

One deliberate asymmetry: the present branch gets no unknown-generation residue. That residue carries a per-capture UUID, which would make two captures of the same sheet incomparable, so a post-gesture poll could never settle on a sheet. A route test pins it.

Scope limit, stated rather than hidden: the guarantee is structural wherever the route attaches an identity. Captures that bypass the route (preferredBackend pinned, customActions) still use legacy matching — but #2438 does not function there anyway, since the bridge serves the occluded app tree and never reports a surface. Pre-existing; left alone.

2. One value, five names — done, −72 lines. SurfaceScopedNodes now travels as one value: ResolvedInteractionTarget.preAction, SettleObservationCommandOptions.baseline. Deleted PreActionBaselineFields, preActionBaselineFields(), preActionBaseline(), evidenceBaseline() (it existed only to re-wrap what the former unwrapped), the local SettleBaseline, the split-then-reassemble in settleObservationCommand, and both 'preActionNodes' in resolved narrowing tests. Your one-caller claim was verified independently.

Two things found in passing. RefResolution was a fourth rebuild-with-conditional-spread at the describeResolvedInteractionNode call; it now carries tree: SurfaceScopedNodes, taking spreads that write surfaceBundleId from 6 down to 2, both genuine mint points. And the rename would have made two guards pass vacuously — the "payload must not carry the resolution baseline" assertions named preActionNodes, so they now name preAction; the get/is/wait guards deliberately still say preActionNodes, because the selector-read family has its own independent flat field.

Seven drift-warning comments were deleted — the ones whose existence was your evidence for the problem. The mint function stayed out of contracts on purpose: moving it would turn a types-only module into a runtime one and shift every importer's eager closure.

3. Thin wrappers — done. iosSystemSurfaceDisclosure() inlined and deleted; iosSystemSurfaceTransitionDisclosure() kept for its different sentence and now has a contracts-level test it previously lacked. The doubled readSystemSurfaceProvenance call is bound once.

4 and 5. Filed the shared system-surface model as #2489 rather than blocking. Probe left as-is — your launchctl analysis matches what I measured, since the cached-target path skips it and folding the check in would force a ~1s call per capture.

Smaller notes. ADR now states the Swift branch precedence, with the ordering verified in source first.

On DataPayload: I'd push back. The var is the local binding in executeSnapshotPrepared, not a struct field — DataPayload's fields were already all var on main, and this diff adds one more among ~40. Constructing it complete means threading the surface through five or six signatures and ~10 call sites, including the boundedBlockingSystemAlertSnapshot early return, which is stamped today — missing it would silently drop the disclosure when a SpringBoard alert is captured while a host is foreground. It also pushes a response-level fact only the command context knows down into capture backends whose job is tree acquisition. Happy to do it if you still want it.

On the red smoke: it is the lane, not this PR. Two runs of this same commit failed with two different assertions — automation-longpress did not become visible after scrolling, then wait for the WebView page to expose its link, which main is also failing right now (run 34592935964). A deterministic regression cannot rotate its symptom. The failing run also contains zero system-surface mentions, so the present/unknown branches never fired and the bridge path was unchanged. I checked before assuming: this assertion appears in none of the six recent main failures, which is why I bisected rather than re-running until green. Filed as #2491 with the rates and signatures.

@thymikee

Copy link
Copy Markdown
Member Author

The latest explanation does not resolve the two comparison concerns at 6ba5b3d. A positive process probe can outlive the sheet, so its host identity cannot stand in for the surface the runner actually returned. The route-bypassing captures also need that returned-surface distinction. Please derive comparison identity at the capture owner and cover dismissal with a lingering host plus a pinned-backend capture; then repeat the live transition run. The current smoke failure is RUNNER_BUSY at the WebView page wait, matching the separately reported main failure, and appears unrelated to these findings.

The `present` path stamped the capture's comparison lineage from the host-side
presence probe. That probe answers about a host PROCESS and deliberately stays
positive while a dismissed host lingers, so during that window the runner
truthfully returned APP content while the route lineaged it to the HOST: the
sheet capture before the dismissal and the app capture after it compared equal,
and a post-gesture poll could read the transition as a stable surface.

Derive the identity from the returned capture's `systemSurface` instead - the
runner stamps the surface it actually served - and say which of the two the
capture holds in the warning. The probe's host is now evidence only: it names
the matched host in a route diagnostic so a lingering window is legible in the
daemon log. Other reasons keep their lineage and wording byte for byte.

Captures that bypass the route's planning (a pinned backend, a custom-actions
read) also reach the runner, and the runner serves the sheet there too. They
carried no comparison key at all, so a sheet and app content fell through to
legacy presentation matching as one presentation and could corroborate a tap
across the two. The capture owner now gives those a surface-scoped identity as
well, with no fallback-source residue: nothing fell back. An app capture off
the route is untouched.
@thymikee

Copy link
Copy Markdown
Member Author

Both points taken; fixed in 28c6e86.

What the identity is derived from now. The present path no longer reads the probe for identity. It runs the runner capture first, then lineages it from the returned capture's systemSurface: the host bundle id when the runner says it served the surface, the app bundle id otherwise. The probe answers about a host process and is documented to stay positive while a dismissed host lingers; only the runner answers about the screen, and it stamps what it served onto the capture. The reason token — and therefore the warning sentence — comes from the same derived value, so the identity and the prose cannot disagree: a lingering-host capture now says system-surface-host-lingering and "used XCTest, which read app content", instead of claiming it read the sheet. The probe's matched host survives as evidence only: it is named in a ios_snapshot_route_fallback debug diagnostic (detectedHost) when the runner disagrees, so the lingering window is legible in the daemon log rather than looking like a missing bridge capture.

The design point from the previous commit is unchanged: the present path still attaches no unknown-generation residue on either branch, so two captures of the same sheet stay comparable and a poll can settle on it. The new regression also pins the mirror case — two app captures taken inside one lingering window still compare equal, so the poll can settle on app content too. Every other reason keeps its lineage, residue and wording byte for byte.

How the bypass paths are covered. isEligible excludes a pinned preferredBackend and customActions: true; you are right that those still go through XCTest and can report a surface. The route is still the capture owner for them (every Apple captureSnapshot goes through route.capture), so the bypass branch now stamps the same surface-scoped identity when the returned capture reports a surface: lineage <device>:<host bundle>, and no fallback-source residue, because nothing fell back — the runner is the requested producer there. An app capture off the route is byte-identical to before (no identity, no warning), which the regression asserts with toEqual. So the distinction is structural in the identity on those paths too; hasMatchingPresentation and snapshotComparisonKey stay plain key equality and still know nothing about system surfaces.

Revert-sensitivity.

  • Fix 1: reverting runSurfaceFallback to lineage from the probe's host → a lingering probe cannot make a sheet capture and an app capture compare equal fails (targetId came back ios-1:com.apple.SafariViewService instead of ios-1:com.example.app, i.e. exactly the cross-surface equality). Restored → green.
  • Fix 2: reverting the !isEligible branch to a bare fallback(input) → both cases of a route-bypassing capture of a system surface is incomparable fail (comparisonIdentity undefined). Restored → green.

One existing test changed, assertions untouched: a presented system surface captures under the host lineage, never the app lineage used a fallback that returned a capture with no systemSurface while the probe said present. Under the corrected semantics that combination is the lingering case, so the fixture now returns the capture the runner actually produces for a sheet. A new surfaceRunnerResult() helper mints it.

Not changed, flagged rather than silently left: the bridge-failure reasons (circuit-disabled, a typed acquisition failure) keep the generation-scoped app lineage per your "byte-identical" constraint. If the probe answers absent and the runner then serves a sheet — a sheet appearing inside the race window between the probe and the capture — two such captures still compare equal across that surface change. It needs both a false-negative probe and a disabled-or-failed bridge, so it is much narrower than what this fixes; say the word and I will extend the same capture-derived lineage to those reasons.

Gates on the pushed commit: pnpm format (oxfmt), pnpm lint, pnpm typecheck, pnpm check:affected --run (all gates pass; vitest related = 775 files / 5887 tests green, including every existing cross-surface regression), test-file-size ratchet green (snapshot-route.test.ts 588 lines). No device or simulator was used — the live dismissal/lingering-host validation is yours to re-run, and the new detectedHost diagnostic should make the lingering window easy to spot in daemon.log.

@thymikee

Copy link
Copy Markdown
Member Author

Both findings were correct, and the second one corrected a claim of mine that was simply wrong. Fixed at 28c6e86, and I repeated the live case you asked for on that commit.

Where the identity comes from now. runFallback was split so the present path awaits the runner capture first and derives the lineage from result.systemSurface — the host bundle only when the runner says it actually served the surface, the app otherwise. The probe's matched host is now evidence only: when the runner disagrees with it, that shows up as detectedHost in the route diagnostic. The derived value also picks the reason token, so a lingering-host capture reports system-surface-host-lingering and the warning can no longer claim it read the sheet while returning app content.

On route-bypassing captures — you were right and I was wrong. I had written that those paths "never report a surface, since the bridge serves the occluded app tree". That conflated bypassing the route with using the bridge. They go to the runner, which serves the sheet and stamps it. The !isEligible early return now stamps a surface-scoped identity when, and only when, the returned capture reports a surface — with no fallback-source residue, since nothing fell back there.

Live evidence, re-run on 28c6e86 (iPhone 17 Pro / iOS 26.2, real ASWebAuthenticationSession sheet, rebuilt dist + runner):

step observed
sheet presented 21 nodes (the form); system-surface-presented; sheet disclosure attached
dismissed, host process still running (pgrep positive) 4 nodes of app content (AuthProbe, Sign In, and the WebAuthenticationSession error 1 label from the cancel); reason system-surface-host-lingering; sheet disclosure correctly absent

That second row is the bug you found: previously the probe stayed positive and the route stamped that app capture with the host lineage. It now follows the capture.

Revert-sensitivity, both proven: reverting fix 1 makes a lingering probe cannot make a sheet capture and an app capture compare equal fail with targetId coming back as the host instead of the app — the cross-surface equality itself. Reverting fix 2 makes both route-bypass cases fail with comparisonIdentity undefined. One existing test's fixture changed (assertions untouched): it previously had the probe say present while the fallback returned a capture with no systemSurface, which under the corrected semantics is the lingering case.

No hand-rolled surface branch was reintroduced in hasMatchingPresentation or snapshotComparisonKey — the distinction stays structural. The deliberate "no unknown-generation residue on the present path" point still holds, and its route test is green; two captures of the same sheet still compare equal, so a poll can settle on a sheet.

One residual I am flagging rather than fixing, because it is outside what you asked and I did not want to widen the diff unasked: the bridge-failure reasons (circuit-disabled, typed acquisition failures) still carry the generation-scoped app lineage. If the probe answers absent and the runner then serves a sheet — a sheet appearing inside the probe→capture race window — two such captures still compare equal across that surface change. It needs a false-negative probe and a disabled or failed bridge. Say the word and I will extend the capture-derived lineage to those reasons too.

@thymikee

Copy link
Copy Markdown
Member Author

The lingering-host and pinned-backend fixes at 28c6e86 address the two reviewed cases, and the new live dismissal run covers the requested transition. One instance of the same comparison bug remains in runFallback (packages/platform-apple/src/snapshot-route.ts:278): it stamps the supplied app lineage even when the runner returns systemSurface. With the bridge circuit already disabled, capture app content, then let the sheet appear after the next absent probe but before the runner capture. Both captures receive the same app-generation key, so tap corroboration can treat different surfaces as comparable. Derive the served-surface identity for every runner fallback at the shared stamping point, preserving fallback reasons and generation evidence where applicable, and add the absent-probe/circuit-disabled regression. This is needed before approval; iOS smoke is also still running.

@thymikee

Copy link
Copy Markdown
Member Author

The iOS smoke job at 28c6e86 has now failed with RUNNER_BUSY at the WebView page wait, matching the main-lane signature tracked in #2491. It appears unrelated to the system-surface changes. The previously reported shared-fallback identity finding remains the code blocker.

A runner fallback's comparison identity was decided per call site. The
`present` path and the off-route path read the runner's `systemSurface`
stamp, but the plain `runFallback` path did not: it stamped the app
lineage the route had planned, whatever the runner returned.

The probe and the capture are separate observations, so a sheet can
appear in the gap between them. With the bridge circuit already disabled
for the generation, an app capture and a later sheet capture both
received the same app-generation key, so tap corroboration could treat
two different surfaces as comparable.

`stampFallback` now owns the decision for every runner fallback: the
surface the runner served outranks the app lineage the route planned.
The reason the bridge was skipped survives either way, and
app-generation evidence leaves with the app lineage it describes, so two
captures of the same sheet still compare equal. `runSurfaceFallback`
keeps only the reason, which is the one thing that path decides.
@thymikee

Copy link
Copy Markdown
Member Author

Fixed at 82b623d0f8. You were right that the same bug survived in runFallback, and moving the decision to the shared point deleted the duplication rather than adding a third copy of it.

What changed. stampFallback is now the one place a runner fallback's identity is decided. It reads the runner's systemSurface stamp off the result it is about to stamp, and that stamp outranks the app lineage the route planned:

const served = result.systemSurface;
comparisonIdentity: runnerComparisonIdentity(
  served ? surfaceLineage(deviceId, served.bundleId) : lineage,
  request,
  [...(served ? [] : residue), { kind: 'fallback-source', producer: 'apple-runner' }],
),

Every arm that reaches the runner goes through it: circuit-disabled, target-resolution-failed, system-surface-probe-unavailable, the typed bridge failures, and the proven-present path. runSurfaceFallback now decides only the reason, which is the one thing that path actually knows (SYSTEM_SURFACE_PRESENTED when the runner served the sheet, SYSTEM_SURFACE_HOST_LINGERING when the probe matched a host but the runner served app content). captureOffRoute shares the same surfaceLineage helper.

Preserving the two things you named.

Fallback reasons. The reason is independent of what the runner found, so it survives untouched. But the generic warning sentence would then have claimed an app generation the capture does not hold, so there is one new branch: a served surface on a non-surface reason reads Simulator AX snapshot unavailable (circuit-disabled); used XCTest, which read the system surface presented over the app. The reason is still named; only the content clause changed.

Generation evidence. App-generation evidence leaves with the app lineage it describes. A surface is not an app generation, and the arm's per-capture unknown-generation residue would otherwise make two captures of the same sheet incomparable with each other — which is the rule the present path already followed and the reason it carried no residue. So generation and that residue are dropped exactly when the served-surface lineage replaces the app lineage, and fallback-source stays, because a fallback did happen.

The regression you asked for, plus its sibling:

  • a sheet that appears after an absent probe is identified by the surface, not the app — absent probe, circuit already disabled from a prior typed failure, then a runner capture that returns systemSurface. Asserts the sheet is lineaged to the host, the app capture keeps targetId and generation, the two are not equal, two sheet captures still are equal, and the warning keeps circuit-disabled.
  • a surface capture drops the app-generation residue of the arm that reached it — the same gap through the system-surface-probe-unavailable arm, which is the one that attaches a per-capture residue id. Without the drop, two captures of the same sheet would be incomparable and a poll could never settle on it.

Seeded. Reverting only the served ? … : lineage choice in stampFallback fails both new tests and both pre-existing surface tests (4 failed / 19 passed), so the assertions are load-bearing rather than incidental.

Size. +19 production code lines and +17 comment lines in snapshot-route.ts, against 5 call sites that can no longer each get this wrong. Full file is 505 lines.

pnpm check:affected --run: 5888 passed, 1 failed — test/integration/provider-scenarios/ios-record-trace.test.ts with /bin/ps timed out after 1000ms. That is the known under-contention spawn timeout, not this change: it passes in isolation on this commit, and the surface probe does not use host-process at all (it goes through runAppleToolCommand).

One thing stated plainly: this round is unit-proven, not live-proven. The window is a race between the presence probe and the runner capture, which I cannot stage deterministically on a simulator — the unit test is the instrument that can hold it. The live runs on 28c6e86 still stand for the paths they covered, including the sheet-presented and host-lingering transition.

Separately, the iOS smoke red on this PR is not this PR: it is the RUNNER_BUSY chain now fixed by #2493. I traced it and recorded the cause on #2491, including a correction to my own first reading of it.

@thymikee

Copy link
Copy Markdown
Member Author

Smoke on 82b623d0f8 is red for a cause outside this PR, and I have deliberately not pushed a fix for it here. Evidence, then why.

Same signature as before and as on main: step wait for the WebView page to expose its link, scenario smoke:webview-remote-content, RUNNER_BUSY from a capture whose predecessor exceeded the runner's execution watchdog.

This PR's feature never ran in that job. The only occurrence of system-surface anywhere in the log is the bundle-size line for dist/src/ios-system-surface.js. No route warning, no SafariViewService, no probe verdict — the scenario has no web sign-in sheet, so the presence probe answers absent and the route behaves exactly as before this branch.

The wedge is in the harness, and it is in this tree because it is in main: acceptDeepLinkConfirmationIfPresent hard-codes its readiness landmark to 'Automation lab' (test/integration/ios-simulator-e2e/live-automation-scenario.ts:251). Off the Automation route that landmark can never match, so the helper always falls through to an alert get probe, and an XCTest alert query against a live WKWebView exceeds the runner's 30 s main-thread watchdog. live-snapshot-depth-frontier.ts:28 calls the same helper, so the wedge is set before smoke:webview-remote-content ever runs and that scenario is the victim, not the cause.

Why no fix here. #2493 already fixes exactly this, with live before/after numbers, and it changes the same harness function plus the wait ride-out. Duplicating its three-line landmark change onto this branch would not make this PR mergeable any sooner, and it would guarantee a conflict on that function between two open PRs. It is also the design decision you asked to keep separate from this PR's validation. So the sequencing is: land #2493, then this branch goes green on a rebase.

pnpm check:affected --run on 82b623d0f8 is green apart from one known under-contention spawn timeout (/bin/ps timed out after 1000ms in ios-record-trace), which passes in isolation and does not touch the probe.

@thymikee

Copy link
Copy Markdown
Member Author

No remaining code findings at 82b623d. The shared fallback now derives identity from the returned surface on every path; the circuit-disabled regression covers the race, and the earlier live sheet/dismissal evidence remains applicable. Ready for human review. The failing iOS smoke still reports the known WebView RUNNER_BUSY harness issue and appears unrelated; land #2493 and revalidate this branch before merge.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

1 participant