feat(macos-helper): drive app sessions through accessibility actions

[thymikee]

Summary

Adds an app surface to the macOS helper so a macOS app session can run without XCTest Automation Mode: no overlay, the app may stay behind other windows, and the user keeps the real pointer.

• snapshot --surface app --bundle-id walks the session app. Chromium/Electron apps get their accessibility tree turned on; only the snapshot that turns it on waits (up to 1 s), and a tree that never populates adds a warnings entry.
• press/fill/type/scroll act through accessibility actions. When the app's hit test names no control (Chromium answers with wrapper groups), the helper takes the smallest control containing the point inside the hit window (or the front window), and responses carry windowTitle. No pointer events are posted: inactive apps drop them while the post reports success.
• screenshot --surface app captures the on-screen front window alone.

Nothing calls the surface yet; #3189 routes app sessions to it, #3195 adds the ghost cursor. 6 files.

Validation

• pnpm check:affected --run passed on 86a9c656b; swift test passed (vocabulary and scroll travel pinned to contracts/fixtures).
• Live helper JSON for the Chromium fallback route is in the review reply below.

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.97 MB	5.00 MB	+33.5 kB
Package (unpacked)	4.97 MB	5.00 MB	+33.5 kB
Package (download)	1.49 MB	1.50 MB	+8.6 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	22.8 ms	22.3 ms	-0.6 ms
CLI --help	69.0 ms	65.5 ms	-3.4 ms

[thymikee]

Thanks for the PR. I reviewed 6ac9e42 and found one defect that needs a fix and one gap in evidence, so it is not ready to merge yet. All 19 checks pass on that commit, but CI only builds the helper and does not run the AX routes live. No conflicts.

When the app's own hit test at (x,y) returns no text input and no pressable control role (a blank area, AXStaticText, or a Chromium wrapper group), resolvePressTarget and fillInBackground fall back to smallestElement. That walk covers every window from windows(of: app). It has no z-order or visibility filter, so a control in an overlapped or minimized window of the same app can win over the window the point actually hit. The helper then runs AXPress or sets AXValue there and reports success with ax-press or ax-value. The response does not name the window. In a multi-window app such as TextEdit, Finder or an Electron app, press or fill in the front window could act on a back window, and the caller would not know. This is inferred from the code, not reproduced. The rule should be that a fallback target belongs to the same AX window the hit test resolved, or to the front on-screen window from onScreenWindows(pid:) when the hit returned nothing. Please apply it once in smallestElement, so press and fill both inherit it, and return the resolved window title or number in the response so the host can check it.

The PR body describes live runs in prose only, and no helper output shows which route ran. Two branches have no evidence: the smallest-containing-control fallback near this line, and the AXEnhancedUserInterface fallback. The Swift tests cover only the vocabulary, scroll arithmetic and a role-set predicate, and resolvePressTarget and smallestElement have no test. Please attach the helper JSON from agent-device-macos-helper press --surface app --bundle-id <electron app> --x --y with the Electron app in the background. Use a point where the hit test returns a wrapper group, so the smallestElement branch runs. The output should show mechanism ax-press, the role, and the visible effect, such as sidebar navigation in the next snapshot. Please do the same for fill on a Chromium text field. A run on a native app whose hit test already names an AXButton does not reach this branch.

Small notes. In readTextAtPosition, the ?? AXUIElementCreateSystemWide() cannot fire, because requireSessionApplication throws first. awaitPopulatedWebContent adds 1 s to every snapshot of a Chromium app with no populated AXWebArea. Is that cost intended? enableRemoteAccessibilityTree still discards both set results, so a failed enable is not reported.

This PR adds 801 net production lines, above the 700-line threshold, and it has no caller yet. Could the host pass back the identity of the node it chose from the app-surface snapshot (path or index plus window)? The helper could then act on that element directly. That would drop resolvePressTarget, the smallestElement walk and the ancestor chain, and the first finding with them. Would splitting help too, with app-window screenshot and Chromium enabling in one PR and the AX actions in another? This would need #3189's routing to send node identity rather than only x/y, which is a contract decision for the ADR 0031 work.

Before merge, please restrict the press and fill fallback to the hit-tested window, then attach the live helper output for the Chromium fallback route.

[thymikee]

Thanks. Addressed in c0fabe6ec.

Fallback window (fixed). actionWindow picks the window the hit landed in (AXWindow of the hit), else the app's front on-screen window matched by frame. smallestElement now walks only that window, so press and fill both inherit the rule. A hit that names no window also falls back to the front window. Press, fill and type responses carry windowTitle, and #3189 passes it through to the CLI response.

Live evidence for the Chromium fallback route. Codex (Electron) in the background, T3 Code frontmost the whole time. The raw AXUIElementCopyElementAtPosition chain at each point has only wrapper groups, so smallestElement runs:

== press Scheduled at 378,272 — raw hit-test chain:
0 AXGroup | actions: ["AXPress", "AXShowMenu", "AXScrollToVisible"]
1 AXGroup | actions: ["AXPress", "AXShowMenu", "AXScrollToVisible"]
2 AXGroup | actions: ["AXPress", "AXShowMenu", "AXScrollToVisible"]
3 AXWebArea | Plan refactors with Claude
$ agent-device-macos-helper press --x 378 --y 272 --surface app --bundle-id com.openai.codex
{"data":{"bundleId":"com.openai.codex","clicks":1,"mechanism":"ax-press","role":"AXButton","surface":"app","windowTitle":"ChatGPT","x":378,"y":272},"ok":true}
headings after: ['Scheduled', 'Schedule a task']

== fill prompt at 1078,819 — raw hit-test chain:
0 AXGroup | 1 AXGroup | 2 AXGroup
$ agent-device-macos-helper fill --x 1078 --y 819 --text "evidence: background fill" --bundle-id com.openai.codex
{"data":{"bundleId":"com.openai.codex","mechanism":"ax-value","role":"AXTextArea","windowTitle":"ChatGPT"},"ok":true}
prompt value: 'evidence: background fill'   (cleared afterwards)
frontmost: T3 Code (Nightly)

AXEnhancedUserInterface fallback. Codex rejects AXManualAccessibility with kAXErrorAttributeUnsupported (-25205) and populates after AXEnhancedUserInterface. That is the first live run in this thread; I did not repeat it cold, because that means quitting your Codex.

Small notes.

• The dead ?? in readTextAtPosition is gone.
• The 1 s wait is now paid only by the snapshot that turns the tree on. A tree already on returns at once; the Codex snapshot above took 253 ms.
• Enable failure is reported: if neither attribute can be set, or the tree does not populate in time, the snapshot carries a warnings entry. feat(macos): opt-in native app backend beside XCTest #3189 types the field and tests that it reaches the result.

Element identity instead of x/y. I kept point dispatch and recorded why in ADR 0031 ("Pointer dispatch, not element identity"):

• Every platform dispatches by point, and ADR 0011's occlusion, offscreen and parent-owned point guarantees are decided on that point. Identity dispatch would be a new path that needs its own guarantee row.
• An AXUIElement cannot outlive the one-shot helper process. An identity would be a tree path re-resolved against a tree that may have changed, which is the same resolution problem with weaker evidence.

Splitting. The stack already moved the ghost cursor into #3195. This layer stays one concern, the helper's app surface, at 926 gross lines. A further split would leave the AX actions PR without the screenshot it is verified with.

[thymikee]

This PR is ready at c0fabe6. The earlier evidence gap is closed: the author's live run now covers the AX routes, and the size question is answered by keeping point dispatch to match ADR 0011.

All 19 checks pass. CI builds the Swift helper but does not run the AX routes, so the live run in the author's comment is the only evidence for them. I did not reproduce that output. The AXEnhancedUserInterface fallback comes from one warm run, and the author says they did not repeat it cold. actionWindow, windowTitle and the snapshot warning have no Swift unit test because they need live AX. I read only the logical patch in the 6 PR files through the range-diff, and I did not read ADR 0031 in the stacked PR #3189. There are no conflicts.

Not blocking, and you can take or leave it: in SnapshotTraversal.swift, enableRemoteAccessibilityTree returns true as soon as AXManualAccessibility or AXEnhancedUserInterface already reads true. If an earlier snapshot set the attribute but timed out before the web area filled in, every later snapshot skips both the wait and the warning. Returning hasPopulatedWebArea(appElement) in that case would fix it and adds no wait.

Cubic has not reviewed this head yet, so please check its threads before merge.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-04 10:20 UTC