From 4b5291259d419ebe2e50fbf65ff7b4180c30c8f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Sat, 3 Oct 2026 21:42:32 +0200 Subject: [PATCH] feat(macos): opt-in native app backend beside XCTest `AGENT_DEVICE_MACOS_APP_BACKEND=native` routes macOS app sessions to the macOS helper's accessibility actions instead of the XCTest runner. The surface routing takes the backend explicitly, the native interactor refuses runner-only commands with a typed reason rather than starting XCTest, and helper refusals surface as UNSUPPORTED_OPERATION. ADR 0031 records the decision; docs cover the env vars and limits. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/adr/0031-macos-native-app-backend.md | 79 ++++++++ docs/adr/README.md | 1 + packages/contracts/src/facades/session.ts | 2 + .../contracts/src/session-surface.test.ts | 58 +++++- packages/contracts/src/session-surface.ts | 48 +++-- packages/platform-apple/src/interactions.ts | 3 +- packages/platform-apple/src/interactor.ts | 43 +++-- .../platform-apple/src/navigation/runtime.ts | 16 +- .../src/os/macos/app-backend.test.ts | 13 ++ .../src/os/macos/app-backend.ts | 7 + .../src/os/macos/helper.test.ts | 23 ++- .../platform-apple/src/os/macos/helper.ts | 78 +++++++- .../os/macos/native-app-interactor.test.ts | 176 ++++++++++++++++++ .../src/os/macos/native-app-interactor.ts | 153 +++++++++++++++ .../src/os/macos/native-backend-facts.test.ts | 41 ++++ .../src/os/macos/native-backend-facts.ts | 28 +++ .../src/os/macos/surface-snapshot.test.ts | 40 ++++ .../src/os/macos/surface-snapshot.ts | 2 +- .../src/runtime-snapshot.test.ts | 54 +++++- .../platform-apple/src/runtime-snapshot.ts | 33 +++- packages/platform-apple/src/runtime.test.ts | 46 +++++ packages/platform-apple/src/runtime.ts | 12 ++ src/daemon/screenshot-crop-target.ts | 5 +- src/platform-runtime-operation-host.test.ts | 21 +-- src/platform-runtime-operation-host.ts | 4 +- website/docs/docs/commands.md | 1 + website/docs/docs/configuration.md | 1 + 27 files changed, 918 insertions(+), 70 deletions(-) create mode 100644 docs/adr/0031-macos-native-app-backend.md create mode 100644 packages/platform-apple/src/os/macos/app-backend.test.ts create mode 100644 packages/platform-apple/src/os/macos/app-backend.ts create mode 100644 packages/platform-apple/src/os/macos/native-app-interactor.test.ts create mode 100644 packages/platform-apple/src/os/macos/native-app-interactor.ts create mode 100644 packages/platform-apple/src/os/macos/native-backend-facts.test.ts create mode 100644 packages/platform-apple/src/os/macos/native-backend-facts.ts create mode 100644 packages/platform-apple/src/os/macos/surface-snapshot.test.ts diff --git a/docs/adr/0031-macos-native-app-backend.md b/docs/adr/0031-macos-native-app-backend.md new file mode 100644 index 0000000000..1b8390cfd3 --- /dev/null +++ b/docs/adr/0031-macos-native-app-backend.md @@ -0,0 +1,79 @@ +# ADR 0031: macOS Native App Backend — Accessibility Actions Beside XCTest + +## Status + +Accepted (2026-10-03). Opt-in; XCTest stays the default app-session backend. + +## Rules at a glance + +1. `AGENT_DEVICE_MACOS_APP_BACKEND` selects the backend for macOS `app` sessions: `xctest` + (default) or `native`. It is a daemon setting read through `readMacOsAppBackend`: once by the + Apple runtime owner, on its first macOS device, and once per macOS interactor. Other Apple + devices never read it. An unknown value fails macOS sessions with `INVALID_ARGS`, never a + silent fallback. +2. `macOsSurfaceBackend(surface, appBackend)` in `packages/contracts/src/session-surface.ts` is the + one routing decision. With `native`, the `app` surface is helper-routed exactly like + `frontmost-app`, `desktop`, and `menubar`. +3. A native daemon never starts XCTest on macOS. `macOsNativeBackendFacts` refuses every macOS + operation only the runner serves (recording, `prepare`, `back`, press and hold, gestures and + their viewport) at admission, for every macOS session, with `UNSUPPORTED_OPERATION` and + `reason: 'unsupported-device-backend'`, the reason the physical-iOS XCTest backend uses. + `macOsNativeAppInteractor` is assembled member by member, so no runner-backed member reaches + it; an action that names no app is refused rather than handed to the runner. +4. Pointer actions are accessibility actions only: `AXPress`, focus, value, selected text, or a + scroll bar value. Press and hold is refused at admission; double, secondary, and middle clicks + are refused by the interactor before the helper runs. A press or fill with no pressable + element, or a scroll with no settable scroll bar, is refused by the helper and carries its + `helperReason` under the same `reason`; that vocabulary is pinned by + `contracts/fixtures/macos-native-helper-outcomes.json`. Only keyboard text falls back to events + posted to the app's process. +5. Snapshots of a Chromium-based session app (one shipping `chrome_100_percent.pak`) enable its + accessibility tree (`AXManualAccessibility`, else `AXEnhancedUserInterface`) before traversal. + Only the snapshot that turns the tree on waits for it (up to 1 s); a tree that does not + populate adds a snapshot warning. The tree stays on for the app's lifetime, as for any + assistive client. The app surface walks up to 48 levels deep and reports a deeper tree as + truncated; other helper surfaces keep 12. +6. Screenshots capture the session app's front window by itself through ScreenCaptureKit. + +## Context + +The XCTest runner drives a macOS app through XCUIApplication, which puts the host in Automation +Mode: a system overlay is shown and the runner moves the shared pointer. The macOS helper already +read the accessibility tree for helper surfaces, and the session snapshot vocabulary is the same, +so an app session can be served without a test session while the app stays behind the user's +windows. + +## Decision details + +**Pointer delivery.** Events posted to a process with `CGEventPostToPid` were measured against +Calculator (SwiftUI): mouse events were dropped whether the app was frontmost or in the background, +and wheel events were dropped in the background, while every post reported success. Keyboard events +were accepted by Calculator and by Electron apps in the background. A fallback that cannot be +observed to work would turn "nothing happened" into success, so pointer actions have no event +fallback. + +**Target resolution.** The helper hit-tests inside the session app (other apps' windows above it do +not answer) and walks at most four ancestors for a text input or a pressable control role. +Chromium answers a hit test with wrapper groups that all claim `AXPress`, so when the chain names +no control the helper picks the smallest such element whose frame contains the point, searching +only the window the hit landed in (the app's front on-screen window when the hit names none). A +group is pressed only when no control contains the point. Responses name the window acted in +(`windowTitle`) when the window has an accessibility title. This resolves the same point the daemon computed from the snapshot node; the +daemon dispatch paths and their ADR 0011 guarantees are unchanged. + +**Pointer dispatch, not element identity.** The daemon dispatches every platform by point, and +the occlusion, offscreen, and parent-owned touch-point guarantees of ADR 0011 are decided on that +point. Sending an element identity instead would be a new dispatch path with its own guarantee +row, and an `AXUIElement` cannot outlive the one-shot helper process that resolved it, so an +identity would be a tree path re-resolved against a tree that may have changed. + +## Rejected alternatives + +- **Suppressing Automation Mode.** `automationmodetool` removes the authentication prompt, not the + overlay, and the runner still owns the pointer. +- **Private SkyLight event delivery** (`SLEventPostToPid`, focus-without-raise). It would cover + pointer-only controls, but it is private API that can break with any macOS release. Revisit only + with evidence of apps the accessibility path cannot drive. +- **Native by default.** The accessibility path cannot express drags, holds, or double-clicks, and + apps with sparse accessibility trees still need the runner. Defaults change only with coverage + evidence across app frameworks. diff --git a/docs/adr/README.md b/docs/adr/README.md index e62cc253c2..7694aab843 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -32,6 +32,7 @@ | [0028 Capability-Family Cell Vocabulary — One Runtime Source (Proposed)](0028-capability-family-cell-vocabulary.md) | adding a capability operation family, `UnavailablePlatformRuntimeFacts` / `UNAVAILABLE_CELLS`, `INTERACTOR_OPERATIONS`, and why an eight-package fan-out recurs | | [0029 Daemon Policy](0029-daemon-policy.md) | `AGENT_DEVICE_DAEMON_POLICY`, confining a daemon's commands, devices, or device shutdown, and where operator rules are enforced for batch/replay steps | | [0030 Process Lock Exclusion](0030-process-lock-exclusion.md) | process-lock publication/reclaim/release, retained mutation guards, and the single-protocol upgrade boundary | +| [0031 macOS Native App Backend](0031-macos-native-app-backend.md) | `AGENT_DEVICE_MACOS_APP_BACKEND`, driving macOS app sessions without XCTest Automation Mode, and why pointer actions are accessibility actions only | ADRs record *why*; the registries and gates they describe are the living source of truth — when prose and a registry disagree, the registry wins and the ADR needs a follow-up. diff --git a/packages/contracts/src/facades/session.ts b/packages/contracts/src/facades/session.ts index 264075cdc0..732fb306d8 100644 --- a/packages/contracts/src/facades/session.ts +++ b/packages/contracts/src/facades/session.ts @@ -5,8 +5,10 @@ export { macOsHelperSurface, macOsSurfaceBackend, parseSessionSurface, + readMacOsAppBackend, } from '../session-surface.ts'; export type { + MacOsAppBackend, MacOsHelperSurface, MacOsSurfaceBackend, SessionSurface, diff --git a/packages/contracts/src/session-surface.test.ts b/packages/contracts/src/session-surface.test.ts index eb42f1cf9e..05a6dd5ebb 100644 --- a/packages/contracts/src/session-surface.test.ts +++ b/packages/contracts/src/session-surface.test.ts @@ -3,21 +3,59 @@ import { SESSION_SURFACES, macOsHelperSurface, macOsSurfaceBackend, + readMacOsAppBackend, + type MacOsAppBackend, type MacOsSurfaceBackend, type SessionSurface, } from './session-surface.ts'; -const EXPECTED_BACKENDS: Record = { - app: 'xctest', - 'frontmost-app': 'macos-helper', - desktop: 'macos-helper', - menubar: 'macos-helper', +const EXPECTED_BACKENDS: Record> = { + xctest: { + app: 'xctest', + 'frontmost-app': 'macos-helper', + desktop: 'macos-helper', + menubar: 'macos-helper', + }, + native: { + app: 'macos-helper', + 'frontmost-app': 'macos-helper', + desktop: 'macos-helper', + menubar: 'macos-helper', + }, }; +test.each( + (['xctest', 'native'] as const).flatMap((appBackend) => [ + ...SESSION_SURFACES.map( + (surface) => [appBackend, surface, EXPECTED_BACKENDS[appBackend][surface]] as const, + ), + [appBackend, undefined, EXPECTED_BACKENDS[appBackend].app] as const, + ]), +)( + 'with the %s app backend the macOS %s surface is served by %s', + (appBackend, surface, backend) => { + expect(macOsSurfaceBackend(surface, appBackend)).toBe(backend); + expect(macOsHelperSurface(surface, appBackend)).toBe( + backend === 'macos-helper' ? (surface ?? 'app') : undefined, + ); + }, +); + test.each([ - ...SESSION_SURFACES.map((surface) => [surface, EXPECTED_BACKENDS[surface]] as const), - [undefined, 'xctest'] as const, -])('the macOS %s surface is served by %s', (surface, backend) => { - expect(macOsSurfaceBackend(surface)).toBe(backend); - expect(macOsHelperSurface(surface)).toBe(backend === 'macos-helper' ? surface : undefined); + [undefined, 'xctest'], + ['', 'xctest'], + ['native', 'native'], + [' Native ', 'native'], + ['xctest', 'xctest'], +] as const)('AGENT_DEVICE_MACOS_APP_BACKEND=%j selects %s', (raw, expected) => { + expect( + readMacOsAppBackend((name) => { + expect(name).toBe('AGENT_DEVICE_MACOS_APP_BACKEND'); + return raw; + }), + ).toBe(expected); +}); + +test('an unknown app backend is refused instead of falling back', () => { + expect(() => readMacOsAppBackend(() => 'vision')).toThrow(/AGENT_DEVICE_MACOS_APP_BACKEND/); }); diff --git a/packages/contracts/src/session-surface.ts b/packages/contracts/src/session-surface.ts index ab3ddcb9b5..49946b2b9f 100644 --- a/packages/contracts/src/session-surface.ts +++ b/packages/contracts/src/session-surface.ts @@ -15,30 +15,54 @@ export function parseSessionSurface(value: string | undefined): SessionSurface { /** The backend that serves every operation on a macOS surface. */ export type MacOsSurfaceBackend = Extract; -const MACOS_SURFACE_BACKENDS = { - app: 'xctest', +/** + * Which backend drives a macOS app session. `xctest` is the runner under XCTest Automation Mode; + * `native` drives the app through the macOS helper's accessibility actions and process-targeted + * events, so the app can stay in the background and the user keeps the pointer. + */ +const MACOS_APP_BACKENDS = ['xctest', 'native'] as const; +export type MacOsAppBackend = (typeof MACOS_APP_BACKENDS)[number]; +const MACOS_APP_BACKEND_ENV = 'AGENT_DEVICE_MACOS_APP_BACKEND'; +const MACOS_APP_BACKEND_ENUM = defineStringEnum(MACOS_APP_BACKENDS, { + normalize: (raw) => raw.trim().toLowerCase(), + message: (value) => + `Invalid ${MACOS_APP_BACKEND_ENV}: ${value}. Use ${MACOS_APP_BACKENDS.join('|')}.`, +}); + +/** The host's app-session backend; unset selects `xctest`. */ +export function readMacOsAppBackend( + readEnvironment: (name: string) => string | undefined, +): MacOsAppBackend { + const raw = readEnvironment(MACOS_APP_BACKEND_ENV); + return raw === undefined || raw.trim() === '' ? 'xctest' : MACOS_APP_BACKEND_ENUM.parse(raw); +} + +const MACOS_HELPER_SURFACE_BACKENDS = { 'frontmost-app': 'macos-helper', desktop: 'macos-helper', menubar: 'macos-helper', -} as const satisfies Record; +} as const satisfies Record, MacOsSurfaceBackend>; /** An absent surface is an app session, the reading every route already gives it. */ -export function macOsSurfaceBackend(surface: SessionSurface | undefined): MacOsSurfaceBackend { - return MACOS_SURFACE_BACKENDS[surface ?? 'app']; +export function macOsSurfaceBackend( + surface: SessionSurface | undefined, + appBackend: MacOsAppBackend, +): MacOsSurfaceBackend { + const resolved = surface ?? 'app'; + if (resolved === 'app') return appBackend === 'native' ? 'macos-helper' : 'xctest'; + return MACOS_HELPER_SURFACE_BACKENDS[resolved]; } -type HelperRoutedSurface = { - [S in SessionSurface]: (typeof MACOS_SURFACE_BACKENDS)[S] extends 'macos-helper' ? S : never; -}[SessionSurface]; - declare const helperSurface: unique symbol; /** A surface the owner routed to the macOS helper; only `macOsHelperSurface` produces one. */ -export type MacOsHelperSurface = HelperRoutedSurface & { readonly [helperSurface]: true }; +export type MacOsHelperSurface = SessionSurface & { readonly [helperSurface]: true }; export function macOsHelperSurface( surface: SessionSurface | undefined, + appBackend: MacOsAppBackend, ): MacOsHelperSurface | undefined { - return surface !== undefined && macOsSurfaceBackend(surface) === 'macos-helper' - ? (surface as MacOsHelperSurface) + const resolved = surface ?? 'app'; + return macOsSurfaceBackend(resolved, appBackend) === 'macos-helper' + ? (resolved as MacOsHelperSurface) : undefined; } diff --git a/packages/platform-apple/src/interactions.ts b/packages/platform-apple/src/interactions.ts index 4cb595c02a..f1477d68f1 100644 --- a/packages/platform-apple/src/interactions.ts +++ b/packages/platform-apple/src/interactions.ts @@ -206,7 +206,8 @@ async function runApplePressPoint( point: { x: number; y: number }, options: PressPointOptions, ): Promise> { - const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined; + // The runner owner's routing: an app session reaches here only on the XCTest backend. + const helper = isMacOs(device) ? macOsHelperSurface(options.surface, 'xctest') : undefined; if (helper) { return await runMacOsSurfacePress(context, point, options, helper); } diff --git a/packages/platform-apple/src/interactor.ts b/packages/platform-apple/src/interactor.ts index fac6673e02..cca3535875 100644 --- a/packages/platform-apple/src/interactor.ts +++ b/packages/platform-apple/src/interactor.ts @@ -14,7 +14,11 @@ import { } from './runner/index.ts'; import { toAppleTvRemoteButton } from '@agent-device/contracts/tv-remote'; import { SCREENSHOT_FULLSCREEN_REASONS } from '@agent-device/contracts/capture'; -import { macOsHelperSurface, type MacOsHelperSurface } from '@agent-device/contracts/session'; +import { + macOsHelperSurface, + type MacOsHelperSurface, + type SessionSurface, +} from '@agent-device/contracts/session'; import { DEVICE_ROTATIONS, type DeviceRotation } from '@agent-device/contracts/device'; import { normalizeSnapshotScope } from '@agent-device/contracts/snapshot'; import { withDiagnosticTimer } from '@agent-device/host-kit/diagnostics'; @@ -31,6 +35,8 @@ import type { SnapshotOptions, } from '@agent-device/contracts/interactor-types'; import { captureMacOsSurfaceSnapshot } from './os/macos/surface-snapshot.ts'; +import { hostMacOsAppBackend } from './os/macos/app-backend.ts'; +import { macOsNativeAppInteractor } from './os/macos/native-app-interactor.ts'; import { presentAppleRunnerSnapshot, readAppleSnapshotResult, @@ -55,6 +61,9 @@ export function createAppleInteractor( ); } const { overrides, runnerOpts } = iosRunnerOverrides(device, runnerContext); + const appBackend = isMacOs(device) ? hostMacOsAppBackend() : 'xctest'; + const helperSurface = (surface: SessionSurface | undefined) => + isMacOs(device) ? macOsHelperSurface(surface, appBackend) : undefined; const interactor: Interactor = { open: (app, options) => openIosApp(device, app, { @@ -67,14 +76,16 @@ export function createAppleInteractor( }), openDevice: () => openIosDevice(device), close: (app) => closeIosApp(device, app, runnerOpts), - screenshot: (outPath, options) => runAppleScreenshot(device, outPath, options, runnerOpts), - snapshot: async (options) => await captureAppleSnapshot(device, options, runnerOpts), + screenshot: (outPath, options) => + runAppleScreenshot(device, outPath, options, runnerOpts, helperSurface(options?.surface)), + snapshot: async (options) => + await captureAppleSnapshot(device, options, runnerOpts, helperSurface(options?.surface)), // The live text at a point: helper for a helper-routed macOS surface, XCTest runner for - // every other Apple leaf including a macOS app session. + // every other Apple leaf. readTextAtPoint: async (point, options) => { - const helper = isMacOs(device) ? macOsHelperSurface(options?.surface) : undefined; + const helper = helperSurface(options?.surface); return helper - ? await readMacOsSurfaceTextAtPoint(point, helper, options?.appBundleId) + ? await readMacOsSurfaceTextAtPoint(point, helper, options?.appBundleId, options?.signal) : await readRunnerTextAtPoint(device, point, options, runnerOpts); }, // The XCTest runner's own text reading: it observes the live accessibility hierarchy @@ -203,16 +214,18 @@ export function createAppleInteractor( dismissAlert: (options) => actOnAppleAlert(device, runnerOpts, 'dismiss', options), ...overrides, }; - if (!runnerProvider) return interactor; - return withInjectedAppleRunnerTransport(device, runnerContext, interactor, runnerProvider); + const served = + appBackend === 'native' ? macOsNativeAppInteractor(interactor, runnerContext) : interactor; + if (!runnerProvider) return served; + return withInjectedAppleRunnerTransport(device, runnerContext, served, runnerProvider); } async function captureAppleSnapshot( device: DeviceInfo, options: SnapshotOptions | undefined, runnerOpts: RunnerCallOptions, + helper: MacOsHelperSurface | undefined, ) { - const helper = isMacOs(device) ? macOsHelperSurface(options?.surface) : undefined; if (helper) { return await captureMacOsSurfaceSnapshot({ ...options, surface: helper }, options?.signal); } @@ -388,20 +401,24 @@ async function runAppleScreenshot( outPath: string, options: ScreenshotOptions = {}, runnerOpts: RunnerCallOptions, + helper: MacOsHelperSurface | undefined, ): Promise { - const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined; if (helper) { if (options.fullscreen) { throw new AppError( 'INVALID_ARGS', - `screenshot --fullscreen is not accepted on the macOS ${helper} surface: it always captures the main display`, + `screenshot --fullscreen is not accepted on the macOS ${helper} surface: its capture frame is fixed`, { reason: SCREENSHOT_FULLSCREEN_REASONS.macOsHelperSurfaceFixedFrame, surface: helper, }, ); } - await runMacOsScreenshotAction(outPath, { surface: helper }); + await runMacOsScreenshotAction(outPath, { + surface: helper, + ...(helper === 'app' ? { bundleId: options.appBundleId } : {}), + signal: runnerOpts.signal, + }); return {}; } if (options.captureBackend === 'runner') { @@ -430,11 +447,13 @@ async function readMacOsSurfaceTextAtPoint( point: Point, surface: MacOsHelperSurface, appBundleId: string | undefined, + signal: AbortSignal | undefined, ): Promise { const { runMacOsReadTextAction } = await import('./os/macos/helper.ts'); const result = await runMacOsReadTextAction(point.x, point.y, { bundleId: appBundleId, surface, + signal, }); return result.text; } diff --git a/packages/platform-apple/src/navigation/runtime.ts b/packages/platform-apple/src/navigation/runtime.ts index eb0d8af2e9..dba50102a3 100644 --- a/packages/platform-apple/src/navigation/runtime.ts +++ b/packages/platform-apple/src/navigation/runtime.ts @@ -183,17 +183,27 @@ export function appleNavigationFacts(device: DeviceInfo) { }); } -/** Binds whichever navigation operations {@link appleNavigationFacts} admitted. */ +/** + * Binds whichever navigation operations the device's admitted facts allow. The cells are those + * {@link appleNavigationFacts} declares, read from the admitted facts so a refusal layered over + * the leaf (a backend that cannot serve one) also withholds the binding. + */ export function createAppleNavigationOperations(params: { host: Pick; device: DeviceInfo; signal: AbortSignal; + admitted: Readonly>; }) { - const { host, device, signal } = params; + const { host, device, signal, admitted } = params; + const navigationKeys = Object.keys(appleNavigationFacts(device)) as Array< + keyof ReturnType + >; return bindAdmittedLocalInteractorOperations({ device, signal, resolveInteractor: host.localInteractors.resolve, - facts: appleNavigationFacts(device), + facts: Object.fromEntries(navigationKeys.map((key) => [key, admitted[key]])) as ReturnType< + typeof appleNavigationFacts + >, }); } diff --git a/packages/platform-apple/src/os/macos/app-backend.test.ts b/packages/platform-apple/src/os/macos/app-backend.test.ts new file mode 100644 index 0000000000..939e1f8415 --- /dev/null +++ b/packages/platform-apple/src/os/macos/app-backend.test.ts @@ -0,0 +1,13 @@ +import { afterEach, expect, test, vi } from 'vitest'; +import { hostMacOsAppBackend } from './app-backend.ts'; + +afterEach(() => { + vi.unstubAllEnvs(); +}); + +test('the host keeps XCTest for app sessions unless it opts into the native backend', () => { + vi.stubEnv('AGENT_DEVICE_MACOS_APP_BACKEND', ''); + expect(hostMacOsAppBackend()).toBe('xctest'); + vi.stubEnv('AGENT_DEVICE_MACOS_APP_BACKEND', 'native'); + expect(hostMacOsAppBackend()).toBe('native'); +}); diff --git a/packages/platform-apple/src/os/macos/app-backend.ts b/packages/platform-apple/src/os/macos/app-backend.ts new file mode 100644 index 0000000000..5f8d6974a4 --- /dev/null +++ b/packages/platform-apple/src/os/macos/app-backend.ts @@ -0,0 +1,7 @@ +import { readMacOsAppBackend, type MacOsAppBackend } from '@agent-device/contracts/session'; +import { readHostEnvironmentVariable } from '@agent-device/host-kit/process'; + +/** The app-session backend the host selected through `AGENT_DEVICE_MACOS_APP_BACKEND`. */ +export function hostMacOsAppBackend(): MacOsAppBackend { + return readMacOsAppBackend(readHostEnvironmentVariable); +} diff --git a/packages/platform-apple/src/os/macos/helper.test.ts b/packages/platform-apple/src/os/macos/helper.test.ts index a6d129fbe6..777c0caa92 100644 --- a/packages/platform-apple/src/os/macos/helper.test.ts +++ b/packages/platform-apple/src/os/macos/helper.test.ts @@ -2,7 +2,10 @@ import assert from 'node:assert/strict'; import { test } from 'vitest'; import { macOsHelperSurface } from '@agent-device/contracts/session'; import { createLocalAppleToolProvider, withAppleToolProvider } from '../../core/tool-provider.ts'; +import { readFileSync } from 'node:fs'; import { + MACOS_DELIVERY_MECHANISMS, + MACOS_HELPER_REFUSAL_REASONS, macOsClickScheduleMs, runMacOsPressAction, runMacOsReadTextAction, @@ -10,9 +13,9 @@ import { runMacOsSnapshotAction, } from './helper.ts'; -const desktop = macOsHelperSurface('desktop')!; -const menubar = macOsHelperSurface('menubar')!; -const frontmostApp = macOsHelperSurface('frontmost-app')!; +const desktop = macOsHelperSurface('desktop', 'xctest')!; +const menubar = macOsHelperSurface('menubar', 'xctest')!; +const frontmostApp = macOsHelperSurface('frontmost-app', 'xctest')!; test('macOS helper snapshot passes cancellation to the helper process', async () => { const controller = new AbortController(); @@ -255,3 +258,17 @@ test('helper entry points accept only an owner-routed surface', () => { }; assert.equal(typeof widenedCalls, 'function'); }); + +test('the helper vocabulary matches the table the Swift helper is held to', () => { + const fixture = JSON.parse( + readFileSync( + new URL( + '../../../../../contracts/fixtures/macos-native-helper-outcomes.json', + import.meta.url, + ), + 'utf8', + ), + ) as { refusalReasons: string[]; deliveryMechanisms: string[] }; + assert.deepEqual([...MACOS_HELPER_REFUSAL_REASONS], fixture.refusalReasons); + assert.deepEqual([...MACOS_DELIVERY_MECHANISMS], fixture.deliveryMechanisms); +}); diff --git a/packages/platform-apple/src/os/macos/helper.ts b/packages/platform-apple/src/os/macos/helper.ts index 599a51e82a..135f65f3f4 100644 --- a/packages/platform-apple/src/os/macos/helper.ts +++ b/packages/platform-apple/src/os/macos/helper.ts @@ -112,6 +112,28 @@ function appendMacOsHelperContextArgs( } } +/** + * The helper's app-surface vocabulary, pinned with its Swift enums by + * `contracts/fixtures/macos-native-helper-outcomes.json`. + */ +export const MACOS_HELPER_REFUSAL_REASONS = [ + 'background-pointer-gesture', + 'no-accessible-target', + 'no-settable-text-input', + 'no-scroll-bar', +] as const; +export const MACOS_DELIVERY_MECHANISMS = [ + 'ax-press', + 'ax-focus', + 'ax-value', + 'ax-selected-text', + 'ax-confirm', + 'ax-scroll-bar', + 'key-events', +] as const; +/** How an app-surface action reached the app, as the helper reports it. */ +export type MacOsDeliveryMechanism = (typeof MACOS_DELIVERY_MECHANISMS)[number]; + export function resolveMacOsHelperPackageRootFrom(modulePath: string): string { let currentDir = path.dirname(modulePath); while (true) { @@ -379,6 +401,8 @@ export async function runMacOsSnapshotAction( nodes: MacOsSnapshotNode[]; truncated: boolean; backend: 'macos-helper'; + /** Set when the app's Chromium accessibility tree could not be turned on in time. */ + warnings?: string[]; }> { const args = ['snapshot', '--surface', surface]; appendMacOsHelperContextArgs(args, options); @@ -388,13 +412,13 @@ export async function runMacOsSnapshotAction( export async function runMacOsReadTextAction( x: number, y: number, - options: { surface: MacOsHelperSurface; bundleId?: string }, + options: { surface: MacOsHelperSurface; bundleId?: string; signal?: AbortSignal }, ): Promise<{ text: string; }> { const args = ['read', '--x', String(x), '--y', String(y)]; appendMacOsHelperContextArgs(args, options); - return await runMacOsHelper(args); + return await runMacOsHelper(args, { signal: options.signal }); } // Mirrors the helper's own floors (`MouseClickSchedule.swift`): the schedule the helper runs @@ -448,6 +472,9 @@ export async function runMacOsPressAction( doubleClick?: boolean; bundleId?: string; surface?: SessionSurface; + mechanism?: MacOsDeliveryMechanism; + /** The window the helper acted in, for an app-surface press. */ + windowTitle?: string; }> { const args = ['press', '--x', String(x), '--y', String(y)]; if (options.holdMs && options.holdMs > 0) { @@ -471,14 +498,57 @@ export async function runMacOsPressAction( }); } +/** Inserts text at the app session's focused element without activating the app. */ +export async function runMacOsTypeAction( + text: string, + options: { bundleId: string; delayMs?: number; signal?: AbortSignal }, +): Promise<{ mechanism?: MacOsDeliveryMechanism; role?: string; windowTitle?: string }> { + const args = ['type', '--text', text]; + if (options.delayMs && options.delayMs > 0) args.push('--delay-ms', String(options.delayMs)); + appendMacOsHelperContextArgs(args, { bundleId: options.bundleId }); + return await runMacOsHelper(args, { + signal: options.signal, + timeoutMs: MACOS_HELPER_TIMEOUT_MS + text.length * (options.delayMs ?? 0), + }); +} + +/** Replaces the value of the app session's text input at a point. */ +export async function runMacOsFillAction( + x: number, + y: number, + text: string, + options: { bundleId: string; signal?: AbortSignal }, +): Promise<{ mechanism?: MacOsDeliveryMechanism; role?: string; windowTitle?: string }> { + const args = ['fill', '--x', String(x), '--y', String(y), '--text', text]; + appendMacOsHelperContextArgs(args, { bundleId: options.bundleId }); + return await runMacOsHelper(args, { signal: options.signal }); +} + +/** Scrolls the scroll area at the center of the app session's front window. */ +export async function runMacOsScrollAction( + direction: 'up' | 'down' | 'left' | 'right', + options: { + bundleId: string; + amount?: number; + pixels?: number; + signal?: AbortSignal; + }, +): Promise> { + const args = ['scroll', '--direction', direction]; + if (options.amount !== undefined) args.push('--amount', String(options.amount)); + if (options.pixels !== undefined) args.push('--pixels', String(options.pixels)); + appendMacOsHelperContextArgs(args, { bundleId: options.bundleId }); + return await runMacOsHelper(args, { signal: options.signal }); +} + export async function runMacOsScreenshotAction( outPath: string, - options: { surface: MacOsHelperSurface }, + options: { surface: MacOsHelperSurface; bundleId?: string; signal?: AbortSignal }, ): Promise<{ path: string; surface?: SessionSurface; }> { const args = ['screenshot', '--out', outPath]; appendMacOsHelperContextArgs(args, options); - return await runMacOsHelper(args); + return await runMacOsHelper(args, { signal: options.signal }); } diff --git a/packages/platform-apple/src/os/macos/native-app-interactor.test.ts b/packages/platform-apple/src/os/macos/native-app-interactor.test.ts new file mode 100644 index 0000000000..6380d86f34 --- /dev/null +++ b/packages/platform-apple/src/os/macos/native-app-interactor.test.ts @@ -0,0 +1,176 @@ +import assert from 'node:assert/strict'; +import { test } from 'vitest'; +import type { Interactor, RunnerContext } from '@agent-device/contracts/interactor-types'; +import { AppError } from '@agent-device/kernel/errors'; +import { createLocalAppleToolProvider, withAppleToolProvider } from '../../core/tool-provider.ts'; +import { macOsNativeAppInteractor } from './native-app-interactor.ts'; + +const context: RunnerContext = { appBundleId: 'com.apple.TextEdit' }; + +/** The XCTest-backed interactor the native one draws from; any member it calls is recorded. */ +function runnerInteractor(reached: string[] = []): Interactor { + return new Proxy({} as Interactor, { + get: (_target, member) => async () => { + reached.push(String(member)); + return {}; + }, + }); +} + +function nativeInteractor(ctx: RunnerContext = context, reached: string[] = []): Interactor { + return macOsNativeAppInteractor(runnerInteractor(reached), ctx); +} + +async function recordHelperCalls( + data: Record, + run: (interactor: Interactor) => Promise, +): Promise<{ calls: string[][]; result: unknown }> { + const calls: string[][] = []; + const reachedRunner: string[] = []; + const provider = createLocalAppleToolProvider({ + macosHelper: { + run: async (args) => { + calls.push([...args]); + return { exitCode: 0, stdout: JSON.stringify({ ok: true, data }), stderr: '' }; + }, + }, + }); + const result = await withAppleToolProvider( + provider, + async () => await run(nativeInteractor(context, reachedRunner)), + ); + // Served through the helper alone: the XCTest-backed interactor never ran. + assert.deepEqual(reachedRunner, []); + return { calls, result }; +} + +test('a native tap presses the app surface and reports the mechanism', async () => { + const { calls, result } = await recordHelperCalls( + { x: 10, y: 20, mechanism: 'ax-press', windowTitle: 'Untitled' }, + async (interactor) => await interactor.tap(10, 20), + ); + assert.deepEqual(calls, [ + ['press', '--x', '10', '--y', '20', '--bundle-id', 'com.apple.TextEdit', '--surface', 'app'], + ]); + assert.deepEqual(result, { mechanism: 'ax-press', windowTitle: 'Untitled' }); +}); + +test('native type and fill address the session app, not the frontmost one', async () => { + const { calls } = await recordHelperCalls({ mechanism: 'ax-value' }, async (interactor) => { + await interactor.type('hello'); + await interactor.fill(5, 6, 'world'); + }); + assert.deepEqual(calls, [ + ['type', '--text', 'hello', '--bundle-id', 'com.apple.TextEdit'], + ['fill', '--x', '5', '--y', '6', '--text', 'world', '--bundle-id', 'com.apple.TextEdit'], + ]); +}); + +test('a native scroll reports travel from the window frame the helper resolved', async () => { + const { calls, result } = await recordHelperCalls( + { + x: 480, + y: 434, + x2: 480, + y2: 134, + referenceWidth: 656, + referenceHeight: 422, + travelPixels: 300, + mechanism: 'ax-scroll-bar', + }, + async (interactor) => await interactor.scroll('down', { pixels: 300 }), + ); + assert.deepEqual(calls, [ + ['scroll', '--direction', 'down', '--pixels', '300', '--bundle-id', 'com.apple.TextEdit'], + ]); + assert.deepEqual(result, { + x1: 480, + y1: 434, + x2: 480, + y2: 134, + referenceWidth: 656, + referenceHeight: 422, + pixels: 300, + mechanism: 'ax-scroll-bar', + }); +}); + +test('runner-only members refuse with the backend reason instead of starting XCTest', async () => { + const reached: string[] = []; + const interactor = nativeInteractor(context, reached); + for (const refused of [ + () => interactor.back(), + () => interactor.setOrientation('portrait'), + () => interactor.pressPoint!({ x: 1, y: 2 }, secondaryClick), + () => interactor.pressPoint!({ x: 1, y: 2 }, { ...primaryClick, doubleTap: true }), + () => interactor.pressPoint!({ x: 1, y: 2 }, { ...primaryClick, holdMs: 500 }), + () => interactor.longPress(1, 2, 500), + ]) { + await assert.rejects(refused, isNativeBackendRefusal); + } + assert.equal(interactor.gestureViewport, undefined); + assert.equal(interactor.doubleTap, undefined); + assert.equal(interactor.findText, undefined); + assert.deepEqual(reached, []); +}); + +test('an action without an app session refuses rather than falling back to the runner', async () => { + const reached: string[] = []; + const interactor = nativeInteractor({}, reached); + await assert.rejects(() => interactor.type('hello'), isNativeBackendRefusal); + await assert.rejects(() => interactor.scroll('down'), isNativeBackendRefusal); + assert.deepEqual(reached, []); +}); + +test('a press on another surface stays with the helper press of that surface', async () => { + const reached: string[] = []; + await nativeInteractor(context, reached).pressPoint!( + { x: 1, y: 2 }, + { ...primaryClick, surface: 'menubar' }, + ); + assert.deepEqual(reached, ['pressPoint']); +}); + +test('a helper refusal surfaces as an unsupported operation that keeps the helper reason', async () => { + const provider = createLocalAppleToolProvider({ + macosHelper: { + run: async () => ({ + exitCode: 1, + stdout: JSON.stringify({ + ok: false, + error: { + message: 'no pressable accessibility element at the point', + details: { reason: 'no-accessible-target', bundleId: 'com.apple.TextEdit' }, + }, + }), + stderr: '', + }), + }, + }); + await assert.rejects( + async () => + await withAppleToolProvider(provider, async () => await nativeInteractor().tap(1, 2)), + (error: unknown) => + isNativeBackendRefusal(error) && + (error as AppError).details?.helperReason === 'no-accessible-target', + ); +}); + +const primaryClick = { + button: 'primary', + count: 1, + intervalMs: 0, + holdMs: 0, + jitterPx: 0, + doubleTap: false, +} as const; + +const secondaryClick = { ...primaryClick, button: 'secondary' } as const; + +function isNativeBackendRefusal(error: unknown): boolean { + return ( + error instanceof AppError && + error.code === 'UNSUPPORTED_OPERATION' && + error.details?.reason === 'unsupported-device-backend' + ); +} diff --git a/packages/platform-apple/src/os/macos/native-app-interactor.ts b/packages/platform-apple/src/os/macos/native-app-interactor.ts new file mode 100644 index 0000000000..630d04c27b --- /dev/null +++ b/packages/platform-apple/src/os/macos/native-app-interactor.ts @@ -0,0 +1,153 @@ +import type { + Interactor, + PressPointOptions, + RunnerContext, +} from '@agent-device/contracts/interactor-types'; +import { macOsHelperSurface } from '@agent-device/contracts/session'; +import { assertScrollGestureInput } from '@agent-device/contracts/scroll-gesture'; +import { AppError } from '@agent-device/kernel/errors'; +import { normalizeAppleScrollResultWithResolvedFrame } from '../../core/scroll.ts'; +import { + MACOS_HELPER_REFUSAL_REASONS, + runMacOsFillAction, + runMacOsPressAction, + runMacOsScrollAction, + runMacOsTypeAction, +} from './helper.ts'; + +const NATIVE_APP_SURFACE = macOsHelperSurface('app', 'native')!; + +const NATIVE_BACKEND_HINT = + 'The native macOS app backend acts through accessibility actions only. Unset AGENT_DEVICE_MACOS_APP_BACKEND to drive this app with XCTest.'; + +const REFUSAL_REASONS: ReadonlySet = new Set(MACOS_HELPER_REFUSAL_REASONS); + +/** + * The macOS interactor of the native backend. It is assembled member by member so that nothing + * reaches the XCTest runner: actions on the session app go through the macOS helper, and only + * members whose macOS implementation uses local tooling or the helper are taken from `base`. + * Runner-only commands are refused at admission by `macOsNativeBackendFacts`; the members the + * interface requires for them refuse the same way. + */ +export function macOsNativeAppInteractor(base: Interactor, ctx: RunnerContext): Interactor { + const bundleId = (): string => { + if (ctx.appBundleId) return ctx.appBundleId; + throw nativeBackendRefusal('an action on a session that names no app'); + }; + const press = async ( + point: { x: number; y: number }, + options: Partial> = {}, + ) => { + const result = await withHelperRefusals( + runMacOsPressAction(point.x, point.y, { + surface: NATIVE_APP_SURFACE, + bundleId: bundleId(), + clicks: options.count, + intervalMs: options.intervalMs, + signal: ctx.signal, + }), + ); + return actedOn(result); + }; + return { + open: base.open, + openDevice: base.openDevice, + close: base.close, + screenshot: base.screenshot, + snapshot: base.snapshot, + readTextAtPoint: base.readTextAtPoint, + readClipboard: base.readClipboard, + writeClipboard: base.writeClipboard, + setSetting: base.setSetting, + readAlert: base.readAlert, + awaitAlert: base.awaitAlert, + acceptAlert: base.acceptAlert, + dismissAlert: base.dismissAlert, + tap: async (x, y) => await press({ x, y }), + pressPoint: async (point, options) => { + // Another surface's press is the helper's own surface press, never the runner's. + if (options.surface !== undefined && options.surface !== 'app') { + return await base.pressPoint!(point, options); + } + if (options.button !== 'primary') throw nativeBackendRefusal(`${options.button} click`); + if (options.doubleTap) throw nativeBackendRefusal('double-click'); + if (options.holdMs > 0) throw nativeBackendRefusal('press and hold'); + return await press(point, options); + }, + longPress: async () => { + throw nativeBackendRefusal('press and hold'); + }, + focus: async (x, y) => await press({ x, y }), + type: async (text, delayMs) => { + await withHelperRefusals( + runMacOsTypeAction(text, { bundleId: bundleId(), delayMs, signal: ctx.signal }), + ); + }, + fill: async (x, y, text) => { + const result = await withHelperRefusals( + runMacOsFillAction(x, y, text, { bundleId: bundleId(), signal: ctx.signal }), + ); + return actedOn(result); + }, + scroll: async (direction, options) => { + assertScrollGestureInput(options ?? {}); + const result = await withHelperRefusals( + runMacOsScrollAction(direction, { + bundleId: bundleId(), + amount: options?.amount, + pixels: options?.pixels, + signal: ctx.signal, + }), + ); + return { + ...normalizeAppleScrollResultWithResolvedFrame(result, direction, options, { + includeDuration: false, + }), + mechanism: result.mechanism, + }; + }, + back: async () => { + throw nativeBackendRefusal('back'); + }, + setOrientation: async () => { + throw nativeBackendRefusal('orientation'); + }, + }; +} + +/** What the helper reports about the element it acted on. */ +function actedOn(result: { mechanism?: string; windowTitle?: string }) { + return { + mechanism: result.mechanism, + ...(result.windowTitle === undefined ? {} : { windowTitle: result.windowTitle }), + }; +} + +function nativeBackendRefusal(action: string): AppError { + return new AppError( + 'UNSUPPORTED_OPERATION', + `${action} is not supported by the native macOS app backend.`, + { reason: 'unsupported-device-backend', hint: NATIVE_BACKEND_HINT }, + ); +} + +async function withHelperRefusals(action: Promise): Promise { + try { + return await action; + } catch (error) { + if (!(error instanceof AppError)) throw error; + const reason = error.details?.reason; + if (typeof reason !== 'string' || !REFUSAL_REASONS.has(reason)) throw error; + throw new AppError( + 'UNSUPPORTED_OPERATION', + error.message, + { + ...error.details, + reason: 'unsupported-device-backend', + helperReason: reason, + hint: NATIVE_BACKEND_HINT, + }, + error, + ); + } +} diff --git a/packages/platform-apple/src/os/macos/native-backend-facts.test.ts b/packages/platform-apple/src/os/macos/native-backend-facts.test.ts new file mode 100644 index 0000000000..a6a5422ee1 --- /dev/null +++ b/packages/platform-apple/src/os/macos/native-backend-facts.test.ts @@ -0,0 +1,41 @@ +import { expect, test } from 'vitest'; +import type { DeviceInfo } from '@agent-device/kernel/device'; +import { macOsNativeBackendFacts } from './native-backend-facts.ts'; + +const mac = { + platform: 'apple', + appleOs: 'macos', + id: 'host', + name: 'Mac', + kind: 'device', + target: 'desktop', +} as const satisfies DeviceInfo; + +test('the native backend refuses every runner-only macOS operation at admission', () => { + const facts = macOsNativeBackendFacts(mac, 'native'); + for (const operation of [ + 'screenRecordingStart', + 'screenRecordingReattach', + 'prepareAppleRunner', + 'longPressPoint', + 'back', + 'performGesturePlan', + 'performDirectionalFlingPlan', + 'performMultiTouchGesturePlan', + 'performTargetAuthoredDrag', + 'gestureViewport', + ]) { + expect(facts).toHaveProperty(operation, { + available: false, + reason: 'unsupported-device-backend', + hint: expect.stringContaining('AGENT_DEVICE_MACOS_APP_BACKEND'), + }); + } +}); + +test('the XCTest backend and other Apple devices keep their own facts', () => { + expect(macOsNativeBackendFacts(mac, 'xctest')).toEqual({}); + expect(macOsNativeBackendFacts({ ...mac, appleOs: 'ios', kind: 'simulator' }, 'native')).toEqual( + {}, + ); +}); diff --git a/packages/platform-apple/src/os/macos/native-backend-facts.ts b/packages/platform-apple/src/os/macos/native-backend-facts.ts new file mode 100644 index 0000000000..2a4544b505 --- /dev/null +++ b/packages/platform-apple/src/os/macos/native-backend-facts.ts @@ -0,0 +1,28 @@ +import { backRuntimeOperationFacts } from '@agent-device/contracts/back-runtime'; +import { gestureRuntimeOperationFacts } from '@agent-device/contracts/gesture-runtime'; +import type { RuntimeOperationUnavailability } from '@agent-device/contracts/platform-runtime'; +import type { MacOsAppBackend } from '@agent-device/contracts/session'; +import { isMacOs, type DeviceInfo } from '@agent-device/kernel/device'; + +const nativeBackendUnavailable: RuntimeOperationUnavailability = Object.freeze({ + available: false, + reason: 'unsupported-device-backend', + hint: 'The native macOS app backend never starts the XCTest runner, which this command needs. Unset AGENT_DEVICE_MACOS_APP_BACKEND to use XCTest.', +}); + +/** + * The macOS operations only the XCTest runner can serve. Under the native backend they are + * refused at admission, before dispatch, for every macOS session of the daemon: the backend is a + * daemon setting, so no session on it may start the runner. Spread last over the leaf's facts. + */ +export function macOsNativeBackendFacts(device: DeviceInfo, appBackend: MacOsAppBackend) { + if (!isMacOs(device) || appBackend !== 'native') return {}; + return Object.freeze({ + screenRecordingStart: nativeBackendUnavailable, + screenRecordingReattach: nativeBackendUnavailable, + prepareAppleRunner: nativeBackendUnavailable, + longPressPoint: nativeBackendUnavailable, + ...backRuntimeOperationFacts({ back: nativeBackendUnavailable }), + ...gestureRuntimeOperationFacts({ unsupported: nativeBackendUnavailable }), + }); +} diff --git a/packages/platform-apple/src/os/macos/surface-snapshot.test.ts b/packages/platform-apple/src/os/macos/surface-snapshot.test.ts new file mode 100644 index 0000000000..35f1639aea --- /dev/null +++ b/packages/platform-apple/src/os/macos/surface-snapshot.test.ts @@ -0,0 +1,40 @@ +import { expect, test } from 'vitest'; +import { macOsHelperSurface } from '@agent-device/contracts/session'; +import { createLocalAppleToolProvider, withAppleToolProvider } from '../../core/tool-provider.ts'; +import { captureMacOsSurfaceSnapshot } from './surface-snapshot.ts'; + +test('an app snapshot carries the helper warning that web content may be missing', async () => { + const warning = "The app's Chromium accessibility tree did not populate."; + const calls: string[][] = []; + const provider = createLocalAppleToolProvider({ + macosHelper: { + run: async (args) => { + calls.push([...args]); + return { + exitCode: 0, + stdout: JSON.stringify({ + ok: true, + data: { + surface: 'app', + nodes: [], + truncated: false, + backend: 'macos-helper', + warnings: [warning], + }, + }), + stderr: '', + }; + }, + }, + }); + const result = await withAppleToolProvider( + provider, + async () => + await captureMacOsSurfaceSnapshot({ + surface: macOsHelperSurface('app', 'native')!, + appBundleId: 'com.openai.codex', + }), + ); + expect(calls).toEqual([['snapshot', '--surface', 'app', '--bundle-id', 'com.openai.codex']]); + expect(result.warnings).toEqual([warning]); +}); diff --git a/packages/platform-apple/src/os/macos/surface-snapshot.ts b/packages/platform-apple/src/os/macos/surface-snapshot.ts index 28df3a9693..36398e05bc 100644 --- a/packages/platform-apple/src/os/macos/surface-snapshot.ts +++ b/packages/platform-apple/src/os/macos/surface-snapshot.ts @@ -13,7 +13,7 @@ export async function captureMacOsSurfaceSnapshot( const surface = options.surface; const { runMacOsSnapshotAction } = await import('./helper.ts'); const result = await runMacOsSnapshotAction(surface, { - bundleId: surface === 'menubar' ? options.appBundleId : undefined, + bundleId: surface === 'menubar' || surface === 'app' ? options.appBundleId : undefined, signal, }); return shapeDesktopSurfaceSnapshot({ ...result, producer: 'macos-helper' }, options); diff --git a/packages/platform-apple/src/runtime-snapshot.test.ts b/packages/platform-apple/src/runtime-snapshot.test.ts index 78ec50a5e0..876a3ceb39 100644 --- a/packages/platform-apple/src/runtime-snapshot.test.ts +++ b/packages/platform-apple/src/runtime-snapshot.test.ts @@ -2,7 +2,7 @@ import { expect, test, vi } from 'vitest'; import type { PlatformRuntimeHost } from '@agent-device/contracts/platform-runtime-operations'; import type { DeviceInfo } from '@agent-device/kernel/device'; import { platformRuntimeHostFixture } from './runtime.fixtures.ts'; -import { bindAppleFindTextRuntime } from './runtime-snapshot.ts'; +import { bindAppleFindTextRuntime, bindAppleSnapshotRuntime } from './runtime-snapshot.ts'; const ios = { platform: 'apple', @@ -14,6 +14,48 @@ const ios = { booted: true, } as const satisfies DeviceInfo; +const mac = { + platform: 'apple', + appleOs: 'macos', + id: 'host', + name: 'Mac', + kind: 'device', + target: 'desktop', +} as const satisfies DeviceInfo; + +function macSnapshotOwners(appBackend: 'native' | 'xctest') { + const helper = vi.fn(async () => ({ nodes: [], truncated: false }) as never); + const runner = vi.fn(async () => ({ nodes: [], truncated: false }) as never); + const fixture = platformRuntimeHostFixture(); + const operation = bindAppleSnapshotRuntime( + { + ...fixture, + snapshot: { ...fixture.snapshot, captureSurface: helper }, + localInteractors: { resolve: vi.fn(async () => ({ snapshot: runner }) as never) }, + }, + { device: mac, signal: new AbortController().signal, appBackend }, + ); + return { operation, helper, runner }; +} + +test('a native app session captures through the helper with its routed surface, never XCTest', async () => { + const { operation, helper, runner } = macSnapshotOwners('native'); + await operation.captureSnapshot({ options: { appBundleId: 'com.apple.TextEdit' } }); + expect(helper).toHaveBeenCalledWith( + mac, + { appBundleId: 'com.apple.TextEdit', surface: 'app' }, + expect.anything(), + ); + expect(runner).not.toHaveBeenCalled(); +}); + +test('an XCTest app session captures through the runner interactor, never the helper', async () => { + const { operation, helper, runner } = macSnapshotOwners('xctest'); + await operation.captureSnapshot({ options: { appBundleId: 'com.apple.TextEdit' } }); + expect(runner).toHaveBeenCalledTimes(1); + expect(helper).not.toHaveBeenCalled(); +}); + test('findText resolves the owner interactor once with request execution and cancellation', async () => { const findText = vi.fn( async (_text: string, _options?: { appBundleId?: string; signal?: AbortSignal }) => ({ @@ -25,7 +67,11 @@ test('findText resolves the owner interactor once with request execution and can const host = hostWithRunner(true, resolve); const request = new AbortController(); const poll = new AbortController(); - const operation = bindAppleFindTextRuntime(host, { device: ios, signal: request.signal }); + const operation = bindAppleFindTextRuntime(host, { + device: ios, + signal: request.signal, + appBackend: 'xctest', + }); await expect( operation.findText({ @@ -66,6 +112,7 @@ test.each([ const operation = bindAppleFindTextRuntime(host, { device, signal: new AbortController().signal, + appBackend: 'xctest', }); await expect( @@ -95,6 +142,7 @@ test('findText on a Simulator without a live runner reports not-proven', async ( const operation = bindAppleFindTextRuntime(hostWithRunner(false, resolve), { device: ios, signal: new AbortController().signal, + appBackend: 'xctest', }); await expect( operation.findText({ text: 'Settings', options: { appBundleId: 'com.example.app' } }), @@ -108,6 +156,7 @@ test('findText on a Simulator with a live runner still asks the runner', async ( const operation = bindAppleFindTextRuntime(hostWithRunner(true, resolve), { device: ios, signal: new AbortController().signal, + appBackend: 'xctest', }); await expect( @@ -123,6 +172,7 @@ test('findText on a physical iOS device resolves the runner regardless of sessio const operation = bindAppleFindTextRuntime(hostWithRunner(false, resolve), { device, signal: new AbortController().signal, + appBackend: 'xctest', }); await expect( diff --git a/packages/platform-apple/src/runtime-snapshot.ts b/packages/platform-apple/src/runtime-snapshot.ts index c3608e6b49..2f7159fd49 100644 --- a/packages/platform-apple/src/runtime-snapshot.ts +++ b/packages/platform-apple/src/runtime-snapshot.ts @@ -11,15 +11,27 @@ import type { PlatformRuntimeHost, PlatformRuntimeOperations, } from '@agent-device/contracts/platform-runtime-operations'; -import { macOsSurfaceBackend, type SessionSurface } from '@agent-device/contracts/session'; +import { + macOsHelperSurface, + macOsSurfaceBackend, + type MacOsAppBackend, + type SessionSurface, +} from '@agent-device/contracts/session'; import { isMacOs, type DeviceInfo } from '@agent-device/kernel/device'; import { hasSimulatorBridge } from './snapshot-observability.ts'; import type { AppleSnapshotRoute } from './snapshot-route.ts'; +/** A request binding, with the macOS app backend the owner resolved for the daemon. */ +type AppleSnapshotRequest = Readonly<{ + device: DeviceInfo; + signal: AbortSignal; + appBackend: MacOsAppBackend; +}>; + /** Apple-owned selection between app snapshots and explicit macOS surface snapshots. */ export function bindAppleSnapshotRuntime( host: PlatformRuntimeHost, - request: Readonly<{ device: DeviceInfo; signal: AbortSignal }>, + request: AppleSnapshotRequest, route?: AppleSnapshotRoute, ): SnapshotRuntimeOperation { const appSnapshot = bindLocalSnapshotInteractor({ @@ -28,10 +40,14 @@ export function bindAppleSnapshotRuntime( resolveInteractor: host.localInteractors.resolve, }); const captureSnapshot = async (input: CaptureSnapshotInput) => { - if (isMacOs(request.device) && macOsSurfaceBackend(input.options?.surface) === 'macos-helper') { + const helperSurface = isMacOs(request.device) + ? macOsHelperSurface(input.options?.surface, request.appBackend) + : undefined; + if (helperSurface) { + // The routed surface travels explicitly: an app session reaches the helper only here. return await host.snapshot.captureSurface( request.device, - input.options, + { ...input.options, surface: helperSurface }, captureSnapshotSignal(request.signal, input), ); } @@ -76,7 +92,7 @@ type SnapshotRuntimeOperation = Pick< */ export function bindAppleFindTextRuntime( host: PlatformRuntimeHost, - request: Readonly<{ device: DeviceInfo; signal: AbortSignal }>, + request: AppleSnapshotRequest, ): Pick { return Object.freeze({ findText: async (input: FindTextInput): Promise => { @@ -101,7 +117,7 @@ type AdmittedAppleNativeFind = Readonly<{ appBundleId: string; signal: AbortSign */ async function admitAppleNativeFind( host: Pick, - request: Readonly<{ device: DeviceInfo; signal: AbortSignal }>, + request: AppleSnapshotRequest, input: Readonly<{ options?: Readonly<{ appBundleId?: string; surface?: SessionSurface }>; execution?: Readonly<{ requestId?: string }>; @@ -110,7 +126,10 @@ async function admitAppleNativeFind( ): Promise { const appBundleId = input.options?.appBundleId; if (appBundleId === undefined) return undefined; - if (isMacOs(request.device) && macOsSurfaceBackend(input.options?.surface) === 'macos-helper') { + if ( + isMacOs(request.device) && + macOsSurfaceBackend(input.options?.surface, request.appBackend) === 'macos-helper' + ) { return undefined; } const signal = input.signal ? AbortSignal.any([request.signal, input.signal]) : request.signal; diff --git a/packages/platform-apple/src/runtime.test.ts b/packages/platform-apple/src/runtime.test.ts index 5499829530..486ae66561 100644 --- a/packages/platform-apple/src/runtime.test.ts +++ b/packages/platform-apple/src/runtime.test.ts @@ -237,6 +237,52 @@ test.each(Object.entries(leaves))( }, ); +test('a misspelled macOS app backend fails macOS bindings only', async () => { + vi.stubEnv('AGENT_DEVICE_MACOS_APP_BACKEND', 'nativ'); + try { + const runtime = createApplePlatformRuntime(platformRuntimeHostFixture()); + const bind = (device: DeviceInfo) => + runtime.bind({ + device, + intent: { kind: 'ordinary' }, + scope: { + signal: new AbortController().signal, + diagnostics: { emit: () => {} }, + progress: { report: () => {} }, + }, + }); + await expect(bind(leaves.ios)).resolves.toBeDefined(); + await expect(bind(leaves.macos)).rejects.toMatchObject({ code: 'INVALID_ARGS' }); + } finally { + vi.unstubAllEnvs(); + } +}); + +test('the native macOS app backend refuses recording and the runner before dispatch', async () => { + vi.stubEnv('AGENT_DEVICE_MACOS_APP_BACKEND', 'native'); + try { + const binding = await createApplePlatformRuntime(platformRuntimeHostFixture()).bind({ + device: leaves.macos, + intent: { kind: 'ordinary' }, + scope: { + signal: new AbortController().signal, + diagnostics: { emit: () => {} }, + progress: { report: () => {} }, + }, + }); + for (const operation of ['screenRecordingStart', 'prepareAppleRunner', 'back'] as const) { + expect(binding.facts.operations[operation]).toMatchObject({ + available: false, + reason: 'unsupported-device-backend', + }); + expect(binding.operations[operation]).toBeTypeOf('undefined'); + } + expect(binding.facts.operations.tapPoint).toEqual({ available: true }); + } finally { + vi.unstubAllEnvs(); + } +}); + test('hover has no Apple interactor route on macOS, iOS, or tvOS; the touch family reports its typed denial', async () => { for (const device of [leaves.macos, leaves.ios, leaves.tvos]) { const binding = await createApplePlatformRuntime(platformRuntimeHostFixture()).bind({ diff --git a/packages/platform-apple/src/runtime.ts b/packages/platform-apple/src/runtime.ts index da0a4f4b6c..31fb84423d 100644 --- a/packages/platform-apple/src/runtime.ts +++ b/packages/platform-apple/src/runtime.ts @@ -71,6 +71,9 @@ import { appleNavigationFacts, createAppleNavigationOperations } from './navigat import { appleSystemFacts, createAppleSystemOperations } from './system/runtime.ts'; import { appleFoldableFacts, createAppleFoldableOperations } from './foldable/runtime.ts'; import { bindAppleFindTextRuntime, bindAppleSnapshotRuntime } from './runtime-snapshot.ts'; +import type { MacOsAppBackend } from '@agent-device/contracts/session'; +import { hostMacOsAppBackend } from './os/macos/app-backend.ts'; +import { macOsNativeBackendFacts } from './os/macos/native-backend-facts.ts'; import { createAppleSnapshotRoute } from './snapshot-route.ts'; const owner = localRuntimeOwner('apple'); @@ -276,6 +279,11 @@ function appleFocusFact(device: DeviceInfo): RuntimeOperationFact { export function createApplePlatformRuntime(host: PlatformRuntimeHost): PlatformRuntimeOwner { const appLogs = createAppleAppLogRuntime(host); const snapshotRoute = createAppleSnapshotRoute(host); + // A daemon setting, read on the first macOS device and kept for the owner's lifetime so facts and + // bindings agree on it. Other Apple devices never read it, so a bad value cannot fail them. + let resolvedMacOsAppBackend: MacOsAppBackend | undefined; + const macOsAppBackend = (device: DeviceInfo): MacOsAppBackend => + isMacOs(device) ? (resolvedMacOsAppBackend ??= hostMacOsAppBackend()) : 'xctest'; const inspectFacts = async (device: DeviceInfo) => { const logs = await appLogs.inspectFacts(device); const deployment = appleAppDeploymentFacts(device); @@ -337,6 +345,7 @@ export function createApplePlatformRuntime(host: PlatformRuntimeHost): PlatformR listApps: apps, ...appleApplicationLifecycleFacts(device), shutdownTarget: shutdownFact(device), + ...macOsNativeBackendFacts(device, macOsAppBackend(device)), }, }); }; @@ -396,6 +405,7 @@ export function createApplePlatformRuntime(host: PlatformRuntimeHost): PlatformR { device: request.device, signal: request.scope.signal, + appBackend: macOsAppBackend(request.device), }, snapshotRoute, ), @@ -455,12 +465,14 @@ export function createApplePlatformRuntime(host: PlatformRuntimeHost): PlatformR bindAppleFindTextRuntime(host, { device: request.device, signal: request.scope.signal, + appBackend: macOsAppBackend(request.device), }), ), ...createAppleNavigationOperations({ host, device: request.device, signal: request.scope.signal, + admitted: facts.operations, }), ...createAppleSystemOperations({ host, diff --git a/src/daemon/screenshot-crop-target.ts b/src/daemon/screenshot-crop-target.ts index 8d71e342cf..bf98152312 100644 --- a/src/daemon/screenshot-crop-target.ts +++ b/src/daemon/screenshot-crop-target.ts @@ -2,7 +2,7 @@ import { SCREENSHOT_CROP_REASONS, type ScreenshotCropReason, } from '@agent-device/contracts/capture'; -import { macOsSurfaceBackend, type SessionSurface } from '@agent-device/contracts/session'; +import type { SessionSurface } from '@agent-device/contracts/session'; import { resolveDeviceAppleOs, type DeviceInfo } from '@agent-device/kernel/device'; import { AppError } from '@agent-device/kernel/errors'; import { validateSelectorExpression } from '@agent-device/selectors'; @@ -99,7 +99,8 @@ function classifyAppleCropTarget( } function classifyMacOsCropTarget(surface: SessionSurface | undefined): CropTarget { - return macOsSurfaceBackend(surface) === 'macos-helper' ? 'macos-helper' : 'macos-app-window'; + // An app session's screenshot frames its window on either backend; the others frame the display. + return (surface ?? 'app') === 'app' ? 'macos-app-window' : 'macos-helper'; } function cropRefusal(target: string, rejectionReason?: ScreenshotCropReason): AppError { diff --git a/src/platform-runtime-operation-host.test.ts b/src/platform-runtime-operation-host.test.ts index d8611ab95f..71b67aa164 100644 --- a/src/platform-runtime-operation-host.test.ts +++ b/src/platform-runtime-operation-host.test.ts @@ -85,18 +85,15 @@ test('composes focused deployment executors instead of a cross-family deployment expect(existsSync(join(directory, 'platform-runtime-app-deployment-host.ts'))).toBe(false); }); -test.each([undefined, 'app'] as const)( - 'the macOS surface loader refuses a %s surface the owner routes to the runner', - async (surface) => { - vi.mocked(captureMacOsSurfaceSnapshot).mockClear(); - const refusal = loadMacOsSurfaceSnapshot({ surface }); - await expect(refusal).rejects.toBeInstanceOf(TypeError); - await expect(refusal).rejects.toThrow( - 'Apple surface capture requires a helper-routed macOS surface', - ); - expect(captureMacOsSurfaceSnapshot).not.toHaveBeenCalled(); - }, -); +test('the macOS surface loader refuses a capture the owner did not route to it', async () => { + vi.mocked(captureMacOsSurfaceSnapshot).mockClear(); + const refusal = loadMacOsSurfaceSnapshot({ depth: 2 }); + await expect(refusal).rejects.toBeInstanceOf(TypeError); + await expect(refusal).rejects.toThrow( + 'Apple surface capture requires a helper-routed macOS surface', + ); + expect(captureMacOsSurfaceSnapshot).not.toHaveBeenCalled(); +}); test('the macOS surface loader forwards a helper-routed surface unchanged', async () => { vi.mocked(captureMacOsSurfaceSnapshot).mockClear(); diff --git a/src/platform-runtime-operation-host.ts b/src/platform-runtime-operation-host.ts index aeaaee1e02..3fbb279478 100644 --- a/src/platform-runtime-operation-host.ts +++ b/src/platform-runtime-operation-host.ts @@ -42,7 +42,9 @@ export async function loadMacOsSurfaceSnapshot( options: CaptureSnapshotInput['options'], signal?: AbortSignal, ): Promise { - const surface = macOsHelperSurface(options?.surface); + // The Apple owner routes and names the surface; one it did not name was never routed here. + const surface = + options?.surface === undefined ? undefined : macOsHelperSurface(options.surface, 'native'); if (!surface) { throw new TypeError('Apple surface capture requires a helper-routed macOS surface'); } diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index 61d23a4eb6..da9b5f5870 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -344,6 +344,7 @@ agent-device snapshot -i --platform apple --target desktop - Prefer selector or `@ref`-driven interactions on macOS. Window position can shift between runs, so raw x/y point commands are less stable than snapshot-derived targets. - Use `click --button secondary` for context menus on macOS, then run `snapshot -i` again. - On `frontmost-app` and `menubar` surfaces, `press` and `click` post synthetic mouse events through the macOS helper (the `desktop` surface inspects only): `--hold-ms` is how long the button stays down (at least 40 ms, 60 ms by default, because AppKit drops a release posted in the same tick as its press), `--count` is that many independent clicks (apps that detect a double-click by timing, such as Finder, still read two clicks at the default interval as one; pass an `--interval-ms` longer than the system double-click time to keep them apart), and `--double-tap` posts each click as a double-click pair. A long schedule such as `--hold-ms 10000 --count 4` is given the time it needs, and a helper stopped mid-hold — by a cancelled request, a dropped client, or its deadline — releases the button before it exits. `--jitter-px` is not applied on these surfaces. +- With `AGENT_DEVICE_MACOS_APP_BACKEND=native`, macOS app sessions run without XCTest: no Automation Mode overlay, the app can stay behind other windows, and the real pointer stays with you. `click`, `press`, and `fill` use accessibility actions on the element at their target point; `type` inserts at the focused control and falls back to key events sent to the app; `scroll` moves the accessibility scroll bar under the front window's center. Chromium-based apps expose their full tree. The daemon then never starts the XCTest runner on macOS: `record`, `prepare`, `back`, `longpress` and `--hold-ms`, `--double-tap`, `--button secondary`, and gestures are refused with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`), as is a click on an element with no accessibility action. Use the default XCTest backend for those. `screenshot` captures only the app's front window, even when other windows cover it. - Mobile-only helpers remain unsupported on macOS: `boot`, `shutdown`, `home`, `orientation`, `app-switcher`, `action-button`, `fold`, `install`, `reinstall`, `install-from-source`, and `push`. Recommended loops: diff --git a/website/docs/docs/configuration.md b/website/docs/docs/configuration.md index ad30d67d35..bb90c911b6 100644 --- a/website/docs/docs/configuration.md +++ b/website/docs/docs/configuration.md @@ -132,6 +132,7 @@ These env vars are the supported user-facing configuration surface. Other `AGENT | App hooks and logs | `AGENT_DEVICE_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_ANDROID_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_APP_LOG_MAX_BYTES`, `AGENT_DEVICE_APP_LOG_MAX_FILES`, `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`, `AGENT_DEVICE_EVENT_LOG_MAX_BYTES` | Public. Byte caps take whole integers (`5242880`), not `5MB`. | | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. | | Install/update and platform helpers | `AGENT_DEVICE_NO_UPDATE_NOTIFIER`, `AGENT_DEVICE_MACOS_HELPER_BIN`, `AGENT_DEVICE_ANDROID_SNAPSHOT_HELPER_SESSION` | Public operator controls | +| macOS app backend | `AGENT_DEVICE_MACOS_APP_BACKEND` | Public operator control, read by the daemon. `native` drives macOS app sessions through the macOS helper instead of XCTest; see [Commands](/docs/commands). Unset or `xctest` keeps the runner. Restart the daemon after changing it. | ## Command-specific defaults