feat(macos): opt-in native app backend beside XCTest

docs/adr/0031-macos-native-app-backend.md

@@ -0,0 +1,79 @@
+# ADR 0031: macOS Native App Backend — Accessibility Actions Beside XCTest
+
+## Status
+
+Accepted (2026-10-03). Opt-in; XCTest stays the default app-session backend.
+
+## Rules at a glance
+
+1. `AGENT_DEVICE_MACOS_APP_BACKEND` selects the backend for macOS `app` sessions: `xctest`
+   (default) or `native`. It is a daemon setting read through `readMacOsAppBackend`: once by the
+   Apple runtime owner, on its first macOS device, and once per macOS interactor. Other Apple
+   devices never read it. An unknown value fails macOS sessions with `INVALID_ARGS`, never a
+   silent fallback.
+2. `macOsSurfaceBackend(surface, appBackend)` in `packages/contracts/src/session-surface.ts` is the
+   one routing decision. With `native`, the `app` surface is helper-routed exactly like
+   `frontmost-app`, `desktop`, and `menubar`.
+3. A native daemon never starts XCTest on macOS. `macOsNativeBackendFacts` refuses every macOS
+   operation only the runner serves (recording, `prepare`, `back`, press and hold, gestures and
+   their viewport) at admission, for every macOS session, with `UNSUPPORTED_OPERATION` and
+   `reason: 'unsupported-device-backend'`, the reason the physical-iOS XCTest backend uses.
+   `macOsNativeAppInteractor` is assembled member by member, so no runner-backed member reaches
+   it; an action that names no app is refused rather than handed to the runner.
+4. Pointer actions are accessibility actions only: `AXPress`, focus, value, selected text, or a
+   scroll bar value. Press and hold is refused at admission; double, secondary, and middle clicks
+   are refused by the interactor before the helper runs. A press or fill with no pressable
+   element, or a scroll with no settable scroll bar, is refused by the helper and carries its
+   `helperReason` under the same `reason`; that vocabulary is pinned by
+   `contracts/fixtures/macos-native-helper-outcomes.json`. Only keyboard text falls back to events
+   posted to the app's process.
+5. Snapshots of a Chromium-based session app (one shipping `chrome_100_percent.pak`) enable its
+   accessibility tree (`AXManualAccessibility`, else `AXEnhancedUserInterface`) before traversal.
+   Only the snapshot that turns the tree on waits for it (up to 1 s); a tree that does not
+   populate adds a snapshot warning. The tree stays on for the app's lifetime, as for any
+   assistive client. The app surface walks up to 48 levels deep and reports a deeper tree as
+   truncated; other helper surfaces keep 12.
+6. Screenshots capture the session app's front window by itself through ScreenCaptureKit.
+
+## Context
+
+The XCTest runner drives a macOS app through XCUIApplication, which puts the host in Automation
+Mode: a system overlay is shown and the runner moves the shared pointer. The macOS helper already
+read the accessibility tree for helper surfaces, and the session snapshot vocabulary is the same,
+so an app session can be served without a test session while the app stays behind the user's
+windows.
+
+## Decision details
+
+**Pointer delivery.** Events posted to a process with `CGEventPostToPid` were measured against
+Calculator (SwiftUI): mouse events were dropped whether the app was frontmost or in the background,
+and wheel events were dropped in the background, while every post reported success. Keyboard events
+were accepted by Calculator and by Electron apps in the background. A fallback that cannot be
+observed to work would turn "nothing happened" into success, so pointer actions have no event
+fallback.
+
+**Target resolution.** The helper hit-tests inside the session app (other apps' windows above it do
+not answer) and walks at most four ancestors for a text input or a pressable control role.
+Chromium answers a hit test with wrapper groups that all claim `AXPress`, so when the chain names
+no control the helper picks the smallest such element whose frame contains the point, searching
+only the window the hit landed in (the app's front on-screen window when the hit names none). A
+group is pressed only when no control contains the point. Responses name the window acted in
+(`windowTitle`) when the window has an accessibility title. This resolves the same point the daemon computed from the snapshot node; the
+daemon dispatch paths and their ADR 0011 guarantees are unchanged.
+
+**Pointer dispatch, not element identity.** The daemon dispatches every platform by point, and
+the occlusion, offscreen, and parent-owned touch-point guarantees of ADR 0011 are decided on that
+point. Sending an element identity instead would be a new dispatch path with its own guarantee
+row, and an `AXUIElement` cannot outlive the one-shot helper process that resolved it, so an
+identity would be a tree path re-resolved against a tree that may have changed.
+
+## Rejected alternatives
+
+- **Suppressing Automation Mode.** `automationmodetool` removes the authentication prompt, not the
+  overlay, and the runner still owns the pointer.
+- **Private SkyLight event delivery** (`SLEventPostToPid`, focus-without-raise). It would cover
+  pointer-only controls, but it is private API that can break with any macOS release. Revisit only
+  with evidence of apps the accessibility path cannot drive.
+- **Native by default.** The accessibility path cannot express drags, holds, or double-clicks, and
+  apps with sparse accessibility trees still need the runner. Defaults change only with coverage
+  evidence across app frameworks.

docs/adr/README.md

@@ -32,6 +32,7 @@
 | [0028 Capability-Family Cell Vocabulary — One Runtime Source (Proposed)](0028-capability-family-cell-vocabulary.md) | adding a capability operation family, `UnavailablePlatformRuntimeFacts` / `UNAVAILABLE_CELLS`, `INTERACTOR_OPERATIONS`, and why an eight-package fan-out recurs |
 | [0029 Daemon Policy](0029-daemon-policy.md) | `AGENT_DEVICE_DAEMON_POLICY`, confining a daemon's commands, devices, or device shutdown, and where operator rules are enforced for batch/replay steps |
 | [0030 Process Lock Exclusion](0030-process-lock-exclusion.md) | process-lock publication/reclaim/release, retained mutation guards, and the single-protocol upgrade boundary |
+| [0031 macOS Native App Backend](0031-macos-native-app-backend.md) | `AGENT_DEVICE_MACOS_APP_BACKEND`, driving macOS app sessions without XCTest Automation Mode, and why pointer actions are accessibility actions only |
 
 ADRs record *why*; the registries and gates they describe are the living source of truth — when
 prose and a registry disagree, the registry wins and the ADR needs a follow-up.

packages/contracts/src/facades/session.ts

@@ -5,8 +5,10 @@ export {
   macOsHelperSurface,
   macOsSurfaceBackend,
   parseSessionSurface,
+  readMacOsAppBackend,
 } from '../session-surface.ts';
 export type {
+  MacOsAppBackend,
   MacOsHelperSurface,
   MacOsSurfaceBackend,
   SessionSurface,

packages/contracts/src/session-surface.test.ts

@@ -3,21 +3,59 @@ import {
   SESSION_SURFACES,
   macOsHelperSurface,
   macOsSurfaceBackend,
+  readMacOsAppBackend,
+  type MacOsAppBackend,
   type MacOsSurfaceBackend,
   type SessionSurface,
 } from './session-surface.ts';
 
-const EXPECTED_BACKENDS: Record<SessionSurface, MacOsSurfaceBackend> = {
-  app: 'xctest',
-  'frontmost-app': 'macos-helper',
-  desktop: 'macos-helper',
-  menubar: 'macos-helper',
+const EXPECTED_BACKENDS: Record<MacOsAppBackend, Record<SessionSurface, MacOsSurfaceBackend>> = {
+  xctest: {
+    app: 'xctest',
+    'frontmost-app': 'macos-helper',
+    desktop: 'macos-helper',
+    menubar: 'macos-helper',
+  },
+  native: {
+    app: 'macos-helper',
+    'frontmost-app': 'macos-helper',
+    desktop: 'macos-helper',
+    menubar: 'macos-helper',
+  },
 };
 
+test.each(
+  (['xctest', 'native'] as const).flatMap((appBackend) => [
+    ...SESSION_SURFACES.map(
+      (surface) => [appBackend, surface, EXPECTED_BACKENDS[appBackend][surface]] as const,
+    ),
+    [appBackend, undefined, EXPECTED_BACKENDS[appBackend].app] as const,
+  ]),
+)(
+  'with the %s app backend the macOS %s surface is served by %s',
+  (appBackend, surface, backend) => {
+    expect(macOsSurfaceBackend(surface, appBackend)).toBe(backend);
+    expect(macOsHelperSurface(surface, appBackend)).toBe(
+      backend === 'macos-helper' ? (surface ?? 'app') : undefined,
+    );
+  },
+);
+
 test.each([
-  ...SESSION_SURFACES.map((surface) => [surface, EXPECTED_BACKENDS[surface]] as const),
-  [undefined, 'xctest'] as const,
-])('the macOS %s surface is served by %s', (surface, backend) => {
-  expect(macOsSurfaceBackend(surface)).toBe(backend);
-  expect(macOsHelperSurface(surface)).toBe(backend === 'macos-helper' ? surface : undefined);
+  [undefined, 'xctest'],
+  ['', 'xctest'],
+  ['native', 'native'],
+  [' Native ', 'native'],
+  ['xctest', 'xctest'],
+] as const)('AGENT_DEVICE_MACOS_APP_BACKEND=%j selects %s', (raw, expected) => {
+  expect(
+    readMacOsAppBackend((name) => {
+      expect(name).toBe('AGENT_DEVICE_MACOS_APP_BACKEND');
+      return raw;
+    }),
+  ).toBe(expected);
+});
+
+test('an unknown app backend is refused instead of falling back', () => {
+  expect(() => readMacOsAppBackend(() => 'vision')).toThrow(/AGENT_DEVICE_MACOS_APP_BACKEND/);
 });

packages/contracts/src/session-surface.ts

@@ -15,30 +15,54 @@ export function parseSessionSurface(value: string | undefined): SessionSurface {
 /** The backend that serves every operation on a macOS surface. */
 export type MacOsSurfaceBackend = Extract<SnapshotBackend, 'xctest' | 'macos-helper'>;
 
-const MACOS_SURFACE_BACKENDS = {
-  app: 'xctest',
+/**
+ * Which backend drives a macOS app session. `xctest` is the runner under XCTest Automation Mode;
+ * `native` drives the app through the macOS helper's accessibility actions and process-targeted
+ * events, so the app can stay in the background and the user keeps the pointer.
+ */
+const MACOS_APP_BACKENDS = ['xctest', 'native'] as const;
+export type MacOsAppBackend = (typeof MACOS_APP_BACKENDS)[number];
+const MACOS_APP_BACKEND_ENV = 'AGENT_DEVICE_MACOS_APP_BACKEND';
+const MACOS_APP_BACKEND_ENUM = defineStringEnum(MACOS_APP_BACKENDS, {
+  normalize: (raw) => raw.trim().toLowerCase(),
+  message: (value) =>
+    `Invalid ${MACOS_APP_BACKEND_ENV}: ${value}. Use ${MACOS_APP_BACKENDS.join('|')}.`,
+});
+
+/** The host's app-session backend; unset selects `xctest`. */
+export function readMacOsAppBackend(
+  readEnvironment: (name: string) => string | undefined,
+): MacOsAppBackend {
+  const raw = readEnvironment(MACOS_APP_BACKEND_ENV);
+  return raw === undefined || raw.trim() === '' ? 'xctest' : MACOS_APP_BACKEND_ENUM.parse(raw);
+}
+
+const MACOS_HELPER_SURFACE_BACKENDS = {
   'frontmost-app': 'macos-helper',
   desktop: 'macos-helper',
   menubar: 'macos-helper',
-} as const satisfies Record<SessionSurface, MacOsSurfaceBackend>;
+} as const satisfies Record<Exclude<SessionSurface, 'app'>, MacOsSurfaceBackend>;
 
 /** An absent surface is an app session, the reading every route already gives it. */
-export function macOsSurfaceBackend(surface: SessionSurface | undefined): MacOsSurfaceBackend {
-  return MACOS_SURFACE_BACKENDS[surface ?? 'app'];
+export function macOsSurfaceBackend(
+  surface: SessionSurface | undefined,
+  appBackend: MacOsAppBackend,
+): MacOsSurfaceBackend {
+  const resolved = surface ?? 'app';
+  if (resolved === 'app') return appBackend === 'native' ? 'macos-helper' : 'xctest';
+  return MACOS_HELPER_SURFACE_BACKENDS[resolved];
 }
 
-type HelperRoutedSurface = {
-  [S in SessionSurface]: (typeof MACOS_SURFACE_BACKENDS)[S] extends 'macos-helper' ? S : never;
-}[SessionSurface];
-
 declare const helperSurface: unique symbol;
 /** A surface the owner routed to the macOS helper; only `macOsHelperSurface` produces one. */
-export type MacOsHelperSurface = HelperRoutedSurface & { readonly [helperSurface]: true };
+export type MacOsHelperSurface = SessionSurface & { readonly [helperSurface]: true };
 
 export function macOsHelperSurface(
   surface: SessionSurface | undefined,
+  appBackend: MacOsAppBackend,
 ): MacOsHelperSurface | undefined {
-  return surface !== undefined && macOsSurfaceBackend(surface) === 'macos-helper'
-    ? (surface as MacOsHelperSurface)
+  const resolved = surface ?? 'app';
+  return macOsSurfaceBackend(resolved, appBackend) === 'macos-helper'
+    ? (resolved as MacOsHelperSurface)
     : undefined;
 }

packages/platform-apple/src/interactions.ts

@@ -206,7 +206,8 @@ async function runApplePressPoint(
   point: { x: number; y: number },
   options: PressPointOptions,
 ): Promise<Record<string, unknown>> {
-  const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined;
+  // The runner owner's routing: an app session reaches here only on the XCTest backend.
+  const helper = isMacOs(device) ? macOsHelperSurface(options.surface, 'xctest') : undefined;
   if (helper) {
     return await runMacOsSurfacePress(context, point, options, helper);
   }

packages/platform-apple/src/interactor.ts

@@ -14,7 +14,11 @@ import {
 } from './runner/index.ts';
 import { toAppleTvRemoteButton } from '@agent-device/contracts/tv-remote';
 import { SCREENSHOT_FULLSCREEN_REASONS } from '@agent-device/contracts/capture';
-import { macOsHelperSurface, type MacOsHelperSurface } from '@agent-device/contracts/session';
+import {
+  macOsHelperSurface,
+  type MacOsHelperSurface,
+  type SessionSurface,
+} from '@agent-device/contracts/session';
 import { DEVICE_ROTATIONS, type DeviceRotation } from '@agent-device/contracts/device';
 import { normalizeSnapshotScope } from '@agent-device/contracts/snapshot';
 import { withDiagnosticTimer } from '@agent-device/host-kit/diagnostics';
@@ -31,6 +35,8 @@ import type {
   SnapshotOptions,
 } from '@agent-device/contracts/interactor-types';
 import { captureMacOsSurfaceSnapshot } from './os/macos/surface-snapshot.ts';
+import { hostMacOsAppBackend } from './os/macos/app-backend.ts';
+import { macOsNativeAppInteractor } from './os/macos/native-app-interactor.ts';
 import {
   presentAppleRunnerSnapshot,
   readAppleSnapshotResult,
@@ -55,6 +61,9 @@ export function createAppleInteractor(
     );
   }
   const { overrides, runnerOpts } = iosRunnerOverrides(device, runnerContext);
+  const appBackend = isMacOs(device) ? hostMacOsAppBackend() : 'xctest';
+  const helperSurface = (surface: SessionSurface | undefined) =>
+    isMacOs(device) ? macOsHelperSurface(surface, appBackend) : undefined;
   const interactor: Interactor = {
     open: (app, options) =>
       openIosApp(device, app, {
@@ -67,14 +76,16 @@ export function createAppleInteractor(
       }),
     openDevice: () => openIosDevice(device),
     close: (app) => closeIosApp(device, app, runnerOpts),
-    screenshot: (outPath, options) => runAppleScreenshot(device, outPath, options, runnerOpts),
-    snapshot: async (options) => await captureAppleSnapshot(device, options, runnerOpts),
+    screenshot: (outPath, options) =>
+      runAppleScreenshot(device, outPath, options, runnerOpts, helperSurface(options?.surface)),
+    snapshot: async (options) =>
+      await captureAppleSnapshot(device, options, runnerOpts, helperSurface(options?.surface)),
     // The live text at a point: helper for a helper-routed macOS surface, XCTest runner for
-    // every other Apple leaf including a macOS app session.
+    // every other Apple leaf.
     readTextAtPoint: async (point, options) => {
-      const helper = isMacOs(device) ? macOsHelperSurface(options?.surface) : undefined;
+      const helper = helperSurface(options?.surface);
       return helper
-        ? await readMacOsSurfaceTextAtPoint(point, helper, options?.appBundleId)
+        ? await readMacOsSurfaceTextAtPoint(point, helper, options?.appBundleId, options?.signal)
         : await readRunnerTextAtPoint(device, point, options, runnerOpts);
     },
     // The XCTest runner's own text reading: it observes the live accessibility hierarchy
@@ -203,16 +214,18 @@ export function createAppleInteractor(
     dismissAlert: (options) => actOnAppleAlert(device, runnerOpts, 'dismiss', options),
     ...overrides,
   };
-  if (!runnerProvider) return interactor;
-  return withInjectedAppleRunnerTransport(device, runnerContext, interactor, runnerProvider);
+  const served =
+    appBackend === 'native' ? macOsNativeAppInteractor(interactor, runnerContext) : interactor;
+  if (!runnerProvider) return served;
+  return withInjectedAppleRunnerTransport(device, runnerContext, served, runnerProvider);
 }
 
 async function captureAppleSnapshot(
   device: DeviceInfo,
   options: SnapshotOptions | undefined,
   runnerOpts: RunnerCallOptions,
+  helper: MacOsHelperSurface | undefined,
 ) {
-  const helper = isMacOs(device) ? macOsHelperSurface(options?.surface) : undefined;
   if (helper) {
     return await captureMacOsSurfaceSnapshot({ ...options, surface: helper }, options?.signal);
   }
@@ -388,20 +401,24 @@ async function runAppleScreenshot(
   outPath: string,
   options: ScreenshotOptions = {},
   runnerOpts: RunnerCallOptions,
+  helper: MacOsHelperSurface | undefined,
 ): Promise<ScreenshotCaptureFacts> {
-  const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined;
   if (helper) {
     if (options.fullscreen) {
       throw new AppError(
         'INVALID_ARGS',
-        `screenshot --fullscreen is not accepted on the macOS ${helper} surface: it always captures the main display`,
+        `screenshot --fullscreen is not accepted on the macOS ${helper} surface: its capture frame is fixed`,
         {
           reason: SCREENSHOT_FULLSCREEN_REASONS.macOsHelperSurfaceFixedFrame,
           surface: helper,
         },
       );
     }
-    await runMacOsScreenshotAction(outPath, { surface: helper });
+    await runMacOsScreenshotAction(outPath, {
+      surface: helper,
+      ...(helper === 'app' ? { bundleId: options.appBundleId } : {}),
+      signal: runnerOpts.signal,
+    });
     return {};
   }
   if (options.captureBackend === 'runner') {
@@ -430,11 +447,13 @@ async function readMacOsSurfaceTextAtPoint(
   point: Point,
   surface: MacOsHelperSurface,
   appBundleId: string | undefined,
+  signal: AbortSignal | undefined,
 ): Promise<string | undefined> {
   const { runMacOsReadTextAction } = await import('./os/macos/helper.ts');
   const result = await runMacOsReadTextAction(point.x, point.y, {
     bundleId: appBundleId,
     surface,
+    signal,
   });
   return result.text;
 }

packages/platform-apple/src/navigation/runtime.ts

@@ -183,17 +183,27 @@ export function appleNavigationFacts(device: DeviceInfo) {
   });
 }
 
-/** Binds whichever navigation operations {@link appleNavigationFacts} admitted. */
+/**
+ * Binds whichever navigation operations the device's admitted facts allow. The cells are those
+ * {@link appleNavigationFacts} declares, read from the admitted facts so a refusal layered over
+ * the leaf (a backend that cannot serve one) also withholds the binding.
+ */
 export function createAppleNavigationOperations(params: {
   host: Pick<PlatformRuntimeHost, 'localInteractors'>;
   device: DeviceInfo;
   signal: AbortSignal;
+  admitted: Readonly<Record<string, RuntimeOperationFact>>;
 }) {
-  const { host, device, signal } = params;
+  const { host, device, signal, admitted } = params;
+  const navigationKeys = Object.keys(appleNavigationFacts(device)) as Array<
+    keyof ReturnType<typeof appleNavigationFacts>
+  >;
   return bindAdmittedLocalInteractorOperations({
     device,
     signal,
     resolveInteractor: host.localInteractors.resolve,
-    facts: appleNavigationFacts(device),
+    facts: Object.fromEntries(navigationKeys.map((key) => [key, admitted[key]])) as ReturnType<
+      typeof appleNavigationFacts
+    >,
   });
 }