AccessorySetupKit picker on physical iOS: system-modal snapshot detection is flaky, `alert` is capability-gated — querying `com.apple.AccessorySetupUI` directly works reliably

[knchst]

Summary

On a physical iPhone, the AccessorySetupKit (ASK) accessory picker — a system sheet hosted out-of-process by /Applications/AccessorySetupUI.app — is only intermittently visible to snapshot (and therefore to wait/selector resolution), and alert is refused outright by the CLI capability gate. In the same experiment, querying the hosting process com.apple.AccessorySetupUI directly from the runner (without activating it) saw every element reliably and could tap them.

This currently forces hard-coded coordinates + fixed sleeps in E2E scripts that need to get through BLE accessory pairing on real devices.

Environment

• agent-device 0.17.6 (invoked via npx agent-device@0.17.6; code references below checked against current main — the relevant logic is unchanged, ios-runner/ is now apple-runner/)
• Physical iPhone 14, iOS 26.5 (kind: device, platform: ios)
• macOS host (Darwin 25.4.0), Xcode toolchain, Node 22
• App under test presents the ASK picker via ASAccessorySession.showPicker; picker shows one discovered accessory ("Mori Device") with Close / Set Up / Learn More buttons

What we observed (single session, picker on screen the whole time)

The picker was opened and confirmed visible via screenshot (sheet shows "Finding Accessories", the accessory card, and a blue Set Up button).

1. snapshot DID capture the picker at first — as the synthesized system-modal Alert:

$ agent-device snapshot -i --json ...   # picker on screen

{ "success": true, "data": { "nodes": [
  { "type": "Alert", "label": "System Alert", "index": 0, "rect": { "x": 0, "y": 0, "width": 390, "height": 844 }, "hittable": true, ... },
  { "type": "StaticText", "label": "Set Up", "parentIndex": 0, ... },
  { "type": "Button", "label": "Close", "parentIndex": 0, ... },
  { "type": "Button", "label": "Set Up", "parentIndex": 0, ... },
  { "type": "Button", "label": "Learn More", "parentIndex": 0, ... }
] } }

Both snapshot -i --json and snapshot --raw --json returned this identical 2,422-byte payload — the signature of the blockingSystemAlertSnapshot() early return in snapshotFast/snapshotRaw (apple-runner/.../RunnerTests+Snapshot.swift). So the springboard-modal machinery does work on a physical device — sometimes.

2. ~1 minute later, same picker still on screen, the same capture path stopped seeing it:

$ agent-device wait 'label="Set Up"' 3000 ...
Error (COMMAND_FAILED): wait timed out for selector: label="Set Up".
Current surface: Connect your device, Plug the USB cable into the back of the device. ...

wait polls the same snapshot capture used above (waitForSelector → captureSelectorSnapshot), and the timeout decoration (inspectCurrentSurface) takes one more interactive snapshot. All of those polls returned only the background app's tree — no Alert node, even though the picker was still on screen (verified by screenshot). So within one minute, the same picker flipped from "detected as blocking modal" to "invisible", with no UI change we could see.

3. alert is gated off entirely on physical iOS:

$ agent-device alert get ...
Error (UNSUPPORTED_OPERATION): alert is not supported on this device

This comes from physicalDeviceSurfaces: false in IOS_FAMILY_CAPABILITIES (src/platforms/apple/capabilities.ts), which gates alert/clipboard/settings to macOS-host-or-simulator via supportsHostOrSimulatorSurface (src/platforms/apple/plugin.ts). The runner-side implementation (resolveAlert → firstBlockingSystemModal(in: springboard), RunnerTests+Alert.swift) is the exact machinery that produced the successful Alert snapshot in observation 1, so on the evidence above the blanket gate looks overly conservative for alert on physical iOS.

4. Querying the hosting process directly works reliably (probe XCUITest added to the runner target):

While the picker is visible, its host process is AccessorySetupUI (com.apple.AccessorySetupUI, state = runningForeground). The picker is NOT a sheet of the app under test (app.sheets.count == 0); it mirrors into springboard as an anonymous alert (springboard.alerts.count == 1, label='', buttons=3, descendants=31).

A plain query — without activate() — against the host sees everything, in ~0.9 s:

askui = XCUIApplication(bundleIdentifier: "com.apple.AccessorySetupUI")
askui.buttons                    → 'Close' (326.0, 396.7, 30, 30), 'Set Up' (40, 694, 310, 50), 'Learn More' (40, 754, 310, 50)
askui.staticTexts                → '', 'Set Up', 'Learn More', 'Finding Accessories', 'Mori Device', 'Mori'
askui.descendants(.any).count    → 38

And synthesized events work: askui.buttons["Close"].tap() dismissed the picker (visually confirmed).

Caveats discovered along the way:

• Never activate()/open com.apple.AccessorySetupUI — activation tears the sheet into a black full-screen state and the picker has to be recovered manually. Query only.
• Immediately after the picker dismisses, further queries against the host throw kAXErrorServerNotFound (the AX server is gone); the runner's existing safely() exception absorber handles this shape of failure.
• Tapping springboard-scoped content logs XCTest's Failed to construct element query matching interruption. Interrupting element Alert, foreground application 'com.apple.springboard', element application: (null) warning, but event synthesis still succeeds.

Analysis

blockingSystemAlertSnapshot() (apple-runner/.../RunnerTests+SystemModal.swift) returns nil — hiding the modal from the snapshot — when actionableElements(in: modal) is empty; safeIsActionableCandidate requires candidate.isHittable. The springboard mirror of the remote-hosted ASK picker exposes its buttons (buttons=3), but their hittability from the springboard scope appears to be unstable: the same picker passed the gate at 11:33 and failed it from 11:34 onward. When the gate fails, snapshot, wait, selector-based click, and (if it were enabled) alert all lose the modal at once, while the screenshot still shows it — the worst case for an agent, which then acts on the background app's tree.

Direct queries in the com.apple.AccessorySetupUI scope were stable in the same window, which suggests probing the hosting process is the reliable path for this class of remote-hosted system UI.

Suggested fixes

(a) Remote-host probe fallback in the runner. In blockingSystemAlertSnapshot() (and resolveAlert), when a springboard alert/sheet exists but yields no hittable actionable elements, probe a small allowlist of known remote system-UI hosts — starting with com.apple.AccessorySetupUI — via plain queries (no activation), and synthesize the modal node from that scope. The probe only runs when the springboard mirror has already been detected but is unusable, so the common path pays nothing. I have a working patch shaped this way and can open a PR.

(b) Revisit the physical-iOS alert gate. physicalDeviceSurfaces: false currently blankets alert/clipboard/settings. The springboard-modal machinery behind alert demonstrably works on physical devices (observation 1); with (a) it would also cover the ASK picker. Splitting alert out of the blanket gate (or gating per-surface) would let physical-device E2E handle system dialogs without coordinate hacks. clipboard/settings may well still need the simulator — no claim about those.

Happy to contribute the PR for (a) (query-only, safely()-absorbed, no activation) and discuss options for (b).