Skip to content

fix(ios): support remote-hosted alerts on physical devices#1232

Open
knchst wants to merge 5 commits into
callstack:mainfrom
knchst:fix/remote-hosted-system-modal-probe
Open

fix(ios): support remote-hosted alerts on physical devices#1232
knchst wants to merge 5 commits into
callstack:mainfrom
knchst:fix/remote-hosted-system-modal-probe

Conversation

@knchst

@knchst knchst commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Fixes #1231.

Summary

Make physical-iOS system alerts reliable, including AccessorySetupKit pickers hosted by com.apple.AccessorySetupUI.

  • When SpringBoard exposes a modal but no hittable actions, query an allowlisted foreground system-UI host without activating it.
  • Share the modal resolution path between snapshots and alert commands.
  • Bound post-action alert checks to the source that was acted on, avoiding stale XCUIElement access after dismissal.
  • Enable alert on physical iOS while keeping physical iPadOS, visionOS, tvOS, clipboard, and settings gated pending separate verification.

Validation

@knchst knchst marked this pull request as ready for review July 13, 2026 03:45
@thymikee

Copy link
Copy Markdown
Member

Review at 320ffb42 found four blockers before readiness:

  1. A successful remote-host alert tap can fail during dismissal verification. The fallback stores ownerApp: remote.host; after the tap dismisses AccessorySetupUI, alertStillVisible re-enters resolveAlert(app: remote.host). With the SpringBoard modal gone, resolveAlert directly evaluates activeApp.alerts.allElementsBoundByIndex, although AccessorySetupKit picker on physical iOS: system-modal snapshot detection is flaky, alert is capability-gated — querying com.apple.AccessorySetupUI directly works reliably #1231 documents that post-dismiss queries against this host can throw kAXErrorServerNotFound. Wrap that fallback query in the existing safe query path and add a regression for a host disappearing after activation.

  2. alert accept cannot select the ASK positive action. The picker exposes Close / Set Up / Learn More, but chooseAlertButton does not recognize Set Up; it returns alert accept button not found. Either support this affirmative label with a focused test or narrow the PR's alert-family claim.

  3. The remote tree is accepted without proving the host is foreground. remoteHostedSystemModal rejects only .notRunning, so .runningBackground* and .unknown hosts may substitute their action tree for an unrelated unusable SpringBoard modal. The issue's evidence observes .runningForeground; fail closed to that state (or add an equivalent binding check) before replacing the modal tree.

  4. The changed production path lacks regression and end-to-end evidence. Green CI compiles the Swift, and the physical evidence proves that direct host queries/taps work, but it does not exercise patched snapshot -i --json, wait 'label="Set Up"', or selector click after the SpringBoard mirror becomes non-actionable. Please add executable coverage for the routing/fail-closed behavior and provide a physical-device run through at least the snapshot/wait/click production path.

Also, Fixes #1231 would close an issue whose physical-iOS alert capability decision remains unresolved. Use a non-closing reference or split/retain a follow-up for that explicitly gated part.

No fixer was dispatched. An external Claude pass was not run because export was not separately authorized.

knchst added a commit to knchst/agent-device that referenced this pull request Jul 13, 2026
…est probe routing

Addresses review on callstack#1232:
- Gate the remote-host probe to a foreground host
  (RemoteHostedSystemModalPolicy.isEligibleHostState); background/unknown hosts
  fail closed instead of substituting an unrelated action tree.
- Wrap the alert-resolution fallback query in safeElementsQuery so a dismissed
  remote host raising kAXErrorServerNotFound is absorbed.
- Extract routing/gating into RemoteHostedSystemModalPolicy and add
  simulator-free unit tests under AGENT_DEVICE_RUNNER_UNIT_TESTS.
@knchst

knchst commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 3f36308 addressing the four blockers. Rundown:

1. Dismissal re-query throwing kAXErrorServerNotFound. The resolveAlert fallback activeApp.alerts.allElementsBoundByIndex is now wrapped in safeElementsQuery, so when alertStillVisible re-enters with the just-dismissed host as activeApp, the raising query is absorbed and reports no alert. The probe path (actionableElements(in:)) was already inside safely() absorbers. I did not add a standalone "host disappears" XCUITest for this — it needs a live remote host to dismiss, so it belongs in the physical-device run below rather than the simulator-free unit surface; happy to add one if you'd rather gate on it.

2. alert accept can't select Set Up. Narrowed the claim instead of widening the accept vocabulary — added a "Scope of the alert command" section: alert dismiss drives Close (already in the dismiss vocabulary), and alert accept selecting an affirmative like Set Up is explicitly out of scope, since the picker is fully operable via snapshot + selector click and adding Set Up to the generic accept list would risk mis-accepting unrelated alerts. Left as a follow-up if you want a context-scoped affirmative match.

3. Foreground not proven before substituting the tree. Now fails closed to .runningForeground via RemoteHostedSystemModalPolicy.isEligibleHostState; .runningBackground / .notRunning / .unknown are all rejected, and the state read falls back to .unknown (ineligible) if it raises.

4. Executable coverage for routing/fail-closed. Extracted the routing + gating into a pure RemoteHostedSystemModalPolicy (mirrors SynthesizedFallbackPolicy) and added unit tests under AGENT_DEVICE_RUNNER_UNIT_TESTS (shouldProbeRemoteHost runs only at zero springboard actions; isEligibleHostState foreground-only). They compile under the Swift Runner Unit Compile job. The physical-device pass through the patched snapshot -i --json / wait 'label="Set Up"' / selector click path is still pending — AccessorySetupUI only exists on real hardware — and I'll attach that run.

Also switched Fixes #1231Refs #1231 so the unresolved physical-iOS alert capability-gate decision stays open.

@knchst

knchst commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

On-device evidence

To back the fixes with something concrete rather than a "will attach later": below are the on-device observations behind each branch. Captured on a physical iPhone 14 / iOS 26.5, picker on screen (one discovered accessory, Close / Set Up / Learn More). These come from the stock 0.17.6 runner plus a throwaway probe test — I'm being upfront that this is not a run of the patched binary (see the note at the end).

Host process is foreground — backs the .runningForeground gate (blocker 3):

ASKPROBE askui.state=4          // XCUIApplication.State.runningForeground
ASKPROBE sb.alerts=1 sb.sheets=0 sb.windows=11

SpringBoard mirror is present but its actions are unusable — backs shouldProbeRemoteHost triggering at zero actions (blocker 4):

ASKPROBE sb.alert label='' buttons=3 descendants=31

The mirror reports 3 buttons, but actionableElements(in: springboard) comes back empty because they don't pass isHittable from the SpringBoard scope — precisely the "detected but unusable" condition that makes blockingSystemAlertSnapshot() return nil today and that now triggers the probe.

Direct host-scope query sees and can act on the real elements — backs the probe path:

ASKPROBE askui.buttons.count=3
ASKPROBE askui.button label='Close'      frame=(326.0, 396.7, 30.0, 30.0)
ASKPROBE askui.button label='Set Up'     frame=(40.0, 694.0, 310.0, 50.0)
ASKPROBE askui.button label='Learn More' frame=(40.0, 754.0, 310.0, 50.0)
ASKPROBE askui.staticTexts: '', 'Set Up', 'Learn More', 'Finding Accessories', ...
ASKPROBE askui.descendants.count=38

Query completed in ~0.92s. askui.buttons["Close"].tap() then dismissed the picker (visually confirmed).

Dismissal raises kAXErrorServerNotFound — backs the safeElementsQuery guard (blocker 1):

ASKPROBE dismiss: tapping Close via ASKUI element query
... Failed to get matching snapshots: Error getting main window kAXErrorServerNotFound

Re-querying the host immediately after it dismisses throws — exactly what the wrapped resolveAlert fallback now absorbs.

SpringBoard path can itself yield the modal snapshot when the mirror is momentarily hittable: an interactive snapshot at one point returned the picker as a 5-node Alert tree (System AlertClose / Set Up / Learn More) — the same node shape the remote-host fallback now emits.


To be precise about what this is and isn't: these are observations from the stock runner + a probe test, mapping 1:1 to the patch's four branches (foreground gate, zero-action probe trigger, direct-scope readability, post-dismiss kAXErrorServerNotFound) — not an end-to-end run of the patched binary through snapshot -i --json / wait / click. Reproducing the picker needs the accessory put back into pairing mode, so I'm not going to promise that run on a timeline. If you want the patched binary exercised end-to-end before merge, I can set it up on the same hardware — just say so and I'll post it.

@thymikee

Copy link
Copy Markdown
Member

Re-review at 3f36308 found no remaining production-code defects: the dismissal query now fails safely, host substitution is foreground-only, the alert claim is narrowed, and Refs #1231 preserves the separate capability decision.

Readiness is still withheld because the attached device evidence explicitly uses the stock 0.17.6 runner plus a probe, not this patched binary. Please exercise the actual PR head on a physical iPhone with the ASK picker visible and the SpringBoard mirror in its zero-action state, then provide:

  • patched snapshot -i --json showing the ASK controls from the fallback;
  • patched wait 'label="Set Up"' succeeding;
  • patched selector click (or Close dismissal) acting on the intended ASK control;
  • confirmation that the session was closed cleanly.

The added policy tests compile, but CI's Swift Runner Unit Compile job does not execute the production host-substitution route. The physical run above is therefore the remaining merge-readiness evidence. No fixer was dispatched.

@thymikee thymikee changed the title fix(ios): probe remote-hosted system modals (AccessorySetupKit picker) when the springboard mirror yields no hittable actions fix(ios): support remote-hosted alerts on physical devices Jul 13, 2026
knchst and others added 5 commits July 13, 2026 19:38
…) when the springboard mirror yields no hittable actions
…est probe routing

Addresses review on callstack#1232:
- Gate the remote-host probe to a foreground host
  (RemoteHostedSystemModalPolicy.isEligibleHostState); background/unknown hosts
  fail closed instead of substituting an unrelated action tree.
- Wrap the alert-resolution fallback query in safeElementsQuery so a dismissed
  remote host raising kAXErrorServerNotFound is absorbed.
- Extract routing/gating into RemoteHostedSystemModalPolicy and add
  simulator-free unit tests under AGENT_DEVICE_RUNNER_UNIT_TESTS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants