diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 6d7180d37..d43bb3841 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -12,6 +12,13 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-13). See [Mig `REPLAY_DIVERGENCE` repair-loop fix (#1223). - Decision 6, the base agent-supervised re-record repair — `replay --save-script` arming, the post-watermark healed slice, `repairHint`, and the writer's bare-`@ref` fail-loud guard (#1228). +- Decision 4 amendment, `screen`'s capture scope and ref selection — the divergence `screen` capture runs + through the same `captureSnapshot` wrapper as plain `snapshot` (full-window scope + Android freshness / + post-action retry parity) under a clean, fixed capture-flags policy (a failed raw/scoped/`-d` action can + no longer narrow the diagnostic tree), with the chrome and meaningful-target filters layered on top as + filters, not scopings; and `screen.refs` is ranked within the byte cap (foreign-window dismiss targets + ahead of app content; mass-covered app nodes surfaced rather than emptied) instead of sliced in document + order, so a captured overlay is never buried past the cap (#1264). **Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle): the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED → @@ -438,6 +445,42 @@ healthy snapshot digest and the only form that issues actionable refs. `{ state: hint }` is returned when capture fails or is sparse; it has no refs or generation and must not fall back to the old session tree. Screen-capture failure never replaces or masks the original replay cause. +> **Amendment (#1264): `screen`'s capture scope and ref selection.** `refs` is a filtered, ranked digest +> of the exact same tree a plain `snapshot` would return at that moment. Two guarantees back the invariant: +> +> 1. **Capture scope.** The capture underneath `screen` runs through the **same `captureSnapshot` wrapper** +> the plain `snapshot` command's backend calls — not a parallel single-shot dispatch — so it inherits +> `snapshot`'s full-window scope (Android: the snapshot-helper route, with the existing graceful +> app-scoped fallback only when the helper is unavailable; iOS: the bounded system-modal probe path; +> macOS/Linux: their surface-scoped branches) AND its Android freshness / post-action retry policy. A +> divergence must not consume a first stale or app-scoped dump while a plain `snapshot` retries to the +> fresh full-window tree — that would make the divergence staler or narrower than `snapshot`, violating +> the invariant. The divergence capture's flags are a **clean, fixed policy** (full-window, non-raw, +> default depth), NOT the failed action's flags: a failed `snapshot --raw`/scoped/`-d` action must never +> narrow or reshape the diagnostic tree below what a plain `snapshot` shows, so `snapshotRaw` / +> `snapshotScope` / `snapshotDepth` are dropped; only the interactive-only policy (full for non-rect +> `get`/`is`/`wait` reads, interactive otherwise) is carried. The chrome filter (#1233/#1256, +> `collectSettleChromeRefs`) and the meaningful-target filter (label/id or `hittable`) are layered ON TOP +> of that full capture as **filters**, not as a separate, narrower scoping — a filter may drop a node the +> full capture contains, but the capture itself must never omit content `snapshot` would show. +> 2. **Ref selection within the cap.** The `screen.refs` cap is a **byte bound**, not a "first N in document +> order" policy. A separate-window overlay enumerates AFTER the app window's nodes, so a document-order +> slice truncates a fully-captured overlay away (its dismiss target sits past the cap) — reporting a +> healthy-looking app even though the capture holds the covering window. So `refs` is RANKED before it is +> capped: foreign-window (non-app-`bundleId`) hittable nodes — the dismiss targets for whatever covers +> the app — are promoted ahead of app content, with document order otherwise preserved (stable within +> each tier). And when a system overlay MASS-COVERS the app so every app node is annotated `covered`, +> those covered nodes are surfaced rather than emitting an empty `refs`: a report whose capture holds +> meaningful nodes but whose `refs` is empty is broken by construction. +> +> This is a hard invariant: **an agent must never see a healthier `screen` in a divergence report than a +> plain `snapshot` would show it.** Concretely, a separate-window system overlay covering the app at the +> moment of capture — a held volume dialog, a persistent quick-settings shade, a permission dialog — must +> appear in `screen.refs` (its actionable/hittable/labeled nodes surviving the filters AND the cap) exactly +> as `snapshot` would present it. `repairHint` (decision 6) and `suggestions` (decision 1) consume the FULL +> captured node list, not the capped `refs` slice, so hint routing is computed over the same full, +> correctly-scoped capture and is never routed as if the app underneath a covering overlay were healthy. + Response levels bound the entire serialized UTF-8 `details.divergence` object, not merely its arrays: compact (`--level digest`) is at most **8 KiB**, default at most **24 KiB**, and full at most **64 KiB**. Compact carries at most **8** screen refs and no suggestion entries — it carries `suggestionCount` (the diff --git a/src/core/__tests__/snapshot-chrome-android-statusbar.test.ts b/src/core/__tests__/snapshot-chrome-android-statusbar.test.ts index 9b07479a7..9a5a568ba 100644 --- a/src/core/__tests__/snapshot-chrome-android-statusbar.test.ts +++ b/src/core/__tests__/snapshot-chrome-android-statusbar.test.ts @@ -104,14 +104,16 @@ test('Android non-raw capture: status-bar leaves are recognized as chrome once t } }); -test('Android actionable systemui overlay (volume dialog) still survives with the status-bar leak fix (#1251)', () => { - // Extend the RAW capture with a disjoint systemui run whose leaf ids look - // like an actionable overlay (volume dialog): no real capture of that - // surface was available, but the point of this test is exactly that the - // new leaf-id set must NOT broaden to "any systemui id". Appended to the - // RAW array (not the already-walked output) so the whole tree — status bar - // AND volume dialog — goes through the real walk together, exactly as a - // single device capture would. +test('Android actionable systemui overlay (volume dialog) still survives the chrome filter (#1251)', () => { + // Filter-logic unit test, NOT a live-capture-path claim (#1264 finding 2): this + // exercises `collectSettleChromeRefs` in isolation over a synthetic systemui + // run appended to an unrelated capture fixture. The leaf ids ARE real, + // live-verified volume-dialog ids (`volume_dialog_container`, + // `volume_new_ringer_active_icon_container`), but appending them to this + // fixture does not reproduce how a live capture reaches the filter — that is + // covered separately by the full-capture invariant test below. The point + // here is narrower: the status-bar/nav-bar leaf-id set (#1251) must NOT + // broaden to "any systemui id" and drop an actionable overlay it is handed. const rawWithVolumeDialog: RawSnapshotNode[] = [ ...ANDROID_IME_CAPTURE_RAW_NODES, { @@ -125,8 +127,8 @@ test('Android actionable systemui overlay (volume dialog) still survives with th parentIndex: 9000, type: 'android.widget.ImageButton', bundleId: 'com.android.systemui', - identifier: 'com.android.systemui:id/volume_dialog_slider', - label: 'Media volume', + identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container', + label: 'Ringer volume', hittable: true, }, ]; @@ -134,13 +136,16 @@ test('Android actionable systemui overlay (volume dialog) still survives with th const chromeRefs = collectSettleChromeRefs(nodes, 'com.callstack.agentdevicelab'); // The container is non-hittable with a generic id, but its hittable - // `volume_dialog_slider` child keeps it in the walked tree (descendantHittable). + // `volume_new_ringer_active_icon_container` child keeps it in the walked + // tree (descendantHittable). assert.equal( chromeRefs.has(refForIdentifier(nodes, 'com.android.systemui:id/volume_dialog_container')), false, ); assert.equal( - chromeRefs.has(refForIdentifier(nodes, 'com.android.systemui:id/volume_dialog_slider')), + chromeRefs.has( + refForIdentifier(nodes, 'com.android.systemui:id/volume_new_ringer_active_icon_container'), + ), false, ); // The status-bar leak fix stays active in the very same tree. diff --git a/src/daemon/handlers/__tests__/session-replay-divergence.test.ts b/src/daemon/handlers/__tests__/session-replay-divergence.test.ts index 9da29bb5b..ec13a0144 100644 --- a/src/daemon/handlers/__tests__/session-replay-divergence.test.ts +++ b/src/daemon/handlers/__tests__/session-replay-divergence.test.ts @@ -8,6 +8,18 @@ vi.mock('../../../core/dispatch.ts', async (importOriginal) => { return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() }; }); +// Stub the Android freshness-retry delay to a no-op so the capture-parity test +// exercises the retry BRANCH without a real wall-clock wait (repo guidance: +// no real-time sleeps in unit tests). This is the exact `sleep` the retry path +// in `snapshot-capture.ts` (`capturePostActionAwareSnapshot`) awaits; making it +// instant does not change control flow — the loop still runs, retries, and +// re-captures — so the test still proves two dispatches and use of the retried +// tree. No other test in this file triggers a sleep path. +vi.mock('../../../utils/timeouts.ts', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, sleep: vi.fn(async () => {}) }; +}); + import { dispatchCommand } from '../../../core/dispatch.ts'; import { makeAndroidSession, @@ -471,3 +483,578 @@ test('buildReplayFailureDivergence excludes Android status-bar/IME chrome from s expect(screen.refs.some((ref) => ref.label === '+48 555 010 010')).toBe(true); // field-phone expect(screen.truncated).toBeUndefined(); }); + +// #1264 coexistence (chrome filter + overlay, real walked fixture): a +// separate-window system overlay (volume dialog) that a full `snapshot` capture +// includes survives into divergence `screen.refs` — its actionable (hittable) +// content passing the meaningful-target filter — while ordinary +// status-bar/nav-bar/IME chrome in the SAME capture is still excluded. This +// runs the real non-raw Android walk over app content + status bar + overlay +// together, proving the chrome filter and the overlay coexist. NOTE: this +// fixture is small enough that the overlay fits inside the 20-cap regardless of +// ordering, so it does NOT by itself prove cap-burial resistance — that is the +// dedicated realistic-tree test below. +test('buildReplayFailureDivergence: a system-overlay window survives into screen.refs while status/nav chrome is filtered (#1264)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-overlay-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set( + sessionName, + makeAndroidSession(sessionName, { appBundleId: 'com.callstack.agentdevicelab' }), + ); + + const rawWithVolumeDialog = [ + ...ANDROID_IME_CAPTURE_RAW_NODES, + { + index: 9000, + type: 'android.widget.FrameLayout', + bundleId: 'com.android.systemui', + identifier: 'com.android.systemui:id/volume_dialog_container', + }, + { + index: 9001, + parentIndex: 9000, + type: 'android.widget.ImageButton', + bundleId: 'com.android.systemui', + identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container', + label: 'Ringer volume', + hittable: true, + }, + ]; + + mockDispatchCommand.mockResolvedValue({ + nodes: walkNonRawAndroidFixture(rawWithVolumeDialog), + truncated: false, + backend: 'android', + }); + + const action = { + ts: 0, + command: 'press', + positionals: ['label="Full name"'], + flags: {}, + result: { selectorChain: ['label="Full name"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + expect(divergence.screen.state).toBe('available'); + const screen = divergence.screen as Extract; + + // The overlay's actionable (hittable) control survives the filters, exactly + // as a plain `snapshot` at the same moment would show it. + expect(screen.refs.some((ref) => ref.label === 'Ringer volume')).toBe(true); + + // Ordinary status-bar/mobile/wifi systemui chrome and IME keys, present in + // the SAME capture, are still filtered — the overlay fix does not broaden + // the chrome filter into a no-op. + expect(screen.refs.some((ref) => ref.label === '7:03')).toBe(false); // clock + expect(screen.refs.some((ref) => ref.label === 'T-Mobile, signal full.')).toBe(false); // mobile_combo + expect(screen.refs.some((ref) => ref.label === 'Wifi signal full.')).toBe(false); // wifi_signal + expect(screen.refs.some((ref) => ref.label === 'Shift')).toBe(false); // IME key + + // App content underneath, from the same capture, is unaffected by either + // the overlay or the chrome filter. + expect(screen.refs.some((ref) => ref.label === 'Checkout form')).toBe(true); + expect(screen.refs.some((ref) => ref.label === 'review name')).toBe(true); // field-name +}); + +// Realistic full capture: 24 app controls enumerate in document order BEFORE a +// separate-window overlay, whose lone actionable dismiss target is captured +// LAST (index 24) — exactly the shape of a live tree where the app window's +// ~77 nodes precede a volume dialog / QS shade. 24 app + 1 overlay = 25 +// meaningful candidates > the 20-cap. Under the previous +// `candidates.slice(0, 20)` in document order, the overlay sits at position 24 +// and is TRUNCATED away (the archived #1264 evidence: `screen.truncated: true`, +// zero volume refs). Under within-cap ranking (foreign-bundle hittable dismiss +// targets promoted ahead of app content) it survives. This test fails on +// document-order slicing and passes with the ranking. +function appSwampWithTrailingOverlay(appBundleId: string) { + const appControls = Array.from({ length: 24 }, (_, i) => ({ + index: i, + depth: 1, + type: 'android.widget.Button', + bundleId: appBundleId, + label: `App control ${String(i + 1).padStart(2, '0')}`, + identifier: `${appBundleId}:id/control_${i + 1}`, + // Non-overlapping small rects down the app window: no occlusion fires. + rect: { x: 0, y: 10 * i, width: 120, height: 8 }, + hittable: true, + })); + return [ + ...appControls, + // The overlay's dismiss target, captured LAST — a foreign (systemui) window. + { + index: 24, + depth: 1, + type: 'android.widget.ImageButton', + bundleId: 'com.android.systemui', + identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container', + label: 'Ringer volume', + rect: { x: 300, y: 300, width: 44, height: 44 }, + hittable: true, + }, + ]; +} + +test('buildReplayFailureDivergence: a fully-captured overlay dismiss-target enumerating LAST still lands within the screen.refs cap (#1264 cap burial)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-burial-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + const appBundleId = 'com.callstack.agentdevicelab'; + sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId })); + + mockDispatchCommand.mockResolvedValue({ + nodes: appSwampWithTrailingOverlay(appBundleId), + truncated: false, + backend: 'android', + }); + + const action = { + ts: 0, + command: 'press', + positionals: ['label="App control 01"'], + flags: {}, + result: { selectorChain: ['label="App control 01"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + expect(divergence.screen.state).toBe('available'); + const screen = divergence.screen as Extract; + + // The overlay dismiss target, captured LAST (position 24, past the 20-cap in + // document order), survives because foreign-window hittable nodes are ranked + // ahead of app content within the cap. + expect(screen.refs.some((ref) => ref.label === 'Ringer volume')).toBe(true); + // It is ranked first, not merely present. + expect(screen.refs[0]?.label).toBe('Ringer volume'); + // Ranking is stable for equal-priority app content: the surviving app refs are + // still the earliest ones in document order (no reshuffle). + expect(screen.refs.some((ref) => ref.label === 'App control 01')).toBe(true); + // The report still honestly reports it dropped candidates beyond the cap. + expect(screen.truncated).toBe(true); +}); + +// Occlusion interaction (#1264): when a system overlay MASS-COVERS the app, +// every app node is annotated `interactionBlocked: 'covered'` (and +// `hittable: false`) by `annotateCoveredSnapshotNodes`. The covering overlay's +// own actionable node is NOT covered and IS the repair surface. A report whose +// capture holds these nodes but whose `refs` is empty is broken by +// construction. The overlay-like `Dialog` container (a later document node with +// a full-screen rect over the app buttons) triggers the occlusion annotation. +function overlayMassCoveringApp(appBundleId: string) { + const appButtons = Array.from({ length: 3 }, (_, i) => ({ + index: i, + depth: 1, + type: 'android.widget.Button', + bundleId: appBundleId, + label: `Field ${String.fromCharCode(65 + i)}`, + identifier: `${appBundleId}:id/field_${i + 1}`, + rect: { x: 20, y: 100 + 60 * i, width: 300, height: 44 }, + hittable: true, + })); + const dialogIndex = appButtons.length; + return [ + ...appButtons, + // Overlay-like container (`Dialog` type) covering the whole app area, later + // in document order — marks every app button `covered`. Non-hittable, so it + // is not itself a dismiss target. + { + index: dialogIndex, + depth: 0, + type: 'android.app.Dialog', + bundleId: 'com.android.systemui', + identifier: 'com.android.systemui:id/volume_dialog_container', + rect: { x: 0, y: 0, width: 402, height: 874 }, + }, + // The overlay's actionable dismiss target, a child of the dialog. + { + index: dialogIndex + 1, + parentIndex: dialogIndex, + depth: 1, + type: 'android.widget.ImageButton', + bundleId: 'com.android.systemui', + identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container', + label: 'Ringer volume', + rect: { x: 300, y: 300, width: 44, height: 44 }, + hittable: true, + }, + ]; +} + +test('buildReplayFailureDivergence: when a system overlay mass-covers the app, the overlay dismiss-target surfaces in screen.refs and refs is non-empty (#1264 occlusion)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-occlusion-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + const appBundleId = 'com.callstack.agentdevicelab'; + sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId })); + + mockDispatchCommand.mockResolvedValue({ + nodes: overlayMassCoveringApp(appBundleId), + truncated: false, + backend: 'android', + }); + + const action = { + ts: 0, + command: 'press', + positionals: ['label="Field A"'], + flags: {}, + result: { selectorChain: ['label="Field A"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + expect(divergence.screen.state).toBe('available'); + const screen = divergence.screen as Extract; + + // A capture with meaningful nodes must never yield an empty screen.refs. + expect(screen.refs.length).toBeGreaterThan(0); + // The overlay's non-covered actionable node is the repair surface and surfaces. + expect(screen.refs.some((ref) => ref.label === 'Ringer volume')).toBe(true); + // The mass-covered app buttons are correctly excluded while a non-covered + // dismiss target exists (they are not tappable under the overlay). + expect(screen.refs.some((ref) => ref.label === 'Field A')).toBe(false); +}); + +// Occlusion fallback (#1264): the covering overlay has NO actionable node of its +// own (a bare scrim), so once the app is mass-covered there is no non-covered +// meaningful candidate at all. Rather than emit an empty `screen.refs` — a +// report broken by construction — the covered app nodes are surfaced so the +// agent at least sees what is underneath. +function bareScrimMassCoveringApp(appBundleId: string) { + const appButtons = Array.from({ length: 3 }, (_, i) => ({ + index: i, + depth: 1, + type: 'android.widget.Button', + bundleId: appBundleId, + label: `Field ${String.fromCharCode(65 + i)}`, + identifier: `${appBundleId}:id/field_${i + 1}`, + rect: { x: 20, y: 100 + 60 * i, width: 300, height: 44 }, + hittable: true, + })); + return [ + ...appButtons, + // Bare overlay scrim: overlay-like `Dialog` type (covers the app) but no + // label, no identifier, non-hittable — not itself a meaningful candidate. + { + index: appButtons.length, + depth: 0, + type: 'android.app.Dialog', + bundleId: 'com.android.systemui', + rect: { x: 0, y: 0, width: 402, height: 874 }, + }, + ]; +} + +test('buildReplayFailureDivergence: a mass-covered app with no actionable overlay node still returns non-empty screen.refs (#1264 occlusion fallback)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-scrim-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + const appBundleId = 'com.callstack.agentdevicelab'; + sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId })); + + mockDispatchCommand.mockResolvedValue({ + nodes: bareScrimMassCoveringApp(appBundleId), + truncated: false, + backend: 'android', + }); + + const action = { + ts: 0, + command: 'press', + positionals: ['label="Field A"'], + flags: {}, + result: { selectorChain: ['label="Field A"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + expect(divergence.screen.state).toBe('available'); + const screen = divergence.screen as Extract; + + // No non-covered meaningful candidate exists, but the capture is not empty — + // the covered app content is surfaced rather than returning an empty report. + expect(screen.refs.length).toBeGreaterThan(0); + expect(screen.refs.some((ref) => ref.label === 'Field A')).toBe(true); +}); + +// #1257 + #1264 reconciliation: the ADR-0014 partial ref frame the divergence +// capture activates (`markSessionPartialRefsIssued`) must authorize EXACTLY the +// refs the `screen.refs` digest renders — both now derive from the shared +// `selectDivergenceScreenRefNodes`. This is load-bearing in the mass-covered +// fallback: the screen surfaces COVERED refs, which #1257's original +// non-covered-only `digestBodies` filter would have EXCLUDED from the frame — +// leaving the agent a ref the screen advertised but the frame rejects. Assert +// the frame scope equals the emitted screen ref set (covered refs included). +test('buildReplayFailureDivergence: the partial ref frame authorizes exactly the emitted screen.refs, including mass-covered fallback refs (#1257 + #1264)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-frame-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + const appBundleId = 'com.callstack.agentdevicelab'; + sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId })); + + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValue({ + nodes: bareScrimMassCoveringApp(appBundleId), + truncated: false, + backend: 'android', + }); + + const action = { + ts: 0, + command: 'press', + positionals: ['label="Field A"'], + flags: {}, + result: { selectorChain: ['label="Field A"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + const screen = divergence.screen as Extract; + const screenRefBodies = new Set(screen.refs.map((ref) => ref.ref)); + expect(screenRefBodies.size).toBeGreaterThan(0); + + // The partial ref frame the capture activated authorizes exactly the emitted + // screen refs — no more (no over-pin surface), no less (every advertised ref + // is usable), even though every one of them is a `covered` node here. + const stored = sessionStore.get(sessionName); + expect(stored?.refFrameState).toBe('active'); + expect(stored?.refFrameScope).toEqual(screenRefBodies); +}); + +// #1264 (capture parity, point 1): the divergence capture must go through the +// SAME `captureSnapshot` wrapper as a plain `snapshot`, so it inherits Android +// freshness + post-action retry. Otherwise a divergence could consume the first +// stale / app-scoped dump while a plain `snapshot` retries to the fresh +// full-window tree — a divergence STALER than `snapshot`. Here the session +// carries an active Android freshness marker (baselineCount 20); the first +// on-device dump is a stale, near-empty tree (sharp node-count drop, no +// meaningful content → the `sharp-drop` retry trigger), and only the RETRIED +// second dump contains the system overlay. The divergence must reflect the +// retried tree. The retry delay (`sleep`) is stubbed to a no-op at the top of +// this file, so the retry BRANCH runs without a real wall-clock wait. +test('buildReplayFailureDivergence: routes through the freshness-retry wrapper and uses the retried fresh tree, not the first stale dump (#1264 capture parity)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-fresh-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + const appBundleId = 'com.callstack.agentdevicelab'; + const session = makeAndroidSession(sessionName, { appBundleId }); + // Active freshness marker: a navigation-sensitive action just ran, and the + // pre-action baseline had 20 nodes, so a near-empty next dump is suspicious. + session.androidSnapshotFreshness = { + action: 'press', + markedAt: Date.now(), + baselineCount: 20, + routeComparable: false, + }; + sessionStore.set(sessionName, session); + + // Capture 1: stale, near-empty dump (a single bare view — no hittable/label/ + // id) → `sharp-drop` vs the 20-node baseline → triggers a retry. + const staleDump = { + nodes: [ + { + index: 0, + type: 'android.view.View', + bundleId: appBundleId, + rect: { x: 0, y: 0, width: 10, height: 10 }, + }, + ], + truncated: false, + backend: 'android', + }; + // Capture 2: the fresh full-window tree, holding the app control AND the + // separate-window system overlay's dismiss target. + const freshDump = { + nodes: [ + { + index: 0, + type: 'android.widget.Button', + bundleId: appBundleId, + label: 'App control', + identifier: `${appBundleId}:id/control`, + rect: { x: 20, y: 100, width: 200, height: 44 }, + hittable: true, + }, + { + index: 1, + type: 'android.widget.ImageButton', + bundleId: 'com.android.systemui', + identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container', + label: 'Ringer volume', + rect: { x: 300, y: 300, width: 44, height: 44 }, + hittable: true, + }, + ], + truncated: false, + backend: 'android', + }; + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValueOnce(staleDump).mockResolvedValueOnce(freshDump); + + const action = { + ts: 0, + command: 'press', + positionals: ['label="App control"'], + flags: {}, + result: { selectorChain: ['label="App control"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + // The freshness wrapper retried past the stale dump (2 on-device captures). + expect(mockDispatchCommand).toHaveBeenCalledTimes(2); + + expect(divergence.screen.state).toBe('available'); + const screen = divergence.screen as Extract; + // The overlay only exists in the RETRIED capture, so its presence proves the + // divergence used the fresh tree — parity with what a plain `snapshot` sees. + expect(screen.refs.some((ref) => ref.label === 'Ringer volume')).toBe(true); + expect(screen.refs.some((ref) => ref.label === 'App control')).toBe(true); +}); + +// #1264 (clean flags policy, point 2): a failed `snapshot --raw`/scoped/`-d` +// action must never narrow the DIAGNOSTIC divergence tree. The divergence +// capture builds its flags from a fixed policy (full-window, non-raw, default +// depth), NOT from the failed action's flags — so `snapshotRaw`/`snapshotScope`/ +// `snapshotDepth` on the action do not reach the capture. This inspects the +// context handed to the snapshot dispatch and asserts those narrowing flags are +// dropped while the interactive-only policy is still applied. +test('buildReplayFailureDivergence: divergence capture drops the action snapshotRaw/scope/depth flags (#1264 clean flags policy)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-flags-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + const appBundleId = 'com.callstack.agentdevicelab'; + sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId })); + + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValue({ + nodes: [ + { + index: 0, + type: 'android.widget.Button', + bundleId: appBundleId, + label: 'Submit', + identifier: `${appBundleId}:id/submit`, + rect: { x: 20, y: 100, width: 200, height: 44 }, + hittable: true, + }, + ], + truncated: false, + backend: 'android', + }); + + // A failed action that itself requested a raw, ref-scoped, depth-limited + // snapshot — none of which may reshape the divergence diagnostic tree. + const action = { + ts: 0, + command: 'press', + positionals: ['label="Submit"'], + flags: { snapshotRaw: true, snapshotScope: '@e5', snapshotDepth: 2 }, + result: { selectorChain: ['label="Submit"'] }, + }; + await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + expect(mockDispatchCommand).toHaveBeenCalled(); + const context = mockDispatchCommand.mock.calls[0]?.[4] as + | { + snapshotRaw?: boolean; + snapshotScope?: string; + snapshotDepth?: number; + snapshotInteractiveOnly?: boolean; + } + | undefined; + // The action's narrowing flags are stripped by the fixed divergence policy. + expect(context?.snapshotRaw).not.toBe(true); + expect(context?.snapshotScope).toBeUndefined(); + expect(context?.snapshotDepth).toBeUndefined(); + // The interactive-only policy (press → interactive) is still applied. + expect(context?.snapshotInteractiveOnly).toBe(true); +}); diff --git a/src/daemon/handlers/session-replay-divergence.ts b/src/daemon/handlers/session-replay-divergence.ts index da88adf5f..e1de6c6ad 100644 --- a/src/daemon/handlers/session-replay-divergence.ts +++ b/src/daemon/handlers/session-replay-divergence.ts @@ -1,14 +1,13 @@ import fs from 'node:fs'; import path from 'node:path'; -import { dispatchCommand } from '../../core/dispatch.ts'; -import { contextFromFlags } from '../context.ts'; import { markSessionPartialRefsIssued, setSessionSnapshot } from '../session-snapshot.ts'; import { isSparseSnapshotQualityVerdict } from '../../snapshot/snapshot-quality.ts'; import { displayLabel, formatRole } from '../../snapshot/snapshot-lines.ts'; import { redactDiagnosticData } from '../../kernel/redaction.ts'; +import type { CommandFlags } from '../../core/dispatch.ts'; import type { DaemonError, ResponseLevel } from '../../kernel/contracts.ts'; -import type { RawSnapshotNode, SnapshotBackend, SnapshotNode } from '../../kernel/snapshot.ts'; -import { buildSnapshotState } from './snapshot-capture.ts'; +import type { SnapshotNode } from '../../kernel/snapshot.ts'; +import { captureSnapshot } from './snapshot-capture.ts'; import { buildSelectorChainForNode, resolveSelectorChain, @@ -199,10 +198,35 @@ export function toReplayRepairHintCapture( } /** - * The single post-failure capture, blessed via the standard ref-issuing - * sequence (setSessionSnapshot -> markSessionSnapshotRefsIssued -> store). + * The single post-failure capture, blessed via the ADR-0014 partial ref-issuing + * sequence (setSessionSnapshot -> markSessionPartialRefsIssued -> store): a + * divergence screen publishes only its bounded ref set, so it activates a + * PARTIAL frame authorizing exactly those bodies, not a complete namespace. * Sparse captures do not write back (selector-capture reliability contract), * so a sparse verdict degrades the whole observation. + * + * ADR 0012 decision 4 amendment (#1264): this routes through `captureSnapshot` + * — the EXACT wrapper the `snapshot` command's backend calls + * (`dispatchSnapshotViaRuntime` -> `createDaemonSnapshotBackend`), which owns + * Android freshness + post-action retry (`capturePostActionAwareSnapshot`) on + * top of the per-platform capture (Android snapshot-helper full-window route + * with its graceful app-scoped fallback; iOS bounded system-modal probe path; + * macOS/Linux surface-scoped branches). Calling the inner single-shot + * `captureSnapshotData` instead would let a divergence consume a first stale / + * app-scoped dump while a plain `snapshot` retries to the fresh full-window + * tree — a divergence STALER or NARROWER than `snapshot`, which is exactly the + * invariant this amendment forbids: an agent must never see a healthier + * `screen` in a divergence report than a plain `snapshot` would show it. + * + * The capture flags are a CLEAN, fixed divergence-capture policy, NOT the + * failed action's flags: `snapshotRaw`/`snapshotScope`/`snapshotDepth` from a + * failed `snapshot --raw`/scoped/`-d` action would narrow or reshape the + * diagnostic tree below what a plain `snapshot` shows, so they are dropped. The + * only carried policy is interactive-only (`divergenceCaptureInteractiveOnly` — + * full for non-rect `get`/`is`/`wait` reads so static-text targets survive, + * interactive otherwise), matching heal's long-standing rule. The chrome filter + * (`collectSettleChromeRefs`) and the meaningful-target filter stay layered ON + * TOP of this full capture as FILTERS, never as a narrower scoping. */ export async function captureDivergenceObservation(params: { session: SessionState; @@ -212,25 +236,15 @@ export async function captureDivergenceObservation(params: { action: SessionAction; }): Promise { const { session, sessionName, sessionStore, logPath, action } = params; - const snapshotInteractiveOnly = divergenceCaptureInteractiveOnly(action); + const flags = divergenceCaptureFlags(action); try { - const data = (await dispatchCommand(session.device, 'snapshot', [], undefined, { - ...contextFromFlags( - logPath, - { ...(action.flags ?? {}), snapshotInteractiveOnly }, - session.appBundleId, - session.trace?.outPath, - ), - })) as { - nodes?: RawSnapshotNode[]; - truncated?: boolean; - backend?: SnapshotBackend; - quality?: unknown; - }; - const snapshot = buildSnapshotState(data, { - ...(action.flags ?? {}), - snapshotInteractiveOnly, + const capture = await captureSnapshot({ + device: session.device, + session, + flags, + logPath, }); + const snapshot = capture.snapshot; if (isSparseSnapshotQualityVerdict(snapshot.snapshotQuality)) { return { state: 'unavailable', @@ -239,19 +253,17 @@ export async function captureDivergenceObservation(params: { }; } setSessionSnapshot(session, snapshot); - // ADR 0014: the divergence screen digest publishes only its capped, - // non-covered, non-chrome refs — the same set `buildReplayDivergenceScreenRefs` - // renders. Activate a PARTIAL frame authorizing exactly those bodies. - const chromeRefs = collectSettleChromeRefs(snapshot.nodes, session.appBundleId); - const digestBodies = snapshot.nodes - .filter( - (node) => - node.ref !== undefined && - node.interactionBlocked !== 'covered' && - !chromeRefs.has(node.ref), - ) - .slice(0, SCREEN_REF_CAPTURE_LIMIT) - .map((node) => node.ref as string); + // ADR 0014 (#1257) + #1264: the divergence screen publishes exactly the + // ranked, occlusion-resolved, capped ref set `screen.refs` renders. Activate + // a PARTIAL frame authorizing precisely THOSE bodies — derived from the same + // `selectDivergenceScreenRefNodes` the digest uses, so the frame never + // authorizes a ref the screen hides (over-pin risk) nor rejects one the + // screen advertised (e.g. the mass-covered fallback surfaces covered refs + // that the old non-covered-only filter would have excluded here). + const digestBodies = selectDivergenceScreenRefNodes( + snapshot.nodes, + session.appBundleId, + ).nodes.map((node) => node.ref as string); markSessionPartialRefsIssued(session, digestBodies); sessionStore.set(sessionName, session); return { @@ -269,6 +281,17 @@ export async function captureDivergenceObservation(params: { } } +/** + * The clean, fixed flags for a divergence capture (#1264): full-window + * (no `snapshotScope`), non-raw (no `snapshotRaw`), default depth (no + * `snapshotDepth`) — a failed scoped/raw/depth-limited action must never + * produce a narrowed divergence `screen`. Only the interactive-only policy is + * carried, since it governs whether static-text suggestion targets survive. + */ +function divergenceCaptureFlags(action: SessionAction): CommandFlags { + return { snapshotInteractiveOnly: divergenceCaptureInteractiveOnly(action) }; +} + /** * Interactive-only capture, except for non-rect selector reads * (`get`/`is`/`wait`) whose suggestion targets include static text — the @@ -324,6 +347,80 @@ function isMeaningfulDivergenceTarget(node: SnapshotNode): boolean { return Boolean(displayLabel(node, formatRole(node.type ?? 'Element'))) || node.hittable === true; } +/** + * ADR 0012 decision 4 amendment (#1264): a hittable node owned by a window + * OTHER than the app under test — a system-overlay window (volume dialog, + * quick-settings shade, permission dialog) whose actionable nodes are the + * dismiss targets for whatever is covering the app. Ownership is the node's own + * `bundleId`/package: Android sets it per node from the accessibility `package`, + * so a systemui/permission-controller/`android` node reads as foreign; iOS and + * macOS leave per-node `bundleId` undefined, so this is inert there (those + * platforms surface separate-window modals through the dedicated probe path, + * not by cap-competing with app content). Guarded on a known `appBundleId` so a + * sessionless capture never reorders — without an app identity there is no + * "foreign" to promote. + */ +function isForeignOverlayDismissTarget( + node: SnapshotNode, + appBundleId: string | undefined, +): boolean { + return ( + appBundleId !== undefined && + node.bundleId !== undefined && + node.bundleId !== appBundleId && + node.hittable === true + ); +} + +/** + * The single source of truth for which nodes a divergence `screen.refs` + * publishes, and in what order. Both the rendered `screen.refs` digest + * (`buildReplayDivergenceScreenRefs`) AND the ADR-0014 partial ref frame the + * capture authorizes (`captureDivergenceObservation` → + * `markSessionPartialRefsIssued`) derive from THIS function, so the authorized + * ref set is exactly the set the agent is shown — never a superset it can pin + * refs outside of, nor a subset that rejects a ref the screen advertised. + * Returns the capped node list plus whether ranking overflowed the cap. + */ +function selectDivergenceScreenRefNodes( + nodes: SnapshotNode[], + appBundleId: string | undefined, +): { nodes: SnapshotNode[]; truncated: boolean } { + // Keyboard/IME chrome must not consume the ref budget: it reuses the exact + // structural classifier `--settle`'s tail already relies on (#1198/#1200) + // rather than a second keyboard/IME node-type list. + const chromeRefs = collectSettleChromeRefs(nodes, appBundleId); + const meaningful = nodes.filter( + (node) => node.ref && !chromeRefs.has(node.ref) && isMeaningfulDivergenceTarget(node), + ); + // Occlusion fallback (#1264): a `covered` node is normally dropped — an agent + // cannot tap what an overlay hides. But when a system overlay MASS-COVERS the + // app, EVERY app node is annotated `covered`; dropping them all would emit an + // empty `screen.refs` while the capture plainly holds meaningful nodes — a + // report broken by construction (the agent is shown nothing to act on). So + // `covered` nodes are excluded only while non-covered candidates remain; if + // the entire meaningful surface is covered, they are surfaced rather than + // returning empty. + const visible = meaningful.filter((node) => node.interactionBlocked !== 'covered'); + const pool = visible.length > 0 ? visible : meaningful; + // Rank within the cap instead of slicing document order (#1264 cap burial): + // `SCREEN_REF_CAPTURE_LIMIT` is a BYTE bound, NOT a "first 20 in tree order" + // policy. A separate-window overlay enumerates AFTER the app window's nodes, + // so on a realistic tree its dismiss target sits past position 20 and is + // truncated away even though it was captured. Foreign-bundle hittable overlay + // nodes (the dismiss targets) are promoted ahead of app content; ordering is + // otherwise STABLE — document order is preserved within each tier, so + // equal-priority app nodes are never reshuffled. `repairHint`/`suggestions` + // consume the FULL captured node list, not this slice, so hint routing is + // unaffected; only the agent-visible `screen.refs` selection changes. + const ranked = [ + ...pool.filter((node) => isForeignOverlayDismissTarget(node, appBundleId)), + ...pool.filter((node) => !isForeignOverlayDismissTarget(node, appBundleId)), + ]; + const selected = ranked.slice(0, SCREEN_REF_CAPTURE_LIMIT); + return { nodes: selected, truncated: ranked.length > selected.length }; +} + function buildReplayDivergenceScreenRefs( nodes: SnapshotNode[], sanitize: DivergenceFieldSanitizer, @@ -332,18 +429,8 @@ function buildReplayDivergenceScreenRefs( refs: ReplayDivergenceScreenRef[]; truncated: boolean; } { - // Keyboard/IME chrome must not consume the ref budget: it reuses the exact - // structural classifier `--settle`'s tail already relies on (#1198/#1200) - // rather than a second keyboard/IME node-type list. - const chromeRefs = collectSettleChromeRefs(nodes, appBundleId); - const candidates = nodes.filter( - (node) => - node.ref && - node.interactionBlocked !== 'covered' && - !chromeRefs.has(node.ref) && - isMeaningfulDivergenceTarget(node), - ); - const refs = candidates.slice(0, SCREEN_REF_CAPTURE_LIMIT).map((node) => { + const { nodes: selected, truncated } = selectDivergenceScreenRefNodes(nodes, appBundleId); + const refs = selected.map((node) => { const role = formatRole(node.type ?? 'Element'); const label = displayLabel(node, role); return { @@ -352,7 +439,7 @@ function buildReplayDivergenceScreenRefs( ...(label ? { label: sanitize(label) } : {}), }; }); - return { refs, truncated: candidates.length > refs.length }; + return { refs, truncated }; } const BASIS_RANK: Record = {