Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Changelog

## Unreleased

- Breaking (ADR 0014, session ref-frame lifetime): a mutation through an `@ref` now expires the session's ref frame, so a later ref mutation without a fresh observation fails closed with a typed `details.reason` (`ref_frame_expired`, `ref_generation_mismatch`, `plain_ref_requires_complete_frame`, or `ref_not_issued`) instead of acting on a possibly-navigated screen. A ref-oriented sequence that performs several mutations must re-`snapshot` between them, consume an honestly issued `--settle` ref in pinned `@eN~s<gen>` form, or use selectors. Enforcement applies on every platform, not just iOS. Legacy hand-written `.ad` scripts that reuse several bare refs from one snapshot must capture between mutations or use selectors.
- Ref reads resolve against the authorized ref frame's source tree, so an internal read-only capture (including Android freshness) can no longer retarget an admitted `@ref` by positional coincidence. Read-only ref consumers keep the structured staleness warning while the frame retains the ref's evidence.

## 0.15.0

- Breaking: `apps` discovery and public app-list helpers now default to user-installed apps. Use `--all` or `filter: 'all'` to include system/OEM apps.
Expand Down
8 changes: 6 additions & 2 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@
- Coverage manifest: `CONTRACT_COVERAGE` export beside each interaction contract test file claiming which matrix cells it proves; the coverage gate requires every enforced/delegated cell to be claimed and rejects overclaims of waived cells.
- Delegation-on-error: a fast path falling back to the runtime path on semantic failure shapes. It closes failure-side guarantee cells only — never success-path parity.
- Ref generation pin: optional `~s<n>` suffix on an @ref carrying the snapshot generation it was minted from. Accepted as input everywhere, emitted by no tree output (snapshot token budget), auto-appended by the MCP layer, stripped and ignored by replay.
- Ref frame (ADR 0014): the session's single authorization namespace for mutation `@ref`s, kept separate from the latest operational observation (`session.snapshot`). It owns a frozen epoch (the `refsGeneration` the client received), an immutable source tree, a lifecycle state (`active`/`expired`), and an issuance scope (`all` for a complete snapshot, or the bounded set of ref bodies a partial publication emitted). Owned solely by `src/daemon/ref-frame.ts`. A complete snapshot activates an `all` frame; `find`/settled diff/replay divergence activate a bounded partial frame that supersedes the prior one; internal read captures never activate or reindex it.
- Frame expiry seam (ADR 0014): every mutating leaf calls `expireRefFrame` synchronously, immediately before the device op that may change element identity (after all pre-action guards), so a post-dispatch failure still leaves the frame expired — there is no success-only rollback. Ref resolution binds `@eN` against the frame's source tree, so an Android freshness (or any read-only) capture cannot retarget an admitted ref by positional coincidence; a fresh capture's coordinates are adopted only when its node's local identity matches.
- Mutation admission (ADR 0014): a ref mutation is admitted only against an active frame whose epoch and issuance scope authorize the ref (`admitRefMutation`, order-sensitive reasons `ref_frame_expired` → `ref_generation_mismatch` → `plain_ref_requires_complete_frame` → `ref_not_issued`). Rejections carry `details.reason` and name the lifetime failure. A ref-oriented sequence that performs several mutations must re-observe (snapshot), consume an honestly issued settled ref in pinned form, or use selectors. Read-only ref consumers stay fail-open with a staleness warning while the frame retains the ref's evidence.
- Settled observation: opt-in (`--settle`) post-action payload on press/click/fill/longpress — the quiet-window stable loop re-captures until the UI settles, and the response carries the diff vs the pre-action tree (changed lines only, added lines with fresh refs, `refsGeneration` when the settled tree was stored). Best-effort: never fails the action; `settled: false` plus a hint on never-quiet content.
- Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree, query sweep, private AX) run by one plan runner under a shared wall-clock budget; recovery ordering is declared data, never a per-call-site branch.
- Snapshot quality verdict: structured outcome (state, backend, reason code, effective depth, collapsed leaves) computed once by the plan runner and shipped with every planned snapshot payload; the daemon and CLI render it instead of re-deriving degradation from node shapes.
Expand Down Expand Up @@ -136,8 +139,9 @@ the observable freshness and failure semantics below before any runtime refactor
direct miss may fall back to the snapshot selector path, but ambiguous matches and runner errors
must surface instead of silently falling back. `get text` uses direct native selectors only for
simple `id` selectors because label/text/value reads need snapshot disambiguation.
- Regular selector reads remain capture-backed. `@ref` reads resolve against stored session
snapshots; selector `get`/`is`/`find`/`wait` capture through the backend. `find` and `wait`
- Regular selector reads remain capture-backed. `@ref`s resolve against the authorized ref frame's
source tree (ADR 0014), not whatever now sits at that index in a newer observation; selector
`get`/`is`/`find`/`wait` capture through the backend. `find` and `wait`
polling must bypass the 750 ms snapshot cache. The cache is also bypassed while Android freshness
recovery or post-gesture stabilization is active.
- Sparse snapshot quality verdicts are observable failures. Sparse captures must not replace
Expand Down
11 changes: 6 additions & 5 deletions docs/adr/0012-interactive-replay.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,11 +463,12 @@ every `screen` ref with `refsGeneration` before returning it, including on the e
client callers receive the unpinned refs and generation already present in the daemon error. No caller
gets a text-only divergence that loses its repair data.

> **Proposed ADR 0014 amendment:** if ADR 0014 is accepted and implemented, replay divergence
> `screen.refs` becomes a partial ref publication. MCP keeps auto-pinning those refs; CLI text renders
> `@eN~s<refsGeneration>`; JSON and Node.js callers pair each plain ref with the response-level
> generation before mutation. Until that migration lands, the current unpinned contract above remains
> accepted. The implementation must update this note and its contract tests in the same change.
> **ADR 0014 amendment (accepted, implemented):** replay divergence `screen.refs` is now a partial ref
> publication — it activates a bounded partial ref frame authorizing exactly the divergence screen's refs
> (`markSessionPartialRefsIssued`). MCP auto-pins those refs; CLI text renders `@eN~s<refsGeneration>`;
> JSON and Node.js callers pair each plain ref with the response-level generation before mutation. Because
> the frame is partial, a mutation through a divergence ref requires the pinned form; a plain ref there
> reports `plain_ref_requires_complete_frame`.

`--from N` is a `replay`-only flag. `test` must reject it as `INVALID_ARGS`; test shares replay execution
but must remain a full, deterministic suite run. `N` is a 1-based index into the fully expanded
Expand Down
52 changes: 35 additions & 17 deletions docs/adr/0014-session-ref-frame-lifetime.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Status

Proposed
Accepted

## Context

Expand Down Expand Up @@ -35,10 +35,9 @@ remain compatible with selector-based replay, and add no automatic capture or pe

## Decision

The terms introduced below describe the proposed target model. They do not replace the current domain
language in `CONTEXT.md` while this ADR is Proposed and the source still implements the snapshot/stale
marker model. The implementation change that establishes these concepts must promote the accepted
terms into `CONTEXT.md`; documentation must not present the target model as current behavior early.
The terms introduced below are now the implemented model; the ref-frame vocabulary is promoted into
`CONTEXT.md`. The coarse `snapshotRefsStale` marker still backs read-only staleness warnings and is
removed once per-platform enforcement is confirmed by live evidence (migration step 8).

### Ref frames are separate from operational observations

Expand Down Expand Up @@ -69,15 +68,15 @@ compatibility.

### Frame transitions

| Event | Frame transition | Plain refs | Pinned refs |
| --- | --- | --- | --- |
| Internal observation that returns no refs | None | Unchanged | Unchanged |
| Complete frame activation | New active epoch, scope `all` | Accepted from this frame | Accepted when epoch matches |
| Non-empty partial publication | New active epoch, scope = emitted refs | Rejected | Accepted only for an emitted ref at this epoch |
| First possible device-side effect | Advance once to `expired` | Rejected | Previous epoch rejected |
| Additional effects while already expired | Idempotent | Rejected | Rejected |
| Sparse, failed, or unusable capture | None | Unchanged | Unchanged |
| Session reopen | New random-seeded lifetime | Rejected until publication | Old lifetime rejected probabilistically as today |
| Event | Frame transition | Plain refs | Pinned refs |
| ----------------------------------------- | -------------------------------------- | -------------------------- | ------------------------------------------------ |
| Internal observation that returns no refs | None | Unchanged | Unchanged |
| Complete frame activation | New active epoch, scope `all` | Accepted from this frame | Accepted when epoch matches |
| Non-empty partial publication | New active epoch, scope = emitted refs | Rejected | Accepted only for an emitted ref at this epoch |
| First possible device-side effect | Advance once to `expired` | Rejected | Previous epoch rejected |
| Additional effects while already expired | Idempotent | Rejected | Rejected |
| Sparse, failed, or unusable capture | None | Unchanged | Unchanged |
| Session reopen | New random-seeded lifetime | Rejected until publication | Old lifetime rejected probabilistically as today |

A complete activation normally accompanies a command result that exposes the complete ref namespace
for the stored frame, including an interactive or intentionally scoped snapshot. An intentionally
Expand Down Expand Up @@ -210,9 +209,7 @@ The command descriptor's daemon facet declares a request policy:
```ts
type RefFrameEffect = 'preserve' | 'may-invalidate' | 'delegated';

type DaemonRefFrameEffect =
| RefFrameEffect
| ((request: DaemonRequest) => RefFrameEffect);
type DaemonRefFrameEffect = RefFrameEffect | ((request: DaemonRequest) => RefFrameEffect);
```

The daemon registry exposes a named resolver. Every daemon command is classified, but the
Expand Down Expand Up @@ -440,6 +437,27 @@ Each step lands green and independently useful:
evidence; promote the implemented vocabulary into `CONTEXT.md`; then remove the superseded coarse
stale marker.

Implementation status: steps 1–7 have landed — the ref-frame module and observation split (1), the
complete daemon classification and gate (2), the pre-side-effect seam at every leaf (3), correct
complete/partial publication with bounded scope, MCP pin retention, and pinned partial CLI text (4),
Android freshness decoupled from positional ref authorization (5), the cross-platform contract and
provider evidence (6), and fail-closed admission enforcement across platforms with typed reasons (7).
Fresh live evidence has exercised nearly every production seam — Apple runtime-ref, direct/native
selector, generation-pin, generic, and lifecycle paths; Android helper freshness (including proven
non-retarget) and Android existing-session relaunch; and a real provider-backed interaction plus
provider-backed lifecycle operation (AWS Device Farm, `backend: webdriver`: a fresh ref succeeded, an
immediate stale ref was rejected before dispatch with the shared typed fields, an `open --relaunch`
lifecycle mutation expired the frame, and a fresh observation restored authorization). ONE seam —
Android blocking-dialog recovery — was NOT live-exercised: the repo harness exposes no deterministic
app-owned ANR trigger, and no reproducible control exists to raise the system dialog on demand. The
team accepted shipping without a live run for it: its transition and abort logic are covered by fixture
regressions (`android-system-dialog-ref-frame.test.ts` proves recovery expires the frame before its
tap; `interaction-android-recovery-abort.test.ts` proves an outstanding ref action then aborts with the
shared `ref_frame_expired` rejection and no dispatch), the seam is enforced in code identically to the
verified paths, and it is recorded here as a documented, accepted evidence gap rather than an open
release blocker. Every other enabled seam is proven on hardware, so step 8's removal of the superseded
coarse `snapshotRefsStale` marker is unblocked.

PR #1241 landed independently as a compatible transitional fix. It rejects a known iOS stale-marker
case before this full lifecycle is implemented; it does not own the architecture migration.

Expand Down
18 changes: 18 additions & 0 deletions src/__tests__/client.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -903,6 +903,24 @@ test('client capture.snapshot preserves visibility metadata from daemon response
});
});

test('client capture.snapshot preserves refsGeneration from daemon responses (ADR 0014)', async () => {
const setup = createTransport(async () => ({
ok: true,
data: {
nodes: [{ ref: 'e1', index: 0, depth: 0, type: 'Button', label: 'Go' }],
truncated: false,
refsGeneration: 752890,
},
}));
const client = createAgentDeviceClient(setup.config, { transport: setup.transport });

const result = await client.capture.snapshot();

// Node.js callers must retain the response-level generation to pin a plain ref
// (`@e1~s752890`) before a mutation.
assert.equal(result.refsGeneration, 752890);
});

test('client capture.snapshot preserves snapshot quality annotation from daemon responses', async () => {
const snapshotQuality = {
state: 'recovered',
Expand Down
4 changes: 4 additions & 0 deletions src/agent-device-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,7 @@ function optionalSnapshotResponseFields(
| 'warnings'
| 'snapshotQuality'
| 'snapshotDiagnostics'
| 'refsGeneration'
>
> {
const visibility = readObject(data.visibility);
Expand All @@ -511,6 +512,9 @@ function optionalSnapshotResponseFields(
...readSerializedSnapshotCaptureAnnotations(data),
...(unchanged ? { unchanged: unchanged as CaptureSnapshotResult['unchanged'] } : {}),
...(snapshotDiagnostics ? { snapshotDiagnostics } : {}),
// ADR 0014: keep the response-level ref-frame generation on Node.js results
// so callers can pin refs (`@e12~s<refsGeneration>`) before a mutation.
...(typeof data.refsGeneration === 'number' ? { refsGeneration: data.refsGeneration } : {}),
};
}

Expand Down
6 changes: 6 additions & 0 deletions src/client/client-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,12 @@ export type CaptureSnapshotResult = {
unchanged?: SnapshotUnchanged;
snapshotDiagnostics?: SnapshotDiagnosticsSummary;
identifiers: AgentDeviceIdentifiers;
/**
* ADR 0014: the response-level ref-frame epoch the plain node refs were minted
* from. A ref-issuing snapshot carries it ONCE (nodes stay plain `@e12` for the
* token budget); pair a ref with it (`@e12~s<refsGeneration>`) before a mutation.
*/
refsGeneration?: number;
} & PublicSnapshotCaptureAnnotations;

export type CaptureScreenshotOptions = AgentDeviceRequestOverrides & {
Expand Down
20 changes: 19 additions & 1 deletion src/commands/interaction/output.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,27 @@ function getCliOutput(params: { result: CommandRequestResult; format?: string })
return defaultCommandCliOutput(data);
}

// ADR 0014: a reusable ref in a PARTIAL result renders in ready-to-copy
// `@eN~s<refsGeneration>` form so a human CLI caller can paste it into the next
// mutation without a separate pin step. A mutating result carries no
// `refsGeneration`, so its acted ref is never pinned.
function pinnedRefText(ref: unknown, refsGeneration: unknown): string | undefined {
if (typeof ref !== 'string' || ref.length === 0) return undefined;
if (typeof refsGeneration !== 'number') return undefined;
const body = ref.startsWith('@') ? ref.slice(1) : ref;
return `@${body}~s${refsGeneration}`;
}

function findCliOutput(result: CommandRequestResult): CliOutput {
const data = result as Record<string, unknown>;
// Interactive find actions (click/fill/focus/type) carry the same success message as
// their direct counterparts; prefer it over the raw text field fill responses include.
const message = readCommandMessage(data);
if (message) return { data, text: message };
if (typeof data.text === 'string') return { data, text: data.text };
// A read-only find that returns a reusable ref renders it pinned (ADR 0014).
const pinned = pinnedRefText(data.ref, data.refsGeneration);
if (pinned) return { data, text: `Found: ${pinned}` };
if (typeof data.found === 'boolean') return { data, text: `Found: ${data.found}` };
if (data.node) return { data, text: JSON.stringify(data.node, null, 2) };
return defaultCommandCliOutput(data);
Expand Down Expand Up @@ -64,6 +78,7 @@ type SettleTextView = {
};
tail?: Array<{ ref?: string; role?: string; label?: string }>;
tailTruncated?: boolean;
refsGeneration?: number;
};

/**
Expand Down Expand Up @@ -100,7 +115,10 @@ function formatSettleTailLines(view: SettleTextView): string[] {
const lines = [`unchanged interactive (${tail.length}):`];
for (const entry of tail) {
const label = entry.label ? ` "${entry.label}"` : '';
lines.push(`= @${entry.ref ?? ''} [${entry.role ?? ''}]${label}`);
// ADR 0014: the settled tail refs are reusable, so render them pinned when
// the settle response carried its generation.
const ref = pinnedRefText(entry.ref, view.refsGeneration) ?? `@${entry.ref ?? ''}`;
lines.push(`= ${ref} [${entry.role ?? ''}]${label}`);
}
if (view.tailTruncated) {
lines.push('… more interactive elements not shown, use snapshot -i');
Expand Down
23 changes: 14 additions & 9 deletions src/commands/interaction/runtime/resolution.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ test('runtime interactions reject unsupported macOS desktop and menubar surfaces
assert.equal(pressed, true);
});

test('runtime ref interactions refresh the snapshot when a stored ref has no usable rect', async () => {
test('runtime ref interactions fail closed when the authorized ref has no usable bounds (ADR 0014)', async () => {
const staleSnapshot = makeSnapshotState([
{
index: 0,
Expand All @@ -422,25 +422,30 @@ test('runtime ref interactions refresh the snapshot when a stored ref has no usa
hittable: true,
},
]);
const freshSnapshot = selectorSnapshot();
const calls: Point[] = [];
let captures = 0;
const device = createInteractionDevice(staleSnapshot, {
captureSnapshot: async () => {
captures += 1;
return { snapshot: freshSnapshot };
return { snapshot: selectorSnapshot() };
},
tap: async (_context, point) => {
calls.push(point);
},
});

const result = await device.interactions.click(ref('@e1'), { session: 'default' });

assert.equal(captures, 1);
assert.deepEqual(calls, [{ x: 60, y: 40 }]);
assert.equal(result.kind, 'ref');
assert.equal(result.node?.rect?.width, 100);
// ADR 0014: the authorized frame's @e1 has no usable rect, so it FAILS rather
// than recapturing and accepting the same index from a newer tree by
// positional coincidence.
await assert.rejects(
() => device.interactions.click(ref('@e1'), { session: 'default' }),
(error: unknown) => {
assert.match((error as Error).message, /Ref @e1 not found or has no bounds/);
return true;
},
);
assert.equal(captures, 0);
assert.deepEqual(calls, []);
});

test('tryResolveRefNode discloses exact for a resolved ref and label-fallback for label recovery', () => {
Expand Down
Loading
Loading