Skip to content

fix: make each Extension ref resolve to exactly one element - #66

Merged
cryptotavares merged 3 commits into
cryptotavares/mmcli-p1-10-ext-speedupsfrom
cryptotavares/mmcli-p1-11-exact-refs
Oct 29, 2026
Merged

cryptotavares merged 3 commits into
cryptotavares/mmcli-p1-10-ext-speedupsfrom
cryptotavares/mmcli-p1-11-exact-refs

Conversation

@cryptotavares

@cryptotavares cryptotavares commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary: Give duplicate browser accessibility refs unique DOM selectors when possible. Resolve ref targets page-wide before checking --within, so a parent cannot select another member of an ambiguous group.

Stack context: mm CLI v2, Phase 1, P1-11. Base: #65 (cryptotavares/mmcli-p1-10-ext-speedups); followed by #67. Review #65 first.

Problem

Browser refs use role-and-name selectors. Two elements with the same role and name therefore share one selector. A click fails in Playwright strict mode. With --within, the parent can narrow that selector to a different element and click the wrong row.

The Extension has repeated labels such as “Show QR code”, “Copy address”, and “Network options”. Ref identity must distinguish these elements without assuming AX-tree order matches Playwright locator order.

Two rows each expose a “Show QR code” button.
Ref for row B + --within row A
Before: the shared role selector can select row A's button.

Solution

Build ref selectors at the browser discovery boundary. Keep the existing selector map consumed by Playwright tools and session managers.

flowchart TD
    A[AX snapshot candidates] --> B[Group normalized role-and-name selectors]
    B -->|unique role| C[Keep exact role selector with snapshot scope]
    B -->|duplicate role or generic| D[Build unique DOM selector in cached isolated world]
    D --> E{Selector result?}
    E -->|unique| H[Store selector in ref map]
    E -->|unavailable| F[Keep strict role or text fallback]
    E -->|closed shadow root| G[Store unreachable marker]
    C --> H
    F --> H
    H --> I[Resolve Playwright locator for action]
    G --> X[Reject ref action before locator creation]

    classDef new fill:#fff3cd,stroke:#d39e00
    classDef stop fill:#f8d7da,stroke:#c82333
    class B,C,D,E,F,G new
    class X stop
Loading

Yellow boxes are new. Red boxes stop the action.

The DOM builder prefers a unique data-testid, then a unique id, then a unique ancestor-anchored path. It uses nth-of-type or nth-child only within a path whose uniqueness it checks. Open shadow roots use chained host selectors: css=<host> >> css=<inside>.

callOnNodeInIsolatedWorld resolves the backend node in the cached session and isolated world introduced by P1-10. This keeps selector construction outside the Extension's scuttled main-world globals. Lookups run concurrently with a fixed bound. Session or world failures retry once; missing nodes and page-function exceptions do not.

Role and text selectors normalize whitespace and escape quoted values. The explicit s flag documents the role engine's existing exact, case-sensitive behavior. Scoped snapshots prefix role and text selectors with css=<root> >> nth=0. Unique DOM selectors remain page-wide; the scope root itself uses the root selector.

For --within plus a ref, waitForTarget checks page-wide ambiguity before containment. It rechecks an outside-parent result after a short delay to allow DOM movement. Late targets wait for attachment within the shared timeout budget.

What changes

Condition or boundary Before After
Duplicate role and name with resolvable DOM nodes Refs share a role selector Each ref gets its own verified unique DOM selector
Duplicate without a unique DOM selector Ambiguous role selector Keeps the strict fallback; adds no AX-order nth= guess
Selector builder finds a closed shadow root Other selectors cannot reach that node Stores an unreachable marker; ref resolution rejects it
Scoped snapshot Role selectors search page-wide Role and text selectors retain the snapshot root; fallback relatives remain page-wide
Ref target with --within Parent can narrow an ambiguous ref to another element Rejects page-wide ambiguity and a confirmed outside-parent target
Consumer interface Map<string, string> and ISessionManager Same contracts; browser discovery and locator resolution own the change

Ref-specific errors include Ref e2 is not inside the within element and Ref e2 is inside a closed shadow root: it cannot be targeted. An ambiguous ref reports Ref e2 is ambiguous (9 matches); take a new snapshot or target it by testId or selector.

Decisions for reviewers

Decision Reason
Keep selector-backed refs instead of the backend-node ref model preferred by P1-11 Tools need Playwright locators. Consumers already implement ISessionManager with selector strings.
Verify DOM paths instead of assigning an AX-order index Dialogs, inert, shadow roots, slots, and aria-owns can produce different AX and Playwright orders.
Reuse the P1-10 isolated-world cache Avoids per-node contexts and keeps shared session recovery in cdp.ts.
Treat --within as containment for ref targets A parent must not choose among the page-wide matches of an ambiguous ref.

Review guide

Read the files in this order:

# File What to check
1 src/tools/utils/discovery.ts Candidate grouping, scoped snapshots, fallback selectors, and page-wide within checks.
2 src/tools/utils/in-page.ts Self-contained selector builder, uniqueness rules, shadow roots, and escaping.
3 src/tools/utils/cdp.ts Cached-world reuse, node resolution, retry classification, and object-group release.
4 src/tools/utils/{discovery,in-page,cdp}.test.ts Duplicate targeting, parser round trips, bounded concurrency, shadow cases, and containment races.
5 src/server/create-server.test.ts Queue-wait test holds the first request longer; discovery tests tolerate the remaining timeout budget.
6 README.md, SKILL.md Browser ref guarantees, strict failures, and --within guidance.
Testing
  • Fresh check at remote head e93a529e34cc3b1c65615e87bbd282a23e16f504: CI=true yarn vitest run src/tools/utils/cdp.test.ts src/tools/utils/discovery.test.ts src/tools/utils/in-page.test.ts src/server/create-server.test.ts src/platform/playwright-driver.test.ts --coverage.enabled=false --maxWorkers=2 passes: 5 files, 352 tests, no type errors.
  • Existing tests cover duplicate selectors (AC1), exact names (AC2), scope roots, strict fallbacks, and delayed or ambiguous within targets.
  • In-page tests execute function source in a VM with a fake DOM. They cover open, nested, slotted, and closed shadow roots, escaping, and scuttled globals.
  • The Proof and Unit tests sections below retain the original historical evidence. The fresh check does not repeat that live validation.

Breaking changes

None. Refs are created per snapshot, so no caller can depend on the old selector strings. Behaviour changes, both towards failing instead of hitting the wrong element:

  • within + a ref outside the parent now fails fast instead of acting inside the parent;
  • a ref that cannot be made unique fails in strict mode, as before.

Consumer impact

Consumer Change needed Done on branch
Extension test/e2e/playwright/llm-workflow/ None n/a
Mobile tests/llm-workflow/ None (browser-only code paths) n/a

Proof

Tested with a yalc build of this branch in the Extension (e2e build, headless, fixture wallet). For the "before" I built #65 (50eb396) in a temp clone and ran the same script.

Check Before (#65) After (f7057e7)
Click the ref of a duplicated label (9 cases: groups of 2, 9 and 13, positions 0–8) 9/9 fail (strict mode: 2 / 9 / 13 matches) 9/9 hit the right element (dialog heading, clipboard address, Edit form name and chain id)
All 13 "Network options" refs → Edit form strict-mode failure 13/13 open their own network
--within <first row> + ref of the Solana row's QR button opened Ethereum's QR dialog fails fast: "not inside the within element" (750 ms wall, mostly CLI startup)
"Privacy" vs "Privacy Policy" links correct correct (no change; role= was already exact)
Smoke (unlock by ref, tabs, send/back, account list, within, scoped snapshot) 7/11 11/11
describe-screen toolMs p50: home / networks / receive 514 / 4033 / 4023 514 / 4031 / 4027

Generic Chromium cases (built dist, real Chromium; none of these occur on the Extension today): modal <dialog>, inert, open, nested and slotted shadow DOM, aria-owns, and a page whose main world makes CSS, Element, Node, window and Array.from throw. 39/41 refs click the right element. The 2 closed-shadow refs fail on purpose, and none hits a wrong element. On 40 duplicate buttons, snapshot time went from 42 to 33 ms, because lookups now run concurrently.

Pre-existing issues found (not caused by this PR, unchanged on both builds):

  • The home token-row ref never clicks: Chromium exposes a link, but Playwright sees no link because the element has no href.
  • describe-screen takes ~4 s on networks, receive, the account list and About; a fixed timeout is suspected.
Unit tests
Test Files  51 passed (51)
     Tests  1673 passed (1673)
Type Errors no errors
All files | 97.42 | 94.48 | 96.55 | 97.57
yarn lint → exit 0

New tests:

  • AC1: duplicates get distinct unique selectors.
  • AC2: "Send" vs "Send max" are exact.
  • Escaping, checked with a round trip through Playwright's own selector parser; skipped when the internal path is missing.
  • Scoped prefix; text= stays strict; no nth fallback.
  • in-page builder in a vm with a fake DOM: testid, then id, then path; shadow roots and host chaining; slotted :nth-child; closed shadow marker; scuttled globals; CSS.escape parity.
  • callOnNodeInIsolatedWorld: retry, node gone, exceptions.
  • within: ambiguous ref, ref not inside the parent, and the race when the element moves into the parent between checks.

41a404f deflakes the P1-01 queue-wait timing test: under host load it saw 45 ms against a 50 ms threshold, so it now holds the first request for 250 ms.

Ticket P1-11.

Browser a11y refs mapped to `role=<role>[name="<name>"]`, which is not
unique: two "Send" buttons shared one selector, so a click failed with a
strict-mode violation, and with `within` it could hit another row.

- Refs whose role and name are shared with other refs of the snapshot get
  a DOM selector that is verified unique the way Playwright's CSS engine
  matches (unique data-testid, then id, then a path anchored at the
  nearest unique ancestor, nth-of-type or nth-child). Elements in open
  shadow roots get a chained `css=<host> >> css=<path>` selector. Built
  concurrently (at most 16 at a time) in a CDP isolated world, with no
  page globals (CSS.escape inlined), so LavaMoat scuttling cannot break it.
- No index fallback: when no unique DOM selector can be built, the ref
  keeps the role selector, so Playwright's strict mode fails the action
  instead of acting on another element. Refs inside a closed shadow root
  cannot be targeted and fail with a clear error.
- Role selectors use the explicit exact flag `s`. text= fallbacks stay
  strict as well.
- Scoped snapshots prefix role selectors with the root
  (`css=<root> >> nth=0 >> ...`).
- `within` + ref: the ref resolves page-wide and must be inside the
  parent; a ref outside it fails at once with a clear error, and an
  ambiguous ref is rejected instead of being narrowed by the parent.

The refMap stays Map<string, string>; ISessionManager is unchanged.
- Queue wait: the second request could take tens of ms to reach the queue under host load, so queueMs came out below the 50 ms threshold. Hold the first request for 250 ms instead of 60 ms.
- waitForTarget: the timeout passed to waitFor is the time left until the deadline, which can be 4999 ms on a slow runner. Accept (budget - 100, budget].
@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-11-exact-refs branch from 0d117be to 2d1bacb Compare October 28, 2026 11:29
@cryptotavares
cryptotavares merged commit 5935825 into main Oct 29, 2026
27 checks passed
@cryptotavares
cryptotavares deleted the cryptotavares/mmcli-p1-11-exact-refs branch October 29, 2026 13:52
cryptotavares added a commit that referenced this pull request Oct 29, 2026
…refs (#64)

> **Summary:** Cancel timed-out requests cooperatively and fence ref-map
commits against aborted or older requests. Capture mobile trees and test
IDs together, and settle mobile observations after mutations.
>
> **Scope change (2026-10-27):** the ref-stability layer (renumbering
refs so unchanged elements keep their numbers) was removed in `6944868`.
Refs are positional again, as on `main`. See [Why no ref
stability](#why-no-ref-stability).

**Stack context:** mm CLI v2, Phase 1, **P1-08 + P1-09**, combined in
one PR. Base: [#62](#62)
(`cryptotavares/mmcli-p1-06-observe-level-no-state`); followed by
[#65](#65).

## Problem

A queue timeout rejects the caller, but the task can keep running and
overwrite another request's refs. An observation timeout also leaves
collection running.

Mobile observations read test IDs and the tree from separate snapshots.
A click can return the old screen or `Unlock, busy`.

## Solution

Keep cancellation and commit fencing at separate boundaries:

1. **Cooperative cancellation:** `RequestQueue` passes an `AbortSignal`
and aborts it before timeout rejection. Tools check supported
checkpoints. Observations use a linked controller and their own
deadline.
2. **Commit fencing:** the server owns a small `RefStore` (94 lines).
Each started task gets a generation. Every ref write
(`sessionManager.setRefMap` / `clearRefMap` from tools, and the
observation) checks the generation and request signal, even when a tool
ignores cancellation.

```mermaid
flowchart TD
    A[Queued task starts] --> B[Create signal and ref generation]
    B --> C[Run tool through guarded context]
    C --> D[Optional observation checks linked abort signal]
    C -->|ref write| F{Request current and not aborted?}
    D -->|ref write| F
    F -->|yes| G[Commit map]
    F -->|no| X[Drop write. Keep current refs.]
    T[Queue timeout aborts request] -.-> C
    T -.-> D
    T -.-> F

    classDef new fill:#fff3cd,stroke:#d39e00
    classDef stop fill:#f8d7da,stroke:#c82333
    class B,C,D,F,G,T new
    class X stop
```

Yellow boxes are new. Red boxes stop the ref write.

Tools write refs through `context.sessionManager`, which the daemon
proxies: `setRefMap` and `clearRefMap` go through the request handle.
Other members retain the real receiver, including `#private` fields.
`ISessionManager` stays unchanged.

Optional `captureScreen()` derives both mobile outputs from one
snapshot; drivers without it retain separate reads. After eligible
mutations, retry busy screens and allow one unchanged-screen retry,
compared against the last tree the agent received (`lastScreen`: the
delivered observation, else the `describe_screen` / full
`accessibility_snapshot` tree). Launch, cleanup, and non-batch tools
that set refs skip settling.

## What changes

| Condition or boundary | Before | After |
|---|---|---|
| Queued task times out | Remaining work can continue and replace refs |
Abort checkpoints stop cooperative work; the commit fence drops later
ref writes |
| Observation reaches its own timeout | Collection continues after the
response | Abort collection; the final checkpoint prevents its late ref
write |
| Observation not delivered (`observe none`) | Still became the diff
baseline | Not recorded; the next diff is against what the agent last
saw |
| `list_testids` | Reads the tree and replaces refs despite returning no
refs | Collect only test IDs; leave refs unchanged |
| Mobile discovery and observations | Separate snapshots can disagree or
show an unsettled screen | Share a snapshot when supported; bound
post-mutation settle retries |

Queue timeout errors use HTTP 500 with `TOOL_EXECUTION_FAILED` and `Tool
execution timed out after ${effectiveTimeout}ms`. `MM_REQUEST_ABORTED`
is the internal checkpoint response. Cancellation does not undo an
action already dispatched to a driver. Refresh with `describe-screen`
before retrying.

## Decisions for reviewers

| Decision | Reason |
|---|---|
| Keep abort checkpoints and the commit fence | A driver operation can
finish after cancellation. Ref ownership must remain safe independently.
Synchronous abort-handler settlement can win the queue's timeout race. |
| No ref stability; refs stay positional | See [Why no ref
stability](#why-no-ref-stability). |
| The observation's refMap always wins, also after a tool set refs | The
observation is the last tree in the response, so every ref in it
resolves. |
| `lastScreen` updates only after a delivered response | A timed-out
request's tree is never the settle baseline. |

## Review guide

Read the files in this order:

| # | File | What to check |
|---|---|---|
| 1 | `src/server/request-queue.ts`, `src/types/http.ts` |
Abort-before-reject ordering, bounded settle wait, timeout clamp, and
optional context additions. |
| 2 | `src/server/ref-store.ts` | Generation and abort checks;
`toolSetRefs`. |
| 3 | `src/server/create-server.ts` | Proxy receiver binding
(`setRefMap`, `clearRefMap`), observation cancellation checkpoints,
`screenAfterResponse` / `lastScreen`, batch behavior, separate diff
baseline. |
| 4 | `src/tools/utils.ts`, `src/tools/interaction.ts`,
`src/tools/batch.ts`, `src/tools/discovery-tools.ts` | Checkpoints and
`list_testids` tree-read removal. |
| 5 | `src/platform/types.ts`, `src/platform/mobile-platform-driver.ts`,
`src/server/screen-settle.ts` | Optional capture boundary, shared
snapshot, busy detection, and bounded retries. |
| 6 | Tests beside these modules; `README.md`, `SKILL.md` | Timeout
isolation, matching exceptions, snapshot consistency, and caller
recovery guidance. |

<details>
<summary>Testing</summary>

At `6944868`: `yarn test` 52 files / 1528 tests pass; coverage 97.03 /
93.95 / 95.21 / 97.19; `yarn lint` passes. Every branch above it in the
stack (#65, #66, #67, #44, #68) was restacked and passes `yarn test` +
`yarn lint` at its own tip.

</details>

## Breaking changes

None.
- `ToolContext.signal`, `IPlatformDriver.captureScreen` and
`ScreenCapture` are optional additions.
- `RequestQueue.enqueue` now passes the signal to its callback. Existing
callbacks that take no argument keep working.

## Consumer impact

| Consumer | Change needed | Done on branch |
|---|---|---|
| Extension `test/e2e/playwright/llm-workflow/` | None. It does not use
`RequestQueue`, and its `setRefMap`/`getRefMap` implementation is
unchanged. | n/a |
| Mobile `tests/llm-workflow/` | None (same). | n/a |

## Why no ref stability

The first version of this PR also renumbered refs so an element kept its
number across observations (`stabilizeRefs`, a session high-water mark,
baseline checks, merging tool and observation refs: about 500 lines of
code and 900 of tests). It was removed because the problem it solves is
rare:

- Refs are deterministic per tree: the same screen gives the same refs,
and an unchanged screen diffs as empty (device proof below).
- Since P1-05/P1-06, read commands collect no observation, so they no
longer renumber anything.
- The only case it fixed is an element inserted above the one the agent
holds. The response then carries the new refs (the diff lists the
shifted nodes, or the full tree when every node shifted), and SKILL.md
tells agents to use the refs from the latest response.
- No baseline defect (B1–B15) was a wrong-element click caused by
renumbering.

If real runs show agents clicking stale refs, it can come back as its
own ticket.

## Proof

**After the shrink (`6944868`, run on the restacked stack tip):**

| Check | Extension | iOS | Android |
|---|---|---|---|
| Timed-out click, then `describe-screen` (queueMs 0), then click by ref
| pass | pass | pass |
| `describe-screen` twice: identical refs | 17/17 | 67/67 | 242/242 |
| Unchanged screen, 3 taps: diff | 4,243 B, +0 −0 | 4,860 B, +0 −0 |
5,306 B, +0 −0 |
| Navigate away and back, click by the latest ref | pass | pass | pass |

Android navigation observations can still show the previous screen (the
new screen mounts about 2 s later). That is the existing Android settle
limit noted in #44, not changed here.

**Earlier revision (with ref stability), kept for history:**

Tested with yalc builds in the Extension (e2e build, headless) and
Mobile (dev client on iOS sim and Android emulator). The first full run
used an earlier revision (e804b95). The final commit got a targeted
mobile recheck plus a smoke run on all three platforms. Details are in
the proof file in the mm-cli-v2 workspace.

| Check | Before (#62) | After |
|---|---|---|
| `click --testid does-not-exist --timeout 500`, then `describe-screen`,
then click by ref | – | works on ext, iOS and Android (queueMs 0,
intended element clicked) |
| Android unlock-click observation | login screen ("Unlock, busy") in
3/3 | no login or busy in 4/4 (3 wallet screen, 1 tab bar only) |
| Unchanged-screen diff, iOS (`wait-for wallet-screen`) | 7495 B | 7518
B, diff +0 −0 |
| Unchanged-screen diff, Android (`click tab-bar-item-Wallet`, p50 of 3)
| 7672 B, observationMs 1924 | 40–110 KB, observationMs **965–1001**
(one snapshot instead of two) |
| Ext bench p50 (click / get-state / list-testids totalMs) | 580 / 5 /
565 | 582 / 6 / **60** (`list_testids` no longer reads the tree) |

The Android unchanged-screen diff gap in that revision is closed by #67
and #44 (see the table above).
cryptotavares added a commit that referenced this pull request Oct 29, 2026
…e call (#65)

> **Summary:** Reduce browser observation work with a bounded animation
wait and one in-page testId collection call. Run both functions in a
cached CDP isolated world, so Extension page-global restrictions do not
break them.

**Stack context:** mm CLI v2, Phase 1, **P1-10 + P1-12**, combined in
one PR. Base: `cryptotavares/mmcli-p1-08-abort-refs`; review
[#64](#64) first.
Followed by [#66](#66).

## Problem

Browser observations wait for every running animation, including
spinners that never finish. The old main-world check also depends on
globals that LavaMoat can restrict. Hidden tabs do not provide the
animation frames that the check needs.

TestId collection adds three Playwright calls per element:
`getAttribute`, `isVisible`, and `textContent`. Extension home therefore
needs many round trips before the accessibility snapshot starts.

These costs share the observation budget. The historical proof below
records a first Send observation that times out without testIds. This PR
changes the browser collection boundary used by observations and
`PlaywrightPlatformDriver.getTestIds`.

## Solution

Keep browser orchestration in the daemon. Move DOM inspection into
self-contained functions that execute in the main frame's isolated
world.

```mermaid
flowchart TD
    A[Browser observation] --> B{Read-only or discovery tool?}
    B -->|no| C[Wait for finite, in-viewport animations]
    B -->|yes| D[Collect testIds in cached isolated world]
    C --> D
    D -->|success| E[Map items to existing output shape]
    D -->|failure or timeout| F[Collect testIds through locator fallback]
    E --> G[Collect accessibility tree]
    F --> G
    G --> H[Return observation]

    classDef new fill:#fff3cd,stroke:#d39e00
    class C,D,E,F new
```

Yellow boxes are new.

`callInIsolatedWorld` owns the CDP session and execution context. A
`WeakMap` caches them per page. Steady-state calls send one
`Runtime.callFunctionOn` request. Protocol or session failures clear the
matching cache entry and retry once. In-page exceptions propagate
without a retry.

The isolated world shares the DOM but has separate globals. Page
functions travel as source text, so they cannot import helpers or
capture module state. Their helpers stay inside each function. The
source wrapper supplies `__name` for keepNames-transformed functions; VM
tests check the serialization boundary.

`discovery.ts` owns the failure policy. Animation timeouts and
unreadable pages end the wait. TestId call failures and timeouts use the
locator fallback. The server still checks its abort signal between
observation stages.

## What changes

| Condition or boundary | Before | After |
|---|---|---|
| Running animations | Every running animation delays collection | Only
finite animations with non-empty target boxes intersecting the viewport
delay collection |
| Animation wait budget | 3000 ms configured wait | 1000 ms in-page cap,
with 250 ms daemon grace |
| Hidden page | Animation-frame polling only | Poll with a 16 ms timer
while hidden |
| TestId collection | Three locator calls per element | One in-page call
in steady state; locator fallback on failure or a 5 s call timeout |
| TestId output | Items carry testId, tag, text, and visibility | Same
shape, order, limit, and text-preview rules; `tag` stays `element` |
| Page navigation or closed CDP session | No shared isolated-world cache
| Drop the failed entry and retry once; late failures cannot replace a
newer entry |

Collection includes hidden testIds with `visible: false`. Visibility
follows Playwright's rules, including `display: contents`, non-empty
boxes, and computed `visibility`. Zero opacity and off-screen position
do not hide an otherwise visible item.

The collector visits document matches before open shadow-root matches.
It skips empty testIds without consuming the limit. Text remains trimmed
and capped at 100 characters; null text maps to `undefined` at the
daemon boundary.

## Decisions for reviewers

| Decision | Reason |
|---|---|
| Combine **P1-10** and **P1-12** | Both improvements need the same
isolated-world execution boundary. |
| Cache one session and world per page | Avoid creating more execution
contexts on each call to a long-lived Extension page. |
| Keep page functions self-contained and exclude them from coverage
instrumentation | Injected counters or module helpers would break
source-text execution. VM tests exercise that boundary directly. |
| Keep locator fallback and fail-open settling | Observation collection
remains best-effort when CDP or page inspection fails. The 5 s cap
covers the in-page call, not the fallback. |
| Preserve the separate accessibility snapshot | This PR targets
animation settling and testIds. **P1-14** addresses the remaining
snapshot cost identified in the historical proof. |

## Review guide

Read the files in this order:

| # | File | What to check |
|---|---|---|
| 1 | `src/server/create-server.ts` | `snapshotBrowser` keeps
category-based settling, collection order, and abort checkpoints. |
| 2 | `src/tools/utils/constants.ts` | Animation cap, daemon grace, and
testId-call budget. |
| 3 | `src/tools/utils/cdp.ts` | Per-page caching, stale-entry reset
guard, single retry, exception handling, and timeout cleanup. |
| 4 | `src/tools/utils/in-page.ts` | Animation eligibility, hidden-page
polling, visibility semantics, shadow-root order, and source-text
independence. |
| 5 | `src/tools/utils/discovery.ts` | Output mapping, locator fallback,
and animation failure policy. |
| 6 | `src/tools/utils/{cdp,in-page,discovery}.test.ts`,
`src/server/create-server.test.ts` | Cache lifecycle, serialized
functions, fallback, timeout, and observation integration. |

<details>
<summary>Testing</summary>

- Fresh check at remote head `50eb3962b30ee8f5e2605dfcf307417e0282f560`:
`CI=true yarn vitest run src/tools/utils/cdp.test.ts
src/tools/utils/in-page.test.ts src/tools/utils/discovery.test.ts
src/server/create-server.test.ts src/platform/playwright-driver.test.ts
src/tools/discovery-tools.test.ts --coverage.enabled=false
--maxWorkers=2 --minWorkers=1` passes: 6 files, 317 tests.
- `yarn lint` passes. `yarn tsc --project tsconfig.build.json --noEmit`
passes for production sources.
- Standalone `yarn tsc --project tsconfig.json --noEmit` fails at the
remote head with 11 test diagnostics. One is the new
`in-page.test.ts:728` type predicate (`TS2677`); ten come from inherited
test code.
- The server suite emits an expected `EACCES` log from its read-only
log-directory fixture and passes.
- The `Proof` and `Unit tests` sections below retain the original
historical evidence. These fresh checks do not repeat the live browser
benchmarks or full coverage run.

</details>

## Breaking changes

None. The constants are internal. Observations may now be taken while an
infinite or off-screen animation is still running; that is intended.

## Consumer impact

| Consumer | Change needed | Done on branch |
|---|---|---|
| Extension `test/e2e/playwright/llm-workflow/` | None | n/a |
| Mobile `tests/llm-workflow/` | None (browser-only code paths) | n/a |

## Proof

Tested with a yalc build of this branch in the Extension (e2e build,
headless). I ran a fresh before/after bench back to back: before = #64
(7d0c565), after = this branch. Every bench has 5 runs, and the 1-minute
host load was 6–9. After a review round I re-checked the final commit
(50eb396) and got the same numbers.

| Check | Before (#64) | After |
|---|---|---|
| `list-testids` on home, toolMs p50/p95 (44 items) | 53 / 61 | **1 /
1** |
| `list-testids` JSON, home + unlock | – | same items, order, tags and
visibility. Only difference: the local chain port in 3 `text` values,
which also changes between two runs of the same build |
| Click on home (Tokens tab), observationMs p50/p95 | 562 / 571 | 515 /
517 |
| Click on Send page, observationMs p50/p95 | 21 / 32 | 13 / 13 |
| Click on account list, observationMs p50/p95 | 529 / 532 | 513 / 515 |
| First Send click of a session, observationMs | **8003** (timeout, no
testIds) | **43** (11 testIds) |
| Send → back smoke, 5 runs | 5/5, but 4/5 Send opens returned no
testIds | 5/5, all with testIds (3 runs total) |

The remaining ~515 ms on home is the accessibility-tree snapshot:
`get-state --observe full` takes ~509 ms without any settle. The settle
and testIds now add about 6 ms together. P1-14 targets that snapshot.

Known gap: the locator fallback logs only through `debugWarn`, so it is
visible only with `DEBUG=true`. In the recheck it never ran: every
`list_testids` took ~1–2 ms; the fallback takes ~50 ms.

<details>
<summary>Unit tests</summary>

```
Test Files  51 passed (51)
     Tests  1610 passed (1610)
All files | 97.31 | 94.18 | 96.35 | 97.47
yarn lint → exit 0
```

New tests:

- `in-page.test.ts`:
- each function is compiled from its own source text in a bare `vm`
context, so a stray import or closure fails the test;
- a guard checks that `toString()` has no bundler helpers other than
`__name`;
- covers the settle predicate (infinite, off-viewport, no target, hidden
page) and visibility (`display: contents`, `visibility`, zero box),
text, limit and shadow-root order.
- `cdp.test.ts`: session/world caching, retry after navigation, in-page
exceptions, timeout.
- `discovery.test.ts`: the single call, plus fallback to locators on
failure and on timeout.

</details>
cryptotavares added a commit that referenced this pull request Oct 29, 2026
…apshot (#67)

> **Summary:** Remove Android status/navigation bar windows and the Expo
DevLauncher button from mobile screen data. Derive visibility from
snapshot frames and avoid an unchanged-screen retry after a slow first
capture.

**Stack context:** mm CLI v2, Phase 1, **P1-13**. Base:
`cryptotavares/mmcli-p1-11-exact-refs`; stacked on
[#66](#66), which
reviewers should read first. Reuses P1-08's shared `captureScreen` and
refines P1-09's mobile settle policy.

## Problem

Mobile snapshots include elements outside the app. Android's status bar
contributes changing clock, signal, and battery data. The Expo
DevLauncher button also appears in app discovery results. These elements
add noise to accessibility diffs, test IDs, and locator suggestions.

Each test ID read also calls `getWindowSize()`, although snapshot frames
can supply the viewport. The server starts its settle budget after the
first capture, then can retry an unchanged screen despite a slow
capture.

## Solution

Put app filtering and snapshot-derived viewport handling in
`MobilePlatformDriver`, before shared tools consume screen data. Keep
settle decisions in the server.

```mermaid
flowchart TD
    A[One device snapshot] --> B[Read viewport from unfiltered root frames]
    A --> C[Filter Android bar windows and optional dev overlay]
    B --> D[Build tree, refs, test IDs, or locators]
    C --> D
    D --> E[For mutating or batch observations, apply bounded settle]
    E -->|retry allowed| A
    E -->|done| F[Return final observation]

    classDef new fill:#fff3cd,stroke:#d39e00
    class B,C,E new
```

Yellow boxes are new.

`#snapshotApp` supplies one filtered hierarchy to accessibility
normalization, test ID collection, and locator generation.
`captureScreen` retains that hierarchy for its test ID reader. The
existing shared-capture boundary remains intact.

Android filtering requires both a bar-shaped window and at least one
identifier, all from `com.android.systemui`. The frame must span the
viewport width or height and be at most 15% of its own length thick.
This keeps non-bar System UI windows, including biometric prompts. Flat
hierarchies remove only status/navigation bar identifier prefixes.

The default overlay filter removes iOS `gearshape.fill` and its
following `StaticText` label, `Tools`. On Android, it removes a `Tools`
`ImageView` inside a Compose view. Empty, unnamed containers that held
only removed elements disappear too. Filtering leaves the input
unchanged and reuses unchanged subtrees.

## What changes

| Condition or boundary | Before | After |
|---|---|---|
| Mobile tree, refs, test IDs, and generated locators | Include system
bars and the dev button | Use the same app-filtered hierarchy |
| iOS first root frame or Android window extent provides a viewport |
Test ID reads still call `getWindowSize()` | Use snapshot coordinates
without that device call |
| Snapshot cannot provide a viewport | Read the device window size each
time | Cache a successful fallback per driver; retry failed reads |
| Unchanged screen after a first capture of at least 1 second | Can
trigger another capture | Skip the unchanged-screen retry |
| Busy screen after a slow first capture | Retry under existing limits |
Still retry; the first capture now counts against the wait budget |
| Observation or discovery read | P1-08 already shares the capture |
Existing tests lock in one snapshot per capture, before settle retries |

## Decisions for reviewers

| Decision | Reason |
|---|---|
| Filter at the mobile driver boundary | Keeps all discovery outputs
consistent without tool-specific exclusions. Browser behavior stays
outside this change. |
| Require bar geometry and System UI ownership | Avoids removing a
biometric prompt or an app window that contains other identifiers. |
| Export optional `MobilePlatformDriverOptions` | `excludeDevOverlay:
false` keeps the dev button. Android bar filtering still applies. |
| Cache only the fallback window size | Snapshot frames reflect the
current orientation. The fallback cache does not refresh after rotation.
|
| Preserve busy-screen retries after slow captures | A slow capture does
not establish that an in-progress action has completed. |
| Keep backend tap lookup outside this change | **P1-13** does not
remove backend tap snapshots. The historical proof below records the
unmet Android latency target. |

## Review guide

Read the files in this order:

| # | File | What to check |
|---|---|---|
| 1 | `src/platform/mobile-hierarchy.ts` | Bar ownership and geometry,
overlay matching, subtree reuse, and viewport derivation from unfiltered
roots. |
| 2 | `src/platform/mobile-platform-driver.ts` | `#snapshotApp`, shared
consumers, fallback caching, and the optional constructor argument. |
| 3 | `src/server/create-server.ts`, `src/tools/utils/constants.ts` |
`snapshotMobile`: first-capture budget, slow unchanged capture, and
busy-screen precedence. |
| 4 | `src/platform/index.ts`, `src/index.ts` | Public
`MobilePlatformDriverOptions` type exports. |
| 5 | `src/platform/mobile-hierarchy.test.ts`,
`src/platform/mobile-platform-driver.test.ts`,
`src/server/create-server.test.ts` | Real Android fixture, preserved
system dialogs, output consistency, snapshot counts, and settle timing.
|
| 6 | `README.md`, `SKILL.md` | Filtering rules, single-capture
contract, and fallback orientation limitation. |

<details>
<summary>Testing</summary>

- Fresh check at remote head `6136dc6e39398cc06ce2a3dcc4a0eda4280f6812`:
`CI=true yarn vitest run src/platform/mobile-hierarchy.test.ts
src/platform/mobile-platform-driver.test.ts
src/server/create-server.test.ts src/tools/discovery-tools.test.ts
--coverage.enabled=false --maxWorkers=2 --minWorkers=1` passes: 4 files,
301 tests. Vitest reports no type errors.
- `yarn lint` passes at that remote head.
- The independent `yarn tsc --project tsconfig.test.json --noEmit` check
reports 11 existing test-project diagnostics. A compiler-API comparison
confirms the same diagnostics at the remote head.
- The existing tests cover app filtering, viewport fallback failures,
subtree identity, shared snapshots, and slow/busy settle cases.
- The `Proof` and `Unit tests` sections below retain historical
evidence. These fresh checks do not repeat live device validation or the
full suite.

</details>

## Breaking changes

None. The constructor option is optional. Mobile testIds and trees no
longer contain Android status/nav bar windows or the Expo dev menu
button.

## Consumer impact

| Consumer | Change needed | Done on branch |
|---|---|---|
| Extension `test/e2e/playwright/llm-workflow/` | None (mobile only) |
n/a |
| Mobile `tests/llm-workflow/` | None. Its agent-device snapshot output
on Android (pixel frames, full resource-ids) is handled. | n/a |

## Proof

Tested with yalc builds in Mobile (dev client) on the iOS simulator and
the Android emulator. Before = #66 build, after = this branch. Click and
benchmark rows are 5 runs each; host load was 5–14.

| Check | Before (#66) | After (3cc8aa9) |
|---|---|---|
| Android login testIds / `com.android.systemui` | 35 / 22 | **13 / 0**
|
| Android wallet `describe-screen` root windows / systemui nodes | 2 /
22 | 1 / 0 |
| Android `generate-locators` on login | – | 13 locators, 0 systemui, 0
dev menu |
| Android click `observationMs` p50 / p95 | 963 / 1048 | 891 / 914 (−7 %
/ −13 %) |
| Android unchanged-screen click: empty diffs | 0 of 5 | 1–2 of 5 (7.8
KB) |
| iOS login testIds | 10 (incl. `gearshape.fill`) | 9 |
| iOS click `observationMs` p50 / p95 | 1170 / 1232 | **972 / 1005**
(−17 %) |
| iOS `list-testids` / `describe-screen` toolMs p50 | 1162 / 1516 |
**895 / 1284** |
| iOS `idb describe` (window size) per click / describe / list | 5 / 2 /
2 | 4 / 1 / 1 |
| Device snapshots per click / describe / list (both platforms) | 3 / 1
/ 1 | 3 / 1 / 1 |

**AC3 ("Android click `observationMs` roughly halves") is not met, and
core cannot meet it.**
- Halving assumed 2 snapshots per observation. P1-08 already brought
that down to 1.
- Each click takes 3 device snapshots: 2 inside the backend tap
(`waitForElement` + `tapElement`, in device-mcp / the Mobile
agent-device helper) and 1 for the observation.
- Removing the tap's snapshots needs a change in device-mcp or the
consumer, for example tapping at coordinates already in the last
observation. That is listed as a follow-up.

**Still noisy on Android:**
- Some unchanged-screen diffs are still 30–135 KB, because the app keeps
mounting zero-height `ViewGroup`s. P1-14 (drop layout-only nodes) covers
this.
- On iOS, 2–4 full-screen `Toolbar` / `FloatingBarHostingView` nodes
remain. They are most likely UIKit's floating bar per navigation
controller, not the dev launcher. They are kept for now, pending a check
on a release build.

<details>
<summary>Unit tests</summary>

```
Test Files  52 passed (52)
     Tests  1723 passed (1723)
Type Errors no errors
All files | 97.49 | 94.79 | 96.69 | 97.63
yarn lint → exit 0
```

New tests:

- `mobile-hierarchy.test.ts`:
  - bar-window rule, plus a biometric prompt window that is kept;
- flat-hierarchy fallback, dev overlay and the `excludeDevOverlay`
option, empty-container pruning;
- viewport from the root frame or Android extent, including an
off-origin window and a status-bar-only snapshot;
- a real Android login snapshot fixture (`src/platform/__fixtures__/`,
excluded from the build), which reduces to 1 root and 0 systemui
testIds.
- Driver: 1 snapshot per call, `getWindowSize` at most once,
`generateLocators` filtered.
- Server: 1 snapshot per observation; the settle budget includes the
first capture; slow first capture → no unchanged-screen retry, busy
screens still retry.

</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants