Repository navigation
fix: make each Extension ref resolve to exactly one element - #66
Merged
cryptotavares merged 3 commits intoOct 29, 2026
Conversation
cryptotavares
force-pushed
the
cryptotavares/mmcli-p1-11-exact-refs
branch
from
October 25, 2026 09:46
41a404f to
e93a529
Compare
cryptotavares
force-pushed
the
cryptotavares/mmcli-p1-11-exact-refs
branch
from
October 27, 2026 15:25
91e888f to
0d117be
Compare
cryptotavares
marked this pull request as ready for review
October 27, 2026 18:07
Ticket P1-11. Browser a11y refs mapped to `role=<role>[name="<name>"]`, which is not unique: two "Send" buttons shared one selector, so a click failed with a strict-mode violation, and with `within` it could hit another row. - Refs whose role and name are shared with other refs of the snapshot get a DOM selector that is verified unique the way Playwright's CSS engine matches (unique data-testid, then id, then a path anchored at the nearest unique ancestor, nth-of-type or nth-child). Elements in open shadow roots get a chained `css=<host> >> css=<path>` selector. Built concurrently (at most 16 at a time) in a CDP isolated world, with no page globals (CSS.escape inlined), so LavaMoat scuttling cannot break it. - No index fallback: when no unique DOM selector can be built, the ref keeps the role selector, so Playwright's strict mode fails the action instead of acting on another element. Refs inside a closed shadow root cannot be targeted and fail with a clear error. - Role selectors use the explicit exact flag `s`. text= fallbacks stay strict as well. - Scoped snapshots prefix role selectors with the root (`css=<root> >> nth=0 >> ...`). - `within` + ref: the ref resolves page-wide and must be inside the parent; a ref outside it fails at once with a clear error, and an ambiguous ref is rejected instead of being narrowed by the parent. The refMap stays Map<string, string>; ISessionManager is unchanged.
- Queue wait: the second request could take tens of ms to reach the queue under host load, so queueMs came out below the 50 ms threshold. Hold the first request for 250 ms instead of 60 ms. - waitForTarget: the timeout passed to waitFor is the time left until the deadline, which can be 4999 ms on a slow runner. Accept (budget - 100, budget].
cryptotavares
force-pushed
the
cryptotavares/mmcli-p1-11-exact-refs
branch
from
October 28, 2026 11:29
0d117be to
2d1bacb
Compare
itsyoboieltr
approved these changes
Oct 28, 2026
seaona
approved these changes
Oct 29, 2026
cryptotavares
added a commit
that referenced
this pull request
Oct 29, 2026
…refs (#64) > **Summary:** Cancel timed-out requests cooperatively and fence ref-map commits against aborted or older requests. Capture mobile trees and test IDs together, and settle mobile observations after mutations. > > **Scope change (2026-10-27):** the ref-stability layer (renumbering refs so unchanged elements keep their numbers) was removed in `6944868`. Refs are positional again, as on `main`. See [Why no ref stability](#why-no-ref-stability). **Stack context:** mm CLI v2, Phase 1, **P1-08 + P1-09**, combined in one PR. Base: [#62](#62) (`cryptotavares/mmcli-p1-06-observe-level-no-state`); followed by [#65](#65). ## Problem A queue timeout rejects the caller, but the task can keep running and overwrite another request's refs. An observation timeout also leaves collection running. Mobile observations read test IDs and the tree from separate snapshots. A click can return the old screen or `Unlock, busy`. ## Solution Keep cancellation and commit fencing at separate boundaries: 1. **Cooperative cancellation:** `RequestQueue` passes an `AbortSignal` and aborts it before timeout rejection. Tools check supported checkpoints. Observations use a linked controller and their own deadline. 2. **Commit fencing:** the server owns a small `RefStore` (94 lines). Each started task gets a generation. Every ref write (`sessionManager.setRefMap` / `clearRefMap` from tools, and the observation) checks the generation and request signal, even when a tool ignores cancellation. ```mermaid flowchart TD A[Queued task starts] --> B[Create signal and ref generation] B --> C[Run tool through guarded context] C --> D[Optional observation checks linked abort signal] C -->|ref write| F{Request current and not aborted?} D -->|ref write| F F -->|yes| G[Commit map] F -->|no| X[Drop write. Keep current refs.] T[Queue timeout aborts request] -.-> C T -.-> D T -.-> F classDef new fill:#fff3cd,stroke:#d39e00 classDef stop fill:#f8d7da,stroke:#c82333 class B,C,D,F,G,T new class X stop ``` Yellow boxes are new. Red boxes stop the ref write. Tools write refs through `context.sessionManager`, which the daemon proxies: `setRefMap` and `clearRefMap` go through the request handle. Other members retain the real receiver, including `#private` fields. `ISessionManager` stays unchanged. Optional `captureScreen()` derives both mobile outputs from one snapshot; drivers without it retain separate reads. After eligible mutations, retry busy screens and allow one unchanged-screen retry, compared against the last tree the agent received (`lastScreen`: the delivered observation, else the `describe_screen` / full `accessibility_snapshot` tree). Launch, cleanup, and non-batch tools that set refs skip settling. ## What changes | Condition or boundary | Before | After | |---|---|---| | Queued task times out | Remaining work can continue and replace refs | Abort checkpoints stop cooperative work; the commit fence drops later ref writes | | Observation reaches its own timeout | Collection continues after the response | Abort collection; the final checkpoint prevents its late ref write | | Observation not delivered (`observe none`) | Still became the diff baseline | Not recorded; the next diff is against what the agent last saw | | `list_testids` | Reads the tree and replaces refs despite returning no refs | Collect only test IDs; leave refs unchanged | | Mobile discovery and observations | Separate snapshots can disagree or show an unsettled screen | Share a snapshot when supported; bound post-mutation settle retries | Queue timeout errors use HTTP 500 with `TOOL_EXECUTION_FAILED` and `Tool execution timed out after ${effectiveTimeout}ms`. `MM_REQUEST_ABORTED` is the internal checkpoint response. Cancellation does not undo an action already dispatched to a driver. Refresh with `describe-screen` before retrying. ## Decisions for reviewers | Decision | Reason | |---|---| | Keep abort checkpoints and the commit fence | A driver operation can finish after cancellation. Ref ownership must remain safe independently. Synchronous abort-handler settlement can win the queue's timeout race. | | No ref stability; refs stay positional | See [Why no ref stability](#why-no-ref-stability). | | The observation's refMap always wins, also after a tool set refs | The observation is the last tree in the response, so every ref in it resolves. | | `lastScreen` updates only after a delivered response | A timed-out request's tree is never the settle baseline. | ## Review guide Read the files in this order: | # | File | What to check | |---|---|---| | 1 | `src/server/request-queue.ts`, `src/types/http.ts` | Abort-before-reject ordering, bounded settle wait, timeout clamp, and optional context additions. | | 2 | `src/server/ref-store.ts` | Generation and abort checks; `toolSetRefs`. | | 3 | `src/server/create-server.ts` | Proxy receiver binding (`setRefMap`, `clearRefMap`), observation cancellation checkpoints, `screenAfterResponse` / `lastScreen`, batch behavior, separate diff baseline. | | 4 | `src/tools/utils.ts`, `src/tools/interaction.ts`, `src/tools/batch.ts`, `src/tools/discovery-tools.ts` | Checkpoints and `list_testids` tree-read removal. | | 5 | `src/platform/types.ts`, `src/platform/mobile-platform-driver.ts`, `src/server/screen-settle.ts` | Optional capture boundary, shared snapshot, busy detection, and bounded retries. | | 6 | Tests beside these modules; `README.md`, `SKILL.md` | Timeout isolation, matching exceptions, snapshot consistency, and caller recovery guidance. | <details> <summary>Testing</summary> At `6944868`: `yarn test` 52 files / 1528 tests pass; coverage 97.03 / 93.95 / 95.21 / 97.19; `yarn lint` passes. Every branch above it in the stack (#65, #66, #67, #44, #68) was restacked and passes `yarn test` + `yarn lint` at its own tip. </details> ## Breaking changes None. - `ToolContext.signal`, `IPlatformDriver.captureScreen` and `ScreenCapture` are optional additions. - `RequestQueue.enqueue` now passes the signal to its callback. Existing callbacks that take no argument keep working. ## Consumer impact | Consumer | Change needed | Done on branch | |---|---|---| | Extension `test/e2e/playwright/llm-workflow/` | None. It does not use `RequestQueue`, and its `setRefMap`/`getRefMap` implementation is unchanged. | n/a | | Mobile `tests/llm-workflow/` | None (same). | n/a | ## Why no ref stability The first version of this PR also renumbered refs so an element kept its number across observations (`stabilizeRefs`, a session high-water mark, baseline checks, merging tool and observation refs: about 500 lines of code and 900 of tests). It was removed because the problem it solves is rare: - Refs are deterministic per tree: the same screen gives the same refs, and an unchanged screen diffs as empty (device proof below). - Since P1-05/P1-06, read commands collect no observation, so they no longer renumber anything. - The only case it fixed is an element inserted above the one the agent holds. The response then carries the new refs (the diff lists the shifted nodes, or the full tree when every node shifted), and SKILL.md tells agents to use the refs from the latest response. - No baseline defect (B1–B15) was a wrong-element click caused by renumbering. If real runs show agents clicking stale refs, it can come back as its own ticket. ## Proof **After the shrink (`6944868`, run on the restacked stack tip):** | Check | Extension | iOS | Android | |---|---|---|---| | Timed-out click, then `describe-screen` (queueMs 0), then click by ref | pass | pass | pass | | `describe-screen` twice: identical refs | 17/17 | 67/67 | 242/242 | | Unchanged screen, 3 taps: diff | 4,243 B, +0 −0 | 4,860 B, +0 −0 | 5,306 B, +0 −0 | | Navigate away and back, click by the latest ref | pass | pass | pass | Android navigation observations can still show the previous screen (the new screen mounts about 2 s later). That is the existing Android settle limit noted in #44, not changed here. **Earlier revision (with ref stability), kept for history:** Tested with yalc builds in the Extension (e2e build, headless) and Mobile (dev client on iOS sim and Android emulator). The first full run used an earlier revision (e804b95). The final commit got a targeted mobile recheck plus a smoke run on all three platforms. Details are in the proof file in the mm-cli-v2 workspace. | Check | Before (#62) | After | |---|---|---| | `click --testid does-not-exist --timeout 500`, then `describe-screen`, then click by ref | – | works on ext, iOS and Android (queueMs 0, intended element clicked) | | Android unlock-click observation | login screen ("Unlock, busy") in 3/3 | no login or busy in 4/4 (3 wallet screen, 1 tab bar only) | | Unchanged-screen diff, iOS (`wait-for wallet-screen`) | 7495 B | 7518 B, diff +0 −0 | | Unchanged-screen diff, Android (`click tab-bar-item-Wallet`, p50 of 3) | 7672 B, observationMs 1924 | 40–110 KB, observationMs **965–1001** (one snapshot instead of two) | | Ext bench p50 (click / get-state / list-testids totalMs) | 580 / 5 / 565 | 582 / 6 / **60** (`list_testids` no longer reads the tree) | The Android unchanged-screen diff gap in that revision is closed by #67 and #44 (see the table above).
cryptotavares
added a commit
that referenced
this pull request
Oct 29, 2026
…e call (#65) > **Summary:** Reduce browser observation work with a bounded animation wait and one in-page testId collection call. Run both functions in a cached CDP isolated world, so Extension page-global restrictions do not break them. **Stack context:** mm CLI v2, Phase 1, **P1-10 + P1-12**, combined in one PR. Base: `cryptotavares/mmcli-p1-08-abort-refs`; review [#64](#64) first. Followed by [#66](#66). ## Problem Browser observations wait for every running animation, including spinners that never finish. The old main-world check also depends on globals that LavaMoat can restrict. Hidden tabs do not provide the animation frames that the check needs. TestId collection adds three Playwright calls per element: `getAttribute`, `isVisible`, and `textContent`. Extension home therefore needs many round trips before the accessibility snapshot starts. These costs share the observation budget. The historical proof below records a first Send observation that times out without testIds. This PR changes the browser collection boundary used by observations and `PlaywrightPlatformDriver.getTestIds`. ## Solution Keep browser orchestration in the daemon. Move DOM inspection into self-contained functions that execute in the main frame's isolated world. ```mermaid flowchart TD A[Browser observation] --> B{Read-only or discovery tool?} B -->|no| C[Wait for finite, in-viewport animations] B -->|yes| D[Collect testIds in cached isolated world] C --> D D -->|success| E[Map items to existing output shape] D -->|failure or timeout| F[Collect testIds through locator fallback] E --> G[Collect accessibility tree] F --> G G --> H[Return observation] classDef new fill:#fff3cd,stroke:#d39e00 class C,D,E,F new ``` Yellow boxes are new. `callInIsolatedWorld` owns the CDP session and execution context. A `WeakMap` caches them per page. Steady-state calls send one `Runtime.callFunctionOn` request. Protocol or session failures clear the matching cache entry and retry once. In-page exceptions propagate without a retry. The isolated world shares the DOM but has separate globals. Page functions travel as source text, so they cannot import helpers or capture module state. Their helpers stay inside each function. The source wrapper supplies `__name` for keepNames-transformed functions; VM tests check the serialization boundary. `discovery.ts` owns the failure policy. Animation timeouts and unreadable pages end the wait. TestId call failures and timeouts use the locator fallback. The server still checks its abort signal between observation stages. ## What changes | Condition or boundary | Before | After | |---|---|---| | Running animations | Every running animation delays collection | Only finite animations with non-empty target boxes intersecting the viewport delay collection | | Animation wait budget | 3000 ms configured wait | 1000 ms in-page cap, with 250 ms daemon grace | | Hidden page | Animation-frame polling only | Poll with a 16 ms timer while hidden | | TestId collection | Three locator calls per element | One in-page call in steady state; locator fallback on failure or a 5 s call timeout | | TestId output | Items carry testId, tag, text, and visibility | Same shape, order, limit, and text-preview rules; `tag` stays `element` | | Page navigation or closed CDP session | No shared isolated-world cache | Drop the failed entry and retry once; late failures cannot replace a newer entry | Collection includes hidden testIds with `visible: false`. Visibility follows Playwright's rules, including `display: contents`, non-empty boxes, and computed `visibility`. Zero opacity and off-screen position do not hide an otherwise visible item. The collector visits document matches before open shadow-root matches. It skips empty testIds without consuming the limit. Text remains trimmed and capped at 100 characters; null text maps to `undefined` at the daemon boundary. ## Decisions for reviewers | Decision | Reason | |---|---| | Combine **P1-10** and **P1-12** | Both improvements need the same isolated-world execution boundary. | | Cache one session and world per page | Avoid creating more execution contexts on each call to a long-lived Extension page. | | Keep page functions self-contained and exclude them from coverage instrumentation | Injected counters or module helpers would break source-text execution. VM tests exercise that boundary directly. | | Keep locator fallback and fail-open settling | Observation collection remains best-effort when CDP or page inspection fails. The 5 s cap covers the in-page call, not the fallback. | | Preserve the separate accessibility snapshot | This PR targets animation settling and testIds. **P1-14** addresses the remaining snapshot cost identified in the historical proof. | ## Review guide Read the files in this order: | # | File | What to check | |---|---|---| | 1 | `src/server/create-server.ts` | `snapshotBrowser` keeps category-based settling, collection order, and abort checkpoints. | | 2 | `src/tools/utils/constants.ts` | Animation cap, daemon grace, and testId-call budget. | | 3 | `src/tools/utils/cdp.ts` | Per-page caching, stale-entry reset guard, single retry, exception handling, and timeout cleanup. | | 4 | `src/tools/utils/in-page.ts` | Animation eligibility, hidden-page polling, visibility semantics, shadow-root order, and source-text independence. | | 5 | `src/tools/utils/discovery.ts` | Output mapping, locator fallback, and animation failure policy. | | 6 | `src/tools/utils/{cdp,in-page,discovery}.test.ts`, `src/server/create-server.test.ts` | Cache lifecycle, serialized functions, fallback, timeout, and observation integration. | <details> <summary>Testing</summary> - Fresh check at remote head `50eb3962b30ee8f5e2605dfcf307417e0282f560`: `CI=true yarn vitest run src/tools/utils/cdp.test.ts src/tools/utils/in-page.test.ts src/tools/utils/discovery.test.ts src/server/create-server.test.ts src/platform/playwright-driver.test.ts src/tools/discovery-tools.test.ts --coverage.enabled=false --maxWorkers=2 --minWorkers=1` passes: 6 files, 317 tests. - `yarn lint` passes. `yarn tsc --project tsconfig.build.json --noEmit` passes for production sources. - Standalone `yarn tsc --project tsconfig.json --noEmit` fails at the remote head with 11 test diagnostics. One is the new `in-page.test.ts:728` type predicate (`TS2677`); ten come from inherited test code. - The server suite emits an expected `EACCES` log from its read-only log-directory fixture and passes. - The `Proof` and `Unit tests` sections below retain the original historical evidence. These fresh checks do not repeat the live browser benchmarks or full coverage run. </details> ## Breaking changes None. The constants are internal. Observations may now be taken while an infinite or off-screen animation is still running; that is intended. ## Consumer impact | Consumer | Change needed | Done on branch | |---|---|---| | Extension `test/e2e/playwright/llm-workflow/` | None | n/a | | Mobile `tests/llm-workflow/` | None (browser-only code paths) | n/a | ## Proof Tested with a yalc build of this branch in the Extension (e2e build, headless). I ran a fresh before/after bench back to back: before = #64 (7d0c565), after = this branch. Every bench has 5 runs, and the 1-minute host load was 6–9. After a review round I re-checked the final commit (50eb396) and got the same numbers. | Check | Before (#64) | After | |---|---|---| | `list-testids` on home, toolMs p50/p95 (44 items) | 53 / 61 | **1 / 1** | | `list-testids` JSON, home + unlock | – | same items, order, tags and visibility. Only difference: the local chain port in 3 `text` values, which also changes between two runs of the same build | | Click on home (Tokens tab), observationMs p50/p95 | 562 / 571 | 515 / 517 | | Click on Send page, observationMs p50/p95 | 21 / 32 | 13 / 13 | | Click on account list, observationMs p50/p95 | 529 / 532 | 513 / 515 | | First Send click of a session, observationMs | **8003** (timeout, no testIds) | **43** (11 testIds) | | Send → back smoke, 5 runs | 5/5, but 4/5 Send opens returned no testIds | 5/5, all with testIds (3 runs total) | The remaining ~515 ms on home is the accessibility-tree snapshot: `get-state --observe full` takes ~509 ms without any settle. The settle and testIds now add about 6 ms together. P1-14 targets that snapshot. Known gap: the locator fallback logs only through `debugWarn`, so it is visible only with `DEBUG=true`. In the recheck it never ran: every `list_testids` took ~1–2 ms; the fallback takes ~50 ms. <details> <summary>Unit tests</summary> ``` Test Files 51 passed (51) Tests 1610 passed (1610) All files | 97.31 | 94.18 | 96.35 | 97.47 yarn lint → exit 0 ``` New tests: - `in-page.test.ts`: - each function is compiled from its own source text in a bare `vm` context, so a stray import or closure fails the test; - a guard checks that `toString()` has no bundler helpers other than `__name`; - covers the settle predicate (infinite, off-viewport, no target, hidden page) and visibility (`display: contents`, `visibility`, zero box), text, limit and shadow-root order. - `cdp.test.ts`: session/world caching, retry after navigation, in-page exceptions, timeout. - `discovery.test.ts`: the single call, plus fallback to locators on failure and on timeout. </details>
cryptotavares
added a commit
that referenced
this pull request
Oct 29, 2026
…apshot (#67) > **Summary:** Remove Android status/navigation bar windows and the Expo DevLauncher button from mobile screen data. Derive visibility from snapshot frames and avoid an unchanged-screen retry after a slow first capture. **Stack context:** mm CLI v2, Phase 1, **P1-13**. Base: `cryptotavares/mmcli-p1-11-exact-refs`; stacked on [#66](#66), which reviewers should read first. Reuses P1-08's shared `captureScreen` and refines P1-09's mobile settle policy. ## Problem Mobile snapshots include elements outside the app. Android's status bar contributes changing clock, signal, and battery data. The Expo DevLauncher button also appears in app discovery results. These elements add noise to accessibility diffs, test IDs, and locator suggestions. Each test ID read also calls `getWindowSize()`, although snapshot frames can supply the viewport. The server starts its settle budget after the first capture, then can retry an unchanged screen despite a slow capture. ## Solution Put app filtering and snapshot-derived viewport handling in `MobilePlatformDriver`, before shared tools consume screen data. Keep settle decisions in the server. ```mermaid flowchart TD A[One device snapshot] --> B[Read viewport from unfiltered root frames] A --> C[Filter Android bar windows and optional dev overlay] B --> D[Build tree, refs, test IDs, or locators] C --> D D --> E[For mutating or batch observations, apply bounded settle] E -->|retry allowed| A E -->|done| F[Return final observation] classDef new fill:#fff3cd,stroke:#d39e00 class B,C,E new ``` Yellow boxes are new. `#snapshotApp` supplies one filtered hierarchy to accessibility normalization, test ID collection, and locator generation. `captureScreen` retains that hierarchy for its test ID reader. The existing shared-capture boundary remains intact. Android filtering requires both a bar-shaped window and at least one identifier, all from `com.android.systemui`. The frame must span the viewport width or height and be at most 15% of its own length thick. This keeps non-bar System UI windows, including biometric prompts. Flat hierarchies remove only status/navigation bar identifier prefixes. The default overlay filter removes iOS `gearshape.fill` and its following `StaticText` label, `Tools`. On Android, it removes a `Tools` `ImageView` inside a Compose view. Empty, unnamed containers that held only removed elements disappear too. Filtering leaves the input unchanged and reuses unchanged subtrees. ## What changes | Condition or boundary | Before | After | |---|---|---| | Mobile tree, refs, test IDs, and generated locators | Include system bars and the dev button | Use the same app-filtered hierarchy | | iOS first root frame or Android window extent provides a viewport | Test ID reads still call `getWindowSize()` | Use snapshot coordinates without that device call | | Snapshot cannot provide a viewport | Read the device window size each time | Cache a successful fallback per driver; retry failed reads | | Unchanged screen after a first capture of at least 1 second | Can trigger another capture | Skip the unchanged-screen retry | | Busy screen after a slow first capture | Retry under existing limits | Still retry; the first capture now counts against the wait budget | | Observation or discovery read | P1-08 already shares the capture | Existing tests lock in one snapshot per capture, before settle retries | ## Decisions for reviewers | Decision | Reason | |---|---| | Filter at the mobile driver boundary | Keeps all discovery outputs consistent without tool-specific exclusions. Browser behavior stays outside this change. | | Require bar geometry and System UI ownership | Avoids removing a biometric prompt or an app window that contains other identifiers. | | Export optional `MobilePlatformDriverOptions` | `excludeDevOverlay: false` keeps the dev button. Android bar filtering still applies. | | Cache only the fallback window size | Snapshot frames reflect the current orientation. The fallback cache does not refresh after rotation. | | Preserve busy-screen retries after slow captures | A slow capture does not establish that an in-progress action has completed. | | Keep backend tap lookup outside this change | **P1-13** does not remove backend tap snapshots. The historical proof below records the unmet Android latency target. | ## Review guide Read the files in this order: | # | File | What to check | |---|---|---| | 1 | `src/platform/mobile-hierarchy.ts` | Bar ownership and geometry, overlay matching, subtree reuse, and viewport derivation from unfiltered roots. | | 2 | `src/platform/mobile-platform-driver.ts` | `#snapshotApp`, shared consumers, fallback caching, and the optional constructor argument. | | 3 | `src/server/create-server.ts`, `src/tools/utils/constants.ts` | `snapshotMobile`: first-capture budget, slow unchanged capture, and busy-screen precedence. | | 4 | `src/platform/index.ts`, `src/index.ts` | Public `MobilePlatformDriverOptions` type exports. | | 5 | `src/platform/mobile-hierarchy.test.ts`, `src/platform/mobile-platform-driver.test.ts`, `src/server/create-server.test.ts` | Real Android fixture, preserved system dialogs, output consistency, snapshot counts, and settle timing. | | 6 | `README.md`, `SKILL.md` | Filtering rules, single-capture contract, and fallback orientation limitation. | <details> <summary>Testing</summary> - Fresh check at remote head `6136dc6e39398cc06ce2a3dcc4a0eda4280f6812`: `CI=true yarn vitest run src/platform/mobile-hierarchy.test.ts src/platform/mobile-platform-driver.test.ts src/server/create-server.test.ts src/tools/discovery-tools.test.ts --coverage.enabled=false --maxWorkers=2 --minWorkers=1` passes: 4 files, 301 tests. Vitest reports no type errors. - `yarn lint` passes at that remote head. - The independent `yarn tsc --project tsconfig.test.json --noEmit` check reports 11 existing test-project diagnostics. A compiler-API comparison confirms the same diagnostics at the remote head. - The existing tests cover app filtering, viewport fallback failures, subtree identity, shared snapshots, and slow/busy settle cases. - The `Proof` and `Unit tests` sections below retain historical evidence. These fresh checks do not repeat live device validation or the full suite. </details> ## Breaking changes None. The constructor option is optional. Mobile testIds and trees no longer contain Android status/nav bar windows or the Expo dev menu button. ## Consumer impact | Consumer | Change needed | Done on branch | |---|---|---| | Extension `test/e2e/playwright/llm-workflow/` | None (mobile only) | n/a | | Mobile `tests/llm-workflow/` | None. Its agent-device snapshot output on Android (pixel frames, full resource-ids) is handled. | n/a | ## Proof Tested with yalc builds in Mobile (dev client) on the iOS simulator and the Android emulator. Before = #66 build, after = this branch. Click and benchmark rows are 5 runs each; host load was 5–14. | Check | Before (#66) | After (3cc8aa9) | |---|---|---| | Android login testIds / `com.android.systemui` | 35 / 22 | **13 / 0** | | Android wallet `describe-screen` root windows / systemui nodes | 2 / 22 | 1 / 0 | | Android `generate-locators` on login | – | 13 locators, 0 systemui, 0 dev menu | | Android click `observationMs` p50 / p95 | 963 / 1048 | 891 / 914 (−7 % / −13 %) | | Android unchanged-screen click: empty diffs | 0 of 5 | 1–2 of 5 (7.8 KB) | | iOS login testIds | 10 (incl. `gearshape.fill`) | 9 | | iOS click `observationMs` p50 / p95 | 1170 / 1232 | **972 / 1005** (−17 %) | | iOS `list-testids` / `describe-screen` toolMs p50 | 1162 / 1516 | **895 / 1284** | | iOS `idb describe` (window size) per click / describe / list | 5 / 2 / 2 | 4 / 1 / 1 | | Device snapshots per click / describe / list (both platforms) | 3 / 1 / 1 | 3 / 1 / 1 | **AC3 ("Android click `observationMs` roughly halves") is not met, and core cannot meet it.** - Halving assumed 2 snapshots per observation. P1-08 already brought that down to 1. - Each click takes 3 device snapshots: 2 inside the backend tap (`waitForElement` + `tapElement`, in device-mcp / the Mobile agent-device helper) and 1 for the observation. - Removing the tap's snapshots needs a change in device-mcp or the consumer, for example tapping at coordinates already in the last observation. That is listed as a follow-up. **Still noisy on Android:** - Some unchanged-screen diffs are still 30–135 KB, because the app keeps mounting zero-height `ViewGroup`s. P1-14 (drop layout-only nodes) covers this. - On iOS, 2–4 full-screen `Toolbar` / `FloatingBarHostingView` nodes remain. They are most likely UIKit's floating bar per navigation controller, not the dev launcher. They are kept for now, pending a check on a release build. <details> <summary>Unit tests</summary> ``` Test Files 52 passed (52) Tests 1723 passed (1723) Type Errors no errors All files | 97.49 | 94.79 | 96.69 | 97.63 yarn lint → exit 0 ``` New tests: - `mobile-hierarchy.test.ts`: - bar-window rule, plus a biometric prompt window that is kept; - flat-hierarchy fallback, dev overlay and the `excludeDevOverlay` option, empty-container pruning; - viewport from the root frame or Android extent, including an off-origin window and a status-bar-only snapshot; - a real Android login snapshot fixture (`src/platform/__fixtures__/`, excluded from the build), which reduces to 1 root and 0 systemui testIds. - Driver: 1 snapshot per call, `getWindowSize` at most once, `generateLocators` filtered. - Server: 1 snapshot per observation; the settle budget includes the first capture; slow first capture → no unchanged-screen retry, busy screens still retry. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack context: mm CLI v2, Phase 1, P1-11. Base: #65 (
cryptotavares/mmcli-p1-10-ext-speedups); followed by #67. Review #65 first.Problem
Browser refs use role-and-name selectors. Two elements with the same role and name therefore share one selector. A click fails in Playwright strict mode. With
--within, the parent can narrow that selector to a different element and click the wrong row.The Extension has repeated labels such as “Show QR code”, “Copy address”, and “Network options”. Ref identity must distinguish these elements without assuming AX-tree order matches Playwright locator order.
Solution
Build ref selectors at the browser discovery boundary. Keep the existing selector map consumed by Playwright tools and session managers.
flowchart TD A[AX snapshot candidates] --> B[Group normalized role-and-name selectors] B -->|unique role| C[Keep exact role selector with snapshot scope] B -->|duplicate role or generic| D[Build unique DOM selector in cached isolated world] D --> E{Selector result?} E -->|unique| H[Store selector in ref map] E -->|unavailable| F[Keep strict role or text fallback] E -->|closed shadow root| G[Store unreachable marker] C --> H F --> H H --> I[Resolve Playwright locator for action] G --> X[Reject ref action before locator creation] classDef new fill:#fff3cd,stroke:#d39e00 classDef stop fill:#f8d7da,stroke:#c82333 class B,C,D,E,F,G new class X stopYellow boxes are new. Red boxes stop the action.
The DOM builder prefers a unique
data-testid, then a uniqueid, then a unique ancestor-anchored path. It usesnth-of-typeornth-childonly within a path whose uniqueness it checks. Open shadow roots use chained host selectors:css=<host> >> css=<inside>.callOnNodeInIsolatedWorldresolves the backend node in the cached session and isolated world introduced by P1-10. This keeps selector construction outside the Extension's scuttled main-world globals. Lookups run concurrently with a fixed bound. Session or world failures retry once; missing nodes and page-function exceptions do not.Role and text selectors normalize whitespace and escape quoted values. The explicit
sflag documents the role engine's existing exact, case-sensitive behavior. Scoped snapshots prefix role and text selectors withcss=<root> >> nth=0. Unique DOM selectors remain page-wide; the scope root itself uses the root selector.For
--withinplus a ref,waitForTargetchecks page-wide ambiguity before containment. It rechecks an outside-parent result after a short delay to allow DOM movement. Late targets wait for attachment within the shared timeout budget.What changes
nth=guess--withinMap<string, string>andISessionManagerRef-specific errors include
Ref e2 is not inside the within elementandRef e2 is inside a closed shadow root: it cannot be targeted. An ambiguous ref reportsRef e2 is ambiguous (9 matches); take a new snapshot or target it by testId or selector.Decisions for reviewers
ISessionManagerwith selector strings.inert, shadow roots, slots, andaria-ownscan produce different AX and Playwright orders.cdp.ts.--withinas containment for ref targetsReview guide
Read the files in this order:
src/tools/utils/discovery.tswithinchecks.src/tools/utils/in-page.tssrc/tools/utils/cdp.tssrc/tools/utils/{discovery,in-page,cdp}.test.tssrc/server/create-server.test.tsREADME.md,SKILL.md--withinguidance.Testing
e93a529e34cc3b1c65615e87bbd282a23e16f504:CI=true yarn vitest run src/tools/utils/cdp.test.ts src/tools/utils/discovery.test.ts src/tools/utils/in-page.test.ts src/server/create-server.test.ts src/platform/playwright-driver.test.ts --coverage.enabled=false --maxWorkers=2passes: 5 files, 352 tests, no type errors.withintargets.ProofandUnit testssections below retain the original historical evidence. The fresh check does not repeat that live validation.Breaking changes
None. Refs are created per snapshot, so no caller can depend on the old selector strings. Behaviour changes, both towards failing instead of hitting the wrong element:
within+ a ref outside the parent now fails fast instead of acting inside the parent;Consumer impact
test/e2e/playwright/llm-workflow/tests/llm-workflow/Proof
Tested with a yalc build of this branch in the Extension (e2e build, headless, fixture wallet). For the "before" I built #65 (50eb396) in a temp clone and ran the same script.
--within <first row>+ ref of the Solana row's QR buttonrole=was already exact)within, scoped snapshot)describe-screentoolMs p50: home / networks / receiveGeneric Chromium cases (built dist, real Chromium; none of these occur on the Extension today): modal
<dialog>,inert, open, nested and slotted shadow DOM,aria-owns, and a page whose main world makesCSS,Element,Node,windowandArray.fromthrow. 39/41 refs click the right element. The 2 closed-shadow refs fail on purpose, and none hits a wrong element. On 40 duplicate buttons, snapshot time went from 42 to 33 ms, because lookups now run concurrently.Pre-existing issues found (not caused by this PR, unchanged on both builds):
href.describe-screentakes ~4 s on networks, receive, the account list and About; a fixed timeout is suspected.Unit tests
New tests:
text=stays strict; nonthfallback.in-pagebuilder in avmwith a fake DOM: testid, then id, then path; shadow roots and host chaining; slotted:nth-child; closed shadow marker; scuttled globals;CSS.escapeparity.callOnNodeInIsolatedWorld: retry, node gone, exceptions.within: ambiguous ref, ref not inside the parent, and the race when the element moves into the parent between checks.41a404fdeflakes the P1-01 queue-wait timing test: under host load it saw 45 ms against a 50 ms threshold, so it now holds the first request for 250 ms.