Skip to content

feat: MM_TARGET_OBSCURED, click --dismiss-keyboard and screenChanged #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cryptotavares wants to merge 1 commit into from
Draft

feat: MM_TARGET_OBSCURED, click --dismiss-keyboard and screenChanged #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Mobile `click` never reports a tap that hit something else: device-mcp refuses a tap when another element covers the target, and the click fails with the new `MM_TARGET_OBSCURED` (`MM_TARGET_NOT_VISIBLE` when the target is off screen). `click --dismiss-keyboard` closes a covering keyboard and taps once more (`keyboardDismissed: true`). With an observation, a mobile click result has `screenChanged`. The hit-test needs a device-mcp release with MetaMask/device-mcp#51.
- Start the CLI faster: find the project root without running `git` and do not load zod. Load `@metamask/device-mcp` only when a mobile session uses it, so browser daemons start without it.
- **BREAKING:** `scroll_to_element` `direction` is now the content direction (`down` reveals content below; it was the finger direction) and defaults to `auto`. The result adds `swipes`, `inViewport` and `bounds`; a target that is found but not brought into view fails with the new `MM_TARGET_NOT_VISIBLE`, and one never found with `MM_TARGET_NOT_FOUND`. The viewport check needs a device-mcp release with MetaMask/device-mcp#50; with device-mcp 0.7.0 only `bounds` is returned.
- Mobile `click` and `type` tap the element that the target wait found, so a tap takes one device snapshot instead of two (needs a device-mcp release with `tapElement(query, { element })`; older releases ignore the option). `describe_screen` reads the app state and the screen at the same time on mobile.
Expand Down
22 changes: 12 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -624,16 +624,16 @@ mm describe-screen

### Interaction

| Command | Description |
| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `mm click <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Clicks an element by its accessibility reference (e.g., `e3`). The ref comes from a prior `describe-screen` call. Waits for the element to be visible before clicking. The `--timeout` covers the entire operation (visibility wait + click action). Default: 15s. Use `--within` to scope the target inside a parent element (`testid:<id>`, `selector:<css>`, or a bare a11y ref). |
| `mm type <ref> <text> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Types text into an input element identified by its accessibility reference. Clears the field first, then sets the new value (uses Playwright's `fill()`). Accepts `--timeout` for the total time budget (default: 15s). Use `--within` to scope the target inside a parent element. |
| `mm get-text <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Reads the text content of an element. Returns the inner text, target descriptor, and character length. Accepts `--timeout` for the total time budget (default: 15s). Useful for asserting visible values without screenshots. |
| `mm describe-screen` | Captures the full screen state: extension info, visible test IDs, and a trimmed accessibility tree with deterministic refs (`e1`, `e2`, ...). This is the primary command for understanding what's on screen before interacting. |
| `mm screenshot [--name <name>]` | Takes a full-page screenshot of the current page. Saves to the artifacts directory. Use `--name` to set a descriptive filename. |
| `mm wait-for <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Blocks until an element identified by its accessibility reference becomes visible, or the timeout expires. Default timeout is 15 seconds. Use `--within` to scope the target inside a parent element. |
| `mm wait-for-notification [--timeout <ms>]` | Waits for the extension notification popup to appear within a timeout. Returns the notification page URL. |
| `mm clipboard <read\|write> [text]` | Reads from or writes to the system clipboard via Chrome DevTools Protocol. Useful for pasting seed phrases or copying addresses. |
| Command | Description |
| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `mm click <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>] [--dismiss-keyboard]` | Clicks an element by its accessibility reference (e.g., `e3`). The ref comes from a prior `describe-screen` call. Waits for the element to be visible before clicking. The `--timeout` covers the entire operation (visibility wait + click action). Default: 15s. Use `--within` to scope the target inside a parent element (`testid:<id>`, `selector:<css>`, or a bare a11y ref). Mobile: fails with `MM_TARGET_OBSCURED` when something covers the target; `--dismiss-keyboard` closes a covering keyboard and taps again. |
| `mm type <ref> <text> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Types text into an input element identified by its accessibility reference. Clears the field first, then sets the new value (uses Playwright's `fill()`). Accepts `--timeout` for the total time budget (default: 15s). Use `--within` to scope the target inside a parent element. |
| `mm get-text <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Reads the text content of an element. Returns the inner text, target descriptor, and character length. Accepts `--timeout` for the total time budget (default: 15s). Useful for asserting visible values without screenshots. |
| `mm describe-screen` | Captures the full screen state: extension info, visible test IDs, and a trimmed accessibility tree with deterministic refs (`e1`, `e2`, ...). This is the primary command for understanding what's on screen before interacting. |
| `mm screenshot [--name <name>]` | Takes a full-page screenshot of the current page. Saves to the artifacts directory. Use `--name` to set a descriptive filename. |
| `mm wait-for <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Blocks until an element identified by its accessibility reference becomes visible, or the timeout expires. Default timeout is 15 seconds. Use `--within` to scope the target inside a parent element. |
| `mm wait-for-notification [--timeout <ms>]` | Waits for the extension notification popup to appear within a timeout. Returns the notification page URL. |
| `mm clipboard <read\|write> [text]` | Reads from or writes to the system clipboard via Chrome DevTools Protocol. Useful for pasting seed phrases or copying addresses. |

### Navigation

Expand Down Expand Up @@ -801,6 +801,8 @@ Tool errors are classified into specific error codes for structured handling:
| `MM_TARGET_NOT_FOUND` | Element not found by ref, testId, or selector |
| `MM_WAIT_TIMEOUT` | Timeout waiting for element visibility |
| `MM_CLICK_FAILED` | Click operation failed |
| `MM_TARGET_OBSCURED` | Mobile: another element (keyboard, sheet, overlay) covers the target; nothing was tapped |
| `MM_TARGET_NOT_VISIBLE` | Mobile: the target is outside the viewport (tap or `scroll_to_element`) |
| `MM_CLICK_TIMEOUT` | Click action timed out (element found, click hung) |
| `MM_TYPE_FAILED` | Type operation failed |
| `MM_TYPE_TIMEOUT` | Fill action timed out |
Expand Down
Loading
Loading