Repository navigation
feat: MM_TARGET_OBSCURED, click --dismiss-keyboard and screenChanged - #77
Draft
cryptotavares wants to merge 1 commit into
Draft
cryptotavares wants to merge 1 commit into
cryptotavares wants to merge 1 commit into
Conversation
B2: a mobile click returned clicked:true while the keyboard covered the target. device-mcp now refuses such taps (TARGET_OBSCURED). - TARGET_OBSCURED -> new MM_TARGET_OBSCURED; ELEMENT_NOT_IN_VIEWPORT -> MM_TARGET_NOT_VISIBLE (typed codes checked before the text patterns) - click --dismiss-keyboard / dismissKeyboard: when the keyboard is the cover, dismiss it, re-read the element, tap once more (keyboardDismissed: true) - mobile click result gains screenChanged when an observation was collected (signature vs the last screen the agent received) - README/SKILL
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
MobilePlatformDriver.clickreturns{clicked: true}once the backend call resolves, so B2 (click --testid log-in-buttonunder the iOS keyboard) was a false success. An agent cannot tell a covered target from a missed one.This PR (with MetaMask/device-mcp#51, which refuses such taps):
MM_TARGET_OBSCUREDfor device-mcpTARGET_OBSCURED. device-mcpELEMENT_NOT_IN_VIEWPORT(tap on an off-screen element) →MM_TARGET_NOT_VISIBLE.classifyInteractionErrorchecks these typed codes before the text patterns (a message with "not found" in it is stillMM_TARGET_OBSCURED).click --dismiss-keyboard(dismissKeyboard: true): when the keyboard is the cover (keyboard: true), dismiss it, look up the element again (the layout can move), and tap once more. The result haskeyboardDismissed: true. An overlay that is not the keyboard is never dismissed.clickresult gainsscreenChangedwhen an observation is collected: whether the observed screen differs (refs ignored) from the last screen the agent received before the click.falseis a hint, not an error. It is omitted with--observe none.--dismiss-keyboard,screenChanged.Part of the mm CLI v2 effort: ticket
P1-42(Phase 1). Stacked on #76 (review that first; GitHub stack #75).Depends on MetaMask/device-mcp#51 (needs a device-mcp release before merge). With device-mcp 0.7.0, nothing changes for taps (no
TARGET_OBSCUREDis thrown).Breaking changes
None. A mobile click that silently tapped something else now fails with
MM_TARGET_OBSCURED/MM_TARGET_NOT_VISIBLE.Consumer impact
tests/llm-workflow/--dismiss-keyboardProof
Same runs as MetaMask/device-mcp#51. Error codes are read from the daemon response (the CLI prints only the message until P1-16).
click --testid log-in-button, keyboard up{"clicked": true}, still on loginMM_TARGET_OBSCUREDnaming the keyboard; nothing tapped--dismiss-keyboard{"clicked": true, "keyboardDismissed": true}→ unlocked{"clicked": true}MM_TARGET_NOT_VISIBLEscreenChanged: Home tab (already open) / profile buttonfalse/truescreenChanged)Unit tests
dismissKeyboard→ dismiss, re-wait, tap the new element,keyboardDismissed: true; no option → error unchanged; overlay (keyboard: false) → no dismiss; still covered → error after one retry; abort during dismiss → no second tap.classifyInteractionError: typedTARGET_OBSCURED/ELEMENT_NOT_IN_VIEWPORTbefore text patterns.withScreenChanged: changed / unchanged (refs ignored) / every skip case (other tool, failed click, no observation, no earlier screen, browser, no driver).--dismiss-keyboardsendsdismissKeyboard: trueand is never read as the target. Schema: boolean only.