diff --git a/.claude/agents/diagnose-maestro-failure.md b/.claude/agents/diagnose-maestro-failure.md new file mode 100644 index 0000000000..bb82160632 --- /dev/null +++ b/.claude/agents/diagnose-maestro-failure.md @@ -0,0 +1,117 @@ +--- +name: diagnose-maestro-failure +description: Use when a Maestro test-android run has failed and a structured per-flow diagnosis is needed without bloating the caller's context. Accepts either a local artifact directory or a GitHub Actions run/job URL. Reads commands JSON, screenshots, and maestro.log; classifies each newly-failing passing/ flow; identifies cascade groups; and returns a structured report with proposed fixes. Does NOT edit code or commit — the caller is responsible for consent and application. +tools: Bash, Read, Agent +--- + +# Diagnose Maestro Failure (subagent) + +You diagnose Maestro `test-android` failures from CI artifacts and return a structured report. You do not modify code, commit, or push — the caller handles consent and application. + +## Input + +The caller passes one of: + +- **Local artifact directory** — e.g. `/Users/.../maestro-root-dir-android` or `/tmp/maestro-android-`. Must contain a `tests/` subtree. +- **GitHub Actions URL** — `https://github.com/mobile-dev-inc/Maestro/actions/runs/` or `.../runs//job/`. + +If a URL, extract the `run_id` and download: + +```bash +RUN_ID= +DEST=/tmp/maestro-android-$RUN_ID +gh run download "$RUN_ID" --repo mobile-dev-inc/Maestro \ + --name maestro-root-dir-android -D "$DEST" +``` + +If the directory has no `tests/demo_app/passing/` subtree (e.g. iOS artifact, or the run failed before tests), report that and stop. + +## Where to Look + +Only `/tests/demo_app/passing/` is signal. `failing/` is the negative-path suite — its commands are *supposed* to fail. Ignore it entirely. + +```bash +cd /tests/demo_app/passing +grep -l '"FAILED"' commands-*.json +``` + +The status field lives at `.[].metadata.status`. Possible values: `COMPLETED` (success) and `FAILED` (terminal failure). `commands-*.json` is the **only** authoritative source — `screenshot-❌-*.png` is not. + +**`commands-(retry).json` is the canonical example.** This flow exercises the `retryCommand` directive: a tap fails, Maestro retries it, the retry succeeds, and the flow passes. In the artifact: + +- All entries (including the recovered `tapOnElement`) have `metadata.status = "COMPLETED"` — Maestro writes only the final outcome of a recovered command, not the failed attempt. +- A `screenshot-❌--(retry).png` still exists, captured at the moment of the failed attempt before the retry succeeded. +- The grep above does **not** match this file (no `"FAILED"` string anywhere), so the agent never investigates it. This is correct behaviour. + +The risk is treating the `❌` screenshot as evidence of failure on its own. Don't. Trust the candidates from grep — they are the flows whose JSON contains a terminal `FAILED`. Any other `❌` screenshot in `passing/` is from a recovered retry and is irrelevant. + +## Per-Flow Investigation + +For each candidate flow returned by the grep above: + +1. **Open `commands-().json`.** Locate the entry whose `metadata.status == "FAILED"` — that's the terminal failure. Record: command type (the single key under `command`), error message, `metadata.timestamp` (ms epoch), `metadata.sequenceNumber`, `metadata.duration` (≈17000 ms = default-wait timeout, indicating UI-blocked, not crash). +2. **Open the screenshot `screenshot-❌--().png` whose timestamp matches the FAILED entry.** This is the highest-signal artifact — JSON and logs alone misclassify when a system dialog is invisible to them. Always read it. Ignore `❌` screenshots from other timestamps in the same file — those are recovered retries. +3. **Grep `maestro.log`** for the flow name and the timestamp window for surrounding driver activity, gRPC errors, and stack traces. + +## Classification + +| Screen at failure | Class | Where the fix lives | +|---|---|---| +| Expected app screen, but selector wouldn't match | Test flow issue | `e2e/demo_app/.maestro/**/*.yaml` | +| Android / GMS system dialog blocking the app | Maestro driver gap (auto-grant / auto-dismiss missing) | `maestro-client/src/main/java/maestro/drivers/AndroidDriver.kt` or `.github/workflows/test-e2e.yaml` (emulator pre-config) | +| Pixel launcher visible behind a system dialog | Same — dialog stole focus across `launchApp` / `clearState` / `stopApp` | Same as above | +| App crash dialog / blank screen | App-side issue | Out of scope — report only | +| Stack trace from `maestro.*` in `maestro.log` | Maestro framework bug | `maestro-client/` / `maestro-orchestra/` / `maestro-android/` | + +For anything classified as **driver gap** or **framework bug**, dispatch an `Explore` subagent into the relevant module(s) to find the offending code path before proposing line numbers. Never guess. + +## Cascade Detection (critical) + +A single system dialog can fail many subsequent flows because `clearState` / `launchApp` / `stopApp` don't dismiss system overlays. List FAILED entries by ascending timestamp; if downstream screenshots show the *same* overlay, group them under the **root trigger** (the earliest flow that raised the dialog). Propose **one** fix per root, not N fixes for the cascade. + +## Required Output Format + +Return this structure verbatim — the caller parses it: + +````markdown +## Summary + + +## Root causes (caller acts on each) + +### 1. — +- **Trigger:** +- **Affected flows:** , , ... (N flows) +- **Cascade?** yes/no +- **Proposed fix:** + - File: `:` + - What it does: + - Diff (unified or pseudo): + ``` + - old line + + new line + ``` + - Side effects: + - Verified via Explore? + +### 2. ... + +## Out of scope +- `` — + +## Notes + +```` + +## Constraints + +- Do NOT edit, write, or commit any file. Diagnosis only. +- Do NOT propose a separate fix for each member of a cascade group. +- Do NOT guess at Kotlin line numbers — verify via Explore subagent first. +- Do NOT report `failing/` flows as failures. +- Do NOT trust `❌` screenshots on their own. The grep filter (`grep -l '"FAILED"' commands-*.json`) is the source of truth for which flows had a terminal failure. A `❌` screenshot inside a flow whose JSON has zero `FAILED` entries is from a recovered retry and is not evidence of regression. `commands-(retry).json` is the canonical example — see "Where to Look" for details. +- Always read the screenshot, even when JSON+log seem to point clearly at a cause — the screenshot is what reveals system overlays. + +## When to Stop and Hand Back + +Hand back as soon as every passing-suite FAILED command has been either (a) classified with a proposed fix, or (b) classified as out-of-scope with a reason. Do not loop, retry, or attempt to apply fixes. diff --git a/.claude/skills/bump-android-version/SKILL.md b/.claude/skills/bump-android-version/SKILL.md new file mode 100644 index 0000000000..fa88aa27bc --- /dev/null +++ b/.claude/skills/bump-android-version/SKILL.md @@ -0,0 +1,151 @@ +--- +name: bump-android-version +description: Use when bumping Maestro's Android compileSdk/targetSdk to a new API level and validating end-to-end against the test-e2e GHA workflow until the test-android job is green. +--- + +# Bump Android Version + +## Overview + +Drive an Android-version bump in Maestro commit-by-commit on a single feature branch: edit gradle, rebuild the on-device driver APKs, push a draft PR, dispatch `test-e2e.yaml` against the new system image, watch the `test-android` job, and diagnose+fix any newly-failing `passing/` flows. Loop until green. + +## When to Use + +- User asks to bump Maestro's Android `compileSdk` / `targetSdk` (e.g. "bump to API 36"). +- User wants to validate Maestro against a new Android system image. + +One commit per logical step. Don't bundle the gradle bump and the APK rebuild — they need to be separately revertible. After a failing run, fixes are committed and the workflow re-dispatched. Loop until green; do not stop on the first re-dispatch. + +**The loop only acts on `passing/` failures.** `tests/demo_app/passing/` is the regression-detection suite — its flows are expected to pass, so a failure there is a real signal. `tests/demo_app/failing/` is the negative-path suite (its flows are *expected* to fail) and is ignored end-to-end. The diagnose agent rejects `failing/` artifacts and rejects retry-recovered flows in `passing/` (see [`.claude/agents/diagnose-maestro-failure.md`](../../agents/diagnose-maestro-failure.md)). If the run is otherwise green except for `failing/`, that's a green run. + +## Pre-flight + +- Working tree clean (`git status`). +- `main` up to date: `git fetch origin && git checkout main && git pull --ff-only origin main`. +- The workflow's `workflow_dispatch` inputs `android_version` (and optional `app` / `flow`) must exist on the branch. +- `android_version` is a `choice` input — its options live in `.github/workflows/test-e2e.yaml`. If `` isn't already listed (e.g. bumping to API 37), add it to the `options:` list in a separate commit before dispatching, otherwise the workflow rejects the input. +- The `validate-inputs` job rejects `android_version <= android-29`. Bumps below API 29 aren't supported. +- Branch name: `bump-android-api--` (e.g. `bump-android-api-34-36`). + +```bash +git checkout -b bump-android-api-- origin/main +``` + +## Commit 1: bump compile/target SDK + +Read current values from `maestro-android/build.gradle.kts` (`compileSdk` and `targetSdk`). Tell the user the current values verbatim, then ask for the new target: + +> Current `compileSdk` / `targetSdk` = ``. What's the new API level? + +Edit both lines. Also `rg -n "compileSdk\s*=|targetSdk\s*=" --type kotlin --type-add 'kotlin:*.kts'` to catch consistent occurrences in other modules — only update those that intentionally track the same SDK. Commit: + +```bash +git add maestro-android/build.gradle.kts +git commit -m "chore(android): bump compile/target SDK from to " +``` + +## Commit 2: rebuild driver APKs + +```bash +./gradlew :maestro-android:assemble :maestro-android:assembleAndroidTest +``` + +The build's `copyMaestroAndroid` / `copyMaestroServer` finalizers update three checked-in files. Stage and commit *only* those: + +```bash +git add maestro-client/src/main/resources/maestro-app.apk \ + maestro-client/src/main/resources/maestro-server.apk \ + maestro-client/src/main/resources/maestro-android-source.sha256 +git commit -m "chore(android): rebuild driver APKs against API " +``` + +If `assemble` fails, **stop and surface the failure to the user** with the gradle error and a suggested fix. Do not proceed without rebuilt driver APKs. Once we have assemble fixed and new drivers are commited, we can proceed to the next step. + +## Step 3: push draft PR + dispatch + +```bash +git push -u origin bump-android-api-- +gh pr create --draft \ + --title "chore(android): bump compile/target SDK to " \ + --body "Validates Maestro against API . Driver APKs rebuilt. Test-e2e dispatched with system-images;android-;google_apis;x86_64." + +gh workflow run test-e2e.yaml --ref bump-android-api-- \ + -f android_version="android-" + +# capture the run id +sleep 3 +gh run list --workflow test-e2e.yaml --branch bump-android-api-- --limit 1 --json databaseId,status +``` + +Always pass `-f android_version=android-` — the default is `android-32` and would silently re-test the old API. The workflow constructs the full `system-images;...` string internally. `app` and `flow` are optional narrowing knobs (defaults: `demo_app`, all flows); leave them off for the full bump-validation run. + +## Step 4: watch the test-android job + +```bash +gh run watch +``` + +Block in the foreground; the run is the bottleneck. **Green here means every `passing/` flow passed (terminally — retry-recovered counts as passed). `failing/` outcomes do not affect the verdict.** On green: update PR from draft → ready, done. + +On red, download artifacts: + +```bash +gh run download --name maestro-root-dir-android -D /tmp/maestro-android- +``` + +## Step 5: diagnose (delegated to subagent) + +Dispatch the **`diagnose-maestro-failure`** subagent with the artifact directory `/tmp/maestro-android-` (or the GHA run URL — the subagent handles both). It diagnoses each passing-suite failure in isolation and returns a structured report grouped by root cause (cascades collapsed). It does **not** apply fixes — that's your job here. + +When the subagent returns: + +**Fixes go in Maestro source — not in `.github/workflows/test-e2e.yaml`.** A failing flow on a new API level reflects a behaviour Maestro users will also hit on their own machines/CI, not just ours. Patching the GHA workflow (e.g. extra `adb shell settings put …`, command-line tweaks, AVD setup steps) hides the regression from real users and ships a Maestro that's only green inside our CI. + +Acceptable fix targets and their roles are documented in [`AGENTS.md`](../../../AGENTS.md) (modules `maestro-android/`, `maestro-client/`, `maestro-orchestra/`, `e2e/demo_app/`). Driver-behaviour fixes belong in `maestro-android/` or `maestro-client/`; fixture-only issues belong in `e2e/demo_app/`. Editing the driver source means rebuilding the driver APKs (`./gradlew :maestro-android:assemble :maestro-android:assembleAndroidTest`) before re-dispatching. + +Valid edits to `test-e2e.yaml` during this loop are workflow-shape changes (matrix, retention, dispatch inputs) — and the narrow exception below. Anything else (e.g. extra `adb shell settings put …` to mask a Maestro driver gap) is off-limits. If the subagent proposes a workflow patch, push back: ask it (or yourself) where the same fix would live in `maestro-android/` or `maestro-client/` so users on the new API level inherit it automatically. + +**Exception — third-party app first-run UI not triggered by a Maestro API.** A pre-installed app's own onboarding (Chrome Welcome / "Make Chrome your own", browser default-app picker, Play Protect prompts that fire at app launch) is environment-harness state, not driver behaviour. Maestro doesn't expose a public API that triggers them — they're side effects of the OS image we picked. These may be pre-disabled in the AVD setup step (typical knobs: `setprop debug.chrome.command_line`, `settings put global …` flags). Test: *if Maestro called nothing related to this dialog, would it still appear?* If yes → CI workaround is fair game. If no (e.g. `setLocation` triggers GMS Location Accuracy via `FusedLocationProviderClient` inside the driver) → fix the driver instead. + +1. For each root cause in its report, present the proposed diff to the user (file path + one-line summary + diff + side effects). Reject any proposal that targets `test-e2e.yaml` for driver-behaviour reasons before showing it — re-scope to Maestro source first. +2. **Ask consent explicitly per fix:** + > Proposed fix for ``: ``. Apply? +3. On approval: edit, commit with a focused message — don't bundle unrelated fixes. + ```bash + git commit -m "fix(android-): " + ``` +4. On rejection: skip that fix; record what was rejected so you don't re-propose it next iteration. + +After all approved fixes for this iteration are committed: + +- **If any fix was applied:** push and re-dispatch, then loop back to **Step 4** with the new `run_id`. + + ```bash + git push + gh workflow run test-e2e.yaml --ref bump-android-api-- \ + -f android_version="android-" + ``` + +- **If no fixes were applied** (everything was out-of-scope, or the user rejected every proposal): pause and ask the user how to proceed. Do not keep dispatching. + +## Termination + +Stop when `test-android` is green on the latest run. Then: + +```bash +gh pr ready +``` + +If the loop has gone three iterations without progress (same flow keeps failing for different reasons after fixes), pause and ask the user how to proceed — don't spin indefinitely. + +## Anti-patterns + +- **Bundling gradle + APK rebuild into one commit** — kills bisectability. Always two commits. +- **Dispatching with default inputs** — `android_version` defaults to `android-32` and hits the old API. Always pass `-f android_version=android-`. +- **Forgetting to extend the `android_version` choice enum** — the workflow rejects API levels not listed in `options:`. Add `` to the enum in a separate commit before the first dispatch. +- **Auto-applying Maestro source fixes without consent** — every Kotlin patch needs explicit user approval first. +- **Patching `.github/workflows/test-e2e.yaml` to mask a driver behaviour gap** — workflow band-aids hide the regression from users running Maestro outside our CI. Fix `maestro-android/` or `e2e/demo_app/` instead so the fix ships with the driver APKs. +- **Treating `failing/` artifacts as regressions** — that suite is expected to fail. +- **Treating `❌` screenshots in `passing/` as regressions on their own** — when `retryCommand` recovers a failed attempt, Maestro writes only the final `COMPLETED` entry to `commands-().json` but leaves the `❌` screenshot from the failed attempt on disk (e.g. `screenshot-❌--(retry).png` for the `retry` flow itself). The diagnose agent uses `commands-*.json` containing a terminal `FAILED` entry as the source of truth — orphan `❌` screenshots without a matching FAILED entry are noise. +- **Committing on `main`** — always work on the bump branch. Verify with `git status` before every commit. +- **Skipping the screenshot read** — the screenshot tells you what `maestro.log` and the JSON cannot. It's the highest-signal artifact. Always read it. diff --git a/.claude/skills/prepare-release/SKILL.md b/.claude/skills/prepare-release/SKILL.md new file mode 100644 index 0000000000..7bd78428eb --- /dev/null +++ b/.claude/skills/prepare-release/SKILL.md @@ -0,0 +1,78 @@ +--- +name: prepare-release +description: Use this skill when the user asks to "prepare a release", "update the changelog", "bump the version", "prepare release notes", or "cut a release". Automates changelog generation from git commits since the last tag and version bumping across gradle.properties files. +version: 1.0.0 +--- + +# Prepare Release Skill + +Automates the three-file release-prep workflow: CHANGELOG.md entry, and version bumps in both gradle.properties files. + +## Steps + +### 1. Determine the version number + +Check if the user has provided a version number in their message. If not, use `AskUserQuestion` to ask: + +> "What version number should this release be? (e.g. 2.5.1)" + +Use the version exactly as provided (no `v` prefix). + +### 2. Find commits since the last tag + +Run these two commands: + +```bash +git describe --tags --abbrev=0 +``` + +Then use the resulting tag (e.g. `v2.5.0`) to list commits: + +```bash +git log ..HEAD --oneline +``` + +### 3. Update CHANGELOG.md + +File: `CHANGELOG.md` + +Insert a new version section **between** the `## Unreleased` line and the first existing version header. Format: + +```markdown +## + +- +- +... +``` + +**Editorial rules for commit → entry conversion:** +- Each commit becomes one `- ` bullet point. +- Strip conventional-commit prefixes (`fix:`, `feat:`, `fix(scope):`, etc.) and capitalise the first word. +- Remove PR number suffixes like `(#1234)` only if the commit message is already clear without them; otherwise keep them. +- Rewrite for clarity and consistency with the existing CHANGELOG tone (sentence case, imperative or noun phrases). +- Do **not** add a "Thanks to" contributors line — that is added manually. + +### 4. Update version numbers in both gradle.properties files + +Update these two files so all three files carry the same version: + +**`gradle.properties`** — change the `VERSION_NAME` line: +``` +VERSION_NAME= +``` + +**`maestro-cli/gradle.properties`** — change the `CLI_VERSION` line: +``` +CLI_VERSION= +``` + +### 5. Run ChangeLogUtilsTest + +```bash +./gradlew :maestro-cli:test --tests "maestro.cli.util.ChangeLogUtilsTest" +``` + +The key test (`test format last version`) reads `CLI_VERSION` from `maestro-cli/gradle.properties` and asserts that the CHANGELOG contains a non-empty entry for that version. All three files must be consistent for the test to pass. + +Report the test result. If it fails, diagnose and fix before finishing. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 8093cd539f..33579f14bc 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -6,4 +6,6 @@ copilot:summary +> **Does this need e2e tests?** The demo app lives in [`e2e/demo_app/`](e2e/demo_app/) — add a test screen and Maestro flow in the same PR. See [`e2e/demo_app/CLAUDE.md`](e2e/demo_app/CLAUDE.md) for details. + ## Issues fixed diff --git a/.github/scripts/boot_simulator.sh b/.github/scripts/boot_simulator.sh index 477666e235..f8f96bfee1 100755 --- a/.github/scripts/boot_simulator.sh +++ b/.github/scripts/boot_simulator.sh @@ -29,4 +29,10 @@ while true; do echo "Waiting for the simulator to be ready..." sleep 5 # sleep for 5 seconds before checking again to avoid spamming fi -done \ No newline at end of file +done + +# Expose the booted simulator UDID to subsequent workflow steps so they don't +# have to rediscover it (e.g. by parsing `xcrun simctl list devices booted`). +if [ -n "$GITHUB_ENV" ]; then + echo "SIM_UDID=$simulator_id" >> "$GITHUB_ENV" +fi \ No newline at end of file diff --git a/.github/scripts/setup_android_device.sh b/.github/scripts/setup_android_device.sh new file mode 100755 index 0000000000..7f6d1536a3 --- /dev/null +++ b/.github/scripts/setup_android_device.sh @@ -0,0 +1,51 @@ +#!/bin/bash +set -euo pipefail + +# Wait for the connected Android emulator to finish booting, then apply CI-specific +# pre-configuration that prevents Maestro flows from being blocked by: +# - Chrome's first-run "Welcome / Make Chrome your own" UI +# - GMS's "Google Location Accuracy" consent dialog (Android 14+ google_apis images) +# +# Both pre-configs are emulator/userdebug-only — real users hit these dialogs on their +# own devices and need driver-level fixes for them. This script is purely a CI bootstrap. +# +# Idempotent. Run after starting the emulator and before installing apps / running tests. + +adb wait-for-device && echo 'Emulator device online' +adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;' && echo 'Emulator booted' + +# Wait for the package manager service to be available +while true; do + adb shell service list | grep 'package' && echo 'service "package" is active!' && break + echo 'waiting for service "package" to start' + sleep 1 +done + +# Skip Chrome first-run experience (Welcome / "Make Chrome your own" sign-in card +# surfaced on Android 35+). This is Chrome's own UI on launch, not a Maestro behaviour +# — Maestro just dispatches the VIEW intent in openLink. The setprop trick is userdebug- +# only, so it lives here in the AVD setup rather than in the maestro-client driver. +# Ref: https://stackoverflow.com/questions/33408138 +adb shell 'echo "chrome --disable-fre --no-default-browser-check --no-first-run" > /data/local/tmp/chrome-command-line' +adb shell setprop debug.chrome.command_line /data/local/tmp/chrome-command-line +adb shell am force-stop com.android.chrome || true + +# Pre-grant GMS "Google Location Accuracy" consent so FusedLocationProviderClient.setMockMode(true) +# in maestro-server doesn't trigger a system overlay during setLocation flows on the +# Android 14+ google_apis image. The flag lives in GMS's private content provider +# (content://com.google.settings/partner) which only `system` uid can write to — +# `adb root` (userdebug emulator only) gives us that. Real users on production-build +# phones can't root and will hit the dialog separately; that's a driver-side fix. +adb root +adb wait-for-device +adb shell content insert \ + --uri content://com.google.settings/partner \ + --bind name:s:network_location_opt_in --bind value:s:1 || true +adb shell content insert \ + --uri content://com.google.settings/partner \ + --bind name:s:use_location_for_services --bind value:s:1 || true + +# Drop back to shell uid so subsequent steps (install_apps, run_tests, maestro driver +# via dadb) exercise the same code path real users hit on production-build devices. +adb unroot +adb wait-for-device diff --git a/.github/workflows/check-drivers.yaml b/.github/workflows/check-drivers.yaml new file mode 100644 index 0000000000..89fdbf39d9 --- /dev/null +++ b/.github/workflows/check-drivers.yaml @@ -0,0 +1,110 @@ +name: Check Drivers Up-to-Date + +on: + pull_request: + paths: + - 'maestro-android/**' + - 'maestro-ios-xctest-runner/**' + +jobs: + changes: + runs-on: ubuntu-latest + outputs: + android: ${{ steps.filter.outputs.android }} + ios: ${{ steps.filter.outputs.ios }} + steps: + - uses: actions/checkout@v6 + - uses: dorny/paths-filter@v4 + id: filter + with: + filters: | + android: + - 'maestro-android/**' + ios: + - 'maestro-ios-xctest-runner/**' + + check-android-driver: + name: Check Android Driver + needs: changes + if: needs.changes.outputs.android == 'true' + runs-on: ubuntu-latest + + steps: + - name: Clone repository + uses: actions/checkout@v6 + + - name: Set up Java + uses: actions/setup-java@v5 + with: + distribution: zulu + java-version: 17 + cache: gradle + + - name: Check Android drivers are fresh + run: ./gradlew :maestro-client:checkAndroidApksFresh + + check-ios-driver: + name: Check iOS Driver + needs: changes + if: needs.changes.outputs.ios == 'true' + runs-on: macos-26 + timeout-minutes: 30 + + steps: + - name: Clone repository + uses: actions/checkout@v6 + + - name: Select Xcode 26.2 + run: sudo xcode-select -s /Applications/Xcode_26.2.app/Contents/Developer + + - name: Rebuild iOS drivers + run: sh maestro-ios-xctest-runner/build-maestro-ios-runner-all.sh + + - name: Fail if drivers are out of date + run: | + # Compare extracted archive contents rather than raw zip bytes. + # `zip -r` is not deterministic (timestamps, file ordering leak into + # the archive), so two builds from identical sources produce + # byte-different zips. We want the check to fail only on meaningful + # content differences. + fail=0 + for rel in \ + driver-iPhoneSimulator/Debug-iphonesimulator/maestro-driver-iosUITests-Runner.zip \ + driver-iPhoneSimulator/Debug-iphonesimulator/maestro-driver-ios.zip \ + driver-iPhoneSimulator/maestro-driver-ios-config.xctestrun \ + driver-iphoneos/Debug-iphoneos/maestro-driver-iosUITests-Runner.zip \ + driver-iphoneos/Debug-iphoneos/maestro-driver-ios.zip \ + driver-iphoneos/maestro-driver-ios-config.xctestrun; do + path="maestro-ios-driver/src/main/resources/$rel" + [ -e "$path" ] || continue + if [[ "$path" == *.zip ]]; then + tmp=$(mktemp -d) + git show "HEAD:$path" > "$tmp/committed.zip" 2>/dev/null || { + echo "NEW FILE: $rel" + fail=1 + continue + } + mkdir -p "$tmp/head" "$tmp/new" + unzip -q "$tmp/committed.zip" -d "$tmp/head" + unzip -q "$path" -d "$tmp/new" + if ! diff -qr "$tmp/head" "$tmp/new" >/dev/null; then + echo "CONTENT DIFFERS: $rel" + diff -r "$tmp/head" "$tmp/new" | head -30 + fail=1 + fi + else + # plist / xctestrun — diff as text + if ! git diff --exit-code -- "$path" >/dev/null; then + echo "CONTENT DIFFERS: $rel" + git diff --stat -- "$path" + fail=1 + fi + fi + done + if [ "$fail" -ne 0 ]; then + echo "" + echo "ERROR: iOS drivers are out of date." + echo "Run the following and commit the updated drivers:" + echo " sh maestro-ios-xctest-runner/build-maestro-ios-runner-all.sh" + exit 1 + fi \ No newline at end of file diff --git a/.github/workflows/lock-closed-issues.yaml b/.github/workflows/lock-closed-issues.yaml index 1453e62217..70cd98aaf2 100644 --- a/.github/workflows/lock-closed-issues.yaml +++ b/.github/workflows/lock-closed-issues.yaml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-latest if: github.repository == 'mobile-dev-inc/maestro' steps: - - uses: dessant/lock-threads@v5 + - uses: dessant/lock-threads@v6 with: process-only: issues github-token: ${{ github.token }} diff --git a/.github/workflows/publish-cli.yaml b/.github/workflows/publish-cli.yaml index d6b9718dd4..133a915122 100644 --- a/.github/workflows/publish-cli.yaml +++ b/.github/workflows/publish-cli.yaml @@ -10,10 +10,10 @@ jobs: steps: - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Java - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: distribution: zulu java-version: 17 diff --git a/.github/workflows/publish-release.yaml b/.github/workflows/publish-release.yaml index c5aa9b2d7b..9214a5cb45 100644 --- a/.github/workflows/publish-release.yaml +++ b/.github/workflows/publish-release.yaml @@ -17,10 +17,10 @@ jobs: steps: - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Java - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: distribution: zulu java-version: 17 diff --git a/.github/workflows/publish-snapshot.yaml b/.github/workflows/publish-snapshot.yaml index 15dd13836b..c0dc1c6764 100644 --- a/.github/workflows/publish-snapshot.yaml +++ b/.github/workflows/publish-snapshot.yaml @@ -13,10 +13,10 @@ jobs: steps: - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Java - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: distribution: zulu java-version: 17 diff --git a/.github/workflows/test-e2e-ios-intel.yaml b/.github/workflows/test-e2e-ios-intel.yaml index c93e88b877..03e419195e 100644 --- a/.github/workflows/test-e2e-ios-intel.yaml +++ b/.github/workflows/test-e2e-ios-intel.yaml @@ -33,14 +33,14 @@ jobs: run: ./gradlew :maestro-cli:distZip - name: Upload zipped Maestro CLI artifact - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: name: maestro-cli-jdk${{ matrix.java-version }}-run_id${{ github.run_id }} path: maestro-cli/build/distributions/maestro.zip retention-days: 1 - name: Upload build/Products to artifacts - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: name: build__Products-jdk${{ matrix.java-version }} path: build/Products @@ -67,7 +67,7 @@ jobs: java-version: 17 - name: Download artifacts - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: maestro-cli-jdk17-run_id${{ github.run_id }} @@ -111,7 +111,7 @@ jobs: run: kill -SIGINT "$(cat ~/screenrecord.pid)" - name: Upload ~/.maestro artifacts - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: maestro-root-dir-ios @@ -119,17 +119,8 @@ jobs: retention-days: 7 include-hidden-files: true - - name: Upload xctest runner logs - uses: actions/upload-artifact@v6 - if: success() || failure() - with: - name: xctest_runner_logs - path: ~/Library/Logs/maestro/xctest_runner_logs - retention-days: 7 - include-hidden-files: true - - name: Upload screen recording of Simulator - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: maestro-screenrecord-ios.mp4 diff --git a/.github/workflows/test-e2e-prod.yaml b/.github/workflows/test-e2e-prod.yaml deleted file mode 100644 index 1390381de7..0000000000 --- a/.github/workflows/test-e2e-prod.yaml +++ /dev/null @@ -1,85 +0,0 @@ -name: Test E2E (prod) - -on: - workflow_dispatch: - schedule: - - cron: '0 * * * *' - -jobs: - test-cloud-production: - # This job is copied from "e2e-production" in mobile-dev-inc/monorepo. - # We want it here so open-source users can also have some visibility into it. - - runs-on: ubuntu-latest - if: github.repository == 'mobile-dev-inc/maestro' - - steps: - - name: Clone repository - uses: actions/checkout@v4 - - - name: Set up Java - uses: actions/setup-java@v4 - with: - distribution: zulu - java-version: 17 - - - name: Install Maestro - run: | - curl -Ls --retry 3 --retry-all-errors "https://get.maestro.mobile.dev" | bash - echo "${HOME}/.maestro/bin" >> $GITHUB_PATH - - - name: Print Maestro version - run: maestro --version - - - name: Download samples - run: maestro download-samples - - - name: Run iOS test - run: | - maestro cloud \ - --apiKey ${{ secrets.E2E_MOBILE_DEV_API_KEY }} \ - --timeout 180 \ - --fail-on-cancellation \ - --include-tags=advanced \ - samples/sample.zip samples - - - name: Run Android test - run: | - maestro cloud \ - --apiKey ${{ secrets.E2E_MOBILE_DEV_API_KEY }} \ - --fail-on-cancellation \ - --include-tags advanced \ - samples/sample.apk samples - - - name: Send Slack message - if: failure() - run: | - curl --request POST \ - --url "${{ secrets.E2E_SLACK_WEBHOOK_URL }}" \ - --header 'Content-Type: application/json' \ - --data '{ - "text": "🚨 *Maestro E2E Test Failed*\nStatus: 'Failure'\nRun: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View details>" - }' - - # - name: Trigger alert on failure - # if: ${{ false }} - # # if: failure() - # run: | - # curl --request POST \ - # --url "https://events.pagerduty.com/v2/enqueue" \ - # --header 'Content-Type: application/json' \ - # --data '{ - # "payload": { - # "summary": "E2E test failed", - # "source": "E2E test", - # "severity": "critical" - # }, - # "routing_key": "${{ secrets.E2E_PAGER_DUTY_INTEGRATION_KEY }}", - # "event_action": "trigger", - # "links": [ - # { - # "href": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}", - # "text": "Failed E2E test - Github Action" - # } - # ] - # }' diff --git a/.github/workflows/test-e2e.yaml b/.github/workflows/test-e2e.yaml index 5cd7335e9f..5ce4f951a7 100644 --- a/.github/workflows/test-e2e.yaml +++ b/.github/workflows/test-e2e.yaml @@ -2,19 +2,158 @@ name: Test E2E on: workflow_dispatch: + inputs: + android_version: + description: 'Android API level to test against' + required: false + type: choice + default: 'android-32' + options: + - 'android-32' + - 'android-33' + - 'android-34' + - 'android-35' + - 'android-36' + app: + description: 'App workspace to test (e.g. demo_app, simple_web_view). Empty = all apps.' + type: string + default: 'demo_app' + flow: + description: 'Optional: flow filename within the chosen app (e.g. setLocation). Empty = all flows. Single-flow mode runs Android only.' + type: string + default: '' + platform: + description: 'Platform to test against (workflow_dispatch only; pull_request runs all three).' + type: choice + default: 'android' + options: + - 'android' + - 'ios' + - 'web' pull_request: +# Cancel an in-progress run when a new one starts on the same ref + same trigger + same dispatch +# inputs. PR pushes cancel earlier PR runs on the same branch; manual dispatches with different +# android_version / platform / app / flow stay independent so we can validate combinations in parallel. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ inputs.android_version || '' }}-${{ inputs.platform || '' }}-${{ inputs.app || '' }}-${{ inputs.flow || '' }} + cancel-in-progress: true + jobs: + validate-inputs: + name: Validate workflow inputs + if: github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + outputs: + flow_path: ${{ steps.resolve.outputs.flow_path }} + steps: + - uses: actions/checkout@v6 + + - name: Validate android_version + run: | + version="${{ inputs.android_version }}" + suffix="${version#android-}" + if ! printf '%s' "$suffix" | grep -qE '^[0-9]+$'; then + echo "::error::android_version malformed: '$version'" + exit 1 + fi + if [ "$suffix" -le 29 ]; then + echo "::error::android_version must be > android-29 (got: $version)" + exit 1 + fi + + - name: Validate app + if: inputs.app != '' + run: | + app="${{ inputs.app }}" + if [ "$app" != "demo_app" ] && [ ! -d "e2e/workspaces/$app" ]; then + echo "::error::Unknown app: '$app'. Available:" + echo " demo_app" + for d in e2e/workspaces/*/; do [ -d "$d" ] && echo " $(basename "$d")"; done + exit 1 + fi + + - name: Resolve flow + id: resolve + if: inputs.flow != '' + run: | + app="${{ inputs.app }}" + [ -z "$app" ] && app="demo_app" + if [ "$app" = "demo_app" ]; then + workspace_dir="e2e/demo_app/.maestro" + else + workspace_dir="e2e/workspaces/$app" + fi + + name="${{ inputs.flow }}" + name="${name%.yaml}" + matches=$(find "$workspace_dir" -type f -name "${name}.yaml") + count=$(printf "%s\n" "$matches" | grep -cv '^$' || true) + + if [ "$count" -eq 0 ]; then + echo "::error::No flow '$name' under $workspace_dir. Available:" + find "$workspace_dir" -type f -name "*.yaml" | xargs -n1 basename | sed 's/\.yaml$//' | sort -u + exit 1 + fi + if [ "$count" -gt 1 ]; then + echo "::error::Multiple matches for '$name':" + printf "%s\n" "$matches" + exit 1 + fi + + # run_tests runs with working-directory: ${{ github.workspace }}/e2e, so strip the + # leading e2e/ to make the path resolve relative to that working directory. + relative_path="${matches#e2e/}" + echo "Resolved: $matches (relative to e2e/: $relative_path)" + echo "flow_path=$relative_path" >> "$GITHUB_OUTPUT" + echo "abs_path=$matches" >> "$GITHUB_OUTPUT" + + - name: Validate flow vs platform + if: inputs.flow != '' + env: + PLATFORM: ${{ inputs.platform }} + FLOW_PATH: ${{ steps.resolve.outputs.abs_path }} + FLOW_REL: ${{ steps.resolve.outputs.flow_path }} + run: | + # Read platform tags from the flow's YAML frontmatter. We only look between + # `tags:` and the first `---` so we don't pick up step-level dashes. + flow_tags=$(awk ' + /^tags:/ {flag=1; next} + /^---/ {exit} + flag && /^[[:space:]]*-[[:space:]]/ {gsub(/^[[:space:]]*-[[:space:]]*/, ""); print} + ' "$FLOW_PATH") + platform_tags=$(printf "%s\n" "$flow_tags" | grep -E '^(android|ios|web)$' || true) + + # No platform tags = cross-platform flow, allow. + if [ -z "$platform_tags" ]; then + echo "Flow $FLOW_REL has no platform tags; allowing on platform '$PLATFORM'." + exit 0 + fi + + # Platform must be one of the declared tags. + if ! printf "%s\n" "$platform_tags" | grep -qx "$PLATFORM"; then + tags_csv=$(printf "%s\n" "$platform_tags" | paste -sd, -) + echo "::error::Flow $FLOW_REL is tagged for [$tags_csv] but you requested platform '$PLATFORM'." + exit 1 + fi + build: - name: Build on Java ${{ matrix.java-version }} - runs-on: macos-latest + name: Build on ${{ matrix.os }} + needs: validate-inputs + if: ${{ always() && (needs.validate-inputs.result == 'success' || needs.validate-inputs.result == 'skipped') }} + runs-on: ${{ matrix.os }} timeout-minutes: 20 strategy: fail-fast: false matrix: + os: [ubuntu-latest, macos-latest, windows-latest] java-version: [17] + defaults: + run: + shell: bash + steps: - name: Clone repository uses: actions/checkout@v6 @@ -27,20 +166,22 @@ jobs: cache: gradle - name: Build xctest-runner + if: runner.os == 'macOS' run: ./maestro-ios-xctest-runner/build-maestro-ios-runner.sh | xcbeautify - name: Build Maestro CLI run: ./gradlew :maestro-cli:distZip - - name: Upload zipped Maestro CLI artifact - uses: actions/upload-artifact@v6 + - name: Upload Maestro CLI artifact + uses: actions/upload-artifact@v7 with: - name: maestro-cli-jdk${{ matrix.java-version }}-run_id${{ github.run_id }} + name: maestro-cli-${{ matrix.os }}-jdk${{ matrix.java-version }}-run_id${{ github.run_id }} path: maestro-cli/build/distributions/maestro.zip retention-days: 1 - name: Upload build/Products to artifacts - uses: actions/upload-artifact@v6 + if: runner.os == 'macOS' + uses: actions/upload-artifact@v7 with: name: build__Products-jdk${{ matrix.java-version }} path: build/Products @@ -49,12 +190,17 @@ jobs: test-web: name: Test on Web runs-on: ubuntu-latest - needs: build + needs: [build, validate-inputs] + if: ${{ always() && needs.build.result == 'success' && (needs.validate-inputs.result == 'success' || needs.validate-inputs.result == 'skipped') && (github.event_name == 'pull_request' || inputs.platform == 'web') && inputs.flow == '' }} + + env: + MAESTRO_APP: ${{ inputs.app }} steps: - name: Clone repository (only needed for the e2e directory) uses: actions/checkout@v6 + - name: Set up Java uses: actions/setup-java@v5 with: @@ -67,9 +213,9 @@ jobs: chrome-version: 142 - name: Download artifacts - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: - name: maestro-cli-jdk17-run_id${{ github.run_id }} + name: maestro-cli-macos-latest-jdk17-run_id${{ github.run_id }} - name: Add Maestro CLI executable to PATH run: | @@ -80,14 +226,14 @@ jobs: run: | maestro --help maestro --version - + - name: Run tests working-directory: ${{ github.workspace }}/e2e timeout-minutes: 20 - run: ./run_tests_web + run: ./run_tests web - name: Upload ~/.maestro artifacts - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: maestro-root-dir-web @@ -98,15 +244,17 @@ jobs: test-android: name: Test on Android runs-on: ubuntu-latest - needs: build + needs: [build, validate-inputs] + if: ${{ always() && needs.build.result == 'success' && (needs.validate-inputs.result == 'success' || needs.validate-inputs.result == 'skipped') && (github.event_name == 'pull_request' || inputs.platform == 'android') }} timeout-minutes: 60 env: ANDROID_HOME: /home/runner/androidsdk ANDROID_SDK_ROOT: /home/runner/androidsdk ANDROID_AVD_HOME: /home/runner/.config/.android/avd/ - ANDROID_OS_IMAGE: system-images;android-32;google_apis;x86_64 - ANDROID_PLATFORM: platforms;android-34 + ANDROID_OS_IMAGE: system-images;${{ inputs.android_version || 'android-32' }};google_apis;x86_64 + MAESTRO_APP: ${{ inputs.app }} + MAESTRO_FLOW_PATH: ${{ needs.validate-inputs.outputs.flow_path }} steps: - name: Enable KVM group perms @@ -118,27 +266,13 @@ jobs: - name: Clone repository (only needed for the e2e directory) uses: actions/checkout@v6 + - name: Set up Java uses: actions/setup-java@v5 with: distribution: zulu java-version: 17 - - name: Download Maestro build from previous job - uses: actions/download-artifact@v7 - with: - name: maestro-cli-jdk17-run_id${{ github.run_id }} - - - name: Add Maestro CLI executable to PATH - run: | - unzip maestro.zip -d maestro_extracted - echo "$PWD/maestro_extracted/maestro/bin" >> $GITHUB_PATH - - - name: Check if Maestro CLI executable starts up - run: | - maestro --help - maestro --version - - name: Set up mobile-dev-inc/bartek-scripts (for install_android_sdk script) run: | git clone https://github.com/mobile-dev-inc/bartek-scripts.git $HOME/scripts @@ -157,9 +291,58 @@ jobs: echo "$ANDROID_HOME/emulator" >> $GITHUB_PATH sdkmanager --install "platform-tools" echo "$ANDROID_HOME/platform-tools" >> $GITHUB_PATH - sdkmanager --install "$ANDROID_PLATFORM" sdkmanager --install "$ANDROID_OS_IMAGE" + - name: Cache pub + uses: actions/cache@v5 + with: + path: | + ~/.pub-cache + e2e/demo_app/.dart_tool + key: pub-${{ runner.os }}-${{ hashFiles('e2e/demo_app/pubspec.lock') }} + restore-keys: | + pub-${{ runner.os }}- + + - name: Cache Gradle + uses: actions/cache@v5 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + e2e/demo_app/android/.gradle + key: gradle-${{ runner.os }}-${{ hashFiles('e2e/demo_app/android/**/*.gradle*', 'e2e/demo_app/android/gradle/wrapper/gradle-wrapper.properties') }} + restore-keys: | + gradle-${{ runner.os }}- + + - name: Set up Flutter + uses: subosito/flutter-action@v2 + with: + channel: stable + flutter-version-file: e2e/demo_app/pubspec.yaml + + - name: Build demo_app APK + working-directory: ${{ github.workspace }}/e2e/demo_app + run: | + flutter pub get + flutter build apk + mkdir -p ${{ github.workspace }}/e2e/apps + cp build/app/outputs/flutter-apk/app-release.apk ${{ github.workspace }}/e2e/apps/demo_app.apk + + - name: Download Maestro build from previous job + uses: actions/download-artifact@v8 + with: + name: maestro-cli-macos-latest-jdk17-run_id${{ github.run_id }} + + - name: Add Maestro CLI executable to PATH + run: | + unzip maestro.zip -d maestro_extracted + echo "$PWD/maestro_extracted/maestro/bin" >> $GITHUB_PATH + + - name: Check if Maestro CLI executable starts up + run: | + maestro --help + maestro --version + - name: Create AVD run: | avdmanager -s create avd \ @@ -180,63 +363,38 @@ jobs: echo "HOME=$HOME" cat << EOF >> ~/.config/.android/avd/MyAVD.avd/config.ini - hw.cpu.ncore=2 + hw.cpu.ncore=4 hw.gpu.enabled=yes hw.gpu.mode=swiftshader_indirect - hw.ramSize=3072 - disk.dataPartition.size=4G - vm.heapSize=576 - hw.lcd.density=440 - hw.lcd.height=2220 - hw.lcd.width=1080 + hw.ramSize=4192 + disk.dataPartition.size=6442450944 + vm.heapSize=112M + hw.lcd.density=240 + hw.lcd.height=1280 + hw.lcd.width=606 EOF - name: Run AVD run: | emulator @MyAVD \ - -verbose -no-snapshot -no-window -no-audio -no-boot-anim -accel on -camera-back none -qemu -m 3072 \ + -verbose -no-snapshot -no-window -no-audio -no-boot-anim -accel on -no-metrics -camera-back emulated \ >~/emulator_stdout.log \ 2>~/emulator_stderr.log & - - name: Wait for AVD to start up - run: | - adb wait-for-device && echo 'Emulator device online' - adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;' && echo 'Emulator booted' - - # This is also a prerequiste - while true; do - adb shell service list | grep 'package' && echo 'service "package" is active!' && break - echo 'waiting for service "package" to start' - sleep 1 - done - - - name: Download apps - working-directory: ${{ github.workspace }}/e2e - run: ./download_apps android + - name: Wait for AVD to start up and configure CI pre-conditions + run: bash .github/scripts/setup_android_device.sh - name: Install apps working-directory: ${{ github.workspace }}/e2e run: ./install_apps android - - name: Start screen recording of AVD - run: | - adb shell screenrecord /sdcard/screenrecord.mp4 & - echo $! > ~/screenrecord.pid - - name: Run tests working-directory: ${{ github.workspace }}/e2e - timeout-minutes: 20 + timeout-minutes: 30 run: ./run_tests android - - name: Stop screen recording of AVD - if: success() || failure() - run: | - kill -SIGINT "$(cat ~/screenrecord.pid)" || echo "failed to kill screenrecord: code $?" && exit 0 - sleep 5 # prevent video file corruption - adb pull /sdcard/screenrecord.mp4 ~/screenrecord.mp4 - - name: Upload ~/.maestro artifacts - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: maestro-root-dir-android @@ -244,38 +402,47 @@ jobs: retention-days: 7 include-hidden-files: true - - name: Upload screen recording of AVD - uses: actions/upload-artifact@v6 - if: success() || failure() - with: - name: maestro-screenrecord-android.mp4 - path: ~/screenrecord.mp4 - retention-days: 7 - test-ios: name: Test on iOS runs-on: macos-26 - needs: build + needs: [build, validate-inputs] + if: ${{ always() && needs.build.result == 'success' && (needs.validate-inputs.result == 'success' || needs.validate-inputs.result == 'skipped') && (github.event_name == 'pull_request' || inputs.platform == 'ios') && inputs.flow == '' }} timeout-minutes: 120 env: MAESTRO_DRIVER_STARTUP_TIMEOUT: 240000 # 240s MAESTRO_CLI_LOG_PATTERN_CONSOLE: '%d{HH:mm:ss.SSS} [%5level] %logger.%method: %msg%n' + MAESTRO_APP: ${{ inputs.app }} steps: - name: Clone repository (only needed for the e2e directory) uses: actions/checkout@v6 + - name: Set up JDK uses: actions/setup-java@v5 with: distribution: zulu java-version: 17 + - name: Set up Flutter + uses: subosito/flutter-action@v2 + with: + channel: stable + flutter-version-file: e2e/demo_app/pubspec.yaml + + - name: Build demo_app for iOS Simulator + working-directory: ${{ github.workspace }}/e2e/demo_app + run: | + flutter pub get + flutter build ios --simulator + mkdir -p ${{ github.workspace }}/e2e/apps + cp -r build/ios/iphonesimulator/Runner.app ${{ github.workspace }}/e2e/apps/demo_app.app + - name: Download artifacts - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: - name: maestro-cli-jdk17-run_id${{ github.run_id }} + name: maestro-cli-macos-latest-jdk17-run_id${{ github.run_id }} - name: Add Maestro CLI executable to PATH run: | @@ -312,12 +479,26 @@ jobs: timeout-minutes: 120 run: ./run_tests ios + - name: Install xcodegen + # Required to generate the Xcode project for the SwiftUI fixture used + # by the preflight regression test below. + run: brew install xcodegen + + - name: Verify iOS UI-interruption preflight is suppressed + # Regression guard for the XCTest UI-interruption preflight hang. + # Builds a minimal SwiftUI app that presents a real UIAlertController, + # runs a gesture against it, and asserts Apple's preflight signature + # lines never appear in the xctest_runner log. Fails if the swizzle in + # maestro-ios-xctest-runner/maestro-driver-iosUITests/Categories/ + # XCUIApplication+Helper.m regresses. + run: ./e2e/alert-repro-swiftui/verify_preflight_suppressed.sh "$SIM_UDID" + - name: Stop screen recording if: success() || failure() run: kill -SIGINT "$(cat ~/screenrecord.pid)" - name: Upload ~/.maestro artifacts - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: maestro-root-dir-ios @@ -325,17 +506,8 @@ jobs: retention-days: 7 include-hidden-files: true - - name: Upload xctest runner logs - uses: actions/upload-artifact@v6 - if: success() || failure() - with: - name: xctest_runner_logs - path: ~/Library/Logs/maestro/xctest_runner_logs - retention-days: 7 - include-hidden-files: true - - name: Upload screen recording of Simulator - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: maestro-screenrecord-ios.mp4 @@ -360,12 +532,12 @@ jobs: java-version: 17 - name: Download Maestro artifact - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: - name: maestro-cli-jdk17-run_id${{ github.run_id }} + name: maestro-cli-macos-latest-jdk17-run_id${{ github.run_id }} - name: Download build/Products artifact - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: build__Products-jdk17 path: build/Products @@ -388,10 +560,64 @@ jobs: run: ./maestro-ios-xctest-runner/test-maestro-ios-runner.sh - name: Upload xc test runner logs - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: success() || failure() with: name: test-ios-xctest-runner__xctest_runner_logs path: ~/Library/Logs/maestro/xctest_runner_logs retention-days: 7 include-hidden-files: true + + test-cli: + name: Verify CLI version on ${{ matrix.os }} + runs-on: ${{ matrix.os }} + needs: build + if: ${{ always() && needs.build.result == 'success' }} + timeout-minutes: 10 + + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + java-version: [17] + + defaults: + run: + shell: bash + + steps: + - name: Clone repository + uses: actions/checkout@v6 + + - name: Set up Java + uses: actions/setup-java@v5 + with: + distribution: zulu + java-version: ${{ matrix.java-version }} + + - name: Download Maestro CLI artifact + uses: actions/download-artifact@v8 + with: + name: maestro-cli-${{ matrix.os }}-jdk${{ matrix.java-version }}-run_id${{ github.run_id }} + + - name: Add Maestro CLI to PATH (Linux / macOS) + if: runner.os != 'Windows' + run: | + unzip maestro.zip -d maestro_extracted + echo "$PWD/maestro_extracted/maestro/bin" >> $GITHUB_PATH + + - name: Add Maestro CLI to PATH (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: | + Expand-Archive -Path maestro.zip -DestinationPath maestro_extracted + "$PWD\maestro_extracted\maestro\bin" | Out-File -FilePath $env:GITHUB_PATH -Append + + - name: Test CLI (Linux / macOS) + if: runner.os != 'Windows' + run: e2e/cli/test-cli.sh + + - name: Test CLI (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: e2e/cli/test-cli.ps1 diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 54c213ed7a..a1cc66b9cb 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -16,10 +16,10 @@ jobs: steps: - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Java - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: distribution: zulu java-version: ${{ matrix.java-version }} @@ -30,7 +30,7 @@ jobs: run: ./gradlew test - name: Upload unit test report - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 if: failure() with: name: maestro-unit-test-report @@ -38,133 +38,47 @@ jobs: retention-days: 1 include-hidden-files: true - android-integration-test: - name: Android Integration Test on Java ${{ matrix.java-version }} - runs-on: ubuntu-latest - - strategy: - fail-fast: false - matrix: - java-version: [17] + ios-driver-lib-test: + name: MaestroDriverLib Unit Tests + runs-on: macos-latest + timeout-minutes: 10 steps: - # This is needed to enable hardware acceleration - - name: Enable KVM group perms - run: | - echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules - sudo udevadm control --reload-rules - sudo udevadm trigger --name-match=kvm - - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - - name: Set up Java - uses: actions/setup-java@v4 - with: - distribution: zulu - java-version: ${{ matrix.java-version }} - cache: gradle + - name: Run MaestroDriverLib Tests + working-directory: ${{ github.workspace }}/maestro-ios-xctest-runner/MaestroDriverLib + run: swift test - - name: Run Integration test - if: matrix.java-version == '17' - id: integration-test - uses: reactivecircus/android-emulator-runner@v2 - with: - api-level: 34 - arch: x86_64 - target: google_apis - script: | - # h/t https://gist.github.com/Tobias-Pe/52c8d6e162f75fb5dff5258a525fbf79 - adb logcat -c # clear logs - touch emulator.log # create log file - chmod 777 emulator.log # allow writing to log file - adb logcat >> emulator.log & # pipe all logcat messages into log file as a background process - ./gradlew integrationTest # run the tests - cp emulator.log ~/.maestro/emulator.log # copy emulator log with other debug artifacts - - - name: Upload Maestro integration debug artifacts - uses: actions/upload-artifact@v4 - if: failure() - with: - name: maestro-root-dir-android - path: ~/.maestro - retention-days: 1 - include-hidden-files: true - - - name: Upload integration test report - uses: actions/upload-artifact@v4 - if: failure() - with: - name: maestro-integration-test-report - path: ./**/build/reports/tests/integrationTest - retention-days: 1 - include-hidden-files: true - - ios-integration-test: - name: iOS Integration Test on Java ${{ matrix.java-version }} + ios-xctest-runner-test: + name: iOS XCTest Runner Unit Tests runs-on: macos-latest - timeout-minutes: 50 - - strategy: - fail-fast: false - matrix: - java-version: [17] + timeout-minutes: 15 steps: - - name: Clone repository (only needed for the e2e directory) - uses: actions/checkout@v4 - - - name: Set up Java - uses: actions/setup-java@v4 - with: - distribution: zulu - java-version: ${{ matrix.java-version }} - cache: gradle - - - name: Set permission to scripts - run: chmod +x .github/scripts/boot_simulator.sh - - - name: Boot Simulator - run: ./.github/scripts/boot_simulator.sh - - - name: Set Simulator - run: | - SIMULATOR_ID=$(xcrun simctl list -j | jq -r '.devices | to_entries[] | .value[] | select(.isAvailable==true and .state=="Booted") | .udid' | head -n 1) - echo "SIMULATOR_ID=$SIMULATOR_ID" >> $GITHUB_ENV - - - name: Download apps - working-directory: ${{ github.workspace }}/e2e - run: ./download_apps iOS - - - name: Install apps - working-directory: ${{ github.workspace }}/e2e - run: | - unzip apps/wikipedia.zip - xcrun simctl install $SIMULATOR_ID ./Wikipedia.app + - name: Clone repository + uses: actions/checkout@v6 - - name: Build for Testing - working-directory: ${{ github.workspace }} - run: | - xcodebuild clean build-for-testing \ - -project ./maestro-ios-xctest-runner/maestro-driver-ios.xcodeproj \ - -derivedDataPath "./driver-verification" \ - -scheme driver-verification-tests \ - -destination "generic/platform=iOS Simulator" + - name: Select Xcode + run: sudo xcode-select -s /Applications/Xcode.app/Contents/Developer - - name: Run Integration tests + - name: Run iOS Unit Tests + working-directory: ${{ github.workspace }}/maestro-ios-xctest-runner run: | - XCTESTRUN_FILE=$(find ./driver-verification/Build/Products -name "driver-verification-tests*.xctestrun" | head -n 1) - echo "Running tests on $SIMULATOR_ID" - xcodebuild test-without-building \ - -xctestrun "$XCTESTRUN_FILE" \ - -destination "id=$SIMULATOR_ID" + xcodebuild test \ + -project maestro-driver-ios.xcodeproj \ + -scheme maestro-driver-iosTests \ + -destination 'platform=iOS Simulator,name=iPhone 16' \ + -only-testing:maestro-driver-iosTests \ + | xcpretty --color || exit ${PIPESTATUS[0]} validate-gradle-wrapper: runs-on: ubuntu-latest steps: - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Validate Gradle Wrapper - uses: gradle/actions/wrapper-validation@v4 + uses: gradle/actions/wrapper-validation@v6 diff --git a/.github/workflows/update-samples.yaml b/.github/workflows/update-samples.yaml index 115801ee08..2882e0fdab 100644 --- a/.github/workflows/update-samples.yaml +++ b/.github/workflows/update-samples.yaml @@ -12,16 +12,16 @@ jobs: steps: - name: Clone repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Authenticate to Google Cloud - uses: google-github-actions/auth@v2 + uses: google-github-actions/auth@v3 with: # These credentials should only have write access to the bucket credentials_json: ${{ secrets.GCP_MOBILEDEV_BUCKET_CREDENTIALS }} - name: Set up Google Cloud CLI - uses: google-github-actions/setup-gcloud@v2 + uses: google-github-actions/setup-gcloud@v3 with: version: '>= 484.0.0' project_id: perf-dev-289002 diff --git a/.github/workflows/warn_build_xctestrunner.yaml b/.github/workflows/warn_build_xctestrunner.yaml deleted file mode 100644 index 9a2cde5799..0000000000 --- a/.github/workflows/warn_build_xctestrunner.yaml +++ /dev/null @@ -1,14 +0,0 @@ -name: Warn Build XCTest runner - -on: - pull_request: - paths: - - 'maestro-ios-xctest-runner/**' - -jobs: - warn: - runs-on: ubuntu-latest - steps: - - uses: mshick/add-pr-comment@v2 - with: - message: "Make sure to run ./maestro-ios-xctest-runner/build-maestro-ios-runner.sh with every swift change" diff --git a/.idea/.gitignore b/.idea/.gitignore index 10a0b510ed..5cd9b02653 100644 --- a/.idea/.gitignore +++ b/.idea/.gitignore @@ -27,4 +27,6 @@ gradle.xml kotlinc.xml vcs.xml -copilot.data.migration.* \ No newline at end of file +copilot.data.migration.* + +libraries/ \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 0000000000..ecd2870752 --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..9d56f63274 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,126 @@ +# AGENTS.md — Maestro + +Shared context for any Claude Code skill or subagent operating in this repo. Skills (`.claude/skills/*`) reference this file rather than restating module roles; if a description here drifts from reality, fix it here once and every skill follows. + +## Module map + +Top-level Gradle modules. Code lives under each module's `src/main/`. + +| Module | Role | +|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `maestro-android/` | On-device Android driver. Kotlin sources compile to two checked-in APKs (`maestro-app.apk`, `maestro-server.apk`) consumed by `maestro-client/`. The build's `copyMaestroAndroid` / `copyMaestroServer` finalizers update those APKs plus a `maestro-android-source.sha256` checksum. | +| `maestro-ios-driver/` | Host side of iOS driver wrapper (Kotlin). The actual XCTest runner lives in `maestro-ios-xctest-runner/`. | +| `maestro-ios-xctest-runner/` | Swift XCTest runner that runs on the iOS device/simulator. The compiled artifacts (`maestro-driver-ios*.zip`) are checked in under `maestro-ios-driver/src/main/resources/driver-iPhoneSimulator/Debug-iphonesimulator/`. | +| `maestro-ios/` | iOS host-side glue (small — most iOS host code lives in `maestro-client/`). | +| `maestro-client/` | Host-side Kotlin SDK that drives devices. Platform drivers live in `src/main/java/maestro/drivers/`: `AndroidDriver.kt`, `IOSDriver.kt`, `WebDriver.kt`, `CdpWebDriver.kt`. This is where most "auto-grant", "auto-dismiss", system-dialog handling and platform-specific quirks belong. | +| `maestro-orchestra/` | Command execution layer. `Orchestra.kt` interprets each Maestro command, applies retries, manages the command lifecycle. Sub-packages: `error/`, `filter/`, `workspace/`, `yaml/`. | +| `maestro-orchestra-models/` | Shared command/data models (used by `maestro-orchestra/` and consumers). | +| `maestro-cli/` | CLI entry point + MCP server. Mixed Kotlin (~100 files) + Swift (~56 files for iOS-related CLI bits). | +| `maestro-utils/` | Shared utilities. | +| `maestro-web/` | Web (browser) driver pieces. | +| `maestro-proto/` | Protobuf definitions shared across modules. | +| `maestro-test/` | Cross-module tests that doesn't require devices. | + +## E2E test fixtures (`e2e/`) + +Shipped fixtures used by `.github/workflows/test-e2e.yaml`. Run via `e2e/run_tests ` (see `e2e/run_tests` for env-var inputs `MAESTRO_APP`, `MAESTRO_FLOW_PATH`). + +| Path | Role | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `e2e/demo_app/` | Flutter demo app whose only purpose is to exercise Maestro features. Contains its own `CLAUDE.md`. Built binaries are uploaded to a GCS bucket and re-downloaded by CI. | +| `e2e/demo_app/.maestro/` | Maestro flow YAMLs that drive the demo app. | +| `e2e/workspaces/` | Additional app workspaces (e.g. `simple_web_view`, `wikipedia`). | +| `e2e/run_tests` | Test driver invoked by the workflow. | + +### `passing/` vs `failing/` suites + +Tag-based filters inside the YAML flows split test runs into two suites at execution time: + +- `passing/` — flows tagged `passing`. **Expected to pass.** Any failure here is a real regression. This is the only suite the diagnose agent reads. +- `failing/` — flows tagged `failing`. **Expected to fail** (negative-path coverage: assertions that should not match, commands that should error). The workflow inverts the success check on this suite. Do not treat `failing/` artifacts as regressions. + +Artifacts land at `/tests///`: + +## `test-e2e.yaml` workflow contract + +`.github/workflows/test-e2e.yaml` is the validation harness for both PR triggers and manual `workflow_dispatch` (e.g. validating a new Android API level or iOS version). Contract: + +- **`workflow_dispatch` inputs** — `android_version` (choice enum), `app` (string, default `demo_app`), `flow` (string, optional single-flow). The `validate-inputs` job rejects `android_version <= android-29`, missing `app` workspace, or ambiguous `flow`. (See PR #3226.) +- **`pull_request` triggers** are byte-identical to the prior behaviour; manual dispatches use the new narrowing knobs. +- **`test-android` job** boots an emulator on `system-images;${android_version};google_apis;x86_64` and runs `e2e/run_tests android`. + +Skills that bump platform versions (Android API levels, iOS versions) drive this workflow via `gh workflow run test-e2e.yaml --ref -f android_version=<...>`. + +## Testing + +Three layers — unit, integration, E2E — plus MCP-specific evals. Each layer has a different cost/coverage trade-off; default to the lowest layer that can express the test. + +### Unit tests (per module, `src/test/kotlin/`) + +Standard per-class tests. Stack: **JUnit 5** (`junit-jupiter-api` + `-params` + `-engine`), **Google Truth** for assertions, **MockK** for mocks. Each module's `build.gradle.kts` enables the platform via `tasks.named("test") { useJUnitPlatform() }`. + +```bash +./gradlew :maestro-orchestra:test # one module +./gradlew test # all modules +``` + +### Integration tests (`maestro-test/`) + +Cross-module tests for behaviour that does **not** require a device or simulator — JS engine integration points, command orchestration end-to-end, cancellation / coroutine semantics. Notable suites: + +- `IntegrationTest.kt` — full `Maestro` orchestration against an in-process `FakeDriver` (defined in `maestro-test/src/main/kotlin/maestro/test/drivers/`: `FakeDriver`, `FakeLayoutElement`, `FakeTimer`). Covers test-run cancellation (`CancellationException`, `withTimeout`, supervisor scopes) and the full command lifecycle without a real device. +- `GraalJsEngineTest.kt` / shared `JsEngineTest.kt` — Maestro's JS extension points (`evalScript`, JS-evaluated assertions/conditions). Exercises `org.graalvm.polyglot` directly. +- `FlowControllerTest.kt`, `DeepestMatchingElementTest.kt` — orchestration and view-hierarchy logic. + +Stack: **JUnit 5**, **Google Truth**, **WireMock JRE8** (HTTP fakes), plus the in-house `FakeDriver` fixtures listed above. No mocks of Maestro's own classes — tests run real `Maestro` against the fakes. + +```bash +./gradlew :maestro-test:test +``` + +### E2E tests (`e2e/`) + +Smoke-test every Maestro command across Android, iOS, and Web on real fixture apps. Maestro is its own dogfood harness: the CLI executes Maestro flow YAMLs against the fixtures, asserting both the framework's commands and the platform drivers behave correctly. + +Stack: **Maestro CLI itself** (dogfood) + `e2e/run_tests` shell driver + GHA workflow (`.github/workflows/test-e2e.yaml`). Fixture and suite layout is in "E2E test fixtures (`e2e/`)" above. + +```bash +cd e2e && ./run_tests # local +gh workflow run test-e2e.yaml --ref -f android_version=android- # CI +``` + +**Two roles for the same E2E setup.** The same suite serves both purposes — treat them identically: + +1. **Regression smoke** — every PR that touches Maestro source runs the suite on the current platform versions, catching behaviour breakage on existing platforms. +2. **New-OS validation** — when launching a new Android API level or iOS version, the same flows are dispatched against the new system image to confirm Maestro still works. This is what `bump-android-version` (and the planned `bump-ios-version`) drives. + +A flow breaking for either reason is a real regression — fix in `maestro-android/`, `maestro-client/`, or `e2e/demo_app/`, not in `test-e2e.yaml` (see "What NOT to do"). + +**Multiple apps for framework-specific coverage.** `demo_app/` (Flutter) is the default fixture and exercises every Maestro command. When a target is **framework-specific** (SwiftUI, React Native, Jetpack Compose specifics, WebView quirks, etc.), add a separate workspace under `e2e/workspaces//` with its own `.maestro/` flow YAMLs and a binary under `e2e/apps/`. Existing examples: `simple_web_view` (WebView coverage), `wikipedia` (real-world third-party app). The workflow's `app` input narrows a manual dispatch to one workspace: `... -f app=simple_web_view`. + +### MCP server evals (`maestro-cli/src/test/mcp/`) + +LLM-behaviour evaluations and tool-functionality tests for the MCP server inside `maestro-cli`. Stack: **`mcp-server-tester`** (npm package, run via `npx`) consuming YAML definitions (`full-evals.yaml`, `inspect-screen-evals.yaml`, `tool-tests-{with,without}-device.yaml`) plus per-platform setup scripts under `setup/`. See `maestro-cli/src/test/mcp/README.md` for the model list, scorers, and how to run. + +```bash +./run_mcp_tool_tests.sh ios # tool-functionality (fast) +./run_mcp_evals.sh ios # LLM behaviour (slower) +``` + +## Conventions + +- Kotlin 1.9 / JVM 17. Gradle. No DI framework — services are constructed manually. +- Protobuf for the on-device wire format (`maestro-proto/`). +- Coroutines with explicit dispatchers; `runBlocking` only at entry points. +- Exposed exceptions classify failures (retryable vs terminal) — see `maestro-orchestra/src/main/java/maestro/orchestra/error/`. + +## Where Claude Code resources live + +- `.claude/skills/*` — skills (workflows). Each skill's `SKILL.md` references this file for module roles. +- `.claude/agents/*.md` — subagents (e.g. `diagnose-maestro-failure.md`). Their input/output contracts are documented in each file. + +## What NOT to do + +- Don't fix driver-behaviour gaps by patching `.github/workflows/test-e2e.yaml` (e.g. extra `adb shell settings put …`, command-line tweaks, AVD pre-config). Workflow band-aids hide the regression from users running Maestro outside our CI. Fix `maestro-android/`, `maestro-client/`, or `e2e/demo_app/` instead so the fix ships with the driver APKs. Workflow edits are valid for shape-changes (matrix, retention, dispatch inputs) and the narrow third-party-FRE exception documented in skill files. +- Don't edit checked-in driver artifacts (`maestro-app.apk`, `maestro-server.apk`, `maestro-android-source.sha256`, `maestro-driver-ios*.zip`) by hand — they are gradle finalizer outputs. +- Don't modify existing flows in `failing/` to make them pass — that's the negative-path suite by design. diff --git a/CHANGELOG.md b/CHANGELOG.md index 91f08059b9..fc7bdcbbbd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,135 @@ ## Unreleased +## 2.6.0 + +- Add Maestro Viewer - a desktop visualizer for live hierarchy and commands +- Remove bundled Maestro Studio from the CLI in favor of the Maestro Studio desktop app +- Remove the deprecated Rhino JS engine +- Enable parallel iOS simulator execution with per-device session tracking +- Added iOS XCTest logs to debug output directory +- Improve YAML parsing error outputs +- Improved rendering of relative paths when using `runScript` commands +- Improved rendering of the source of failing commands +- Improved rendering of JS errors +- Make `maestro hierarchy` command output only JSON +- Improve performance when cancelling a screenshot operation +- Remove unnecessary device interaction after running commands that won't affect the app (e.g. `assertVisible`) +- Fix `assertScreenshot` match percent so it aligns with the configured threshold +- Add `file` attribute to JUnit report +- Make JUnit XML `id` and `classname` attributes configurable +- Include cloud URL and app binary ID in the HTML report +- Stop assuming a lowercase `.yaml` extension when parsing flow names +- MCP `cheat_sheet` works without Maestro Cloud authentication +- MCP `take_screenshot` downscales screenshots to 2000px +- Throw the correct test error when a timeout value is invalid +- Fail fast with a helpful message when an env value is null +- Warn when the deprecated `--android-api-level` / `--ios-version` flags are used +- Fix `inputText` for native date inputs on web +- Swallow CDP `ServiceConfigurationError` in `WebDriver.open()` +- Fix `androidWebViewHierarchy: devtools` by preserving hostname in `DummyDns` +- Use emoji in Windows terminal output - consistent with other platforms +- Bump Selenium to 4.43.0 and CDP bindings to v147 +- Additional CI for building Maestro on multiple OSes and adding console tests + +Thanks to @LVSant, @hSATAC, @mikamikaWolt, @dezsibalint, @sidferreira and @qwertey6 who contributed changes included in this release ❤️ + +## 2.5.1 + +- Fix `NoSuchMethodError` on Ktor server startup (login, studio, and cloud commands) +- Fix emulator discovery on Windows, restoring `start-device` functionality +- Improve Windows developer experience +- Improve device info help in `maestro cloud --help` +- Fix detection of successful emulator installation in `start-device` +- Improve CLI help for the `start-device` command +- Fail-fast on iOS transport failure instead of rethrowing raw `SocketTimeoutException` + +## 2.5.0 + +- MCP server improvements + - Consolidate `run_flow` and `run_flow_files` into a single `run` tool, mirroring the shape of `maestro test` + - Drop 8 redundant tools (`tap_on`, `input_text`, `back`, `launch_app`, `stop_app`, `check_flow_syntax`, `start_device`, `query_docs`), shrinking the surface from 15 to 8 + - Add native Maestro Cloud tools: `run_on_cloud` submits a flow + app binary asynchronously and returns a dashboard URL; `get_cloud_run_status` polls status and per-flow results + - Add `list_cloud_devices` and a `device_model` parameter on `run_on_cloud`, so agents can look up valid `{platform, model, supported_os}` triples instead of guessing + - Add server instructions and a clean stdout handshake - MCP startup no longer corrupts the JSON-RPC channel, fixing `Unexpected token 'k'` errors on strict clients like Claude Desktop + - Drop `use_fuzzy_matching` from the `tap_on` schema - the flag had no YAML equivalent and produced surprising matches + - A compact JSON output by default + - Upgrade MCP Kotlin SDK to the official 0.11.1 +- Fix `--config=` on `maestro cloud` for any filename other than `config.yaml`/`config.yml`, and allow configs to live outside the workspace directory +- Detect workspace config files by content (top-level YAML keys) instead of filename, so files like `platform_settings.yaml` are no longer mis-parsed as flows +- Fix workspace ZIP path traversal for single-flow cloud uploads - zip entries are now rooted at the deepest common ancestor, eliminating broken `../` segments +- Fix `--android-api-level` and `--ios-version` being silently ignored on cloud uploads +- Normalize resolved workspace paths so `..` segments collapse on zip filesystems +- Fix a file dependency resolution problem when users provided absolute paths +- Show upload progress in `maestro cloud` CLI output for the whole upload (workspace zip + app binary + mapping), not just the app binary +- Add a CLI warning when the workspace zip produced for `maestro cloud` exceeds 20 MB, since oversized uploads slow queue and run times across the fleet +- Replace Android TCP port forwarding with a direct ADB socket for gRPC - around 41% faster on a simple `assertVisible` flow, and removes the flaky forwarding thread that fought with Studio/CLI/MCP running side-by-side +- Fixes to cancelling flow execution when inside a long-running command +- Allow JavaScript expressions in the `timeout` field of `waitForAnimationToEnd` +- Add missing string-command config for `inputRandomColorName` and related random-input commands +- Detect Azure Pipelines as a CI provider (via `TF_BUILD`), so promotional messages are suppressed +- Populate the `text` attribute for `