From 015f9144365835cc1260e1ba8d38460d308ad8ff Mon Sep 17 00:00:00 2001 From: JP Cottin Date: Sun, 26 Jul 2026 12:02:00 -0700 Subject: [PATCH 1/2] Add a local replay harness for the Emulator Preview multi-run job Pushing to see how the preview emulator behaves is a 25-minute round trip. scripts/replay-preview-multirun.sh runs the same boot/snapshot cycles on a dev machine in a few minutes: create the AVD, install the APK with auto-play enabled, play, screenshot, shut down so the emulator writes its quickboot snapshot, and repeat. It reports the app pid per cycle, which is the property the experiment exists to check. It is written for a developer machine rather than a runner, so unlike CI it cannot assume it owns the host: - every adb call is pinned to the emulator it launched. The emulator is identified as the device that appears after launch, not by assuming emulator-5554 -- another emulator may already hold the default ports, and adb may still list a stale entry for one that has exited. If ours never appears and the log shows the console port was taken, it says so instead of driving somebody else's device. - shutdown is scoped to the emulator's own process group. - an AVD that already existed is reused and never deleted; only one the script created is cleaned up, and -k keeps that too. - an existing console auth token is left untouched. README gains the local-replay instructions and an updated CI/CD table: it still described three jobs, and the smoke matrix had grown to cover API 37.0 and 37.1 across two GPU backends and both channels. --- README.md | 35 +++- scripts/replay-preview-multirun.sh | 290 +++++++++++++++++++++++++++++ 2 files changed, 323 insertions(+), 2 deletions(-) create mode 100755 scripts/replay-preview-multirun.sh diff --git a/README.md b/README.md index ad7c715..6378626 100644 --- a/README.md +++ b/README.md @@ -238,13 +238,44 @@ screenshot: ## CI/CD -Three GitHub Actions jobs run on every push and pull request to `main`: +GitHub Actions runs on every push and pull request to `main`. The first three +jobs gate merges. The last three explore newer Android emulator tooling and are +marked `continue-on-error`, so a preview package that moves underneath us +reports its findings without ever blocking a PR. | Job | What it does | Artifacts | |-----|-------------|-----------| | **Build APK** | Compiles the debug APK | `debug-apk` | | **Native Tests** | Runs the 103 Google Test cases on x86\_64 emulators (API 34 + API 36) | — | -| **Smoke Test** | Runs the Android instrumented test on x86\_64 emulators and captures an in-game screenshot via `UiAutomation`. The test asserts the activity is RESUMED **and** that the renderer logged `Swapchain ready`, so a dead Vulkan path fails loudly. Blocking on API 34 + 36; non-blocking preview legs on API 37.0 (`google_apis_ps16k`, 16 KB pages) across the swiftshader / lavapipe / auto GPU backends | `smoke-screenshot-api*`, `smoke-test-results-api*`, `smoke-logcat-api*` (suffixed per leg) | +| **Smoke Test** | Runs the Android instrumented test on x86\_64 emulators and captures an in-game screenshot via `UiAutomation`. The test asserts the activity is RESUMED **and** that the renderer logged `Swapchain ready`, so a dead Vulkan path fails loudly. Blocking on API 34 + 36; non-blocking legs on API 37.0 and 37.1 (`google_apis_ps16k`, 16 KB pages) across the lavapipe and auto GPU backends, from both the stable and canary channels | `smoke-screenshot-api*`, `smoke-test-results-api*`, `smoke-logcat-api*` (suffixed per leg) | +| **Android CLI experiment** | Drives the same instrumented test through the `android` CLI — SDK install, AVD creation, boot and teardown — instead of `sdkmanager`/`avdmanager` plus the emulator-runner action | `cli-smoke-*` | +| **Emulator Preview** | Boots the Android Emulator Preview package (`emulators;latest`, which installs alongside the stable emulator under `emulators/latest/`) and runs the instrumented test against it | `preview-smoke-*` | +| **Emulator Preview multi-run** | Four boot cycles against the same AVD with quickboot snapshots enabled: each cycle plays the game briefly, screenshots it, then shuts down so the emulator saves its snapshot. Checks whether a live Vulkan app survives snapshot save/restore — it does: the app keeps its pid and the game continues across cycles | `preview-multirun-screenshots`, `preview-multirun-emulator-logs` | + +The preview jobs share their setup through the composite action in +`.github/actions/preview-emulator`, which installs the system image, creates +the AVD, installs the preview emulator and its host dependencies. + +### Replaying the Emulator Preview job locally + +Pushing to see what a preview emulator does is a slow way to iterate, so the +multi-run job can be replayed on a local machine in a few minutes: + +```bash +scripts/replay-preview-multirun.sh # 4 cycles, throwaway AVD, cleaned up after +scripts/replay-preview-multirun.sh -n 2 -k # 2 cycles, keep the AVD for inspection +scripts/replay-preview-multirun.sh -h # options +``` + +It needs `emulators;latest`, the API 37.0 `google_apis_ps16k` system image and +KVM. Every `adb` call is pinned to the emulator it launches and shutdown is +scoped to that emulator's process group, so it is safe to run while other +devices or emulators are attached. + +The script writes `~/.emulator_console_auth_token` if you do not already have +one. The emulator console authenticates against that file before it offers its +full command set, including `kill` — which is how both the script and CI ask +the emulator to shut down cleanly so that it writes its quickboot snapshot. ## License diff --git a/scripts/replay-preview-multirun.sh b/scripts/replay-preview-multirun.sh new file mode 100755 index 0000000..3af5b44 --- /dev/null +++ b/scripts/replay-preview-multirun.sh @@ -0,0 +1,290 @@ +#!/usr/bin/env bash +# +# Replay the CI job "Emulator Preview experiment multi-run" on this machine. +# +# The job boots the Android Emulator Preview (emulators;latest) several times +# WITHOUT -no-snapshot: each cycle plays the game for a few seconds, screenshots +# it, then shuts down gracefully so the emulator writes its quickboot snapshot. +# If restore preserves the running app, the app keeps the same pid across cycles +# and the game continues where it left off. +# +# Running it here instead of pushing turns a ~25 minute CI round trip into a few +# minutes, which matters when iterating on the emulator itself. +# +# Usage: +# scripts/replay-preview-multirun.sh [-n RUNS] [-a AVD_NAME] [-o OUTDIR] [-k] +# +# -n RUNS number of boot/snapshot cycles (default 4) +# -a NAME AVD to create and use (default preview_replay) +# -o DIR output directory for screenshots and logs (default /tmp/preview-replay) +# -k keep the AVD afterwards (default: delete it -- it is several GB) +# +# Requirements: $ANDROID_HOME with the emulators;latest package and the +# system image below, plus KVM. Install with: +# sdkmanager --channel=3 --install 'emulators;latest' \ +# 'system-images;android-37.0;google_apis_ps16k;x86_64' +set -u + +RUNS=4 +AVD=preview_replay +OUT=/tmp/preview-replay +KEEP=0 +while getopts ':n:a:o:kh' opt; do + case "$opt" in + n) RUNS="$OPTARG" ;; + a) AVD="$OPTARG" ;; + o) OUT="$OPTARG" ;; + k) KEEP=1 ;; + h) sed -n '2,28p' "$0"; exit 0 ;; + *) echo "unknown option -$OPTARG (try -h)" >&2; exit 2 ;; + esac +done + +case "$AVD" in + ''|*/*) echo "ERROR: -a needs a plain AVD name" >&2; exit 2 ;; +esac +case "$RUNS" in + ''|*[!0-9]*|0) echo "ERROR: -n needs a positive number of cycles" >&2; exit 2 ;; +esac + +SDK="${ANDROID_HOME:-${ANDROID_SDK_ROOT:-$HOME/Android/Sdk}}" +export ANDROID_HOME="$SDK" ANDROID_SDK_ROOT="$SDK" +ADB="$SDK/platform-tools/adb" +EMULATOR="$SDK/emulators/latest/emulator" +API=37.0 +TARGET=google_apis_ps16k +ABI=x86_64 +PKG=com.jpcottin.vulkanspaceinvaders +REPO="$(cd "$(dirname "$0")/.." && pwd)" +APK="$REPO/app/build/outputs/apk/debug/app-debug.apk" + +[ -x "$EMULATOR" ] || { echo "ERROR: no preview emulator at $EMULATOR"; exit 1; } +[ -x "$ADB" ] || { echo "ERROR: no adb at $ADB"; exit 1; } +[ -e /dev/kvm ] || echo "WARNING: /dev/kvm missing; the emulator will be extremely slow" + +mkdir -p "$OUT/screenshots" +rm -f "$OUT"/screenshots/*.png "$OUT"/emulator_run*.txt "$OUT"/logcat_run*.txt "$OUT"/pids.txt + +# This host may have other devices on adb (another emulator, a Cuttlefish +# instance, a phone). Every adb call is pinned to the emulator we launch, and +# process cleanup is scoped to its own process group -- never a pattern match +# across all emulator processes. +SERIAL="" +A() { "$ADB" -s "$SERIAL" "$@"; } +SELF_PGID="$(ps -o pgid= -p $$ | tr -d ' ')" + +banner() { echo; echo "################ $* ################"; } + +# The emulator console starts unauthenticated, where it offers only +# help/ping/auth/quit/avd -- 'kill' and 'avd pause' are not available, so +# 'adb emu kill' returns "KO: unknown command". The emulator reads this token +# file but does not create it, so create it if absent. An existing token is +# left alone. +TOKEN="$HOME/.emulator_console_auth_token" +if [ ! -s "$TOKEN" ]; then + printf 'replayConsoleToken' > "$TOKEN" + chmod 600 "$TOKEN" + echo "created $TOKEN" +fi + +banner "SETUP: AVD $AVD" +# Only an AVD this script created may be deleted at the end -- never one that +# was already on the machine, even if -a named it explicitly. +CREATED_AVD=0 +if [ -d "$HOME/.android/avd/$AVD.avd" ]; then + echo "reusing the existing AVD $AVD (it will be left in place)" +else + CREATED_AVD=1 +fi +if [ ! -d "$HOME/.android/avd/$AVD.avd" ]; then + # avdmanager silently produces nothing once emulators;latest is installed, + # so write the AVD files directly. This mirrors what the CI job does. + echo no | "$SDK/cmdline-tools/latest/bin/avdmanager" create avd --force -n "$AVD" \ + --abi "$TARGET/$ABI" --device 'pixel_6' \ + --package "system-images;android-$API;$TARGET;$ABI" >/dev/null 2>&1 || true +fi +if [ ! -d "$HOME/.android/avd/$AVD.avd" ]; then + echo "avdmanager produced nothing; writing the AVD files by hand" + mkdir -p "$HOME/.android/avd/$AVD.avd" + printf 'avd.ini.encoding=UTF-8\npath=%s/.android/avd/%s.avd\npath.rel=avd/%s.avd\ntarget=android-%s\n' \ + "$HOME" "$AVD" "$AVD" "$API" > "$HOME/.android/avd/$AVD.ini" + cat > "$HOME/.android/avd/$AVD.avd/config.ini" << CFG +AvdId=$AVD +avd.ini.displayname=$AVD +avd.ini.encoding=UTF-8 +abi.type=$ABI +hw.cpu.arch=$ABI +image.sysdir.1=system-images/android-$API/$TARGET/$ABI/ +tag.id=google_apis +tag.display=Google APIs +PlayStore.enabled=no +hw.lcd.density=420 +hw.lcd.width=1080 +hw.lcd.height=2400 +hw.keyboard=yes +hw.gpu.enabled=yes +hw.gpu.mode=auto +disk.dataPartition.size=8192M +hw.ramSize=4096 +hw.cpu.ncore=4 +CFG +fi + +if [ ! -f "$APK" ]; then + banner "Building the debug APK" + (cd "$REPO" && ./gradlew assembleDebug --no-daemon) || exit 1 +fi + +cleanup() { + # On an interrupted run, take down the emulator we started -- by process + # group when we managed to isolate one, otherwise by pid. + if [ -n "${EMU_PGID:-}" ] && kill -0 -- "-$EMU_PGID" 2>/dev/null; then + kill -TERM -- "-$EMU_PGID" 2>/dev/null || true + elif [ -n "${EMU_PID:-}" ] && kill -0 "$EMU_PID" 2>/dev/null; then + kill -TERM "$EMU_PID" 2>/dev/null || true + fi +} +trap cleanup EXIT INT TERM + +boot_and_play() { + N="$1" + banner "RUN $N: launching emulator (snapshots enabled)" + # Record the emulators already attached, so we can identify ours as the one + # that appears afterwards. Never assume emulator-5554 is ours: this machine + # may already be running an emulator on the default ports. + DEVICES_BEFORE="$("$ADB" devices | awk '/^emulator-/{print $1}' | sort)" + # No -no-snapshot: load the quickboot snapshot if present, save it on exit. + setsid "$EMULATOR" @"$AVD" -no-window -gpu auto -noaudio -no-boot-anim \ + -camera-back none -memory 4096 -verbose -show-kernel \ + > "$OUT/emulator_run$N.txt" 2>&1 & + EMU_PID=$! + # setsid(2) happens asynchronously, so poll until the process group settles. + EMU_PGID="" + for _ in $(seq 1 20); do + P="$(ps -o pgid= -p "$EMU_PID" 2>/dev/null | tr -d ' ')" + if [ -n "$P" ] && [ "$P" != "$SELF_PGID" ]; then EMU_PGID="$P"; break; fi + kill -0 "$EMU_PID" 2>/dev/null || break + sleep 0.5 + done + [ -n "$EMU_PGID" ] || echo "WARNING: could not isolate a process group; using single-pid shutdown" + + SERIAL="" + for _ in $(seq 1 36); do + kill -0 "$EMU_PID" 2>/dev/null || { echo "ERROR: emulator exited early"; tail -30 "$OUT/emulator_run$N.txt"; return 1; } + cand="$(comm -13 <(printf '%s\n' "$DEVICES_BEFORE") \ + <("$ADB" devices | awk '/^emulator-/{print $1}' | sort) | head -1)" + [ -n "$cand" ] && { SERIAL="$cand"; break; } + sleep 5 + done + if [ -z "$SERIAL" ]; then + echo "ERROR: the emulator we launched never appeared on adb after 180s" + if grep -q "address already in use" "$OUT/emulator_run$N.txt" 2>/dev/null; then + echo " its console port is taken -- another emulator is already running:" + "$ADB" devices | sed 's/^/ /' + echo " stop it, or free the console ports, and try again." + fi + return 1 + fi + echo "RUN $N serial=$SERIAL pid=$EMU_PID pgid=${EMU_PGID:-none}" + + for _ in $(seq 1 48); do + [ "$(A shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" = "1" ] && break + sleep 10 + done + [ "$(A shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" = "1" ] || { echo "ERROR: boot timeout"; return 1; } + T_RESUME=$(date +%s) + + if [ "$N" = 1 ]; then + # A guest error dialog would block all input and be preserved by the + # snapshot, so suppress dialogs for the duration of the replay. + A shell settings put global hide_error_dialogs 1 >/dev/null || true + A install -r "$APK" >/dev/null || return 1 + # Auto-play must be on BEFORE the first start: the game reads + # /settings.bin at startup (magic "SETT" LE, soundOn=0, autoPlay=1). + A shell "run-as $PKG sh -c 'mkdir -p files; echo VFRFUwAAAAABAAAA | base64 -d > files/settings.bin'" + A shell input keyevent KEYCODE_WAKEUP >/dev/null || true + A shell wm dismiss-keyguard >/dev/null || true + else + # The restored frame, before any input: should match the previous cycle's + # exit screenshot, which was taken immediately before that cycle froze. + A exec-out screencap -p > "$OUT/screenshots/run$N-entry.png" || true + fi + + RUN_PID="$(A shell pidof "$PKG" 2>/dev/null | tr -d '\r')" + echo "RUN $N app pid after boot/restore: ${RUN_PID:-none}" + echo "cycle $N: ${RUN_PID:-none}" >> "$OUT/pids.txt" + A shell am start -n "$PKG/android.app.NativeActivity" >/dev/null 2>&1 || true + for _ in $(seq 1 30); do + A shell dumpsys window 2>/dev/null | grep -qi "ocus.*vulkanspaceinvaders" && break + sleep 1 + done + sleep 3 + # The TITLE screen needs a tap to start a round; auto-play only flies during + # PLAYING. Taps are spaced out because input to an unfocused window is dropped. + for _ in 1 2 3 4; do A shell input tap 540 1500 >/dev/null || true; sleep 2; done + sleep 5 + + A logcat -d -t 500 > "$OUT/logcat_run$N.txt" 2>/dev/null || true + # Liveness: two captures 2s apart must differ, or rendering has stalled. + A exec-out screencap -p > "$OUT/liveness-a.png" || true + sleep 2 + A exec-out screencap -p > "$OUT/liveness-b.png" || true + if cmp -s "$OUT/liveness-a.png" "$OUT/liveness-b.png"; then + echo "RUN $N display: FROZEN (two captures 2s apart are identical)" + else + echo "RUN $N display: LIVE" + fi + + # Exit screenshot last, immediately before shutdown, so it is as close as + # possible to the state the snapshot captures. A few seconds still elapse + # while the emulator shuts down and the guest keeps running, so the next + # cycle's entry screenshot is close to this one rather than identical. + A exec-out screencap -p > "$OUT/screenshots/run$N.png" || true + echo "RUN $N app pid before shutdown: $(A shell pidof "$PKG" 2>/dev/null | tr -d '\r' || echo none)" + + T_KILL=$(date +%s) + echo "RUN $N: shutting down via adb emu kill" + A emu kill >/dev/null 2>&1 || true + for _ in $(seq 1 90); do kill -0 "$EMU_PID" 2>/dev/null || break; sleep 1; done + if kill -0 "$EMU_PID" 2>/dev/null; then + echo "WARNING: still up 90s after 'adb emu kill'; terminating" + if [ -n "$EMU_PGID" ]; then kill -TERM -- "-$EMU_PGID" 2>/dev/null || true + else kill "$EMU_PID" 2>/dev/null || true; fi + sleep 10 + fi + if [ -n "$EMU_PGID" ]; then + for _ in $(seq 1 120); do kill -0 -- "-$EMU_PGID" 2>/dev/null || break; sleep 1; done + fi + EMU_PGID="" + echo "RUN $N: down $(( $(date +%s) - T_KILL ))s after the shutdown request (played ~$(( T_KILL - T_RESUME ))s)" + LD_LIBRARY_PATH="$SDK/emulators/latest/lib64" \ + "$SDK/emulators/latest/bin/qemu-img" snapshot -l \ + "$HOME/.android/avd/$AVD.avd/userdata-qemu.img.qcow2" 2>&1 | sed 's/^/ /' || true + sleep 3 +} + +for n in $(seq 1 "$RUNS"); do + boot_and_play "$n" || { echo "run $n failed"; break; } +done + +banner "RESULT" +echo "app pid per cycle (identical from cycle 2 on = the app survived every restore):" +sed 's/^/ /' "$OUT/pids.txt" 2>/dev/null || true +for n in $(seq 2 "$RUNS"); do + p=$((n-1)) + if [ -f "$OUT/screenshots/run$n-entry.png" ] && [ -f "$OUT/screenshots/run$p.png" ]; then + if cmp -s "$OUT/screenshots/run$n-entry.png" "$OUT/screenshots/run$p.png"; then + echo " run$n-entry == run$p exit (identical: the snapshot froze exactly at the screenshot)" + else + echo " run$n-entry != run$p exit (differs: the guest advanced between capture and freeze)" + fi + fi +done +echo +echo "screenshots and logs: $OUT" +if [ "$KEEP" = 0 ] && [ "$CREATED_AVD" = 1 ]; then + rm -rf "$HOME/.android/avd/$AVD.avd" "$HOME/.android/avd/$AVD.ini" + echo "removed the AVD it created ($AVD); pass -k to keep it" +else + echo "kept AVD $AVD ($(du -sh "$HOME/.android/avd/$AVD.avd" 2>/dev/null | cut -f1))" +fi From 5f878f7c075c8eb0c0aef64bd5a5a11fcab2742c Mon Sep 17 00:00:00 2001 From: JP Cottin Date: Sun, 26 Jul 2026 12:27:52 -0700 Subject: [PATCH 2/2] Replay harness: do not clean up while the emulator is still shutting down A cycle that failed part-way left the emulator running, and the script went straight to removing the AVD. The emulator was still writing, so it recreated the directory after the delete -- leaving an .avd with no .ini, which the next run then reused and could not boot. Take the emulator down and wait for its process group before reporting or removing anything, and keep the AVD if it somehow will not stop. Treat an .avd/.ini pair that is missing one half as incomplete: say so and replace it, rather than trying to boot it. --- scripts/replay-preview-multirun.sh | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/scripts/replay-preview-multirun.sh b/scripts/replay-preview-multirun.sh index 3af5b44..dffc9d8 100755 --- a/scripts/replay-preview-multirun.sh +++ b/scripts/replay-preview-multirun.sh @@ -91,8 +91,15 @@ banner "SETUP: AVD $AVD" # Only an AVD this script created may be deleted at the end -- never one that # was already on the machine, even if -a named it explicitly. CREATED_AVD=0 -if [ -d "$HOME/.android/avd/$AVD.avd" ]; then +if [ -d "$HOME/.android/avd/$AVD.avd" ] && [ -f "$HOME/.android/avd/$AVD.ini" ]; then echo "reusing the existing AVD $AVD (it will be left in place)" +elif [ -d "$HOME/.android/avd/$AVD.avd" ] || [ -f "$HOME/.android/avd/$AVD.ini" ]; then + # A directory without its .ini (or vice versa) is not a usable AVD -- an + # emulator that was still shutting down when a previous run cleaned up can + # leave one behind. Replace it rather than trying to boot it. + echo "found an incomplete AVD $AVD (missing its .ini or .avd); replacing it" + rm -rf "$HOME/.android/avd/$AVD.avd" "$HOME/.android/avd/$AVD.ini" + CREATED_AVD=1 else CREATED_AVD=1 fi @@ -267,6 +274,15 @@ for n in $(seq 1 "$RUNS"); do boot_and_play "$n" || { echo "run $n failed"; break; } done +# A cycle that failed part-way may leave the emulator running. Take it down and +# wait for it before reporting or removing the AVD: an emulator still shutting +# down will recreate the directory underneath us. +cleanup +if [ -n "${EMU_PGID:-}" ]; then + for _ in $(seq 1 60); do kill -0 -- "-$EMU_PGID" 2>/dev/null || break; sleep 1; done + kill -0 -- "-$EMU_PGID" 2>/dev/null && echo "WARNING: the emulator is still running; leaving the AVD alone" && KEEP=1 +fi + banner "RESULT" echo "app pid per cycle (identical from cycle 2 on = the app survived every restore):" sed 's/^/ /' "$OUT/pids.txt" 2>/dev/null || true