diff --git a/README.md b/README.md index ad7c715..6378626 100644 --- a/README.md +++ b/README.md @@ -238,13 +238,44 @@ screenshot: ## CI/CD -Three GitHub Actions jobs run on every push and pull request to `main`: +GitHub Actions runs on every push and pull request to `main`. The first three +jobs gate merges. The last three explore newer Android emulator tooling and are +marked `continue-on-error`, so a preview package that moves underneath us +reports its findings without ever blocking a PR. | Job | What it does | Artifacts | |-----|-------------|-----------| | **Build APK** | Compiles the debug APK | `debug-apk` | | **Native Tests** | Runs the 103 Google Test cases on x86\_64 emulators (API 34 + API 36) | — | -| **Smoke Test** | Runs the Android instrumented test on x86\_64 emulators and captures an in-game screenshot via `UiAutomation`. The test asserts the activity is RESUMED **and** that the renderer logged `Swapchain ready`, so a dead Vulkan path fails loudly. Blocking on API 34 + 36; non-blocking preview legs on API 37.0 (`google_apis_ps16k`, 16 KB pages) across the swiftshader / lavapipe / auto GPU backends | `smoke-screenshot-api*`, `smoke-test-results-api*`, `smoke-logcat-api*` (suffixed per leg) | +| **Smoke Test** | Runs the Android instrumented test on x86\_64 emulators and captures an in-game screenshot via `UiAutomation`. The test asserts the activity is RESUMED **and** that the renderer logged `Swapchain ready`, so a dead Vulkan path fails loudly. Blocking on API 34 + 36; non-blocking legs on API 37.0 and 37.1 (`google_apis_ps16k`, 16 KB pages) across the lavapipe and auto GPU backends, from both the stable and canary channels | `smoke-screenshot-api*`, `smoke-test-results-api*`, `smoke-logcat-api*` (suffixed per leg) | +| **Android CLI experiment** | Drives the same instrumented test through the `android` CLI — SDK install, AVD creation, boot and teardown — instead of `sdkmanager`/`avdmanager` plus the emulator-runner action | `cli-smoke-*` | +| **Emulator Preview** | Boots the Android Emulator Preview package (`emulators;latest`, which installs alongside the stable emulator under `emulators/latest/`) and runs the instrumented test against it | `preview-smoke-*` | +| **Emulator Preview multi-run** | Four boot cycles against the same AVD with quickboot snapshots enabled: each cycle plays the game briefly, screenshots it, then shuts down so the emulator saves its snapshot. Checks whether a live Vulkan app survives snapshot save/restore — it does: the app keeps its pid and the game continues across cycles | `preview-multirun-screenshots`, `preview-multirun-emulator-logs` | + +The preview jobs share their setup through the composite action in +`.github/actions/preview-emulator`, which installs the system image, creates +the AVD, installs the preview emulator and its host dependencies. + +### Replaying the Emulator Preview job locally + +Pushing to see what a preview emulator does is a slow way to iterate, so the +multi-run job can be replayed on a local machine in a few minutes: + +```bash +scripts/replay-preview-multirun.sh # 4 cycles, throwaway AVD, cleaned up after +scripts/replay-preview-multirun.sh -n 2 -k # 2 cycles, keep the AVD for inspection +scripts/replay-preview-multirun.sh -h # options +``` + +It needs `emulators;latest`, the API 37.0 `google_apis_ps16k` system image and +KVM. Every `adb` call is pinned to the emulator it launches and shutdown is +scoped to that emulator's process group, so it is safe to run while other +devices or emulators are attached. + +The script writes `~/.emulator_console_auth_token` if you do not already have +one. The emulator console authenticates against that file before it offers its +full command set, including `kill` — which is how both the script and CI ask +the emulator to shut down cleanly so that it writes its quickboot snapshot. ## License diff --git a/scripts/replay-preview-multirun.sh b/scripts/replay-preview-multirun.sh new file mode 100755 index 0000000..dffc9d8 --- /dev/null +++ b/scripts/replay-preview-multirun.sh @@ -0,0 +1,306 @@ +#!/usr/bin/env bash +# +# Replay the CI job "Emulator Preview experiment multi-run" on this machine. +# +# The job boots the Android Emulator Preview (emulators;latest) several times +# WITHOUT -no-snapshot: each cycle plays the game for a few seconds, screenshots +# it, then shuts down gracefully so the emulator writes its quickboot snapshot. +# If restore preserves the running app, the app keeps the same pid across cycles +# and the game continues where it left off. +# +# Running it here instead of pushing turns a ~25 minute CI round trip into a few +# minutes, which matters when iterating on the emulator itself. +# +# Usage: +# scripts/replay-preview-multirun.sh [-n RUNS] [-a AVD_NAME] [-o OUTDIR] [-k] +# +# -n RUNS number of boot/snapshot cycles (default 4) +# -a NAME AVD to create and use (default preview_replay) +# -o DIR output directory for screenshots and logs (default /tmp/preview-replay) +# -k keep the AVD afterwards (default: delete it -- it is several GB) +# +# Requirements: $ANDROID_HOME with the emulators;latest package and the +# system image below, plus KVM. Install with: +# sdkmanager --channel=3 --install 'emulators;latest' \ +# 'system-images;android-37.0;google_apis_ps16k;x86_64' +set -u + +RUNS=4 +AVD=preview_replay +OUT=/tmp/preview-replay +KEEP=0 +while getopts ':n:a:o:kh' opt; do + case "$opt" in + n) RUNS="$OPTARG" ;; + a) AVD="$OPTARG" ;; + o) OUT="$OPTARG" ;; + k) KEEP=1 ;; + h) sed -n '2,28p' "$0"; exit 0 ;; + *) echo "unknown option -$OPTARG (try -h)" >&2; exit 2 ;; + esac +done + +case "$AVD" in + ''|*/*) echo "ERROR: -a needs a plain AVD name" >&2; exit 2 ;; +esac +case "$RUNS" in + ''|*[!0-9]*|0) echo "ERROR: -n needs a positive number of cycles" >&2; exit 2 ;; +esac + +SDK="${ANDROID_HOME:-${ANDROID_SDK_ROOT:-$HOME/Android/Sdk}}" +export ANDROID_HOME="$SDK" ANDROID_SDK_ROOT="$SDK" +ADB="$SDK/platform-tools/adb" +EMULATOR="$SDK/emulators/latest/emulator" +API=37.0 +TARGET=google_apis_ps16k +ABI=x86_64 +PKG=com.jpcottin.vulkanspaceinvaders +REPO="$(cd "$(dirname "$0")/.." && pwd)" +APK="$REPO/app/build/outputs/apk/debug/app-debug.apk" + +[ -x "$EMULATOR" ] || { echo "ERROR: no preview emulator at $EMULATOR"; exit 1; } +[ -x "$ADB" ] || { echo "ERROR: no adb at $ADB"; exit 1; } +[ -e /dev/kvm ] || echo "WARNING: /dev/kvm missing; the emulator will be extremely slow" + +mkdir -p "$OUT/screenshots" +rm -f "$OUT"/screenshots/*.png "$OUT"/emulator_run*.txt "$OUT"/logcat_run*.txt "$OUT"/pids.txt + +# This host may have other devices on adb (another emulator, a Cuttlefish +# instance, a phone). Every adb call is pinned to the emulator we launch, and +# process cleanup is scoped to its own process group -- never a pattern match +# across all emulator processes. +SERIAL="" +A() { "$ADB" -s "$SERIAL" "$@"; } +SELF_PGID="$(ps -o pgid= -p $$ | tr -d ' ')" + +banner() { echo; echo "################ $* ################"; } + +# The emulator console starts unauthenticated, where it offers only +# help/ping/auth/quit/avd -- 'kill' and 'avd pause' are not available, so +# 'adb emu kill' returns "KO: unknown command". The emulator reads this token +# file but does not create it, so create it if absent. An existing token is +# left alone. +TOKEN="$HOME/.emulator_console_auth_token" +if [ ! -s "$TOKEN" ]; then + printf 'replayConsoleToken' > "$TOKEN" + chmod 600 "$TOKEN" + echo "created $TOKEN" +fi + +banner "SETUP: AVD $AVD" +# Only an AVD this script created may be deleted at the end -- never one that +# was already on the machine, even if -a named it explicitly. +CREATED_AVD=0 +if [ -d "$HOME/.android/avd/$AVD.avd" ] && [ -f "$HOME/.android/avd/$AVD.ini" ]; then + echo "reusing the existing AVD $AVD (it will be left in place)" +elif [ -d "$HOME/.android/avd/$AVD.avd" ] || [ -f "$HOME/.android/avd/$AVD.ini" ]; then + # A directory without its .ini (or vice versa) is not a usable AVD -- an + # emulator that was still shutting down when a previous run cleaned up can + # leave one behind. Replace it rather than trying to boot it. + echo "found an incomplete AVD $AVD (missing its .ini or .avd); replacing it" + rm -rf "$HOME/.android/avd/$AVD.avd" "$HOME/.android/avd/$AVD.ini" + CREATED_AVD=1 +else + CREATED_AVD=1 +fi +if [ ! -d "$HOME/.android/avd/$AVD.avd" ]; then + # avdmanager silently produces nothing once emulators;latest is installed, + # so write the AVD files directly. This mirrors what the CI job does. + echo no | "$SDK/cmdline-tools/latest/bin/avdmanager" create avd --force -n "$AVD" \ + --abi "$TARGET/$ABI" --device 'pixel_6' \ + --package "system-images;android-$API;$TARGET;$ABI" >/dev/null 2>&1 || true +fi +if [ ! -d "$HOME/.android/avd/$AVD.avd" ]; then + echo "avdmanager produced nothing; writing the AVD files by hand" + mkdir -p "$HOME/.android/avd/$AVD.avd" + printf 'avd.ini.encoding=UTF-8\npath=%s/.android/avd/%s.avd\npath.rel=avd/%s.avd\ntarget=android-%s\n' \ + "$HOME" "$AVD" "$AVD" "$API" > "$HOME/.android/avd/$AVD.ini" + cat > "$HOME/.android/avd/$AVD.avd/config.ini" << CFG +AvdId=$AVD +avd.ini.displayname=$AVD +avd.ini.encoding=UTF-8 +abi.type=$ABI +hw.cpu.arch=$ABI +image.sysdir.1=system-images/android-$API/$TARGET/$ABI/ +tag.id=google_apis +tag.display=Google APIs +PlayStore.enabled=no +hw.lcd.density=420 +hw.lcd.width=1080 +hw.lcd.height=2400 +hw.keyboard=yes +hw.gpu.enabled=yes +hw.gpu.mode=auto +disk.dataPartition.size=8192M +hw.ramSize=4096 +hw.cpu.ncore=4 +CFG +fi + +if [ ! -f "$APK" ]; then + banner "Building the debug APK" + (cd "$REPO" && ./gradlew assembleDebug --no-daemon) || exit 1 +fi + +cleanup() { + # On an interrupted run, take down the emulator we started -- by process + # group when we managed to isolate one, otherwise by pid. + if [ -n "${EMU_PGID:-}" ] && kill -0 -- "-$EMU_PGID" 2>/dev/null; then + kill -TERM -- "-$EMU_PGID" 2>/dev/null || true + elif [ -n "${EMU_PID:-}" ] && kill -0 "$EMU_PID" 2>/dev/null; then + kill -TERM "$EMU_PID" 2>/dev/null || true + fi +} +trap cleanup EXIT INT TERM + +boot_and_play() { + N="$1" + banner "RUN $N: launching emulator (snapshots enabled)" + # Record the emulators already attached, so we can identify ours as the one + # that appears afterwards. Never assume emulator-5554 is ours: this machine + # may already be running an emulator on the default ports. + DEVICES_BEFORE="$("$ADB" devices | awk '/^emulator-/{print $1}' | sort)" + # No -no-snapshot: load the quickboot snapshot if present, save it on exit. + setsid "$EMULATOR" @"$AVD" -no-window -gpu auto -noaudio -no-boot-anim \ + -camera-back none -memory 4096 -verbose -show-kernel \ + > "$OUT/emulator_run$N.txt" 2>&1 & + EMU_PID=$! + # setsid(2) happens asynchronously, so poll until the process group settles. + EMU_PGID="" + for _ in $(seq 1 20); do + P="$(ps -o pgid= -p "$EMU_PID" 2>/dev/null | tr -d ' ')" + if [ -n "$P" ] && [ "$P" != "$SELF_PGID" ]; then EMU_PGID="$P"; break; fi + kill -0 "$EMU_PID" 2>/dev/null || break + sleep 0.5 + done + [ -n "$EMU_PGID" ] || echo "WARNING: could not isolate a process group; using single-pid shutdown" + + SERIAL="" + for _ in $(seq 1 36); do + kill -0 "$EMU_PID" 2>/dev/null || { echo "ERROR: emulator exited early"; tail -30 "$OUT/emulator_run$N.txt"; return 1; } + cand="$(comm -13 <(printf '%s\n' "$DEVICES_BEFORE") \ + <("$ADB" devices | awk '/^emulator-/{print $1}' | sort) | head -1)" + [ -n "$cand" ] && { SERIAL="$cand"; break; } + sleep 5 + done + if [ -z "$SERIAL" ]; then + echo "ERROR: the emulator we launched never appeared on adb after 180s" + if grep -q "address already in use" "$OUT/emulator_run$N.txt" 2>/dev/null; then + echo " its console port is taken -- another emulator is already running:" + "$ADB" devices | sed 's/^/ /' + echo " stop it, or free the console ports, and try again." + fi + return 1 + fi + echo "RUN $N serial=$SERIAL pid=$EMU_PID pgid=${EMU_PGID:-none}" + + for _ in $(seq 1 48); do + [ "$(A shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" = "1" ] && break + sleep 10 + done + [ "$(A shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" = "1" ] || { echo "ERROR: boot timeout"; return 1; } + T_RESUME=$(date +%s) + + if [ "$N" = 1 ]; then + # A guest error dialog would block all input and be preserved by the + # snapshot, so suppress dialogs for the duration of the replay. + A shell settings put global hide_error_dialogs 1 >/dev/null || true + A install -r "$APK" >/dev/null || return 1 + # Auto-play must be on BEFORE the first start: the game reads + # /settings.bin at startup (magic "SETT" LE, soundOn=0, autoPlay=1). + A shell "run-as $PKG sh -c 'mkdir -p files; echo VFRFUwAAAAABAAAA | base64 -d > files/settings.bin'" + A shell input keyevent KEYCODE_WAKEUP >/dev/null || true + A shell wm dismiss-keyguard >/dev/null || true + else + # The restored frame, before any input: should match the previous cycle's + # exit screenshot, which was taken immediately before that cycle froze. + A exec-out screencap -p > "$OUT/screenshots/run$N-entry.png" || true + fi + + RUN_PID="$(A shell pidof "$PKG" 2>/dev/null | tr -d '\r')" + echo "RUN $N app pid after boot/restore: ${RUN_PID:-none}" + echo "cycle $N: ${RUN_PID:-none}" >> "$OUT/pids.txt" + A shell am start -n "$PKG/android.app.NativeActivity" >/dev/null 2>&1 || true + for _ in $(seq 1 30); do + A shell dumpsys window 2>/dev/null | grep -qi "ocus.*vulkanspaceinvaders" && break + sleep 1 + done + sleep 3 + # The TITLE screen needs a tap to start a round; auto-play only flies during + # PLAYING. Taps are spaced out because input to an unfocused window is dropped. + for _ in 1 2 3 4; do A shell input tap 540 1500 >/dev/null || true; sleep 2; done + sleep 5 + + A logcat -d -t 500 > "$OUT/logcat_run$N.txt" 2>/dev/null || true + # Liveness: two captures 2s apart must differ, or rendering has stalled. + A exec-out screencap -p > "$OUT/liveness-a.png" || true + sleep 2 + A exec-out screencap -p > "$OUT/liveness-b.png" || true + if cmp -s "$OUT/liveness-a.png" "$OUT/liveness-b.png"; then + echo "RUN $N display: FROZEN (two captures 2s apart are identical)" + else + echo "RUN $N display: LIVE" + fi + + # Exit screenshot last, immediately before shutdown, so it is as close as + # possible to the state the snapshot captures. A few seconds still elapse + # while the emulator shuts down and the guest keeps running, so the next + # cycle's entry screenshot is close to this one rather than identical. + A exec-out screencap -p > "$OUT/screenshots/run$N.png" || true + echo "RUN $N app pid before shutdown: $(A shell pidof "$PKG" 2>/dev/null | tr -d '\r' || echo none)" + + T_KILL=$(date +%s) + echo "RUN $N: shutting down via adb emu kill" + A emu kill >/dev/null 2>&1 || true + for _ in $(seq 1 90); do kill -0 "$EMU_PID" 2>/dev/null || break; sleep 1; done + if kill -0 "$EMU_PID" 2>/dev/null; then + echo "WARNING: still up 90s after 'adb emu kill'; terminating" + if [ -n "$EMU_PGID" ]; then kill -TERM -- "-$EMU_PGID" 2>/dev/null || true + else kill "$EMU_PID" 2>/dev/null || true; fi + sleep 10 + fi + if [ -n "$EMU_PGID" ]; then + for _ in $(seq 1 120); do kill -0 -- "-$EMU_PGID" 2>/dev/null || break; sleep 1; done + fi + EMU_PGID="" + echo "RUN $N: down $(( $(date +%s) - T_KILL ))s after the shutdown request (played ~$(( T_KILL - T_RESUME ))s)" + LD_LIBRARY_PATH="$SDK/emulators/latest/lib64" \ + "$SDK/emulators/latest/bin/qemu-img" snapshot -l \ + "$HOME/.android/avd/$AVD.avd/userdata-qemu.img.qcow2" 2>&1 | sed 's/^/ /' || true + sleep 3 +} + +for n in $(seq 1 "$RUNS"); do + boot_and_play "$n" || { echo "run $n failed"; break; } +done + +# A cycle that failed part-way may leave the emulator running. Take it down and +# wait for it before reporting or removing the AVD: an emulator still shutting +# down will recreate the directory underneath us. +cleanup +if [ -n "${EMU_PGID:-}" ]; then + for _ in $(seq 1 60); do kill -0 -- "-$EMU_PGID" 2>/dev/null || break; sleep 1; done + kill -0 -- "-$EMU_PGID" 2>/dev/null && echo "WARNING: the emulator is still running; leaving the AVD alone" && KEEP=1 +fi + +banner "RESULT" +echo "app pid per cycle (identical from cycle 2 on = the app survived every restore):" +sed 's/^/ /' "$OUT/pids.txt" 2>/dev/null || true +for n in $(seq 2 "$RUNS"); do + p=$((n-1)) + if [ -f "$OUT/screenshots/run$n-entry.png" ] && [ -f "$OUT/screenshots/run$p.png" ]; then + if cmp -s "$OUT/screenshots/run$n-entry.png" "$OUT/screenshots/run$p.png"; then + echo " run$n-entry == run$p exit (identical: the snapshot froze exactly at the screenshot)" + else + echo " run$n-entry != run$p exit (differs: the guest advanced between capture and freeze)" + fi + fi +done +echo +echo "screenshots and logs: $OUT" +if [ "$KEEP" = 0 ] && [ "$CREATED_AVD" = 1 ]; then + rm -rf "$HOME/.android/avd/$AVD.avd" "$HOME/.android/avd/$AVD.ini" + echo "removed the AVD it created ($AVD); pass -k to keep it" +else + echo "kept AVD $AVD ($(du -sh "$HOME/.android/avd/$AVD.avd" 2>/dev/null | cut -f1))" +fi