diff --git a/.gitignore b/.gitignore index 6d824ce5..5e0a0fa4 100644 --- a/.gitignore +++ b/.gitignore @@ -82,6 +82,9 @@ ios/Frameworks/ build/ *.tsbuildinfo +# scripts/benchmark-boot.sh output (per-boot logcat captures + results JSON) +benchmark-results/ + # eslint .eslintcache diff --git a/AGENTS.md b/AGENTS.md index 295fe643..ff8d152b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -113,7 +113,7 @@ Mirrored verbatim from Expo's official [`expo/skills`](https://github.com/expo/s - [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — process model, IPC channels and framing, boot handshake, lifecycle state machines, error handling, Sentry observability, alternatives considered. - [`docs/BUILD.md`](docs/BUILD.md) — the `build-backend.ts` pipeline, the versioned addon-filename scheme, runtime addon loading, the native-modules source-of-truth model. - [`docs/TESTING.md`](docs/TESTING.md) — the seven test layers, the workflows, the merge queue and required checks, the e2e device suite, the secrets/trust boundary. -- [`docs/BENCHMARKING.md`](docs/BENCHMARKING.md) — measuring the backend's memory footprint: the Sentry gauges that answer the question across the fleet, what they measure and why, including the consent-tier reasoning. +- [`docs/BENCHMARKING.md`](docs/BENCHMARKING.md) — measuring the backend's memory footprint: the local A/B cold-boot harness (`npm run bench:android`) and the Sentry gauges that answer the same question across the fleet, including the consent-tier reasoning. - [`CONTRIBUTING.md`](./CONTRIBUTING.md) — setup, every `npm run` script, per-layer test commands, commit/PR/release conventions. Open work is tracked in [GitHub issues](https://github.com/digidem/comapeo-core-react-native/issues). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 67cb5bac..40124843 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -135,6 +135,11 @@ what each one verifies) has a root script: | iOS integration | `npm run test:ios` | needs the integration app prebuilt + a simulator | | End-to-end (local) | `npm run e2e:ios` / `e2e:android`, then `npm run e2e:test` | Release build + Maestro; see below | +Measuring the backend's memory footprint (comparing two builds on a device, +and the Sentry gauges that answer the same question across the fleet) is a +separate exercise from testing: see +[docs/BENCHMARKING.md](./docs/BENCHMARKING.md) and `npm run bench:android`. + The native/integration commands are slower and platform-specific. The CI workflows remain the source of truth for the exact flags and device matrix: [ios-tests.yml](.github/workflows/ios-tests.yml), diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md index 13ee2d73..21231a00 100644 --- a/docs/BENCHMARKING.md +++ b/docs/BENCHMARKING.md @@ -7,8 +7,14 @@ backend makes it a less attractive victim than the foreground UI. That makes "what does this change cost in memory" a question worth being able to answer cheaply and repeatedly. -The backend measures its own footprint and reports it across the fleet as -Sentry gauges, from every install with diagnostics on. The numbers come from +There are two ways to answer it, and they share one implementation. + +| | Where | What it answers | +|---|---|---| +| [`scripts/benchmark-boot.sh`](../scripts/benchmark-boot.sh) | one device you control | "Is build B lighter than build A?" — an A/B you run before merging | +| [Sentry gauges](#the-fleet-view-sentry-gauges) | every install with diagnostics on | "Did it hold up on real devices?" — after shipping | + +Both read the same numbers from [`backend/lib/memory-snapshot.js`](../backend/lib/memory-snapshot.js). ## What gets measured, and why those numbers @@ -39,6 +45,102 @@ Android. On iOS node runs in-process, so the same fields would describe the UI too; the reader returns `null` there and only the V8 numbers are reported. The platform gate is the filesystem, not a flag that could drift. +## Running an A/B locally + +The harness is built for comparing two builds, not for producing absolute +figures. Emulator numbers do not transfer to a physical device — V8 sizes its +heap from physical RAM, among other things — but the *difference* between two +builds measured the same way usually does. + +The simple case, when you have two APKs: + +```sh +./scripts/benchmark-boot.sh --ab /tmp/before.apk /tmp/after.apk \ + --rounds 3 --per-round 5 +``` + +That runs 15 cold boots of each, alternating which build goes first every +round, and prints a comparison at the end. If you only have one thing to +measure, run it twice and compare the files: + +```sh +./scripts/benchmark-boot.sh --label before --iterations 10 +# …change something, rebuild… +./scripts/benchmark-boot.sh --label after --iterations 10 +node scripts/benchmark-report.mjs benchmark-results/before.json \ + benchmark-results/after.json +``` + +With no `--apk` or `--ab` it builds `apps/integration` in Release first. +Benchmark against Release, never debug: `startForeground` alone is about ten +times slower in a debug build. + +Useful flags: `--device ` (required when more than one is attached), +`--iterations`, `--rounds` / `--per-round`, `--duration` (seconds to watch each +boot, default 20), `--abi`, `--skip-build`, `--out`. + +Output lands in `benchmark-results/` (gitignored): one JSON per label plus the +raw per-boot logcat captures under a per-invocation `raw//` +directory — the second invocation of a before/after comparison does not +overwrite the first's captures, which are worth keeping when a run looks +strange. + +### Reading the output + +``` +metric baseline candidate delta % p +----------------------------------------------------------------------- +Peak RSS (MB) 274.4 261.0 -13.4 -4.9 <0.001 +Anonymous RSS (MB) 103.3 92.5 -10.8 -10.5 0.029 +V8 live heap (MB) 25.4 15.8 -9.6 -37.8 <0.001 + +per-round medians (peak RSS, MB): +round baseline candidate +1 274.2 261.2 +2 274.4 261.0 +3 275.1 261.1 +``` + +Medians, not means: boot measurements are skewed by the occasional scheduling +stall, and one stalled boot drags a mean around. The p-values are two-sided +Mann–Whitney U over the individual boots — they say whether *these* boots +differ, not whether the effect holds on hardware you did not test. + +**The per-round table is the one to check first.** A development machine is +usually doing other things, and host load drifts over the twenty-odd minutes a +full run takes. If the two builds separate the same way in every round, the +difference is the build. If the rounds disagree, you measured the machine. + +### Things that will bite you + +- **Steady-state RSS is bimodal.** By the time the sample is taken, a ~24 MB + block has either been released back or not, and which happens varies boot to + boot. Peak RSS does not have this problem, which is another reason to lead + with it. If you need the settled number, take enough boots to see both modes + in each build and compare them mode-for-mode. +- **The compile cache is only wiped when adb runs as root.** The wipe exists + so that swapping APKs doesn't leave one build paying for a + `NODE_COMPILE_CACHE` the other wrote, but app cache directories are + private, so it only works after `adb root` (emulators, userdebug builds). + With root, measured boots are cold-cache — the worst case, and the one that + matters for a process that may be killed at startup. Without it the harness + prints a warning and the boots are warm-cache; the memory numbers are still + valid, but treat the timing comparison with suspicion, especially right + after swapping APKs. +- **The first boot after each install is discarded** — it faults the freshly + written APK in. +- **`VmSize` is not a footprint.** A V8 build with pointer compression reserves + a 4 GB cage per isolate, which never becomes resident. The harness does not + report it for exactly this reason. + +### Without root + +The harness works unmodified on a production device: the boot timings come +from the module's `[comapeo.*]` lifecycle crumbs in logcat, and peak RSS, RSS +and the heap breakdown come from the backend's own `[comapeo.memory] boot` +log line, which reads `/proc/self` and needs no privilege. The only thing +root buys is the compile-cache wipe above. + ## The fleet view: Sentry gauges The same snapshot feeds four gauges, emitted once about three seconds after @@ -77,7 +179,7 @@ question worth asking of it. It only separates builds that carry different revisions, though. A libnode built from an unmerged branch reports whatever tag it was based on, so two such builds are indistinguishable by this attribute — locally that does not matter -(you know which APK you installed), but anything shipped to devices for +(the harness knows which APK it installed), but anything shipped to devices for comparison needs its own revision. Bump the nodejs-mobile tag before a staged rollout you intend to measure. diff --git a/package.json b/package.json index 62988cdc..b0b76b49 100644 --- a/package.json +++ b/package.json @@ -73,6 +73,8 @@ "prebackend:build": "npm run backend:install", "backend:build": "node ./scripts/build-backend.ts", "backend:test": "npm --prefix backend test", + "bench:android": "./scripts/benchmark-boot.sh", + "bench:report": "node ./scripts/benchmark-report.mjs", "e2e:ios": "npm --prefix apps/e2e run build:ios", "e2e:android": "npm --prefix apps/e2e run build:android", "e2e:test": "maestro test maestro/e2e.yaml" diff --git a/scripts/benchmark-boot.sh b/scripts/benchmark-boot.sh new file mode 100755 index 00000000..dd10a917 --- /dev/null +++ b/scripts/benchmark-boot.sh @@ -0,0 +1,273 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Cold-boot benchmark for the embedded Node backend on Android. +# +# Measures what the `:ComapeoCore` foreground-service process costs to start: +# peak RSS, settled RSS, the V8 heap breakdown, and launch → `started` → +# `ready` timings — all read from the module's `[comapeo.*]` lifecycle crumbs +# and the backend's own `[comapeo.memory] boot` log line, so no root is +# needed for the numbers. Built for comparing two builds of anything that +# changes the backend's footprint — a different libnode.so, a dependency +# bump, a lazy import — not for absolute numbers. Run --help for usage. + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +APP_DIR="$PROJECT_ROOT/apps/integration" + +PKG="com.comapeo.core.integration" +ACTIVITY=".MainActivity" +ABI="arm64-v8a" + +LABEL="current" +ITERATIONS=5 +DURATION=20 +ROUNDS=3 +PER_ROUND=5 +DEVICE="" +APK="" +AB_A="" +AB_B="" +OUT_DIR="$PROJECT_ROOT/benchmark-results" +SKIP_BUILD=0 + +usage() { + cat <<'EOF' +Cold-boot benchmark for the embedded Node backend on Android. + +Single series — measure whatever is installed (or install one APK first): + ./scripts/benchmark-boot.sh --label before --iterations 10 + ./scripts/benchmark-boot.sh --label after --apk /tmp/after.apk + node scripts/benchmark-report.mjs benchmark-results/before.json \ + benchmark-results/after.json + +Interleaved A/B — alternates blocks between two APKs, which is the only +honest way to run this on a machine that is doing other work: + ./scripts/benchmark-boot.sh --ab /tmp/before.apk /tmp/after.apk \ + --rounds 3 --per-round 5 + +Options: + --label series label for a single-series run (default: current) + --iterations boots in a single-series run (default: 5) + --ab interleaved A/B between two APKs + --rounds A/B rounds (default: 3) + --per-round boots per build per round (default: 5) + --duration seconds to watch each boot (default: 20) + --device adb device (required when several are attached) + --apk install this APK before the run + --abi ABI for the Release build (default: arm64-v8a) + --out results directory (default: benchmark-results) + --skip-build measure what is already installed + +With no --apk/--ab it builds the integration app in Release first (debug +start-up is ~10x slower and not representative — see CONTRIBUTING.md). +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --label) LABEL="$2"; shift 2 ;; + --iterations) ITERATIONS="$2"; shift 2 ;; + --duration) DURATION="$2"; shift 2 ;; + --device) DEVICE="$2"; shift 2 ;; + --apk) APK="$2"; shift 2 ;; + --ab) + [ $# -ge 3 ] || { echo "Error: --ab takes two APK paths." >&2; exit 1; } + AB_A="$2"; AB_B="$3"; shift 3 ;; + --rounds) ROUNDS="$2"; shift 2 ;; + --per-round) PER_ROUND="$2"; shift 2 ;; + --abi) ABI="$2"; shift 2 ;; + --out) OUT_DIR="$2"; shift 2 ;; + --skip-build) SKIP_BUILD=1; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;; + esac +done + +# The interactive shell aliases `adb` to a multi-device picker that ignores +# -s and blocks on stdin; go straight to the binary. +ADB=(command adb) +[ -n "$DEVICE" ] && ADB=(command adb -s "$DEVICE") + +# Run ids ("