feat(scripts): local A/B boot benchmark harness

.gitignore

@@ -82,6 +82,9 @@ ios/Frameworks/
 build/
 *.tsbuildinfo
 
+# scripts/benchmark-boot.sh output (per-boot logcat captures + results JSON)
+benchmark-results/
+
 # eslint
 .eslintcache
 

AGENTS.md

@@ -113,7 +113,7 @@ Mirrored verbatim from Expo's official [`expo/skills`](https://github.com/expo/s
 - [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — process model, IPC channels and framing, boot handshake, lifecycle state machines, error handling, Sentry observability, alternatives considered.
 - [`docs/BUILD.md`](docs/BUILD.md) — the `build-backend.ts` pipeline, the versioned addon-filename scheme, runtime addon loading, the native-modules source-of-truth model.
 - [`docs/TESTING.md`](docs/TESTING.md) — the seven test layers, the workflows, the merge queue and required checks, the e2e device suite, the secrets/trust boundary.
-- [`docs/BENCHMARKING.md`](docs/BENCHMARKING.md) — measuring the backend's memory footprint: the Sentry gauges that answer the question across the fleet, what they measure and why, including the consent-tier reasoning.
+- [`docs/BENCHMARKING.md`](docs/BENCHMARKING.md) — measuring the backend's memory footprint: the local A/B cold-boot harness (`npm run bench:android`) and the Sentry gauges that answer the same question across the fleet, including the consent-tier reasoning.
 - [`CONTRIBUTING.md`](./CONTRIBUTING.md) — setup, every `npm run` script, per-layer test commands, commit/PR/release conventions.
 
 Open work is tracked in [GitHub issues](https://github.com/digidem/comapeo-core-react-native/issues).

CONTRIBUTING.md

@@ -135,6 +135,11 @@ what each one verifies) has a root script:
 | iOS integration      | `npm run test:ios`                                         | needs the integration app prebuilt + a simulator |
 | End-to-end (local)   | `npm run e2e:ios` / `e2e:android`, then `npm run e2e:test` | Release build + Maestro; see below               |
 
+Measuring the backend's memory footprint (comparing two builds on a device,
+and the Sentry gauges that answer the same question across the fleet) is a
+separate exercise from testing: see
+[docs/BENCHMARKING.md](./docs/BENCHMARKING.md) and `npm run bench:android`.
+
 The native/integration commands are slower and platform-specific. The CI
 workflows remain the source of truth for the exact flags and device matrix:
 [ios-tests.yml](.github/workflows/ios-tests.yml),

docs/BENCHMARKING.md

@@ -7,8 +7,14 @@ backend makes it a less attractive victim than the foreground UI. That makes
 "what does this change cost in memory" a question worth being able to answer
 cheaply and repeatedly.
 
-The backend measures its own footprint and reports it across the fleet as
-Sentry gauges, from every install with diagnostics on. The numbers come from
+There are two ways to answer it, and they share one implementation.
+
+| | Where | What it answers |
+|---|---|---|
+| [`scripts/benchmark-boot.sh`](../scripts/benchmark-boot.sh) | one device you control | "Is build B lighter than build A?" — an A/B you run before merging |
+| [Sentry gauges](#the-fleet-view-sentry-gauges) | every install with diagnostics on | "Did it hold up on real devices?" — after shipping |
+
+Both read the same numbers from
 [`backend/lib/memory-snapshot.js`](../backend/lib/memory-snapshot.js).
 
 ## What gets measured, and why those numbers
@@ -39,6 +45,102 @@ Android. On iOS node runs in-process, so the same fields would describe the UI
 too; the reader returns `null` there and only the V8 numbers are reported. The
 platform gate is the filesystem, not a flag that could drift.
 
+## Running an A/B locally
+
+The harness is built for comparing two builds, not for producing absolute
+figures. Emulator numbers do not transfer to a physical device — V8 sizes its
+heap from physical RAM, among other things — but the *difference* between two
+builds measured the same way usually does.
+
+The simple case, when you have two APKs:
+
+```sh
+./scripts/benchmark-boot.sh --ab /tmp/before.apk /tmp/after.apk \
+    --rounds 3 --per-round 5
+```
+
+That runs 15 cold boots of each, alternating which build goes first every
+round, and prints a comparison at the end. If you only have one thing to
+measure, run it twice and compare the files:
+
+```sh
+./scripts/benchmark-boot.sh --label before --iterations 10
+# …change something, rebuild…
+./scripts/benchmark-boot.sh --label after --iterations 10
+node scripts/benchmark-report.mjs benchmark-results/before.json \
+                                  benchmark-results/after.json
+```
+
+With no `--apk` or `--ab` it builds `apps/integration` in Release first.
+Benchmark against Release, never debug: `startForeground` alone is about ten
+times slower in a debug build.
+
+Useful flags: `--device <serial>` (required when more than one is attached),
+`--iterations`, `--rounds` / `--per-round`, `--duration` (seconds to watch each
+boot, default 20), `--abi`, `--skip-build`, `--out`.
+
+Output lands in `benchmark-results/` (gitignored): one JSON per label plus the
+raw per-boot logcat captures under a per-invocation `raw/<timestamp>/`
+directory — the second invocation of a before/after comparison does not
+overwrite the first's captures, which are worth keeping when a run looks
+strange.
+
+### Reading the output
+
+```
+metric                   baseline  candidate     delta       %        p
+-----------------------------------------------------------------------
+Peak RSS (MB)               274.4      261.0     -13.4    -4.9   <0.001
+Anonymous RSS (MB)          103.3       92.5     -10.8   -10.5    0.029
+V8 live heap (MB)            25.4       15.8      -9.6   -37.8   <0.001
+
+per-round medians (peak RSS, MB):
+round      baseline  candidate
+1             274.2      261.2
+2             274.4      261.0
+3             275.1      261.1
+```
+
+Medians, not means: boot measurements are skewed by the occasional scheduling
+stall, and one stalled boot drags a mean around. The p-values are two-sided
+Mann–Whitney U over the individual boots — they say whether *these* boots
+differ, not whether the effect holds on hardware you did not test.
+
+**The per-round table is the one to check first.** A development machine is
+usually doing other things, and host load drifts over the twenty-odd minutes a
+full run takes. If the two builds separate the same way in every round, the
+difference is the build. If the rounds disagree, you measured the machine.
+
+### Things that will bite you
+
+- **Steady-state RSS is bimodal.** By the time the sample is taken, a ~24 MB
+  block has either been released back or not, and which happens varies boot to
+  boot. Peak RSS does not have this problem, which is another reason to lead
+  with it. If you need the settled number, take enough boots to see both modes
+  in each build and compare them mode-for-mode.
+- **The compile cache is only wiped when adb runs as root.** The wipe exists
+  so that swapping APKs doesn't leave one build paying for a
+  `NODE_COMPILE_CACHE` the other wrote, but app cache directories are
+  private, so it only works after `adb root` (emulators, userdebug builds).
+  With root, measured boots are cold-cache — the worst case, and the one that
+  matters for a process that may be killed at startup. Without it the harness
+  prints a warning and the boots are warm-cache; the memory numbers are still
+  valid, but treat the timing comparison with suspicion, especially right
+  after swapping APKs.
+- **The first boot after each install is discarded** — it faults the freshly
+  written APK in.
+- **`VmSize` is not a footprint.** A V8 build with pointer compression reserves
+  a 4 GB cage per isolate, which never becomes resident. The harness does not
+  report it for exactly this reason.
+
+### Without root
+
+The harness works unmodified on a production device: the boot timings come
+from the module's `[comapeo.*]` lifecycle crumbs in logcat, and peak RSS, RSS
+and the heap breakdown come from the backend's own `[comapeo.memory] boot`
+log line, which reads `/proc/self` and needs no privilege. The only thing
+root buys is the compile-cache wipe above.
+
 ## The fleet view: Sentry gauges
 
 The same snapshot feeds four gauges, emitted once about three seconds after
@@ -77,7 +179,7 @@ question worth asking of it.
 It only separates builds that carry different revisions, though. A libnode
 built from an unmerged branch reports whatever tag it was based on, so two such
 builds are indistinguishable by this attribute — locally that does not matter
-(you know which APK you installed), but anything shipped to devices for
+(the harness knows which APK it installed), but anything shipped to devices for
 comparison needs its own revision. Bump the nodejs-mobile tag before a staged
 rollout you intend to measure.
 

package.json

@@ -73,6 +73,8 @@
     "prebackend:build": "npm run backend:install",
     "backend:build": "node ./scripts/build-backend.ts",
     "backend:test": "npm --prefix backend test",
+    "bench:android": "./scripts/benchmark-boot.sh",
+    "bench:report": "node ./scripts/benchmark-report.mjs",
     "e2e:ios": "npm --prefix apps/e2e run build:ios",
     "e2e:android": "npm --prefix apps/e2e run build:android",
     "e2e:test": "maestro test maestro/e2e.yaml"