feat(scripts): local A/B boot benchmark harness - #253
Open
gmaclennan wants to merge 2 commits into
Open
Conversation
gmaclennan
force-pushed
the
feat/boot-benchmark-harness
branch
from
August 22, 2026 14:26
5460adf to
2a8b8e7
Compare
`scripts/benchmark-boot.sh --ab before.apk after.apk` runs interleaved cold boots of `apps/integration`, alternating which build goes first each round, discarding the warm-up boot after each install and wiping the compile cache before every boot so neither build inherits the other's. It prints medians, deltas, Mann–Whitney p-values and per-round medians — the last being the check that matters on a machine that is also doing other work. Peak RSS, RSS and the heap breakdown come from the `[comapeo.memory] boot` line the backend already logs once per launch, so no root is needed and the harness works unmodified on a production device; `adb root` only adds a 50 ms `/proc` trajectory on top. Parsing and statistics live in a pure `scripts/lib/benchmark-core.mjs` covered by `npm run test:scripts`; the shell wrapper is only device orchestration. Usage, the flags, how to read the output and the ways a run can mislead you are in docs/BENCHMARKING.md.
gmaclennan
force-pushed
the
feat/boot-benchmark-harness
branch
from
August 24, 2026 13:50
2a8b8e7 to
c87f3d9
Compare
Cut the root-only 50 ms /proc timeline end to end, the dumpsys meminfo capture, the unused memoryEpoch/alive outputs, and the free-text boot-milestone regexes — every number the harness reports now comes from the structured [comapeo.*] crumbs and the backend's own [comapeo.memory] boot line. Fixes from review: exact run-id matching via one tested parseRunId (a label that was a prefix of another absorbed its runs), safe_label applied to --label, a warning when the compile-cache wipe cannot work without root (and docs corrected to stop claiming cold-cache there), adb pull failures reported instead of crashing collect, per-invocation raw capture directories so the before/after workflow keeps both sides, a date +%s.%N fallback for older toybox, a sleep in the PID wait loop that busy-spun during the measured launch window, logcat filtered at the source by message regex (the Comapeo:NodeJS tag's colon makes a tag:priority filterspec impossible), an 8 s warm-up boot instead of the full duration, and node:util parseArgs in benchmark-collect.mjs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #249, which adds
backend/lib/memory-snapshot.jsand the[comapeo.memory] bootlog line this harness reads. Review that one first; the base branch here isfeat/backend-memory-telemetry.Answering "is build B lighter than build A?" has so far meant a throwaway script each time. This makes it one command:
It runs interleaved cold boots of
apps/integration, alternating which build goes first each round and discarding the warm-up boot after each install. It prints medians (not means — one scheduling stall drags a mean around), deltas, two-sided Mann–Whitney p-values, and per-round medians.The per-round table is the part worth reading first. A development machine is doing other things and host load drifts over the twenty-odd minutes a full run takes, so the question is not whether the two medians differ but whether they separate the same way in every round. If the rounds disagree, you measured the machine.
Every number comes from logcat: boot timings from the module's structured
[comapeo.*]lifecycle crumbs, and peak RSS, RSS and the heap breakdown from the backend logging its own reading once per launch, so the harness works unmodified on an unrooted production device. The one thing root changes is the compile-cache wipe between boots — app cache directories are private, so withoutadb rootthe measured boots are warm-cache and the harness warns up front. With no--apk/--abit buildsapps/integrationin Release first — benchmark against Release, never debug, wherestartForegroundalone is about ten times slower.Parsing and statistics live in a pure
scripts/lib/benchmark-core.mjswith no device or filesystem dependency, covered bynpm run test:scripts; the shell wrapper is only device orchestration. Output lands inbenchmark-results/(gitignored): one JSON per label plus the raw per-boot logcat captures, kept in a per-invocation directory so the second half of a before/after comparison does not overwrite the first.docs/BENCHMARKING.mdgains the usage sections — the flags, how to read the output, and the ways a run can mislead you (steady-state RSS is bimodal,VmSizeis not a footprint).Following review, the harness was cut to this core: the root-only 50 ms
/proctimeline, thedumpsys meminfocapture and the free-text log-message matching are gone (all recoverable from history), run ids are matched exactly, and logcat is filtered at the source to the[comapeo.*]lines the collector reads.Validation
Used to re-run the V8 pointer-compression A/B end to end — 24 boots, one command — reproducing an earlier throwaway-script measurement independently: peak RSS −12.4 MB, live V8 heap −37.9%, per-round medians non-overlapping. That run predates the review slim-down; the protocol and statistics are unchanged since, but the slimmed scripts have not yet been re-run against a device.
npm run lint,npm test(78) andnpm run test:scripts(40) pass, and both shell scripts passbash -n.Depends on #248 for the default build path (
--apk/--abwork without it).