From aeda6fddd389692bc6b0e8103acb7200f7af5ae4 Mon Sep 17 00:00:00 2001 From: CJ Pais Date: Tue, 20 Oct 2026 00:17:38 -0700 Subject: [PATCH] scripts/bench: add memory.py, the peak-memory harness the plan needs Every memory branch is accepted on a number, and notes/memory-reduction-plan.md assumes a harness that makes those numbers comparable: one transcribe-cli process at a time (a 16 GB host corrupts peaks otherwise), the same sample, the same metric, appended to a CSV. Until now that was a throwaway shell script in tmp/, which is why the numbers in the audit and the pack-gate-up work were hand-copied. uv run scripts/bench/memory.py --models granite-4.0-1b-speech --samples jfk uv run scripts/bench/memory.py --models --samples dots-full --repeat 2 uv run scripts/bench/memory.py --models sensevoice --duration 1200 Reuses run.py's model/sample resolution and machine detection, so variants and quants are spelled the same way as the perf matrices. Per cell it records: - peak RSS (the kernel's high-water mark, never sampled: no missed peak) - peak footprint (macOS /usr/bin/time -l; what the OS actually accounts) - peak RssAnon (Linux, sampled - the share an OOM killer charges, and the reason #195 exists) - scheduler scratch, summed over backends from the wave-0 instrumentation, which is the portable number on Metal where RSS under-reports GPU scratch - load time, wall time, exit code, and a transcript hash so a branch can be diffed against another run without keeping the text Anything hostile to measurement is left out rather than faked: columns the host cannot measure stay empty. --repeat 2 gives a warm number; --report min is there for noisy hosts but never mixed with last in one table. --env and `--` pass branch-specific switches through without the script knowing about them. --- scripts/bench/memory.py | 530 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 530 insertions(+) create mode 100755 scripts/bench/memory.py diff --git a/scripts/bench/memory.py b/scripts/bench/memory.py new file mode 100755 index 000000000..591d1b0a7 --- /dev/null +++ b/scripts/bench/memory.py @@ -0,0 +1,530 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = [] +# /// +""" +memory.py - peak-memory harness for one (variant, sample) cell at a time. + +Every memory branch in notes/memory-reduction-plan.md is accepted on a number, +not on "it got better", and those numbers are only comparable if they come from +the same harness, run one process at a time (a 16 GB host OOMs and corrupts +peaks otherwise). This is that harness: it runs transcribe-cli serially, records +peak RSS, the platform's own footprint number, and the scheduler scratch the +run committed, and appends one CSV row per cell. + +Usage: + # one variant, one sample + uv run scripts/bench/memory.py --models granite-4.0-1b-speech --samples jfk + + # a family sweep at the plan's reference length, warm (2 runs, keep the 2nd) + uv run scripts/bench/memory.py \\ + --models granite-4.0-1b-speech,granite-speech-4.1-2b,canary-qwen-2.5b \\ + --samples dots-full --repeat 2 + + # duration scaling (generates a cut with ffmpeg if missing) + uv run scripts/bench/memory.py --models sensevoice --samples dots-full \\ + --duration 1200 + + # A/B a branch-specific switch; --env for library env vars, `--` for CLI flags + uv run scripts/bench/memory.py --models granite-4.0-1b-speech --samples jfk \\ + --label pack-gate-up-copy --env TRANSCRIBE_NO_GATE_UP_ALIAS=1 + uv run scripts/bench/memory.py --models qwen3-asr-0.6b --samples jfk \\ + --backend cpu -- --n-ctx 4096 + +Everything after `--` is passed straight to transcribe-cli, so branch-specific +CLI switches work without teaching this script about them. + +Output: reports/memory//