Repository navigation
mmap model weights instead of copying them into anonymous memory - #195
Conversation
Weights are currently read tensor-by-tensor with fin.read() into a staging vector and copied in with ggml_backend_tensor_set, so the whole model lives in anonymous heap. Anonymous pages cannot be reclaimed, so on Android every weight byte is charged to the process by the low-memory killer; file-backed pages can simply be dropped and re-read. On a 5.5 GB phone this is the difference between a 1.7B model running and being killed mid-session. Add load_common::map_tensor_data_cpu(), which mmaps the GGUF read-only, wraps the tensor-data region in ggml_backend_cpu_buffer_from_ptr (whose free_buffer is NULL, so it does not own the pages) and points each tensor at its file offset with ggml_backend_tensor_alloc. A MappedWeights RAII member on the model owns the mapping and is released after the buffer and ctx_meta, both of which point into it. It returns bool rather than transcribe_status: mapping is an optimization, so every failure - unsupported platform, non-CPU backend, open/mmap failure, or a misaligned or truncated file - falls back to the existing alloc_ctx_tensors + stream_tensor_data path, which re-validates and reports real corruption itself. Every tensor is checked before any is allocated so a bad file falls back cleanly rather than half-mapping ctx_meta. Wired up for qwen3_asr, parakeet and whisper. Each was audited first for writes to ctx_meta tensors after load, since mapping read-only turns such a write into a crash. Notably whisper's legacy .bin loader does write frontend.mel_filterbank and frontend.window in place, so only the GGUF path is mapped and bin_load.cpp keeps its own allocation. voxtral, granite, funasr_nano and medasr are untouched and still stream. Peak RssAnon, x86 Linux, Release, 4 threads, one 8.6 s clip: model file before after qwen3-asr 1.7B Q4_0 1114 MiB 1936 MiB 817 MiB parakeet 0.6B Q8_0 705 MiB 1128 MiB 423 MiB whisper distil Q8_0 792 MiB 897 MiB 102 MiB The reduction tracks file size, as expected. Peak RSS is roughly unchanged (file pages still count) but the anonymous share, which is what the OOM killer charges, drops by the size of the model. Transcripts are identical on all three, and dropping the staging copy makes load slightly faster; the only cost is page-fault latency on the first run against a cold page cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Benchmarks via
|
| model | file | base | mmap | |
|---|---|---|---|---|
| qwen3-asr 1.7B Q4_0 | 1114 MiB | load_ms |
1154.2 | 350.6 |
wall_ms mean |
4209.7 | 4331.7 | ||
rtf_wall |
2.05 | 2.00 | ||
| peak RssAnon | 1936 MiB | 828 MiB | ||
| peak RSS | 1943 MiB | 1942 MiB | ||
| parakeet 0.6B Q8_0 | 705 MiB | load_ms |
859.4 | 390.3 |
wall_ms mean |
723.9 | 724.5 | ||
rtf_wall |
11.94 | 11.93 | ||
| peak RssAnon | 1133 MiB | 428 MiB | ||
| peak RSS | 1139 MiB | 1139 MiB | ||
| whisper distil Q8_0 | 792 MiB | load_ms |
633.4 | 33.9 |
wall_ms mean |
11081.5 | 11193.1 | ||
rtf_wall |
0.78 | 0.77 | ||
| peak RssAnon | 916 MiB | 114 MiB | ||
| peak RSS | 923 MiB | 908 MiB |
Output is unchanged. hyp_text is byte-identical between the two builds for all three models (sha256 of the hypothesis matches), and for parakeet — the one family where transcribe-bench populates token_ids_csv — all 39 token IDs are identical too. qwen3_asr and whisper report n_tokens: 0 in the bench JSON so there were no IDs to compare; their hypothesis strings match exactly.
Load gets faster, which I had not expected going in: 18.7× for whisper (633 → 34 ms), 3.3× for qwen (1154 → 351 ms), 2.2× for parakeet (859 → 390 ms). Mapping replaces a full read-plus-copy of the file with page-table setup, and pages fault in lazily during the first run.
rtf_wall is unchanged in every case (2.05→2.00, 11.94→11.93, 0.78→0.77 — all within run-to-run noise). Peak RSS is also unchanged, as expected: the mapped pages still count toward RSS. The point is the split, not the total — RssAnon drops by very close to the file size in each case, and that is what an OOM killer charges against the process.
On-device
Galaxy S21 FE (5.5 GB RAM, Android 16), measured through a host app, RssAnon in kB:
| model | 0.1.3 | 0.3.1 | 0.3.1 + this patch |
|---|---|---|---|
| qwen3-asr 1.7B Q4_0 | 2,064,316 | 1,786,472 | 700,504 |
| parakeet 0.6B Q8_0 | 1,204,496 | 1,160,548 | 439,436 |
| whisper distil Q8_0 | — | — | 97,064 |
The middle column is worth noting on its own: the scratch-memory work already merged in #150/#161 is worth ~278 MB on the 1.7B between 0.1.3 and 0.3.1. This patch is additive to that, and on this device it is the difference between the 1.7B being reclaimed mid-session and running with zero lmkd kills.
Warm-run latency on-device is unchanged (parakeet 1.24 s vs 1.20–1.29 s before); only the first run against a cold page cache costs ~0.7 s extra.
The patch applies unmodified to the released transcribe-cpp-sys 0.3.1 crate as well as to main.
|
Thanks we definitely should do this |
|
Thanks, my phone has 6 gb ram and 6 gb swap, no other way 1.5 gb asr models fit in it only with mmap. |
ggml_backend_metal_device_buffer_mapped wrapped the result of ggml_metal_buffer_map in a backend buffer without checking it, so a failed mapping (newBufferWithBytesNoCopy or residency-set setup returning nil) produced a buffer with a NULL context that crashed on first use. Return NULL instead, as ggml_backend_dev_buffer_from_host_ptr's other implementations do, so a caller can fall back to a copied buffer. Downstream patch 0003, recorded in ggml/UPSTREAM.
fuse_encoder_q_bias added each block's linear_q bias into pos_bias_u / pos_bias_v in place with ggml_backend_tensor_set. That is the one place any family writes into a GGUF-backed weight after load; with the weights memory-mapped read-only it would crash, and it made the loaded tensors differ from the file. Write the sums into new F32 tensors (same shape and name) in their own q_bias_fused ctx + buffer on the primary backend, the way fuse_batch_norm handles BN, and repoint the block's pos_u/pos_v slots. Inputs that are missing or not F32 [d_model] leave that block unfused (q_b add kept). Same arithmetic, so outputs are unchanged.
Builds on the previous commit's approach (map the GGUF, wrap it as a
host-pointer buffer, point the tensors into it) and makes it the one
weight load every family uses, everywhere the device can use host memory.
Loader (src/transcribe-mmap.*, transcribe-load-common.*):
- MappedFile maps the whole file read-only: mmap(PROT_READ, MAP_SHARED) on
POSIX, CreateFileMappingW + MapViewOfFile on Windows (UTF-8 path through
the wide conversion). It then starts reading the file in the background:
F_RDADVISE on Apple, POSIX_FADV_SEQUENTIAL + WILLNEED on Linux/Android,
PrefetchVirtualMemory on Windows. MADV_RANDOM made a cold-cache first
run 70%+ slower; MADV_WILLNEED left macOS ~15% behind the copying load;
F_RDADVISE matches it.
- load_common::load_weights / allocate_weights replace the per-family
alloc_ctx_tensors + stream_tensor_data pair. When mapping is allowed and
the primary device reports buffer_from_host_ptr + mmap_support (CPU
everywhere, Apple7+ Metal), the mapping is wrapped with
ggml_backend_dev_buffer_from_host_ptr and every tensor whose file bytes
are usable as-is (same type and size as in the file, in bounds, aligned
for the device) points into it. Anything else goes into an ordinary
buffer and is streamed, so a family that converts tensors at load
(ecapa_tdnn widens Q8_0 to F16) maps the rest. Vulkan/CUDA/ROCm keep
copying into device memory.
- Wrapping the page-aligned mapping instead of base + data_offset fixes an
abort: ggml_backend_cpu_buffer_from_ptr asserts a 32-byte-aligned base,
which a valid GGUF with general.alignment < 32 does not give.
- WeightStorage owns the mapped and copied buffers plus the mapping and
frees them in order; it replaces each family's backend_buffer (and the
previous commit's MappedWeights). Mapping or wrapping failures log and
fall back to copying.
- Legacy whisper.cpp / Silero .bin weights are converted on the host and
stay copied (allocate_copied_weights).
Mapped weights are read-only. An audit of every family found no writes
into GGUF-backed tensors after load besides cohere's Q-bias fusion (fixed
in the previous commit); CONTRIBUTING and the porting skill now say so.
API: transcribe_model_load_params gains `bool no_mmap` (default false,
zero-default like every field); TRANSCRIBE_NO_MMAP=1 forces it process
wide. transcribe_model_load_file now hands families a full-size params
view so callers built before the field existed read the default. CLI and
transcribe-bench take --no-mmap. The header documents the mmap hazard: do
not truncate or rewrite a loaded model file in place.
Measured (same binary, --no-mmap vs default), peak footprint / RssAnon:
macOS M4 CPU parakeet-tdt-0.6b Q8_0 1198 -> 504 MiB, qwen3-asr-0.6b
Q8_0 1705 -> 745, whisper-large-v3-turbo F16 1860 -> 182,
cohere Q8_0 3481 -> 1173
macOS M4 Metal parakeet 868 -> 172, qwen3-asr 1522 -> 550,
whisper turbo 1877 -> 189, cohere 2493 -> 170
Linux aarch64 RssAnon parakeet 1092 -> 397, qwen3-asr 1471 -> 655,
whisper turbo 1684 -> 129
Warm loads are 3-10x faster; steady-state encode/decode is unchanged;
cold-cache first runs match the copying load. Every family's output is
identical with and without --no-mmap on CPU and Metal (one model each,
plus the multitalker bundle and a whisper.cpp .bin).
Tests: transcribe_weights_mmap_smoke (fixtures, CI) checks the mapping
decision for every loadable fixture family, bit-identical results mapped
vs copied (silero VAD, ecapa langid incl. the partly-mapped Q8_0 model),
the old struct_size, TRANSCRIBE_NO_MMAP, a misaligned GGUF
(arch_silero_vad_misaligned.gguf: copied, same results, no abort), a
truncated file, two models on one file, a non-ASCII path, and Metal when
present. It crashes with SIGBUS if cohere's fusion writes in place again.
Passes on macOS (CPU + Metal), Linux aarch64, and Windows x86_64 (MinGW
build under Wine); the mmap code also compiles for Android API 21.
Co-authored-by: montvid <montvid@users.noreply.github.com>
Regenerate the Python/TypeScript FFI (+ include/transcribe.abihash) and the Rust sys bindings for transcribe_model_load_params.no_mmap, pin the Swift ABI hash, and expose the option: Python Model(no_mmap=) and transcribe(), Rust ModelOptions::no_mmap, TypeScript ModelOptions.noMmap, Swift ModelOptions(noMmap:). READMEs note the memory-mapping behavior and the TRANSCRIBE_NO_MMAP override.
stream_tensor_data read each tensor whole into a staging vector grown to the largest tensor, then handed it to ggml_backend_tensor_set in one call. On every load that copies weights (GPU backends, no_mmap, tensors converted at load) that held the largest tensor in anonymous memory a second time: 243 MiB for Qwen3-ASR-1.7B's token embedding. On device memory the host cannot map, Vulkan also stages each call whole through a per-device buffer it keeps for the life of the process, sized to the largest write. Read host-buffer (CPU) tensors straight into t->data, with no staging copy, and upload everything else in 16 MiB chunks via ggml_backend_tensor_set's offset. That bounds both staging buffers. Every weight buffer is a default buffer type, all of which accept partial writes. Measured on a Ryzen 7840U (RADV iGPU), interleaved A/B, identical transcripts: Vulkan qwen3-asr-1.7b peak anon 311 -> 86 MiB, RSS 364 -> 138 MiB Vulkan whisper-turbo peak anon 100 -> 42 MiB, load 491 -> 447 ms CPU no_mmap qwen3-asr-1.7b load 1316 -> 1023 ms Mapped CPU loads are unchanged; Metal (M4) transcripts are unchanged.
…ile::valid fuse_encoder_q_bias re-checked what weights.cpp already guarantees: q_bias_fusable skipped a block unless attn_q_b and pos_bias_u/v were F32 with d_model elements, and an early return skipped the whole fusion when d_model was not head_dim * n_heads. GET_F32 binds all three as required F32 tensors of exact shape, and read_cohere_hparams rejects d_model not divisible by n_heads, so neither check could fire; if the binding ever broke they would have quietly skipped the fold instead of surfacing it. Fuse every block unconditionally. MappedFile::valid() had no callers. Cohere 03-2026 and arabic-07-2026 (Q8_0, Q4_K_M) give identical transcripts on CPU (mapped and no_mmap) and Vulkan; all 48 blocks fuse.
|
Just a note the PR changed quite significantly from the original. I moved things so that it works for all models, and follows llama.cpp patterns more closely. As well as changing some of the mmap flags, etc. Cheers for the PR |
… layout (handy-computer#219, handy-computer#195) handy-computer#219 multitalker: GGUF layout flattened to the repo root (catalog, docs, compose/bench scripts); compose script refuses to overwrite its input. handy-computer#195 mmap model weights: - new src/transcribe-mmap.{cpp,h} + load_common WeightStorage; allocate_weights/load_weights/stream_tensor_data(..., skip) with file-backed weight serving on CPU/Metal, copies elsewhere - transcribe_model_load_params.no_mmap + TRANSCRIBE_NO_MMAP env - weights_mmap smoke test, api_smoke extension Fork resolutions (protect fork-owned features): - silero_vad loader keeps the fork in-memory branch (embedded VAD asset: stream_tensor_data_from_memory); the file branch adopts the new weight_storage skip argument. init_storage with an empty (adopted) path degrades to copied weights, so embedded VAD never attempts mmap. - bindings + include/transcribe.abihash kept on the fork side (0.5.0 line, fork-generated FFI with the sortformer/VAD surface); the no_mmap option port and abihash recompute land in a follow-up adaptation commit on this branch.
The 0.4.1 mmap sync (handy-computer#195) grafts upstream's params_view normalization into the fork's model_load_prologue helper. Upstream builds the view inside transcribe_model_load_file_impl, where `params = ¶ms_view` rebinds the impl's own pointer; inside the helper the same line only reassigned a by-value copy of the pointer, so the view died at return and the family loaders saw the caller's raw struct. Consequences before this fix, caught by weights_mmap_smoke's old-struct_size mode: a caller whose struct_size predates no_mmap had the garbage byte beyond its struct read as no_mmap (mmap silently disabled), and any caller-built non-0/1 bool byte reached the library as UB-loaded bool. The documented contract in transcribe-load-common.h ("the full-size view that transcribe_model_load_file passes to the family loaders") was not actually delivered. model_load_prologue now writes the view into caller-owned storage; both entry points (file load, embedded VAD) declare the view in their frame and rebind params to it after the prologue returns OK. Error paths never read the view. weights_mmap_smoke (all four modes incl. old struct_size), api_smoke and the parakeet mask/longform/pack_gate_up units pass; transcribe-cli rebuilds clean on MSVC. Generated-by: zcode
…, mmap weights handy-computer#195, memory handy-computer#199/handy-computer#208/handy-computer#222/handy-computer#224, Phonon-2 handy-computer#213, NeMo padding frame handy-computer#252, CBLAS check, x86 XSAVE patch) into fork Upstream handy-computer/transcribe.cpp 18947cf..275cd50, merged by hand. Resolution decisions, so the next merge can follow them: Loader / memory - Upstream's WeightStorage + load_weights (memory-mapped weights where the device can use file bytes) replaces every family's alloc + stream pair. The fork's OOM reclaim retry (alloc_ctx_tensors_with_reclaim) moved into allocate_copied_weights, so every family keeps it. - stream_tensor_data: upstream's direct host reads / 16 MiB chunked uploads when the dtype is unchanged; the fork's whole-tensor staging when a dtype conversion is planned (TQ1_G128 -> Q4_0/Q2_0, float conversions) or the destination is a CPU extra buffer (CPU_REPACK asserts on partial writes). - Parakeet keeps retype_ternary_for_runtime + alloc_cpu_repack_weights ahead of load_weights; repacked tensors are skipped by the mapping. - Graph allocation: alloc_inference_graph (fork graph optimizer, opt-in) now also logs the scheduler high-water mark like upstream's TRANSCRIBE_ALLOC_SCHED_GRAPH; new TRANSCRIBE_ALLOC_INFERENCE_GRAPH macros. Families - Conformer causal pre-encode: upstream handy-computer#199 (symmetric p=2 + tail zeroing, no pad copies) replaces the fork's pad_causal/time-chunked depthwise; the fork's in-place bias/ReLU and offline time-chunked depthwise are kept. Nemotron offline and streaming transcripts are byte-identical. - qwen3_asr: the fork's encoder is kept as is (windowed cu_seqlens attention, R2T2 encoder prefix cache, encoder flash off by default per the drift-free policy). Upstream handy-computer#224's separate subsample graph and flash-on default are NOT adopted here (tracked in the master plan); its early input-length gates (run and run_batch) are adopted. - moonshine_streaming and granite_nar: upstream's reviewed handy-computer#222/handy-computer#224 replace the fork's earlier import of PR handy-computer#162; the fork's per-run GPU release is re-applied as scope guards. gigaam: upstream windowing (handy-computer#208) + per-run scheduler release guard. - Parakeet: Phonon-2 masking rule (mask_after_pointwise) applied on top of the fork's pre-encode policy. Phonon-2 conversion goes through the fork's ternary-aware HF loader (_hf_to_nemo_name / extract_tokenizer_hf). - tests: moonshine_streaming_batch_truncation replaced by upstream's input_cap test; granite_nar_bpe_pool_unit gated on the family being compiled in; weights_mmap_smoke made C++23-clean (u8string). VAD - Reuses the verified codex/vad-integration reconciliation (3-way merged against upstream, base c63b18e): the fork's embedded Earshot API moves to src/transcribe-earshot.* and bindings/rust/.../earshot.rs; upstream's Silero VAD role API keeps transcribe-vad.* and vad.rs. Both build together. Architecture plugin ABI 2 -> 3 (Arch gains the VAD ops table). Build / ggml - src/CMakeLists: upstream's CBLAS symbol check ported into the fork's hoisted BLAS probe (shared with arch plugins, include dirs propagated). - patches/ggml: upstream's 0003 (Metal mapped buffer) and 0004 (x86 XSAVE scoring) adopted; fork patches renumbered 0101-0103 so upstream keeps the low numbers. ggml/ re-vendored at the same pin with all seven patches. - Bindings regenerated: ABI hash 1db3bfd2795a0ad8 (Python, TypeScript, Rust bindgen, Swift pin). Verification (Windows, MSVC 14.52, CPU, full model set): full build incl. tests; ctest 66/66 (1 skipped by design); Rust lib/no_model/vad/serde 28/28; lint_teardown, generator/bindgen/Swift drift checks and the pinned clang-format diff gate pass. Transcript A/B vs the pre-merge build: 23/25 byte-identical (nemotron, parakeet v3, parakeet-redux TQ1, granite 4.1, qwen3-asr 1.7B x 5 clips) plus nemotron streaming 4/4; the two differences are punctuation on the 35 s clip for parakeet v3/redux, from upstream handy-computer#252 dropping the NeMo padding mel frame (both are non-Kestrel checkpoints). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
Weights are read tensor-by-tensor with
fin.read()into a staging vector and copied in withggml_backend_tensor_set(transcribe-load-common.cpp), so the entire model lives in anonymous memory.grep -i mmap src/currently returns nothing.Anonymous pages can't be reclaimed, so Android's low-memory killer charges every weight byte to the process. On a 5.5 GB phone I measured
lmkd: Reclaim 'dev.notune.transcribe' ... to free 1531868kB rss; reason: min2x watermark is breached even after kill— a 1.7B Qwen3-ASR would load and transcribe a one-off file fine, then get killed as soon as a second component (the voice-input panel) was resident at the same time.This is orthogonal to #150/#161/#162, which target compute/scratch growth. This one is purely about where the weights live. #162's note that "Qwen3-ASR 0.6B — ~7.6 GiB RSS ... for an ~811 MiB model" is a good illustration: this patch removes that 811 MiB component from the anonymous total; the rest is still scratch.
Approach
load_common::map_tensor_data_cpu()mmaps the GGUFPROT_READ/MAP_PRIVATE,madvise(MADV_RANDOM)s the tensor region, wraps it inggml_backend_cpu_buffer_from_ptr(whosefree_bufferisNULL, so it doesn't own the pages) and points each tensor at its file offset withggml_backend_tensor_alloc. AMappedWeightsRAII member on the model owns the mapping and is released after the buffer andctx_meta, both of which point into it. Same shape as llama.cpp's mmap path.It returns
bool, nottranscribe_status, on purpose: mapping is an optimization, and there's no suitable "not supported, carry on" code in the enum. Every failure — Windows, non-CPU primary,open/mmapfailure, or a file whose tensors are misaligned or run past EOF — falls back to the existingalloc_ctx_tensors+stream_tensor_datapath, which re-validates and reports genuine corruption itself. All tensors are checked before any is allocated, so a bad file falls back cleanly instead of half-mappingctx_meta.Safety
Mapping read-only turns a post-load write to a weight tensor into a crash, so each arch was audited for
ggml_backend_tensor_settargets before wiring:bn_fusedbuffer; the conv_pw F32 promotion emits new tensors; the decoder's targets are its own joint/LSTM buffers.frontend.mel_filterbank/frontend.windowinto host buffers. The legacy.binloader does write those two in place, so only the GGUF path is mapped andbin_load.cppkeeps its own allocation.voxtral,voxtral_realtime,granite,funasr_nanoandmedasrare untouched and still stream — each needs the same audit before adopting it. The doc comment onmap_tensor_data_cpusays so.Measurements
Peak
RssAnon, x86 Linux, Release,-DGGML_NATIVE=ON, 4 threads, one 8.6 s clip:The reduction tracks file size. Peak RSS is roughly unchanged — file pages still count toward RSS — but the anonymous share, which is what the OOM killer charges, drops by the size of the model.
On the device that motivated this (Galaxy S21 FE, Android 16), the 1.7B went from being reclaimed to running the voice-input panel with 0 lmkd kills.
Verification
Transcripts are byte-identical before and after on all three architectures. Dropping the staging copy makes load slightly faster (8.27 s → 7.38 s for the 1.7B on-device); the only cost is page-fault latency on the first run against a cold page cache — I measured 1.74 s then 1.22/1.23 s for parakeet, against 1.20 s unmapped.
Happy to split this per-arch, gate it behind an option, or drop the whisper/parakeet wiring if you'd rather land the helper alone first.