Repository navigation
moonshine_streaming: bound encoder memory, and fix an out-of-bounds read past 82 s - #222
Merged
Merged
Conversation
The one-shot (and batch) encoder ran the whole utterance as one graph: one
dense [T_enc, T_enc] f32 mask per layer on host and device, plus
[T_enc, T_enc, n_heads] attention scores. Memory grew with T^2 - 12 / 20 /
26 GB of footprint for tiny / small / medium on 5 minutes of audio.
Encode in windows of 1024 emitted frames (~20 s) instead, each carrying the
encoder's full receptive field: cumulative_left_context + the frontend conv
pad on the left, cumulative_right_context on the right. Every layer attends
through a finite sliding window and the convs are causal, so this is exact in
the mathematical sense - the same algebra the streaming path already relies on
and parity-tests. Window cuts sit on encoder-frame boundaries so both strided
convs stay phase-aligned. Inputs that fit one window take the old single-pass
path unchanged.
Not bit-exact: Metal's kernels order reductions by a frame's position in the
graph, so windows that don't start at frame 0 differ at ~5e-4 of the
activation range, uniformly, with no rise at the cuts. Shortening the left
context by 10 or 30 frames produces 1e-2 / 4e-1 spikes at every cut, so the
check would catch a missing-context bug. The residual is about half of what
toggling flash attention does to the single pass.
Peak footprint, Metal, Q8_0, inside the model's 81.9 s input range:
30 s 60 s 80 s
tiny 303 -> 240 669 -> 283 1040 -> 305
small 510 -> 413 1125 -> 515 1734 -> 561
medium 742 -> 609 1612 -> 786 2465 -> 863
Transcripts are byte-identical to main on jfk and 30/60/80/120 s for all three
sizes; stream_parity passes with and without a forced 64-frame window.
TRANSCRIBE_MOONSHINE_STREAMING_ENC_WINDOW overrides the window size.
Beyond 81.9 s (4096 encoder frames) the adapter's pos_emb table runs out:
that is a separate, pre-existing bug (OOB get_rows: CPU aborts, Metal reads
garbage) and not fixed here.
The adapter adds pos_emb[frame] to every encoder frame from a table of
dec_max_position_embeddings (4096) rows - 4096 encoder frames, 81.92 s of
audio per decode. The reference raises IndexError past it. The port had no
bound, so a longer input fed frame indices >= 4096 to ggml_get_rows:
- CPU: GGML_ASSERT(i01 >= 0 && i01 < ne01) aborts the process (rc 134), for
one-shot input over 82 s and for any stream that runs past 82 s - a host
app dictating for a minute and a half on a CPU backend crashes.
- Metal: silent out-of-bounds reads, i.e. garbage transcripts (<unk>,
repetition loops on long files).
Enforce the cap with the contract docs/input-limits.md already defines for an
absolute position wall (voxtral_realtime):
- transcribe_run: input whose padded length needs more than 4096 encoder
frames is rejected with TRANSCRIBE_ERR_INPUT_TOO_LONG before encoding.
- transcribe_run_batch: the same, per utterance (the batched GPU path now
carries a per-row fail status instead of hardcoding INVALID_ARG); the
serial path inherits it from run().
- Streaming: emission stops at frame 4096. The stream is flagged
transcribe_was_truncated() with a single WARN, later audio is dropped (and
not buffered), and feeds/finalize keep returning TRANSCRIBE_OK. A stream
that ends inside the last right-context window past the cap is handled at
finalize the same way.
- apply_adapter_window refuses any frame range outside the table, as the
backstop that keeps get_rows in bounds whatever a caller does.
- capabilities.max_audio_ms reports 81920 (it advertised ~1024 s, derived
from the decoder token cap, which is not the binding limit).
Docs move moonshine_streaming from the soft-window bucket ("output-bound,
~17 min") to the hard-cap bucket and spell out the streaming behaviour.
Verified on tiny/small/medium: 4096 frames (81.92 s) runs, 4097 is rejected,
on Metal and CPU; a 120 s stream on CPU (which aborted before) finalizes OK,
truncated, and its transcript equals one-shot over the first 81.92 s.
tests/moonshine_streaming_batch_truncation is replaced by
tests/moonshine_streaming_input_cap: its long row (love-loss.wav, 197 s) only
"truncated" because the decode was running on out-of-bounds positions. The new
gated test pins the boundary, one-shot rejection, the per-row batch status, and
a stream past the cap (exact transcript on the default backend, word for word
on CPU, whose shape-dependent kernels can move a comma), on both backends.
- Compare the streamed transcript to one-shot exactly on the default backend (byte for byte proves the stream consumed the first 81.92 s and nothing else); on CPU assert only the invariants, since ggml's shape-dependent kernels can move a comma between a windowed and a single-pass run. Drops the word-level comparison helper, which was the weak part of the test. - Feed 90 s instead of the whole 306 s sample: enough to cross the cap. (The runtime is dominated by the at-cap encode's masks either way, ~37 s.)
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
…r#220, handy-computer#223, handy-computer#222, handy-computer#225, handy-computer#224) - handy-computer#220 causal_lm: alias the packed gate+up projection onto the existing weights (no duplicate conversion copies at load) - handy-computer#223 scripts/bench: memory.py peak-memory harness - handy-computer#222 moonshine_streaming: bounded encoder memory past the 82 s training window and an out-of-bounds read fix; input-cap smoke replaces the truncation test - handy-computer#225 catalog: Apple M4 speed cells published; weak-machine targets capped - handy-computer#224 qwen3-asr nar: encoder graph memory bounded Fork note: no fork-owned files in the conflict set.
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
transcribe-e2e (tools/, workspace member) loads the fork's self-contained transcribe.dll exactly the way sound-server does — the transcribe-cpp safe wrapper with the dynload feature, runtime dlopen, no compile-time native dependency — and runs 23 checks over the public safe API only: - library: init_dynamic full-symbol resolve, >=0.5.0 version gate, backend/device enumeration, embedded-VAD probe - ASR core (whisper-tiny): offline run, no_mmap parity (0.4.1 mmap via dynload), run_batch == serial, progress monotonicity + cancel-at-first- tick -> Aborted with partial transcript, de->en translate - families + 0.4.1 behavior: moonshine, sensevoice ITN default ON, fun-asr-mlt ITN default ON (handy-computer#205), qwen3 NAR (handy-computer#224), gigaam windowed 33.8 s Russian (handy-computer#208), parakeet v3 [256,256] local attention (handy-computer#199), multitalker bundle diarize=ON (handy-computer#219, bundle composed on demand via compose-multitalker-bundle.py) - streaming: buffered stream == offline (multitalker streaming ckpt), moonshine-streaming feed/finalize (handy-computer#222) - sortformer: DIARIZE role + LowLatency preset, push-stream session with tentative->committed convergence (fork SFPS) - embedded VAD: one-shot segments, VAD-prechunk -> whisper run_batch (the sound-rs pipeline), stream_feed/flush prob parity Check audio reuses the committed oracle mixes (sortformer-2spk-mix.wav, multitalker-2spk-mix.wav) so results stay comparable with the pinned goldens in tests/. Missing model files SKIP; a FAIL exits 1. scripts/e2e/run-e2e.ps1 drives the whole flow: fetch+verify the embedded VAD asset (sha256 single-sourced from cmake), build the SHARED_EMBED DLL, compose the multitalker bundle, build the CLI, run the matrix. Validated live: 23 pass / 0 skip / 0 fail on the merged 0.4.1 tree. Generated-by: zcode
NairoDorian
added a commit
to NairoDorian/transcribe.cpp
that referenced
this pull request
Nov 14, 2026
…, mmap weights handy-computer#195, memory handy-computer#199/handy-computer#208/handy-computer#222/handy-computer#224, Phonon-2 handy-computer#213, NeMo padding frame handy-computer#252, CBLAS check, x86 XSAVE patch) into fork Upstream handy-computer/transcribe.cpp 18947cf..275cd50, merged by hand. Resolution decisions, so the next merge can follow them: Loader / memory - Upstream's WeightStorage + load_weights (memory-mapped weights where the device can use file bytes) replaces every family's alloc + stream pair. The fork's OOM reclaim retry (alloc_ctx_tensors_with_reclaim) moved into allocate_copied_weights, so every family keeps it. - stream_tensor_data: upstream's direct host reads / 16 MiB chunked uploads when the dtype is unchanged; the fork's whole-tensor staging when a dtype conversion is planned (TQ1_G128 -> Q4_0/Q2_0, float conversions) or the destination is a CPU extra buffer (CPU_REPACK asserts on partial writes). - Parakeet keeps retype_ternary_for_runtime + alloc_cpu_repack_weights ahead of load_weights; repacked tensors are skipped by the mapping. - Graph allocation: alloc_inference_graph (fork graph optimizer, opt-in) now also logs the scheduler high-water mark like upstream's TRANSCRIBE_ALLOC_SCHED_GRAPH; new TRANSCRIBE_ALLOC_INFERENCE_GRAPH macros. Families - Conformer causal pre-encode: upstream handy-computer#199 (symmetric p=2 + tail zeroing, no pad copies) replaces the fork's pad_causal/time-chunked depthwise; the fork's in-place bias/ReLU and offline time-chunked depthwise are kept. Nemotron offline and streaming transcripts are byte-identical. - qwen3_asr: the fork's encoder is kept as is (windowed cu_seqlens attention, R2T2 encoder prefix cache, encoder flash off by default per the drift-free policy). Upstream handy-computer#224's separate subsample graph and flash-on default are NOT adopted here (tracked in the master plan); its early input-length gates (run and run_batch) are adopted. - moonshine_streaming and granite_nar: upstream's reviewed handy-computer#222/handy-computer#224 replace the fork's earlier import of PR handy-computer#162; the fork's per-run GPU release is re-applied as scope guards. gigaam: upstream windowing (handy-computer#208) + per-run scheduler release guard. - Parakeet: Phonon-2 masking rule (mask_after_pointwise) applied on top of the fork's pre-encode policy. Phonon-2 conversion goes through the fork's ternary-aware HF loader (_hf_to_nemo_name / extract_tokenizer_hf). - tests: moonshine_streaming_batch_truncation replaced by upstream's input_cap test; granite_nar_bpe_pool_unit gated on the family being compiled in; weights_mmap_smoke made C++23-clean (u8string). VAD - Reuses the verified codex/vad-integration reconciliation (3-way merged against upstream, base c63b18e): the fork's embedded Earshot API moves to src/transcribe-earshot.* and bindings/rust/.../earshot.rs; upstream's Silero VAD role API keeps transcribe-vad.* and vad.rs. Both build together. Architecture plugin ABI 2 -> 3 (Arch gains the VAD ops table). Build / ggml - src/CMakeLists: upstream's CBLAS symbol check ported into the fork's hoisted BLAS probe (shared with arch plugins, include dirs propagated). - patches/ggml: upstream's 0003 (Metal mapped buffer) and 0004 (x86 XSAVE scoring) adopted; fork patches renumbered 0101-0103 so upstream keeps the low numbers. ggml/ re-vendored at the same pin with all seven patches. - Bindings regenerated: ABI hash 1db3bfd2795a0ad8 (Python, TypeScript, Rust bindgen, Swift pin). Verification (Windows, MSVC 14.52, CPU, full model set): full build incl. tests; ctest 66/66 (1 skipped by design); Rust lib/no_model/vad/serde 28/28; lint_teardown, generator/bindgen/Swift drift checks and the pinned clang-format diff gate pass. Transcript A/B vs the pre-merge build: 23/25 byte-identical (nemotron, parakeet v3, parakeet-redux TQ1, granite 4.1, qwen3-asr 1.7B x 5 clips) plus nemotron streaming 4/4; the two differences are punctuation on the 35 s clip for parakeet v3/redux, from upstream handy-computer#252 dropping the NeMo padding mel frame (both are non-Kestrel checkpoints). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
actually cap moonshine and also reduce its mem usage