Repository navigation
Conversation
Contributor
|
Thanks I'll try it and merge when I have a chance |
This was referenced Oct 24, 2026
NairoDorian
added a commit
to NairoDorian/transcribe.cpp
that referenced
this pull request
Nov 14, 2026
ggml v0.26.0 (d7cb5741) -> v1.0.0 (417a024b, = master on 2026-11-14), via scripts/sync-ggml.sh. Upstream releases in between: 0.27 (scheduler copy / realloc callbacks, KleidiAI SME2 k-quants, AVX2 Q2_0 dot), 0.28 (TOP_K_MASK, -1 skip ids in MUL_MAT_ID), 0.29 (F8_E4M3, flash_attn_ext_rows, batched tensor copy/set/get), 1.0 (deprecated API and DIAG_MASK ops pruned, YaRN truncation, CUDA few-row quantized matmuls, Vulkan prefill FA). None of the pruned APIs is used by this tree. Patch refresh (0001, 0003, 0004 apply unchanged): - 0002 backend-reg-filter: context moved in ggml_backend_load_all_from_path. - 0101 CUDA pool trim + graph evict: v1.0 added invalidate_cuda_graphs() to the legacy pool; the exported clear() now syncs, frees and invalidates like the pool's own OOM path, and ggml_backend_cuda_trim_pools() drops the backend's captured graphs after trimming (a replayed graph skips the ops that would re-map a trimmed VMM tail). - 0102 TQ1_G128: F8_E4M3 took type id 43, TQ1_G128 stays parked at 96 (COUNT 97). ggml now ships its own AVX2 Q2_0 dot product, so the patch's duplicate is dropped. Re-added the TQ1_G128 MMVQ case, the coopmat2 dequant functions (v1.0 readonly block style) and the backend-ops entry. - 0103 decode KV-trim: K->ne[1] became n_kv_cols in the FA mask-scan gate. Notable for this fork: - CUDA unary ops no longer require a contiguous source, so the conformer GLU sigmoid stays on the GPU (what upstream PR handy-computer#228 works around). - The block_reduce shared-memory race (llama.cpp #26385, backported by audio.cpp #837) is fixed in this ggml. Verification: - CPU (Windows, MSVC 14.52, full model set): full build incl. tests; CTest 66/66 (incl. ternary_tq1_g128_unit, threadpool_oversubscription); Rust lib/no_model/vad/serde 28/28. - CUDA (sm_89, RTX 4070 Laptop, CUDA 13.5): transcribe-cli builds with all patches. Granite 4.1 2B: graph splits 38 -> 6, sigmoids on CPU 16 -> 0, encode 461.7 -> 216.6 ms (german.wav) and 497.1 -> 241.9 ms (dots.wav), 2.05-2.13x, transcripts byte-identical (interleaved, 4 rounds, first dropped). Nemotron 3.5 and Qwen3-ASR 1.7B within noise. - Android arm64 (NDK r30, android-arm64 preset): all 7 CPU ISA variants (armv8.0 .. armv9.2), Vulkan and libtranscribe cross-compile. - Transcripts vs the pre-merge build (5 models x 5 clips + Nemotron streaming x 4, CPU and CUDA): beyond the known upstream handy-computer#252 differences, ggml 1.0 changes single punctuation marks in four cases - CPU: Qwen3 dots.wav, Nemotron streaming german.wav; CUDA: Qwen3 german.wav and fleurs-fr.wav. Deterministic run to run; rounding-level drift from the reworked kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On CUDA, the GLU in the conformer conv module runs its sigmoid on the CPU. The value half of the GLU is a strided view of the pointwise conv output, and ggml-cuda only supports unary ops on contiguous tensors, so the scheduler moves every SIGMOID node to the CPU and copies the activations to the host and back in every block.
This PR adds
transcribe::conformer::glu(), which makes the value half contiguous before the sigmoid, and uses it in the four places that had their own copy of the same GLU code: the shared conv module and the Granite, Granite NAR and Granite5 CTC encoders.The copy does not change any values. Metal and Vulkan already accept the strided input, so for them the only difference is one small extra copy per block.
Scope
glu()helper, used byconv_moduleon the direct pointwise path. The im2col path splits along ne[1], so its views were already contiguous and it is unchanged.conformer_glu_unitchecks the GLU values and that the sigmoid input is contiguous.AI Assistance
Yes. English is not my first language, and I used an AI assistant to help build the code and the wording of this description. I reviewed the changes and ran the measurements below. I found the issue while profiling one of my projects, which uses transcribe.cpp for Cohere Arabic.
Validation
GTX 1660 SUPER (6 GB, sm_75), CUDA 12.9, i5-8500, Linux. Release builds of main (4d7c1cc) and this branch. The machine had other load, so the timings are medians of runs in ABBA order.
Node placement with
GGML_SCHED_DEBUG=2(transcribe-cli --backend cuda):transcribe-bench --backend cuda --warmup 2 --iters 10, median of 20 iterations per build:transcribe-batch-bench --backend cuda, cohere Q8_0, ar-long.wav, time per utterance:In my project, this change alone took a 69 minute Arabic lecture from 259.6s to 223.6s on the same GPU.
Accuracy on CUDA: on 500 human transcribed Arabic clips (FLEURS, Common Voice, SADA, Casablanca and Quran, 79 minutes) with
transcribe-cli --batch --batch-size 8, WER went from 21.31% to 21.38% and CER from 9.69% to 9.68%. The paired bootstrap 95% interval for the WER change is -0.15 to +0.33 points, so I don't see an accuracy change. 26 clips changed (4 better, 6 worse, 16 with the same error count), which I think comes from the sigmoid now running in the CUDA kernel instead of the CPU one. Parakeet gave identical transcripts on jfk, dots, dots-full, death, cj-swimming-drop and fleurs-en.CPU: the tensors dumped with
TRANSCRIBE_DUMP_DIRfor parakeet on jfk.wav are byte identical between main and this branch, andtranscribe-bench --backend cpushowed no slowdown.scripts/ci/clang-format.sh --checkpasses, andctestpasses (62 tests; the diarize CLI smoke is skipped without a model). The new unit test fails if theggml_contis removed.Not tested: Metal, Vulkan, ROCm, and the Granite models, which I don't have. ROCm uses the same ggml-cuda support check, so it should get the same benefit.
The encoder attention has the same kind of fallback: FLASH_ATTN still runs on the CPU in these encoders (48 nodes for cohere, 24 for parakeet). I plan to send a separate PR for that.