Skip to content

conformer: make the GLU sigmoid input contiguous so CUDA keeps it on the GPU - #228

Open
AliOsm wants to merge 1 commit into
handy-computer:mainfrom
AliOsm:conformer-glu-contiguous-sigmoid
Open

AliOsm wants to merge 1 commit into
handy-computer:mainfrom
AliOsm:conformer-glu-contiguous-sigmoid

Conversation

@AliOsm

@AliOsm AliOsm commented Oct 24, 2026

Copy link
Copy Markdown

Summary

On CUDA, the GLU in the conformer conv module runs its sigmoid on the CPU. The value half of the GLU is a strided view of the pointwise conv output, and ggml-cuda only supports unary ops on contiguous tensors, so the scheduler moves every SIGMOID node to the CPU and copies the activations to the host and back in every block.

This PR adds transcribe::conformer::glu(), which makes the value half contiguous before the sigmoid, and uses it in the four places that had their own copy of the same GLU code: the shared conv module and the Granite, Granite NAR and Granite5 CTC encoders.

The copy does not change any values. Metal and Vulkan already accept the strided input, so for them the only difference is one small extra copy per block.

Scope

  • src/conformer: new glu() helper, used by conv_module on the direct pointwise path. The im2col path splits along ne[1], so its views were already contiguous and it is unchanged.
  • src/arch/granite, granite_nar, granite5_ctc: use the helper instead of their own GLU code.
  • tests: conformer_glu_unit checks the GLU values and that the sigmoid input is contiguous.

AI Assistance

Yes. English is not my first language, and I used an AI assistant to help build the code and the wording of this description. I reviewed the changes and ran the measurements below. I found the issue while profiling one of my projects, which uses transcribe.cpp for Cohere Arabic.

Validation

GTX 1660 SUPER (6 GB, sm_75), CUDA 12.9, i5-8500, Linux. Release builds of main (4d7c1cc) and this branch. The machine had other load, so the timings are medians of runs in ABBA order.

Node placement with GGML_SCHED_DEBUG=2 (transcribe-cli --backend cuda):

                                  SIGMOID on CPU    graph splits
                                  main -> branch    main -> branch
cohere-transcribe-arabic Q8_0     48 -> 0           196 -> 100
parakeet-tdt-0.6b-v3 Q8_0         24 -> 0            97 -> 49

transcribe-bench --backend cuda --warmup 2 --iters 10, median of 20 iterations per build:

                                    encode ms               total ms
cohere Q8_0, ar-long.wav (25.7 s)   1499 -> 1337 (1.12x)    1686 -> 1524 (1.11x)
parakeet Q8_0, dots.wav (35.3 s)     793 ->  680 (1.17x)    2038 -> 1869 (1.09x)

transcribe-batch-bench --backend cuda, cohere Q8_0, ar-long.wav, time per utterance:

batch 1   1717 -> 1563 ms
batch 4   1601 -> 1491 ms
batch 8   1595 -> 1451 ms

In my project, this change alone took a 69 minute Arabic lecture from 259.6s to 223.6s on the same GPU.

Accuracy on CUDA: on 500 human transcribed Arabic clips (FLEURS, Common Voice, SADA, Casablanca and Quran, 79 minutes) with transcribe-cli --batch --batch-size 8, WER went from 21.31% to 21.38% and CER from 9.69% to 9.68%. The paired bootstrap 95% interval for the WER change is -0.15 to +0.33 points, so I don't see an accuracy change. 26 clips changed (4 better, 6 worse, 16 with the same error count), which I think comes from the sigmoid now running in the CUDA kernel instead of the CPU one. Parakeet gave identical transcripts on jfk, dots, dots-full, death, cj-swimming-drop and fleurs-en.

CPU: the tensors dumped with TRANSCRIBE_DUMP_DIR for parakeet on jfk.wav are byte identical between main and this branch, and transcribe-bench --backend cpu showed no slowdown.

scripts/ci/clang-format.sh --check passes, and ctest passes (62 tests; the diarize CLI smoke is skipped without a model). The new unit test fails if the ggml_cont is removed.

Not tested: Metal, Vulkan, ROCm, and the Granite models, which I don't have. ROCm uses the same ggml-cuda support check, so it should get the same benefit.

The encoder attention has the same kind of fallback: FLASH_ATTN still runs on the CPU in these encoders (48 nodes for cohere, 24 for parakeet). I plan to send a separate PR for that.

@AliOsm
AliOsm requested a review from cjpais as a code owner October 24, 2026 12:57
@cjpais

cjpais commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Thanks I'll try it and merge when I have a chance

NairoDorian added a commit to NairoDorian/transcribe.cpp that referenced this pull request Nov 14, 2026
ggml v0.26.0 (d7cb5741) -> v1.0.0 (417a024b, = master on 2026-11-14), via
scripts/sync-ggml.sh. Upstream releases in between: 0.27 (scheduler copy /
realloc callbacks, KleidiAI SME2 k-quants, AVX2 Q2_0 dot), 0.28 (TOP_K_MASK,
-1 skip ids in MUL_MAT_ID), 0.29 (F8_E4M3, flash_attn_ext_rows, batched
tensor copy/set/get), 1.0 (deprecated API and DIAG_MASK ops pruned, YaRN
truncation, CUDA few-row quantized matmuls, Vulkan prefill FA). None of the
pruned APIs is used by this tree.

Patch refresh (0001, 0003, 0004 apply unchanged):
- 0002 backend-reg-filter: context moved in ggml_backend_load_all_from_path.
- 0101 CUDA pool trim + graph evict: v1.0 added invalidate_cuda_graphs() to
  the legacy pool; the exported clear() now syncs, frees and invalidates
  like the pool's own OOM path, and ggml_backend_cuda_trim_pools() drops the
  backend's captured graphs after trimming (a replayed graph skips the ops
  that would re-map a trimmed VMM tail).
- 0102 TQ1_G128: F8_E4M3 took type id 43, TQ1_G128 stays parked at 96
  (COUNT 97). ggml now ships its own AVX2 Q2_0 dot product, so the patch's
  duplicate is dropped. Re-added the TQ1_G128 MMVQ case, the coopmat2
  dequant functions (v1.0 readonly block style) and the backend-ops entry.
- 0103 decode KV-trim: K->ne[1] became n_kv_cols in the FA mask-scan gate.

Notable for this fork:
- CUDA unary ops no longer require a contiguous source, so the conformer
  GLU sigmoid stays on the GPU (what upstream PR handy-computer#228 works around).
- The block_reduce shared-memory race (llama.cpp #26385, backported by
  audio.cpp #837) is fixed in this ggml.

Verification:
- CPU (Windows, MSVC 14.52, full model set): full build incl. tests;
  CTest 66/66 (incl. ternary_tq1_g128_unit, threadpool_oversubscription);
  Rust lib/no_model/vad/serde 28/28.
- CUDA (sm_89, RTX 4070 Laptop, CUDA 13.5): transcribe-cli builds with all
  patches. Granite 4.1 2B: graph splits 38 -> 6, sigmoids on CPU 16 -> 0,
  encode 461.7 -> 216.6 ms (german.wav) and 497.1 -> 241.9 ms (dots.wav),
  2.05-2.13x, transcripts byte-identical (interleaved, 4 rounds, first
  dropped). Nemotron 3.5 and Qwen3-ASR 1.7B within noise.
- Android arm64 (NDK r30, android-arm64 preset): all 7 CPU ISA variants
  (armv8.0 .. armv9.2), Vulkan and libtranscribe cross-compile.
- Transcripts vs the pre-merge build (5 models x 5 clips + Nemotron
  streaming x 4, CPU and CUDA): beyond the known upstream handy-computer#252 differences,
  ggml 1.0 changes single punctuation marks in four cases - CPU: Qwen3
  dots.wav, Nemotron streaming german.wav; CUDA: Qwen3 german.wav and
  fleurs-fr.wav. Deterministic run to run; rounding-level drift from the
  reworked kernels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants