Static batched decode (LLaMA + Qwen3): B independent sequences per step, up to 41x aggregate throughput - #129

Merged

orionpapadakis merged 11 commits into

beehive-lab:mainfrom

mikepapadim:feat/static-batched-decode

Aug 28, 2026

Commits on Jul 11, 2026

Add static batched decode: B independent LLaMA sequences per step, per-slot KV, 41x aggregate throughput
mikepapadim
committed
0c501a6
Extend static batched decode to Qwen3 (per-head QK-norm + split-half RoPE decode kernels); CUDA-backend docs + multi-model perf
mikepapadim
committed
b7cf126
Add continuous (iteration-level) batching: evict-on-stop + admit-from-queue, +20% throughput / +24% utilization vs static wave
mikepapadim
committed
4843943
Add PagedAttention: block-pool KV + per-slot block table, 10.7x less KV memory at ~1% overhead (LLaMA)
mikepapadim
committed
37e7f61
Add prefix caching: shared prompt prefix prefilled once into pinned blocks, +85% throughput / 85% fewer prefill tokens
mikepapadim
committed
a81268f
Extend paging + prefix caching to Qwen3 (split-half paged RoPE; shared paged flash kernel); +104% throughput with prefix cache
mikepapadim
committed
50eca7d
Doc: add exact prompts + per-result engine flags for reproduction
mikepapadim
committed
19ce1b0
Skip full-vocab logits GEMM on pure-prefill steps (logits only for sampling slots)
mikepapadim
committed
3dd54da
On-device greedy sampling (GPU argmax over logits): transfer B token ids not the 65MB logits tensor, +30% throughput
mikepapadim
committed
fc1c16e
Doc: on-device sampling section + numbers
mikepapadim
committed
0cc44dc

Commits on Jul 14, 2026

Doc: Qwen3 batched-decode quick-reproduce (copy-paste static + continuous serving) with fresh verified numbers; engine javadoc reproduce steps
mikepapadim
committed
dacb7b7