Run mixture-of-experts language models larger than your GPU, with routed experts streamed from NVMe.
A single-GPU CUDA inference engine for MoE models whose weights do not fit in VRAM. Dense weights stay resident on the card; routed experts live on an SSD and are hauled up on demand through an LRU cache in VRAM. CUDA C++, no framework dependencies.
Currently runs GLM-5.2 (753B total / 39B active, MXFP4) on a single 96 GB card:
32 GB dense weights, resident in VRAM
62 GB LRU cache, ~3,094 of 19,200 routed experts
385 GB routed experts on NVMe, fetched per layer per token
Correctness is established. Throughput is not competitive, and the measurements below explain why — that result is the main thing this repository has to offer.
| measured | |
|---|---|
| Correctness | top-5 token ids exact vs. a numpy reference; worst substep 1.69e-06 against a derived 1e-5 gate |
| Single layers | ≤2.2e-08 against a transformers oracle |
| Throughput | 1.227 tok/s warm (RTX PRO 6000 Blackwell, Samsung 9100 PRO) |
| Expert-cache hit rate | 56.5% at 16.1% residency |
See docs/RESULTS.md for the full measurements, the negative controls, and the throughput analysis.
The point of the engine. Three real PRs from a working Gitea instance, reviewed end to end:
prefill 9.30 tok/s decode 0.889 tok/s 21m30s per review, all three ending on EOS
Each review is structured to the prompt and file-scoped. On a C++ Levinson-Durbin solver it identified a division by zero when the prediction error reaches zero, with the trigger condition and a fix; on a TypeScript player it traced parseFloat("") → NaN through both branches of a validator to show the fallback is bypassed. Twenty minutes a review is a nightly-batch tool, not an interactive one.
Expert fetch is 87.6% of layer time, and the bottleneck is a host memory copy rather than the SSD.
Measured on the device and the real packed files: NVMe random reads of the 20 MB expert stride reach 8.78 GB/s at queue depth 1 and 9.26 GB/s at QD 4, with latency scaling linearly past that — the device saturates at depth 1. But the fetch path opens with plain O_RDONLY, and buffered reads measure 6.7 GB/s even with the page cache warm, because every read pays copy_to_user. Effective decode rate works out to 6.0 GB/s: 336 misses × 20.05 MB ÷ 1.125 s per token, predicting 1006 ms of fetch against 985 ms measured.
So the engine sits on the buffered-read ceiling, 35% below what the storage already delivers, while the GPU is idle 96.5% of the time and host-to-device runs at 49.4 GB/s from pinned memory — 5.3× the SSD, and never used as a cache tier.
An earlier version of this section argued the pipeline was structurally shallow-queued: expert selection is data-dependent per layer, so a decode step never has more than 8 — mean 3.5 — reads in flight. The shallow queue is real, and it is not what binds. The 4→8 io-thread null (+0.5%) was read as "nothing left to issue" when it means "the device was already full at 1."
An independent implementation, Colibri (pure C, CPU-first), reports 1.23 tok/s peak on GLM-5.2, against 1.227 here. That was read as two implementations converging on a limit belonging to the technique. A GPU engine tying a CPU engine on a pure data-movement problem is better evidence of an unoptimised data path.
Requires CUDA 12.8+ and a GPU with enough VRAM for the dense weights plus a useful cache.
make ARCH=sm_120 # Blackwell (RTX PRO 6000, RTX 50xx)
make ARCH=sm_89 tests # Ada# 1. Fetch weights (MXFP4 checkpoint, ~420 GB)
huggingface-cli download amd/GLM-5.2-MXFP4 --local-dir ./glm52-mxfp4
# 2. Repack routed experts into per-layer files with a fixed stride
python3 tools/repack_experts_glm.py --model ./glm52-mxfp4 --out ./packed_experts --layers 3-77
# 3. Verify the packed bytes against the checkpoint (samples 4 experts x 6 sub-tensors per layer)
python3 tools/repack_experts_glm.py --model ./glm52-mxfp4 --out ./packed_experts --layers 3 --verify-only
# 4. Generate
./infer_glm --model-dir ./glm52-mxfp4 --packed ./packed_experts \
--prompt "def quicksort(arr):" --tokens 40 --io-threads 4--io-threads selects the fetch strategy: 0 pinned staging only, 1 double-buffered overlap, 4 batched issue (best measured). Beyond 4 there is nothing left to overlap.
./infer_glm --model-dir ./glm52-mxfp4 --packed ./packed_experts \
--serve --port 8081 --max-seq 8192 --tokens 600 --no-thinkAn OpenAI-compatible endpoint: POST /v1/chat/completions (add "stream": true for SSE), GET /v1/models, GET /health. One request at a time — a second concurrent request gets 503 rather than queueing behind a generation that runs for tens of minutes. A request whose prompt plus max_tokens exceeds --max-seq is refused with a 400 naming all three numbers, since truncating to fit would answer a prompt the caller did not send.
Two decode defences exist and are off by default, because greedy argmax with no penalty is the configuration every correctness result here was measured under. --rep-penalty F divides the logits of recently-emitted tokens (multiplying the negative ones, which is the half that is easy to get backwards). --degen-window N watches the decoded character mix and stops a generation whose recent output has lost its letters — the failure mode where a model emits valid distinct tokens in an endless word-association loop, which no token-id repeat check can see. Any byte with the high bit set counts as a letter, so Russian review text is not mistaken for degeneration.
Two properties follow from the measured speed rather than from taste. Prefill takes minutes, so the stream sends SSE keepalive comments until the first token; a client that waits for any byte would otherwise time out first. And --no-think matters: with reasoning on, a review-sized budget is spent entirely on the reasoning trace and no answer is reached. Sampling is greedy, so temperature is ignored rather than silently approximated.
--max-seq is a standing decision here, because the KV cache is allocated once at load and trades directly against the expert pool — roughly 180 KB per position across the 78 layers.
Tokenization is delegated to a Python sidecar using AutoTokenizer, so any model with a tokenizer.json works without a bespoke exporter. Stop tokens come from generation_config.json where the model ships one, which is the only place GLM-5.2 records that <|user|> and <|observation|> end a turn — deriving them from the tokenizer's own metadata gets a plausible-looking set that stops nothing.
The correctness claims are backed by negative controls — checks are only meaningful if they can fail:
make tests
./test_glm_mxfp4 # MXFP4 dequant, 8 seeds x 2 shapes
./test_glm_expert_cache --packed-dir ./packed_experts --layer 3
./test_glm_layer --model-dir ./glm52-mxfp4 --oracle glm-oracle --layer 3
./test_glm_chain --model-dir ./glm52-mxfp4 --packed ./packed_experts --ref glm-ref
python3 tools/check_ref_vs_oracle.py --negative-controlsThe last command injects known defects and asserts they are caught. Three of them — a flipped MXFP4 nibble order, routed_scaling_factor 1.0 instead of 2.5, and the wrong LoRA-norm epsilon — are separated from baseline by 3.4e+04x to 1.9e+05x.
One of those deserves emphasis: the LoRA-epsilon defect produced the exactly correct top-5 tokens while getting layer-40 and layer-77 expert selection wrong. An end-to-end token-match gate would have passed it. Only per-substep comparison caught it.
- Long-context arithmetic is validated one layer at a time, not end to end. GLM-5.2's DSA sparse-attention indexer is implemented and wired into attention, so the old
index_topk(2048-token) abort is gone. Belowindex_topkthe top-k selects every key, the index mask is a no-op, and the sparse path is bit-identical to the dense one — asserted intest_glm_layer. Above it, at 4096 tokens,test_glm_layercompares an indexer-owning layer and a consuming layer against atransformersoracle: with the oracle's key selection forced in, every substep agrees to the bf16 fixture floor (worst 0.83 bf16 ulp of its own scale); with the CUDA indexer choosing, the two agree on 2043 of 2048 keys and the worst substep is 10.05 ulp. The measured limits are recorded honestly: the test detects a selection error of ≳8 keys in 2048 but not 1, and it cannot see ak_normeps of 1e-5 instead of 1e-6 (2.4e-03 on index scores, the same size as the bf16 floor). The full chain above 2048 still has no oracle —ref_glm_chain.pycarries the same 2048 limit. - Single GPU. No tensor/pipeline parallelism.
- Greedy decode. No batching. The serving endpoint handles one request at a time.
- One model family so far.
MIT — see LICENSE. Copyright (c) 2026 Sergey Subbotin.
Model weights are not covered by this licence and are not distributed here. GLM-5.2's weights are released by Z.ai under MIT; the MXFP4 quantisation used above is published by AMD.
This is an independent implementation. No code from any other inference engine is included.