ssd-offloading

⚡ Zero-Stall MoE Inference via Lookahead Prediction & Async DMA Prefetching. Optimized for SSD I/O with Hybrid MLA+Sliding Window Attention.

125B MoE (Qwen3.8-Flash-Next) on 16GB RAM + 8GB GPU: expert cache, next-layer read-ahead and GPU keep-alive for llama.cpp on Windows

A llama.cpp fork that runs Qwen3.8-Flash-Next (180B MoE, 97.5 GB at true Q4) on a 64 GB M1 Max at ~30 tok/s: experts streamed from the SSD, Metal 4, sparse attention, MTP drafts. 1.9-6x faster than other engines on the same file.

Run the full 2.8T-parameter Kimi-K3 checkpoint from SSD on a 64 GB M4 Pro Mac mini: all 93 layers, per-expert streaming, native MLX/Metal. Slow (~32 s/token) but complete.

Kimi K3 (2.8T MoE) on a 16GB-RAM Windows PC: SSD streaming for llama.cpp, 45 -> 4.5 s/token

Run an 816B-parameter MoE (Muse Spark Open) on a 128 GB Mac by streaming experts from SSD with MLX

Run DeepSeek-V4.1-Flash (551B MoE, IQ2_XXS) locally on Apple Silicon with 128K context, even though it's bigger than RAM. Patched llama.cpp Metal runtime streams experts from SSD with a GPU expert cache. Tested on M5 Pro 64 GB: 252 tok/s prefill, 16.3 tok/s decode.