Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) + 64 GB RAM: 128K context, up to 2,100 tok/s prefill, 43–51 tok/s decode. vLLM runtime with hot MoE experts on the GPU and cold experts computed on the CPU, INT8 KV cache, Docker, OpenAI-compatible API.
moe mixture-of-experts single-gpu rtx3090 int4 llm long-context vllm local-llm llm-inference qwen kv-cache-quantization openai-compatible-api consumer-gpu rtx-3090 cpu-offloading qwen38 qwen3-8-flash-next 128k-context 24gb-vram
-
Updated
Sep 30, 2026 - Python