Dense Transformer Inference Engine — Pure GGUF · From-Scratch Parser · Zero HuggingFace Dependency
🏷️ LLM Inference · GGUF · KV Cache · Flash Attention · Speculative Decoding · Self-Spec (ACL'24) · Long Context · Pure PyTorch
🌐 English · 中文
PyDense is a production-grade inference engine for dense Transformer models, built entirely from scratch with only two dependencies: PyTorch and NumPy.
Unlike most LLM engines that rely on HuggingFace Transformers, xFormers, llama-cpp, or bitsandbytes, PyDense ships its own:
| Component | Description |
|---|---|
| 🧬 GGUF Parser | Hand-written binary parser: magic-number validation, metadata KV parsing, tensor offset computation, mmap zero-copy |
| 🧠 Dense Transformer | Manual implementations of LLaMA / Qwen2 / Gemma architectures: RoPE, SwiGLU, GQA, RMSNorm |
| 🔤 Tokenizer | Built directly from GGUF metadata tokens — no external tokenizer libraries |
| 📐 Dequantization | Native dequant kernels for all GGML types (Q2_K ~ Q8_0, IQ1~IQ4, BF16, F16, F32) |
All optimizations are enabled automatically — no config files, no tuning knobs, no environment variables.
- Python 3.11+
- CUDA GPU with Compute Capability ≥ 8.0 (Ampere or newer)
- CUDA Driver ≥ 535.x
pip install -r requirements.txt# Point to any GGUF model file
python main.py --model /path/to/model.gguf
# Interactive chat (Ctrl+D or /exit to quit)
python main.py --model /path/to/model.gguf chat
# HTTP API server (OpenAI-compatible)
python main.py --model /path/to/model.gguf --port 8080
# Benchmark
python main.py --model /path/to/model.gguf --benchmark# Interactive guided setup
python launch.py
# One-click start with benchmark
python launch.py --model /path/to/model.gguf --auto
# Skip dependency checks, launch directly
python launch.py --model /path/to/model.gguf --quick# Text completion
curl http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{"prompt": "Hello, how are you?", "max_tokens": 100, "stream": false}'
# Chat completion
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "Hello!"}], "stream": true}'
# List models
curl http://localhost:8080/v1/models
# Health check
curl http://localhost:8080/healthPyDense requires a CUDA GPU with Compute Capability ≥ 8.0 (Ampere or newer).
| Tier | GPUs | Tuning |
|---|---|---|
| 🔬 Deep | NVIDIA A100, H100, RTX 3080, RTX 4090 (24GB), RTX 5090 | Hand-tuned profile (KV frac, batch, block size, compile mode) |
| 🧪 Basic | RTX 30 / 40 / 50 series (e.g. 3060–5090) | Family profile, VRAM-capped |
| 🚫 Dropped | T4 / V100 / Pascal and older | Not supported — engine refuses to start |
Other GPUs (e.g. T4, V100, Pascal) are intentionally not supported; the engine exits with a clear message on startup.
Any GGUF-quantized dense causal language model with these architectures:
| Architecture | Models |
|---|---|
| LLaMA | LLaMA 2 / 3 / 3.1 / 3.2 |
| Qwen2 | Qwen2 / Qwen2.5 (0.5B ~ 72B) |
| Gemma | Gemma 1 / 2 |
Note: PyDense is for dense models only. MoE models (Mixtral, DeepSeek-V2/V3, Qwen2-MoE) are not supported.
All GGML quantization formats are supported with native dequant kernels:
Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K, Q8_0, IQ1_S, IQ2_XXS, IQ2_XS, IQ3_XXS, IQ4_XS, BF16, F16, F32
main.py ← CLI entry point + model loading + server orchestration
launch.py ← Guided interactive launcher (newcomer-friendly)
│
├── GGUF Engine (zero-dependency binary parser)
│ ├── gguf_loader.py ← GGUF v2/v3 binary parser + mmap zero-copy
│ ├── dense_model.py ← Hand-written Transformer (LLaMA/Qwen2/Gemma)
│ └── tokenizer_from_gguf.py ← Tokenizer from GGUF metadata
│
├── Inference Pipeline
│ ├── scheduler.py ← Unified scheduler (Chunked Prefill + Decode dual-stream)
│ ├── goose_core.py ← Goose speculative decoding (PLD + tree attention)
│ ├── attention_kernel.py ← FlashAttention SDPA (torch.compile)
│ └── cache_manager.py ← Hybrid KV cache (PagedAttention + RadixAttention)
│
├── Memory & Hardware
│ ├── vram_budget.py ← VRAM budget manager (startup eval + runtime OOM guard)
│ └── hardware_probe.py ← Hardware detection (CC ≥ 8.0, profile-gated)
│
├── Long Context (ICML'24 / ICLR'25)
│ ├── long_context/self_extend.py ← SelfExtend: 8-32x context extension
│ └── long_context/re_attention.py ← ReAttention: 1M+ token support
│
├── Tools & Services
│ ├── tool_sink.py ← In-model tool-calling framework ([[tool(...)]])
│ ├── api_server.py ← OpenAI-compatible HTTP API (asyncio + stdlib)
│ └── engine_logger.py ← Unified logging system
│
└── requirements.txt ← Only PyTorch + NumPy (ruff/pytest for dev)
| Optimization | Effect |
|---|---|
| 🔥 FlashAttention SDPA | 2–4× attention speedup; math/mem_efficient fallbacks disabled |
| ⚡ cuDNN benchmark + TF32 | Maximum matmul throughput on Ampere+ GPUs |
| 🚀 torch.compile | 10–30% end-to-end speedup (reduce-overhead → default fallback) |
| 📦 PagedAttention KV Cache | Efficient memory management with block-level allocation |
| 🌳 RadixAttention Prefix Cache | BLAKE2b incremental hashing; automatic prefix reuse |
| 📐 Chunked Prefill | Sarathi-style; auto-tuned block size; eliminates first-token latency |
| 🚀 Goose Speculative Decoding | PLD pattern matching + tree attention; 10–40% decode speedup |
| 🔄 Self-Spec Skeleton Draft | Skip-layer draft generation + full-model verification (ACL'24) |
| 🧠 Adaptive KV Compression | H2O + StreamingLLM hybrid; auto-tuned sink/window params |
| 📊 Dual CUDA Stream Pipeline | Prefill stream + Decode stream running concurrently |
| 🧮 VRAM Budget Management | Precise startup evaluation; real-time waterline monitoring; auto-degradation |
No config files, no tuning parameters, no environment variables needed. Everything just works.
# Verbose output
python main.py --model /path/to/model.gguf -v
# Debug mode
export MOE_LOG_LEVEL=debug
python main.py --model /path/to/model.gguf# Default: 128 prompt + 128 generation
python main.py --model /path/to/model.gguf --benchmark
# Long-context test
python main.py --model /path/to/model.gguf --benchmark --prompt-len 2048 --gen-len 256| Metric | Improvement |
|---|---|
| Attention computation | 2–4× faster (Flash SDPA) |
| End-to-end throughput | 10–30% faster (torch.compile) |
| Decode latency | 10–40% faster (Goose + Self-Spec) |
| KV cache memory | ~50% reduction (Paged + Radix) |
| Long-context memory | ~50% reduction (adaptive KV compression) |
| Prefill latency | Eliminated first-token stall (chunked prefill) |
CC BY-NC-SA 4.0 — Attribution-NonCommercial-ShareAlike 4.0 International
- ✅ Personal use, research, education — free
- ✅ Share and adapt — with attribution and same license
- ❌ Commercial use requires separate licensing
PyDense 是一个面向稠密 Transformer 模型的生产级推理引擎，仅依赖 PyTorch + NumPy 两项核心库。
与大多数依赖 HuggingFace Transformers、xFormers、llama-cpp 或 bitsandbytes 的推理引擎不同，PyDense 完全自研：
- GGUF 解析器：手写二进制解析（魔数校验、元数据键值对读取、张量偏移量计算、mmap 零拷贝）
- 稠密 Transformer：手动实现 LLaMA / Qwen2 / Gemma 架构（RoPE、SwiGLU、GQA、RMSNorm）
- 分词器：直接从 GGUF 元数据中读取 tokenizer.ggml.tokens 自行构建
- 反量化：内置全部 GGML 量化类型的反量化内核
pip install -r requirements.txt
# 指定 GGUF 模型文件
python main.py --model /path/to/model.gguf
# 交互式聊天
python main.py --model /path/to/model.gguf chat
# HTTP API 服务
python main.py --model /path/to/model.gguf --port 8080LLaMA 2/3/3.1/3.2、Qwen2/2.5、Gemma 1/2。不支持 MoE 模型。
CC BY-NC-SA 4.0 — 署名-非商业性使用-相同方式共享 4.0 国际