fsdp

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

Best practices & guides on how to write distributed pytorch training code

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

A PyTorch native library for training speculative decoding models

Repo for Qwen Image Finetune

Find why PyTorch training is slow, compare runs, and catch performance regressions in CI.

Educational distributed training and inference library for heterogeneous local hardware: Mac minis, Raspberry Pis and GPUs over plain Python sockets. FSDP, GRPO, parameter-server, expert and data parallelism.

Write once, run anywhere; ezpz 🍋

A comprehensive hands-on guide to building production-grade distributed applications with Ray - from distributed training and multimodal data processing to inference and reinforcement learning.

Research platform for model training, evaluation, and experimentation across architectures, benchmarks, and recipes.

META LLAMA3 GENAI Real World UseCases End To End Implementation Guide

Fast and easy distributed model training examples.

Interactive exercises for understanding compute, communication, and dependencies in distributed LLM training traces.

Billus 大模型技能库：面向 LLM、VL、多模态与图像生成模型的训练、调参、剪枝、量化与工程化技能库

Forge kernels — fused Triton kernels for faster, leaner LLM fine-tuning. One-call patching into Hugging Face models via forge.patch(model), with FSDP2 multi-GPU support and a growing set of kernels and architectures. Every speedup backed by a committed benchmark. Apache-2.0.

A minimal, hackable pre-training stack for GPT-style language models

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem