From 2d6bf872dbf8674c7168e59452c922b346dcd809 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 22:14:12 -0700 Subject: [PATCH 01/17] blog: Advanced CUDA Graph Techniques in Inference Adds the CUDA Graph post covering the runner/backend refactor, Breakable CUDA Graph, full CUDA Graph for prefill, and CUDA Graph memory footprint. Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 189 ++++++++++++++++++ .../blog/breakable_cuda_graph/bcg-design.svg | 183 +++++++++++++++++ .../blog/breakable_cuda_graph/cg-memory.svg | 89 +++++++++ .../blog/breakable_cuda_graph/diffusion.svg | 51 +++++ .../breakable_cuda_graph/full-prefill.svg | 52 +++++ .../breakable_cuda_graph/prefill-build.svg | 52 +++++ 6 files changed, 616 insertions(+) create mode 100644 blog/2026-08-15-advanced-cuda-graph.md create mode 100644 public/images/blog/breakable_cuda_graph/bcg-design.svg create mode 100644 public/images/blog/breakable_cuda_graph/cg-memory.svg create mode 100644 public/images/blog/breakable_cuda_graph/diffusion.svg create mode 100644 public/images/blog/breakable_cuda_graph/full-prefill.svg create mode 100644 public/images/blog/breakable_cuda_graph/prefill-build.svg diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md new file mode 100644 index 000000000..5083cab8b --- /dev/null +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -0,0 +1,189 @@ +--- +title: "Advanced CUDA Graph Techniques in Inference" +author: "SGLang Team" +date: "August 15, 2026" +previewImg: /images/blog/breakable_cuda_graph/bcg-design.svg +type: blog +--- + +## TL;DR + +CUDA Graphs promise to remove kernel-launch overhead, but getting close to that benefit in a real inference engine requires graphing as much of the workload as possible without sacrificing compatibility, startup time, or memory. + +In SGLang, we refactored CUDA Graph support around a common runner/backend interface, making different capture strategies reusable across execution paths. For the more complex prefill path, the SGLang community introduced Breakable CUDA Graph and pioneered full CUDA Graph support on the FA4 and FlashInfer attention backends, both of which were first developed by the SGLang community as open-source serving techniques. We also dive deeper into CUDA Graph memory management, including memory reuse across shapes and graph segments, which is becoming an increasingly important part of SGLang’s overall memory management. + +## Background + +An inference step is not a single kernel but a sequence of many GPU operations. In modern LLM serving engines, repeatedly launching these operations from the CPU can introduce noticeable overhead, especially for latency-sensitive workloads. CUDA Graph reduces this overhead by recording the GPU work once and replaying it with much lower launch overhead. + +But applying CUDA Graphs effectively in a modern inference engine is not straightforward. The graph design must fit different execution phases, remain compatible with complex kernels and runtime-dependent behavior, and control the capture-time and memory overhead introduced by the graphs themselves. As inference stacks become more complex, proper CUDA Graph integration becomes increasingly important. + +This post walks through how CUDA Graph support is built in SGLang and what we changed: + + + +## CUDA Graph in SGLang: the Runner/Backend Split and Flexible Combinations + +Before this refactor, CUDA Graph support had grown around individual execution paths. Decode, prefill, and speculative decoding each had their own CUDA Graph runners, with overlapping logic for capture shapes, static buffers, replay, and graph configuration. As more execution modes and capture strategies were added, this duplication made it harder to reuse infrastructure and made CUDA Graph-related server arguments increasingly ambiguous. + +The [refactor](https://github.com/sgl-project/sglang/pull/23906) separates these responsibilities into two layers. A **runner** manages the execution-specific state needed for capture and replay: captured shapes, static input buffers, attention metadata, and the padding of live batches into captured shapes. A **backend** determines how that execution is captured, whether as one full graph, a sequence of breakable segments, or compiler-generated pieces. + +Because runners depend only on a common backend interface, each execution path can choose its capture strategy independently. Prefill and decode have separate runners, and speculative decoding adds more: the EAGLE draft, draft-extend and frozen-KV MTP draft steps each get their own runner built on the decode runner, while target verify is the decode runner itself, capturing more than one token per request. + + + +

The runner prepares each execution path for capture and replay, while the backend determines how the forward is turned into replayable graphs: as one full graph, segmented during capture, or traced and split before capture.

+ +### Full CUDA Graph + +The full backend captures one `torch.cuda.CUDAGraph` for each selected shape, with no eager regions and the fewest replay-time launches of the three backends. This works naturally for decode: each request contributes one token, so the primary shape variable is batch size, which can be covered by a set of captured batch-size buckets. Prefill varies along more dimensions and is therefore harder; we discuss it in [its own section](#full-cuda-graph-for-prefill). + +### Breakable CUDA Graph + +Breakable CUDA Graph (BCG) captures graph-safe regions while allowing selected operations to run eagerly between graph segments. An incompatible operation can be marked with `@eager_on_graph`; capture stops before the marked function and resumes afterward, producing a sequence of CUDA Graph segments separated by eager regions. + +Unlike compiler-based piecewise capture, these breaks are inserted directly during capture rather than discovered by tracing the full model first. We discuss the mechanism and why SGLang moved to this design in the next section. + +### TC piecewise CUDA Graph + +The third backend reaches similar segmentation through a compiler. `torch.compile` traces the forward with `fullgraph=True`, the resulting FX graph is split at registered split points, and each piece is compiled and captured on its own. It was SGLang's first answer to partial CUDA Graph capture and still ships for platforms where breakable capture has not been validated. + +## Breakable CUDA Graph: Eager Breaks without a Compiler + +CUDA Graph traditionally requires the captured region to be fully graph-compatible. In practice, modern inference workloads contain operations that cannot be captured directly. Prefill attention is a common example: some attention backends depend on runtime metadata and host-side preparation. A single incompatible operation can therefore prevent CUDA Graph from covering a much larger part of the forward. + +We introduced **Breakable CUDA Graph (BCG)** to make capture more flexible. The mechanism and the `@eager_on_graph` decorator landed first as part of CUDA Graph debug mode in [#19102](https://github.com/sgl-project/sglang/pull/19102), and were then built into a breakable piecewise backend for prefill in [#22218](https://github.com/sgl-project/sglang/pull/22218). Instead of requiring the entire forward to be graph-compatible, BCG allows selected operations to run eagerly while capturing the graph-compatible regions around them. At a high level, the forward becomes a sequence of CUDA Graph segments connected by explicit eager breaks. + +### Design and Mechanism + +CUDA Graph works best when replay follows a fixed sequence of GPU operations without host participation. Real inference forwards, however, contain operations that do not fit naturally inside that model: attention backends may plan from live sequence lengths, collectives may involve runtime coordination, and serving features may update state dynamically. + +Giving up on CUDA Graph whenever one such operation appears would leave much of the forward uncaptured. BCG instead lets developers mark the incompatible region directly with `@eager_on_graph`. During capture, the current graph segment is closed when execution reaches the marked function, the function runs eagerly, and capture resumes afterward in a new segment. + +At replay time, the recorded graph segments and eager functions run in the same order. The tensor crossing an eager break is created by the preceding captured segment and registered as a persistent boundary buffer, so its device address remains fixed. The following captured segment is captured against that same address. During replay, the eager function therefore writes its newly computed result back into this boundary buffer rather than returning a newly allocated tensor, allowing the next segment to read the updated value from the address it was originally captured with. BCG never inspects or traces the operations inside the eager region: they only need to execute correctly. + +From a functionality perspective, BCG and the earlier torch-compile-based piecewise backend produce the same kind of replayable structure: CUDA Graph segments separated by eager regions. The key difference is how that structure is constructed. TC piecewise first asks the compiler to understand the full forward and then splits the resulting graph. BCG places the splits directly while capture is happening. + +### Benefits + +**Faster startup.** For compiler-based piecewise graphs, compilation — not CUDA Graph capture — became the dominant setup cost. Measured separately, `torch.compile` accounts for 78–86% of the time spent preparing prefill graphs. The cost also grows with model complexity: compilation alone takes about 90 seconds on a 235B MoE and 158 seconds on GLM-5.2. BCG removes that phase entirely and reaches segmented execution in a single capture pass. + + + +

Time to build the prefill CUDA Graphs, 42 captured shapes, TP4 on 4×GB300.

+ +The compilation overhead was also visible in day-to-day development. In our CI setup at the time, compilation was often repeated across test runs, making CUDA Graph tests noticeably slower. Better caching could mitigate this, but removing the compiler from the capture path also removed this extra source of complexity from the development loop. + +**Broader compatibility.** SGLang relies heavily on custom CUDA, Triton, and JIT-compiled kernels that are not native PyTorch operators. To make these kernels visible to `torch.compile`, we often had to wrap them through `torch.library` and provide fake implementations for tracing. This introduced compiler-specific scaffolding throughout the kernel stack. + +More importantly, the compiler also constrained **where graph boundaries could be placed**. Inputs and outputs crossing a registered operator boundary had to be representable by the compiler. When the natural boundary involved more specialized runtime state or return types, we sometimes had to search for a different cutting point or enlarge the eager region simply to expose an interface the compiler could handle. As the serving stack grew, the compiler boundary increasingly influenced the structure of code that was otherwise unrelated to compilation. + +BCG removes this constraint at eager breaks: the graph system does not need to understand how the marked function is implemented or trace through its internals, allowing graph boundaries to follow serving logic rather than compiler tracing and type requirements. As CUDA Graph had to coexist with DP attention, MoE all-to-all backends, LoRA, PD disaggregation, hierarchical cache, deterministic inference, and other rapidly evolving features, making CUDA Graph work increasingly started to feel like a torch.compile integration project. New kernels often meant custom-op registrations and fake implementations, while new features could force us to move graph boundaries simply to satisfy the compiler. With BCG, incompatible regions can remain ordinary eager execution, substantially reducing this compiler-specific engineering overhead. + +**Debuggable by construction.** A captured CUDA Graph replays as an opaque unit: ordinary Python does not execute inside it, which makes prints, assertions, and step-by-step inspection difficult. BCG naturally leaves eager regions where normal Python still runs on every replay. + +SGLang extends this idea with [`--debug-cuda-graph`](https://github.com/sgl-project/sglang/pull/19102), which effectively wraps the whole forward in an eager break. The model then executes eagerly while still going through the CUDA Graph runner, static buffers, replay path, and metadata preparation. This provides a useful debugging boundary: if the problem remains, it is likely in the model or runner path; if it disappears, capture itself becomes the primary suspect. + +### BCG in Diffusion + +BCG has also been [adopted by SGLang’s diffusion stack](https://github.com/sgl-project/sglang/pull/27436). Diffusion repeatedly executes the same DiT forward during denoising, making CUDA Graph especially useful when those forwards contain many small, launch-bound kernels. + + + +This is particularly effective when execution is launch-bound. For example, after warmup, Qwen-Image at 512×512 on a single B200 improves from 6.48 s to 2.45 s end-to-end latency, and Z-Image improves from 1.231 s to 0.662 s. + + + +

End-to-end latency after warmup. Each bar pair uses the same model workload and seed.

+ +The broader lesson is that BCG removes launch overhead; it does not reduce model FLOPs or make compute-bound kernels cheaper. Its advantage is largest when exposed launch gaps are a meaningful fraction of execution time. + +## Full CUDA Graph for Prefill + +Full CUDA Graph is straightforward for decode because each request contributes one token: the main varying dimension is batch size. Prefill is harder because a batch varies in two dimensions at once — the total number of tokens and the number of requests those tokens belong to — while a captured graph requires both to remain fixed. Together with attention backends that depend on runtime metadata, this made full CUDA Graph difficult to apply to prefill and was one of the main reasons we adopted Breakable CUDA Graph there. + +More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph ([#27988](https://github.com/sgl-project/sglang/pull/27988)), including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. This is an exciting experimental feature that is still under active development: backend coverage is limited today, and we are continuing to improve compatibility, capture policies, and performance. + +### Making prefill static + +SGLang fixes the token dimension with token buckets. A live batch is padded to the nearest captured token count, much like decode pads batch size to a captured bucket. + +The request dimension is handled separately. Each captured graph reserves a fixed number of request slots. Live requests occupy the first slots; unused ones are rewritten as zero-length sentinels, with zero sequence and extend lengths and offsets parked after the real tokens. If a batch contains more requests than the graph has slots, it falls back to eager execution. + + + +

At replay, tokens are padded to the captured bucket while unused request slots are filled with zero-length sentinels.

+ +The sentinel metadata must be rewritten on every replay because the captured graph still reads the entire request table. Attention metadata is likewise rebuilt outside the graph for the padded batch before replay. Today, full prefill capture therefore requires attention backends that support this style of metadata preparation, including FlashAttention and FlashInfer. + +### What does the padding cost? + +The two forms of padding have very different costs. + +Padded tokens are real work. They become actual rows in the captured batch and therefore pass through dense projections as part of the same GEMMs. SGLang carries the true token count separately, allowing MoE routing, attention, and linear-attention kernels to skip much of the padded region, but dense computation still pays for those extra rows. + +Empty request slots are much cheaper. In FlashAttention's variable-length scheduler, work is derived from each sequence's actual length rather than assigning a fixed amount of computation to every request slot. A zero-length request therefore contributes essentially no attention work; it mainly adds metadata and a small amount of scheduling overhead. + +This asymmetry is important: token padding is the expensive dimension, while request-slot padding is comparatively cheap. + +Full prefill capture is still an experimental feature. It has to be enabled explicitly — the engine warns that `full` is experimental and points to breakable or tc_piecewise for production workloads — and it currently works mainly on the FlashAttention (fa4) and FlashInfer backends, which are the ones that build extend-mode metadata the way the captured path needs. Broadening backend support and tuning the bucket and slot choices is still ahead of us. + +## Memory Footprint of CUDA Graphs + +Memory poses two separate challenges: keeping a segmented capture from multiplying resident memory, and capturing far enough that resident graph memory actually replaces the worst eager activation peak. + +### Reuse inside a segmented capture + +A segmented backend could easily multiply graph memory: every captured shape contains multiple graph segments, and each segment has intermediates that must remain valid for replay. BCG avoids that multiplication through three forms of reuse. + + + +One value cannot be treated this way: the tensor that carries data across an eager break. The next graph segment is captured against its address, so that buffer must stay alive and be updated in place on every replay. + +With these reuse mechanisms, even a large capture table remains modest: 42 shapes across a 78-layer MoE add 2.4 GB of graph memory on GLM-5.2. + +### Capture through the chunked-prefill size + +CUDA Graphs change the shape of prefill memory usage. Graph memory is resident: it is allocated during capture and remains for the lifetime of the server. Eager activations are transient: each prefill allocates working memory, and the largest supported prefill determines the peak. + +Capturing a prefill shape moves much of that transient working set into the graph's resident memory pool. But this only helps for shapes that actually replay a graph. If the capture ladder stops below the maximum prefill size, the largest prefill still falls back to eager execution and retains the original activation peak — while the server also pays for all of the resident graphs below it. + +This makes the capture ceiling more important than the number of captured shapes. Since `chunked_prefill_size` bounds the largest single prefill forward, capturing through that size removes the worst eager activation peak. + + + +

Prefill memory above the no-graph resident baseline, measured after one prefill at exactly the chunked-prefill size.

+ +Ceilings below the chunk size sit slightly *above* the no-graph baseline: they add resident graphs while the activation peak stays exactly where it was. Once the ceiling reaches the chunk size, that peak goes away entirely. + +Capturing through the chunked-prefill size buys two things: + + + +## Acknowledgments + +This work was a collaboration between the SGLang team and the Meta team. + +SGLang: Yuwei An*, Cheng Wan, Xiaoyu Zhang, Mick Qian + +Meta: Shiyang Chen*, Lianmin Zheng + +We also thank the NVIDIA, AMD, and Meta PyTorch teams for their help along the way. + +\* Equal contribution. diff --git a/public/images/blog/breakable_cuda_graph/bcg-design.svg b/public/images/blog/breakable_cuda_graph/bcg-design.svg new file mode 100644 index 000000000..37ac0e42e --- /dev/null +++ b/public/images/blog/breakable_cuda_graph/bcg-design.svg @@ -0,0 +1,183 @@ + + + + + + + + + + + + + + + + + + + + + + + + Runner + + capture time + + + capture_prepare + + + capture + + + capture_one + + + replay time + + + can_run_graph + + + load_batch + + + replay + + + + + + + + + + + + capture_one() + + replay() + + + Backend + + + Full CUDA Graph + capture the whole forward as one graph + + one capture + + + + + proj + + norm + + attention + + moe + + a2a + + proj + + norm + + mlp + + + + + + Breakable CUDA Graph + cut in time, while capturing — no whole-graph pass + + one pass + + + + capture + capture + capture + + + exit ✂ + exit ✂ + + + + + segment + + attention + + segment + + a2a + + segment + + + + one shared memory pool — keeps tensors alive across every break + + + + + TC Piecewise CUDA Graph + walk the whole graph first, then cut + + ① trace entire forward + + Torch Dynamo traces every op + + + + + + proj + + norm + + attention + + moe + + a2a + + proj + + norm + + mlp + + + ② split at registered + split points + + + + + + + ✂ + ✂ + + ③ compile + capture + each piece + + + compile → graph + + eager + + graph + + eager + + compile → graph + + diff --git a/public/images/blog/breakable_cuda_graph/cg-memory.svg b/public/images/blog/breakable_cuda_graph/cg-memory.svg new file mode 100644 index 000000000..46f1ee895 --- /dev/null +++ b/public/images/blog/breakable_cuda_graph/cg-memory.svg @@ -0,0 +1,89 @@ + + + + Prefill memory above the no-graph resident baseline + after one prefill at exactly the chunked-prefill size + + + + captured graphs (resident) + + activation peak (per request) + + + + + + + + + + + + + 00.250.50 + 0.751.001.25 GB + 00.250.50 + 0.751.001.25 GB + + + + Llama-3.3-70B · TP4 · chunk 8192 + + + none + 0.83 + + + + 1024 + 0.85 + + + + 2048 + 0.87 + + + + 4096 + 0.90 + + + + 8192 + = chunk + 0.14 + + + gpt-oss-120b · TP2 · chunk 16384 + + + none + 1.00 + + + + 2048 + 1.02 + + + + 4096 + 1.03 + + + + 8192 + 1.05 + + + + 16384 + = chunk + 0.10 + + The no-graph bar is not zero: it is the activation peak every prefill pays. Capturing below the chunk size adds resident graphs without removing that peak. + Once the largest captured shape covers the chunk size, the peak disappears and total memory drops below the no-graph baseline. + diff --git a/public/images/blog/breakable_cuda_graph/diffusion.svg b/public/images/blog/breakable_cuda_graph/diffusion.svg new file mode 100644 index 000000000..bde968d84 --- /dev/null +++ b/public/images/blog/breakable_cuda_graph/diffusion.svg @@ -0,0 +1,51 @@ + + SGLang diffusion end-to-end latency with eager execution and Breakable CUDA Graph + Five model workloads normalized independently to their eager latency. Breakable CUDA Graph reduces end-to-end latency for launch-bound diffusion models on B200 and H200. + + Diffusion latency after warmup — eager vs BCG + Each row is normalized to its own eager baseline · shorter is better · absolute e2e latency shown at right + + eager + BCG + + + B200 · single GPU + + 025%50%75%100% + + eager → BCG + speedup + + + + + + Qwen-Image512×512 + + 6.480 → 2.450 s2.64× + + Z-Image256×256 + + 1.231 → 0.662 s1.86× + + Ideogram 4512×512 + + 1.625 → 0.991 s1.64× + + + + H200 + + SANA1.5-1.6B1024×1024 · 1 GPU · 20 steps + + 0.821 → 0.608 s1.35× + + LTX-2 two-stage768×512×121f · 2 GPUs · CFG parallel + + 8.538 → 6.893 s1.24× + + + + Bars compare only eager and BCG within each row; workloads, step counts, resolutions, and hardware differ across rows. + BCG targets exposed launch gaps. It does not reduce model FLOPs, text encoding, or VAE work. + \ No newline at end of file diff --git a/public/images/blog/breakable_cuda_graph/full-prefill.svg b/public/images/blog/breakable_cuda_graph/full-prefill.svg new file mode 100644 index 000000000..6a7ce088b --- /dev/null +++ b/public/images/blog/breakable_cuda_graph/full-prefill.svg @@ -0,0 +1,52 @@ + + + + + + + + + + + the batch + 2 requests + 100 tokens + + + request A · 60 tokens + + request B · 40 tokens + + + the captured + graph + 128-token bucket + 4 request slots + + tokens — padded up to the bucket + + A + + B + + 28 padded + + request slots — unused ones zeroed + + slot 0 — A + + slot 1 — B + + slot 2 — length 0 + + slot 3 — length 0 + + + + + + padded tokens — real rows through every dense projection + + + empty slots — no attention tiles; a metadata row and a spare block + diff --git a/public/images/blog/breakable_cuda_graph/prefill-build.svg b/public/images/blog/breakable_cuda_graph/prefill-build.svg new file mode 100644 index 000000000..d6f94257f --- /dev/null +++ b/public/images/blog/breakable_cuda_graph/prefill-build.svg @@ -0,0 +1,52 @@ + + + + Cold-start time to build prefill CUDA Graphs + TP4 on 4×GB300 · 42 captured shapes per configuration · weight loading and kernel JIT excluded + + + + compile + + capture + + + + + + Qwen3-235B-A22B + 94-layer MoE + + BCG + + 27.7 + 27.7 s + + TC piecewise + + 90.4 + + 16.2 + 106.6 s + 3.8× slower + + + GLM-5.2 + 78-layer MoE + DSA + + BCG + + 35.2 + 35.2 s + + TC piecewise + + 158.2 + + 24.9 + 183.1 s + 5.2× slower + + BCG removes the compiler phase; its graph capture alone is slightly slower than TC piecewise capture. + From db55d870a7de53fe2ecf1c2fd92581ebf31b8519 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 22:44:35 -0700 Subject: [PATCH 02/17] blog: add Baizhou Zhang, Yusheng Su, Ke Bao to acknowledgments Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index 5083cab8b..5d6376973 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -180,7 +180,7 @@ Capturing through the chunked-prefill size buys two things: This work was a collaboration between the SGLang team and the Meta team. -SGLang: Yuwei An*, Cheng Wan, Xiaoyu Zhang, Mick Qian +SGLang: Yuwei An*, Cheng Wan, Xiaoyu Zhang, Mick Qian, Baizhou Zhang, Yusheng Su, Ke Bao Meta: Shiyang Chen*, Lianmin Zheng From a7bc6b228c4bb92498eb19c138d19e3edb7ece8d Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 22:46:07 -0700 Subject: [PATCH 03/17] blog: thank Thinking Machines Lab in acknowledgments Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index 5d6376973..c67c3d19c 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -184,6 +184,6 @@ SGLang: Yuwei An*, Cheng Wan, Xiaoyu Zhang, Mick Qian, Baizhou Zhang, Yusheng Su Meta: Shiyang Chen*, Lianmin Zheng -We also thank the NVIDIA, AMD, and Meta PyTorch teams for their help along the way. +We also thank the NVIDIA, AMD, Thinking Machines Lab, and Meta PyTorch teams for their help along the way. \* Equal contribution. From f2f26280cc43239099bcc3682be3cbb3283faa83 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:00:59 -0700 Subject: [PATCH 04/17] blog: use gpt-oss and GLM-5.2 for the prefill memory figure Both panels are now measured at the same chunked-prefill size (8192), so they differ only by model. Updates the accompanying numbers, and corrects the claim that the activation peak disappears entirely: on GLM it falls to 0.35 GB because the sparse-attention indexer still runs eagerly at a break. Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 4 +- .../blog/breakable_cuda_graph/cg-memory.svg | 116 +++++++++--------- 2 files changed, 62 insertions(+), 58 deletions(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index c67c3d19c..aede5f503 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -167,12 +167,12 @@ This makes the capture ceiling more important than the number of captured shapes

Prefill memory above the no-graph resident baseline, measured after one prefill at exactly the chunked-prefill size.

-Ceilings below the chunk size sit slightly *above* the no-graph baseline: they add resident graphs while the activation peak stays exactly where it was. Once the ceiling reaches the chunk size, that peak goes away entirely. +Ceilings below the chunk size sit slightly *above* the no-graph baseline: they add resident graphs while the activation peak stays exactly where it was. Once the ceiling reaches the chunk size, the largest prefill finally replays a graph and that peak collapses — to essentially nothing on gpt-oss-120b (0.56 GB to 0.001 GB), and from 1.55 GB to 0.35 GB on GLM-5.2, whose sparse-attention indexer still runs eagerly at a break. Capturing through the chunked-prefill size buys two things:
    -
  • Lower total memory. The activation peak stops being paid per request, and the total lands below the no-graph baseline — 0.69 GB lower on Llama-3.3-70B, 0.90 GB on gpt-oss-120b. Modest against a footprint of a few hundred gigabytes, but a saving rather than a cost.
    • +
  • Lower total memory. The activation peak stops being paid per request, and the total lands below the no-graph baseline — 0.51 GB lower on gpt-oss-120b, 1.10 GB on GLM-5.2. Modest against a footprint of a few hundred gigabytes, but a saving rather than a cost.
  • Predictable memory usage. A workload-dependent activation spike becomes a fixed allocation established at capture time. The engine can account for that memory up front instead of reserving headroom for a transient peak that appears only during large prefills.
diff --git a/public/images/blog/breakable_cuda_graph/cg-memory.svg b/public/images/blog/breakable_cuda_graph/cg-memory.svg index 46f1ee895..f535b3921 100644 --- a/public/images/blog/breakable_cuda_graph/cg-memory.svg +++ b/public/images/blog/breakable_cuda_graph/cg-memory.svg @@ -1,9 +1,9 @@ - - + Prefill memory above the no-graph resident baseline - after one prefill at exactly the chunked-prefill size + after one prefill at exactly the chunked-prefill size (8192 tokens) · each panel has its own y-scale @@ -12,78 +12,82 @@ activation peak (per request) - - - - - - - - + + + + + + - 00.250.50 - 0.751.001.25 GB - 00.250.50 - 0.751.001.25 GB + 00.20 + 0.400.60 GB - - Llama-3.3-70B · TP4 · chunk 8192 + gpt-oss-120b · TP2 · chunk 8192 - + none - 0.83 + 0.560 - + 1024 - 0.85 + 0.575 - - + + 2048 - 0.87 + 0.580 - - + + 4096 - 0.90 + 0.591 - - + + 8192 = chunk - 0.14 + 0.055 + + + + + + + + + + 00.50 + 1.001.50 GB + - - gpt-oss-120b · TP2 · chunk 16384 + GLM-5.2-FP8 · TP4 · chunk 8192 - + none - 1.00 - - - - 2048 - 1.02 - - - - 4096 - 1.03 - - - - 8192 - 1.05 - - - - 16384 + 1.552 + + + + 1024 + 1.572 + + + + 2048 + 1.584 + + + + 4096 + 1.607 + + + + 8192 = chunk - 0.10 + 0.456 - The no-graph bar is not zero: it is the activation peak every prefill pays. Capturing below the chunk size adds resident graphs without removing that peak. - Once the largest captured shape covers the chunk size, the peak disappears and total memory drops below the no-graph baseline. From 84679d3f0b75d395547fa96eb955161bdf7b456b Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:05:05 -0700 Subject: [PATCH 05/17] blog: scale down the figures Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index aede5f503..3648eb473 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -35,7 +35,7 @@ The [refactor](https://github.com/sgl-project/sglang/pull/23906) separates these Because runners depend only on a common backend interface, each execution path can choose its capture strategy independently. Prefill and decode have separate runners, and speculative decoding adds more: the EAGLE draft, draft-extend and frozen-KV MTP draft steps each get their own runner built on the decode runner, while target verify is the decode runner itself, capturing more than one token per request. - +

The runner prepares each execution path for capture and replay, while the backend determines how the forward is turned into replayable graphs: as one full graph, segmented during capture, or traced and split before capture.

@@ -73,7 +73,7 @@ From a functionality perspective, BCG and the earlier torch-compile-based piecew **Faster startup.** For compiler-based piecewise graphs, compilation — not CUDA Graph capture — became the dominant setup cost. Measured separately, `torch.compile` accounts for 78–86% of the time spent preparing prefill graphs. The cost also grows with model complexity: compilation alone takes about 90 seconds on a 235B MoE and 158 seconds on GLM-5.2. BCG removes that phase entirely and reaches segmented execution in a single capture pass. - +

Time to build the prefill CUDA Graphs, 42 captured shapes, TP4 on 4×GB300.

@@ -101,7 +101,7 @@ BCG has also been [adopted by SGLang’s diffusion stack](https://github.com/sgl This is particularly effective when execution is launch-bound. For example, after warmup, Qwen-Image at 512×512 on a single B200 improves from 6.48 s to 2.45 s end-to-end latency, and Z-Image improves from 1.231 s to 0.662 s. - +

End-to-end latency after warmup. Each bar pair uses the same model workload and seed.

@@ -119,7 +119,7 @@ SGLang fixes the token dimension with token buckets. A live batch is padded to t The request dimension is handled separately. Each captured graph reserves a fixed number of request slots. Live requests occupy the first slots; unused ones are rewritten as zero-length sentinels, with zero sequence and extend lengths and offsets parked after the real tokens. If a batch contains more requests than the graph has slots, it falls back to eager execution. - +

At replay, tokens are padded to the captured bucket while unused request slots are filled with zero-length sentinels.

@@ -163,7 +163,7 @@ Capturing a prefill shape moves much of that transient working set into the grap This makes the capture ceiling more important than the number of captured shapes. Since `chunked_prefill_size` bounds the largest single prefill forward, capturing through that size removes the worst eager activation peak. - +

Prefill memory above the no-graph resident baseline, measured after one prefill at exactly the chunked-prefill size.

From 1d170803b9042d2ddbc40c2281c64d74c820b31f Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:08:40 -0700 Subject: [PATCH 06/17] blog: use a consistent "text #PR" form for all references Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index 3648eb473..b13d5f8c5 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -31,7 +31,7 @@ This post walks through how CUDA Graph support is built in SGLang and what we ch Before this refactor, CUDA Graph support had grown around individual execution paths. Decode, prefill, and speculative decoding each had their own CUDA Graph runners, with overlapping logic for capture shapes, static buffers, replay, and graph configuration. As more execution modes and capture strategies were added, this duplication made it harder to reuse infrastructure and made CUDA Graph-related server arguments increasingly ambiguous. -The [refactor](https://github.com/sgl-project/sglang/pull/23906) separates these responsibilities into two layers. A **runner** manages the execution-specific state needed for capture and replay: captured shapes, static input buffers, attention metadata, and the padding of live batches into captured shapes. A **backend** determines how that execution is captured, whether as one full graph, a sequence of breakable segments, or compiler-generated pieces. +The refactor [#23906](https://github.com/sgl-project/sglang/pull/23906) separates these responsibilities into two layers. A **runner** manages the execution-specific state needed for capture and replay: captured shapes, static input buffers, attention metadata, and the padding of live batches into captured shapes. A **backend** determines how that execution is captured, whether as one full graph, a sequence of breakable segments, or compiler-generated pieces. Because runners depend only on a common backend interface, each execution path can choose its capture strategy independently. Prefill and decode have separate runners, and speculative decoding adds more: the EAGLE draft, draft-extend and frozen-KV MTP draft steps each get their own runner built on the decode runner, while target verify is the decode runner itself, capturing more than one token per request. @@ -87,11 +87,11 @@ BCG removes this constraint at eager breaks: the graph system does not need to u **Debuggable by construction.** A captured CUDA Graph replays as an opaque unit: ordinary Python does not execute inside it, which makes prints, assertions, and step-by-step inspection difficult. BCG naturally leaves eager regions where normal Python still runs on every replay. -SGLang extends this idea with [`--debug-cuda-graph`](https://github.com/sgl-project/sglang/pull/19102), which effectively wraps the whole forward in an eager break. The model then executes eagerly while still going through the CUDA Graph runner, static buffers, replay path, and metadata preparation. This provides a useful debugging boundary: if the problem remains, it is likely in the model or runner path; if it disappears, capture itself becomes the primary suspect. +SGLang extends this idea with `--debug-cuda-graph` [#19102](https://github.com/sgl-project/sglang/pull/19102), which effectively wraps the whole forward in an eager break. The model then executes eagerly while still going through the CUDA Graph runner, static buffers, replay path, and metadata preparation. This provides a useful debugging boundary: if the problem remains, it is likely in the model or runner path; if it disappears, capture itself becomes the primary suspect. ### BCG in Diffusion -BCG has also been [adopted by SGLang’s diffusion stack](https://github.com/sgl-project/sglang/pull/27436). Diffusion repeatedly executes the same DiT forward during denoising, making CUDA Graph especially useful when those forwards contain many small, launch-bound kernels. +BCG has also been adopted by SGLang’s diffusion stack [#27436](https://github.com/sgl-project/sglang/pull/27436). Diffusion repeatedly executes the same DiT forward during denoising, making CUDA Graph especially useful when those forwards contain many small, launch-bound kernels.
  • Capture the real serving shapes. Resolution, video frame count, prompt-conditioning length, CFG mode, and the selected transformer can all affect the capture signature. We warm up the shapes that are actually served and fall back to eager execution for unseen signatures.
    • @@ -111,7 +111,7 @@ The broader lesson is that BCG removes launch overhead; it does not reduce model Full CUDA Graph is straightforward for decode because each request contributes one token: the main varying dimension is batch size. Prefill is harder because a batch varies in two dimensions at once — the total number of tokens and the number of requests those tokens belong to — while a captured graph requires both to remain fixed. Together with attention backends that depend on runtime metadata, this made full CUDA Graph difficult to apply to prefill and was one of the main reasons we adopted Breakable CUDA Graph there. -More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph ([#27988](https://github.com/sgl-project/sglang/pull/27988)), including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. This is an exciting experimental feature that is still under active development: backend coverage is limited today, and we are continuing to improve compatibility, capture policies, and performance. +More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph [#27988](https://github.com/sgl-project/sglang/pull/27988), including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. This is an exciting experimental feature that is still under active development: backend coverage is limited today, and we are continuing to improve compatibility, capture policies, and performance. ### Making prefill static @@ -147,7 +147,7 @@ A segmented backend could easily multiply graph memory: every captured shape con
    • One shared memory pool across segments. Every segment for a captured shape uses the same CUDA Graph pool, allowing intermediate storage to be reused rather than pinned separately for each segment.
      • -
    • Weak references at eager breaks. Tensors passed into a break are held weakly when the graph pool already owns their storage, avoiding unnecessary Python references that would extend tensor lifetimes. The tensor weak-reference technique comes from vLLM #9724, which introduced it so that captured graphs could share output buffers instead of each pinning its own.
      • +
    • Weak references at eager breaks. Tensors passed into a break are held weakly when the graph pool already owns their storage, avoiding unnecessary Python references that would extend tensor lifetimes. The tensor weak-reference technique comes from vLLM #9724, which introduced it so that captured graphs could share output buffers instead of each pinning its own.
    • One output buffer across capture sizes. Capture sizes share a single maximum-sized output buffer, sliced to the rows needed by each shape, instead of allocating one output buffer per shape.
    From ccf42156355d87d5942bc46be72771b60712dc03 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:10:08 -0700 Subject: [PATCH 07/17] blog: fix broken diffusion figure (missing xmlns) The chart was extracted from an inline block, where the SVG namespace is implicit. As a standalone .svg file the browser cannot parse it without an explicit xmlns, so the image rendered broken. Co-Authored-By: Claude Opus 5 (1M context) --- public/images/blog/breakable_cuda_graph/diffusion.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/public/images/blog/breakable_cuda_graph/diffusion.svg b/public/images/blog/breakable_cuda_graph/diffusion.svg index bde968d84..b6bfa9c6a 100644 --- a/public/images/blog/breakable_cuda_graph/diffusion.svg +++ b/public/images/blog/breakable_cuda_graph/diffusion.svg @@ -1,4 +1,4 @@ - + SGLang diffusion end-to-end latency with eager execution and Breakable CUDA Graph Five model workloads normalized independently to their eager latency. Breakable CUDA Graph reduces end-to-end latency for launch-bound diffusion models on B200 and H200. From 45412cae2ea07fb03a65a3d8f04e3bae48b85d85 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:14:47 -0700 Subject: [PATCH 08/17] blog: drop the diffusion SVG's inline root style Left over from when the chart was inline HTML; as a standalone file used via it fought the page's own sizing and cropped the footnotes. Co-Authored-By: Claude Opus 5 (1M context) --- public/images/blog/breakable_cuda_graph/diffusion.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/public/images/blog/breakable_cuda_graph/diffusion.svg b/public/images/blog/breakable_cuda_graph/diffusion.svg index b6bfa9c6a..e77d8265f 100644 --- a/public/images/blog/breakable_cuda_graph/diffusion.svg +++ b/public/images/blog/breakable_cuda_graph/diffusion.svg @@ -1,4 +1,4 @@ - + SGLang diffusion end-to-end latency with eager execution and Breakable CUDA Graph Five model workloads normalized independently to their eager latency. Breakable CUDA Graph reduces end-to-end latency for launch-bound diffusion models on B200 and H200. From 2a0e469840c24a21baddf8299576fba464a9cece Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:27:12 -0700 Subject: [PATCH 09/17] blog: add the prefill-only latency comparison Adds a "Faster prefill" benefit with a two-panel figure: gpt-oss-120b, where every backend runs (full 1.85x, breakable 1.62x, tc_piecewise 1.39x over eager), and GLM-5.2, where only BCG can capture at all (1.60x). Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 5 + .../breakable_cuda_graph/prefill-ttft.svg | 107 ++++++++++++++++++ 2 files changed, 112 insertions(+) create mode 100644 public/images/blog/breakable_cuda_graph/prefill-ttft.svg diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index b13d5f8c5..465c212f4 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -79,6 +79,11 @@ From a functionality perspective, BCG and the earlier torch-compile-based piecew The compilation overhead was also visible in day-to-day development. In our CI setup at the time, compilation was often repeated across test runs, making CUDA Graph tests noticeably slower. Better caching could mitigate this, but removing the compiler from the capture path also removed this extra source of complexity from the development loop. +**Faster prefill.** Segmented capture also pays off at replay, not only at startup. On gpt-oss-120b (TP4, 4×GB300) where every backend runs, prefill is 1.62× faster than eager with BCG and 1.85× with full capture, while the compiler-based backend reaches 1.39× — BCG is 17% faster than TC piecewise at replay. On GLM-5.2 the comparison is shorter, because only BCG can capture it at all: 1.60× over eager, while TC piecewise cannot trace the forward and full capture has no path for its sparse attention. Both models are flat across a 32× range in prompt length, which is the signature of launch overhead rather than compute. + + +

    Prefill-only latency: a fixed input length with one output token, one request at a time. Measured with decode graphs disabled in every arm, so the only difference is how prefill is captured.

    + **Broader compatibility.** SGLang relies heavily on custom CUDA, Triton, and JIT-compiled kernels that are not native PyTorch operators. To make these kernels visible to `torch.compile`, we often had to wrap them through `torch.library` and provide fake implementations for tracing. This introduced compiler-specific scaffolding throughout the kernel stack. More importantly, the compiler also constrained **where graph boundaries could be placed**. Inputs and outputs crossing a registered operator boundary had to be representable by the compiler. When the natural boundary involved more specialized runtime state or return types, we sometimes had to search for a different cutting point or enlarge the eager region simply to expose an interface the compiler could handle. As the serving stack grew, the compiler boundary increasingly influenced the structure of code that was otherwise unrelated to compilation. diff --git a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg new file mode 100644 index 000000000..074cb1214 --- /dev/null +++ b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg @@ -0,0 +1,107 @@ + + + Prefill latency by capture strategy + median TTFT · one request at a time · fixed input length, one output token · lower is better + + + no graph + + + tc_piecewise + + + breakable + + + full + gpt-oss-120b · TP4 + every backend works here + + 0 + + 40 + + 80 + + 120 + + 160 ms + + + + + + + + no graph + + + + + + + + tc_piecewise + + + + + + + + breakable + + + + + + + + full + 64 + 128 + 256 + 512 + 1024 + 2048 + input length (tokens) + GLM-5.2-FP8 · TP4 + only BCG can capture this model + + 0 + + 150 + + 300 + + 450 + + 600 ms + + + + + + + + no graph + + + + + + + + breakable + 64 + 128 + 256 + 512 + 1024 + 2048 + input length (tokens) + tc_piecewise — Torch Dynamo cannot trace this forward + full — sparse attention has no captured-metadata path + Both curves are flat in prompt length: at these sizes prefill is dominated by launch overhead, and capture removes a roughly fixed cost. + From ec6382726b2ccef42ac263d36d85d6864a6c5474 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:36:36 -0700 Subject: [PATCH 10/17] blog: keep only gpt-oss in the prefill latency figure Single panel, so the four curves have room and the line labels are not clipped. The GLM result stays in the prose. Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 4 +- .../breakable_cuda_graph/prefill-ttft.svg | 169 +++++++----------- 2 files changed, 67 insertions(+), 106 deletions(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index 465c212f4..b163b20dd 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -79,10 +79,10 @@ From a functionality perspective, BCG and the earlier torch-compile-based piecew The compilation overhead was also visible in day-to-day development. In our CI setup at the time, compilation was often repeated across test runs, making CUDA Graph tests noticeably slower. Better caching could mitigate this, but removing the compiler from the capture path also removed this extra source of complexity from the development loop. -**Faster prefill.** Segmented capture also pays off at replay, not only at startup. On gpt-oss-120b (TP4, 4×GB300) where every backend runs, prefill is 1.62× faster than eager with BCG and 1.85× with full capture, while the compiler-based backend reaches 1.39× — BCG is 17% faster than TC piecewise at replay. On GLM-5.2 the comparison is shorter, because only BCG can capture it at all: 1.60× over eager, while TC piecewise cannot trace the forward and full capture has no path for its sparse attention. Both models are flat across a 32× range in prompt length, which is the signature of launch overhead rather than compute. +**Faster prefill.** Segmented capture also pays off at replay, not only at startup. On gpt-oss-120b (TP4, 4×GB300) where every backend runs, prefill is 1.62× faster than eager with BCG and 1.85× with full capture, while the compiler-based backend reaches 1.39× — BCG is 17% faster than TC piecewise at replay. On GLM-5.2 only BCG can capture at all — TC piecewise cannot trace the forward and full capture has no path for its sparse attention — and it is 1.60× over eager there. Every curve is flat across a 32× range in prompt length, which is the signature of launch overhead rather than compute. -

    Prefill-only latency: a fixed input length with one output token, one request at a time. Measured with decode graphs disabled in every arm, so the only difference is how prefill is captured.

    +

    Prefill-only latency on gpt-oss-120b, where all four backends run. Decode graphs are disabled in every arm, so the only difference is how prefill is captured.

    **Broader compatibility.** SGLang relies heavily on custom CUDA, Triton, and JIT-compiled kernels that are not native PyTorch operators. To make these kernels visible to `torch.compile`, we often had to wrap them through `torch.library` and provide fake implementations for tracing. This introduced compiler-specific scaffolding throughout the kernel stack. diff --git a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg index 074cb1214..7eca5bbf1 100644 --- a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg +++ b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg @@ -1,107 +1,68 @@ - - + Prefill latency by capture strategy - median TTFT · one request at a time · fixed input length, one output token · lower is better - - - no graph - - - tc_piecewise - - - breakable - - - full - gpt-oss-120b · TP4 - every backend works here - - 0 - - 40 - - 80 - - 120 - - 160 ms - - - - - - - - no graph - - - - - - - - tc_piecewise - - - - - - - - breakable - - - - - - - - full - 64 - 128 - 256 - 512 - 1024 - 2048 - input length (tokens) - GLM-5.2-FP8 · TP4 - only BCG can capture this model - - 0 - - 150 - - 300 - - 450 - - 600 ms - - - - - - - - no graph - - - - - - - - breakable - 64 - 128 - 256 - 512 - 1024 - 2048 - input length (tokens) - tc_piecewise — Torch Dynamo cannot trace this forward - full — sparse attention has no captured-metadata path - Both curves are flat in prompt length: at these sizes prefill is dominated by launch overhead, and capture removes a roughly fixed cost. + gpt-oss-120b · TP4 · 4×GB300 · median TTFT, one request at a time, fixed input length and one output token + + + no graph + + + tc_piecewise + + + breakable + + + full + + 0 + + 40 + + 80 + + 120 + + 160 ms + + + + + + + + no graph + + + + + + + + tc_piecewise 1.39× + + + + + + + + breakable 1.62× + + + + + + + + full 1.85× + 64 + 128 + 256 + 512 + 1024 + 2048 + input length (tokens) + Every curve is flat across a 32× range in prompt length: at these sizes prefill is dominated by launch overhead, so capture removes a roughly fixed cost. From 01bea742c9ffe1699c4f50bf368299e39fea3297 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:40:56 -0700 Subject: [PATCH 11/17] blog: drop the footnote line from the prefill latency figure Co-Authored-By: Claude Opus 5 (1M context) --- public/images/blog/breakable_cuda_graph/prefill-ttft.svg | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg index 7eca5bbf1..8f52cdd59 100644 --- a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg +++ b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg @@ -1,6 +1,6 @@ - - + Prefill latency by capture strategy gpt-oss-120b · TP4 · 4×GB300 · median TTFT, one request at a time, fixed input length and one output token @@ -64,5 +64,4 @@ 1024 2048 input length (tokens) - Every curve is flat across a 32× range in prompt length: at these sizes prefill is dominated by launch overhead, so capture removes a roughly fixed cost. From e03f6ce2b07751ab5e07213d6db49769524104a6 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:45:52 -0700 Subject: [PATCH 12/17] blog: drop the end-of-line labels from the prefill figure The legend already names each series. Co-Authored-By: Claude Opus 5 (1M context) --- public/images/blog/breakable_cuda_graph/prefill-ttft.svg | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg index 8f52cdd59..b3af43db9 100644 --- a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg +++ b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg @@ -1,6 +1,6 @@ - - + Prefill latency by capture strategy gpt-oss-120b · TP4 · 4×GB300 · median TTFT, one request at a time, fixed input length and one output token @@ -32,7 +32,6 @@ - no graph @@ -40,7 +39,6 @@ - tc_piecewise 1.39× @@ -48,7 +46,6 @@ - breakable 1.62× @@ -56,7 +53,6 @@ - full 1.85× 64 128 256 From 2666d6377c2ca116a60c24b568853051b2b6c143 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:50:16 -0700 Subject: [PATCH 13/17] blog: summarise the results in the TL;DR Build time, compile share, prefill latency, model coverage, implementation size, and memory footprint. Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index b163b20dd..1df31339c 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -12,6 +12,18 @@ CUDA Graphs promise to remove kernel-launch overhead, but getting close to that In SGLang, we refactored CUDA Graph support around a common runner/backend interface, making different capture strategies reusable across execution paths. For the more complex prefill path, the SGLang community introduced Breakable CUDA Graph and pioneered full CUDA Graph support on the FA4 and FlashInfer attention backends, both of which were first developed by the SGLang community as open-source serving techniques. We also dive deeper into CUDA Graph memory management, including memory reuse across shapes and graph segments, which is becoming an increasingly important part of SGLang’s overall memory management. +| | TC piecewise | Breakable CUDA Graph | +| --- | --- | --- | +| Prefill graph build, Qwen3-235B | 106.6 s | **27.7 s** | +| Prefill graph build, GLM-5.2 | 183.1 s | **35.2 s** | +| Share of build spent compiling | 78–86% | **none** | +| Prefill latency vs eager, gpt-oss-120b | 1.39× | **1.62×** (full capture 1.85×) | +| Models it can capture | fails on GLM-5.2, Qwen3-235B, Qwen3-Next | **all of them** | +| Implementation size | 783 LoC | **521 LoC** | + +Memory stays modest: 42 captured shapes across a 78-layer MoE add 2.4 GB of graph memory, and capturing through the chunked-prefill size lands 0.5–1.1 GB *below* the no-graph baseline, because the activation peak it replaces is larger than the graphs themselves. + + ## Background An inference step is not a single kernel but a sequence of many GPU operations. In modern LLM serving engines, repeatedly launching these operations from the CPU can introduce noticeable overhead, especially for latency-sensitive workloads. CUDA Graph reduces this overhead by recording the GPU work once and replaying it with much lower launch overhead. From 582c0244faf6a6892447de7b0af3b3316b58dbac Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Sun, 16 Aug 2026 23:52:40 -0700 Subject: [PATCH 14/17] blog: reconcile the TL;DR build times with the coverage row The two TC piecewise build times were measured with a local workaround for a lazy import that Torch Dynamo refuses to trace; without it neither model compiles at all. Mark them and say so. Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-15-advanced-cuda-graph.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-15-advanced-cuda-graph.md index 1df31339c..3ab2f3f4c 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-15-advanced-cuda-graph.md @@ -14,13 +14,15 @@ In SGLang, we refactored CUDA Graph support around a common runner/backend inter | | TC piecewise | Breakable CUDA Graph | | --- | --- | --- | -| Prefill graph build, Qwen3-235B | 106.6 s | **27.7 s** | -| Prefill graph build, GLM-5.2 | 183.1 s | **35.2 s** | +| Prefill graph build, Qwen3-235B | 106.6 s\* | **27.7 s** | +| Prefill graph build, GLM-5.2 | 183.1 s\* | **35.2 s** | | Share of build spent compiling | 78–86% | **none** | | Prefill latency vs eager, gpt-oss-120b | 1.39× | **1.62×** (full capture 1.85×) | -| Models it can capture | fails on GLM-5.2, Qwen3-235B, Qwen3-Next | **all of them** | +| Models it can capture | needs a local patch to trace Qwen3-235B or GLM-5.2; fails on Qwen3-Next | **all of them** | | Implementation size | 783 LoC | **521 LoC** | +\* TC piecewise cannot trace either model as-is — a lazy import inside the traced region makes Torch Dynamo bail — so those two build times are measured with a local workaround. + Memory stays modest: 42 captured shapes across a 78-layer MoE add 2.4 GB of graph memory, and capturing through the chunked-prefill size lands 0.5–1.1 GB *below* the no-graph baseline, because the activation peak it replaces is larger than the graphs themselves. From 2e22d5717d8740d176a56c9f44758949c7b57271 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Mon, 17 Aug 2026 00:56:04 -0700 Subject: [PATCH 15/17] blog: date to Aug 17, TL;DR summary paragraph, prefill benchmark section - date -> August 17, 2026, post renamed to match - TL;DR: drop the comparison table, add a short paragraph with the code-size, build-time and prefill-latency numbers - move the prefill latency comparison out of the BCG benefits and into "Full CUDA Graph for Prefill" as its own subsection - no-graph curve re-measured at n=4 and plotted as medians; the earlier dip at 256/512 was noise, so the speedups become 1.70x (BCG) and 1.93x (full) Co-Authored-By: Claude Opus 5 (1M context) --- ...h.md => 2026-08-17-advanced-cuda-graph.md} | 28 ++++++------------- .../breakable_cuda_graph/prefill-ttft.svg | 14 +++++----- 2 files changed, 16 insertions(+), 26 deletions(-) rename blog/{2026-08-15-advanced-cuda-graph.md => 2026-08-17-advanced-cuda-graph.md} (93%) diff --git a/blog/2026-08-15-advanced-cuda-graph.md b/blog/2026-08-17-advanced-cuda-graph.md similarity index 93% rename from blog/2026-08-15-advanced-cuda-graph.md rename to blog/2026-08-17-advanced-cuda-graph.md index 3ab2f3f4c..5d01bb03c 100644 --- a/blog/2026-08-15-advanced-cuda-graph.md +++ b/blog/2026-08-17-advanced-cuda-graph.md @@ -1,7 +1,7 @@ --- title: "Advanced CUDA Graph Techniques in Inference" author: "SGLang Team" -date: "August 15, 2026" +date: "August 17, 2026" previewImg: /images/blog/breakable_cuda_graph/bcg-design.svg type: blog --- @@ -12,19 +12,7 @@ CUDA Graphs promise to remove kernel-launch overhead, but getting close to that In SGLang, we refactored CUDA Graph support around a common runner/backend interface, making different capture strategies reusable across execution paths. For the more complex prefill path, the SGLang community introduced Breakable CUDA Graph and pioneered full CUDA Graph support on the FA4 and FlashInfer attention backends, both of which were first developed by the SGLang community as open-source serving techniques. We also dive deeper into CUDA Graph memory management, including memory reuse across shapes and graph segments, which is becoming an increasingly important part of SGLang’s overall memory management. -| | TC piecewise | Breakable CUDA Graph | -| --- | --- | --- | -| Prefill graph build, Qwen3-235B | 106.6 s\* | **27.7 s** | -| Prefill graph build, GLM-5.2 | 183.1 s\* | **35.2 s** | -| Share of build spent compiling | 78–86% | **none** | -| Prefill latency vs eager, gpt-oss-120b | 1.39× | **1.62×** (full capture 1.85×) | -| Models it can capture | needs a local patch to trace Qwen3-235B or GLM-5.2; fails on Qwen3-Next | **all of them** | -| Implementation size | 783 LoC | **521 LoC** | - -\* TC piecewise cannot trace either model as-is — a lazy import inside the traced region makes Torch Dynamo bail — so those two build times are measured with a local workaround. - -Memory stays modest: 42 captured shapes across a 78-layer MoE add 2.4 GB of graph memory, and capturing through the chunked-prefill size lands 0.5–1.1 GB *below* the no-graph baseline, because the activation peak it replaces is larger than the graphs themselves. - +For prefill, Breakable CUDA Graph is now SGLang's default. It reaches the same segmented execution as the `torch.compile`-based piecewise backend in roughly a quarter of the code (521 versus 1,771 lines), builds prefill graphs 3.8–5.2× faster because no compilation is involved, and has broader coverage for complex functionality naturally. Full CUDA Graph for prefill goes further, using request padding to capture the whole forward even for dynamic prefill workloads. Measured on prefill alone, BCG is 1.70× faster than eager execution and full capture reaches 1.93×. ## Background @@ -93,11 +81,6 @@ From a functionality perspective, BCG and the earlier torch-compile-based piecew The compilation overhead was also visible in day-to-day development. In our CI setup at the time, compilation was often repeated across test runs, making CUDA Graph tests noticeably slower. Better caching could mitigate this, but removing the compiler from the capture path also removed this extra source of complexity from the development loop. -**Faster prefill.** Segmented capture also pays off at replay, not only at startup. On gpt-oss-120b (TP4, 4×GB300) where every backend runs, prefill is 1.62× faster than eager with BCG and 1.85× with full capture, while the compiler-based backend reaches 1.39× — BCG is 17% faster than TC piecewise at replay. On GLM-5.2 only BCG can capture at all — TC piecewise cannot trace the forward and full capture has no path for its sparse attention — and it is 1.60× over eager there. Every curve is flat across a 32× range in prompt length, which is the signature of launch overhead rather than compute. - - -

    Prefill-only latency on gpt-oss-120b, where all four backends run. Decode graphs are disabled in every arm, so the only difference is how prefill is captured.

    - **Broader compatibility.** SGLang relies heavily on custom CUDA, Triton, and JIT-compiled kernels that are not native PyTorch operators. To make these kernels visible to `torch.compile`, we often had to wrap them through `torch.library` and provide fake implementations for tracing. This introduced compiler-specific scaffolding throughout the kernel stack. More importantly, the compiler also constrained **where graph boundaries could be placed**. Inputs and outputs crossing a registered operator boundary had to be representable by the compiler. When the natural boundary involved more specialized runtime state or return types, we sometimes had to search for a different cutting point or enlarge the eager region simply to expose an interface the compiler could handle. As the serving stack grew, the compiler boundary increasingly influenced the structure of code that was otherwise unrelated to compilation. @@ -156,6 +139,13 @@ This asymmetry is important: token padding is the expensive dimension, while req Full prefill capture is still an experimental feature. It has to be enabled explicitly — the engine warns that `full` is experimental and points to breakable or tc_piecewise for production workloads — and it currently works mainly on the FlashAttention (fa4) and FlashInfer backends, which are the ones that build extend-mode metadata the way the captured path needs. Broadening backend support and tuning the bucket and slot choices is still ahead of us. +### Prefill benchmark + +With three ways to capture prefill and an eager baseline, the remaining question is what each one costs at replay. Measuring prefill on its own — a fixed input length with a single output token, one request at a time, decode graphs disabled in every arm — on gpt-oss-120b (TP4, 4×GB300), where all four paths run: full capture is 1.93× faster than eager, BCG 1.70×, and TC piecewise 1.45×, so BCG is also 17% faster than the compiler-based backend at replay, not only at build time. On GLM-5.2 only BCG can capture at all — TC piecewise cannot trace the forward and full capture has no path for its sparse attention — and it is 1.60× over eager there. Every curve is flat across a 32× range in prompt length, which is the signature of launch overhead rather than compute. + + +

    Prefill-only latency on gpt-oss-120b, where all four backends run. Decode graphs are disabled in every arm, so the only difference is how prefill is captured.

    + ## Memory Footprint of CUDA Graphs Memory poses two separate challenges: keeping a segmented capture from multiplying resident memory, and capturing far enough that resident graph memory actually replaces the worst eager activation peak. diff --git a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg index b3af43db9..e60cb313e 100644 --- a/public/images/blog/breakable_cuda_graph/prefill-ttft.svg +++ b/public/images/blog/breakable_cuda_graph/prefill-ttft.svg @@ -25,13 +25,13 @@ 120 160 ms - - - - - - - + + + + + + + From e19270588c715f5fb8acee7a8f9623d4d6faff19 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Mon, 17 Aug 2026 01:23:24 -0700 Subject: [PATCH 16/17] blog: bracket the PR references, tighten two passages - wrap every PR reference as [#NNNNN] so the number reads as a citation - shorten the "Faster startup" paragraph - drop the decode-graphs sentence from the figure caption - parenthesise the equal-contribution note Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-17-advanced-cuda-graph.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/blog/2026-08-17-advanced-cuda-graph.md b/blog/2026-08-17-advanced-cuda-graph.md index 5d01bb03c..4fa89e996 100644 --- a/blog/2026-08-17-advanced-cuda-graph.md +++ b/blog/2026-08-17-advanced-cuda-graph.md @@ -33,7 +33,7 @@ This post walks through how CUDA Graph support is built in SGLang and what we ch Before this refactor, CUDA Graph support had grown around individual execution paths. Decode, prefill, and speculative decoding each had their own CUDA Graph runners, with overlapping logic for capture shapes, static buffers, replay, and graph configuration. As more execution modes and capture strategies were added, this duplication made it harder to reuse infrastructure and made CUDA Graph-related server arguments increasingly ambiguous. -The refactor [#23906](https://github.com/sgl-project/sglang/pull/23906) separates these responsibilities into two layers. A **runner** manages the execution-specific state needed for capture and replay: captured shapes, static input buffers, attention metadata, and the padding of live batches into captured shapes. A **backend** determines how that execution is captured, whether as one full graph, a sequence of breakable segments, or compiler-generated pieces. +The refactor [[#23906](https://github.com/sgl-project/sglang/pull/23906)] separates these responsibilities into two layers. A **runner** manages the execution-specific state needed for capture and replay: captured shapes, static input buffers, attention metadata, and the padding of live batches into captured shapes. A **backend** determines how that execution is captured, whether as one full graph, a sequence of breakable segments, or compiler-generated pieces. Because runners depend only on a common backend interface, each execution path can choose its capture strategy independently. Prefill and decode have separate runners, and speculative decoding adds more: the EAGLE draft, draft-extend and frozen-KV MTP draft steps each get their own runner built on the decode runner, while target verify is the decode runner itself, capturing more than one token per request. @@ -59,7 +59,7 @@ The third backend reaches similar segmentation through a compiler. `torch.compil CUDA Graph traditionally requires the captured region to be fully graph-compatible. In practice, modern inference workloads contain operations that cannot be captured directly. Prefill attention is a common example: some attention backends depend on runtime metadata and host-side preparation. A single incompatible operation can therefore prevent CUDA Graph from covering a much larger part of the forward. -We introduced **Breakable CUDA Graph (BCG)** to make capture more flexible. The mechanism and the `@eager_on_graph` decorator landed first as part of CUDA Graph debug mode in [#19102](https://github.com/sgl-project/sglang/pull/19102), and were then built into a breakable piecewise backend for prefill in [#22218](https://github.com/sgl-project/sglang/pull/22218). Instead of requiring the entire forward to be graph-compatible, BCG allows selected operations to run eagerly while capturing the graph-compatible regions around them. At a high level, the forward becomes a sequence of CUDA Graph segments connected by explicit eager breaks. +We introduced **Breakable CUDA Graph (BCG)** to make capture more flexible. The mechanism and the `@eager_on_graph` decorator landed first as part of CUDA Graph debug mode in [[#19102](https://github.com/sgl-project/sglang/pull/19102)], and were then built into a breakable piecewise backend for prefill in [[#22218](https://github.com/sgl-project/sglang/pull/22218)]. Instead of requiring the entire forward to be graph-compatible, BCG allows selected operations to run eagerly while capturing the graph-compatible regions around them. At a high level, the forward becomes a sequence of CUDA Graph segments connected by explicit eager breaks. ### Design and Mechanism @@ -73,7 +73,7 @@ From a functionality perspective, BCG and the earlier torch-compile-based piecew ### Benefits -**Faster startup.** For compiler-based piecewise graphs, compilation — not CUDA Graph capture — became the dominant setup cost. Measured separately, `torch.compile` accounts for 78–86% of the time spent preparing prefill graphs. The cost also grows with model complexity: compilation alone takes about 90 seconds on a 235B MoE and 158 seconds on GLM-5.2. BCG removes that phase entirely and reaches segmented execution in a single capture pass. +**Faster startup.** For compiler-based piecewise graphs, compilation — not capture — dominates setup: `torch.compile` accounts for 78–86% of the time spent preparing prefill graphs, and it grows with model complexity, reaching 90 seconds on a 235B MoE and 158 seconds on GLM-5.2. BCG removes that phase entirely, reaching segmented execution in a single capture pass. @@ -89,11 +89,11 @@ BCG removes this constraint at eager breaks: the graph system does not need to u **Debuggable by construction.** A captured CUDA Graph replays as an opaque unit: ordinary Python does not execute inside it, which makes prints, assertions, and step-by-step inspection difficult. BCG naturally leaves eager regions where normal Python still runs on every replay. -SGLang extends this idea with `--debug-cuda-graph` [#19102](https://github.com/sgl-project/sglang/pull/19102), which effectively wraps the whole forward in an eager break. The model then executes eagerly while still going through the CUDA Graph runner, static buffers, replay path, and metadata preparation. This provides a useful debugging boundary: if the problem remains, it is likely in the model or runner path; if it disappears, capture itself becomes the primary suspect. +SGLang extends this idea with `--debug-cuda-graph` [[#19102](https://github.com/sgl-project/sglang/pull/19102)], which effectively wraps the whole forward in an eager break. The model then executes eagerly while still going through the CUDA Graph runner, static buffers, replay path, and metadata preparation. This provides a useful debugging boundary: if the problem remains, it is likely in the model or runner path; if it disappears, capture itself becomes the primary suspect. ### BCG in Diffusion -BCG has also been adopted by SGLang’s diffusion stack [#27436](https://github.com/sgl-project/sglang/pull/27436). Diffusion repeatedly executes the same DiT forward during denoising, making CUDA Graph especially useful when those forwards contain many small, launch-bound kernels. +BCG has also been adopted by SGLang’s diffusion stack [[#27436](https://github.com/sgl-project/sglang/pull/27436)]. Diffusion repeatedly executes the same DiT forward during denoising, making CUDA Graph especially useful when those forwards contain many small, launch-bound kernels.
    • Capture the real serving shapes. Resolution, video frame count, prompt-conditioning length, CFG mode, and the selected transformer can all affect the capture signature. We warm up the shapes that are actually served and fall back to eager execution for unseen signatures.
      • @@ -113,7 +113,7 @@ The broader lesson is that BCG removes launch overhead; it does not reduce model Full CUDA Graph is straightforward for decode because each request contributes one token: the main varying dimension is batch size. Prefill is harder because a batch varies in two dimensions at once — the total number of tokens and the number of requests those tokens belong to — while a captured graph requires both to remain fixed. Together with attention backends that depend on runtime metadata, this made full CUDA Graph difficult to apply to prefill and was one of the main reasons we adopted Breakable CUDA Graph there. -More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph [#27988](https://github.com/sgl-project/sglang/pull/27988), including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. This is an exciting experimental feature that is still under active development: backend coverage is limited today, and we are continuing to improve compatibility, capture policies, and performance. +More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph [[#27988](https://github.com/sgl-project/sglang/pull/27988)], including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. This is an exciting experimental feature that is still under active development: backend coverage is limited today, and we are continuing to improve compatibility, capture policies, and performance. ### Making prefill static @@ -144,7 +144,7 @@ Full prefill capture is still an experimental feature. It has to be enabled expl With three ways to capture prefill and an eager baseline, the remaining question is what each one costs at replay. Measuring prefill on its own — a fixed input length with a single output token, one request at a time, decode graphs disabled in every arm — on gpt-oss-120b (TP4, 4×GB300), where all four paths run: full capture is 1.93× faster than eager, BCG 1.70×, and TC piecewise 1.45×, so BCG is also 17% faster than the compiler-based backend at replay, not only at build time. On GLM-5.2 only BCG can capture at all — TC piecewise cannot trace the forward and full capture has no path for its sparse attention — and it is 1.60× over eager there. Every curve is flat across a 32× range in prompt length, which is the signature of launch overhead rather than compute. -

      Prefill-only latency on gpt-oss-120b, where all four backends run. Decode graphs are disabled in every arm, so the only difference is how prefill is captured.

      +

      Prefill-only latency on gpt-oss-120b, where all four backends run.

      ## Memory Footprint of CUDA Graphs @@ -156,7 +156,7 @@ A segmented backend could easily multiply graph memory: every captured shape con
      • One shared memory pool across segments. Every segment for a captured shape uses the same CUDA Graph pool, allowing intermediate storage to be reused rather than pinned separately for each segment.
        • -
      • Weak references at eager breaks. Tensors passed into a break are held weakly when the graph pool already owns their storage, avoiding unnecessary Python references that would extend tensor lifetimes. The tensor weak-reference technique comes from vLLM #9724, which introduced it so that captured graphs could share output buffers instead of each pinning its own.
        • +
      • Weak references at eager breaks. Tensors passed into a break are held weakly when the graph pool already owns their storage, avoiding unnecessary Python references that would extend tensor lifetimes. The tensor weak-reference technique comes from vLLM [#9724], which introduced it so that captured graphs could share output buffers instead of each pinning its own.
      • One output buffer across capture sizes. Capture sizes share a single maximum-sized output buffer, sliced to the rows needed by each shape, instead of allocating one output buffer per shape.
      @@ -195,4 +195,4 @@ Meta: Shiyang Chen*, Lianmin Zheng We also thank the NVIDIA, AMD, Thinking Machines Lab, and Meta PyTorch teams for their help along the way. -\* Equal contribution. +(\* Equal contribution) From 1f7fdcc6fd5c722cb1f268fc397658ae3fcbb404 Mon Sep 17 00:00:00 2001 From: Yuwei An Date: Mon, 17 Aug 2026 01:42:09 -0700 Subject: [PATCH 17/17] blog: fix an antecedent in the TL;DR, drop a duplicate disclaimer "both of which" most naturally attached to the FA4 and FlashInfer backends, which read as claiming SGLang developed them. Split the sentence so "both techniques" clearly means BCG and full CUDA Graph. Also remove the experimental caveat from the prefill section opening; the same point is made properly at the end of that section. Co-Authored-By: Claude Opus 5 (1M context) --- blog/2026-08-17-advanced-cuda-graph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/blog/2026-08-17-advanced-cuda-graph.md b/blog/2026-08-17-advanced-cuda-graph.md index 4fa89e996..97ee018f0 100644 --- a/blog/2026-08-17-advanced-cuda-graph.md +++ b/blog/2026-08-17-advanced-cuda-graph.md @@ -10,7 +10,7 @@ type: blog CUDA Graphs promise to remove kernel-launch overhead, but getting close to that benefit in a real inference engine requires graphing as much of the workload as possible without sacrificing compatibility, startup time, or memory. -In SGLang, we refactored CUDA Graph support around a common runner/backend interface, making different capture strategies reusable across execution paths. For the more complex prefill path, the SGLang community introduced Breakable CUDA Graph and pioneered full CUDA Graph support on the FA4 and FlashInfer attention backends, both of which were first developed by the SGLang community as open-source serving techniques. We also dive deeper into CUDA Graph memory management, including memory reuse across shapes and graph segments, which is becoming an increasingly important part of SGLang’s overall memory management. +In SGLang, we refactored CUDA Graph support around a common runner/backend interface, making different capture strategies reusable across execution paths. For the more complex prefill path, the SGLang community introduced Breakable CUDA Graph and pioneered full CUDA Graph support with the FA4 and FlashInfer attention backends. Both techniques were first developed in SGLang as open-source serving techniques. We also dive deeper into CUDA Graph memory management, including memory reuse across shapes and graph segments, which is becoming an increasingly important part of SGLang’s overall memory management. For prefill, Breakable CUDA Graph is now SGLang's default. It reaches the same segmented execution as the `torch.compile`-based piecewise backend in roughly a quarter of the code (521 versus 1,771 lines), builds prefill graphs 3.8–5.2× faster because no compilation is involved, and has broader coverage for complex functionality naturally. Full CUDA Graph for prefill goes further, using request padding to capture the whole forward even for dynamic prefill workloads. Measured on prefill alone, BCG is 1.70× faster than eager execution and full capture reaches 1.93×. @@ -113,7 +113,7 @@ The broader lesson is that BCG removes launch overhead; it does not reduce model Full CUDA Graph is straightforward for decode because each request contributes one token: the main varying dimension is batch size. Prefill is harder because a batch varies in two dimensions at once — the total number of tokens and the number of requests those tokens belong to — while a captured graph requires both to remain fixed. Together with attention backends that depend on runtime metadata, this made full CUDA Graph difficult to apply to prefill and was one of the main reasons we adopted Breakable CUDA Graph there. -More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph [[#27988](https://github.com/sgl-project/sglang/pull/27988)], including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. This is an exciting experimental feature that is still under active development: backend coverage is limited today, and we are continuing to improve compatibility, capture policies, and performance. +More recently, we found ways to make prefill execution sufficiently static for full CUDA Graph [[#27988](https://github.com/sgl-project/sglang/pull/27988)], including restructuring how request slots and attention metadata are represented so that supported attention backends no longer have to remain outside the graph. ### Making prefill static