blog: Advanced CUDA Graph Techniques in Inference - #399
Merged
Conversation
Adds the CUDA Graph post covering the runner/backend refactor, Breakable CUDA Graph, full CUDA Graph for prefill, and CUDA Graph memory footprint. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both panels are now measured at the same chunked-prefill size (8192), so they differ only by model. Updates the accompanying numbers, and corrects the claim that the activation peak disappears entirely: on GLM it falls to 0.35 GB because the sparse-attention indexer still runs eagerly at a break. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chart was extracted from an inline <svg> block, where the SVG namespace is implicit. As a standalone .svg file the browser cannot parse it without an explicit xmlns, so the image rendered broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Left over from when the chart was inline HTML; as a standalone file used via <img> it fought the page's own sizing and cropped the footnotes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a "Faster prefill" benefit with a two-panel figure: gpt-oss-120b, where every backend runs (full 1.85x, breakable 1.62x, tc_piecewise 1.39x over eager), and GLM-5.2, where only BCG can capture at all (1.60x). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Single panel, so the four curves have room and the line labels are not clipped. The GLM result stays in the prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The legend already names each series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Build time, compile share, prefill latency, model coverage, implementation size, and memory footprint. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two TC piecewise build times were measured with a local workaround for a lazy import that Torch Dynamo refuses to trace; without it neither model compiles at all. Mark them and say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- date -> August 17, 2026, post renamed to match - TL;DR: drop the comparison table, add a short paragraph with the code-size, build-time and prefill-latency numbers - move the prefill latency comparison out of the BCG benefits and into "Full CUDA Graph for Prefill" as its own subsection - no-graph curve re-measured at n=4 and plotted as medians; the earlier dip at 256/512 was noise, so the speedups become 1.70x (BCG) and 1.93x (full) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- wrap every PR reference as [#NNNNN] so the number reads as a citation - shorten the "Faster startup" paragraph - drop the decode-graphs sentence from the figure caption - parenthesise the equal-contribution note Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"both of which" most naturally attached to the FA4 and FlashInfer backends, which read as claiming SGLang developed them. Split the sentence so "both techniques" clearly means BCG and full CUDA Graph. Also remove the experimental caveat from the prefill section opening; the same point is made properly at the end of that section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ispobock
approved these changes
Aug 17, 2026
BBuf
reviewed
Aug 17, 2026
|
|
||
| Giving up on CUDA Graph whenever one such operation appears would leave much of the forward uncaptured. BCG instead lets developers mark the incompatible region directly with `@eager_on_graph`. During capture, the current graph segment is closed when execution reaches the marked function, the function runs eagerly, and capture resumes afterward in a new segment. | ||
|
|
||
| At replay time, the recorded graph segments and eager functions run in the same order. The tensor crossing an eager break is created by the preceding captured segment and registered as a persistent boundary buffer, so its device address remains fixed. The following captured segment is captured against that same address. During replay, the eager function therefore writes its newly computed result back into this boundary buffer rather than returning a newly allocated tensor, allowing the next segment to read the updated value from the address it was originally captured with. BCG never inspects or traces the operations inside the eager region: they only need to execute correctly. |
Contributor
There was a problem hiding this comment.
Is the boundary buffer actually the eager function’s captured output, rather than the output of the preceding graph segment? During replay, the new eager output seems to be copied into that retained buffer. Could you clarify this wording?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the CUDA Graph post covering the runner/backend refactor, Breakable CUDA Graph, full CUDA Graph for prefill, and CUDA Graph memory footprint.