Conversation
The skill reported that speculative decoding crosses over — a win below C=4 and
a 3.1x loss at C=32 — and recommended shipping two serving profiles. Those were
real measurements taken on a GPU that was power throttled (HW Slowdown asserted,
module limit capped at 950 W of a 1600 W default, drawing 190-240 W where a
healthy peer drew 600 W).
Re-measured on healthy hardware across ISL 256/8K/32K/128K at C=1/8/16:
speculation wins every single case by 29-145%. There is no crossover up to
C=16. Its only cost is KV pool — freeing the draft cache gave +23% (6.0M ->
7.4M tokens), which does not pay for the throughput.
On a starved part the k+1 verify batch is extra work the GPU cannot absorb; with
headroom it is nearly free and accepted tokens are pure gain. The plausible
mechanism ("verify batch exceeds the tuned kernel range") made a hardware
artifact look like a software law.
Adds the generalized lesson to methodology.md: a hardware fault invalidates
conclusions drawn before it was found, so re-validate everything measured on
that machine, not just the experiment in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The skill shipped a "speculative decoding has a concurrency crossover" finding —
a win below C=4, a 3.1x loss at C=32 — and recommended shipping two serving
profiles. That was measured on a faulty GPU and is wrong.
What happened
The original numbers came from a DGX Station whose GB300 was power throttled:
HW Slowdownasserted, module power limit capped at 950 W against a 1600 Wdefault, drawing 190-240 W where a healthy peer drew 600 W. The skill's own
hardware-validation.mddocuments that machine — but the crossover had alreadybeen measured on it and written up before the fault was found.
Re-measured on healthy hardware
Single GB300, k=8,
max_num_seqs=32, OSL=256, aggregate decode tok/s:No crossover up to C=16. Speculation wins every case by 29-145%. Its only
cost is KV pool: freeing the draft cache gives +23% (6.0M -> 7.4M tokens), which
does not come close to paying for the throughput.
On a starved part the
k+1verify batch is extra work the GPU cannot absorb;with headroom it is nearly free and the accepted tokens are pure gain.
Changes
reference/vllm-tuning.md— replaces the crossover section with the healthyhardware table, and keeps the old result as a cautionary tale rather than
deleting it, since the failure mode is the instructive part.
SKILL.md— the Phase 3 tuning bullet no longer asserts a crossover.reference/methodology.md— adds the generalized lesson: a hardware faultinvalidates conclusions drawn before you found it, so re-validate everything
measured on that machine, not just the experiment in flight.
The plausible mechanism ("verify batch exceeds the tuned kernel range") is what
made a hardware artifact look like a software law. That shape warning is real,
but on healthy hardware it was never large enough to flip the decision.
Validation
./test.shpasses (including the frontmatter check added with the skill).