Skip to content

Correct the speculative-decoding crossover in inference-optimization - #20

Open
theFong wants to merge 1 commit into
mainfrom
fix-spec-decode-crossover
Open

theFong wants to merge 1 commit into
mainfrom
fix-spec-decode-crossover

Conversation

@theFong

@theFong theFong commented Aug 6, 2026

Copy link
Copy Markdown
Owner

The skill shipped a "speculative decoding has a concurrency crossover" finding —
a win below C=4, a 3.1x loss at C=32 — and recommended shipping two serving
profiles. That was measured on a faulty GPU and is wrong.

What happened

The original numbers came from a DGX Station whose GB300 was power throttled:
HW Slowdown asserted, module power limit capped at 950 W against a 1600 W
default, drawing 190-240 W where a healthy peer drew 600 W. The skill's own
hardware-validation.md documents that machine — but the crossover had already
been measured on it and written up before the fault was found.

Re-measured on healthy hardware

Single GB300, k=8, max_num_seqs=32, OSL=256, aggregate decode tok/s:

ISL C Spec ON Spec OFF Winner
256 1 278 158 ON +76%
256 8 1198 866 ON +38%
256 16 1875 1431 ON +31%
8K 8 591 409 ON +44%
8K 16 654 456 ON +43%
32K 8 225 92 ON +145%
32K 16 218 112 ON +95%
128K 8 46 35 ON +29%
128K 16 54 37 ON +45%

No crossover up to C=16. Speculation wins every case by 29-145%. Its only
cost is KV pool: freeing the draft cache gives +23% (6.0M -> 7.4M tokens), which
does not come close to paying for the throughput.

On a starved part the k+1 verify batch is extra work the GPU cannot absorb;
with headroom it is nearly free and the accepted tokens are pure gain.

Changes

  • reference/vllm-tuning.md — replaces the crossover section with the healthy
    hardware table, and keeps the old result as a cautionary tale rather than
    deleting it, since the failure mode is the instructive part.
  • SKILL.md — the Phase 3 tuning bullet no longer asserts a crossover.
  • reference/methodology.md — adds the generalized lesson: a hardware fault
    invalidates conclusions drawn before you found it, so re-validate everything
    measured on that machine, not just the experiment in flight.

The plausible mechanism ("verify batch exceeds the tuned kernel range") is what
made a hardware artifact look like a software law. That shape warning is real,
but on healthy hardware it was never large enough to flip the decision.

Validation

./test.sh passes (including the frontmatter check added with the skill).

The skill reported that speculative decoding crosses over — a win below C=4 and
a 3.1x loss at C=32 — and recommended shipping two serving profiles. Those were
real measurements taken on a GPU that was power throttled (HW Slowdown asserted,
module limit capped at 950 W of a 1600 W default, drawing 190-240 W where a
healthy peer drew 600 W).

Re-measured on healthy hardware across ISL 256/8K/32K/128K at C=1/8/16:
speculation wins every single case by 29-145%. There is no crossover up to
C=16. Its only cost is KV pool — freeing the draft cache gave +23% (6.0M ->
7.4M tokens), which does not pay for the throughput.

On a starved part the k+1 verify batch is extra work the GPU cannot absorb; with
headroom it is nearly free and accepted tokens are pure gain. The plausible
mechanism ("verify batch exceeds the tuned kernel range") made a hardware
artifact look like a software law.

Adds the generalized lesson to methodology.md: a hardware fault invalidates
conclusions drawn before it was found, so re-validate everything measured on
that machine, not just the experiment in flight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant