Adds skip-softmax calibration through the vLLM serving path#1992
Draft
kaix-nv wants to merge 8 commits into
Draft
Adds skip-softmax calibration through the vLLM serving path#1992kaix-nv wants to merge 8 commits into
kaix-nv wants to merge 8 commits into
Conversation
- Route active skip-softmax launches (prefill and paged decode) through the fixed 128x128 calibration tile instead of the autotuner, so realized sparsity matches the granularity thresholds were calibrated at (autotuned BLOCK_N=32 tiles skip differently than the 128x128 measurement/calibration blocks). BLOCK_M steps down on shared-memory pressure (fp32 inputs on ~100KB-smem GPUs) with a warning; BLOCK_N -- the calibrated KV skip granularity -- is never reduced. - flash_skip_softmax: exclude padded query rows from the block keep decision; fully-padded rows voted keep (block_diff == 0) and forced the last partial block row dense, under-counting sparsity. - vLLM runtime: validate the calibrated-decode CUDA-graph guard for sparse-only installs, not just quantized ones. Signed-off-by: Kai Xu <kaix@nvidia.com>
- attention_calibrate: read K/V through a paged cache (vLLM NHD layout) via block_table, reusing the shared paged tile loaders. Contiguous-KV behavior is unchanged; paged and contiguous launches produce identical counters and output. - vLLM adapters: calibration mode (enable_calibration / disable_calibration / iter_sparse_impls). When active, forward routes to the paged calibration kernel -- full dense attention plus multi-threshold tile-skip counting -- so generation is numerically unchanged while stats accumulate. Each scheduled request is measured independently (decode vs prefill phase per request), and raw per-threshold tile counts are recorded so tensor-parallel ranks can be aggregated by summing counts before the fit. - FlashInfer adapter writes the current K/V to the cache before the calibrate kernel reads it (releases that update the cache inside forward); fp16/bf16-only and NHD-only, both validated explicitly. Signed-off-by: Kai Xu <kaix@nvidia.com>
…mple - install_vllm_skip_softmax_calibration: validation-before-mutation install of calibration adapters on every attention layer -- requires eager execution, fp16/bf16 model and KV dtypes, no active attention Q/K/P/V fakequant, and a FlashAttention/FlashInfer backend; disables cascade. Measurement starts separately (enable_calibration RPC), so warmup launches are never recorded. - DynamicThresholdCalibrator.calibrate_from_stats: backend-agnostic fit stage extracted from calibrate(), preserving result fields (fit_logspace, log_a) and reporting per-sample sparsity; the HF path delegates to it unchanged. - plugins/sparse_attn_calibration (vLLM-free): raw tile-count merging across layers and tensor-parallel ranks (counts are additive; ratios are formed only after the global merge, one fit per phase -- never per rank, never by averaging fitted coefficients), plus a canonical sparse_attention_config builder matching export_sparse_attention_config so the serving loader round-trips it; existing N:M groups are preserved. - Thin example: SkipSoftmaxCalibWorker (load hook + enable/status/counts RPCs) and calibrate_sparse_attn.py driver (CLI, prompts, rank-count aggregation, fit, checkpoint-config writing). Signed-off-by: Kai Xu <kaix@nvidia.com>
Tests: - Paged calibrate kernel: paged == contiguous (counters bit-equal, output match) across aligned and non-128-aligned lengths with shuffled block tables; decode-shaped measurement covers the full cache; high-block-ID pointer regression (int64 paged offsets, memory-gated). - Calibration/serving tile contract: an active skip-softmax launch skips exactly the tiles the calibration kernel counted at the same threshold (same 128x128 geometry, same prefix-max criterion); skip composes with P/V QDQ (sm89+). - Adapter forward: mixed prefill/decode batches record per-request phases and raw counts while output stays dense vs an SDPA reference; NHD and fp16/bf16 cache validation; layer count summing; FlashInfer cache write ordered before the calibrate-kernel read. - Installer: install-without-measure lifecycle, eager/quantizer/fp8-KV/ no-layer rejections with validation-before-mutation, and the sparse-only CUDA-graph guard for calibrated decode configs. - vLLM-free helpers: count merging/alignment, TP-rank aggregation, synthetic exponential fit recovery, calibrate_from_stats field contract, canonical config round trip through the serving loader with N:M group preservation. Docs: vLLM calibration workflow section in examples/vllm_serve/README.md. Signed-off-by: Kai Xu <kaix@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1992 +/- ##
==========================================
- Coverage 77.79% 77.64% -0.15%
==========================================
Files 518 519 +1
Lines 58204 58437 +233
==========================================
+ Hits 45279 45375 +96
- Misses 12925 13062 +137
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Kai Xu <kaix@nvidia.com>
Build calibration prompts with RulerDatasetBuilder — the same dataset the PyTorch (HF) calibration path uses — so vLLM- and HF-calibrated thresholds are fit on identical data (defaults mirror the HF path: 24 samples, max_seqlen 32768; NIAH essay haystack via download_ruler_data.sh and --calib_data_dir). The built-in demo prompt fallback is removed; --prompts_file remains as an explicit override for custom calibration data. Signed-off-by: Kai Xu <kaix@nvidia.com>
…a-flow hardening - Reject skip-softmax combined with ANY attention quantization at plan time: quantized Q/K/P change the score distribution the thresholds were calibrated on (N:M sparse softmax still composes with quantization). - The 128x128 skip tile is now a hard contract: configurations that cannot compile it (fp32 on ~100KB-smem GPUs) are rejected with a clear error instead of re-tiled (the BLOCK_M step-down changed realized sparsity). - FlashInfer layout: query vLLM layout metadata and reject HND at install and before the calibration cache write; the NHD shape check remains as a fallback (it is ambiguous when page_size equals the per-rank KV heads). - One canonical threshold sweep shared by the HF and vLLM calibration paths (DEFAULT_THRESHOLD_TRIALS hoisted from DynamicThresholdCalibrator); ignore_eos forces the full decode length; the driver exits nonzero unless every requested phase produced a valid fit (no silent partial export). - Count merging treats alignment as a contract: mismatched sample counts, lengths, threshold widths, or per-rank/per-layer phase coverage raise instead of silently truncating. - Preserve legacy top-level sparse_softmax metadata through recalibration. - Docs: no-sparsification wording (kernel numerics differ from the native backend); clarify prefix-caching support (sparse-only serving supports suffix attention; quantized installs and calibration reject it). - High-block-ID regression test halves its allocation (V aliases K storage). Signed-off-by: Kai Xu <kaix@nvidia.com>
- Skip-softmax + attention quantization is now unreachable from every direction: sparse-only installs reject calibrated skip onto layers with active attention quantizers (not just quantized installs adding skip), and the raw kernel API itself rejects P/V QDQ with an active skip threshold -- which makes the P-QDQ 16-row measurement tile dead code, so the fixed skip tile is unconditionally 128x128. - FlashInfer layout helper preserves a genuine None from get_kv_cache_layout (str(None) became the truthy string None and both guards hard-rejected valid configurations instead of using the shape fallback). - Counter vectors are validated against len(threshold_trials) in both calibrate_from_stats and fit_from_counts: consistently short vectors previously zipped silently and could misattribute sparsities. - Driver decode semantics match vLLM: the first output token comes from the prefill forward, so --decode_tokens now means decode-attention steps and generation runs decode_tokens + 1 output tokens. - Calibrate kernel mirrors the serving kernel IEEE fp32 QK dot so raw fp32 calibration and serving round near-threshold scores identically. - target_sparsity validated to [0, 1] (same range as the HF config). Signed-off-by: Kai Xu <kaix@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change: ?
new feature
Adds skip-softmax threshold calibration through the vLLM serving path, so thresholds can be calibrated on the same inference engine they will be served with.
Usage
Testing
Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: ✅ / ❌ / N/AAdditional Information