[None][feat] Two-level GVR decode top-K dispatch; remove the CUDA heuristic and temporal-only prior state by longcheng-nv · Pull Request #18446 · NVIDIA/TensorRT-LLM

longcheng-nv · 2026-08-31T09:26:55Z

Dev Engineer Review

Added configuration-based two-level GVR dispatch.
Added use_self_sampling_topk and use_gvr_emission across LLM, model, DSA, indexer, and warmup paths.
Removed CUDA heuristic GVR Top-K code and obsolete APIs.
Limited prior-state allocation and updates to temporal GVR.
Replaced retired GVR environment variables with configuration fields.
Updated telemetry manifests and configuration round-trip handling.
Updated radix auxiliary-buffer validation and Top-K operator schemas.
No test-list changes were identified.
Verdict: sufficient based on the reported unit, GPU, dispatch, configuration, and end-to-end validation.

QA Engineer Review

Modified test files:

tests/unittest/_torch/attention/sparse/dsa/test_dsa_indexer.py
- Modified test_metadata_cache_geometry_comes_from_sparse_metadata_params.
- Added coverage for self-sampling and temporal GVR dispatch, prior-state handling, emission propagation, retired environment variables, metadata consistency, shared-Top-K lifecycle, and defaults.
tests/unittest/_torch/modules/test_top_k.py
- Updated GVR dispatch and prior-state tests.
- Updated kernel-call expectations for the removed pre_idx and heuristic_scratch arguments.
tests/unittest/_torch/thop/parallel/test_indexer_topk.py
- Removed obsolete heuristic, distribution, MTP, DSv4, and overflow-policy tests.
- Retained radix, insertion, radix split-work, auxiliary-buffer, CUDA Graph, and validation coverage.
- Added GVR regression coverage for hostile hints and large tie plateaus.

No corresponding tests/integration/test_lists/ changes were identified. Test-list coverage in test-db/ and qa/ is therefore not demonstrated by this change set.

Verdict: needs follow-up to confirm that the modified and added unit tests are registered in the required CI or manual QA test lists.

Description

Rebased onto main after #18410 (PR-1), #18625 (self-sampling +inf fix) and #18646 (device-side prior seeding) merged. This PR folds the former PR-2 (two-level dispatch) and PR-3 (#18447, CUDA heuristic removal) into one to save CI: five logical commits, one pipeline. Design prerequisite #18339 (DSL v1 race fix) is merged.

Resulting dispatch, all from the sparse-attention config (no env vars):

enable_heuristic_topk=false (default)         -> exact insertion/radix top-K
enable_heuristic_topk=true
  use_self_sampling_topk=true (default)       -> GVR V2, self-sampling (emission n/a)
  use_self_sampling_topk=false                -> GVR V1, temporal previous-step hint
    use_gvr_emission=true (FP4 + paged-MQA)   -> V1 + emission block-skip

Commit 1 — two-level dispatch from the sparse-attention config:

enable_heuristic_topk (existing, default False) — first level: the GVR family vs the exact insertion/radix path. The default stays radix.
use_self_sampling_topk (new, default True) — second level: the hint-free self-sampling engine vs the temporal previous-step-hint engine. Only meaningful when the first level is on.
The CUTE_DSL_GVR_V2 enum folds into CUTE_DSL_GVR behind a gvr_self_sampling module flag; TopK.needs_gvr_prior follows the two-level decision. The field threads llm_args -> model_config (both V4 and V3.2 rebuild blocks) -> DSAParams/DSAMetadataParams -> indexer selection and the warmup mirror. TRTLLM_GVR_SELF_SAMPLING is retired (warns, ignored). Drive-by: the self-sampling warmup's top_k source moves off a dead getattr(..., "index_topk") to sparse_mla_topk.

Commit 2 — CUDA GVR heuristic removal (nothing selects it under the two-level dispatch):

heuristicTopKDecode.{cu,h} + heuristic_topk.cuh deleted (2,352 lines; the [None][fix] GVR indexer top-K: repair the non-converged threshold search #17550 exactness-fix code retires with them); indexerTopK.cu drops canUseHeuristic/SchemeX/TRTLLM_HEURISTIC_NMIN/BSMAX; insertion/radix/split-work tiers untouched.
thop indexer_topk_decode: pre_idx/heuristic_scratch drop out of the schema and register_fake; top_k.py loses the CUDA_GVR enum/branch/workspace; heuristic-only test arms retire with the kernel.

Commit 3 — prior state only when the temporal engine runs:

Shared predicate use_self_sampling_gvr() (dsa/params.py) keeps the indexer's per-layer TopK construction and the metadata's allocation decision in agreement (live indexers exist only on cr in {1,4} layers — the V4 backend builds indexers on cr=4 layers only — so the metadata's representative ratio matches every live indexer).
needs_gvr_prior gates the per-layer gvr_prior_indices arena, the LJF row-reorder buffer, prefill seeding, the aux-stream write-back and the indexer-side prior slice; the default self-sampling engine allocates and copies nothing.
With the CUDA heuristic gone the temporal engine requires the CuTe DSL on SM100/103; enable_heuristic_topk without it falls back to exact radix with a one-time warning.

Commit 4 — emission block-skip as a third config flag, use_gvr_emission (default False):

Promotes the emission-assisted block-skip ([None][perf] Emission-assisted GVR top-K decode for the DeepSeek V4 indexer #16953) from the TRTLLM_GVR_EMISSION env var to a sparse-attention config field, threaded llm_args -> model_config (V4 and V3.2 rebuild blocks) -> DSAParams/DSAMetadataParams -> the indexer gate. It is an option on the temporal leaf, not a third dispatch level.
The indexer enables it only when use_gvr_emission and not use_self_sampling_topk, the layer selects CUTE_DSL_GVR, and the FP4 paged-MQA-logits path is in use. The self-sampling engine takes its bracket from the current row and never consumes emission, so the flag is inert there and on non-FP4 layers.

Commit 5 — telemetry golden manifest: tensorrt_llm/usage/llm_args_golden_manifest.json regenerated for the two new fields (use_self_sampling_topk, use_gvr_emission; both captured by value).

Usage — selecting the GVR engine

All flags live on the DSA sparse-attention config
(DeepSeekSparseAttentionConfig, algorithm="dsa" for V3.2;
DeepSeekV4SparseAttentionConfig, algorithm="deepseek_v4" for V4).
enable_heuristic_topk turns the GVR family on; use_self_sampling_topk picks
the generation. enable_heuristic_topk=False (the default) keeps the exact
insertion/radix path.

Python (LLM API):

from tensorrt_llm import LLM
from tensorrt_llm.llmapi import DeepSeekSparseAttentionConfig  # V4: DeepSeekV4SparseAttentionConfig

# GVR V2 — hint-free self-sampling (the default generation once the family is on)
llm = LLM(
    model=...,  # HF id or local checkpoint path
    sparse_attention_config=DeepSeekSparseAttentionConfig(
        enable_heuristic_topk=True,      # turn on the GVR family
        use_self_sampling_topk=True,     # V2 (default; may be omitted)
    ),
)

# GVR V1 — temporal previous-step hint
llm = LLM(
    model=...,
    sparse_attention_config=DeepSeekSparseAttentionConfig(
        enable_heuristic_topk=True,
        use_self_sampling_topk=False,    # V1
        # use_gvr_emission=True,         # optional: FP4 emission block-skip (V1 only)
    ),
)

trtllm-serve (--extra_llm_api_options YAML):

# GVR V2 (default self-sampling)
sparse_attention_config:
  algorithm: dsa              # deepseek_v4 for V4
  enable_heuristic_topk: true
  use_self_sampling_topk: true

# GVR V1 (temporal hint) + FP4 emission block-skip
sparse_attention_config:
  algorithm: dsa              # deepseek_v4 for V4
  enable_heuristic_topk: true
  use_self_sampling_topk: false
  use_gvr_emission: true

Validation

B200, machine-local full build at the pre-restack head (37aa6be3, commits 1-3):

C++ compiles clean after the removal (build_wheel.py; GLOB-based kernels CMake needs no edits)
test_indexer_topk.py (thop): 1392 passed — radix / DSL radix / prefill / overflow-policy arms against the shrunken schema
test_dsa_indexer.py: 116 passed, 4 skipped (full suite; prior-allocation mirror + two-level dispatch expectations)
test_top_k.py: 13 passed; test_gvr_selfsampling_topk.py: 77 passed
Config round-trip: DSA and V4 configs carry use_self_sampling_topk through to_sparse_params/to_sparse_metadata_params
Baseline sanity: the pre-removal thop suite (5037 tests incl. heuristic arms) passes on the same host at PR-1's head

Commit 4 (config-only Python) on this head: use_gvr_emission round-trip — default False on both param dataclasses; every (use_self_sampling_topk, use_gvr_emission) combination threads through, and the indexer gate turns it off on the self-sampling path. Two new test_dsa_indexer.py tests cover this (test_use_gvr_emission_threads_to_params, test_use_gvr_emission_defaults_off); no test depends on the retired env var. The GPU suites on this exact head run in CI.

UT runtime / 30-minute guard

B200 (SM100), one GPU, current head 66ac38f; each suite ran in a fresh process/TMPDIR with a hard timeout 30m. The current Python/CuTe sources use the already validated pre-restack 37aa6be3 C++ artifacts for the unchanged indexer thop ABI.

Suite	Result	Pytest time	End-to-end wall time
`test_indexer_topk.py`	1392 passed	18m 10.11s	18m 24.72s
`test_gvr_selfsampling_topk.py` (3 fresh-process runs)	83 passed/run	48.18-66.16s	53.16-71.88s (median 53.93s)
`test_top_k.py`	14 passed	0.18s	41.35s (import/init dominated)
`test_dsa_indexer.py`, PR-relevant subset	125 passed, 4 skipped	65.01s	1m 51.57s

The two GPU kernel suites take at most 19m 36.60s when run serially in separate fresh processes, leaving more than 10 minutes of a 30-minute guard. The DSA subset excludes 16 unchanged grouped-remap tests that require a newer main-branch convert_req_index_to_global_grouped binary than this local pre-restack build.

CI estimate: the corresponding full unittest/_torch/thop/parallel task on stacked-base #18625 completed in 568.62s (9m 28.62s). This PR reduces test_indexer_topk.py from the pre-removal 5037 cases to 1392, while the comparable standalone GVR run is within 0.8% of the base, so 10-20 minutes is a conservative expected range for this CI task; exceeding 30 minutes is unlikely. PR #18446 pipeline 71180 did not yield a usable UT duration because unrelated pod-launch/JDK posix_spawn failures triggered fail-fast before the B200 shard completed, not because a GVR UT timed out.

e2e — three-param config dispatch (operator level)

DeepSeek-V4-Flash, MTP0, op47 REF (SWE-bench-64K, OSL1024), TEP4 on one B200 node (GPU0 was tenant-occupied, so the 4 clean GPUs; relative A/B holds, absolute TPOT is TEP4 not production TEP8). Ran on a GVR-code-identical head. 17 runs, 0 arm-proof violations, 1 unrelated MPI-init flake (rep6 v1, dropped from its pair).

Arm	Config (sparse-attention, no env)	Dispatch (log-proof)
`ori`	`enable_heuristic_topk=false`	exact radix, GVR not engaged
`v1`	`+ use_self_sampling_topk=false`	temporal, self-sampling not engaged
`v2`	`+ use_self_sampling_topk=true` (default)	self-sampling engaged

Pair (decode per-user TPOT geomean)	gm	n	Gate `gm <= 1 + max(0.5%, 2*SE)`
v2 / ori	0.8803	6	PASS — ~12% faster than radix
v2 / v1	0.9955	5	PASS — on par with temporal
v1 / ori	0.9040	5	—

Status / pending before undraft

[None][feat] GVR V2 decode top-k goes hint-free by default: the bracket comes from the current row #18410 (PR-1) merged; restacked onto [None][fix] Self-sampling top-k drops +inf from the top-k (register family) #18625 (73af6653)
[None][fix] Self-sampling top-k drops +inf from the top-k (register family) #18625 lands; rebase to drop its commits
CI build + full pipeline
e2e trio on PR-1 head (ori/v1/hf arms, 3x8xB200): pass, 104 runs / 0 failures — [None][feat] GVR V2 decode top-k goes hint-free by default: the bracket comes from the current row #18410 (comment)

PR Checklist

PR title and description convey what and why
Test coverage for the change
CI passing

longcheng-nv · 2026-09-03T05:19:16Z

/bot run

tensorrt-cicd · 2026-09-03T05:25:07Z

PR_Github #71180 [ run ] triggered by Bot. Commit: 66ac38f Link to invocation

tensorrt-cicd · 2026-09-03T06:49:29Z

PR_Github #71180 [ run ] completed with state FAILURE. Commit: 66ac38f
/LLM/main/L0_MergeRequest_PR pipeline #58316 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-09-03T06:57:28Z

/bot run --reuse-test --disable-fail-fast

tensorrt-cicd · 2026-09-03T07:05:42Z

PR_Github #71202 [ run ] triggered by Bot. Commit: 66ac38f Link to invocation

tensorrt-cicd · 2026-09-03T09:38:56Z

PR_Github #71202 [ run ] completed with state FAILURE. Commit: 66ac38f
/LLM/main/L0_MergeRequest_PR pipeline #58337 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-09-03T19:30:11Z

/bot run --reuse-test

…ntion config enable_heuristic_topk keeps selecting the GVR family over the exact radix path; a new use_self_sampling_topk config field (default True) selects the hint-free self-sampling engine over the temporal-hint engines. The CUTE_DSL_GVR_V2 enum folds into CUTE_DSL_GVR behind a gvr_self_sampling module flag, TopK.needs_gvr_prior follows the two-level decision, and the retired TRTLLM_GVR_SELF_SAMPLING env only warns. The field threads llm_args -> model_config -> DSAParams/DSAMetadataParams -> indexer and the warmup mirror (whose top_k source also moves off a dead index_topk getattr to sparse_mla_topk). Made-with: Claude Code (Fable 5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

Nothing selects the CUDA heuristic once the unified DSL GVR router is in: delete heuristicTopKDecode.{cu,h} / heuristic_topk.cuh, the canUseHeuristic dispatch and the GVR SchemeX bounds in indexerTopK.cu (radix keeps a cached SM-count helper), shrink the indexer_topk_decode thop schema and its register_fake (pre_idx / heuristic_scratch gone), drop the CUDA_GVR enum plus module branch, and retire the heuristic-only distribution / hostile-hint / tie-plateau test arms. The radix insertion / histogram / split-work tiers are untouched. C++ changes are not compiled yet: build + CI plus the 886x11 CUDA-v1 vs DSL-v1 paired A/B sign-off gate this draft. Made-with: Claude Code (Fable 5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

The self-sampling engine keeps no cross-step state, so the framework no longer allocates it any: the per-layer gvr_prior_indices arena, the LJF row-reorder buffer, prefill seeding, the aux-stream write-back, and the indexer-side prior slice all key on needs_gvr_prior = two-level dispatch selecting the temporal engine. A shared use_self_sampling_gvr() predicate in dsa/params.py keeps the indexer's per-layer TopK construction and the metadata's allocation decision in agreement (live indexers only exist on cr in {1, 4} layers, matching the metadata's representative ratio). With the CUDA heuristic gone, the temporal engine requires the CuTe DSL on SM100/103; enable_heuristic_topk without it falls back to exact radix with a one-time warning. Made-with: Claude Code (Fable 5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

… param Promote the emission-assisted block-skip optimization from the TRTLLM_GVR_EMISSION env var to a `use_gvr_emission` sparse-attention config field (default False). It only takes effect on the temporal-hint (V1) GVR path with FP4 paged-MQA logits; the self-sampling (V2) engine derives its bracket from the current row and never uses emission. Threads the field through llm_args -> model_config (V4 + V3.2 rebuilds) -> DSAParams / DSAMetadataParams -> the indexer gate, and adds config-threading unit tests. Made-with: Claude Code (Fable 5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

…e GVR dispatch fields Adds the two new sparse-attention config fields (use_self_sampling_topk, use_gvr_emission; both bool, captured by value) via scripts/generate_llm_args_golden_manifest.py so test_build_capture_manifest_matches_committed_golden passes again. Made-with: Claude Code (Fable 5.1) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

tensorrt-cicd · 2026-09-03T19:36:45Z

PR_Github #71322 [ run ] triggered by Bot. Commit: 3a8324a Link to invocation

longcheng-nv · 2026-09-03T19:43:36Z

/bot run --reuse-test

tensorrt-cicd · 2026-09-03T19:50:53Z

PR_Github #71324 [ run ] triggered by Bot. Commit: 59d051a Link to invocation

tensorrt-cicd · 2026-09-03T19:56:13Z

PR_Github #71322 [ run ] completed with state ABORTED. Commit: 3a8324a

Link to invocation

tensorrt-cicd · 2026-09-03T22:03:51Z

PR_Github #71324 [ run ] completed with state SUCCESS. Commit: 59d051a
/LLM/main/L0_MergeRequest_PR pipeline #58449 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-09-04T00:49:03Z

CI triage for the latest run (PR_Github #71324 / L0 #58449, head 59d051a): 67 failures, none from this PR.

60 are unittest/_torch/visual_gen/test_trtllm_serve_endpoints.py on CPU-Generic-x86-1 and CPU-Generic-arm-1 (image-generation endpoints returning HTTP 500). These are identical to main's own post-merge run L0_PostMerge doc enhancement for disagg #2947 (Sep 3 12:26Z; status testing #2946 at 00:26Z had none), i.e. a main regression from the VisualGen serve changes merged that morning; main has since waived the file in [None][infra] Waive 1 failed cases for main in pre-merge 58439 #18685 (nvbugs/6720250). The same lanes passed on the previous base with this PR's content.
7 are Test terminated unexpectedly on DGX_H100-1/5/6: fail-fast collateral of the above.

Next: rebase onto main with the waive and re-run with --reuse-test.

Picks up the main-side waive of unittest/_torch/visual_gen/test_trtllm_serve_endpoints.py (NVIDIA#18685) that failed the CPU lanes in the previous CI round. Made-with: Claude Code (Fable 5.1) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

longcheng-nv · 2026-09-04T00:55:34Z

/bot run --reuse-test

coderabbitai · 2026-09-06T09:25:18Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 63741287-05db-450f-9e66-6f6e6ddfe573

📥 Commits

Reviewing files that changed from the base of the PR and between 4656c4b and b3a4fb2.

📒 Files selected for processing (18)

cpp/tensorrt_llm/kernels/IndexerTopK.h
cpp/tensorrt_llm/kernels/heuristicTopKDecode.cu
cpp/tensorrt_llm/kernels/heuristicTopKDecode.h
cpp/tensorrt_llm/kernels/heuristic_topk.cuh
cpp/tensorrt_llm/kernels/indexerTopK.cu
cpp/tensorrt_llm/thop/IndexerTopKOp.cpp
tensorrt_llm/_torch/attention/backends/sparse/dsa/indexer.py
tensorrt_llm/_torch/attention/backends/sparse/dsa/metadata.py
tensorrt_llm/_torch/attention/backends/sparse/dsa/params.py
tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py
tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py
tensorrt_llm/_torch/model_config.py
tensorrt_llm/_torch/modules/top_k.py
tensorrt_llm/llmapi/llm_args.py
tensorrt_llm/usage/llm_args_golden_manifest.json
tests/unittest/_torch/attention/sparse/dsa/test_dsa_indexer.py
tests/unittest/_torch/modules/test_top_k.py
tests/unittest/_torch/thop/parallel/test_indexer_topk.py

💤 Files with no reviewable changes (4)

cpp/tensorrt_llm/kernels/heuristicTopKDecode.h
tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py
cpp/tensorrt_llm/kernels/heuristicTopKDecode.cu
cpp/tensorrt_llm/kernels/heuristic_topk.cuh

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The change removes legacy heuristic Top-K CUDA paths and their public inputs. It adds configuration-driven self-sampling and temporal GVR dispatch, updates sparse-attention metadata and propagation, and expands tests for dispatch, prior state, API calls, and exact Top-K results.

Changes

Indexer Top-K CUDA dispatch

Layer / File(s)	Summary
Remove heuristic CUDA dispatch `cpp/tensorrt_llm/kernels/IndexerTopK.h`, `cpp/tensorrt_llm/kernels/indexerTopK.cu`	Decode APIs no longer accept pre-index or heuristic scratch buffers. Heuristic dispatch and its capability helper were removed. Dispatch now uses insertion, radix, and radix split-work paths.
Update the Torch Top-K boundary `cpp/tensorrt_llm/thop/IndexerTopKOp.cpp`, `tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py`	The Torch operator schema and fake registration remove `pre_idx` and `heuristic_scratch`. FP32 uses caller-owned radix auxiliary buffers.
Add GVR mode configuration `tensorrt_llm/_torch/modules/top_k.py`, `tensorrt_llm/_torch/attention/backends/sparse/dsa/params.py`, `tensorrt_llm/llmapi/llm_args.py`, `tensorrt_llm/_torch/model_config.py`, `tensorrt_llm/usage/llm_args_golden_manifest.json`, `tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py`	Top-K distinguishes self-sampling from temporal CUTE DSL GVR. Sparse-attention configuration propagates `use_self_sampling_topk` and `use_gvr_emission`.
Align DSA metadata and prior state `tensorrt_llm/_torch/attention/backends/sparse/dsa/indexer.py`, `tensorrt_llm/_torch/attention/backends/sparse/dsa/metadata.py`	DSA resolves GVR mode from configuration and prerequisites. Prior-index allocation and row reordering apply only when temporal GVR requires prior state.
Validate dispatch and API changes `tests/unittest/_torch/attention/sparse/dsa/test_dsa_indexer.py`, `tests/unittest/_torch/modules/test_top_k.py`, `tests/unittest/_torch/thop/parallel/test_indexer_topk.py`	Tests cover revised calls, GVR mode selection, prior handling, configuration propagation, CUDA Graph buffers, and exact results across supported dtypes.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to b3a4f

The configuration, dispatch, prior-state, and operator API changes are aligned and validated, with no actionable merge-blocking risk identified.

Sequence Diagram(s)

sequenceDiagram
  participant Config
  participant DSAIndexer
  participant TopK
  participant CUDAIndexer
  Config->>DSAIndexer: provide self-sampling and emission settings
  DSAIndexer->>TopK: select GVR mode and prior requirements
  TopK->>CUDAIndexer: invoke insertion/radix Top-K when fallback applies
  TopK->>DSAIndexer: return selected indices and update temporal state

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.29% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 12 files. (2 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the primary changes: two-level GVR decode Top-K dispatch and removal of the CUDA heuristic and temporal-only prior state.
Description check	✅ Passed	The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the design, configuration changes, removed implementation, validation results, and CI status in…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 37.29% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 12 files. (2 skipped: 1 unsupported, 1 too large.)

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

longcheng-nv · 2026-09-06T09:36:15Z

CI is green (L0 Pipeline SUCCESS) and this is rebased on latest main — ready for code-owner review. Requesting approvals from the owning areas:

@NVIDIA/trt-llm-torch-attention-devs — DSA indexer + C++ IndexerTopK (two-level dispatch, prior-state scoping)
@NVIDIA/trt-llm-topk-devs — modules/top_k.py + tests
@NVIDIA/trt-llm-kernels-devs — CUDA heuristic top-K kernel removal
@NVIDIA/trt-llm-runtime-devs — custom_ops / model_config / llm_args config threading
@NVIDIA/trt-llm-usage-telemetry-devs / @NVIDIA/trt-llm-oss-compliance / @NVIDIA/trt-llm-noncommitted-api-review-committee — llm_args_golden_manifest.json (two new config fields; any one of the three)

Thanks!

longcheng-nv · 2026-09-07T03:35:53Z

/bot run --reuse-test --extra-stage "DGX_B200-8_GPUs-PyTorch-1"

tensorrt-cicd · 2026-09-07T03:41:33Z

PR_Github #71845 [ run ] triggered by Bot. Commit: b3a4fb2 Link to invocation

tensorrt-cicd · 2026-09-07T05:58:52Z

PR_Github #71845 [ run ] completed with state SUCCESS. Commit: b3a4fb2
/LLM/main/L0_MergeRequest_PR pipeline #58923 completed with status: 'SUCCESS'

CI Report

Link to invocation

github-actions Bot assigned longcheng-nv Aug 31, 2026

longcheng-nv mentioned this pull request Aug 31, 2026

[None][refactor] Remove the CUDA GVR heuristic top-K decode and skip prior state for the self-sampling engine #18447

Closed

6 tasks

longcheng-nv force-pushed the feat/gvr-unified-router branch from 5616970 to 67d7135 Compare September 1, 2026 15:50

longcheng-nv changed the title ~~[None][feat] Unified GVR decode top-K: single CUTE_DSL_GVR engine with capture-stable shape routing~~ [None][feat] Two-level GVR decode top-K dispatch from the sparse-attention config Sep 1, 2026

longcheng-nv changed the title ~~[None][feat] Two-level GVR decode top-K dispatch from the sparse-attention config~~ [None][feat] Two-level GVR decode top-K dispatch; remove the CUDA heuristic and temporal-only prior state Sep 2, 2026

longcheng-nv mentioned this pull request Sep 2, 2026

[None][feat] GVR V2 decode top-k goes hint-free by default: the bracket comes from the current row #18410

Merged

longcheng-nv force-pushed the feat/gvr-unified-router branch from 37aa6be to 66ac38f Compare September 3, 2026 05:18

longcheng-nv mentioned this pull request Sep 3, 2026

[None][perf] Seed the GVR prior from device-side prefill lengths #18646

Merged

1 task

longcheng-nv and others added 5 commits September 3, 2026 19:35

longcheng-nv force-pushed the feat/gvr-unified-router branch from 3a8324a to 59d051a Compare September 3, 2026 19:43

longcheng-nv added the api-compatible Accepted LLM API contract change that is backwards-compatible label Sep 4, 2026

longcheng-nv marked this pull request as ready for review September 6, 2026 09:15

longcheng-nv requested review from a team as code owners September 6, 2026 09:15

longcheng-nv requested review from Mgluhovskoi, QiJune and zhaoyangwang-nvidia September 6, 2026 09:15

longcheng-nv requested review from Kefeng-Duan, Linda-Stadter, QiJune, dcampora, juney-nvidia and lfr-0531 and removed request for Mgluhovskoi, QiJune and zhaoyangwang-nvidia September 6, 2026 09:25

juney-nvidia added the ci: full pre-merge approved label Sep 7, 2026

zongfeijing approved these changes Sep 7, 2026

View reviewed changes

lfr-0531 approved these changes Sep 7, 2026

View reviewed changes

longcheng-nv requested a review from lori-ren September 7, 2026 08:20

yuxianq approved these changes Sep 7, 2026

View reviewed changes

Uh oh!

Conversation

longcheng-nv commented Aug 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Usage — selecting the GVR engine

Validation

UT runtime / 30-minute guard

e2e — three-param config dispatch (operator level)

Status / pending before undraft

PR Checklist

Uh oh!

longcheng-nv commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

longcheng-nv commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

longcheng-nv commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

longcheng-nv commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

tensorrt-cicd commented Sep 3, 2026

Uh oh!

longcheng-nv commented Sep 4, 2026

Uh oh!

longcheng-nv commented Sep 4, 2026

Uh oh!

coderabbitai Bot commented Sep 6, 2026

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

longcheng-nv commented Sep 6, 2026

Uh oh!

longcheng-nv commented Sep 7, 2026

Uh oh!

tensorrt-cicd commented Sep 7, 2026

Uh oh!

tensorrt-cicd commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

longcheng-nv commented Aug 31, 2026 •

edited by coderabbitai Bot

Loading