[https://nvbugs/6458213][test] use TRTLLM MoE for Blackwell skip-softmax tests

[bobboli]

Description

Use the TRTLLM MoE backend for the Qwen3-30B-A3B skip-softmax accuracy tests on SM100 and SM103 Blackwell GPUs. Other architectures continue to use the AUTO backend, so Hopper remains on its supported MoE path.

The B200 4-GPU test intermittently produced NaN sampling inputs. CUDA error 710 and the later CUTLASS TMA descriptor failure are downstream effects of the device-side NaN assertion, so the log does not prove CUTLASS is the root cause. This change treats CUTLASS MoE as the leading hypothesis and switches the Blackwell tests to TRTLLM as a practical mitigation.

The PR removes both remaining Qwen3 skip-softmax waivers: the single-GPU B200 waiver associated with NVBug 6606551 and the H100 4-GPU waiver associated with duplicate NVBug 6523809. The DeepSeekR1 waiver previously included in this PR has already been removed from current main and is no longer part of the rebased diff.

Test Coverage

• python3 -m py_compile tests/integration/defs/accuracy/test_llm_api_pytorch.py
• pre-commit run --files tests/integration/defs/accuracy/test_llm_api_pytorch.py tests/integration/test_lists/waives.txt

Both passed. GPU validation is required in CI on the applicable Blackwell and H100/H200 stages.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

• Configured the TRTLLM MoE backend for Qwen3 skip-softmax accuracy tests on SM100 and SM103.
• Kept AUTO for other architectures.
• Removed two Qwen3 waiver entries, including the H100 waiver.
• No runtime code or public API changes were made.
• Test-list formatting and scope are consistent.
• No duplicate or unrelated waiver entries were introduced.

QA Engineer Review

• Modified tests/integration/defs/accuracy/test_llm_api_pytorch.py.
• Modified tests/integration/test_lists/waives.txt.
• Updated the single-GPU and four-GPU skip-softmax accuracy tests.
• Removed the general GPU and H100 Qwen3 waiver entries.
• No corresponding test-db/ or qa/ files were modified.
• Coverage mapping for the changed test functions is incomplete.
• Verdict: needs follow-up.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5044402d-f6e0-4cd6-b308-7705b9cc45a9

📥 Commits

Reviewing files that changed from the base of the PR and between a79c220 and ddcf249.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

Walkthrough

The change adds architecture-dependent MoeConfig values to the single-GPU and four-GPU skip-softmax attention accuracy tests. It also removes two Qwen3 30B A3B Instruct 2507 softmax-attention waivers.

Changes

Skip-softmax attention test configuration

Layer / File(s)	Summary
Configure architecture-dependent MoE backends tests/integration/defs/accuracy/test_llm_api_pytorch.py	The single-GPU and four-GPU tests use TRTLLM on SM100 and SM103, and AUTO on other architectures.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to ddcf2

This PR changes the backend used by Blackwell accuracy tests and removes two test waivers; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the NVBug, test-related change, TRTLLM MoE backend, and Blackwell skip-softmax tests.
Description check	✅ Passed	The description explains the issue, mitigation, affected architectures, removed waivers, and relevant validation. It also identifies that GPU validation remains required in CI.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[bobboli]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #65836 [ run ] triggered by Bot. Commit: 359cb30 Link to invocation

[tensorrt-cicd]

PR_Github #65836 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

[github-actions]

Removed the "ci: post-merge approved" label because @bobboli could not be verified as an active member of NVIDIA/trt-llm-ci-approvers. Ask a member of that team to apply it.

[bobboli]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #65842 [ run ] triggered by Bot. Commit: 359cb30 Link to invocation

[tensorrt-cicd]

PR_Github #65842 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

[xxi-nv]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #65880 [ run ] triggered by Bot. Commit: 359cb30 Link to invocation

[tensorrt-cicd]

PR_Github #65880 [ run ] completed with state SUCCESS. Commit: 359cb30
/LLM/main/L0_MergeRequest_PR pipeline #53573 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[bobboli]

/bot run --disable-fail-fast --post-merge --reuse-test

[tensorrt-cicd]

PR_Github #65969 [ run ] triggered by Bot. Commit: 359cb30 Link to invocation

[tensorrt-cicd]

PR_Github #65969 [ run ] completed with state FAILURE. Commit: 359cb30
/LLM/main/L0_MergeRequest_PR pipeline #53658 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[bobboli]

/bot run --disable-fail-fast --post-merge --reuse-test

[tensorrt-cicd]

PR_Github #66161 [ run ] triggered by Bot. Commit: 359cb30 Link to invocation

[tensorrt-cicd]

PR_Github #66161 [ run ] completed with state FAILURE. Commit: 359cb30
/LLM/main/L0_MergeRequest_PR pipeline #53835 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[bobboli]

/bot run --disable-fail-fast --post-merge --reuse-test

[tensorrt-cicd]

PR_Github #66219 [ run ] triggered by Bot. Commit: 8be625b Link to invocation

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[bobboli]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #69056 [ run ] triggered by Bot. Commit: fbe551f Link to invocation

[bobboli]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #69066 [ run ] triggered by Bot. Commit: a79c220 Link to invocation

[tensorrt-cicd]

PR_Github #69056 [ run ] completed with state ABORTED. Commit: fbe551f

Link to invocation

[bobboli]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #69070 [ run ] triggered by Bot. Commit: ddcf249 Link to invocation

[tensorrt-cicd]

PR_Github #69066 [ run ] completed with state ABORTED. Commit: a79c220

Link to invocation

[bobboli]

/bot run --disable-fail-fast --post-merge

[tensorrt-cicd]

PR_Github #69086 [ run ] triggered by Bot. Commit: ddcf249 Link to invocation

[tensorrt-cicd]

PR_Github #69070 [ run ] completed with state ABORTED. Commit: ddcf249

Link to invocation

[tensorrt-cicd]

PR_Github #69086 [ run ] completed with state SUCCESS. Commit: ddcf249
/LLM/main/L0_MergeRequest_PR pipeline #56457 completed with status: 'SUCCESS'

CI Report

Link to invocation