[https://nvbugs/6566734][test] Unwaive test_disaggregated_qwen3_32b_fp8 and accept both greedy completions

[moraxu]

Dev Engineer Review

• Updated qwen3_32b_fp8 output validation to accept both known-good greedy completion starts.
• Reused the existing any-of check pattern.
• Removed the matching waiver entry from tests/integration/test_lists/waives.txt.
• No public API, performance, or error-handling changes were identified.
• Offline validation and local B200 validation passed.

QA Engineer Review

• Modified test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8].
• Removed its waiver entry from tests/integration/test_lists/waives.txt.
• The test-list change uses the existing test path and has no duplicate entry.
• The test is no longer waived and is covered for CI execution.
• Verdict: sufficient.

Description

disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] failed in post-merge builds 2886 and 2887 on DGX_H200-8_GPUs-PyTorch-Post-Merge-1 and was auto-waived in #17340 (nvbugs/6566734).

Root cause: the test's output verification requires the exact substring "Asyncio in Python is a library" in the completion of the raw prompt "What are the benefits of using asyncio in Python?" (greedy, temperature=0, --ignore-eos, 100 tokens). That prompt sits on a near-tie between two greedy continuations:

• mode A (historical passes): " Asyncio in Python is a library..." — answers the question
• mode B: " I have read that it is used for asynchronous programming..." — continues the question; a coherent, on-topic raw completion

Both failing CI runs produced byte-identical mode-B text, and the other four prompts' completions were byte-identical to each other and healthy (no corruption; the "Berlin Berlin" canary absent). The 5 prompts are sent concurrently (asyncio.gather in disagg_client.py) into a 4x(ctx TP1) + 1x(gen TP4) cluster with Eagle3 speculation and FP8 KV cache, so batch composition — and therefore kernel-level numerics — is not run-invariant; a near-tied top-2 first token can legitimately flip.

The fix mirrors the existing precedent for deepseek_v3_lite (#6301, nvbugs/5390810): accept either known-good greedy completion via the check's existing any-of list mechanism, and remove the waive.

Test Coverage

• disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] — unwaived by this PR (post-merge l0_dgx_h200).
• Offline validation: the new check passes on the exact output.json contents captured from both failing CI runs (2886, 2887), while the old check reproduces the CI assertion on both.
• Local 8xB200 validation (clean dual-arch wheel at 682fa40, the failing run's CI image): 3 server boots; the asyncio completion took the same mode-B start each time, byte-identical across boots. Fixed test: 2/2 PASSED (5:26 cold cache, 2:07 warm). A control run with the unmodified check also passed on B200 — but only because there the completion happens to wander into a sentence containing the literal phrase "Asyncio in Python is a library" ~30 tokens deep; on H200 the tail went elsewhere within the 100-token budget and CI failed. That tail-luck dependence is exactly the fragility this PR removes: the check now anchors on the two observed completion starts.
• H200 validation on this PR: /bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" (as a multi-GPU stage on a PR this additionally requires the ci: full pre-merge approved label from an NVIDIA/trt-llm-ci-approvers member).

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f9b9b7aa-b183-4919-8040-56385338e9bd

📥 Commits

Reviewing files that changed from the base of the PR and between 75ca082 and 7956df6.

📒 Files selected for processing (2)

• tests/integration/defs/disaggregated/test_disaggregated.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/integration/defs/disaggregated/test_disaggregated.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The Qwen3 32B FP8 disaggregated integration test now accepts two valid completion responses and is removed from the skipped-test list.

Changes

Qwen3 FP8 test enablement

Layer / File(s)	Summary
Completion validation and test activation tests/integration/defs/disaggregated/test_disaggregated.py, tests/integration/test_lists/waives.txt	The completion check accepts two valid asynchronous-programming responses and still requires the Berlin response. The test waiver entry is removed.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 7956d

The Qwen3 32B FP8 disaggregated test is re-enabled and now accepts either observed valid greedy completion, avoiding failures caused by batch-dependent near-tied tokens. No concrete current-head merge risk remains.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description clearly explains the failure, root cause, solution, and relevant test coverage. It includes the required sections and completed checklist.
Title check	✅ Passed	The title accurately identifies the NVBugs issue, the test type, the waiver removal, and the acceptance of both valid greedy completions.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #69801 [ run ] triggered by Bot. Commit: 20bcbb9 Link to invocation

[tensorrt-cicd]

PR_Github #69801 [ run ] completed with state SUCCESS. Commit: 20bcbb9
/LLM/main/L0_MergeRequest_PR pipeline #57095 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #69982 [ run ] triggered by Bot. Commit: 20bcbb9 Link to invocation

[tensorrt-cicd]

PR_Github #69982 [ run ] completed with state ABORTED. Commit: 20bcbb9

Link to invocation

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #70424 [ run ] triggered by Bot. Commit: 20bcbb9 Link to invocation

[tensorrt-cicd]

PR_Github #70424 [ run ] completed with state SUCCESS. Commit: 20bcbb9
/LLM/main/L0_MergeRequest_PR pipeline #57653 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #70571 [ run ] triggered by Bot. Commit: 571a637 Link to invocation

[tensorrt-cicd]

PR_Github #70571 [ run ] completed with state SUCCESS. Commit: 571a637
/LLM/main/L0_MergeRequest_PR pipeline #57783 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #70712 [ run ] triggered by Bot. Commit: 571a637 Link to invocation

[tensorrt-cicd]

PR_Github #70712 [ run ] completed with state SUCCESS. Commit: 571a637
/LLM/main/L0_MergeRequest_PR pipeline #57904 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #71021 [ run ] triggered by Bot. Commit: 571a637 Link to invocation

[tensorrt-cicd]

PR_Github #71021 [ run ] completed with state SUCCESS. Commit: 571a637
/LLM/main/L0_MergeRequest_PR pipeline #58177 completed with status: 'UNSTABLE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #71177 [ run ] triggered by Bot. Commit: 7956df6 Link to invocation

[tensorrt-cicd]

PR_Github #71177 [ run ] completed with state SUCCESS. Commit: 7956df6
/LLM/main/L0_MergeRequest_PR pipeline #58314 completed with status: 'SUCCESS'

CI Report

Link to invocation