[None][test] Unwaive 17 recovered perf-sanity test_e2e cases

[chenfeiz0326]

Summary

Re-ran every currently-waived perf/test_perf_sanity.py::test_e2e[...]
case on real GPUs to check whether the originally-waived failures have
recovered. A case counts as recovered only if it runs to completion with
full request accounting (Total == Successful == num_prompts, Failed == 0,
non-null throughput) and no crash/OOM/hang/timeout.

17 of 27 waived cases now pass, so this PR removes only those 17 waive
lines from tests/integration/test_lists/waives.txt. The other 10 still
fail and stay waived.

• Verified at main commit 3253b64043b9d8ceb0a3c802c5e76eb7c51af58d
  (harness read from the commit under test).
• Hardware: GB300 cases on GB300, GB200 cases on GB200, B200 cases on DGX B200.
• Diff is waiver-only (waives.txt, 17 deletions, 0 insertions).

Unwaived cases (bracket id → parent NVBug)

GB300 (11)

case	NVBug
aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL	6581075
aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL	6517846
aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL	6517846
disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL	6572843
disagg_upload-e2e-gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL	6601537
disagg_upload-e2e-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL	6566777
disagg_upload-gen_only-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL	6581075
disagg_upload-gen_only-gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL	6601537
disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL	6581075
disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL	6581075
disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL	6581075

GB200 (2)

case	NVBug
aggr_upload-dynamo_gpt_oss_120b_fp4_blackwell-gpt_oss_fp4_tep4_adp_cutlass_8k1k	6374910
disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL	6581075

B200 / DGX B200 (4)

case	NVBug
full:DGX_B200/...aggr_upload-gemma4_26b_a4b_nvfp4_blackwell-gemma4_26b_a4b_nvfp4_tp1_1k1k	6571410
full:DGX_B200/...aggr_upload-host_perf_llama8b_spec_decode-llama8b_spec_bs1_128_128	6571408
aggr_upload-deepseek_r1_fp8_blackwell-r1_fp8_tp8_mtp3_8k1k	6432948
aggr_upload-super_ad_blackwell-super_ad_ws1_1k1k	6153575

Verification logs

Recovery of each unwaived case was confirmed from its run log below. All logs are now stored on the internal repair-bot host under /home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/<GPU>/. GB300/GB200 rows carry the trtllm-benchmark accounting log (Total == Successful, 0 failed); B200 rows carry the full CI-pytest slurm-<id>.out stream.

GB300 — aws_cmh (11)

case	log
aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c03-glm5-con1024-ctxonly.log
aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c04-glm5-con1-ctxonly.log
aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c05-glm5-con512-ctxonly.log
disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c06-r1-128k8k-con256-e2e.log
disagg_upload-e2e-gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c10-v4pro-con8-e2e.log
disagg_upload-e2e-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c11-glm5-con1024-e2e.log
disagg_upload-gen_only-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c12-r1-128k8k-con256-genonly.log
disagg_upload-gen_only-gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c16-v4pro-con8-genonly.log
disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c17-glm5-con1024-genonly.log
disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c18-glm5-con1-genonly.log
disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB300/c19-glm5-con512-genonly.log

GB200 — lyris (2)

case	log
aggr_upload-dynamo_gpt_oss_120b_fp4_blackwell-gpt_oss_fp4_tep4_adp_cutlass_8k1k	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB200/gb200-gptoss-aggr-8k1k-2734693.log
disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/GB200/gb200-gptoss-disagg-genonly-con1024-2735090.log

B200 / DGX B200 — computelab (4)

case	log
aggr_upload-gemma4_26b_a4b_nvfp4_blackwell-gemma4_26b_a4b_nvfp4_tp1_1k1k	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/B200/b200-gemma-1k1k-3749341.log
aggr_upload-host_perf_llama8b_spec_decode-llama8b_spec_bs1_128_128	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/B200/b200-llama8b-spec-3747851.log
aggr_upload-deepseek_r1_fp8_blackwell-r1_fp8_tp8_mtp3_8k1k	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/B200/b200-dsr1-fp8-8k1k-3750394.log
aggr_upload-super_ad_blackwell-super_ad_ws1_1k1k	/home/scratch.chenfeiz_gpu/repo/20260818-waive-triage/logs/B200/b200-superad-1k1k-3749445.log

Still waived (10, unchanged)

DeepSeek-V4-Pro-fp4 host-RAM OOM during weight load on the multi-ctx-server
topologies (con180/con666/con4301 → ctx3/ctx6/ctx12), the glm5 tep8_mtp3_8k1k
max_num_tokens config mismatch, and the deepseek_r1_fp4_v2 dep4_mtp1_1k8k
KV-cache-v2 scheduler deadlock. Fixes for these are tracked separately.

Test Coverage

Re-enables 17 perf-sanity test_e2e cases in CI that were previously skipped.

Dev Engineer Review

• tests/integration/test_lists/waives.txt removes 17 recovered perf/test_perf_sanity.py::test_e2e waivers.
• The change contains no insertions and no code or public API changes.
• The waiver format and test paths remain consistent.
• Ten cases remain waived.
• The scope matches validation on GB300, GB200, and DGX B200 hardware.

Verdict: sufficient.

QA Engineer Review

• No test-db/ or qa/ files were modified.
• The change removes 17 entries from tests/integration/test_lists/waives.txt.
• The affected cases cover perf/test_perf_sanity.py::test_e2e on GB300, GB200, and DGX B200 hardware.
• The 17 removed waivers passed reruns with complete request accounting, zero failures, non-null throughput, and no crash, OOM, hang, or timeout.
• CBTS coverage data is not provided.

Verdict: needs follow-up.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d324e2e2-c681-46bf-b121-39545d0b159d

📥 Commits

Reviewing files that changed from the base of the PR and between 0c96b06 and e45d9d3.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

The change adds one LagunaXS NVFP4 accuracy waiver and removes obsolete DGX_B200 and performance waivers from the integration-test waiver list.

Changes

Integration waivers

Layer / File(s)	Summary
Add LagunaXS accuracy waiver tests/integration/test_lists/waives.txt	Adds an NVFP4 accuracy waiver for LagunaXS.
Rework performance waivers tests/integration/test_lists/waives.txt	Removes obsolete DGX_B200, GLM-5, DeepSeek R1 FP8, GPT-OSS, SuperAD, and disaggregated performance waivers. Retains selected DeepSeek R1 and DeepSeek V4 Pro skips.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to e45d9

This change only re-enables 17 previously waived performance tests after reported successful runs on the matching hardware; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: brnguyen2, mzweilz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: removing 17 recovered performance-test waivers.
Description check	✅ Passed	The description explains the scope, rationale, affected cases, verification, remaining waivers, and test coverage; the checklist is omitted but non-critical.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[brnguyen2]

Suggestion on how to land this, plus two evidence asks.

Land these as post-merge first, then promote. Removing the waive puts each case straight back into its test-db stage, and for the ones whose entry is stage: pre_merge a single re-failure blocks everyone's PRs, not just yours. Since the recovery evidence here is one green run per case (see below), the low-risk path is: unwaive and flip the entry to post-merge in the same PR, let them soak on main for a week or two of post-merge runs, then a follow-up PR promotes the ones with a clean streak back to pre-merge. You keep the coverage you're restoring, and an intermittent case costs a post-merge alert instead of an L0 outage. Concretely, gemma4_26b_a4b_nvfp4_tp1_1k1k and llama8b_spec_bs1_128_128 are stage: pre_merge in l0_b200_perf_sanity.yml — those are the two worth moving; the rest are already post-merge, so for them "unwaive as-is" is fine risk-wise and only points 2/3 apply.

1. Evidence that these are actually fixed, not intermittently passing. The PR rests on one re-run of each case, with a functional pass bar (Total == Successful, Failed == 0, non-null throughput, no crash/OOM/hang). For perf-sanity and disagg cases, one green run doesn't distinguish "fixed" from "passed this time". Please post, per unwaived ID: the pipeline/job URLs, how many runs each got, and on which platform. A case with a single pass is exactly the one that should land post-merge rather than pre-merge.

2. Tie each unwaive to a root cause. None of the 8 cited bugs (6517846, 6572843, 6566777, 6374910, 6571410, 6571408, 6432948, 6153575) appears to have a fix identified. A bug that's still open with no fixing commit and a case that passed once is intermittency, not recovery. For each bug, please name what fixed it (commit/MR, or an infra/config change such as a driver/container/baseline update) and then move it to V2C so it isn't left open with nothing tracking it. Where you can't name a cause, say so explicitly — those are the strongest candidates for post-merge-only. 6153575 in particular is 105 days open and titled a perf regression in super_ad_blackwell; a regression doesn't recover on its own, so something specific must have changed.

3. Recovery criterion doesn't cover the pre-merge gate. perf_regression_utils.process_and_upload_test_results sets fail_on_regression = not is_post_merge, so a pre-merge case fails on an OpenSearch baseline regression even when the run completes with full request accounting. Your stated bar (Total == Successful == num_prompts, Failed == 0, non-null throughput) doesn't exercise that comparison at all, which is another reason to land the two pre-merge cases as post-merge: post-merge runs upload a baseline without gating on it, so you get real signal before they can block anyone. If you'd rather keep them pre-merge, please /bot run so those two actually hit the regression gate first — 6571410/6571408 were both filed as pre-merge failures.

Happy to re-review once the staging decision and the run evidence are in.

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-PerfSanity-1,DGX_B200-PyTorch-1,DGX_B200-PyTorch-2,DGX_B200-PyTorch-3,DGX_B200-PyTorch-4,DGX_B200-PyTorch-5,DGX_B200-PyTorch-6,DGX_B200-PyTorch-7,DGX_B200-PyTorch-8,DGX_B200-PyTorch-9,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge*,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge*,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge*,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge*,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge*,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8*,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge*,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE8-GPU32-Post-Merge*,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge*"

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-PerfSanity-1,DGX_B200-PyTorch-1,DGX_B200-PyTorch-2,DGX_B200-PyTorch-3,DGX_B200-PyTorch-4,DGX_B200-PyTorch-5,DGX_B200-PyTorch-6,DGX_B200-PyTorch-7,DGX_B200-PyTorch-8,DGX_B200-PyTorch-9,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge*,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge*,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge*,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge*,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge*,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8*,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge*,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE8-GPU32-Post-Merge*,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge*"

[tensorrt-cicd]

PR_Github #67756 [ run ] triggered by Bot. Commit: 110fc67 Link to invocation

[tensorrt-cicd]

PR_Github #67756 [ run ] completed with state FAILURE. Commit: 110fc67
/LLM/main/L0_MergeRequest_PR pipeline #55235 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chenfeiz0326]

/bot skip --comment "Unwaive a perf test, No need to run the whole CI pipeline"

[tensorrt-cicd]

PR_Github #68316 [ skip ] triggered by Bot. Commit: 0c96b06 Link to invocation

[chenfeiz0326]

/bot skip --comment "Unwaive a perf test, No need to run the whole CI pipeline"

[tensorrt-cicd]

PR_Github #68316 [ skip ] completed with state SUCCESS. Commit: 0c96b06
Skipping testing for commit 0c96b06

Link to invocation

[tensorrt-cicd]

PR_Github #68320 [ skip ] triggered by Bot. Commit: e45d9d3 Link to invocation

[tensorrt-cicd]

PR_Github #68320 [ skip ] completed with state SUCCESS. Commit: e45d9d3
Skipping testing for commit e45d9d3

Link to invocation