[https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix - #18732
[https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix#18732chang-l wants to merge 1 commit into
Conversation
…checkpoint storage fix The failure was not a code regression: the Wan2.2-T2V-A14B-Diffusers-NVFP4 checkpoint on the CI model share had a corrupt safetensors shard (transformer/diffusion_pytorch_model-00002-of-00002.safetensors, 'header too large') after the llm-models scratch disk filled up on 2026-08-26. The infra owner cleaned and repaired the share on 2026-08-27, so the waiver added in NVIDIA#18303 can be dropped. The test runs in the pre-merge DGX_B200-PyTorch stage (l0_b200.yml), so this PR's CI run verifies that the NVFP4 checkpoint loads again. Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
|
/bot run |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughThe integration waiver list no longer skips ChangesWan test execution
Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: ⚪ Minimal · up to This change re-enables the Wan text-to-video example test in pre-merge CI. No concrete merge-blocking risk remains in the supplied change context. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #71566 [ run ] triggered by Bot. Commit: |
|
PR_Github #71566 [ run ] completed with state
|
|
CI status note for L0 #58652 (this PR's first run):
|
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Removing the waiver is the right recovery test because the original failure was attributed to corrupted shared checkpoint storage rather than a source regression. Since this PR changes only test admission, I would treat the exact pre-merge B200 execution of test_wan_t2v_example as the merge gate; that proves the repaired checkpoint path actually loads under the same environment that failed before. No code-path risk beyond that validation.
Dev Engineer Review
examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_examplefromtests/integration/test_lists/waives.txt.QA Engineer Review
tests/integration/test_lists/waives.txt.examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example.Description
Unwaive
examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example, waived in #18303 for https://nvbugs/6676844.The failure was not a code regression. The example died during weight loading with
Error while deserializing header: header too largeonWan2.2-T2V-A14B-Diffusers-NVFP4/transformer/diffusion_pytorch_model-00002-of-00002.safetensorson the CI model share (/scratch.trt_llm_data/llm-models). The share's disk filled up on 2026-08-26 and the shard was left corrupt: every other test in the sameDGX_B200-PyTorch-4stage passed (28 passed / 1 skipped / this 1 failure), the Qwen-Image example loaded fine through the same loader, and no visual_gen source changed between the last good commit (ca939b7) and the first bad one (128950d). The infra owner cleaned and repaired the share on 2026-08-27.This PR only removes the waiver line. The test is in the pre-merge
DGX_B200-PyTorchstage (l0_b200.yml), so this PR's CI verifies the NVFP4 checkpoint loads again.Test Coverage
examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example(l0_b200.ymlpre_merge; also listed inl0_dgx_b200.ymlpost_merge)PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.