Skip to content

[https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix - #18732

Open
chang-l wants to merge 1 commit into
NVIDIA:mainfrom
chang-l:unwaive-6676844
Open

[https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix#18732
chang-l wants to merge 1 commit into
NVIDIA:mainfrom
chang-l:unwaive-6676844

Conversation

@chang-l

@chang-l chang-l commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • Removed the waiver for examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example from tests/integration/test_lists/waives.txt.
  • The change has the intended scope and does not modify production code or public APIs.
  • The test-list entry is correctly formatted and has no duplicate or unrelated changes.

QA Engineer Review

  • Modified tests/integration/test_lists/waives.txt.
  • Removed the waiver for examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example.
  • CBTS touchmap coverage is not applicable because this is a test-list-only change.
  • Verdict: sufficient.

Description

Unwaive examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example, waived in #18303 for https://nvbugs/6676844.

The failure was not a code regression. The example died during weight loading with Error while deserializing header: header too large on Wan2.2-T2V-A14B-Diffusers-NVFP4/transformer/diffusion_pytorch_model-00002-of-00002.safetensors on the CI model share (/scratch.trt_llm_data/llm-models). The share's disk filled up on 2026-08-26 and the shard was left corrupt: every other test in the same DGX_B200-PyTorch-4 stage passed (28 passed / 1 skipped / this 1 failure), the Qwen-Image example loaded fine through the same loader, and no visual_gen source changed between the last good commit (ca939b7) and the first bad one (128950d). The infra owner cleaned and repaired the share on 2026-08-27.

This PR only removes the waiver line. The test is in the pre-merge DGX_B200-PyTorch stage (l0_b200.yml), so this PR's CI verifies the NVFP4 checkpoint loads again.

Test Coverage

  • examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example (l0_b200.yml pre_merge; also listed in l0_dgx_b200.yml post_merge)

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…checkpoint storage fix

The failure was not a code regression: the Wan2.2-T2V-A14B-Diffusers-NVFP4
checkpoint on the CI model share had a corrupt safetensors shard
(transformer/diffusion_pytorch_model-00002-of-00002.safetensors, 'header too
large') after the llm-models scratch disk filled up on 2026-08-26. The infra
owner cleaned and repaired the share on 2026-08-27, so the waiver added in
NVIDIA#18303 can be dropped.

The test runs in the pre-merge DGX_B200-PyTorch stage (l0_b200.yml), so this
PR's CI run verifies that the NVFP4 checkpoint loads again.

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@chang-l

chang-l commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 093b8ef5-a9c4-4e86-aa1b-291c991a603b

📥 Commits

Reviewing files that changed from the base of the PR and between f7b7f22 and e47637f.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The integration waiver list no longer skips test_wan_t2v_example in examples/visual_gen/test_visual_gen_wan.py.

Changes

Wan test execution

Layer / File(s) Summary
Remove Wan waiver
tests/integration/test_lists/waives.txt
Removed the waiver and associated bug reference for the Wan text-to-video example test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to e4763

This change re-enables the Wan text-to-video example test in pre-merge CI. No concrete merge-blocking risk remains in the supplied change context.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description check ✅ Passed The description explains the failure cause, the infrastructure repair, the single waiver removal, and the relevant pre-merge test coverage. It includes the required Description, Test Coverage, and PR …
Title check ✅ Passed The title identifies the NVBugs issue, test change, and reason for removing the waiver. It clearly summarizes the primary change.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #71566 [ run ] triggered by Bot. Commit: e47637f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #71566 [ run ] completed with state SUCCESS. Commit: e47637f
/LLM/main/L0_MergeRequest_PR pipeline #58652 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chang-l

chang-l commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

CI status note for L0 #58652 (this PR's first run):

  • The unwaived test passed on B200: CBTS narrowed the DGX_B200-PyTorch stage to exactly examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example and it reported 1 passed (call 511 s, exit code 0). So the Wan2.2 NVFP4 checkpoint on the CI share is restored and the waiver can go.
  • The pipeline FAILURE comes from CPU-Generic-x86-1, unrelated to this change: unittest/others (test_time_breakdown.TestRequestTimeBreakdown::test_parse_records_matches_parse_json_file, Lists differ) and unittest/llmapi/apps/test_request_metrics.py. Both tests were added by [None][feat] perf-sanity: upload per-request disagg lifecycle spans to OpenSearch #18445 (737af47, merged a few hours before this branch), and the same two failures show up today in every other PR run whose base includes that commit, so this is a main-branch breakage. I'll rebase and re-run once a fix or waiver for those lands on main.

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the waiver is the right recovery test because the original failure was attributed to corrupted shared checkpoint storage rather than a source regression. Since this PR changes only test admission, I would treat the exact pre-merge B200 execution of test_wan_t2v_example as the merge gate; that proves the repaired checkpoint path actually loads under the same environment that failed before. No code-path risk beyond that validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants