[https://nvbugs/6480621][test] Revert to 60-second KV transfer timeout for GB300 DeepSeek V4 Pro disaggregated perf-sanity by chienchunhung · Pull Request #17137 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-31T19:21:20Z

Summary

This PR updates one GB300 DeepSeek V4 Pro disaggregated perf-sanity target:

explicitly enables cache_transceiver_precheck;
sets GEN and CTX kv_transfer_timeout_ms from 600000 ms to 60000 ms;
keeps the exact gen-only test unwaived so the modified target is exercised.

PRs #17223 and #18185 have merged into main. The current PR head a4293f296272d202ffad8ebcbd4953066b377580 is rebased directly on main at 36138a2c8e0350d5f23bba41c1c12962a187009b. Merged #18185 supplies the DeepSeek V4 EPLB weight-loading OOM fix and removes this target's waiver.

Validation

Current-head targeted validation: PASSED

Command:

/bot run --disable-fail-fast --disable-reuse-test --only-multi-gpu-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

Exact test:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_deepseek-v4-pro-fp4_8k1k_con180_ctx3_dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL]

Evidence:

PR_Github #69570 and L0 #56888 ran against commit a4293f296 with test reuse disabled.
The exact test passed in 1089.791 seconds; its JUnit result has zero failures.
Slurm job 3325958 completed with exit code 0:0 across nvl72d217-T[01-11].
The disaggregated server and benchmark completed successfully using the Python transceiver, 60-second GEN/CTX KV-transfer timeouts, and the cache-transceiver precheck.
No host OOM, KV-transfer timeout, or data-corruption failure was observed.

This validates the current #17223 + #18185 + #17137 combination for the 11-node/44-GPU, 3-CTX, concurrency-180 CI proxy. It does not replace rerunning the original 8P1D, concurrency-1760 AgentPerf workload reported in NVBUG #6480621.

Related PRs

[https://nvbugs/6480621][fix] Preserve KV ownership in disaggregated precheck #17223: merged ownership/deadline fix.
[https://nvbugs/6272397][fix] Prevent host OOM during checkpoint prefetch #17430: merged checkpoint-prefetch host-OOM fix.
[https://nvbugs/6644226][fix] Reduce DeepSeek V4 EPLB loading memory #18185: merged DeepSeek V4 EPLB weight-loading host-OOM fix; also re-enabled the target.
[None][feat] cache transceiver test in Perf sanity #16674: introduced the precheck being exercised.

chienchunhung · 2026-07-31T19:21:53Z

/bot run --disable-fail-fast --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

chienchunhung · 2026-07-31T19:25:04Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-07-31T19:27:43Z

PR_Github #63105 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-07-31T19:31:38Z

PR_Github #63106 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-07-31T19:35:23Z

PR_Github #63105 [ run ] completed with state ABORTED. Commit: 38ca970

Link to invocation

tensorrt-cicd · 2026-07-31T21:00:54Z

PR_Github #63106 [ run ] completed with state SUCCESS. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51197 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chienchunhung · 2026-07-31T23:00:28Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-07-31T23:06:22Z

PR_Github #63144 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-07-31T23:13:50Z

PR_Github #63144 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51231 (Partly Tested) completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chienchunhung · 2026-07-31T23:35:35Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-07-31T23:41:53Z

PR_Github #63150 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-08-01T00:38:05Z

PR_Github #63150 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51234 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-01T01:22:37Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-08-01T01:28:42Z

PR_Github #63165 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-08-01T02:53:55Z

PR_Github #63165 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51249 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-03T20:38:32Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-08-03T20:44:14Z

PR_Github #63555 [ run ] triggered by Bot. Commit: 3e6c120 Link to invocation

tensorrt-cicd · 2026-08-03T21:27:26Z

PR_Github #63555 [ run ] completed with state SUCCESS. Commit: 3e6c120
/LLM/main/L0_MergeRequest_PR pipeline #51521 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-03T21:42:12Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-03T21:47:58Z

PR_Github #63564 [ run ] triggered by Bot. Commit: b7bd98a Link to invocation

tensorrt-cicd · 2026-08-03T22:59:04Z

PR_Github #63564 [ run ] completed with state SUCCESS. Commit: b7bd98a
/LLM/main/L0_MergeRequest_PR pipeline #51530 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chienchunhung · 2026-08-26T17:51:22Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-26T17:57:12Z

PR_Github #69490 [ run ] triggered by Bot. Commit: f66c303 Link to invocation

tensorrt-cicd · 2026-08-26T20:34:18Z

PR_Github #69490 [ run ] completed with state SUCCESS. Commit: f66c303
/LLM/main/L0_MergeRequest_PR pipeline #56817 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-26T21:39:19Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-26T21:47:44Z

PR_Github #69539 [ run ] triggered by Bot. Commit: 6b5d69c Link to invocation

tensorrt-cicd · 2026-08-26T22:32:03Z

PR_Github #69539 [ run ] completed with state FAILURE. Commit: 6b5d69c
/LLM/main/L0_MergeRequest_PR pipeline #56859 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine

Stamp on behalf of runtime-devs, delegating review to @NVIDIA/trt-llm-torch-attention-devs

chienchunhung · 2026-08-27T00:26:04Z

/bot run --disable-fail-fast --disable-reuse-test --only-multi-gpu-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-27T00:32:23Z

PR_Github #69570 [ run ] triggered by Bot. Commit: a4293f2 Link to invocation

tensorrt-cicd · 2026-08-27T01:46:02Z

PR_Github #69570 [ run ] completed with state SUCCESS. Commit: a4293f2
/LLM/main/L0_MergeRequest_PR pipeline #56888 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chienchunhung · 2026-08-27T18:00:04Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-27T18:06:06Z

PR_Github #69783 [ run ] triggered by Bot. Commit: a4293f2 Link to invocation

tensorrt-cicd · 2026-08-27T20:11:21Z

PR_Github #69783 [ run ] completed with state SUCCESS. Commit: a4293f2
/LLM/main/L0_MergeRequest_PR pipeline #57078 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-08-27T23:08:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-27T23:13:33Z

PR_Github #69820 [ run ] triggered by Bot. Commit: ea57698 Link to invocation

tensorrt-cicd · 2026-08-28T03:53:28Z

PR_Github #69820 [ run ] completed with state SUCCESS. Commit: ea57698
/LLM/main/L0_MergeRequest_PR pipeline #57112 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-28T16:40:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-28T16:46:26Z

PR_Github #70010 [ run ] triggered by Bot. Commit: ea57698 Link to invocation

tensorrt-cicd · 2026-08-28T17:37:52Z

PR_Github #70010 [ run ] completed with state SUCCESS. Commit: ea57698
/LLM/main/L0_MergeRequest_PR pipeline #57289 completed with status: 'SUCCESS'

CI Report

Link to invocation

github-actions Bot assigned chienchunhung Jul 31, 2026

chienchunhung changed the title ~~[https://nvbugs/6480621][test] Validate 60-second KV transfer timeout~~ [https://nvbugs/6480621][test] Revert to 60-second KV transfer timeout for GB300 DeepSeek V4 Pro disaggregated perf-sanity Jul 31, 2026

Shixiaowei02 approved these changes Aug 3, 2026

View reviewed changes

chienchunhung mentioned this pull request Aug 3, 2026

[https://nvbugs/6480621][fix] Preserve KV ownership in disaggregated precheck #17223

Merged

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 38ca970 to 3e6c120 Compare August 3, 2026 20:37

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 3e6c120 to b7bd98a Compare August 3, 2026 21:40

chienchunhung marked this pull request as ready for review August 3, 2026 23:41

chienchunhung requested review from a team as code owners August 3, 2026 23:41

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 6573ef3 to f66c303 Compare August 26, 2026 17:51

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from f66c303 to 6b5d69c Compare August 26, 2026 21:38

chienchunhung requested a review from a team as a code owner August 26, 2026 21:38

chienchunhung requested a review from WeiHaocheng August 26, 2026 21:38

mikeiovine approved these changes Aug 26, 2026

View reviewed changes

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 6b5d69c to a4293f2 Compare August 27, 2026 00:18

chienchunhung enabled auto-merge (squash) August 27, 2026 18:02

chienchunhung mentioned this pull request Aug 27, 2026

[https://nvbugs/6681216][infra] Waive DeepSeek V3 Lite NIXL on B200 and B300 #18325

Merged

[https://nvbugs/6480621][test] Validate 60-second KV transfer timeout

ea57698

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from a4293f2 to ea57698 Compare August 27, 2026 23:07

BowenFu approved these changes Aug 28, 2026

View reviewed changes

chienchunhung merged commit 3c4dc51 into NVIDIA:main Aug 28, 2026
7 checks passed

Uh oh!

Conversation

chienchunhung commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Related PRs

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

chienchunhung commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

chienchunhung commented Aug 26, 2026

Uh oh!

tensorrt-cicd commented Aug 26, 2026

Uh oh!

tensorrt-cicd commented Aug 26, 2026

Uh oh!

chienchunhung commented Aug 26, 2026

Uh oh!

tensorrt-cicd commented Aug 26, 2026

Uh oh!

tensorrt-cicd commented Aug 26, 2026

Uh oh!

mikeiovine left a comment

Choose a reason for hiding this comment

Uh oh!

chienchunhung commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

chienchunhung commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

chienchunhung commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

chienchunhung commented Jul 31, 2026 •

edited

Loading