Skip to content

[https://nvbugs/6561778][fix] Fence all ranks before pytest launch in multi-node slurm_run.sh #17372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
EmmaQiaoCh merged 4 commits into from
Aug 26, 2026
Merged

[https://nvbugs/6561778][fix] Fence all ranks before pytest launch in multi-node slurm_run.sh #17372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
85754cf
[nvbugs/6561778][fix] Fence all Slurm ranks before launching pytest
trtllm-agent Aug 6, 2026
e55a9b6
Merge branch 'main' into repair-bot-bug6561778
EmmaQiaoCh Aug 25, 2026
7f9e33a
Simplify progress logging in slurm_run.sh
EmmaQiaoCh Aug 25, 2026
c6bcc0f
Fix comments' format
EmmaQiaoCh Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions jenkins/scripts/slurm_run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,9 @@ slurm_wait_all_ranks() {
touch "$readyDir/rank_${SLURM_PROCID}.ready"

# Bounded so a dead rank fails the stage loudly instead of hanging until the
# partition walltime kills it; the ceiling exceeds the 2700s pip3 retry budget
# partition walltime kills it. This bounds arrival skew between ranks (each
# rank's deadline starts after its own install finished) -- comfortably above
# the ~10min skew seen in the bug - not any single install-phase timeout;
# in slurm_install.sh so a merely slow rank still releases the barrier.
local timeoutSecs=3600
local deadline=$((SECONDS + timeoutSecs))
Expand All @@ -117,10 +119,7 @@ slurm_wait_all_ranks() {
"all $numRanks ranks to be ready; ready: $ready/$numRanks"
return 1
fi
# One rank reports progress; all of them would spam the log every 10s.
if [ "$SLURM_PROCID" -eq 0 ]; then
echo "(Waiting for all $numRanks ranks to be ready) ready: $ready/$numRanks"
fi
echo "(Waiting for all $numRanks ranks to be ready) ready: $ready/$numRanks"
sleep 10
done
}
Expand Down
1 change: 0 additions & 1 deletion tests/integration/test_lists/waives.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[nvfp4_ws8_80g
accuracy/test_llm_api_autodeploy.py::TestQwen3_5_397B_MoE::test_bf16_small[4] SKIP (https://nvbugs/6507114)
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput_mtp] SKIP (https://nvbugs/6428101)
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput_mtp_trtllm] SKIP (https://nvbugs/6426868)
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency] SKIP (https://nvbugs/6561778)
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency_trtllmgen_adp_lmtp] SKIP (https://nvbugs/6655987)
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_pp4_mtp] SKIP (https://nvbugs/6481323)
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1LongBenchV2::test_fp8_8gpus SKIP (https://nvbugs/6621358)
Expand Down
Loading