[docs] debug guide: tell a cold Triton compile from a colocate hang (en + zh) - #2364
Open
David553 wants to merge 1 commit into
Open
[docs] debug guide: tell a cold Triton compile from a colocate hang (en + zh)#2364David553 wants to merge 1 commit into
David553 wants to merge 1 commit into
Conversation
David553
force-pushed
the
docs/debug-colocate-first-step-compile
branch
from
September 6, 2026 08:31
0bd34f9 to
ef043c9
Compare
…en + zh) A colocate run that finishes its rollout and goes quiet at `Timer actor_train start` is usually not a communication deadlock (THUDM#1487): one pipeline stage is JIT-compiling a Triton kernel (fla/ops/gated_delta_rule on GDN models) while the next waits in recv_forward, and it is the waiting stage's NCCL watchdog that fires. Measured on a slime fork with Qwen3.6-35B-A3B on 2x8 H20: 639 s / 394 s cold, 60 s / 13 s once the per-container Triton cache is warm. Before THUDM#2208 (v0.3.1) the rebuilt process groups also fell back to the default 10-minute timeout regardless of --distributed-timeout-minutes, which is why the hang appeared to ignore the flag and die at exactly 600 s. Adds a section to developer_guide/debug.md (en and zh) with the py-spy check that tells the two apart, the three ways out, and how to tell when it is a real collective mismatch instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
David553
force-pushed
the
docs/debug-colocate-first-step-compile
branch
from
September 7, 2026 04:28
ef043c9 to
472b7d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#1487 — a colocate run finishes its rollout, prints
Timer actor_train start, and then only SGLang health checks; several people in that thread report the same, and "turning colocate off helps". We hit the identical shape on Miles, a slime fork that shares slime'sreloadable_process_groupandsleep()/wake_up()path; what it turned out to be is written up in #1487 (comment).developer_guide/debug.mdcurrently says nothing about colocate offload, process-group rebuild, py-spy, watchdog timeouts, or Triton's per-container cache, so the natural next step for anyone hitting this is to raise the NCCL timeout and re-run — which on releases before v0.3.1 did nothing (see below).What
A new section in
docs/en/developer_guide/debug.mdanddocs/zh/developer_guide/debug.md, placed before the INT4 section, with:fla/ops/gated_delta_ruleon GDN models) while the next waits inrecv_forward→_communicate_shapes→torch.cuda.synchronize; and, on releases before Support reloading the default process group #2208 / v0.3.1, the rebuilt process groups silently falling back to the default 10-minute timeout regardless of--distributed-timeout-minutes;TRITON_CACHE_DIR, size the timeout to cover the first step);Where the numbers come from
2 × 8 H20, Qwen3.6-35B-A3B, TP2 / PP2 / EP8, colocate, on the slime fork: cold first
actor_train639 s and the preceding log-prob pass 394 s; the same passes 60 s and 13 s once~/.triton/cachewas warm. py-spy on all 16 ranks during the stall showed 8/8 stage-0 ranks intriton/backends/nvidia/compiler.py:make_llirand 8/8 stage-1 ranks inrecv_forward. The timeout claim was verified with a two-process probe againstreload_process_groups()before and after #2208:_get_backend(device).options._timeoutgoes2:00:00 → 0:10:00on the old code and stays2:00:00after.Docs only; en and zh updated together.
🤖 Generated with Claude Code