Skip to content

[docs] debug guide: tell a cold Triton compile from a colocate hang (en + zh) - #2364

Open
David553 wants to merge 1 commit into
THUDM:mainfrom
David553:docs/debug-colocate-first-step-compile
Open

[docs] debug guide: tell a cold Triton compile from a colocate hang (en + zh)#2364
David553 wants to merge 1 commit into
THUDM:mainfrom
David553:docs/debug-colocate-first-step-compile

Conversation

@David553

@David553 David553 commented Sep 6, 2026

Copy link
Copy Markdown

Why

#1487 — a colocate run finishes its rollout, prints Timer actor_train start, and then only SGLang health checks; several people in that thread report the same, and "turning colocate off helps". We hit the identical shape on Miles, a slime fork that shares slime's reloadable_process_group and sleep() / wake_up() path; what it turned out to be is written up in #1487 (comment).

developer_guide/debug.md currently says nothing about colocate offload, process-group rebuild, py-spy, watchdog timeouts, or Triton's per-container cache, so the natural next step for anyone hitting this is to raise the NCCL timeout and re-run — which on releases before v0.3.1 did nothing (see below).

What

A new section in docs/en/developer_guide/debug.md and docs/zh/developer_guide/debug.md, placed before the INT4 section, with:

  • the py-spy check (one rank per node / pipeline stage);
  • the two stack shapes that explain most of these hangs: one pipeline stage inside the Triton compiler (fla/ops/gated_delta_rule on GDN models) while the next waits in recv_forward_communicate_shapestorch.cuda.synchronize; and, on releases before Support reloading the default process group #2208 / v0.3.1, the rebuilt process groups silently falling back to the default 10-minute timeout regardless of --distributed-timeout-minutes;
  • the three ways out (warm the Triton cache before a multi-node launch, share TRITON_CACHE_DIR, size the timeout to cover the first step);
  • how to recognize a real collective mismatch instead, and what to attach to an issue.

Where the numbers come from

2 × 8 H20, Qwen3.6-35B-A3B, TP2 / PP2 / EP8, colocate, on the slime fork: cold first actor_train 639 s and the preceding log-prob pass 394 s; the same passes 60 s and 13 s once ~/.triton/cache was warm. py-spy on all 16 ranks during the stall showed 8/8 stage-0 ranks in triton/backends/nvidia/compiler.py:make_llir and 8/8 stage-1 ranks in recv_forward. The timeout claim was verified with a two-process probe against reload_process_groups() before and after #2208: _get_backend(device).options._timeout goes 2:00:00 → 0:10:00 on the old code and stays 2:00:00 after.

Docs only; en and zh updated together.

🤖 Generated with Claude Code

@David553
David553 force-pushed the docs/debug-colocate-first-step-compile branch from 0bd34f9 to ef043c9 Compare September 6, 2026 08:31
…en + zh)

A colocate run that finishes its rollout and goes quiet at `Timer actor_train
start` is usually not a communication deadlock (THUDM#1487): one pipeline stage is
JIT-compiling a Triton kernel (fla/ops/gated_delta_rule on GDN models) while
the next waits in recv_forward, and it is the waiting stage's NCCL watchdog
that fires. Measured on a slime fork with Qwen3.6-35B-A3B on 2x8 H20: 639 s /
394 s cold, 60 s / 13 s once the per-container Triton cache is warm.

Before THUDM#2208 (v0.3.1) the rebuilt process groups also fell back to the
default 10-minute timeout regardless of --distributed-timeout-minutes, which
is why the hang appeared to ignore the flag and die at exactly 600 s.

Adds a section to developer_guide/debug.md (en and zh) with the py-spy check
that tells the two apart, the three ways out, and how to tell when it is a
real collective mismatch instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@David553
David553 force-pushed the docs/debug-colocate-first-step-compile branch from ef043c9 to 472b7d7 Compare September 7, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant