feat(weight-sync): add engine-group wave scheduling

[0z5a]

Draft update — 2026-09-05

Includes the 2/4-scale follow-up and production wave-helper callsite validation. The callsite harness uses control-plane stubs; it is not a complete Ray/SGLang engine-load run. No production performance claim is made.

AI assistance: OpenAI Codex. Remains Draft; implementation, CPU/Gloo checks and production GPU validation are separate acceptance gates. No new GPU results were generated during publication.

The original submission description and validation history follow.

---

Summary

• add optional deterministic wave admission for logical rollout engine groups during weight sync
• apply the same bound to non-colocated NCCL, colocated tensor/IPC, full and delta disk reloads, and quantized post-load processing
• add English/Chinese documentation and CPU-only CI coverage

Motivation

Updating every rollout engine at once maximizes fan-out, but can create a short burst across GPU memory, PCIe/NVLink, network, or storage. A deployment should be able to cap that fan-out without hardware-specific sleeps and without assuming a particular rank count.

Semantics and compatibility

--update-weight-max-inflight-engine-groups N admits at most N logical SGLang engines at a time. An engine remains indivisible across its TP, PP, and multi-node workers. Admission follows stable engine order; for example, five engines at a limit of two form (0, 1), (2, 3), (4).

The default is 0, preserving the existing all-at-once path and aggregate process-group topology. A value greater than or equal to the resolved engine count is also effectively unbounded. There are no world-size, rank-count, or six-rank special cases.

Pause, flush, source-buffer reuse, and weight-version commit boundaries remain outside individual waves, so one completed wave does not resume serving early.

Validation

Correctness validation completed before publication:

• scheduler contract tests cover empty, bounded, unbounded, oversized-limit, invalid-limit, and non-special engine counts (1, 2, 4, 5, and 7)
• distributed contract tests check default aggregate topology, bounded per-engine topology, partial setup cleanup, asynchronous launches within a wave, and completion before the next wave is admitted
• affected argument-validation and colocated-weight-bucket regression tests passed
• repository pre-commit hooks and git diff --check passed
• a single-card CUDA/NCCL smoke was limited to physical GPU 6; it checks that the CUDA path remains viable but cannot validate multi-engine wave behavior

Performance validation is deliberately limited: event-order tests verify that active logical engine groups never exceed the configured bound, but no throughput or latency claim is made from synthetic tests. A real Ray/SGLang end-to-end run was not performed. Before merge, the option should be benchmarked on the target multi-engine topology against the default (0), including weight-sync duration and rollout tail latency.

Duplicate check

I searched current open issues and PRs for engine-group waves, staggered/bounded weight updates, in-flight engine scheduling, process groups, and the exact new option name. I found no substantive duplicate. #2331 is related but distinct: it orders pipeline process-group connect/disconnect, while this PR controls steady-state weight-transfer/reload admission.

AI assistance and required review

This implementation, tests, documentation, and PR text were prepared with OpenAI Codex assistance. A human contributor/maintainer must review every changed line, validate the design assumptions, and reproduce the relevant tests and deployment benchmark before this draft is considered ready to merge.