[profiling] Add trainer communication timeline

[0z5a]

Draft update — 2026-09-05

Labels CUDA-event timing as event brackets rather than kernel-observed timestamps, and removes disabled-timeline hot-path work. The targeted timeline follow-up passed 10 tests. No new full Ray/SGLang GPU run or cross-rank clock-calibration claim is made.

AI assistance: OpenAI Codex. Remains Draft; implementation, CPU/Gloo checks and production GPU validation are separate acceptance gates. No new GPU results were generated during publication.

The original submission description and validation history follow.

---

Summary

• add a default-off trainer-side communication timeline under slime.observability, with semantic JSONL records, CUDA-event timing, and optional NVTX ranges
• propagate one run identity and the active rollout identity across training and weight synchronization
• instrument forward/backward, gradient synchronization, optimizer, weight conversion, bucket readiness/send, engine receive/load, and synchronization completion
• cover distributed NCCL, colocated CUDA IPC/tensor, full-disk, and delta-disk weight update paths
• expose a custom expert-parallel hook API without assuming a fixed world size or rank count
• register the timeline regression test in the generated CPU PR workflow

Rebase and duplicate audit

This draft is rebased onto main at 4c193f1f. Conflict resolution preserves the current accelerator abstraction, weight-updater factory, and observability refactor.

The newer slime.observability.trace_utils implementation records rollout/sample spans for offline trajectory inspection. This draft records trainer-process and weight-sync phases with CUDA events and a shared run/rollout identity, so the two facilities are complementary rather than duplicate implementations. A fresh open issue/PR search found no equivalent trainer communication timeline.

Compatibility and scope

• tracing remains disabled unless --communication-timeline or its environment-variable equivalent is explicitly configured
• the disabled path does not create timeline files or issue rollout-ID RPCs
• this is observability only: it does not change collective ordering or introduce a runtime scheduling policy
• no 5-rank or 6-rank special case is present

Validation

• targeted CPU/regression tests: 31 passed (tests/observability/test_communication_timeline.py, tests/test_empty_colocated_weight_bucket.py, and tests/test_megatron_argument_validation.py)
• full repository pre-commit suite passed: YAML/case/private-key/large-file/requirements checks, Ruff, autoflake, isort, and Black
• changed Python files passed py_compile; the rebased diff passed git diff --check
• an earlier single-GPU CUDA smoke for this implementation emitted 7 timeline records, each with a real CUDA-event duration; the conflict-only rebase was validated with CPU tests and static checks

Limitations

• no end-to-end multi-node Ray/SGLang run or performance claim is made in this draft
• PyTorch-dependent upstream factory/trace tests were not runnable in the local macOS test environment; repository CI remains the source of truth for those jobs

AI assistance disclosure

This draft was developed with OpenAI Codex assistance. The human submitter must review and understand every changed line and be prepared to defend the design and test coverage before marking it ready for review.