Feat: USP sequence parallelism (Ulysses x Ring) via SequenceParallelPlan; Wan enabled - #21

Merged

Rockdu merged 40 commits into

mainfrom

Jul 25, 2026

Commits on Jul 24, 2026

diffusion: sequence-parallel config and Option B parallel state

zhihengy
and
claude
committed
diffusion: USP sequence parallelism for Wan training

zhihengy
and
claude
committed
diffusion: align engine grouping span with rollout, assert full coverage

zhihengy
and
claude
committed
diffusion: SP test suite (mesh, init smoke, parity, weight sync, import guard)

zhihengy
and
claude
committed
diffusion: SP guardrail runner + fp32 mode for grad-sync parity

zhihengy
and
claude
committed
diffusion: guardrail runner cleanup preamble + expert-selection override

zhihengy
and
claude
committed
diffusion: drive SP shard/gather hooks from the model's diffusers _cp_plan

zhihengy
and
claude
committed
diffusion: guardrail runner supports rollout-data save/replay for same-data band comparison

zhihengy
and
claude
committed
diffusion: own the differentiable USP operators, drop the sglang training dependency

zhihengy
and
claude
committed
diffusion: first-principles SP cleanup

zhihengy
and
claude
committed
diffusion: fsdp_shard_mode dp_sp — shard parameters over the dp x sp mesh

zhihengy
and
claude
committed
diffusion: materialize clip_grad_norm's DTensor before logging

zhihengy
and
claude
committed
diffusion: SequenceParallelPlan — per-family SP declaration behind ModelBackend

zhihengy
and
claude
committed
style: pre-commit formatting across sp files

zhihengy
and
claude
committed
diffusion: fail-fast SP validation — driver-side support check, wildcard and key-resolution guards

zhihengy
and
claude
committed
tests(sp): determinism smoke — SP attention path bitwise under deterministic mode

zhihengy
and
claude
committed
diffusion: parameter placement is Option A unconditionally — Option B demoted to a test anchor

zhihengy
and
claude
committed
diffusion: ring_degree is derived, not configured — drop it from every input surface

zhihengy
and
claude
committed
docs(sp_mesh): note the sp_subgroups <-> locate_rank inverse relation

zhihengy
and
claude
committed
diffusion: default SP attention intercepts at diffusers' dispatch_attention_fn

zhihengy
and
claude
committed
Revert "diffusion: align engine grouping span with rollout, assert full coverage"
zhihengy
committed
Revert the clip_grad_norm logging materialization (split to #35)

zhihengy
and
claude
committed
tests: keep the SP suite local — it lands in a separate test-infra PR

zhihengy
and
claude
committed
refactor: sp_plan.py owns the plan contract; sp_attention.py is dispatch-only

zhihengy
and
claude
committed
diffusion: unify cp_*/sp_* in ParallelState — sp is the only sequence axis

zhihengy
and
claude
committed
diffusion: pin ring templates to their torch>=2.11 home, flag the experimental dependency

zhihengy
and
claude
committed
style: isort on sp_ops.py

zhihengy
and
claude
committed
refactor(sp): make sequence parallel plan model-owned
zhihengy
committed
feat(sp): honor attention backend under pure Ulysses
zhihengy
committed
refactor(sp): trim test-only paths and harden validation
zhihengy
committed
refactor(sp): align validation with ownership boundaries
zhihengy
committed
refactor(sp): group sequence-parallel implementation
zhihengy
committed
test(sp): track attention parity and determinism
zhihengy
committed
sp: install USP via Wan attention processor; ring kernel follows the attention backend
zhihengy
committed
Revert "sp: install USP via Wan attention processor; ring kernel follows the attention backend"
zhihengy
committed
sp: ring attention honors --fsdp-attention-backend via RING_KERNELS
zhihengy
committed
refactor: signature bind
Rockdu
committed
refactor: absorb sp attention installation into ModelBackend
Rockdu
committed
refactor: absorb support sp as backend attribute
Rockdu
committed
style: format model backend

Rockdu
and
cursoragent
committed