Skip to content

fix: respect packed document boundaries in Qwen3.5 DeltaNet CP#3003

Open
hallerite wants to merge 1 commit into
mainfrom
fix/qwen3-5-deltanet-cp-packed-boundaries
Open

fix: respect packed document boundaries in Qwen3.5 DeltaNet CP#3003
hallerite wants to merge 1 commit into
mainfrom
fix/qwen3-5-deltanet-cp-packed-boundaries

Conversation

@hallerite

@hallerite hallerite commented Jul 11, 2026

Copy link
Copy Markdown
Member

Fixes #3002.

Under Ulysses CP the DeltaNet layer built its fla CP context from a fabricated single-sequence cu_seqlens ([0, global_seq_len]), so the gated delta rule carried recurrent state across packed documents. Independently, the causal convolution derived its seq_idx from shard-local cu_seqlens, resetting mid-document at every CP shard boundary — wrong even for unpacked rows.

The fix hands fla the information it already knows how to use:

  • _build_cp_context now uses the full pre-shard cu_seqlens published by setup_cp_params (same source as the Nemotron-H fix in fix: reset Nemotron-H CP state at packed boundaries #3001). fla derives per-rank boundaries, document-aware state passing, and conv halo sizes from them.
  • Under CP the convolution runs through fla's CP conv (fla.modules.conv.causal_conv1d with cp_context), which resets at true document boundaries and exchanges tail tokens with the previous rank for documents straddling the shard boundary, in both forward and backward.

The non-CP paths are untouched.

Validation

2-GPU (CP=2, bf16, real fla kernels, layer wired exactly as the trainer wires it), packed row [37, 51, 40] with doc 2 straddling the shard boundary, vs the trusted non-CP varlen path:

check before after
doc 2 max|CP − ref| (packed) 4.6e-01 7.8e-03 (bf16 noise)
single unpacked doc, shard-boundary tokens 4.6e-01 2.0e-03 (1 ulp)
doc 3 response to doc-2 tail perturbation, slow decay (must be 0) 7.8e-01 0.0

tests/unit/train/models/test_qwen3_5_moe.py passes (4 passed); ruff check/format clean.

🤖 Generated with Claude Code


Note

Medium Risk
Targets distributed-training correctness for linear-attention layers under Ulysses CP and packed batches; scope is limited to the DeltaNet CP path with non-CP behavior preserved.

Overview
Fixes incorrect recurrent state and convolution boundaries in Qwen3_5MoeGatedDeltaNet when Ulysses context parallelism is enabled with packed sequences.

_build_cp_context now feeds fla the full pre-shard cu_seqlens from ULYSSES_PARAMS (same pattern as Ulysses flash attention) instead of synthesizing [0, global_seq_len], so the gated delta rule no longer carries state across packed documents.

When CP is active, the causal conv path switches to fla’s CP causal_conv1d with the same cp_context, so resets happen at real document boundaries and halo tokens are exchanged for documents that span shard edges. Non-CP conv and attention fallbacks are unchanged.

Reviewed by Cursor Bugbot for commit 6f068ea. Bugbot is set up for automated code reviews on this repo. Configure here.

Under Ulysses CP the DeltaNet layer built its fla CP context from a
fabricated single-sequence cu_seqlens, so the gated delta rule carried
state across packed documents. The causal conv used shard-local
cu_seqlens, resetting mid-document at every shard boundary even for
unpacked rows.

Build the CP context from the full pre-shard cu_seqlens published by
setup_cp_params (fla derives per-rank boundaries and document-aware
state passing from them) and run the convolution through fla's CP conv,
which resets at true document boundaries and exchanges tail tokens
across ranks for straddling documents.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hallerite hallerite requested a review from samsja July 11, 2026 23:08
@hallerite hallerite marked this pull request as ready for review July 11, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen3.5 hybrid DeltaNet CP ignores packed-document boundaries

1 participant