fix: respect packed document boundaries in Qwen3.5 DeltaNet CP#3003
Open
hallerite wants to merge 1 commit into
Open
fix: respect packed document boundaries in Qwen3.5 DeltaNet CP#3003hallerite wants to merge 1 commit into
hallerite wants to merge 1 commit into
Conversation
Under Ulysses CP the DeltaNet layer built its fla CP context from a fabricated single-sequence cu_seqlens, so the gated delta rule carried state across packed documents. The causal conv used shard-local cu_seqlens, resetting mid-document at every shard boundary even for unpacked rows. Build the CP context from the full pre-shard cu_seqlens published by setup_cp_params (fla derives per-rank boundaries and document-aware state passing from them) and run the convolution through fla's CP conv, which resets at true document boundaries and exchanges tail tokens across ranks for straddling documents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3002.
Under Ulysses CP the DeltaNet layer built its fla CP context from a fabricated single-sequence
cu_seqlens([0, global_seq_len]), so the gated delta rule carried recurrent state across packed documents. Independently, the causal convolution derived itsseq_idxfrom shard-localcu_seqlens, resetting mid-document at every CP shard boundary — wrong even for unpacked rows.The fix hands fla the information it already knows how to use:
_build_cp_contextnow uses the full pre-shardcu_seqlenspublished bysetup_cp_params(same source as the Nemotron-H fix in fix: reset Nemotron-H CP state at packed boundaries #3001). fla derives per-rank boundaries, document-aware state passing, and conv halo sizes from them.fla.modules.conv.causal_conv1dwithcp_context), which resets at true document boundaries and exchanges tail tokens with the previous rank for documents straddling the shard boundary, in both forward and backward.The non-CP paths are untouched.
Validation
2-GPU (CP=2, bf16, real fla kernels, layer wired exactly as the trainer wires it), packed row
[37, 51, 40]with doc 2 straddling the shard boundary, vs the trusted non-CP varlen path:tests/unit/train/models/test_qwen3_5_moe.pypasses (4 passed); ruff check/format clean.🤖 Generated with Claude Code
Note
Medium Risk
Targets distributed-training correctness for linear-attention layers under Ulysses CP and packed batches; scope is limited to the DeltaNet CP path with non-CP behavior preserved.
Overview
Fixes incorrect recurrent state and convolution boundaries in
Qwen3_5MoeGatedDeltaNetwhen Ulysses context parallelism is enabled with packed sequences._build_cp_contextnow feeds fla the full pre-shardcu_seqlensfromULYSSES_PARAMS(same pattern as Ulysses flash attention) instead of synthesizing[0, global_seq_len], so the gated delta rule no longer carries state across packed documents.When CP is active, the causal conv path switches to fla’s CP
causal_conv1dwith the samecp_context, so resets happen at real document boundaries and halo tokens are exchanged for documents that span shard edges. Non-CP conv and attention fallbacks are unchanged.Reviewed by Cursor Bugbot for commit 6f068ea. Bugbot is set up for automated code reviews on this repo. Configure here.