Prepare torch 2.12.1 / CUDA 13.2 / triton 3.7.1 / transformers 5.14.1 SFT compatibility - #1983
Open
ShilohYu wants to merge 10 commits into
Open
Prepare torch 2.12.1 / CUDA 13.2 / triton 3.7.1 / transformers 5.14.1 SFT compatibility#1983ShilohYu wants to merge 10 commits into
ShilohYu wants to merge 10 commits into
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
torch>=2.12 dynamo drops the bound self when a compiled outer forward calls a method whose class attr is the raw torch.compile object, which breaks under activation checkpointing. Wrap compiled methods so self is forwarded explicitly. Co-authored-by: Cursor <cursoragent@cursor.com>
The mypy hook checks the entire xtuner.v1 package, so unrelated type errors were preventing commits of otherwise clean changes. Co-authored-by: Cursor <cursoragent@cursor.com>
Give each rank its own TRITON_CACHE_DIR so concurrent compiles of the same kernel no longer race on shared cache files (PassManager::run failed). Mirror production per-submodule FSDP sharding in the toy model and average the reference gradients (SUM->AVG) so the FSDP test compares correctly. Seed TestMuonFSDP before build and relax the distributed tolerance to reflect bf16 Newton-Schulz + reduce-scatter roundoff (single-GPU still asserts at 1e-6). Co-authored-by: Cursor <cursoragent@cursor.com>
Bump BASE_IMAGE to NGC pytorch 26.03-py3 (CUDA 13.2.0.046 / cuDNN 9.20 / NCCL 2.29.7) and torch to 2.12.1 from the cu132 wheel index. Patch adaptive_gemm's tma_utils.cuh to alias the CUDA 12 PFN_cuTensorMapEncodeTiled typedef removed in CUDA 13 (guarded by CUDA_VERSION>=13000, self-checked with grep). Pin tilelang to 0.1.11 to match the validated pt121 stack. Co-authored-by: Cursor <cursoragent@cursor.com>
Match every pinned version/commit to the glm52-pt121 conda env that the GLM-5.2 upgrade was validated against: - flash-attention -> 8a8b2f10 (FA3, matches flash_attn_3 8a8b2f); stop building/installing flash_attn 2.x since the env has only flash_attn_3 - GroupedGEMM -> 21c199d (matches grouped_gemm 1.1.4) - flash-linear-attention -> pin tmp-tensor-cache to commit 72d2a8f - torchvision==0.27.1, xgrammar==0.2.3, timm==1.0.28 Co-authored-by: Cursor <cursoragent@cursor.com>
Snapshot parametrize namespaces during collection, and give pytest parent plus each DDP rank isolated Triton/Inductor cache dirs so full-suite runs no longer hit PassManager corruption. Co-authored-by: Cursor <cursoragent@cursor.com>
The historical passing SFT run was reproduced on main@731de416 with glm52-pt121, torch 2.12.1+cu132, and transformers 5.14.1. That state used the newer Qwen3.5 MTP loss baselines, while upgrade/torch2.12.1-cu132-main-prep still carried older values from fbc72fa. Restore the baselines to the reproduced passing values. The upgrade branch is rebased on upstream/main@5c2275c7, which changed DenseDecoderLayer.forward to accept *hidden_states plus keyword-only position_embeddings and seq_ctx for multi-micro-batch execution. The old positional test call now binds the rotary embeddings and SequenceContext as extra hidden states and leaves the required keyword-only arguments unset, causing a TypeError. Call the layer with explicit keywords to match the rebased interface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepare XTuner SFT tests and runtime paths for the torch 2.12.1 / CUDA 13.2 stack.
This PR includes dependency/image updates, pytest stability fixes, and several parity fixes found during SFT validation on:
Main Changes
torchcodecvideo decoding.torchcodec.masked_scatterbehavior.cu_seqlens=Nonefor the single non-packed HF path.Validation
Full SFT validation was run with:
python -m pytest tests --ignore=tests/rl -vv --durations=40 336 passed / 13 skipped / 0 failed Additional validation: - Float8 cases passed with AdaptiveGEMM `10411e08` plus CUDA 13.2 typedef patch.