Skip to content

[Refactor] Pass GLM DSA top-k IDs explicitly - #1978

Open
jayhenry wants to merge 1 commit into
InternLM:mainfrom
jayhenry:unify_topk_offload
Open

[Refactor] Pass GLM DSA top-k IDs explicitly#1978
jayhenry wants to merge 1 commit into
InternLM:mainfrom
jayhenry:unify_topk_offload

Conversation

@jayhenry

@jayhenry jayhenry commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Pass contiguous int32 dsa_topk_ids explicitly through GLM decoder layers and MTP inputs/results, removing the SequenceContext cache and manual cross-layer lifecycle runtime.
  • Use one saved-tensor offload window per decoder call for hidden states and the final shared-layer top-k IDs, with prefetch and reserved pinned buffers; MTP top-k offload remains excluded.
  • Keep sharing topology stateless and add coverage for explicit top-k propagation, checkpointing, SP, EP, micro-batches, and MTP.

Validation

  • Full pre-commit suite: docformatter, pyupgrade, ruff, ruff-format, mypy, and pydantic-extra-check.
  • Relevant unit/integration regressions passed for DSA Torch/TileLang/cuDNN backends, SP2, checkpoint, MTP shared logical depth, EP2 micro2, the optimized 8-GPU engine path, and generic non-DSA MoE.
  • 16K, 8xGPU AdamW: rank-0 effective TGS 10,893.7 vs 10,957.3 baseline (-0.58%; below the 11K absolute target); peak allocated memory 104.69 vs 105.26 GB.
  • 32K, 8xGPU Muon SP2+EP2+micro2: full-rank sequence TGS 1,769.0 vs 1,599.3 baseline (+10.61%); peak allocated memory 101.87 vs 102.29 GB; VmPin 42.87 GiB in both runs.

Follow-ups

  • Use selective checkpoint recompute to keep the top-k indexer out of replay recomputation.
  • Evaluate fullgraph=True separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant