qwen3 muon fork - #4779

Open

Shuwen-Fang wants to merge 12 commits into

AI-Hypercomputer:mainfrom

Shuwen-Fang:shuwen-qwen3-muon

Commits on Aug 5, 2026

Update qwen3-next-80b-a3b.yml config parameters
darisoy
committed
cbcaaeb
Integrate mHC into Qwen3NextDecoderLayer (mhc_expansion_rate=4)
darisoy
committed
00302b2
Implement Newton-Schulz log-depth custom VJP triangular matrix solver for GDN (PR #4577 / cl/955390793)
darisoy
committed
0360d02
Implement Hybrid GDN v3 Tokamax forward + Newton-Schulz Custom VJP backward (with shard_map sharding support)
darisoy
committed
4960fea
feat: implement Muon optimizer support for Qwen3-Next architecture (RoutedMoE and GatedDeltaNet)
darisoy
committed
78d09f4
add scan support for qwen3-next model

NuojCheng
authored and
darisoy
committed
1476823

Commits on Aug 7, 2026

add muon impl
Shuwen-Fang
committed
c5ef832
integrate muon
Shuwen-Fang
committed
056abb1
use adamw instead of muon for expert weights
Shuwen-Fang
committed
a9a6b33
explicitly_weight_ag for shard exp on fsdp
Shuwen-Fang
committed
6506eba
update default nesterov style
Shuwen-Fang
committed
36efee5

Commits on Aug 8, 2026

apply muon to expert weights and update optimizer config
Shuwen-Fang
committed
589cd74