[WIP] Add Qwen4Exp text model support

[alifurkanstahl]

Warning

Work in progress — do not merge yet.

This draft is available early so the Qwen4Exp kernels can already be tested
and used while the implementation is still evolving. APIs and implementation
details may still change before this PR leaves draft.

Summary

Adds Liger Kernel support for the Qwen4Exp text stack.

Upstream Qwen4Exp implementation: huggingface/transformers#48337

This PR:

• adds Qwen4Exp-specific Triton paths for HyperConnection pre, GRWrite, EOS-aware n-gram hashing, grouped RMSNorm multi-consumer backward, and grouped RMSNorm + Write4;
• integrates the optimized Qwen4Exp gated-residual / decoder path while preserving Hugging Face behavior;
• adds FusedLinearCrossEntropy support and AutoLiger dispatch for Qwen4Exp composite configs;
• adds correctness, monkey-patch, convergence, and standardized operator benchmarks.

Scope

This PR currently targets the Qwen4Exp text stack.

Follow-ups before leaving draft

• add RoPE support (currently uses the native Hugging Face RoPE path);
• add full Engram kernel coverage beyond the n-gram/hash path included here;
• generalize the GRWrite / HyperConnection kernels beyond hc_count=4;
• add vision / multimodal support.

Performance

Measured on an NVIDIA GeForce RTX 5060 Ti, BF16, B=1, S=2048, H=2048, hc_count=4.

Values below are median timings from the standardized Liger benchmark harness.

Speed

Operation	Mode	Reference	Baseline	Liger	Speedup
GRWrite	fwd+bwd	PyTorch ref	1.345 ms	0.387 ms	3.48x
HyperConnection pre	fwd+bwd	PyTorch ref	1.423 ms	0.590 ms	2.41x
Gated Residual read/write boundary	fwd+bwd	HF Qwen4Exp	11.677 ms	3.000 ms	3.89x
EOS-aware n-gram hash	forward	PyTorch ref	0.272 ms	0.0125 ms	21.71x
Grouped RMS multi-consumer	fwd+bwd	Liger RMSNorm	0.950 ms	0.618 ms	1.54x
Grouped RMS + Write4	fwd+bwd	Liger RMSNorm	1.138 ms	0.624 ms	1.82x

Peak memory

Operation	Reference	Baseline	Liger	Memory saved
GRWrite	PyTorch ref	216.06 MB	152.05 MB	29.63%
HyperConnection pre	PyTorch ref	272.00 MB	208.00 MB	23.53%
Gated Residual read/write boundary	HF Qwen4Exp	548.41 MB	295.25 MB	46.16%
EOS-aware n-gram hash	PyTorch ref	0.728 MB	0.268 MB	63.25%
Grouped RMS multi-consumer	Liger RMSNorm	225.22 MB	193.22 MB	14.21%
Grouped RMS + Write4	Liger RMSNorm	209.63 MB	177.64 MB	15.26%

The Gated Residual benchmark covers the complete residual read/write boundary around an arbitrary attention/MLP result; it intentionally excludes the intervening attention/MLP block compute.

The grouped-RMS rows compare the Qwen4Exp-specialized fused paths against the generic Liger grouped RMSNorm path rather than against PyTorch.

These are operator microbenchmarks, not end-to-end model training speedups.

Full benchmark results are included in benchmark/data/all_benchmark_data.csv.

Testing Done

• Hardware Type: NVIDIA GeForce RTX 5060 Ti

• Python: 3.13.9

• PyTorch: 2.10.0+cu128

• Transformers: 5.16.1

• Triton: 3.6.0

• run make test to ensure correctness

• run make checkstyle to ensure code style

• run make test-convergence to ensure convergence

All Qwen4Exp convergence cases in the suite pass.

make test-convergence currently exits non-zero because of four failures unrelated to Qwen4Exp:

• fp32 mini_qwen3_5_moe multimodal — top-k logprob mismatch;
• bf16 mini_qwen3_5_moe multimodal — top-k logprob mismatch;
• bf16 mini_llama4 with materialized logits;
• bf16 mini_qwen3_moe with materialized logits.

All four failures were reproduced on base commit 6459426 using the same
Python / PyTorch / Transformers / Triton environment and the same isolated
pytest cases.

The base and this branch reproduce the same failure signatures. The fp32
Qwen3.5-MoE case has minor run-to-run numerical variation while retaining the
same mismatch count and indices; the other checked signatures are deterministic.

These failures therefore appear to be pre-existing and are not introduced by
the Qwen4Exp changes.

Test output