Skip to content

[ROCm] Add Qwen3 Next FP8 QKV preparation - #5002

Draft
nholmber wants to merge 5 commits into
ROCm:mainfrom
nholmber:qwen3-fp8-mha
Draft

[ROCm] Add Qwen3 Next FP8 QKV preparation#5002
nholmber wants to merge 5 commits into
ROCm:mainfrom
nholmber:qwen3-fp8-mha

Conversation

@nholmber

@nholmber nholmber commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add the Qwen3 Next FP8 QKV preparation kernel and allow varlen FMHA callers to
forward precomputed FP8 descales.

The kernel combines Q/K RMSNorm, RoPE, output-gate extraction, and FP8
quantization while preserving BF16 model-visible outputs. It supports the
vLLM prequantized-QKV attention integration without requiring attention to
requantize Q/K/V.

Changes

  • Add the public Qwen3 Next FP8-QKV preparation wrapper.
  • Add the corresponding Triton kernel implementation.
  • Return stable scalar/per-head descale shapes.
  • Forward Q/K/V descales through the varlen FMHA wrapper.
  • Add correctness, mixed decode/extend, dtype, CPU-rejection, and FMHA ABI
    tests.

Dependency

The end-to-end model integration is in the matching vLLM draft:

vllm-project/vllm#53823

Duplicate-work check

No issue number was provided. Searches of open ROCm/AITER PRs for
Qwen3 Next FP8 QKV and FP8 descales varlen FMHA found no matching PR.

Validation

Post-rebase checks on current origin/main:

ruff check:
  passed

ruff format --check:
  passed

python bytecode compilation:
  passed

git diff --check:
  passed

Previously completed MI355 GPU validation for the unchanged feature code:

Qwen3 Next FP8 QKV GPU tests: passed
BF16-KV gather-to-FP8 integration tests: 2/2 passed

End-to-end validation with the paired vLLM branch:

long-context and packed coherence: passed
GSM8K: 97.57% flexible-extract
GPQA-Diamond: 92.42%
FP8 ASM FMHA dispatch: confirmed

AI assistance disclosure

This change was developed with OpenAI Codex assistance. The human submitter
must review every changed line and confirm they understand and can defend the
implementation before marking the PR ready.

Assisted-by: OpenAI Codex
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com>
Assisted-by: OpenAI Codex
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com>
Assisted-by: OpenAI Codex
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com>
Assisted-by: OpenAI Codex
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5002 --add-label <label>

PR title tags:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf] and op tags like [MLA] are left untouched. Add the no-auto-title label to opt this PR out of title tagging.

Assisted-by: OpenAI Codex
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant