From f08697e7c53c8750b94e432d7a23e72da3d131de Mon Sep 17 00:00:00 2001 From: realAsma Date: Wed, 16 Sep 2026 14:13:08 +0000 Subject: [PATCH] [Feature] Add Qwen3.8 local-Hessian PTQ recipes Signed-off-by: realAsma --- CHANGELOG.rst | 1 + .../nvfp4_default-kv_none-local_hessian.yaml | 27 ++++++++++ ...nvfp4_w4a4_mlp_fp8_attn_local_hessian.yaml | 33 ++++++++++++ .../ptq/nvfp4_w4a4_mlp_fp8_attn_max.yaml | 51 +++++++++++++++++++ modelopt_recipes/ptq.md | 13 ++++- 5 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 modelopt_recipes/general/ptq/nvfp4_default-kv_none-local_hessian.yaml create mode 100644 modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_local_hessian.yaml create mode 100644 modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_max.yaml diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 3a158bdb1ae..505e168aa6d 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -11,6 +11,7 @@ Changelog - Add ``layerwise.export_dir``: layerwise calibration writes each decoder layer to its own quantized checkpoint shard as it finishes, so no separate ``export_hf_checkpoint()`` pass is needed and, with ``layerwise.checkpoint_dir``, an interrupted run resumes without redoing finished layers. Calibration writes the layer shards; ``finalize()`` on the exporter left on the model adds the tail shard, the index and the config artifacts, and the checkpoint does not load until it runs. ``examples/hf_ptq`` does this for you. Supports FP8 and NVFP4 on single-process models, resident or offloaded, including multimodal models and models with MTP layers; other formats and placements raise ``NotImplementedError`` before calibration starts. - Add support for quantizing and calibrating enabled operators outside the transformer layers, such as ``lm_head``, when using layerwise calibration. - Add an end-to-end BEVFormer ONNX PTQ example with temporal calibration data generation, INT8 and FP8 quantization, TensorRT engine building, and nuScenes accuracy evaluation. See `examples/onnx_ptq/bevformer/README.md `_ for details. +- Add a reusable local-Hessian NVFP4 PTQ recipe and the quantization recipe used for ``nvidia/Qwen3.8-27B-NVFP4``. *Misc* diff --git a/modelopt_recipes/general/ptq/nvfp4_default-kv_none-local_hessian.yaml b/modelopt_recipes/general/ptq/nvfp4_default-kv_none-local_hessian.yaml new file mode 100644 index 00000000000..abf9745cf83 --- /dev/null +++ b/modelopt_recipes/general/ptq/nvfp4_default-kv_none-local_hessian.yaml @@ -0,0 +1,27 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# PTQ recipe wrapping the shipped NVFP4 W4A4 local-Hessian preset. + +imports: + preset: configs/ptq/presets/model/nvfp4_w4a4_weight_local_hessian + +metadata: + recipe_type: ptq + description: >- + Applies NVFP4 W4A4 with static weight scales, dynamic inputs, KV quantizers disabled, and + local-Hessian weight calibration with an FP8-scale sweep. +quantize: + $import: preset diff --git a/modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_local_hessian.yaml b/modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_local_hessian.yaml new file mode 100644 index 00000000000..e007fe5d464 --- /dev/null +++ b/modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_local_hessian.yaml @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# modelopt-schema: modelopt.recipe.config.ModelOptPTQRecipe +imports: + base: models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_max + +metadata: + recipe_type: ptq + description: >- + Qwen3.8-27B local-Hessian PTQ using the NVFP4 W4A4 and FP8 W8A8 assignment exported by + the 5.5-bit NVFP4-max AutoQuantize sweep. + +quantize: + $import: base + algorithm: + method: local_hessian + fp8_scale_sweep: true + layerwise: + enable: true + get_qdq_activations_from_prev_layer: true diff --git a/modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_max.yaml b/modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_max.yaml new file mode 100644 index 00000000000..a71a4573818 --- /dev/null +++ b/modelopt_recipes/models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_max.yaml @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# QuantizeConfig for the precision assignment exported by a 5.5-bit NVFP4-max AutoQuantize sweep. + +# modelopt-schema: modelopt.torch.quantization.config.QuantizeConfig +imports: + base_disable_all: configs/ptq/units/base_disable_all + fp8: configs/numerics/fp8 + nvfp4: configs/numerics/nvfp4 + nvfp4_static: configs/numerics/nvfp4_static + +algorithm: max +quant_cfg: + - $import: base_disable_all + + - quantizer_name: '*language_model.layers.*.mlp.*.weight_quantizer' + cfg: {$import: nvfp4_static} + - quantizer_name: '*language_model.layers.*.mlp.*.input_quantizer' + cfg: {$import: nvfp4} + + - quantizer_name: '*language_model.layers.*.self_attn.*_proj.weight_quantizer' + cfg: {$import: fp8} + - quantizer_name: '*language_model.layers.*.self_attn.*_proj.input_quantizer' + cfg: {$import: fp8} + - quantizer_name: '*language_model.layers.*.linear_attn.*_proj*.weight_quantizer' + cfg: {$import: fp8} + - quantizer_name: '*language_model.layers.*.linear_attn.*_proj*.input_quantizer' + cfg: {$import: fp8} + + - quantizer_name: '*lm_head.weight_quantizer' + cfg: {$import: nvfp4_static} + - quantizer_name: '*lm_head.input_quantizer' + cfg: {$import: nvfp4} + + - quantizer_name: '*linear_attn.in_proj_a*' + enable: false + - quantizer_name: '*linear_attn.in_proj_b*' + enable: false diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 963889e98aa..31902f0b076 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -28,7 +28,7 @@ supported combinations. ### The shipped recipes
-All 25 general/ptq/ recipes (click to expand) +All 26 general/ptq/ recipes (click to expand) | Recipe | Model body | KV cache | Calibration | |--------|-----------|----------|-------------| @@ -38,6 +38,7 @@ supported combinations. | `nvfp4_default-kv_fp8_cast` | NVFP4 W4A4, all linears | FP8 (constant amax) | max | | `nvfp4_act_headroom-kv_fp8_cast` | NVFP4 W4A4, all linears | FP8 (constant amax) | nvfp4_act_headroom | | `nvfp4_default-kv_nvfp4_cast` | NVFP4 W4A4, all linears | NVFP4 (constant amax) | max | +| `nvfp4_default-kv_none-local_hessian` | NVFP4 W4A4 (static W), all linears | none | local Hessian + FP8 sweep | | `nvfp4_default-kv_none-gptq` | NVFP4 W4A4 (static W), all linears | none | GPTQ (layerwise) | | `nvfp4_mlp_only-kv_fp8` | NVFP4 W4A4, MLP + MoE experts | FP8 (calibrated) | max | | `nvfp4_mlp_only-novit-kv_fp8` | NVFP4 W4A4, MLP + MoE experts (VL vision tower excluded) | FP8 (calibrated) | max | @@ -171,6 +172,11 @@ How the quantization scales are searched. The default (no suffix) is `max`. (amax) calibrated as in the default recipes. Costs more calibration time but recovers accuracy NVFP4 W4A4 can lose under plain max. Reach for it when a `max` recipe regresses. +- **`local_hessian`** (`nvfp4_default-kv_none-local_hessian`) — searches static + NVFP4 weight scales by minimizing output reconstruction error using a local + Hessian approximation, with an FP8-scale sweep. It can recover accuracy when + plain max calibration regresses, but currently supports only single-rank + calibration. - **`input_scale1`** (`nvfp4_experts_only_input_scale1-kv_fp8_cast`) — pins the expert **activation** per-tensor amax to a constant `2688.0` (= E2M1_MAX × E4M3_MAX = 6 × 448) via `constant_amax`, so the exported NVFP4 @@ -393,6 +399,11 @@ checkpoint's** quant config verbatim: `nvidia/Mistral-Medium-3.5-128B-NVFP4`: decoder MLP layers 4–86 use NVFP4 W4A4, edge MLP layers 0–3 and 87 use FP8 W8A8, and all attention projections and the KV cache use FP8. It uses max calibration. +- **`models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_local_hessian`** + reproduces NVFP4 W4A4 mixed precision quantization recipe used for `nvidia/Qwen3.8-27B-NVFP4`: + MLP projections and `lm_head` use NVFP4 W4A4, self-attention and + linear-attention projections use FP8 W8A8. Static NVFP4 weight + scales are calibrated with local-Hessian algorithm. - **`models/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse`** mirrors `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` exactly — a hybrid **Mamba-MoE** with a hand-mapped, **per-component** precision scheme: