From a6f472e8c0b5cafe89c8464646fc1759d0588a20 Mon Sep 17 00:00:00 2001 From: Gwenaelle Cunha Sergio Date: Fri, 17 Jul 2026 19:10:33 +0000 Subject: [PATCH 1/6] Clarify ONNX autotune guidance Signed-off-by: Gwenaelle Cunha Sergio --- docs/source/guides/9_autotune.rst | 15 ++++++++++++--- docs/source/guides/_onnx_quantization.rst | 12 ++++++++++++ examples/onnx_ptq/README.md | 3 ++- examples/onnx_ptq/autotune/README.md | 12 ++++++++++++ modelopt/onnx/quantization/autotune/__main__.py | 7 ++++++- 5 files changed, 44 insertions(+), 5 deletions(-) diff --git a/docs/source/guides/9_autotune.rst b/docs/source/guides/9_autotune.rst index 583dfcb6ee8..4c7f7b4d172 100644 --- a/docs/source/guides/9_autotune.rst +++ b/docs/source/guides/9_autotune.rst @@ -2,6 +2,12 @@ Autotune (ONNX) =============================================== +.. warning:: + + Direct Autotune is an advanced ONNX quantization subtool for optimizing Q/DQ placement using TensorRT latency measurements. It does not replace the full calibrated ONNX quantization workflow. + + To quantize an ONNX model taking into consideration accuracy, please use ``python -m modelopt.onnx.quantization ... --autotune=`` with representative calibration data. See the `ONNX quantization Autotune options <_onnx_quantization.html#python-m-modelopt.onnx.quantization-autotune-only-applicable-when-autotune-is-set>`_. + .. contents:: Table of Contents :local: :depth: 2 @@ -22,9 +28,10 @@ The ``modelopt.onnx.quantization.autotune`` module automates Q/DQ (Quantize/Dequ **When to Use This Tool:** -* Quantizing an ONNX model for TensorRT deployment -* Optimizing Q/DQ placement for best performance -* The model has repeating structures (e.g., transformer blocks, ResNet layers) +* Debugging or developing the Q/DQ placement autotuning algorithm +* Running the lower-level workflow without invoking the full quantization CLI +* Programmatic experiments with direct Autotune classes and workflow functions +* Expert workflows that intentionally start from already-quantized or pre-patterned Q/DQ models Quick Start =========== @@ -54,6 +61,8 @@ The command will: 4. Select the best scheme based on TensorRT latency measurements 5. Export an optimized ONNX model with Q/DQ nodes +Autotune searches for Q/DQ placement schemes that improve TensorRT runtime. It does not by itself define the full calibration and quantization policy for an accuracy-sensitive deployment. For end-to-end ONNX PTQ that starts from an unquantized model, run ONNX quantization with calibration data and enable ``--autotune`` there. See the `ONNX quantization Autotune options <_onnx_quantization.html#python-m-modelopt.onnx.quantization-autotune-only-applicable-when-autotune-is-set>`_. + **Output Files:** Files are written under the output directory (default ``./autotuner_output``, or the path given by ``--output_dir``): diff --git a/docs/source/guides/_onnx_quantization.rst b/docs/source/guides/_onnx_quantization.rst index 8a761adfa8f..1c836630976 100644 --- a/docs/source/guides/_onnx_quantization.rst +++ b/docs/source/guides/_onnx_quantization.rst @@ -74,6 +74,18 @@ Call PTQ function quantize_mode="int8", ) +To optimize Q/DQ placement as part of the calibrated ONNX PTQ workflow, enable Autotune on the quantization command or API call. Prefer this path over direct ``modelopt.onnx.quantization.autotune`` when starting from an unquantized model and validating accuracy. + +.. code-block:: python + + moq.quantize( + ... + # Optional: increases quantization time, but should produce more optimal + # Q/DQ node placements. This uses the default Autotune settings; tune + # those with the autotune_* arguments below when needed. + autotune=True, + ) + Alternatively, you can call PTQ function in command line: .. argparse:: diff --git a/examples/onnx_ptq/README.md b/examples/onnx_ptq/README.md index 9e83ec16399..52ccd28a5c7 100644 --- a/examples/onnx_ptq/README.md +++ b/examples/onnx_ptq/README.md @@ -210,7 +210,8 @@ trtexec --onnx=/tmp/identity_neural_network.quant.onnx \ ### Optimize Q/DQ node placement with Autotune This feature automates Q/DQ (Quantize/Dequantize) node placement optimization for ONNX models using TensorRT performance measurements. -For more information on the standalone toolkit, please refer to [autotune](./autotune). +For accuracy-sensitive ONNX PTQ that starts from an unquantized model, prefer enabling Autotune in this quantization workflow with representative calibration data and validate model accuracy after quantization. +For lower-level Q/DQ placement workflows such as direct pattern cache experiments, `qdq_baseline` import, or custom benchmarking, refer to the [Autotune guide](https://nvidia.github.io/Model-Optimizer/guides/9_autotune.html). To access this feature in the ONNX quantization workflow, simply add `--autotune` in your CLI: diff --git a/examples/onnx_ptq/autotune/README.md b/examples/onnx_ptq/autotune/README.md index 443c91d2877..6dcd3a8cc9e 100644 --- a/examples/onnx_ptq/autotune/README.md +++ b/examples/onnx_ptq/autotune/README.md @@ -2,6 +2,18 @@ This example demonstrates automated Q/DQ (Quantize/Dequantize) node placement optimization for ONNX models using TensorRT performance measurements. +> **Warning:** This example uses the direct Autotune entry point for lower-level Q/DQ placement experiments. If you are starting from an unquantized ONNX model and care about **accuracy**, please use the ONNX PTQ workflow with `--autotune` enabled and with representative calibration data: +> +> ```bash +> python -m modelopt.onnx.quantization \ +> --onnx_path=model.onnx \ +> --quantize_mode= \ +> --calibration_data_path=calib.npy \ +> --calibration_method= \ +> --output_path=model.quant.onnx \ +> --autotune= +> ``` + ## Table of Contents
diff --git a/modelopt/onnx/quantization/autotune/__main__.py b/modelopt/onnx/quantization/autotune/__main__.py index 268b6251414..db50ab46bcf 100644 --- a/modelopt/onnx/quantization/autotune/__main__.py +++ b/modelopt/onnx/quantization/autotune/__main__.py @@ -154,7 +154,12 @@ def get_parser() -> argparse.ArgumentParser: """ parser = argparse.ArgumentParser( prog="modelopt.onnx.quantization.autotune", - description="ONNX Q/DQ Autotuning with TensorRT", + description=( + "Advanced ONNX Q/DQ placement autotuning with TensorRT latency measurements. " + "For end-to-end ONNX post-training quantization that starts from an unquantized " + "model and uses calibration data, use " + "`python -m modelopt.onnx.quantization ... --autotune=`." + ), formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: From 214f1af86d7d0577cf77eb571ac04d8af559500f Mon Sep 17 00:00:00 2001 From: Gwenaelle Cunha Sergio Date: Fri, 17 Jul 2026 19:31:08 +0000 Subject: [PATCH 2/6] Clarify autotune example warning Signed-off-by: Gwenaelle Cunha Sergio --- examples/onnx_ptq/autotune/README.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/examples/onnx_ptq/autotune/README.md b/examples/onnx_ptq/autotune/README.md index 6dcd3a8cc9e..d8a86b8bae0 100644 --- a/examples/onnx_ptq/autotune/README.md +++ b/examples/onnx_ptq/autotune/README.md @@ -2,17 +2,7 @@ This example demonstrates automated Q/DQ (Quantize/Dequantize) node placement optimization for ONNX models using TensorRT performance measurements. -> **Warning:** This example uses the direct Autotune entry point for lower-level Q/DQ placement experiments. If you are starting from an unquantized ONNX model and care about **accuracy**, please use the ONNX PTQ workflow with `--autotune` enabled and with representative calibration data: -> -> ```bash -> python -m modelopt.onnx.quantization \ -> --onnx_path=model.onnx \ -> --quantize_mode= \ -> --calibration_data_path=calib.npy \ -> --calibration_method= \ -> --output_path=model.quant.onnx \ -> --autotune= -> ``` +> **Warning:** This example uses the direct Autotune entry point for lower-level Q/DQ placement experiments. If you are starting from an unquantized ONNX model and care about **accuracy**, please use the ONNX PTQ workflow with `--autotune` enabled and with representative calibration data. See [../README#optimize-qdq-node-placement-with-autotune](../README#optimize-qdq-node-placement-with-autotune). ## Table of Contents From aed1116931f7ad3e6d033fad3c15245ef00bcf5f Mon Sep 17 00:00:00 2001 From: Gwenaelle Cunha Sergio Date: Fri, 17 Jul 2026 19:43:05 +0000 Subject: [PATCH 3/6] Refine ONNX autotune wording Signed-off-by: Gwenaelle Cunha Sergio --- docs/source/guides/_onnx_quantization.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/source/guides/_onnx_quantization.rst b/docs/source/guides/_onnx_quantization.rst index 1c836630976..e4d0c2d93d6 100644 --- a/docs/source/guides/_onnx_quantization.rst +++ b/docs/source/guides/_onnx_quantization.rst @@ -74,15 +74,13 @@ Call PTQ function quantize_mode="int8", ) -To optimize Q/DQ placement as part of the calibrated ONNX PTQ workflow, enable Autotune on the quantization command or API call. Prefer this path over direct ``modelopt.onnx.quantization.autotune`` when starting from an unquantized model and validating accuracy. +Optionally enable Autotune for more optimized Q/DQ placement. Note that this will likely increase the time required to calibrate the model. .. code-block:: python moq.quantize( ... - # Optional: increases quantization time, but should produce more optimal - # Q/DQ node placements. This uses the default Autotune settings; tune - # those with the autotune_* arguments below when needed. + # Default Autotune settings, can be tuned with the autotune_* arguments below. autotune=True, ) From 10b8ba07d9b7a72665498cb7d5953e6f09066e8e Mon Sep 17 00:00:00 2001 From: Gwenaelle Cunha Sergio Date: Fri, 17 Jul 2026 19:50:37 +0000 Subject: [PATCH 4/6] Refine autotune CLI description Signed-off-by: Gwenaelle Cunha Sergio --- modelopt/onnx/quantization/autotune/__main__.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/modelopt/onnx/quantization/autotune/__main__.py b/modelopt/onnx/quantization/autotune/__main__.py index db50ab46bcf..68ae101d5ee 100644 --- a/modelopt/onnx/quantization/autotune/__main__.py +++ b/modelopt/onnx/quantization/autotune/__main__.py @@ -155,9 +155,8 @@ def get_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="modelopt.onnx.quantization.autotune", description=( - "Advanced ONNX Q/DQ placement autotuning with TensorRT latency measurements. " - "For end-to-end ONNX post-training quantization that starts from an unquantized " - "model and uses calibration data, use " + "ONNX Q/DQ placement autotuning with TensorRT latency measurements. " + "For end-to-end ONNX quantization, use " "`python -m modelopt.onnx.quantization ... --autotune=`." ), formatter_class=argparse.RawDescriptionHelpFormatter, From 69a52f52a92bd0eaf11575c534eca2c9625d07db Mon Sep 17 00:00:00 2001 From: Gwenaelle Cunha Sergio Date: Fri, 17 Jul 2026 19:52:27 +0000 Subject: [PATCH 5/6] Tighten ONNX autotune guidance Signed-off-by: Gwenaelle Cunha Sergio --- examples/onnx_ptq/README.md | 5 ++--- modelopt/onnx/quantization/autotune/__main__.py | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/examples/onnx_ptq/README.md b/examples/onnx_ptq/README.md index 52ccd28a5c7..5183011dbd0 100644 --- a/examples/onnx_ptq/README.md +++ b/examples/onnx_ptq/README.md @@ -210,8 +210,6 @@ trtexec --onnx=/tmp/identity_neural_network.quant.onnx \ ### Optimize Q/DQ node placement with Autotune This feature automates Q/DQ (Quantize/Dequantize) node placement optimization for ONNX models using TensorRT performance measurements. -For accuracy-sensitive ONNX PTQ that starts from an unquantized model, prefer enabling Autotune in this quantization workflow with representative calibration data and validate model accuracy after quantization. -For lower-level Q/DQ placement workflows such as direct pattern cache experiments, `qdq_baseline` import, or custom benchmarking, refer to the [Autotune guide](https://nvidia.github.io/Model-Optimizer/guides/9_autotune.html). To access this feature in the ONNX quantization workflow, simply add `--autotune` in your CLI: @@ -225,7 +223,8 @@ python -m modelopt.onnx.quantization \ --autotune= ``` -For more fine-tuned Autotune flags, please refer to the [API guide](https://nvidia.github.io/Model-Optimizer/guides/_onnx_quantization.html). +For more fine-tuned Autotune flags, please refer to the [API guide](https://nvidia.github.io/Model-Optimizer/guides/_onnx_quantization.html) and the [Autotune guide](https://nvidia.github.io/Model-Optimizer/guides/9_autotune.html). + ## Resources diff --git a/modelopt/onnx/quantization/autotune/__main__.py b/modelopt/onnx/quantization/autotune/__main__.py index 68ae101d5ee..ccb826c291b 100644 --- a/modelopt/onnx/quantization/autotune/__main__.py +++ b/modelopt/onnx/quantization/autotune/__main__.py @@ -156,8 +156,9 @@ def get_parser() -> argparse.ArgumentParser: prog="modelopt.onnx.quantization.autotune", description=( "ONNX Q/DQ placement autotuning with TensorRT latency measurements. " - "For end-to-end ONNX quantization, use " - "`python -m modelopt.onnx.quantization ... --autotune=`." + "For end-to-end ONNX quantization taking into consideration accuracy, use " + "`python -m modelopt.onnx.quantization ... --autotune=` " + "with representative calibration data." ), formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" From 1ceaf68866d38b4b6409bc0f1db945e82c0a3793 Mon Sep 17 00:00:00 2001 From: Gwenaelle Cunha Sergio Date: Mon, 20 Jul 2026 14:36:56 +0000 Subject: [PATCH 6/6] Fix ONNX PTQ README formatting Signed-off-by: Gwenaelle Cunha Sergio --- examples/onnx_ptq/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/examples/onnx_ptq/README.md b/examples/onnx_ptq/README.md index 5183011dbd0..24d3d97b410 100644 --- a/examples/onnx_ptq/README.md +++ b/examples/onnx_ptq/README.md @@ -225,7 +225,6 @@ python -m modelopt.onnx.quantization \ For more fine-tuned Autotune flags, please refer to the [API guide](https://nvidia.github.io/Model-Optimizer/guides/_onnx_quantization.html) and the [Autotune guide](https://nvidia.github.io/Model-Optimizer/guides/9_autotune.html). - ## Resources - 📅 [Roadmap](https://github.com/NVIDIA/Model-Optimizer/issues/1699)