Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions docs/source/guides/9_autotune.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
Autotune (ONNX)
===============================================

.. warning::

Direct Autotune is an advanced ONNX quantization subtool for optimizing Q/DQ placement using TensorRT latency measurements. It does not replace the full calibrated ONNX quantization workflow.

To quantize an ONNX model taking into consideration accuracy, please use ``python -m modelopt.onnx.quantization ... --autotune=<quick|default|extensive>`` with representative calibration data. See the `ONNX quantization Autotune options <_onnx_quantization.html#python-m-modelopt.onnx.quantization-autotune-only-applicable-when-autotune-is-set>`_.

.. contents:: Table of Contents
:local:
:depth: 2
Expand All @@ -22,9 +28,10 @@ The ``modelopt.onnx.quantization.autotune`` module automates Q/DQ (Quantize/Dequ

**When to Use This Tool:**

* Quantizing an ONNX model for TensorRT deployment
* Optimizing Q/DQ placement for best performance
* The model has repeating structures (e.g., transformer blocks, ResNet layers)
* Debugging or developing the Q/DQ placement autotuning algorithm
* Running the lower-level workflow without invoking the full quantization CLI
* Programmatic experiments with direct Autotune classes and workflow functions
* Expert workflows that intentionally start from already-quantized or pre-patterned Q/DQ models

Quick Start
===========
Expand Down Expand Up @@ -54,6 +61,8 @@ The command will:
4. Select the best scheme based on TensorRT latency measurements
5. Export an optimized ONNX model with Q/DQ nodes

Autotune searches for Q/DQ placement schemes that improve TensorRT runtime. It does not by itself define the full calibration and quantization policy for an accuracy-sensitive deployment. For end-to-end ONNX PTQ that starts from an unquantized model, run ONNX quantization with calibration data and enable ``--autotune`` there. See the `ONNX quantization Autotune options <_onnx_quantization.html#python-m-modelopt.onnx.quantization-autotune-only-applicable-when-autotune-is-set>`_.

Comment on lines +64 to +65

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reconcile the calibration guidance in this guide.

This new guidance correctly directs accuracy-sensitive users to calibrated PTQ with --autotune, but the FAQ at Lines 855-858 still says calibration can be performed separately after Direct Autotune. Update the FAQ so the guide presents one consistent workflow.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/source/guides/9_autotune.rst` around lines 64 - 65, Update the FAQ entry
near the Direct Autotune calibration guidance to align with the workflow
described in the new introduction: for accuracy-sensitive end-to-end ONNX PTQ,
perform calibration during ONNX quantization and enable --autotune, rather than
recommending calibration separately afterward. Remove or revise the conflicting
separate-calibration statement while preserving the surrounding FAQ content.

**Output Files:**

Files are written under the output directory (default ``./autotuner_output``, or the path given by ``--output_dir``):
Expand Down
10 changes: 10 additions & 0 deletions docs/source/guides/_onnx_quantization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,16 @@ Call PTQ function
quantize_mode="int8",
)

Optionally enable Autotune for more optimized Q/DQ placement. Note that this will likely increase the time required to calibrate the model.

.. code-block:: python

moq.quantize(
...
# Default Autotune settings, can be tuned with the autotune_* arguments below.
autotune=True,
)

Alternatively, you can call PTQ function in command line:

.. argparse::
Expand Down
3 changes: 1 addition & 2 deletions examples/onnx_ptq/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,6 @@ trtexec --onnx=/tmp/identity_neural_network.quant.onnx \
### Optimize Q/DQ node placement with Autotune

This feature automates Q/DQ (Quantize/Dequantize) node placement optimization for ONNX models using TensorRT performance measurements.
For more information on the standalone toolkit, please refer to [autotune](./autotune).

To access this feature in the ONNX quantization workflow, simply add `--autotune` in your CLI:

Expand All @@ -224,7 +223,7 @@ python -m modelopt.onnx.quantization \
--autotune=<quick,default,extensive>
```

For more fine-tuned Autotune flags, please refer to the [API guide](https://nvidia.github.io/Model-Optimizer/guides/_onnx_quantization.html).
For more fine-tuned Autotune flags, please refer to the [API guide](https://nvidia.github.io/Model-Optimizer/guides/_onnx_quantization.html) and the [Autotune guide](https://nvidia.github.io/Model-Optimizer/guides/9_autotune.html).

## Resources

Expand Down
2 changes: 2 additions & 0 deletions examples/onnx_ptq/autotune/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

This example demonstrates automated Q/DQ (Quantize/Dequantize) node placement optimization for ONNX models using TensorRT performance measurements.

> **Warning:** This example uses the direct Autotune entry point for lower-level Q/DQ placement experiments. If you are starting from an unquantized ONNX model and care about **accuracy**, please use the ONNX PTQ workflow with `--autotune` enabled and with representative calibration data. See [../README#optimize-qdq-node-placement-with-autotune](../README#optimize-qdq-node-placement-with-autotune).

## Table of Contents

<div align="center">
Expand Down
7 changes: 6 additions & 1 deletion modelopt/onnx/quantization/autotune/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,12 @@ def get_parser() -> argparse.ArgumentParser:
"""
parser = argparse.ArgumentParser(
prog="modelopt.onnx.quantization.autotune",
description="ONNX Q/DQ Autotuning with TensorRT",
description=(
"ONNX Q/DQ placement autotuning with TensorRT latency measurements. "
"For end-to-end ONNX quantization taking into consideration accuracy, use "
"`python -m modelopt.onnx.quantization ... --autotune=<mode>` "
"with representative calibration data."
),
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
Expand Down
Loading