Skip to content

[Fix][Relax] Preserve 16-bit quantization saturation - #20987

Open
DiadkoShmek wants to merge 1 commit into
apache:mainfrom
DiadkoShmek:DiadkoShmek/fix-relax-quantize-int16
Open

DiadkoShmek wants to merge 1 commit into
apache:mainfrom
DiadkoShmek:DiadkoShmek/fix-relax-quantize-int16

Conversation

@DiadkoShmek

Copy link
Copy Markdown

Fixes #20891.

Relax quantization to 16-bit integers can clip in float16. The signed upper bound 32767 rounds to 32768, and the unsigned bound 65535 rounds to infinity. The final integer cast can turn positive saturation into -32768 or 0, respectively. Adding the zero point in float16 can also lose valid interior codes such as 4096 / 2 + 15 = 2063.

For int16 and uint16 outputs, use float32 for the division, rounding, zero-point addition, and clip. The clip bounds are exactly representable in float32, and the cast to the requested integer dtype happens after clipping.

Validation on 23a6dbe, Linux x86-64, LLVM from TIRx 0.1.0.

  • The signed and unsigned float16 regressions fail on the base. Both float32 controls pass. The patched QDQ legalization and operator tests pass 24/24.
  • An additional LLVM sweep enumerated all finite binary16 encodings and checked those with integral quotients for each of five scale/zero-point pairs per output dtype. Across 143,360 checked cases, there were 58,261 mismatches on the base and zero with the patch.
  • Ruff lint and formatting checks pass for the changed files.

Codex assisted with the patch and validation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [Relax] FLOAT16 quantize wraps positive INT16 saturation to -32768

1 participant