Skip to content

Loosen fp16 atol in ONNX export tests for onnxruntime 1.29 CPU numerics - #37

Open
pggPL wants to merge 1 commit into
mainfrom
onnx_fp16_tolerance
Open

Loosen fp16 atol in ONNX export tests for onnxruntime 1.29 CPU numerics#37
pggPL wants to merge 1 commit into
mainfrom
onnx_fp16_tolerance

Conversation

@pggPL

@pggPL pggPL commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Description

L1_pytorch_onnx_unittest fails in nightly CI since 2026-08-18 on the fp16 (precision1) variants of test_export_linear_recipe, test_export_layernorm_linear_recipe and test_export_layernorm_mlp. The trigger is the onnxruntime 1.28.0 → 1.29.0 bump in the unpinned CI image: ORT executes the exported graph on the CPU EP, and 1.29 changed fp16 CPU numerics, so the ORT-vs-TE mismatch now reaches up to ~9e-3 (2–9 fp16 ULPs) against the hardcoded atol=1e-3. Verified by A/B on the same TE/torch: ORT 1.28 passes, ORT 1.29 fails; no TE commit is involved.

Since the comparison is half-precision GPU (TE) vs half-precision CPU (ORT), differences of a few ULPs from accumulation order are expected and do not indicate an incorrect ONNX representation. This PR makes the tolerance precision-aware: atol=2e-2 for fp16 (the value already used for FP8 CurrentScaling and FP8 MLP in the same file), keeping 1e-3 for fp32.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add get_atol(precision) helper in tests/pytorch/test_onnx_export.py returning 2e-2 for fp16 and the default 1e-3 otherwise.
  • Use it in the non-FP8 validation paths of _test_export_linear, _test_export_layernorm_linear, _test_export_layernorm_mlp and in the DelayedScaling branch of _test_export_layernorm_linear.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

🤖 Generated with Claude Code

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant