Repository navigation
Golden tests: pin CPU maths paths so results match on every runner CPU - #830
Merged
Merged
Conversation
GitHub runners differ in CPU (some have AVX-512); MKL and PyTorch kernels pick per-CPU code paths, which moved golden voltages by up to 1.5e-5 (Izhikevich 0.23 mV) on CI run 40800432540. Pin ATEN_CPU_CAPABILITY=avx2 and MKL_CBWR=COMPATIBLE before torch starts (test/conftest.py, regen.py), switch oneDNN off in the golden runs, and regenerate the goldens with that reason. Spikes unchanged in all 16; floats moved at most 2.3e-5 in 4 files. CI prints the runner CPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hananel-Hazan
added
the
tests-changed-ok
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes and why
The golden-output tests from #828 failed on
test (3.11)in #829 (CI run 40800432540). #829 changes only a README comment, and the same code passed on #828. The difference is the GitHub runner's CPU. MKL (used for matrix products) and PyTorch's own kernels each pick the fastest code path for the processor. Some runners have AVX-512; the golden files were made on an AVX2-only CPU. The rounding moved voltages by up to 1.5e-5, and the Izhikevich neuron amplified this to 0.23 mV. The tolerance stays at 1e-6.The fix pins the libraries to one code path on every CPU (
test/regression/cpu_pins.py):ATEN_CPU_CAPABILITY=avx2: PyTorch's kernels use AVX2 even on AVX-512 machines.MKL_CBWR=COMPATIBLE: MKL's mode that gives the same results on all x86 processors.MKL_CBWR.test/conftest.pyandregen.pyapply the pins before torch starts.networks.deterministic()raises an error if the PyTorch pin came too late.The evidence that MKL's path matters, measured on one AVX2 machine: changing only
MKL_CBWRchanged 1 to 4 of the 16 goldens. Changing onlyATEN_CPU_CAPABILITYchanged none of them on that machine.The goldens were regenerated with that reason; there is no library change. Spikes are identical in all 16 files. Float values moved by at most 2.3e-5, in 4 files:
dc2015_b16,conn_postpre_b8,csrm_windowandconv2d_if_izh. The other 12 are bit-identical. CI now prints each runner's CPU model and whether it has AVX-512.Verified on both CPU kinds (CI run 40803279232, two rounds, six jobs): five jobs ran on AMD EPYC 7763 without AVX-512 and one (
test (3.12), round 2) on AMD EPYC 9V45 with AVX-512. Every job passed: 512 passed, 4 skipped (the CUDA test), 40 xfailed. Only one AVX-512 sample has run so far.Checklist
--reason. Theweakened-testscheck will list them.🤖 Generated with Claude Code