Skip to content

perf: SIMD inner product (AVX2/FMA, NEON) for Dot, MatVec and Dot1DP - #16

Merged
tannevaled merged 1 commit into
mainfrom
perf/simd-dot
Oct 4, 2026
Merged

tannevaled merged 1 commit into
mainfrom
perf/simd-dot

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

dotRange, under Dot (1-D), MatVec and Dot1DP, was a four-chain scalar loop
(gc does not vectorise it); on one Zen 3 core a 2^20 dot took 464 us, 2x
NumPy's single thread. Two go-asmgen kernels replace it:

  • amd64 dotFMA: four YMM accumulators, b read straight from memory by
    VFMADD231PD; gated on the FMA probe like the GEMM kernel (the flag is
    renamed useFMA, since it now covers both).
  • arm64 dotNEON: eight D2 accumulators for the four FP pipes of an Apple
    core, folded with FMLA against 1.0.

The other targets keep dotRange. Zen 3, interleaved with main:
Dot 2^20 1 core 464 -> 229 us (NumPy 231) 16 cores 159 -> 99 us
MatVec 1024^2 1 core 482 -> 134 us (NumPy 153) 16 cores 165 -> 89 us

TestDotSIMD: exact on integer data for every length 0..70 (main loop,
folds, tail), within nepssum|a_i*b_i| on random data, NaN and Inf
propagate; on amd64 it also runs with FMA forced off. Passes on a real
Zen 3 (FMA path).

🤖 Generated with Claude Code

dotRange, under Dot (1-D), MatVec and Dot1DP, was a four-chain scalar loop
(gc does not vectorise it); on one Zen 3 core a 2^20 dot took 464 us, 2x
NumPy's single thread. Two go-asmgen kernels replace it:

- amd64 dotFMA: four YMM accumulators, b read straight from memory by
  VFMADD231PD; gated on the FMA probe like the GEMM kernel (the flag is
  renamed useFMA, since it now covers both).
- arm64 dotNEON: eight D2 accumulators for the four FP pipes of an Apple
  core, folded with FMLA against 1.0.

The other targets keep dotRange. Zen 3, interleaved with main:
  Dot 2^20     1 core 464 -> 229 us (NumPy 231)   16 cores 159 -> 99 us
  MatVec 1024^2 1 core 482 -> 134 us (NumPy 153)  16 cores 165 -> 89 us

TestDotSIMD: exact on integer data for every length 0..70 (main loop,
folds, tail), within n*eps*sum|a_i*b_i| on random data, NaN and Inf
propagate; on amd64 it also runs with FMA forced off. Passes on a real
Zen 3 (FMA path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit c405cbe into main Oct 4, 2026
12 checks passed
@tannevaled
tannevaled deleted the perf/simd-dot branch October 4, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant