Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,13 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-11-02 (v0.33.0).** The AVX2 float32 exp kernel (`exp32CRAVX2`)
reads and writes `n` floats, a positive multiple of 8 chosen by the Go
wrapper. It gathers table entries at `z & 63`, so the index cannot leave the
64-entry table, whatever the input. It reads a constant block the wrapper
passes. It is gated on AVX2 and FMA (CPU and OS) and fenced with PROT_NONE
on both sides of each operand.

**2026-11-01 (v0.32.0).** float32 `Log` checks each result for a float32
rounding midpoint with one integer comparison. On a hit it looks the input
up by binary search in a fixed 24-entry table, compiled in and read only.
Expand Down
44 changes: 39 additions & 5 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1481,11 +1481,45 @@ So float32 `Exp` and `Log` are **correctly rounded**: every result is the
float32 nearest the exact value. NumPy's float32 SIMD kernels are documented
at 2.52 and 3.83 ULP in their sources.

**Open defect: still 3× slower than NumPy.** NumPy computes float32 exp and
log in float32 SIMD (`simd_exp_f32`/`simd_log_f32` in
`loops_exponent_log.dispatch.c.src`). Porting those kernels per architecture
is the next step. float64 `Log` is also 1.23× slower than NumPy at 16 Ki on
Zen 3 (85 µs against 69 µs), while float64 `Exp` is 16% faster.
#### float32 `Exp` in AVX2, correctly rounded and as fast as NumPy (2026-11-02)

NumPy's float32 exp gets its speed in float32 SIMD and gives up accuracy to
do it: 2.52 ULP in its source. We wanted both. Arm's optimized-routines has a
vector `expf` that is correctly rounded (`math/aarch64/advsimd/cr_expf.c`,
MIT). It widens to float64 and uses a 64-entry table of 2^(i/64) plus a
degree-5 polynomial, cheap enough for SIMD.

- **`expCR`** transcribes it one operation per line, every fused
multiply-add as `math.FMA`, so a kernel can repeat it bit for bit. Its
table is re-derived at 300 bits (`TestExpCRTable`). Its float64 error is
1.006 ULP, so the exhaustive certifier uses a 4-ULP margin for it.
Over all 2³² float32 inputs: 7 decided by the 300-bit reference,
**0 incorrectly rounded**, with no exception table needed.
- **`exp32CRAVX2`** (go-asmgen, amd64 with AVX2 and FMA) runs the same
operations eight floats at a time. It widens with `VCVTPS2PD`, gathers the
table with `VPGATHERQQ`, assembles the exponent from `z << 46`, and rounds
once with `VCVTPD2PS`. Inputs are clamped to [−512, 512], which keeps the
float32 result: +Inf above, 0 below. NaN lanes are restored from the input.
It is bit-identical to `expCR` on 100 000 inputs, every special value and
every length 0–70, it is fenced, and on a Zen 3 the exhaustive test, which
runs it, finds **0 incorrectly rounded** of 4 294 967 296.
- Every architecture now runs `expCR`: the AVX2 kernel where there is one,
the Go transcription elsewhere. So the bits are the same everywhere.

| float32 `Exp`, one Zen 3 core | v0.32.0 | AVX2 | NumPy 2.5.4 | AVX2 / NumPy |
|---|---:|---:|---:|---:|
| 16 Ki | 90 µs | **26.9 µs** | 27.1 µs | **0.99** |
| 1 Mi | 5.8 ms | **1.48 ms** | 1.66 ms | **0.89** |

These are +235% and +289% in throughput (`abbench`, 8 rounds, p = 0.000). The
host was at load 59 on 128 cores; NumPy ran in the same session on the same
pinned core.

**Open defect: float32 `Log` is still 3× slower than NumPy** (`simd_log_f32`,
3.83 ULP in its source). The next step is the same treatment for log, and
the other architectures for exp. float64 `Log` is also 1.23× slower than
NumPy at 16 Ki on Zen 3 (85 µs against 69 µs), while float64 `Exp` is 16%
faster.

## SIMD coverage

Expand Down
78 changes: 78 additions & 0 deletions internal/kernels/asmgen/amd64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ func main() {
f.Add(dotFMAKernel())
f.Add(gemmSSE2HalfKernel())
f.Add(amd64.FeatureProbe("hasFMA", amd64.FMA))
f.Add(amd64.FeatureProbe("hasAVX2", amd64.AVX2))
f.Add(exp32CRKernel())
f.Add(blocks8Kernel())
f.Add(bin32Kernel("add32SSE2", "ADDPS", "ADDSS"))
f.Add(bin32Kernel("sub32SSE2", "SUBPS", "SUBSS"))
Expand Down Expand Up @@ -368,6 +370,82 @@ func sqrt32Kernel() *emit.Function {
return b.Func()
}

// exp32CRKernel builds exp32CRAVX2(dst, src *float32, n int, k *expCRConsts)
// for n a positive multiple of 8: dst[i] = float32(expCR(clamp(src[i]))), the
// correctly rounded float32 exp, bit for bit the Go expCR (kernels/expcr.go,
// a transcription of Arm optimized-routines' cr_expf).
//
// Eight floats per iteration: clamped to [-512, 512] (exp of the clamped
// value rounds to the same float32 +Inf or 0 as the input's), widened to two
// groups of four doubles, and each group run through expCR's operations in
// the same order and with the same fusing: z = x*invln2 + shift, n = z -
// shift, r = x - n*ln2hi - n*ln2lo (VFNMADD231PD), the degree-5 polynomial,
// the scale assembled from the table entry z&63 (VPGATHERQQ) and the
// exponent bits z<<46, and s*y + s; then rounded once to float32. NaN lanes
// are restored from the input at the end (the clamp turned them into -512).
//
// The constants are 4-lane vectors in the Go block k (offsets below), used as
// memory operands, so all sixteen YMM registers stay free. Needs AVX2 (the
// gather) and FMA.
func exp32CRKernel() *emit.Function {
sig := amd64.Layout(
[]string{"dst", "src", "n", "k"},
[]amd64.Type{amd64.Ptr, amd64.Ptr, amd64.Int64, amd64.Ptr},
nil, nil,
)
const (
shift, invLn2, ln2Hi, ln2Lo = 0, 32, 64, 96
c0, c1, c2, c3 = 128, 160, 192, 224
idxMask, expMask = 256, 288
lo, hi = 320, 352
table = 384
)
b := amd64.NewFunc("exp32CRAVX2", sig, 0)
b.LoadArg("dst", "DI").LoadArg("src", "SI").LoadArg("n", "CX").LoadArg("k", "R8")
b.Raw(fmt.Sprintf("LEAQ %d(R8), R9", table)) // the 64 mantissas of 2^(i/64)
// half runs expCR on the four doubles in h, leaving the four float32
// results in out. Temporaries Y4-Y10.
half := func(h, out string) {
b.Raw(fmt.Sprintf("VMOVUPD %d(R8), Y4", shift))
b.Raw(fmt.Sprintf("VFMADD231PD %d(R8), %s, Y4", invLn2, h)) // z = x*invln2 + shift
b.Raw(fmt.Sprintf("VSUBPD %d(R8), Y4, Y5", shift)) // n = z - shift
b.Raw(fmt.Sprintf("VFNMADD231PD %d(R8), Y5, %s", ln2Hi, h)) // r = x - n*ln2hi
b.Raw(fmt.Sprintf("VFNMADD231PD %d(R8), Y5, %s", ln2Lo, h)) // r -= n*ln2lo
b.Raw(fmt.Sprintf("VMULPD %s, %s, Y6", h, h)) // r2
b.Raw(fmt.Sprintf("VMOVUPD %d(R8), Y7", c0))
b.Raw(fmt.Sprintf("VFMADD231PD %d(R8), %s, Y7", c1, h)) // p01 = r*c1 + c0
b.Raw(fmt.Sprintf("VMOVUPD %d(R8), Y8", c2))
b.Raw(fmt.Sprintf("VFMADD231PD %d(R8), %s, Y8", c3, h)) // p23 = r*c3 + c2
b.Raw("VFMADD231PD Y8, Y6, Y7") // p04 = r2*p23 + p01
b.Raw(fmt.Sprintf("VFMADD231PD Y7, Y6, %s", h)) // y = r2*p04 + r
b.Raw(fmt.Sprintf("VPAND %d(R8), Y4, Y5", idxMask)) // z & 63
b.Raw("VPCMPEQQ Y9, Y9, Y9") // gather every lane
b.Raw("VPGATHERQQ Y9, (R9)(Y5*8), Y10") // mantissa of 2^(i/64)
b.Raw("VPSLLQ $46, Y4, Y4")
b.Raw(fmt.Sprintf("VPAND %d(R8), Y4, Y4", expMask)) // exponent and sign bits
b.Raw("VPOR Y10, Y4, Y4") // s
b.Raw(fmt.Sprintf("VFMADD231PD %s, Y4, Y4", h)) // s*y + s
b.Raw(fmt.Sprintf("VCVTPD2PSY Y4, %s", out)) // round once to float32
}
b.Raw("eloop:")
b.Raw("VMOVUPS (SI), Y0")
b.Raw("VCMPPS $3, Y0, Y0, Y15") // unordered: the NaN lanes
b.Raw(fmt.Sprintf("VMAXPS %d(R8), Y0, Y1", lo))
b.Raw(fmt.Sprintf("VMINPS %d(R8), Y1, Y1", hi))
b.Raw("VCVTPS2PD X1, Y2")
b.Raw("VEXTRACTF128 $1, Y1, X3")
b.Raw("VCVTPS2PD X3, Y3")
half("Y2", "X11")
half("Y3", "X12")
b.Raw("VINSERTF128 $1, X12, Y11, Y11")
b.Raw("VBLENDVPS Y15, Y0, Y11, Y11") // NaN lanes: the input NaN
b.Raw("VMOVUPS Y11, (DI)")
b.Raw("ADDQ $32, SI").Raw("ADDQ $32, DI").Raw("SUBQ $8, CX").Raw("JNZ eloop")
b.Raw("VZEROUPPER")
b.Ret()
return b.Func()
}

// bin32Kernel builds NAME(dst, a, b *float32, n int): dst[i] = a[i] OP b[i],
// 16 floats per iteration over four XMM registers with the packed op, then a
// scalar tail with sca. Each lane is one IEEE float32 operation, so the result
Expand Down
94 changes: 94 additions & 0 deletions internal/kernels/expcr.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
package kernels

import "math"

// expCR is the double-precision exp of Arm's correctly rounded vector expf
// (optimized-routines, math/aarch64/advsimd/cr_expf.c and cr_expf_utils.h,
// MIT), transcribed one operation per line so that a SIMD kernel can repeat
// it bit for bit: every vfmaq/vfmsq is math.FMA.
//
// x/ln2 is rounded to a multiple of 1/64 by adding a shift whose ULP is 2^-6;
// the low 6 bits of the sum index a table of 2^(i/64), and the bits above
// (with the bias 0x3ff folded into the shift) are the exponent of the scale.
// exp(r) - 1 on |r| <= ln2/128 is a degree-5 polynomial. Valid for
// |x| <= 512, which covers every float32 whose exp is neither 0 nor +Inf.
func expCR(x float64) float64 {
z := math.FMA(x, crInvLn2, crShift)
n := z - crShift
r := math.FMA(-n, crLn2Hi, x)
r = math.FMA(-n, crLn2Lo, r)
r2 := r * r
p01 := math.FMA(r, crC1, crC0)
p23 := math.FMA(r, crC3, crC2)
p04 := math.FMA(r2, p23, p01)
y := math.FMA(r2, p04, r)
u := math.Float64bits(z)
s := math.Float64frombits(u<<46&0xfff0000000000000 | crMant[u&63])
return math.FMA(s, y, s)
}

const (
crShift = 0x1.800000000ffc0p+46
crInvLn2 = 0x1.71547652b82fep+0
crLn2Hi = 0x1.62e42fefa39efp-1
crLn2Lo = 0x1.abc9e3b39803fp-56
crC0 = 0x1.fffffffffdbcep-2
crC1 = 0x1.55555555543c2p-3
crC2 = 0x1.555573c64f2e3p-5
crC3 = 0x1.111126b4eff73p-7
)

// crMant are the mantissas of 2^(i/64), i = 0..63 (cr_expf_utils.h).
var crMant = [64]uint64{
0x0000000000000, 0x02c9a3e778061, 0x059b0d3158574, 0x0874518759bc8,
0x0b5586cf9890f, 0x0e3ec32d3d1a2, 0x11301d0125b51, 0x1429aaea92de0,
0x172b83c7d517b, 0x1a35beb6fcb75, 0x1d4873168b9aa, 0x2063b88628cd6,
0x2387a6e756238, 0x26b4565e27cdd, 0x29e9df51fdee1, 0x2d285a6e4030b,
0x306fe0a31b715, 0x33c08b26416ff, 0x371a7373aa9cb, 0x3a7db34e59ff7,
0x3dea64c123422, 0x4160a21f72e2a, 0x44e086061892d, 0x486a2b5c13cd0,
0x4bfdad5362a27, 0x4f9b2769d2ca7, 0x5342b569d4f82, 0x56f4736b527da,
0x5ab07dd485429, 0x5e76f15ad2148, 0x6247eb03a5585, 0x6623882552225,
0x6a09e667f3bcd, 0x6dfb23c651a2f, 0x71f75e8ec5f74, 0x75feb564267c9,
0x7a11473eb0187, 0x7e2f336cf4e62, 0x82589994cce13, 0x868d99b4492ed,
0x8ace5422aa0db, 0x8f1ae99157736, 0x93737b0cdc5e5, 0x97d829fde4e50,
0x9c49182a3f090, 0xa0c667b5de565, 0xa5503b23e255d, 0xa9e6b5579fdbf,
0xae89f995ad3ad, 0xb33a2b84f15fb, 0xb7f76f2fb5e47, 0xbcc1e904bc1d2,
0xc199bdd85529c, 0xc67f12e57d14b, 0xcb720dcef9069, 0xd072d4a07897c,
0xd5818dcfba487, 0xda9e603db3285, 0xdfc97337b9b5f, 0xe502ee78b3ff6,
0xea4afa2a490da, 0xefa1bee615a27, 0xf50765b6e4540, 0xfa7c1819e90d8,
}

// expCRSlice writes expCR(src[i]) into dst[i] for |src[i]| <= 512, and the
// limits outside: +Inf above, 0 below, NaN for NaN (the float32 results
// there are the same).
func expCRSlice(dst, src []float64) {
dst = dst[:len(src)]
for i, x := range src {
switch {
case math.Abs(x) <= 512:
dst[i] = expCR(x)
case x > 0:
dst[i] = math.Inf(1)
case x < 0:
dst[i] = 0
default:
dst[i] = x // NaN
}
}
}

// exp32CR is float32 exp on expCR, rounded once.
func exp32CR(dst, src []float32) {
src = src[:len(dst)]
var buf [via64Chunk]float64
for i := 0; i < len(dst); i += via64Chunk {
b := buf[:min(via64Chunk, len(dst)-i)]
for j := range b {
b[j] = float64(src[i+j])
}
expCRSlice(b, b)
for j, v := range b {
dst[i+j] = float32(v)
}
}
}
50 changes: 50 additions & 0 deletions internal/kernels/expcr_amd64.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
package kernels

// expCRConsts is the constant block exp32CRAVX2 reads (asmgen/amd64/gen.go,
// exp32CRKernel): each constant as a 4-lane vector at the offset the kernel
// uses, then the clamp bounds as 8 floats, then the 64 table mantissas.
type expCRConsts struct {
shift, invLn2, ln2Hi, ln2Lo [4]float64 // offsets 0, 32, 64, 96
c0, c1, c2, c3 [4]float64 // 128, 160, 192, 224
idxMask, expMask [4]uint64 // 256, 288
lo, hi [8]float32 // 320, 352
table [64]uint64 // 384
}

var expCRK = func() *expCRConsts {
k := new(expCRConsts)
for i := range 4 {
k.shift[i], k.invLn2[i], k.ln2Hi[i], k.ln2Lo[i] = crShift, crInvLn2, crLn2Hi, crLn2Lo
k.c0[i], k.c1[i], k.c2[i], k.c3[i] = crC0, crC1, crC2, crC3
k.idxMask[i], k.expMask[i] = 63, 0xfff0000000000000
}
for i := range 8 {
k.lo[i], k.hi[i] = -512, 512
}
k.table = crMant
return k
}()

// useExp32AVX2 selects the AVX2 kernel; a var so tests can force the Go path.
var useExp32AVX2 = hasAVX2() && useFMA

// exp32Fast is the correctly rounded float32 exp: blocks of 8 on the AVX2
// kernel, the rest (and every element without AVX2+FMA) on exp32CR. Both
// compute expCR's operations in the same order, so they agree bit for bit.
func exp32Fast(dst, src []float32) {
src = src[:len(dst)]
n := 0
if useExp32AVX2 {
n = len(dst) &^ 7
if n > 0 {
exp32CRAVX2(&dst[0], &src[0], n, expCRK)
}
}
exp32CR(dst[n:], src[n:])
}

// hasAVX2 reports CPU and OS support for AVX2 (go-asmgen FeatureProbe).
func hasAVX2() bool

//go:noescape
func exp32CRAVX2(dst, src *float32, n int, k *expCRConsts)
45 changes: 45 additions & 0 deletions internal/kernels/expcr_amd64_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
package kernels

import (
"math"
"math/rand"
"testing"
)

// TestExp32AVX2MatchesGo holds the AVX2 kernel to exp32CR bit for bit (NaN
// to NaN) on random inputs across the float32 range, the special values and
// the clamp and overflow edges, at every length 0..70 (blocks of 8 and the Go
// tail).
func TestExp32AVX2MatchesGo(t *testing.T) {
if !useExp32AVX2 {
t.Skip("no AVX2+FMA here")
}
r := rand.New(rand.NewSource(3))
src := make([]float32, 0, 100000)
for _, v := range []float32{0, float32(math.Copysign(0, -1)), 1, -1, 88.72283, 88.72284, 89, -103.972, -103.973, -104,
511.9, 512, 512.1, -512, -512.1, 1e30, -1e30, float32(math.Inf(1)), float32(math.Inf(-1)),
float32(math.NaN()), math.Float32frombits(0xffc00000), math.SmallestNonzeroFloat32} {
src = append(src, v)
}
for len(src) < cap(src) {
src = append(src, math.Float32frombits(r.Uint32()))
}
want := make([]float32, len(src))
exp32CR(want, src)
got := make([]float32, len(src))
exp32Fast(got, src)
for i := range src {
if math.Float32bits(got[i]) != math.Float32bits(want[i]) && !(got[i] != got[i] && want[i] != want[i]) {
t.Fatalf("exp(0x%08x): AVX2 0x%08x, Go 0x%08x", math.Float32bits(src[i]), math.Float32bits(got[i]), math.Float32bits(want[i]))
}
}
for n := 0; n <= 70; n++ {
d := make([]float32, n)
exp32Fast(d, src[:n])
for i := range d {
if math.Float32bits(d[i]) != math.Float32bits(want[i]) && !(d[i] != d[i] && want[i] != want[i]) {
t.Fatalf("length %d [%d]: 0x%08x, want 0x%08x", n, i, math.Float32bits(d[i]), math.Float32bits(want[i]))
}
}
}
}
20 changes: 20 additions & 0 deletions internal/kernels/expcr_fence_amd64_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
//go:build linux || darwin

package kernels

import "testing"

// TestExp32AVX2StaysInside fences src and dst of the AVX2 exp kernel
// (fences32 needs mmap/mprotect, hence the build tag of fence_test.go).
func TestExp32AVX2StaysInside(t *testing.T) {
if !useExp32AVX2 {
t.Skip("no AVX2+FMA here")
}
for n := 8; n <= 64; n += 8 {
for _, s := range fences32(t, n) {
for _, d := range fences32(t, n) {
exp32CRAVX2(&d[0], &s[0], n, expCRK)
}
}
}
}
7 changes: 7 additions & 0 deletions internal/kernels/expcr_other.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
//go:build !amd64

package kernels

// exp32Fast is the correctly rounded float32 exp: expCR in Go on these
// architectures (no SIMD kernel yet).
func exp32Fast(dst, src []float32) { exp32CR(dst, src) }
Loading
Loading