Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@
`ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, bit-identical to its scalar loops
64-bit target, and `Max`/`Min` on amd64 and arm64, bit-identical to its
scalar loops
([docs/design-dtypes.md](docs/design-dtypes.md)).
- **Creation** — `New`/`Zeros`/`Ones`/`Full`/`FromData`/`Arange`/`Linspace`/
`Eye`/`Identity`.
Expand Down
9 changes: 9 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,15 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-26 (v0.20.0).** The float32 extremum kernels (`max32SSE2`,
`min32SSE2`, `max32NEON`, `min32NEON`) read `len(a)` elements from a
non-empty slice. The Go wrappers pass `len(a)` and never an empty slice
(`Max`/`Min` refuse an empty array before), the NEON ones only from 16 up.
`TestBin32StaysInsideItsOperands` now fences them too: it passes on Zen 3,
Neoverse N1 and Apple M. They are tested against one sequential pass of
Go's max/min, with NaN, real signed zeros and infinities at every position,
and a sabotaged kernel fails that test on both architectures.

**2026-10-25 (v0.19.1).** A correctness defect, found while reading the
float64 extremum kernels before writing float32 ones. On amd64, `Max`, `Min`,
`MaxAxis` and `MinAxis` over a mix of signed zeros returned the sign of
Expand Down
37 changes: 35 additions & 2 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1015,8 +1015,41 @@ alternated, 8 rounds, pinned core, `--max-busy 15`):
The controls (float64 `Sqrt`, `Add`, `AddInto`) are flat. The gap left to
NumPy is the allocation, the open defect below.

Still to do, in this order: `Max`/`Min` (float32 everywhere, float64 on
ppc64le, loong64 and riscv64), then a float32 GEMM.
### float32 `Max`/`Min` (2026-10-26)

Two changes:

- **Every architecture.** float32 `Max`/`Min` converted the whole array to a
new float64 array and reduced that. They now read the float32 elements
(`kernels.Max32`/`Min32`), with four independent chains of Go's `max`/`min`.
The value is the same, since the conversion is exact.
- **amd64 and arm64: vector kernels.**
- **amd64:** `MAXPS`/`MINPS` with a `CMPPS`-unordered NaN mask. A zero
result goes back to the scalar loop for its sign, the v0.19.1 rule.
- **arm64:** NEON `VFMAX`/`VFMIN` on `.S4`, which propagate NaN and order
−0 below +0 themselves.

Each is bit for bit one sequential pass of Go's `max`/`min`. That is tested
with NaN, real signed zeros and infinities at every position up to length 70,
and a sabotaged kernel fails it on both architectures. Measured with
`abbench` (alternated, 6 rounds, pinned core):

| float32 `Max`, 2¹⁴ / 2²⁰ | change | control (float64) |
|---|---:|---|
| Zen 3 (SSE2) | **−92% / −88%** | ~ |
| Neoverse N1 (NEON) | **−95% / −93%** | ~ |
| POWER9 (Go loop, no kernel yet) | −42% / −34% | ~ |

On the POWER9 float32 `Max` is still about 7× slower than NumPy, because
Go's NaN-correct `max` is scalar there.

Still to do:
- ppc64le and loong64 need registry entries in go-asmgen first. Go's
assemblers have no VSX float compare (for the NaN mask), and on loong64 no
vector float max, min or compare at all.
- s390x (`VFMAXSB` mode 1) waits for a z machine to test on.
- riscv64 (`vfmax.vv` with a `vmfne.vv` NaN mask) is next.
- float32 GEMM.

**Open defect: the allocating operations.** `Add` is still 1.5–3.2× slower
than NumPy's `a+b`, for float64 as well (21.9 against 8.48 µs at 2¹⁴). The
Expand Down
7 changes: 4 additions & 3 deletions docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,10 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
`fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has
SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for
`Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go
for float32: `Max`/`Min` (11.9–15× slower than NumPy on a POWER9),
`Dot`/`MatMul` (9× slower than our float64 GEMM) and the transcendental
ufuncs.
for float32: `Max`/`Min` on ppc64le, loong64, s390x and riscv64 (v0.20.0
vectorised them on amd64 and arm64 and stopped converting to float64
everywhere), `Dot`/`MatMul` (9× slower than our float64 GEMM) and the
transcendental ufuncs.
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
comparisons returning Bool masks, `Maximum`/`Minimum`, `Logical*`, `Power`,
`Where` and the per-axis reductions, in every dtype. Not done: a public
Expand Down
36 changes: 36 additions & 0 deletions fidelity_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,42 @@ func TestLog2Accuracy(t *testing.T) {
}
}

// TestFloat32ExtremaMatchTheFloat64Path: the float32 Max/Min read the float32
// elements directly; converting to float64 first (the path they replace) gives
// the same value, since the conversion is exact. Strided views are included.
func TestFloat32ExtremaMatchTheFloat64Path(t *testing.T) {
nan := float32(math.NaN())
for _, d := range [][]float32{
{3}, {1, -2, 5.5, 0.25}, {-1, -7, float32(math.Copysign(0, -1)), -3, 0},
{1, nan, 3}, {float32(math.Inf(-1)), -1e30, 1e-45},
} {
a := mustArr(t, ok(FromSlice(d, len(d))))
for _, v := range []*Array{a, mustArr(t, ok(a.Slice(Step(-1))))} {
mx, _ := v.Max()
mn, _ := v.Min()
wmx, _ := v.AsType(Float64).Max()
wmn, _ := v.AsType(Float64).Min()
if !sameFloat(mx, wmx) || !sameFloat(mn, wmn) {
t.Errorf("%v: Max=%v Min=%v, float64 path %v %v", d, mx, mn, wmx, wmn)
}
}
}
if _, err := mustArr(t, ok(ZerosOf(Float32, 0))).Max(); err == nil {
t.Error("Max of an empty float32 array: no error")
}
if _, err := mustArr(t, ok(ZerosOf(Float32, 0))).Min(); err == nil {
t.Error("Min of an empty float32 array: no error")
}
}

// sameFloat is bit equality, any NaN matching any NaN.
func sameFloat(a, b float64) bool {
if a != a && b != b {
return true
}
return math.Float64bits(a) == math.Float64bits(b)
}

// TestExtremaOfSignedZeros: Max is +0 and Min is -0 over a mix of signed
// zeros, whatever their order, as Go's max/min and math.Max/Min order them.
// Until v0.19.1 amd64 returned the sign of whichever zero its kernel met last
Expand Down
54 changes: 54 additions & 0 deletions internal/kernels/asmgen/amd64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ func main() {
f.Add(bin32Kernel("mul32SSE2", "MULPS", "MULSS"))
f.Add(bin32Kernel("div32SSE2", "DIVPS", "DIVSS"))
f.Add(sqrt32Kernel())
f.Add(extremum32Kernel("max32SSE2", "MAXPS", "MAXSS"))
f.Add(extremum32Kernel("min32SSE2", "MINPS", "MINSS"))

if err := os.WriteFile("sum_amd64.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
Expand Down Expand Up @@ -606,6 +608,58 @@ func gemmSSE2HalfKernel() *emit.Function {

// nanScan emits: X9 = (reg unordered reg) ? all-ones : 0 ; X8 |= X9, i.e. it
// ORs into the NaN-seen mask X8 a lane of all-ones wherever reg holds a NaN.
// extremum32Kernel builds max32SSE2/min32SSE2(a *float32, n int) float32, for
// n >= 1: the extremum over 16 floats per iteration in four XMM accumulators
// (MAXPS/MINPS), with X8 the OR of every CMPPS-unordered mask so a NaN anywhere
// gives NaN. MAXPS/MINPS return their source operand on a zero tie, so a zero
// result may carry the wrong sign; the Go wrapper hands that case to the scalar
// loop, as maxSIMD does.
func extremum32Kernel(name, vec, sca string) *emit.Function {
sig := amd64.Layout([]string{"a", "n"}, []amd64.Type{amd64.Ptr, amd64.Int64},
[]string{"ret"}, []amd64.Type{amd64.Float32})
b := amd64.NewFunc(name, sig, 0)
b.LoadArg("a", "AX").LoadArg("n", "CX")
// Seed the accumulators with a[0] in every lane.
b.Raw("MOVSS (AX), X0").Raw("SHUFPS $0x00, X0, X0").
Raw("MOVAPS X0, X1").Raw("MOVAPS X0, X2").Raw("MOVAPS X0, X3").
Raw("XORPS X8, X8")
nanScan32(b, "X0")
b.Raw("xblock:").Raw("CMPQ CX, $16").Raw("JL xfold")
for r := 0; r < 4; r++ {
b.Raw("MOVUPS %d(AX), X%d", 16*r, 4+r)
}
for r := 0; r < 4; r++ {
nanScan32(b, fmt.Sprintf("X%d", 4+r))
}
for r := 0; r < 4; r++ {
b.Raw("%s X%d, X%d", vec, 4+r, r)
}
b.Raw("ADDQ $64, AX").Raw("SUBQ $16, CX").Raw("JMP xblock")
// Fold the accumulators, then the four lanes, into X0's low lane.
b.Raw("xfold:").
Raw("%s X1, X0", vec).Raw("%s X3, X2", vec).Raw("%s X2, X0", vec).
Raw("MOVAPS X0, X1").Raw("MOVHLPS X1, X1").Raw("%s X1, X0", vec).
Raw("MOVAPS X0, X1").Raw("SHUFPS $0x55, X1, X1").Raw("%s X1, X0", sca).
Raw("xtail:").Raw("TESTQ CX, CX").Raw("JZ xnan").
Raw("MOVSS (AX), X4")
nanScan32(b, "X4")
b.Raw("%s X4, X0", sca).
Raw("ADDQ $4, AX").Raw("DECQ CX").Raw("JMP xtail").
// Any lane of X8 set: some element was NaN.
Raw("xnan:").Raw("MOVMSKPS X8, DX").Raw("TESTQ DX, DX").Raw("JZ xdone").
Raw("XORPS X1, X1").Raw("DIVSS X1, X1"). // 0/0 = NaN
Raw("MOVSS X1, X0").
Raw("xdone:").
StoreRet("X0", "ret").
Ret()
return b.Func()
}

// nanScan32 ORs the unordered (NaN) lanes of reg into X8.
func nanScan32(b *amd64.Builder, reg string) {
b.Raw("MOVAPS %s, X9", reg).Raw("CMPPS X9, X9, $3").Raw("ORPS X9, X8")
}

func nanScan(b *amd64.Builder, reg string) {
b.Raw("MOVAPS " + reg + ", X9")
b.Raw("CMPPD X9, X9, $3")
Expand Down
40 changes: 40 additions & 0 deletions internal/kernels/asmgen/arm64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ func main() {
f.Add(sqrt32Kernel())
f.Add(extremumKernel("maxNEON", "VFMAX", "FMAXD"))
f.Add(extremumKernel("minNEON", "VFMIN", "FMIND"))
f.Add(extremum32Kernel("max32NEON", "VFMAX", "FMAXS"))
f.Add(extremum32Kernel("min32NEON", "VFMIN", "FMINS"))
f.Add(gemmKernel())
f.Add(blocks8Kernel())
if err := os.WriteFile("sum_arm64.s", []byte(f.String()), 0o644); err != nil {
Expand Down Expand Up @@ -268,6 +270,44 @@ func extremumKernel(name, vop, sop string) *emit.Function {
return b.Func()
}

// extremum32Kernel builds NAME(a *float32, n int) float32, the maximum (or
// minimum) of a, for n >= 16: four .S4 accumulators seeded with the first 16
// elements, folded together, then the four lanes, then the tail one at a time.
// VFMAX/VFMIN, like the scalar FMAXS/FMINS Go's builtin max/min compile to,
// return NaN when either operand is NaN and order -0 below +0, so the result is
// the sequential max's bit for bit (the operation is associative under them).
func extremum32Kernel(name, vop, sop string) *emit.Function {
sig := arm64.Layout([]string{"a", "n"}, []arm64.Type{arm64.Ptr, arm64.Int64},
[]string{"ret"}, []arm64.Type{arm64.Float32})
b := arm64.NewFunc(name, sig, 0)
b.LoadArg("a", "R0").LoadArg("n", "R1").
Raw("VLD1.P 64(R0), [V0.S4, V1.S4, V2.S4, V3.S4]").
Raw("SUB $16, R1").
Raw("block:").Raw("CMP $16, R1").Raw("BLT fold").
Raw("VLD1.P 64(R0), [V4.S4, V5.S4, V6.S4, V7.S4]")
for r := 0; r < 4; r++ {
b.Raw(fmt.Sprintf("%s V%d.S4, V%d.S4, V%d.S4", vop, 4+r, r, r))
}
b.Raw("SUB $16, R1").Raw("B block").
Raw("fold:").
Raw(vop+" V1.S4, V0.S4, V0.S4").
Raw(vop+" V3.S4, V2.S4, V2.S4").
Raw(vop+" V2.S4, V0.S4, V0.S4")
for l := 0; l < 4; l++ {
b.Raw(fmt.Sprintf("VMOV V0.S[%d], R%d", l, 3+l))
}
b.Raw("FMOVS R3, F0").Raw("FMOVS R4, F9").Raw("FMOVS R5, F10").Raw("FMOVS R6, F11").
Raw(sop+" F9, F0, F0").Raw(sop+" F11, F10, F10").Raw(sop+" F10, F0, F0").
Raw("tail:").Raw("CBZ R1, done").
Raw("FMOVS.P 4(R0), F10").
Raw(sop+" F10, F0, F0").
Raw("SUB $1, R1").Raw("B tail").
Raw("done:").
StoreRet("F0", "ret").
Ret()
return b.Func()
}

// gemmSig is the micro-kernel signature
//
// gemmMicro4x8(kc int, pa, pb, c *float64, ldc int)
Expand Down
57 changes: 57 additions & 0 deletions internal/kernels/extrema32.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package kernels

// Max32 returns the maximum of a (non-empty) as numpy.max computes it: NaN if
// any element is NaN, and -0 below +0 (Go's max). It reads the float32
// elements directly. Converting to float64 first gives the same result (the
// conversion is exact) at the cost of a second array.
func Max32(a []float32) float32 { return max32(a) }

// Min32 is the dual of Max32: NaN if any element is NaN, -0 below +0.
func Min32(a []float32) float32 { return min32(a) }

// max32Go is the oracle for the float32 maximum, and the implementation where
// there is no kernel: four accumulators, so four independent chains, folded
// with the same NaN-propagating max. Max is associative under Go's rule, so
// the grouping does not change the result.
func max32Go(a []float32) float32 {
if len(a) < 4 {
m := a[0]
for _, v := range a[1:] {
m = max(m, v)
}
return m
}
m0, m1, m2, m3 := a[0], a[1], a[2], a[3]
s := a[4:]
for len(s) >= 4 {
m0, m1, m2, m3 = max(m0, s[0]), max(m1, s[1]), max(m2, s[2]), max(m3, s[3])
s = s[4:]
}
m := max(max(m0, m1), max(m2, m3))
for _, v := range s {
m = max(m, v)
}
return m
}

// min32Go is the dual of max32Go.
func min32Go(a []float32) float32 {
if len(a) < 4 {
m := a[0]
for _, v := range a[1:] {
m = min(m, v)
}
return m
}
m0, m1, m2, m3 := a[0], a[1], a[2], a[3]
s := a[4:]
for len(s) >= 4 {
m0, m1, m2, m3 = min(m0, s[0]), min(m1, s[1]), min(m2, s[2]), min(m3, s[3])
s = s[4:]
}
m := min(min(m0, m1), min(m2, m3))
for _, v := range s {
m = min(m, v)
}
return m
}
27 changes: 27 additions & 0 deletions internal/kernels/extrema32_amd64.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package kernels

// max32 runs the SSE2 kernel (sum_amd64.s: MAXPS, four accumulators, a
// CMPPS-unordered NaN mask). MAXPS returns its source operand on a zero tie,
// so a zero result may have the wrong sign: that case, which needs every
// element <= 0 with a zero among them, goes to the scalar loop (Go's max: +0
// if any +0), as maxSIMD does for float64.
func max32(a []float32) float32 {
if r := max32SSE2(&a[0], len(a)); r != 0 {
return r
}
return max32Go(a)
}

// min32 is the dual of max32 (a zero result: -0 if any -0).
func min32(a []float32) float32 {
if r := min32SSE2(&a[0], len(a)); r != 0 {
return r
}
return min32Go(a)
}

//go:noescape
func max32SSE2(a *float32, n int) float32

//go:noescape
func min32SSE2(a *float32, n int) float32
25 changes: 25 additions & 0 deletions internal/kernels/extrema32_arm64.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package kernels

// max32 runs the NEON kernel (sum_arm64.s: VFMAX on .S4, four accumulators)
// from 16 elements up, the four-chain Go loop below that. VFMAX propagates NaN
// and orders -0 below +0, so no correction is needed.
func max32(a []float32) float32 {
if len(a) < 16 {
return max32Go(a)
}
return max32NEON(&a[0], len(a))
}

// min32 is the dual of max32, with VFMIN.
func min32(a []float32) float32 {
if len(a) < 16 {
return min32Go(a)
}
return min32NEON(&a[0], len(a))
}

//go:noescape
func max32NEON(a *float32, n int) float32

//go:noescape
func min32NEON(a *float32, n int) float32
9 changes: 9 additions & 0 deletions internal/kernels/extrema32_generic.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
//go:build !amd64 && !arm64

package kernels

// The float32 extrema run the four-chain Go loops on the architectures
// without a vector kernel for them yet.

func max32(a []float32) float32 { return max32Go(a) }
func min32(a []float32) float32 { return min32Go(a) }
Loading
Loading