From f9c5131206df906ac7ed6ec0cd516a035a02492e Mon Sep 17 00:00:00 2001 From: tannevaled Date: Mon, 26 Oct 2026 11:21:34 +0100 Subject: [PATCH] perf: float32 Max/Min read float32 directly; SSE2 and NEON kernels float32 Max/Min converted the whole array to float64 and reduced that: on a POWER9 they were 11.9-15x slower than NumPy. kernels.Max32/Min32 now read the float32 elements, with four chains of Go's max/min (the value is the same: the conversion is exact), and run vector kernels on amd64 (MAXPS/MINPS + CMPPS-unordered NaN mask; a zero result goes to the scalar loop for its sign, the v0.19.1 rule) and arm64 (NEON VFMAX/VFMIN .S4, NaN-propagating, -0 below +0). abbench, alternated, pinned core, float32 Max 2^14 / 2^20: Zen 3 -92% / -88%, Neoverse N1 -95% / -93%, POWER9 (Go loop) -42% / -34%; float64 controls flat. Tested bit for bit against one sequential pass of Go's max/min, with NaN, real signed zeros and infinities at every position to length 70; a sabotaged kernel fails it on both architectures; fenced. Co-Authored-By: Claude Opus 5.5 --- README.md | 3 +- SECURITY.md | 9 ++ docs/perf.md | 37 ++++++- docs/plan-ndarray.md | 7 +- fidelity_test.go | 36 +++++++ internal/kernels/asmgen/amd64/gen.go | 54 ++++++++++ internal/kernels/asmgen/arm64/gen.go | 40 ++++++++ internal/kernels/extrema32.go | 57 +++++++++++ internal/kernels/extrema32_amd64.go | 27 +++++ internal/kernels/extrema32_arm64.go | 25 +++++ internal/kernels/extrema32_generic.go | 9 ++ internal/kernels/extrema32_test.go | 82 +++++++++++++++ internal/kernels/fence_test.go | 1 + internal/kernels/sum_amd64.s | 140 ++++++++++++++++++++++++++ internal/kernels/sum_arm64.s | 80 +++++++++++++++ ndarray.go | 6 ++ 16 files changed, 607 insertions(+), 6 deletions(-) create mode 100644 internal/kernels/extrema32.go create mode 100644 internal/kernels/extrema32_amd64.go create mode 100644 internal/kernels/extrema32_arm64.go create mode 100644 internal/kernels/extrema32_generic.go create mode 100644 internal/kernels/extrema32_test.go diff --git a/README.md b/README.md index d3acbee..04e7c18 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,8 @@ `ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every - 64-bit target, bit-identical to its scalar loops + 64-bit target, and `Max`/`Min` on amd64 and arm64, bit-identical to its + scalar loops ([docs/design-dtypes.md](docs/design-dtypes.md)). - **Creation** — `New`/`Zeros`/`Ones`/`Full`/`FromData`/`Arange`/`Linspace`/ `Eye`/`Identity`. diff --git a/SECURITY.md b/SECURITY.md index dcadfcb..7332799 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -66,6 +66,15 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none. ## Audit log +**2026-10-26 (v0.20.0).** The float32 extremum kernels (`max32SSE2`, +`min32SSE2`, `max32NEON`, `min32NEON`) read `len(a)` elements from a +non-empty slice. The Go wrappers pass `len(a)` and never an empty slice +(`Max`/`Min` refuse an empty array before), the NEON ones only from 16 up. +`TestBin32StaysInsideItsOperands` now fences them too: it passes on Zen 3, +Neoverse N1 and Apple M. They are tested against one sequential pass of +Go's max/min, with NaN, real signed zeros and infinities at every position, +and a sabotaged kernel fails that test on both architectures. + **2026-10-25 (v0.19.1).** A correctness defect, found while reading the float64 extremum kernels before writing float32 ones. On amd64, `Max`, `Min`, `MaxAxis` and `MinAxis` over a mix of signed zeros returned the sign of diff --git a/docs/perf.md b/docs/perf.md index 2694ef8..3055348 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -1015,8 +1015,41 @@ alternated, 8 rounds, pinned core, `--max-busy 15`): The controls (float64 `Sqrt`, `Add`, `AddInto`) are flat. The gap left to NumPy is the allocation, the open defect below. -Still to do, in this order: `Max`/`Min` (float32 everywhere, float64 on -ppc64le, loong64 and riscv64), then a float32 GEMM. +### float32 `Max`/`Min` (2026-10-26) + +Two changes: + +- **Every architecture.** float32 `Max`/`Min` converted the whole array to a + new float64 array and reduced that. They now read the float32 elements + (`kernels.Max32`/`Min32`), with four independent chains of Go's `max`/`min`. + The value is the same, since the conversion is exact. +- **amd64 and arm64: vector kernels.** + - **amd64:** `MAXPS`/`MINPS` with a `CMPPS`-unordered NaN mask. A zero + result goes back to the scalar loop for its sign, the v0.19.1 rule. + - **arm64:** NEON `VFMAX`/`VFMIN` on `.S4`, which propagate NaN and order + −0 below +0 themselves. + +Each is bit for bit one sequential pass of Go's `max`/`min`. That is tested +with NaN, real signed zeros and infinities at every position up to length 70, +and a sabotaged kernel fails it on both architectures. Measured with +`abbench` (alternated, 6 rounds, pinned core): + +| float32 `Max`, 2¹⁴ / 2²⁰ | change | control (float64) | +|---|---:|---| +| Zen 3 (SSE2) | **−92% / −88%** | ~ | +| Neoverse N1 (NEON) | **−95% / −93%** | ~ | +| POWER9 (Go loop, no kernel yet) | −42% / −34% | ~ | + +On the POWER9 float32 `Max` is still about 7× slower than NumPy, because +Go's NaN-correct `max` is scalar there. + +Still to do: +- ppc64le and loong64 need registry entries in go-asmgen first. Go's + assemblers have no VSX float compare (for the NaN mask), and on loong64 no + vector float max, min or compare at all. +- s390x (`VFMAXSB` mode 1) waits for a z machine to test on. +- riscv64 (`vfmax.vv` with a `vmfne.vv` NaN mask) is next. +- float32 GEMM. **Open defect: the allocating operations.** `Add` is still 1.5–3.2× slower than NumPy's `a+b`, for float64 as well (21.9 against 8.48 µs at 2¹⁴). The diff --git a/docs/plan-ndarray.md b/docs/plan-ndarray.md index 8df0b08..d93c8f6 100644 --- a/docs/plan-ndarray.md +++ b/docs/plan-ndarray.md @@ -157,9 +157,10 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`). `fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for `Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go - for float32: `Max`/`Min` (11.9–15× slower than NumPy on a POWER9), - `Dot`/`MatMul` (9× slower than our float64 GEMM) and the transcendental - ufuncs. + for float32: `Max`/`Min` on ppc64le, loong64, s390x and riscv64 (v0.20.0 + vectorised them on amd64 and arm64 and stopped converting to float64 + everywhere), `Dot`/`MatMul` (9× slower than our float64 GEMM) and the + transcendental ufuncs. - **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic, comparisons returning Bool masks, `Maximum`/`Minimum`, `Logical*`, `Power`, `Where` and the per-axis reductions, in every dtype. Not done: a public diff --git a/fidelity_test.go b/fidelity_test.go index 6edeb5a..1da32da 100644 --- a/fidelity_test.go +++ b/fidelity_test.go @@ -123,6 +123,42 @@ func TestLog2Accuracy(t *testing.T) { } } +// TestFloat32ExtremaMatchTheFloat64Path: the float32 Max/Min read the float32 +// elements directly; converting to float64 first (the path they replace) gives +// the same value, since the conversion is exact. Strided views are included. +func TestFloat32ExtremaMatchTheFloat64Path(t *testing.T) { + nan := float32(math.NaN()) + for _, d := range [][]float32{ + {3}, {1, -2, 5.5, 0.25}, {-1, -7, float32(math.Copysign(0, -1)), -3, 0}, + {1, nan, 3}, {float32(math.Inf(-1)), -1e30, 1e-45}, + } { + a := mustArr(t, ok(FromSlice(d, len(d)))) + for _, v := range []*Array{a, mustArr(t, ok(a.Slice(Step(-1))))} { + mx, _ := v.Max() + mn, _ := v.Min() + wmx, _ := v.AsType(Float64).Max() + wmn, _ := v.AsType(Float64).Min() + if !sameFloat(mx, wmx) || !sameFloat(mn, wmn) { + t.Errorf("%v: Max=%v Min=%v, float64 path %v %v", d, mx, mn, wmx, wmn) + } + } + } + if _, err := mustArr(t, ok(ZerosOf(Float32, 0))).Max(); err == nil { + t.Error("Max of an empty float32 array: no error") + } + if _, err := mustArr(t, ok(ZerosOf(Float32, 0))).Min(); err == nil { + t.Error("Min of an empty float32 array: no error") + } +} + +// sameFloat is bit equality, any NaN matching any NaN. +func sameFloat(a, b float64) bool { + if a != a && b != b { + return true + } + return math.Float64bits(a) == math.Float64bits(b) +} + // TestExtremaOfSignedZeros: Max is +0 and Min is -0 over a mix of signed // zeros, whatever their order, as Go's max/min and math.Max/Min order them. // Until v0.19.1 amd64 returned the sign of whichever zero its kernel met last diff --git a/internal/kernels/asmgen/amd64/gen.go b/internal/kernels/asmgen/amd64/gen.go index 507541a..dda3fc6 100644 --- a/internal/kernels/asmgen/amd64/gen.go +++ b/internal/kernels/asmgen/amd64/gen.go @@ -55,6 +55,8 @@ func main() { f.Add(bin32Kernel("mul32SSE2", "MULPS", "MULSS")) f.Add(bin32Kernel("div32SSE2", "DIVPS", "DIVSS")) f.Add(sqrt32Kernel()) + f.Add(extremum32Kernel("max32SSE2", "MAXPS", "MAXSS")) + f.Add(extremum32Kernel("min32SSE2", "MINPS", "MINSS")) if err := os.WriteFile("sum_amd64.s", []byte(f.String()), 0o644); err != nil { fmt.Fprintln(os.Stderr, err) @@ -606,6 +608,58 @@ func gemmSSE2HalfKernel() *emit.Function { // nanScan emits: X9 = (reg unordered reg) ? all-ones : 0 ; X8 |= X9, i.e. it // ORs into the NaN-seen mask X8 a lane of all-ones wherever reg holds a NaN. +// extremum32Kernel builds max32SSE2/min32SSE2(a *float32, n int) float32, for +// n >= 1: the extremum over 16 floats per iteration in four XMM accumulators +// (MAXPS/MINPS), with X8 the OR of every CMPPS-unordered mask so a NaN anywhere +// gives NaN. MAXPS/MINPS return their source operand on a zero tie, so a zero +// result may carry the wrong sign; the Go wrapper hands that case to the scalar +// loop, as maxSIMD does. +func extremum32Kernel(name, vec, sca string) *emit.Function { + sig := amd64.Layout([]string{"a", "n"}, []amd64.Type{amd64.Ptr, amd64.Int64}, + []string{"ret"}, []amd64.Type{amd64.Float32}) + b := amd64.NewFunc(name, sig, 0) + b.LoadArg("a", "AX").LoadArg("n", "CX") + // Seed the accumulators with a[0] in every lane. + b.Raw("MOVSS (AX), X0").Raw("SHUFPS $0x00, X0, X0"). + Raw("MOVAPS X0, X1").Raw("MOVAPS X0, X2").Raw("MOVAPS X0, X3"). + Raw("XORPS X8, X8") + nanScan32(b, "X0") + b.Raw("xblock:").Raw("CMPQ CX, $16").Raw("JL xfold") + for r := 0; r < 4; r++ { + b.Raw("MOVUPS %d(AX), X%d", 16*r, 4+r) + } + for r := 0; r < 4; r++ { + nanScan32(b, fmt.Sprintf("X%d", 4+r)) + } + for r := 0; r < 4; r++ { + b.Raw("%s X%d, X%d", vec, 4+r, r) + } + b.Raw("ADDQ $64, AX").Raw("SUBQ $16, CX").Raw("JMP xblock") + // Fold the accumulators, then the four lanes, into X0's low lane. + b.Raw("xfold:"). + Raw("%s X1, X0", vec).Raw("%s X3, X2", vec).Raw("%s X2, X0", vec). + Raw("MOVAPS X0, X1").Raw("MOVHLPS X1, X1").Raw("%s X1, X0", vec). + Raw("MOVAPS X0, X1").Raw("SHUFPS $0x55, X1, X1").Raw("%s X1, X0", sca). + Raw("xtail:").Raw("TESTQ CX, CX").Raw("JZ xnan"). + Raw("MOVSS (AX), X4") + nanScan32(b, "X4") + b.Raw("%s X4, X0", sca). + Raw("ADDQ $4, AX").Raw("DECQ CX").Raw("JMP xtail"). + // Any lane of X8 set: some element was NaN. + Raw("xnan:").Raw("MOVMSKPS X8, DX").Raw("TESTQ DX, DX").Raw("JZ xdone"). + Raw("XORPS X1, X1").Raw("DIVSS X1, X1"). // 0/0 = NaN + Raw("MOVSS X1, X0"). + Raw("xdone:"). + StoreRet("X0", "ret"). + Ret() + return b.Func() +} + +// nanScan32 ORs the unordered (NaN) lanes of reg into X8. +func nanScan32(b *amd64.Builder, reg string) { + b.Raw("MOVAPS %s, X9", reg).Raw("CMPPS X9, X9, $3").Raw("ORPS X9, X8") +} + func nanScan(b *amd64.Builder, reg string) { b.Raw("MOVAPS " + reg + ", X9") b.Raw("CMPPD X9, X9, $3") diff --git a/internal/kernels/asmgen/arm64/gen.go b/internal/kernels/asmgen/arm64/gen.go index a1f9d10..d18d8a7 100644 --- a/internal/kernels/asmgen/arm64/gen.go +++ b/internal/kernels/asmgen/arm64/gen.go @@ -54,6 +54,8 @@ func main() { f.Add(sqrt32Kernel()) f.Add(extremumKernel("maxNEON", "VFMAX", "FMAXD")) f.Add(extremumKernel("minNEON", "VFMIN", "FMIND")) + f.Add(extremum32Kernel("max32NEON", "VFMAX", "FMAXS")) + f.Add(extremum32Kernel("min32NEON", "VFMIN", "FMINS")) f.Add(gemmKernel()) f.Add(blocks8Kernel()) if err := os.WriteFile("sum_arm64.s", []byte(f.String()), 0o644); err != nil { @@ -268,6 +270,44 @@ func extremumKernel(name, vop, sop string) *emit.Function { return b.Func() } +// extremum32Kernel builds NAME(a *float32, n int) float32, the maximum (or +// minimum) of a, for n >= 16: four .S4 accumulators seeded with the first 16 +// elements, folded together, then the four lanes, then the tail one at a time. +// VFMAX/VFMIN, like the scalar FMAXS/FMINS Go's builtin max/min compile to, +// return NaN when either operand is NaN and order -0 below +0, so the result is +// the sequential max's bit for bit (the operation is associative under them). +func extremum32Kernel(name, vop, sop string) *emit.Function { + sig := arm64.Layout([]string{"a", "n"}, []arm64.Type{arm64.Ptr, arm64.Int64}, + []string{"ret"}, []arm64.Type{arm64.Float32}) + b := arm64.NewFunc(name, sig, 0) + b.LoadArg("a", "R0").LoadArg("n", "R1"). + Raw("VLD1.P 64(R0), [V0.S4, V1.S4, V2.S4, V3.S4]"). + Raw("SUB $16, R1"). + Raw("block:").Raw("CMP $16, R1").Raw("BLT fold"). + Raw("VLD1.P 64(R0), [V4.S4, V5.S4, V6.S4, V7.S4]") + for r := 0; r < 4; r++ { + b.Raw(fmt.Sprintf("%s V%d.S4, V%d.S4, V%d.S4", vop, 4+r, r, r)) + } + b.Raw("SUB $16, R1").Raw("B block"). + Raw("fold:"). + Raw(vop+" V1.S4, V0.S4, V0.S4"). + Raw(vop+" V3.S4, V2.S4, V2.S4"). + Raw(vop+" V2.S4, V0.S4, V0.S4") + for l := 0; l < 4; l++ { + b.Raw(fmt.Sprintf("VMOV V0.S[%d], R%d", l, 3+l)) + } + b.Raw("FMOVS R3, F0").Raw("FMOVS R4, F9").Raw("FMOVS R5, F10").Raw("FMOVS R6, F11"). + Raw(sop+" F9, F0, F0").Raw(sop+" F11, F10, F10").Raw(sop+" F10, F0, F0"). + Raw("tail:").Raw("CBZ R1, done"). + Raw("FMOVS.P 4(R0), F10"). + Raw(sop+" F10, F0, F0"). + Raw("SUB $1, R1").Raw("B tail"). + Raw("done:"). + StoreRet("F0", "ret"). + Ret() + return b.Func() +} + // gemmSig is the micro-kernel signature // // gemmMicro4x8(kc int, pa, pb, c *float64, ldc int) diff --git a/internal/kernels/extrema32.go b/internal/kernels/extrema32.go new file mode 100644 index 0000000..585efca --- /dev/null +++ b/internal/kernels/extrema32.go @@ -0,0 +1,57 @@ +package kernels + +// Max32 returns the maximum of a (non-empty) as numpy.max computes it: NaN if +// any element is NaN, and -0 below +0 (Go's max). It reads the float32 +// elements directly. Converting to float64 first gives the same result (the +// conversion is exact) at the cost of a second array. +func Max32(a []float32) float32 { return max32(a) } + +// Min32 is the dual of Max32: NaN if any element is NaN, -0 below +0. +func Min32(a []float32) float32 { return min32(a) } + +// max32Go is the oracle for the float32 maximum, and the implementation where +// there is no kernel: four accumulators, so four independent chains, folded +// with the same NaN-propagating max. Max is associative under Go's rule, so +// the grouping does not change the result. +func max32Go(a []float32) float32 { + if len(a) < 4 { + m := a[0] + for _, v := range a[1:] { + m = max(m, v) + } + return m + } + m0, m1, m2, m3 := a[0], a[1], a[2], a[3] + s := a[4:] + for len(s) >= 4 { + m0, m1, m2, m3 = max(m0, s[0]), max(m1, s[1]), max(m2, s[2]), max(m3, s[3]) + s = s[4:] + } + m := max(max(m0, m1), max(m2, m3)) + for _, v := range s { + m = max(m, v) + } + return m +} + +// min32Go is the dual of max32Go. +func min32Go(a []float32) float32 { + if len(a) < 4 { + m := a[0] + for _, v := range a[1:] { + m = min(m, v) + } + return m + } + m0, m1, m2, m3 := a[0], a[1], a[2], a[3] + s := a[4:] + for len(s) >= 4 { + m0, m1, m2, m3 = min(m0, s[0]), min(m1, s[1]), min(m2, s[2]), min(m3, s[3]) + s = s[4:] + } + m := min(min(m0, m1), min(m2, m3)) + for _, v := range s { + m = min(m, v) + } + return m +} diff --git a/internal/kernels/extrema32_amd64.go b/internal/kernels/extrema32_amd64.go new file mode 100644 index 0000000..94ca93d --- /dev/null +++ b/internal/kernels/extrema32_amd64.go @@ -0,0 +1,27 @@ +package kernels + +// max32 runs the SSE2 kernel (sum_amd64.s: MAXPS, four accumulators, a +// CMPPS-unordered NaN mask). MAXPS returns its source operand on a zero tie, +// so a zero result may have the wrong sign: that case, which needs every +// element <= 0 with a zero among them, goes to the scalar loop (Go's max: +0 +// if any +0), as maxSIMD does for float64. +func max32(a []float32) float32 { + if r := max32SSE2(&a[0], len(a)); r != 0 { + return r + } + return max32Go(a) +} + +// min32 is the dual of max32 (a zero result: -0 if any -0). +func min32(a []float32) float32 { + if r := min32SSE2(&a[0], len(a)); r != 0 { + return r + } + return min32Go(a) +} + +//go:noescape +func max32SSE2(a *float32, n int) float32 + +//go:noescape +func min32SSE2(a *float32, n int) float32 diff --git a/internal/kernels/extrema32_arm64.go b/internal/kernels/extrema32_arm64.go new file mode 100644 index 0000000..90629e2 --- /dev/null +++ b/internal/kernels/extrema32_arm64.go @@ -0,0 +1,25 @@ +package kernels + +// max32 runs the NEON kernel (sum_arm64.s: VFMAX on .S4, four accumulators) +// from 16 elements up, the four-chain Go loop below that. VFMAX propagates NaN +// and orders -0 below +0, so no correction is needed. +func max32(a []float32) float32 { + if len(a) < 16 { + return max32Go(a) + } + return max32NEON(&a[0], len(a)) +} + +// min32 is the dual of max32, with VFMIN. +func min32(a []float32) float32 { + if len(a) < 16 { + return min32Go(a) + } + return min32NEON(&a[0], len(a)) +} + +//go:noescape +func max32NEON(a *float32, n int) float32 + +//go:noescape +func min32NEON(a *float32, n int) float32 diff --git a/internal/kernels/extrema32_generic.go b/internal/kernels/extrema32_generic.go new file mode 100644 index 0000000..69ec906 --- /dev/null +++ b/internal/kernels/extrema32_generic.go @@ -0,0 +1,9 @@ +//go:build !amd64 && !arm64 + +package kernels + +// The float32 extrema run the four-chain Go loops on the architectures +// without a vector kernel for them yet. + +func max32(a []float32) float32 { return max32Go(a) } +func min32(a []float32) float32 { return min32Go(a) } diff --git a/internal/kernels/extrema32_test.go b/internal/kernels/extrema32_test.go new file mode 100644 index 0000000..9381c77 --- /dev/null +++ b/internal/kernels/extrema32_test.go @@ -0,0 +1,82 @@ +package kernels + +import ( + "math" + "testing" +) + +// seqMax32 and seqMin32 are the plainest oracle: one sequential pass with Go's +// builtin max and min, which propagate NaN and order -0 below +0. +func seqMax32(a []float32) float32 { + m := a[0] + for _, v := range a[1:] { + m = max(m, v) + } + return m +} + +func seqMin32(a []float32) float32 { + m := a[0] + for _, v := range a[1:] { + m = min(m, v) + } + return m +} + +// TestExtrema32MatchSequential: Max32/Min32 (the kernel where there is one) +// and the four-chain Go loops agree with one sequential pass, bit for bit, at +// every length around the blocks, with NaN, signed zeros and infinities placed +// at every position. +func TestExtrema32MatchSequential(t *testing.T) { + nz := float32(math.Copysign(0, -1)) + inf := float32(math.Inf(1)) + nan := float32(math.NaN()) + check := func(name string, a []float32) { + t.Helper() + for _, f := range []struct { + name string + got, want float32 + }{ + {"Max32", Max32(a), seqMax32(a)}, + {"Min32", Min32(a), seqMin32(a)}, + {"max32Go", max32Go(a), seqMax32(a)}, + {"min32Go", min32Go(a), seqMin32(a)}, + } { + if !sameBits32(f.got, f.want) { + t.Fatalf("%s %s n=%d: %v, sequential %v", name, f.name, len(a), f.got, f.want) + } + } + } + for n := 1; n <= 70; n++ { + check("mixed", operands32(n, 5)) + for pos := 0; pos < n; pos++ { + for _, fill := range []float32{nz, 0, -1, 1} { + for _, v := range []float32{nz, 0, inf, -inf, nan} { + a := make([]float32, n) + for i := range a { + a[i] = fill + } + a[pos] = v + check("special", a) + } + } + } + } +} + +func TestExtrema32SignedZeros(t *testing.T) { + nz := float32(math.Copysign(0, -1)) + for n := 2; n <= 40; n++ { + for first := 0; first < 2; first++ { + a := make([]float32, n) + for i := range a { + if (i+first)%2 == 0 { + a[i] = nz + } + } + if mx, mn := Max32(a), Min32(a); math.Signbit(float64(mx)) || !math.Signbit(float64(mn)) { + t.Fatalf("n=%d first=%d: Max32=%v Min32=%v, want 0 and -0", n, first, mx, mn) + } + } + } +} diff --git a/internal/kernels/fence_test.go b/internal/kernels/fence_test.go index e6155f7..2ed7217 100644 --- a/internal/kernels/fence_test.go +++ b/internal/kernels/fence_test.go @@ -91,6 +91,7 @@ func TestBin32StaysInsideItsOperands(t *testing.T) { Bin32(op, dst, a, b) } Sqrt32(dst, a) + _, _ = Max32(a), Min32(a) } } } diff --git a/internal/kernels/sum_amd64.s b/internal/kernels/sum_amd64.s index 8edd087..1c55394 100644 --- a/internal/kernels/sum_amd64.s +++ b/internal/kernels/sum_amd64.s @@ -934,3 +934,143 @@ TEXT ·sqrt32SSE2(SB), NOSPLIT, $0-24 qdone: RET +TEXT ·max32SSE2(SB), NOSPLIT, $0-20 + MOVQ a+0(FP), AX + MOVQ n+8(FP), CX + MOVSS (AX), X0 + SHUFPS $0x00, X0, X0 + MOVAPS X0, X1 + MOVAPS X0, X2 + MOVAPS X0, X3 + XORPS X8, X8 + MOVAPS X0, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + xblock: + CMPQ CX, $16 + JL xfold + MOVUPS 0(AX), X4 + MOVUPS 16(AX), X5 + MOVUPS 32(AX), X6 + MOVUPS 48(AX), X7 + MOVAPS X4, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MOVAPS X5, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MOVAPS X6, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MOVAPS X7, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MAXPS X4, X0 + MAXPS X5, X1 + MAXPS X6, X2 + MAXPS X7, X3 + ADDQ $64, AX + SUBQ $16, CX + JMP xblock + xfold: + MAXPS X1, X0 + MAXPS X3, X2 + MAXPS X2, X0 + MOVAPS X0, X1 + MOVHLPS X1, X1 + MAXPS X1, X0 + MOVAPS X0, X1 + SHUFPS $0x55, X1, X1 + MAXSS X1, X0 + xtail: + TESTQ CX, CX + JZ xnan + MOVSS (AX), X4 + MOVAPS X4, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MAXSS X4, X0 + ADDQ $4, AX + DECQ CX + JMP xtail + xnan: + MOVMSKPS X8, DX + TESTQ DX, DX + JZ xdone + XORPS X1, X1 + DIVSS X1, X1 + MOVSS X1, X0 + xdone: + MOVSS X0, ret+16(FP) + RET + +TEXT ·min32SSE2(SB), NOSPLIT, $0-20 + MOVQ a+0(FP), AX + MOVQ n+8(FP), CX + MOVSS (AX), X0 + SHUFPS $0x00, X0, X0 + MOVAPS X0, X1 + MOVAPS X0, X2 + MOVAPS X0, X3 + XORPS X8, X8 + MOVAPS X0, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + xblock: + CMPQ CX, $16 + JL xfold + MOVUPS 0(AX), X4 + MOVUPS 16(AX), X5 + MOVUPS 32(AX), X6 + MOVUPS 48(AX), X7 + MOVAPS X4, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MOVAPS X5, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MOVAPS X6, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MOVAPS X7, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MINPS X4, X0 + MINPS X5, X1 + MINPS X6, X2 + MINPS X7, X3 + ADDQ $64, AX + SUBQ $16, CX + JMP xblock + xfold: + MINPS X1, X0 + MINPS X3, X2 + MINPS X2, X0 + MOVAPS X0, X1 + MOVHLPS X1, X1 + MINPS X1, X0 + MOVAPS X0, X1 + SHUFPS $0x55, X1, X1 + MINSS X1, X0 + xtail: + TESTQ CX, CX + JZ xnan + MOVSS (AX), X4 + MOVAPS X4, X9 + CMPPS X9, X9, $3 + ORPS X9, X8 + MINSS X4, X0 + ADDQ $4, AX + DECQ CX + JMP xtail + xnan: + MOVMSKPS X8, DX + TESTQ DX, DX + JZ xdone + XORPS X1, X1 + DIVSS X1, X1 + MOVSS X1, X0 + xdone: + MOVSS X0, ret+16(FP) + RET + diff --git a/internal/kernels/sum_arm64.s b/internal/kernels/sum_arm64.s index 0bdc449..fc8bb74 100644 --- a/internal/kernels/sum_arm64.s +++ b/internal/kernels/sum_arm64.s @@ -439,6 +439,86 @@ TEXT ·minNEON(SB), NOSPLIT, $0-24 FMOVD F0, ret+16(FP) RET +TEXT ·max32NEON(SB), NOSPLIT, $0-20 + MOVD a+0(FP), R0 + MOVD n+8(FP), R1 + VLD1.P 64(R0), [V0.S4, V1.S4, V2.S4, V3.S4] + SUB $16, R1 + block: + CMP $16, R1 + BLT fold + VLD1.P 64(R0), [V4.S4, V5.S4, V6.S4, V7.S4] + VFMAX V4.S4, V0.S4, V0.S4 + VFMAX V5.S4, V1.S4, V1.S4 + VFMAX V6.S4, V2.S4, V2.S4 + VFMAX V7.S4, V3.S4, V3.S4 + SUB $16, R1 + B block + fold: + VFMAX V1.S4, V0.S4, V0.S4 + VFMAX V3.S4, V2.S4, V2.S4 + VFMAX V2.S4, V0.S4, V0.S4 + VMOV V0.S[0], R3 + VMOV V0.S[1], R4 + VMOV V0.S[2], R5 + VMOV V0.S[3], R6 + FMOVS R3, F0 + FMOVS R4, F9 + FMOVS R5, F10 + FMOVS R6, F11 + FMAXS F9, F0, F0 + FMAXS F11, F10, F10 + FMAXS F10, F0, F0 + tail: + CBZ R1, done + FMOVS.P 4(R0), F10 + FMAXS F10, F0, F0 + SUB $1, R1 + B tail + done: + FMOVS F0, ret+16(FP) + RET + +TEXT ·min32NEON(SB), NOSPLIT, $0-20 + MOVD a+0(FP), R0 + MOVD n+8(FP), R1 + VLD1.P 64(R0), [V0.S4, V1.S4, V2.S4, V3.S4] + SUB $16, R1 + block: + CMP $16, R1 + BLT fold + VLD1.P 64(R0), [V4.S4, V5.S4, V6.S4, V7.S4] + VFMIN V4.S4, V0.S4, V0.S4 + VFMIN V5.S4, V1.S4, V1.S4 + VFMIN V6.S4, V2.S4, V2.S4 + VFMIN V7.S4, V3.S4, V3.S4 + SUB $16, R1 + B block + fold: + VFMIN V1.S4, V0.S4, V0.S4 + VFMIN V3.S4, V2.S4, V2.S4 + VFMIN V2.S4, V0.S4, V0.S4 + VMOV V0.S[0], R3 + VMOV V0.S[1], R4 + VMOV V0.S[2], R5 + VMOV V0.S[3], R6 + FMOVS R3, F0 + FMOVS R4, F9 + FMOVS R5, F10 + FMOVS R6, F11 + FMINS F9, F0, F0 + FMINS F11, F10, F10 + FMINS F10, F0, F0 + tail: + CBZ R1, done + FMOVS.P 4(R0), F10 + FMINS F10, F0, F0 + SUB $1, R1 + B tail + done: + FMOVS F0, ret+16(FP) + RET + TEXT ·gemmMicro4x8(SB), NOSPLIT, $0-40 MOVD kc+0(FP), R0 MOVD pa+8(FP), R1 diff --git a/ndarray.go b/ndarray.go index 6023627..3335650 100644 --- a/ndarray.go +++ b/ndarray.go @@ -877,6 +877,9 @@ func (a *Array) Mean() (float64, error) { // Max returns the maximum element. It returns an error for an empty array. // On a complex array it panics; use MaxAxis. func (a *Array) Max() (float64, error) { + if a.dtype == Float32 && a.Size() > 0 { + return float64(kernels.Max32(a.contiguousStore().([]float32))), nil + } if a.dtype != Float64 { a.realOnly("Max", "MaxAxis") return a.AsType(Float64).Max() @@ -890,6 +893,9 @@ func (a *Array) Max() (float64, error) { // Min returns the minimum element. It returns an error for an empty array. // On a complex array it panics; use MinAxis. func (a *Array) Min() (float64, error) { + if a.dtype == Float32 && a.Size() > 0 { + return float64(kernels.Min32(a.contiguousStore().([]float32))), nil + } if a.dtype != Float64 { a.realOnly("Min", "MinAxis") return a.AsType(Float64).Min()