Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
`ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, and `Max`/`Min` on amd64, arm64, riscv64 and s390x, bit-identical to its
64-bit target, and `Max`/`Min` on every 64-bit target but loong64, bit-identical to its
scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64,
rounded once)
([docs/design-dtypes.md](docs/design-dtypes.md)).
Expand Down
8 changes: 8 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-26 (v0.24.0).** The ppc64le extremum kernels (`maxVSX`, `minVSX`,
`max32VSX`, `min32VSX`) read the whole blocks of `a` (`len(a) &^ 7` or
`&^ 15`, never 0, since the wrapper takes the scalar path below a block). They
write one 16-byte vector to an `out` array on the wrapper's stack, and read
two 16-byte read-only constants. They are fenced and tested against the
sequential oracle on a POWER9, where a sabotaged kernel (one NaN-mask OR
removed) fails.

**2026-10-26 (v0.23.0).** The s390x float32 extremum kernels (`max32VXE`,
`min32VXE`) read `len(a)` elements of a slice of at least 16, behind the
`useVXE` gate. On a real IBM Z the PROT_NONE fence and the sequential-oracle
Expand Down
28 changes: 26 additions & 2 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1044,9 +1044,33 @@ On the POWER9 float32 `Max` is still about 7× slower than NumPy, because
Go's NaN-correct `max` is scalar there.

Still to do:
- ppc64le and loong64 need registry entries in go-asmgen first. Go's
assemblers have no VSX float compare (for the NaN mask), and on loong64 no
- ppc64le: done in v0.24.0, below, without a registry entry.
- loong64 needs registry entries in go-asmgen first: Go's assembler has no
vector float max, min or compare at all.

### ppc64le `Max`/`Min` (2026-10-26)

`xvmaxdp`/`xvmindp` (and the `sp` forms) do not propagate NaN as Go and
NumPy do, and Go's assembler has no VSX float compare to build a NaN mask
from. The mask is built in integer arithmetic instead:

- A lane is NaN iff its magnitude bits (`x AND 0x7FF…F`) exceed +Inf's bits
as an unsigned integer. `vcmpgtud`/`vcmpgtuw` test that on the VMX
registers, against constants loaded from a read-only DATA table, and the
results are OR-accumulated.
- The kernel works only vertically: four accumulators folded into one vector
stored to memory, plus the NaN flag.
- Go folds the lanes and the tail with its `max`/`min`. A zero result goes to
the scalar oracle for its sign, as on amd64.

| POWER9, `Max` (`abbench`) | 2¹⁴ | 2²⁰ | NumPy 2.5.3, one core |
|---|---:|---:|---|
| float64 | 53.5 → **7.28 µs** (−86%) | 3.48 → **0.472 ms** (−86%) | 11.8 µs / 0.615 ms |
| float32 | 49.9 → **4.58 µs** (−91%) | 3.13 → **0.289 ms** (−91%) | 7.13 µs / 0.324 ms |

`Max` is now faster than NumPy on the POWER9 for both widths, and the `Sum`
controls are flat. A sabotaged kernel, with one of the four NaN-mask ORs
removed, fails the float32 and float64 oracle tests there.
- s390x: done in v0.23.0, below.

### s390x float32 `Max`/`Min`, and the z14 float32 kernels measured (2026-10-26)
Expand Down
6 changes: 3 additions & 3 deletions docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,9 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
`fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has
SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for
`Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go
for float32: `Max`/`Min` on ppc64le and loong64 (v0.20.0 vectorised them
on amd64 and arm64 and stopped converting to float64 everywhere, v0.22.0
on riscv64 for float32 and float64, v0.23.0 on s390x), and the transcendental ufuncs. `MatMul` runs on the float64
for float32: `Max`/`Min` on loong64 (v0.20.0 vectorised them on amd64 and
arm64 and stopped converting to float64 everywhere, v0.22.0 on riscv64,
v0.23.0 on s390x, v0.24.0 on ppc64le for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32
GEMM is still to write.
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
Expand Down
91 changes: 91 additions & 0 deletions internal/kernels/asmgen/ppc64le/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ func main() {
f.Add(gemmKernel())
f.Add(blocks8Kernel())
f.Add(sqrt32Kernel())
abs64 := f.Data("abs64", lanes(8, 0x7fffffffffffffff))
inf64 := f.Data("inf64", lanes(8, 0x7ff0000000000000))
abs32 := f.Data("abs32", lanes(4, 0x7fffffff))
inf32 := f.Data("inf32", lanes(4, 0x7f800000))
f.Add(extremumKernel("maxVSX", 64, true, abs64, inf64))
f.Add(extremumKernel("minVSX", 64, false, abs64, inf64))
f.Add(extremumKernel("max32VSX", 32, true, abs32, inf32))
f.Add(extremumKernel("min32VSX", 32, false, abs32, inf32))
if err := os.WriteFile("sum_ppc64le.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
Expand Down Expand Up @@ -196,6 +204,89 @@ func binKernel(name, op, sca string) *emit.Function {
return b.Func()
}

// lanes is a 16-byte vector constant: v in every lane of size bytes, each lane
// little-endian, as LXVD2X/LXVW4X load it on ppc64le (every lane equal, so
// their lane order does not matter).
func lanes(size int, v uint64) []byte {
b := make([]byte, 0, 16)
for len(b) < 16 {
for i := 0; i < size; i++ {
b = append(b, byte(v>>(8*i)))
}
}
return b
}

// extremumKernel builds NAME(a *floatN, n int, out *[16]byte) int64 for n a
// positive multiple of the block (8 doubles or 16 floats). It reduces a with
// xvmaxdp/xvmindp (xvmaxsp/xvminsp) into four accumulators, folds them
// vertically into one vector stored at out, and returns non-zero if any
// element was NaN. The Go wrapper folds the lanes and the tail with Go's
// max/min. So neither the instruction's NaN rule nor its sign of a zero tie
// matters: NaN is detected separately, and a zero result goes to the scalar
// oracle.
//
// NaN detection is integer arithmetic on the VMX registers, because Go's
// assembler has no VSX float compare: a lane is NaN iff its magnitude bits
// (x AND abs) exceed +Inf's bits as an unsigned integer (vcmpgtud/vcmpgtuw).
// The masks are OR-accumulated in V22. Registers: VS0-VS3 accumulators;
// V4-V7 (VS36-VS39) the block; V8-V11 scratch; V20, V21 the abs and +Inf
// constants; V22 the NaN mask.
func extremumKernel(name string, bits int, isMax bool, abs, inf string) *emit.Function {
sig := ppc64.Layout([]string{"a", "n", "out"},
[]ppc64.Type{ppc64.Ptr, ppc64.Int64, ppc64.Ptr}, []string{"ret"}, []ppc64.Type{ppc64.Int64})
b := ppc64.NewFunc(name, sig, 0)
b.LoadArg("a", "R3").LoadArg("n", "R4").LoadArg("out", "R5")
offsets(b, false)
ld, st, cmp, block := "LXVD2X", "STXVD2X", "VCMPGTUD", 8
op := b.XVMAXDP
if !isMax {
op = b.XVMINDP
}
if bits == 32 {
ld, st, cmp, block = "LXVW4X", "STXVW4X", "VCMPGTUW", 16
op = b.XVMAXSP
if !isMax {
op = b.XVMINSP
}
}
b.Raw("MOVD $%s+0(SB), R7", abs).Raw("%s (R7)(R0), VS52", ld).
Raw("MOVD $%s+0(SB), R7", inf).Raw("%s (R7)(R0), VS53", ld).
Raw("XXLXOR VS54, VS54, VS54")
loadAndScan := func() {
for k := 0; k < 4; k++ {
b.Raw("%s (R3)(%s), VS%d", ld, vidx(k), 36+k)
}
for k := 0; k < 4; k++ {
b.Raw("VAND V%d, V20, V%d", 4+k, 8+k).
Raw("%s V%d, V21, V%d", cmp, 8+k, 8+k). // magnitude > +Inf: NaN
Raw("VOR V%d, V22, V22", 8+k)
}
b.Raw("ADD $64, R3").Raw("ADD $-%d, R4", block)
}
// The first block seeds the accumulators.
loadAndScan()
for k := 0; k < 4; k++ {
b.Raw("XXLOR VS%d, VS%d, VS%d", 36+k, 36+k, k)
}
b.Label(name + "loop").Raw("CMP R4, $0").Raw("BEQ %sfold", name)
loadAndScan()
for k := 0; k < 4; k++ {
op(k, k, 36+k)
}
b.Raw("BR %sloop", name)
b.Label(name + "fold")
op(0, 0, 1)
op(2, 2, 3)
op(0, 0, 2)
b.Raw("%s VS0, (R5)(R0)", st).
// The NaN mask's two doublewords, ORed into R6.
Raw("XXPERMDI VS54, VS54, $2, VS55").
Raw("MFVSRD VS54, R6").Raw("MFVSRD VS55, R7").Raw("OR R7, R6").
StoreRet("R6", "ret").Ret()
return b.Func()
}

// sqrt32Kernel builds sqrt32VSX(dst, src *float32, n int): xvsqrtsp over 16
// floats per iteration, FSQRTS for the tail; the correctly rounded float32
// square root, bit-identical to float32(math.Sqrt(float64(x))).
Expand Down
2 changes: 1 addition & 1 deletion internal/kernels/extrema32_generic.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build !amd64 && !arm64 && !riscv64 && !s390x
//go:build !amd64 && !arm64 && !ppc64le && !riscv64 && !s390x

package kernels

Expand Down
52 changes: 52 additions & 0 deletions internal/kernels/extrema32_ppc64le.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
package kernels

import "math"

// max32 reduces whole blocks of 16 with xvmaxsp (sum_ppc64le.s), then folds
// the four lanes and the tail with Go's max. As for float64 (maxSIMD), NaN is
// reported separately by an integer magnitude test, and a zero result goes to
// the scalar loop for its sign.
func max32(a []float32) float32 {
n := len(a) &^ 15
if n == 0 {
return max32Go(a)
}
var out [4]float32
if max32VSX(&a[0], n, &out) != 0 {
return float32(math.NaN())
}
m := max(max(out[0], out[1]), max(out[2], out[3]))
for _, v := range a[n:] {
m = max(m, v)
}
if m == 0 {
return max32Go(a)
}
return m
}

// min32 is the dual of max32, with xvminsp.
func min32(a []float32) float32 {
n := len(a) &^ 15
if n == 0 {
return min32Go(a)
}
var out [4]float32
if min32VSX(&a[0], n, &out) != 0 {
return float32(math.NaN())
}
m := min(min(out[0], out[1]), min(out[2], out[3]))
for _, v := range a[n:] {
m = min(m, v)
}
if m == 0 {
return min32Go(a)
}
return m
}

//go:noescape
func max32VSX(a *float32, n int, out *[4]float32) int64

//go:noescape
func min32VSX(a *float32, n int, out *[4]float32) int64
55 changes: 51 additions & 4 deletions internal/kernels/reduce_ppc64le.go
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package kernels

import "math"

// ppc64le VSX kernels generated by go-asmgen (sum_ppc64le.s). VSX is part of
// the ppc64le baseline (POWER8, GOPPC64=power8), so they are always callable.
// Go's ppc64 assembler has no vector double arithmetic; go-asmgen encodes it
Expand Down Expand Up @@ -30,10 +32,55 @@ func sqrtSIMD(dst, src []float64) {
sqrtVSX(&dst[0], &src[0], len(dst))
}

// max/min stay on the four-accumulator scalar reducers: XVMAXDP/XVMINDP do not
// propagate NaN the way Go's max/min and numpy.max do.
func maxSIMD(a []float64) float64 { return maxUnrolled(a) }
func minSIMD(a []float64) float64 { return minUnrolled(a) }
// maxSIMD / minSIMD reduce whole blocks of 8 with xvmaxdp/xvmindp
// (sum_ppc64le.s) and fold the two lanes and the tail with Go's max/min.
// XVMAXDP/XVMINDP do not propagate NaN as Go and numpy.max do, so the kernel
// reports NaN separately: an integer magnitude test, OR-accumulated. Their sign
// of a zero tie is not Go's either, so a zero result goes to the scalar
// reducer, as on amd64.
func maxSIMD(a []float64) float64 {
n := len(a) &^ 7
if n == 0 {
return maxUnrolled(a)
}
var out [2]float64
if maxVSX(&a[0], n, &out) != 0 {
return math.NaN()
}
m := max(out[0], out[1])
for _, v := range a[n:] {
m = max(m, v)
}
if m == 0 {
return maxUnrolled(a)
}
return m
}

func minSIMD(a []float64) float64 {
n := len(a) &^ 7
if n == 0 {
return minUnrolled(a)
}
var out [2]float64
if minVSX(&a[0], n, &out) != 0 {
return math.NaN()
}
m := min(out[0], out[1])
for _, v := range a[n:] {
m = min(m, v)
}
if m == 0 {
return minUnrolled(a)
}
return m
}

//go:noescape
func maxVSX(a *float64, n int, out *[2]float64) int64

//go:noescape
func minVSX(a *float64, n int, out *[2]float64) int64

// addBin, subBin, mulBin, divBin write dst[i] = a[i] OP b[i] with the VSX
// kernels, each lane one IEEE operation, bit-identical to the scalar oracle.
Expand Down
Loading
Loading