From cc1683b0edfccdda712d5228438e49297f41aa8d Mon Sep 17 00:00:00 2001 From: tannevaled Date: Mon, 26 Oct 2026 14:10:14 +0100 Subject: [PATCH] perf: ppc64le Max/Min (float64, float32) with an integer NaN mask xvmaxdp/xvmindp (and sp) do not propagate NaN as Go/NumPy do, and Go's assembler has no VSX float compare. The kernel builds the NaN mask in integer arithmetic: a lane is NaN iff (x AND abs) >u +Inf's bits (vcmpgtud/vcmpgtuw against DATA constants), OR-accumulated. It reduces whole blocks into four accumulators folded into one stored vector, plus the NaN flag; Go folds the lanes and the tail with its max/min and sends a zero result to the scalar oracle for its sign. POWER9 (abbench): Max float64 -86% / -86%, float32 -91% / -91% (2^14 / 2^20), now faster than NumPy 2.5.3 on one core (7.28 vs 11.8 us and 0.472 vs 0.615 ms float64; 4.58 vs 7.13 us and 0.289 vs 0.324 ms float32). Sum controls flat. A sabotaged kernel (one NaN-mask OR removed) fails the float32 and float64 oracle tests on the POWER9. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- SECURITY.md | 8 + docs/perf.md | 28 +- docs/plan-ndarray.md | 6 +- internal/kernels/asmgen/ppc64le/gen.go | 91 +++++++ internal/kernels/extrema32_generic.go | 2 +- internal/kernels/extrema32_ppc64le.go | 52 ++++ internal/kernels/reduce_ppc64le.go | 55 +++- internal/kernels/sum_ppc64le.s | 360 +++++++++++++++++++++++++ 9 files changed, 593 insertions(+), 11 deletions(-) create mode 100644 internal/kernels/extrema32_ppc64le.go diff --git a/README.md b/README.md index d5540ca..d292a0e 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ `ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every - 64-bit target, and `Max`/`Min` on amd64, arm64, riscv64 and s390x, bit-identical to its + 64-bit target, and `Max`/`Min` on every 64-bit target but loong64, bit-identical to its scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64, rounded once) ([docs/design-dtypes.md](docs/design-dtypes.md)). diff --git a/SECURITY.md b/SECURITY.md index 2e0e318..f7ab405 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -66,6 +66,14 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none. ## Audit log +**2026-10-26 (v0.24.0).** The ppc64le extremum kernels (`maxVSX`, `minVSX`, +`max32VSX`, `min32VSX`) read the whole blocks of `a` (`len(a) &^ 7` or +`&^ 15`, never 0, since the wrapper takes the scalar path below a block). They +write one 16-byte vector to an `out` array on the wrapper's stack, and read +two 16-byte read-only constants. They are fenced and tested against the +sequential oracle on a POWER9, where a sabotaged kernel (one NaN-mask OR +removed) fails. + **2026-10-26 (v0.23.0).** The s390x float32 extremum kernels (`max32VXE`, `min32VXE`) read `len(a)` elements of a slice of at least 16, behind the `useVXE` gate. On a real IBM Z the PROT_NONE fence and the sequential-oracle diff --git a/docs/perf.md b/docs/perf.md index 20283dd..6b8ce57 100644 --- a/docs/perf.md +++ b/docs/perf.md @@ -1044,9 +1044,33 @@ On the POWER9 float32 `Max` is still about 7× slower than NumPy, because Go's NaN-correct `max` is scalar there. Still to do: -- ppc64le and loong64 need registry entries in go-asmgen first. Go's - assemblers have no VSX float compare (for the NaN mask), and on loong64 no +- ppc64le: done in v0.24.0, below, without a registry entry. +- loong64 needs registry entries in go-asmgen first: Go's assembler has no vector float max, min or compare at all. + +### ppc64le `Max`/`Min` (2026-10-26) + +`xvmaxdp`/`xvmindp` (and the `sp` forms) do not propagate NaN as Go and +NumPy do, and Go's assembler has no VSX float compare to build a NaN mask +from. The mask is built in integer arithmetic instead: + +- A lane is NaN iff its magnitude bits (`x AND 0x7FF…F`) exceed +Inf's bits + as an unsigned integer. `vcmpgtud`/`vcmpgtuw` test that on the VMX + registers, against constants loaded from a read-only DATA table, and the + results are OR-accumulated. +- The kernel works only vertically: four accumulators folded into one vector + stored to memory, plus the NaN flag. +- Go folds the lanes and the tail with its `max`/`min`. A zero result goes to + the scalar oracle for its sign, as on amd64. + +| POWER9, `Max` (`abbench`) | 2¹⁴ | 2²⁰ | NumPy 2.5.3, one core | +|---|---:|---:|---| +| float64 | 53.5 → **7.28 µs** (−86%) | 3.48 → **0.472 ms** (−86%) | 11.8 µs / 0.615 ms | +| float32 | 49.9 → **4.58 µs** (−91%) | 3.13 → **0.289 ms** (−91%) | 7.13 µs / 0.324 ms | + +`Max` is now faster than NumPy on the POWER9 for both widths, and the `Sum` +controls are flat. A sabotaged kernel, with one of the four NaN-mask ORs +removed, fails the float32 and float64 oracle tests there. - s390x: done in v0.23.0, below. ### s390x float32 `Max`/`Min`, and the z14 float32 kernels measured (2026-10-26) diff --git a/docs/plan-ndarray.md b/docs/plan-ndarray.md index bdd4f34..062e208 100644 --- a/docs/plan-ndarray.md +++ b/docs/plan-ndarray.md @@ -157,9 +157,9 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`). `fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for `Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go - for float32: `Max`/`Min` on ppc64le and loong64 (v0.20.0 vectorised them - on amd64 and arm64 and stopped converting to float64 everywhere, v0.22.0 - on riscv64 for float32 and float64, v0.23.0 on s390x), and the transcendental ufuncs. `MatMul` runs on the float64 + for float32: `Max`/`Min` on loong64 (v0.20.0 vectorised them on amd64 and + arm64 and stopped converting to float64 everywhere, v0.22.0 on riscv64, + v0.23.0 on s390x, v0.24.0 on ppc64le for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64 GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32 GEMM is still to write. - **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic, diff --git a/internal/kernels/asmgen/ppc64le/gen.go b/internal/kernels/asmgen/ppc64le/gen.go index cec69a0..b8cc686 100644 --- a/internal/kernels/asmgen/ppc64le/gen.go +++ b/internal/kernels/asmgen/ppc64le/gen.go @@ -43,6 +43,14 @@ func main() { f.Add(gemmKernel()) f.Add(blocks8Kernel()) f.Add(sqrt32Kernel()) + abs64 := f.Data("abs64", lanes(8, 0x7fffffffffffffff)) + inf64 := f.Data("inf64", lanes(8, 0x7ff0000000000000)) + abs32 := f.Data("abs32", lanes(4, 0x7fffffff)) + inf32 := f.Data("inf32", lanes(4, 0x7f800000)) + f.Add(extremumKernel("maxVSX", 64, true, abs64, inf64)) + f.Add(extremumKernel("minVSX", 64, false, abs64, inf64)) + f.Add(extremumKernel("max32VSX", 32, true, abs32, inf32)) + f.Add(extremumKernel("min32VSX", 32, false, abs32, inf32)) if err := os.WriteFile("sum_ppc64le.s", []byte(f.String()), 0o644); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) @@ -196,6 +204,89 @@ func binKernel(name, op, sca string) *emit.Function { return b.Func() } +// lanes is a 16-byte vector constant: v in every lane of size bytes, each lane +// little-endian, as LXVD2X/LXVW4X load it on ppc64le (every lane equal, so +// their lane order does not matter). +func lanes(size int, v uint64) []byte { + b := make([]byte, 0, 16) + for len(b) < 16 { + for i := 0; i < size; i++ { + b = append(b, byte(v>>(8*i))) + } + } + return b +} + +// extremumKernel builds NAME(a *floatN, n int, out *[16]byte) int64 for n a +// positive multiple of the block (8 doubles or 16 floats). It reduces a with +// xvmaxdp/xvmindp (xvmaxsp/xvminsp) into four accumulators, folds them +// vertically into one vector stored at out, and returns non-zero if any +// element was NaN. The Go wrapper folds the lanes and the tail with Go's +// max/min. So neither the instruction's NaN rule nor its sign of a zero tie +// matters: NaN is detected separately, and a zero result goes to the scalar +// oracle. +// +// NaN detection is integer arithmetic on the VMX registers, because Go's +// assembler has no VSX float compare: a lane is NaN iff its magnitude bits +// (x AND abs) exceed +Inf's bits as an unsigned integer (vcmpgtud/vcmpgtuw). +// The masks are OR-accumulated in V22. Registers: VS0-VS3 accumulators; +// V4-V7 (VS36-VS39) the block; V8-V11 scratch; V20, V21 the abs and +Inf +// constants; V22 the NaN mask. +func extremumKernel(name string, bits int, isMax bool, abs, inf string) *emit.Function { + sig := ppc64.Layout([]string{"a", "n", "out"}, + []ppc64.Type{ppc64.Ptr, ppc64.Int64, ppc64.Ptr}, []string{"ret"}, []ppc64.Type{ppc64.Int64}) + b := ppc64.NewFunc(name, sig, 0) + b.LoadArg("a", "R3").LoadArg("n", "R4").LoadArg("out", "R5") + offsets(b, false) + ld, st, cmp, block := "LXVD2X", "STXVD2X", "VCMPGTUD", 8 + op := b.XVMAXDP + if !isMax { + op = b.XVMINDP + } + if bits == 32 { + ld, st, cmp, block = "LXVW4X", "STXVW4X", "VCMPGTUW", 16 + op = b.XVMAXSP + if !isMax { + op = b.XVMINSP + } + } + b.Raw("MOVD $%s+0(SB), R7", abs).Raw("%s (R7)(R0), VS52", ld). + Raw("MOVD $%s+0(SB), R7", inf).Raw("%s (R7)(R0), VS53", ld). + Raw("XXLXOR VS54, VS54, VS54") + loadAndScan := func() { + for k := 0; k < 4; k++ { + b.Raw("%s (R3)(%s), VS%d", ld, vidx(k), 36+k) + } + for k := 0; k < 4; k++ { + b.Raw("VAND V%d, V20, V%d", 4+k, 8+k). + Raw("%s V%d, V21, V%d", cmp, 8+k, 8+k). // magnitude > +Inf: NaN + Raw("VOR V%d, V22, V22", 8+k) + } + b.Raw("ADD $64, R3").Raw("ADD $-%d, R4", block) + } + // The first block seeds the accumulators. + loadAndScan() + for k := 0; k < 4; k++ { + b.Raw("XXLOR VS%d, VS%d, VS%d", 36+k, 36+k, k) + } + b.Label(name + "loop").Raw("CMP R4, $0").Raw("BEQ %sfold", name) + loadAndScan() + for k := 0; k < 4; k++ { + op(k, k, 36+k) + } + b.Raw("BR %sloop", name) + b.Label(name + "fold") + op(0, 0, 1) + op(2, 2, 3) + op(0, 0, 2) + b.Raw("%s VS0, (R5)(R0)", st). + // The NaN mask's two doublewords, ORed into R6. + Raw("XXPERMDI VS54, VS54, $2, VS55"). + Raw("MFVSRD VS54, R6").Raw("MFVSRD VS55, R7").Raw("OR R7, R6"). + StoreRet("R6", "ret").Ret() + return b.Func() +} + // sqrt32Kernel builds sqrt32VSX(dst, src *float32, n int): xvsqrtsp over 16 // floats per iteration, FSQRTS for the tail; the correctly rounded float32 // square root, bit-identical to float32(math.Sqrt(float64(x))). diff --git a/internal/kernels/extrema32_generic.go b/internal/kernels/extrema32_generic.go index 7a7e558..f77b2a2 100644 --- a/internal/kernels/extrema32_generic.go +++ b/internal/kernels/extrema32_generic.go @@ -1,4 +1,4 @@ -//go:build !amd64 && !arm64 && !riscv64 && !s390x +//go:build !amd64 && !arm64 && !ppc64le && !riscv64 && !s390x package kernels diff --git a/internal/kernels/extrema32_ppc64le.go b/internal/kernels/extrema32_ppc64le.go new file mode 100644 index 0000000..302b186 --- /dev/null +++ b/internal/kernels/extrema32_ppc64le.go @@ -0,0 +1,52 @@ +package kernels + +import "math" + +// max32 reduces whole blocks of 16 with xvmaxsp (sum_ppc64le.s), then folds +// the four lanes and the tail with Go's max. As for float64 (maxSIMD), NaN is +// reported separately by an integer magnitude test, and a zero result goes to +// the scalar loop for its sign. +func max32(a []float32) float32 { + n := len(a) &^ 15 + if n == 0 { + return max32Go(a) + } + var out [4]float32 + if max32VSX(&a[0], n, &out) != 0 { + return float32(math.NaN()) + } + m := max(max(out[0], out[1]), max(out[2], out[3])) + for _, v := range a[n:] { + m = max(m, v) + } + if m == 0 { + return max32Go(a) + } + return m +} + +// min32 is the dual of max32, with xvminsp. +func min32(a []float32) float32 { + n := len(a) &^ 15 + if n == 0 { + return min32Go(a) + } + var out [4]float32 + if min32VSX(&a[0], n, &out) != 0 { + return float32(math.NaN()) + } + m := min(min(out[0], out[1]), min(out[2], out[3])) + for _, v := range a[n:] { + m = min(m, v) + } + if m == 0 { + return min32Go(a) + } + return m +} + +//go:noescape +func max32VSX(a *float32, n int, out *[4]float32) int64 + +//go:noescape +func min32VSX(a *float32, n int, out *[4]float32) int64 diff --git a/internal/kernels/reduce_ppc64le.go b/internal/kernels/reduce_ppc64le.go index 9f18d70..b91e779 100644 --- a/internal/kernels/reduce_ppc64le.go +++ b/internal/kernels/reduce_ppc64le.go @@ -1,5 +1,7 @@ package kernels +import "math" + // ppc64le VSX kernels generated by go-asmgen (sum_ppc64le.s). VSX is part of // the ppc64le baseline (POWER8, GOPPC64=power8), so they are always callable. // Go's ppc64 assembler has no vector double arithmetic; go-asmgen encodes it @@ -30,10 +32,55 @@ func sqrtSIMD(dst, src []float64) { sqrtVSX(&dst[0], &src[0], len(dst)) } -// max/min stay on the four-accumulator scalar reducers: XVMAXDP/XVMINDP do not -// propagate NaN the way Go's max/min and numpy.max do. -func maxSIMD(a []float64) float64 { return maxUnrolled(a) } -func minSIMD(a []float64) float64 { return minUnrolled(a) } +// maxSIMD / minSIMD reduce whole blocks of 8 with xvmaxdp/xvmindp +// (sum_ppc64le.s) and fold the two lanes and the tail with Go's max/min. +// XVMAXDP/XVMINDP do not propagate NaN as Go and numpy.max do, so the kernel +// reports NaN separately: an integer magnitude test, OR-accumulated. Their sign +// of a zero tie is not Go's either, so a zero result goes to the scalar +// reducer, as on amd64. +func maxSIMD(a []float64) float64 { + n := len(a) &^ 7 + if n == 0 { + return maxUnrolled(a) + } + var out [2]float64 + if maxVSX(&a[0], n, &out) != 0 { + return math.NaN() + } + m := max(out[0], out[1]) + for _, v := range a[n:] { + m = max(m, v) + } + if m == 0 { + return maxUnrolled(a) + } + return m +} + +func minSIMD(a []float64) float64 { + n := len(a) &^ 7 + if n == 0 { + return minUnrolled(a) + } + var out [2]float64 + if minVSX(&a[0], n, &out) != 0 { + return math.NaN() + } + m := min(out[0], out[1]) + for _, v := range a[n:] { + m = min(m, v) + } + if m == 0 { + return minUnrolled(a) + } + return m +} + +//go:noescape +func maxVSX(a *float64, n int, out *[2]float64) int64 + +//go:noescape +func minVSX(a *float64, n int, out *[2]float64) int64 // addBin, subBin, mulBin, divBin write dst[i] = a[i] OP b[i] with the VSX // kernels, each lane one IEEE operation, bit-identical to the scalar oracle. diff --git a/internal/kernels/sum_ppc64le.s b/internal/kernels/sum_ppc64le.s index 78b6b0b..fbdff56 100644 --- a/internal/kernels/sum_ppc64le.s +++ b/internal/kernels/sum_ppc64le.s @@ -897,3 +897,363 @@ qtail: qdone: RET +TEXT ·maxVSX(SB), NOSPLIT, $0-32 + MOVD a+0(FP), R3 + MOVD n+8(FP), R4 + MOVD out+16(FP), R5 + MOVD $16, R14 + MOVD $32, R15 + MOVD $48, R16 + MOVD $abs64<>+0(SB), R7 + LXVD2X (R7)(R0), VS52 + MOVD $inf64<>+0(SB), R7 + LXVD2X (R7)(R0), VS53 + XXLXOR VS54, VS54, VS54 + LXVD2X (R3)(R0), VS36 + LXVD2X (R3)(R14), VS37 + LXVD2X (R3)(R15), VS38 + LXVD2X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUD V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUD V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUD V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUD V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-8, R4 + XXLOR VS36, VS36, VS0 + XXLOR VS37, VS37, VS1 + XXLOR VS38, VS38, VS2 + XXLOR VS39, VS39, VS3 +maxVSXloop: + CMP R4, $0 + BEQ maxVSXfold + LXVD2X (R3)(R0), VS36 + LXVD2X (R3)(R14), VS37 + LXVD2X (R3)(R15), VS38 + LXVD2X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUD V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUD V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUD V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUD V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-8, R4 + WORD $0xf0002702 // XVMAXDP VS0, VS36, VS0 + WORD $0xf0212f02 // XVMAXDP VS1, VS37, VS1 + WORD $0xf0423702 // XVMAXDP VS2, VS38, VS2 + WORD $0xf0633f02 // XVMAXDP VS3, VS39, VS3 + BR maxVSXloop +maxVSXfold: + WORD $0xf0000f00 // XVMAXDP VS0, VS1, VS0 + WORD $0xf0421f00 // XVMAXDP VS2, VS3, VS2 + WORD $0xf0001700 // XVMAXDP VS0, VS2, VS0 + STXVD2X VS0, (R5)(R0) + XXPERMDI VS54, VS54, $2, VS55 + MFVSRD VS54, R6 + MFVSRD VS55, R7 + OR R7, R6 + MOVD R6, ret+24(FP) + RET + +TEXT ·minVSX(SB), NOSPLIT, $0-32 + MOVD a+0(FP), R3 + MOVD n+8(FP), R4 + MOVD out+16(FP), R5 + MOVD $16, R14 + MOVD $32, R15 + MOVD $48, R16 + MOVD $abs64<>+0(SB), R7 + LXVD2X (R7)(R0), VS52 + MOVD $inf64<>+0(SB), R7 + LXVD2X (R7)(R0), VS53 + XXLXOR VS54, VS54, VS54 + LXVD2X (R3)(R0), VS36 + LXVD2X (R3)(R14), VS37 + LXVD2X (R3)(R15), VS38 + LXVD2X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUD V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUD V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUD V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUD V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-8, R4 + XXLOR VS36, VS36, VS0 + XXLOR VS37, VS37, VS1 + XXLOR VS38, VS38, VS2 + XXLOR VS39, VS39, VS3 +minVSXloop: + CMP R4, $0 + BEQ minVSXfold + LXVD2X (R3)(R0), VS36 + LXVD2X (R3)(R14), VS37 + LXVD2X (R3)(R15), VS38 + LXVD2X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUD V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUD V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUD V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUD V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-8, R4 + WORD $0xf0002742 // XVMINDP VS0, VS36, VS0 + WORD $0xf0212f42 // XVMINDP VS1, VS37, VS1 + WORD $0xf0423742 // XVMINDP VS2, VS38, VS2 + WORD $0xf0633f42 // XVMINDP VS3, VS39, VS3 + BR minVSXloop +minVSXfold: + WORD $0xf0000f40 // XVMINDP VS0, VS1, VS0 + WORD $0xf0421f40 // XVMINDP VS2, VS3, VS2 + WORD $0xf0001740 // XVMINDP VS0, VS2, VS0 + STXVD2X VS0, (R5)(R0) + XXPERMDI VS54, VS54, $2, VS55 + MFVSRD VS54, R6 + MFVSRD VS55, R7 + OR R7, R6 + MOVD R6, ret+24(FP) + RET + +TEXT ·max32VSX(SB), NOSPLIT, $0-32 + MOVD a+0(FP), R3 + MOVD n+8(FP), R4 + MOVD out+16(FP), R5 + MOVD $16, R14 + MOVD $32, R15 + MOVD $48, R16 + MOVD $abs32<>+0(SB), R7 + LXVW4X (R7)(R0), VS52 + MOVD $inf32<>+0(SB), R7 + LXVW4X (R7)(R0), VS53 + XXLXOR VS54, VS54, VS54 + LXVW4X (R3)(R0), VS36 + LXVW4X (R3)(R14), VS37 + LXVW4X (R3)(R15), VS38 + LXVW4X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUW V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUW V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUW V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUW V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-16, R4 + XXLOR VS36, VS36, VS0 + XXLOR VS37, VS37, VS1 + XXLOR VS38, VS38, VS2 + XXLOR VS39, VS39, VS3 +max32VSXloop: + CMP R4, $0 + BEQ max32VSXfold + LXVW4X (R3)(R0), VS36 + LXVW4X (R3)(R14), VS37 + LXVW4X (R3)(R15), VS38 + LXVW4X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUW V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUW V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUW V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUW V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-16, R4 + WORD $0xf0002602 // XVMAXSP VS0, VS36, VS0 + WORD $0xf0212e02 // XVMAXSP VS1, VS37, VS1 + WORD $0xf0423602 // XVMAXSP VS2, VS38, VS2 + WORD $0xf0633e02 // XVMAXSP VS3, VS39, VS3 + BR max32VSXloop +max32VSXfold: + WORD $0xf0000e00 // XVMAXSP VS0, VS1, VS0 + WORD $0xf0421e00 // XVMAXSP VS2, VS3, VS2 + WORD $0xf0001600 // XVMAXSP VS0, VS2, VS0 + STXVW4X VS0, (R5)(R0) + XXPERMDI VS54, VS54, $2, VS55 + MFVSRD VS54, R6 + MFVSRD VS55, R7 + OR R7, R6 + MOVD R6, ret+24(FP) + RET + +TEXT ·min32VSX(SB), NOSPLIT, $0-32 + MOVD a+0(FP), R3 + MOVD n+8(FP), R4 + MOVD out+16(FP), R5 + MOVD $16, R14 + MOVD $32, R15 + MOVD $48, R16 + MOVD $abs32<>+0(SB), R7 + LXVW4X (R7)(R0), VS52 + MOVD $inf32<>+0(SB), R7 + LXVW4X (R7)(R0), VS53 + XXLXOR VS54, VS54, VS54 + LXVW4X (R3)(R0), VS36 + LXVW4X (R3)(R14), VS37 + LXVW4X (R3)(R15), VS38 + LXVW4X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUW V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUW V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUW V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUW V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-16, R4 + XXLOR VS36, VS36, VS0 + XXLOR VS37, VS37, VS1 + XXLOR VS38, VS38, VS2 + XXLOR VS39, VS39, VS3 +min32VSXloop: + CMP R4, $0 + BEQ min32VSXfold + LXVW4X (R3)(R0), VS36 + LXVW4X (R3)(R14), VS37 + LXVW4X (R3)(R15), VS38 + LXVW4X (R3)(R16), VS39 + VAND V4, V20, V8 + VCMPGTUW V8, V21, V8 + VOR V8, V22, V22 + VAND V5, V20, V9 + VCMPGTUW V9, V21, V9 + VOR V9, V22, V22 + VAND V6, V20, V10 + VCMPGTUW V10, V21, V10 + VOR V10, V22, V22 + VAND V7, V20, V11 + VCMPGTUW V11, V21, V11 + VOR V11, V22, V22 + ADD $64, R3 + ADD $-16, R4 + WORD $0xf0002642 // XVMINSP VS0, VS36, VS0 + WORD $0xf0212e42 // XVMINSP VS1, VS37, VS1 + WORD $0xf0423642 // XVMINSP VS2, VS38, VS2 + WORD $0xf0633e42 // XVMINSP VS3, VS39, VS3 + BR min32VSXloop +min32VSXfold: + WORD $0xf0000e40 // XVMINSP VS0, VS1, VS0 + WORD $0xf0421e40 // XVMINSP VS2, VS3, VS2 + WORD $0xf0001640 // XVMINSP VS0, VS2, VS0 + STXVW4X VS0, (R5)(R0) + XXPERMDI VS54, VS54, $2, VS55 + MFVSRD VS54, R6 + MFVSRD VS55, R7 + OR R7, R6 + MOVD R6, ret+24(FP) + RET + +DATA abs64<>+0(SB)/1, $0xff +DATA abs64<>+1(SB)/1, $0xff +DATA abs64<>+2(SB)/1, $0xff +DATA abs64<>+3(SB)/1, $0xff +DATA abs64<>+4(SB)/1, $0xff +DATA abs64<>+5(SB)/1, $0xff +DATA abs64<>+6(SB)/1, $0xff +DATA abs64<>+7(SB)/1, $0x7f +DATA abs64<>+8(SB)/1, $0xff +DATA abs64<>+9(SB)/1, $0xff +DATA abs64<>+10(SB)/1, $0xff +DATA abs64<>+11(SB)/1, $0xff +DATA abs64<>+12(SB)/1, $0xff +DATA abs64<>+13(SB)/1, $0xff +DATA abs64<>+14(SB)/1, $0xff +DATA abs64<>+15(SB)/1, $0x7f +GLOBL abs64<>(SB), RODATA|NOPTR, $16 + +DATA inf64<>+0(SB)/1, $0x00 +DATA inf64<>+1(SB)/1, $0x00 +DATA inf64<>+2(SB)/1, $0x00 +DATA inf64<>+3(SB)/1, $0x00 +DATA inf64<>+4(SB)/1, $0x00 +DATA inf64<>+5(SB)/1, $0x00 +DATA inf64<>+6(SB)/1, $0xf0 +DATA inf64<>+7(SB)/1, $0x7f +DATA inf64<>+8(SB)/1, $0x00 +DATA inf64<>+9(SB)/1, $0x00 +DATA inf64<>+10(SB)/1, $0x00 +DATA inf64<>+11(SB)/1, $0x00 +DATA inf64<>+12(SB)/1, $0x00 +DATA inf64<>+13(SB)/1, $0x00 +DATA inf64<>+14(SB)/1, $0xf0 +DATA inf64<>+15(SB)/1, $0x7f +GLOBL inf64<>(SB), RODATA|NOPTR, $16 + +DATA abs32<>+0(SB)/1, $0xff +DATA abs32<>+1(SB)/1, $0xff +DATA abs32<>+2(SB)/1, $0xff +DATA abs32<>+3(SB)/1, $0x7f +DATA abs32<>+4(SB)/1, $0xff +DATA abs32<>+5(SB)/1, $0xff +DATA abs32<>+6(SB)/1, $0xff +DATA abs32<>+7(SB)/1, $0x7f +DATA abs32<>+8(SB)/1, $0xff +DATA abs32<>+9(SB)/1, $0xff +DATA abs32<>+10(SB)/1, $0xff +DATA abs32<>+11(SB)/1, $0x7f +DATA abs32<>+12(SB)/1, $0xff +DATA abs32<>+13(SB)/1, $0xff +DATA abs32<>+14(SB)/1, $0xff +DATA abs32<>+15(SB)/1, $0x7f +GLOBL abs32<>(SB), RODATA|NOPTR, $16 + +DATA inf32<>+0(SB)/1, $0x00 +DATA inf32<>+1(SB)/1, $0x00 +DATA inf32<>+2(SB)/1, $0x80 +DATA inf32<>+3(SB)/1, $0x7f +DATA inf32<>+4(SB)/1, $0x00 +DATA inf32<>+5(SB)/1, $0x00 +DATA inf32<>+6(SB)/1, $0x80 +DATA inf32<>+7(SB)/1, $0x7f +DATA inf32<>+8(SB)/1, $0x00 +DATA inf32<>+9(SB)/1, $0x00 +DATA inf32<>+10(SB)/1, $0x80 +DATA inf32<>+11(SB)/1, $0x7f +DATA inf32<>+12(SB)/1, $0x00 +DATA inf32<>+13(SB)/1, $0x00 +DATA inf32<>+14(SB)/1, $0x80 +DATA inf32<>+15(SB)/1, $0x7f +GLOBL inf32<>(SB), RODATA|NOPTR, $16 +