Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
`ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, and `Max`/`Min` on every 64-bit target but loong64, bit-identical to its
64-bit target, and `Max`/`Min` on every 64-bit target, bit-identical to its
scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64,
rounded once)
([docs/design-dtypes.md](docs/design-dtypes.md)).
Expand Down
8 changes: 8 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-26 (v0.25.0).** The loong64 extremum kernels (`maxLSX`, `minLSX`,
`max32LSX`, `min32LSX`) read the whole blocks of `a` behind the `useLASX`
gate. They write one 16-byte vector to an `out` array on the wrapper's stack,
and read two 16-byte read-only constants. A test gap was found and closed: no
extremum test used a negative NaN, so a kernel whose NaN test failed could
pass, as a sabotaged one did. Both signs are tested now. The sabotage then
failed CI's loong64 lane, and every other kernel passed.

**2026-10-26 (v0.24.0).** The ppc64le extremum kernels (`maxVSX`, `minVSX`,
`max32VSX`, `min32VSX`) read the whole blocks of `a` (`len(a) &^ 7` or
`&^ 15`, never 0, since the wrapper takes the scalar path below a block). They
Expand Down
34 changes: 32 additions & 2 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1045,8 +1045,38 @@ Go's NaN-correct `max` is scalar there.

Still to do:
- ppc64le: done in v0.24.0, below, without a registry entry.
- loong64 needs registry entries in go-asmgen first: Go's assembler has no
vector float max, min or compare at all.
- loong64: done in v0.25.0, below, without a registry entry.

### loong64 `Max`/`Min`, and negative NaNs (2026-10-26)

Go's loong64 assembler has no vector float max, min or compare, so the LSX
kernels work entirely in integer arithmetic:

- **Order:** `t = x XOR ((x >>a N−1) >>l 1)` flips the magnitude bits of the
negatives. The signed order of `t` is the float order, with −0 below +0
(Go's rule), and the map is its own inverse.
- **Extremum:** `max(acc, t) = acc XOR ((acc XOR t) AND (acc < t))`, with
`vslt`.
- **NaN:** a lane is NaN iff `(x AND abs) >u +Inf's bits` (`vsltu`),
OR-accumulated.

**What the sabotage run found.** A sabotaged kernel, with the NaN mask of
`max32LSX`'s first lane dropped, *passed* every test in CI's loong64 lane. In
that integer order a **positive** NaN sorts above +Inf, so a max finds it
without any NaN test. Every NaN in the tests was `math.NaN()`, which is
positive. A **negative** NaN sorts below everything, and only the NaN test
catches it. That is no corner case: x86's default NaN (from `0×∞` or `∞−∞`)
is negative.

The extremum tests now cover both signs, for float32 and float64 on every
architecture. With them the sabotage failed CI's loong64 lane
(`special Max32 n=16: -0, sequential NaN`), and every other architecture's
kernels passed.

**Validated under emulation only.** The kernels pass on qemu `la464` (LSX and
LASX, required since #67). The Loongson they would be measured on
(cfarm401) accepted TCP but never sent its SSH banner on 2026-10-26, so their
speed is not measured yet.

### ppc64le `Max`/`Min` (2026-10-26)

Expand Down
5 changes: 2 additions & 3 deletions docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,8 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
`fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has
SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for
`Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go
for float32: `Max`/`Min` on loong64 (v0.20.0 vectorised them on amd64 and
arm64 and stopped converting to float64 everywhere, v0.22.0 on riscv64,
v0.23.0 on s390x, v0.24.0 on ppc64le for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
for float32: nothing in `Max`/`Min` any more (v0.20.0 to v0.25.0
vectorised them on all six 64-bit targets, for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32
GEMM is still to write.
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
Expand Down
98 changes: 98 additions & 0 deletions internal/kernels/asmgen/loong64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ func main() {
f.Add(gemmKernel())
f.Add(blocks8Kernel())
f.Add(sqrt32Kernel())
abs64 := f.Data("abs64", lanes(8, 0x7fffffffffffffff))
inf64 := f.Data("inf64", lanes(8, 0x7ff0000000000000))
abs32 := f.Data("abs32", lanes(4, 0x7fffffff))
inf32 := f.Data("inf32", lanes(4, 0x7f800000))
f.Add(extremumKernel("maxLSX", 64, true, abs64, inf64))
f.Add(extremumKernel("minLSX", 64, false, abs64, inf64))
f.Add(extremumKernel("max32LSX", 32, true, abs32, inf32))
f.Add(extremumKernel("min32LSX", 32, false, abs32, inf32))
if err := os.WriteFile("sum_loong64.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
Expand Down Expand Up @@ -171,6 +179,96 @@ func binKernel(name, vec, sca string) *emit.Function {
return b.Func()
}

// lanes is a 16-byte vector constant: v in every lane of size bytes, each lane
// little-endian.
func lanes(size int, v uint64) []byte {
b := make([]byte, 0, 16)
for len(b) < 16 {
for i := 0; i < size; i++ {
b = append(b, byte(v>>(8*i)))
}
}
return b
}

// extremumKernel builds NAME(a *floatN, n int, out *[16]byte) int64 for n a
// positive multiple of the block (8 doubles or 16 floats), entirely in LSX
// integer arithmetic: Go's loong64 assembler has no vector float max, min or
// compare.
//
// - Order: t = x XOR ((x >>a N-1) >>l 1) flips the magnitude bits of the
// negatives, so the signed order of t is the float order, with -0 below
// +0, Go's rule. The map is its own inverse.
// - Extremum: the max of t and acc is acc XOR ((acc XOR t) AND (acc < t))
// (vslt.d/vslt.w), and the min swaps the compare.
// - NaN: a lane is NaN iff its magnitude bits (x AND abs) exceed +Inf's
// bits as an unsigned integer (vsltu). The masks are OR-accumulated, and
// a non-zero result is returned so the wrapper can answer NaN.
//
// Four accumulators (V0-V3) are folded the same way into one vector, which is
// stored, still in the t order, at out. The Go wrapper maps it back and folds
// the lanes and the tail with Go's max/min. Registers: V4-V7 the block,
// V8-V19 scratch, V20 abs, V21 +Inf, V22 the NaN mask.
func extremumKernel(name string, bits int, isMax bool, abs, inf string) *emit.Function {
sig := loong64.Layout([]string{"a", "n", "out"},
[]loong64.Type{loong64.Ptr, loong64.Int64, loong64.Ptr}, []string{"ret"}, []loong64.Type{loong64.Int64})
b := loong64.NewFunc(name, sig, 0)
b.LoadArg("a", "R4").LoadArg("n", "R5").LoadArg("out", "R6")
w, block := "V", 8
if bits == 32 {
w, block = "W", 16
}
sra, srl, slt, sltu := "VSRA"+w, "VSRL"+w, "VSLT"+w, "VSLT"+w+"U"
b.Raw("MOVV $%s+0(SB), R12", abs).Raw("VMOVQ (R12), V20").
Raw("MOVV $%s+0(SB), R12", inf).Raw("VMOVQ (R12), V21").
Raw("VXORV V22, V22, V22")
// sel sets acc to the extremum of acc and t, with m and d as scratch.
sel := func(acc, t, m, d int) {
if isMax {
b.Raw("%s V%d, V%d, V%d", slt, t, acc, m) // m = acc < t
} else {
b.Raw("%s V%d, V%d, V%d", slt, acc, t, m) // m = t < acc
}
b.Raw("VXORV V%d, V%d, V%d", t, acc, d).
Raw("VANDV V%d, V%d, V%d", m, d, d).
Raw("VXORV V%d, V%d, V%d", d, acc, acc)
}
// load reads one block into V4-V7, ORs its NaN lanes into V22 and leaves
// the ordered integers in V8-V11.
load := func() {
for k := 0; k < 4; k++ {
b.Raw("VMOVQ %d(R4), V%d", 16*k, 4+k)
}
for k := 0; k < 4; k++ {
b.Raw("VANDV V20, V%d, V%d", 4+k, 12+k).
Raw("%s V%d, V21, V%d", sltu, 12+k, 12+k). // +Inf < |x|: NaN
Raw("VORV V%d, V22, V22", 12+k).
Raw("%s $%d, V%d, V%d", sra, bits-1, 4+k, 8+k).
Raw("%s $1, V%d, V%d", srl, 8+k, 8+k).
Raw("VXORV V%d, V%d, V%d", 4+k, 8+k, 8+k)
}
b.Raw("ADDV $64, R4").Raw("ADDV $-%d, R5", block)
}
load()
for k := 0; k < 4; k++ {
b.Raw("VORV V%d, V%d, V%d", 8+k, 8+k, k) // the first block seeds acc
}
b.Label(name+"loop").Raw("BEQ R5, %sfold", name)
load()
for k := 0; k < 4; k++ {
sel(k, 8+k, 12+k, 16+k)
}
b.Raw("JMP %sloop", name)
b.Label(name + "fold")
sel(0, 1, 12, 16)
sel(2, 3, 13, 17)
sel(0, 2, 14, 18)
b.Raw("VMOVQ V0, (R6)").
Raw("VMOVQ V22.V[0], R13").Raw("VMOVQ V22.V[1], R14").Raw("OR R14, R13").
StoreRet("R13", "ret").Ret()
return b.Func()
}

// sqrt32Kernel builds sqrt32LSX(dst, src *float32, n int): vfsqrt.s over four
// 128-bit registers, 16 floats per iteration, SQRTF for the tail; the
// correctly rounded float32 square root, bit-identical to
Expand Down
2 changes: 1 addition & 1 deletion internal/kernels/extrema32_generic.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build !amd64 && !arm64 && !ppc64le && !riscv64 && !s390x
//go:build !amd64 && !arm64 && !loong64 && !ppc64le && !riscv64 && !s390x

package kernels

Expand Down
50 changes: 50 additions & 0 deletions internal/kernels/extrema32_loong64.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
package kernels

import "math"

// max32 reduces whole blocks of 16 with the LSX kernel on the floats' bits in
// integer order (see fromOrdered32), then folds the four lanes and the tail
// with Go's max. A NaN anywhere, reported by the kernel, is NaN.
func max32(a []float32) float32 {
n := len(a) &^ 15
if !useLASX || n == 0 {
return max32Go(a)
}
var out [4]int32
if max32LSX(&a[0], n, &out) != 0 {
return float32(math.NaN())
}
m := max(max(fromOrdered32(out[0]), fromOrdered32(out[1])), max(fromOrdered32(out[2]), fromOrdered32(out[3])))
for _, v := range a[n:] {
m = max(m, v)
}
return m
}

// min32 is the dual of max32.
func min32(a []float32) float32 {
n := len(a) &^ 15
if !useLASX || n == 0 {
return min32Go(a)
}
var out [4]int32
if min32LSX(&a[0], n, &out) != 0 {
return float32(math.NaN())
}
m := min(min(fromOrdered32(out[0]), fromOrdered32(out[1])), min(fromOrdered32(out[2]), fromOrdered32(out[3])))
for _, v := range a[n:] {
m = min(m, v)
}
return m
}

// fromOrdered32 undoes the kernels' order map t = x XOR ((x >>a 31) >>l 1).
func fromOrdered32(t int32) float32 {
return math.Float32frombits(uint32(t ^ int32(uint32(t>>31)>>1)))
}

//go:noescape
func max32LSX(a *float32, n int, out *[4]int32) int64

//go:noescape
func min32LSX(a *float32, n int, out *[4]int32) int64
8 changes: 7 additions & 1 deletion internal/kernels/extrema32_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ func TestExtrema32MatchSequential(t *testing.T) {
nz := float32(math.Copysign(0, -1))
inf := float32(math.Inf(1))
nan := float32(math.NaN())
// A negative NaN too: x86's default NaN (0*Inf, Inf-Inf) has the sign bit
// set. A kernel that orders the bits as integers sorts a positive NaN above
// +Inf, so a max finds it without any NaN test; only a negative NaN shows
// whether the NaN test works. (A sabotaged loong64 NaN mask passed every
// test until this case was added.)
negNaN := math.Float32frombits(0xffc00000)
check := func(name string, a []float32) {
t.Helper()
for _, f := range []struct {
Expand All @@ -51,7 +57,7 @@ func TestExtrema32MatchSequential(t *testing.T) {
check("mixed", operands32(n, 5))
for pos := 0; pos < n; pos++ {
for _, fill := range []float32{nz, 0, -1, 1} {
for _, v := range []float32{nz, 0, inf, -inf, nan} {
for _, v := range []float32{nz, 0, inf, -inf, nan, negNaN} {
a := make([]float32, n)
for i := range a {
a[i] = fill
Expand Down
55 changes: 52 additions & 3 deletions internal/kernels/reduce_loong64.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
package kernels

import "os"
import (
"math"
"os"
)

// loong64 LASX kernels generated by go-asmgen (sum_loong64.s). LASX (256-bit,
// four float64 lanes) is not on every LoongArch CPU, so each wrapper uses it
Expand Down Expand Up @@ -42,8 +45,54 @@ func sqrtSIMD(dst, src []float64) {
}

// max/min stay on the four-accumulator scalar reducers.
func maxSIMD(a []float64) float64 { return maxUnrolled(a) }
func minSIMD(a []float64) float64 { return minUnrolled(a) }
// maxSIMD / minSIMD reduce whole blocks of 8 with the LSX kernels
// (sum_loong64.s), which work on the floats' bits mapped to an integer order
// (see fromOrdered64) because Go's loong64 assembler has no vector float max.
// The wrapper maps the two lanes back and folds them and the tail with Go's
// max/min; a NaN anywhere, reported by the kernel, is NaN.
func maxSIMD(a []float64) float64 {
n := len(a) &^ 7
if !useLASX || n == 0 {
return maxUnrolled(a)
}
var out [2]int64
if maxLSX(&a[0], n, &out) != 0 {
return math.NaN()
}
m := max(fromOrdered64(out[0]), fromOrdered64(out[1]))
for _, v := range a[n:] {
m = max(m, v)
}
return m
}

func minSIMD(a []float64) float64 {
n := len(a) &^ 7
if !useLASX || n == 0 {
return minUnrolled(a)
}
var out [2]int64
if minLSX(&a[0], n, &out) != 0 {
return math.NaN()
}
m := min(fromOrdered64(out[0]), fromOrdered64(out[1]))
for _, v := range a[n:] {
m = min(m, v)
}
return m
}

// fromOrdered64 undoes the kernels' order map t = x XOR ((x >>a 63) >>l 1),
// which is its own inverse.
func fromOrdered64(t int64) float64 {
return math.Float64frombits(uint64(t ^ int64(uint64(t>>63)>>1)))
}

//go:noescape
func maxLSX(a *float64, n int, out *[2]int64) int64

//go:noescape
func minLSX(a *float64, n int, out *[2]int64) int64

func addBin(dst, a, b []float64) {
if !useLASX || len(dst) == 0 {
Expand Down
31 changes: 18 additions & 13 deletions internal/kernels/reduce_simd_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -183,19 +183,24 @@ func TestMaxMinSIMD(t *testing.T) {
if g, w := minSIMD(a), Min(a); !bitEq(g, w) {
t.Fatalf("n=%d: minSIMD=%v, Min=%v", n, g, w)
}
// Inject a NaN at several positions; both must propagate it.
for _, pos := range []int{0, n / 2, n - 1} {
b := append([]float64(nil), a...)
b[pos] = math.NaN()
if g := maxSIMD(b); !math.IsNaN(g) {
t.Fatalf("n=%d NaN@%d: maxSIMD=%v, want NaN", n, pos, g)
}
if g := minSIMD(b); !math.IsNaN(g) {
t.Fatalf("n=%d NaN@%d: minSIMD=%v, want NaN", n, pos, g)
}
// And the oracle agrees it is NaN (semantics match).
if !math.IsNaN(Max(b)) || !math.IsNaN(Min(b)) {
t.Fatalf("n=%d NaN@%d: oracle did not propagate NaN", n, pos)
// Inject a NaN at several positions; both must propagate it. Both
// signs: x86's default NaN is negative, and a kernel that orders the
// bits as integers finds a positive NaN by its size alone, so only the
// negative one shows whether its NaN test works.
for _, nan := range []float64{math.NaN(), math.Float64frombits(0xfff8000000000000)} {
for _, pos := range []int{0, n / 2, n - 1} {
b := append([]float64(nil), a...)
b[pos] = nan
if g := maxSIMD(b); !math.IsNaN(g) {
t.Fatalf("n=%d NaN@%d: maxSIMD=%v, want NaN", n, pos, g)
}
if g := minSIMD(b); !math.IsNaN(g) {
t.Fatalf("n=%d NaN@%d: minSIMD=%v, want NaN", n, pos, g)
}
// And the oracle agrees it is NaN (semantics match).
if !math.IsNaN(Max(b)) || !math.IsNaN(Min(b)) {
t.Fatalf("n=%d NaN@%d: oracle did not propagate NaN", n, pos)
}
}
}
}
Expand Down
Loading
Loading