Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,10 @@ jobs:
fail-fast: false
matrix:
include:
- { goarch: riscv64, qemu: qemu-riscv64-static }
# V enabled: qemu's default riscv64 CPU has none, and the RVV kernels
# went untested here until 2026-10-26. NDARRAY_REQUIRE_RVV makes a
# CPU without V fail the lane (TestRVVWhenRequired).
- { goarch: riscv64, qemu: qemu-riscv64-static, cpu: "rv64,v=true,vlen=256", require: NDARRAY_REQUIRE_RVV }
- { goarch: loong64, qemu: qemu-loongarch64-static }
- { goarch: ppc64le, qemu: qemu-ppc64le-static }
- { goarch: s390x, qemu: qemu-s390x-static }
Expand Down Expand Up @@ -164,7 +167,11 @@ jobs:
fi
# Printed on purpose: without it, "ran under emulation" and "skipped" are
# indistinguishable from a green tick.
echo "running ${{ matrix.goarch }} under $q"
if [ -n "${{ matrix.cpu }}" ]; then
export QEMU_CPU="${{ matrix.cpu }}"
export ${{ matrix.require }}=1
fi
echo "running ${{ matrix.goarch }} under $q (QEMU_CPU=${QEMU_CPU:-default})"
GOARCH=${{ matrix.goarch }} go test -exec="$q" ./...

# 386 runs natively on the amd64 runner (the kernel executes 32-bit x86
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
`ZerosOf`, `FromSlice`, typed access with `Data[T]`/`Item[T]`, weak Go
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, and `Max`/`Min` on amd64 and arm64, bit-identical to its
64-bit target, and `Max`/`Min` on amd64, arm64 and riscv64, bit-identical to its
scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64,
rounded once)
([docs/design-dtypes.md](docs/design-dtypes.md)).
Expand Down
17 changes: 17 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,23 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-26 (v0.22.0).**

- **Correctness defect.** On riscv64 with V, `MeanAxis(1)` of the int64 row
`[MaxInt64, MinInt64, -3]` was 0, NumPy's -1. The strip-mined float64 sum
took even 3 elements as one vector, and its unordered reduction cancelled
`MaxInt64 + (-3)` against `MinInt64`. Below 8 elements the sum is now
sequential, as on the other targets. Found by running the root suite on a
real SpacemiT X60.
- **CI gap behind it.** The riscv64 qemu lane ran on qemu's default CPU,
which has no V, so the RVV kernels had never run in CI. It now enables V
and sets `NDARRAY_REQUIRE_RVV`, so a CPU without V fails the lane
(`TestRVVWhenRequired`).
- **New kernels.** The riscv64 extremum kernels (`maxRVV`, `minRVV`,
`max32RVV`, `min32RVV`) read `len(a)` elements of a non-empty slice, behind
the `useRVV` gate. They are fenced and tested against the sequential oracle
on the X60. A sabotaged one, which drops its NaN count, fails that test.

**2026-10-26 (v0.20.0).** The float32 extremum kernels (`max32SSE2`,
`min32SSE2`, `max32NEON`, `min32NEON`) read `len(a)` elements from a
non-empty slice. The Go wrappers pass `len(a)` and never an empty slice
Expand Down
36 changes: 35 additions & 1 deletion docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1048,7 +1048,41 @@ Still to do:
assemblers have no VSX float compare (for the NaN mask), and on loong64 no
vector float max, min or compare at all.
- s390x (`VFMAXSB` mode 1) waits for a z machine to test on.
- riscv64 (`vfmax.vv` with a `vmfne.vv` NaN mask) is next.
- riscv64: done in v0.22.0, below.

### riscv64 `Max`/`Min` and a sum defect (2026-10-26)

The RVV kernels for float64 and float32 work in three steps:

- **Accumulate:** each strip is folded into a register group with `vfmax.vv`
(or `vfmin.vv`). The tail-undisturbed policy keeps a short last strip from
touching the other lanes.
- **Count NaNs:** RISC-V's `fmax` returns the non-NaN operand, so each strip
also counts its NaN lanes (`vmfne.vv x, x`, then `vcpop.m`), and a non-zero
count gives NaN.
- **Fold:** `vfredmax.vs` reduces the group at the end. `fmax` already orders
−0 below +0.

On a SpacemiT X60 (`abbench`, 6 rounds, alternated):

| `Max`, 2¹⁴ / 2²⁰ | change |
|---|---:|
| float64 | **−89% / −83%** |
| float32 | **−93% / −90%** |
| `Sum` (controls) | ~ |

Running the root suite on that real X60 found a defect:
`MeanAxis(1)` of the int64 row `[MaxInt64, MinInt64, −3]` was 0, NumPy's −1.

- **Cause:** the strip-mined sum kernel took even 3 elements as one vector,
and its unordered reduction added `MaxInt64 + (−3)` first. In float64 that
rounds back to `MaxInt64` and cancels.
- **Fix:** below 8 elements the sum is now sequential, the order NumPy and the
other targets' kernels (whose scalar tails take n < 8 whole) use there.
- **Why CI never saw it:** its riscv64 lane runs under qemu, whose default CPU
has no V, so no RVV kernel had ever run in CI. The lane now enables V
(`QEMU_CPU=rv64,v=true,vlen=256`). It also sets `NDARRAY_REQUIRE_RVV`, so a
CPU without V fails the lane instead of passing on the scalar paths.

### float32 `MatMul` on the float64 GEMM (2026-10-26)

Expand Down
6 changes: 3 additions & 3 deletions docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,9 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
`fft` bridge shipped in v0.10.0 (package `fft`, over go-fft). float32 has
SIMD kernels on all six 64-bit targets for `Sum` (v0.14.0–v0.17.0) and for
`Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go
for float32: `Max`/`Min` on ppc64le, loong64, s390x and riscv64 (v0.20.0
vectorised them on amd64 and arm64 and stopped converting to float64
everywhere), and the transcendental ufuncs. `MatMul` runs on the float64
for float32: `Max`/`Min` on ppc64le, loong64 and s390x (v0.20.0 vectorised
them on amd64 and arm64 and stopped converting to float64 everywhere,
v0.22.0 on riscv64 for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32
GEMM is still to write.
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
Expand Down
61 changes: 61 additions & 0 deletions internal/kernels/asmgen/riscv64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,16 @@ func main() {
f.Add(gemmKernel())
f.Add(blocks8Kernel())
f.Add(sqrt32Kernel())
for _, k := range []struct {
name string
bits int
vop, rd string
}{
{"maxRVV", 64, "VFMAXVV", "VFREDMAXVS"}, {"minRVV", 64, "VFMINVV", "VFREDMINVS"},
{"max32RVV", 32, "VFMAXVV", "VFREDMAXVS"}, {"min32RVV", 32, "VFMINVV", "VFREDMINVS"},
} {
f.Add(extremumKernel(k.name, k.bits, k.vop, k.rd))
}
if err := os.WriteFile("sum_riscv64.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
Expand Down Expand Up @@ -130,6 +140,57 @@ func binKernel(name, op string) *emit.Function {
return b.Func()
}

// extremumKernel builds NAME(a *floatN, n int) floatN for n >= 1, the
// NaN-propagating maximum (or minimum), N = bits:
//
// - The accumulator group V16 (LMUL=4) is seeded with a[0] in every lane.
// - Each strip of vl elements is folded in with vfmax.vv/vfmin.vv under
// the tail-undisturbed policy, so a short last strip leaves the other
// lanes alone.
// - At the end the group is reduced with vfredmax.vs/vfredmin.vs.
//
// RISC-V's fmax/fmin order -0 below +0, as Go does, but return the non-NaN
// operand when one is NaN. So each strip also counts its NaN lanes (vmfne.vv
// x, x, then vcpop.m), and a non-zero count makes the result a quiet NaN.
func extremumKernel(name string, bits int, vop, red string) *emit.Function {
ret, mov, ew, shift, fmv := riscv64.Float64, "MOVD", "E64", 3, "FMVDX"
nan := "$0x7ff8000000000000"
if bits == 32 {
ret, mov, ew, shift, fmv = riscv64.Float32, "MOVF", "E32", 2, "FMVWX"
nan = "$0x7fc00000"
}
sig := riscv64.Layout([]string{"a", "n"}, []riscv64.Type{riscv64.Ptr, riscv64.Int64},
[]string{"ret"}, []riscv64.Type{ret})
b := riscv64.NewFunc(name, sig, 0)
b.LoadArg("a", "X10").LoadArg("n", "X11")
b.Raw("%s (X10), F0", mov).
Raw("VSETVLI X0, %s, M4, TA, MA, X16", ew). // vl = VLMAX
Raw("VFMVVF F0, V16").
Raw("MOV $0, X17"). // NaN lanes seen
Raw("loop:").
Raw("VSETVLI X11, %s, M4, TU, MA, X14", ew).
Raw("VLE%dV (X10), V8", bits).
Raw("VMFNEVV V8, V8, V4"). // V4 = the NaN lanes of the strip
Raw("VCPOPM V4, X15").
Raw("ADD X15, X17, X17").
Raw("%s V8, V16, V16", vop).
Raw("SLLI $%d, X14, X18", shift).
Raw("ADD X18, X10, X10").
Raw("SUB X14, X11, X11").
Raw("BNEZ X11, loop").
Raw("VSETVLI X0, %s, M4, TA, MA, X16", ew).
Raw("VFMVSF F0, V1").
Raw("%s V1, V16, V1", red). // V1[0] = extremum of V1[0] and every lane of V16
Raw("VFMVFS V1, F0").
Raw("BEQZ X17, done").
Raw("MOV %s, X5", nan).
Raw("%s X5, F0", fmv).
Raw("done:").
StoreRet("F0", "ret").
Ret()
return b.Func()
}

// sqrt32Kernel builds sqrt32RVV(dst, src *float32, n int) for n >= 1:
// vfsqrt.v on 32-bit elements, strip-mined; the correctly rounded float32
// square root, bit-identical to float32(math.Sqrt(float64(x))).
Expand Down
2 changes: 1 addition & 1 deletion internal/kernels/extrema32_generic.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build !amd64 && !arm64
//go:build !amd64 && !arm64 && !riscv64

package kernels

Expand Down
24 changes: 24 additions & 0 deletions internal/kernels/extrema32_riscv64.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
package kernels

// max32 runs the RVV kernel (sum_riscv64.s: vfmax.vv with a vmfne NaN count)
// when the CPU reports V, the four-chain Go loop otherwise.
func max32(a []float32) float32 {
if !useRVV {
return max32Go(a)
}
return max32RVV(&a[0], len(a))
}

// min32 is the dual of max32, with vfmin.vv.
func min32(a []float32) float32 {
if !useRVV {
return min32Go(a)
}
return min32RVV(&a[0], len(a))
}

//go:noescape
func max32RVV(a *float32, n int) float32

//go:noescape
func min32RVV(a *float32, n int) float32
33 changes: 30 additions & 3 deletions internal/kernels/reduce_riscv64.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,15 @@ func rvvFromAuxv() bool {
return err == nil && hwcapFrom(auxv)&hwcapV != 0
}

// sumSIMD is the RVV sum from 8 elements up. Below that it is the sequential
// sum, the order NumPy's pairwise sum and the other targets' kernels (whose
// scalar tails take n < 8 whole) use there. The strip-mined kernel would take
// even 3 elements as one vector, and its unordered reduction regrouped
// [MaxInt64, MinInt64, -3] as float64 so that max + (-3) rounded back to max
// and cancelled: the mean was 0, NumPy's -1. Seen on a real SpacemiT X60,
// never under CI's qemu, whose default CPU has no V.
func sumSIMD(a []float64) float64 {
if !useRVV || len(a) == 0 {
if !useRVV || len(a) < 8 {
return Sum(a)
}
return sumRVV(&a[0], len(a))
Expand All @@ -48,8 +55,28 @@ func sqrtSIMD(dst, src []float64) {
sqrtRVV(&dst[0], &src[0], len(dst))
}

func maxSIMD(a []float64) float64 { return maxUnrolled(a) }
func minSIMD(a []float64) float64 { return minUnrolled(a) }
// maxSIMD / minSIMD are the NaN-propagating extrema of a (non-empty): the RVV
// kernels (vfmax/vfmin with a vmfne NaN count) when the CPU reports V, the
// four-chain scalar reducers otherwise.
func maxSIMD(a []float64) float64 {
if !useRVV {
return maxUnrolled(a)
}
return maxRVV(&a[0], len(a))
}

func minSIMD(a []float64) float64 {
if !useRVV {
return minUnrolled(a)
}
return minRVV(&a[0], len(a))
}

//go:noescape
func maxRVV(a *float64, n int) float64

//go:noescape
func minRVV(a *float64, n int) float64

func addBin(dst, a, b []float64) {
if !useRVV || len(dst) == 0 {
Expand Down
18 changes: 18 additions & 0 deletions internal/kernels/rvv_required_riscv64_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package kernels

import (
"os"
"testing"
)

// TestRVVWhenRequired fails when NDARRAY_REQUIRE_RVV is set and the CPU does
// not report V. CI sets it on the riscv64 qemu lane, which runs with V
// enabled: without this check, a qemu whose CPU lost V would take the scalar
// paths and the RVV kernels would go untested behind a green tick, as they
// did until 2026-10-26.
func TestRVVWhenRequired(t *testing.T) {
t.Logf("useRVV = %v", useRVV)
if os.Getenv("NDARRAY_REQUIRE_RVV") != "" && !useRVV {
t.Fatal("NDARRAY_REQUIRE_RVV is set but the CPU does not report V: the RVV kernels did not run")
}
}
Loading
Loading