Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, and `Max`/`Min` on every 64-bit target, bit-identical to its
scalar loops. Its `MatMul` runs on a native float32 GEMM on arm64 (NEON),
scalar loops. Its `MatMul` runs on go-blas/blas's native float32 GEMM on arm64 (NEON),
amd64 with FMA (AVX2), ppc64le (VSX), s390x (z14 vector) and riscv64 (RVV 1.0), accumulating in float32 like NumPy's sgemm, and
elsewhere on the float64 GEMM (accumulated in float64, rounded once).
Its `Exp` and `Log` are **correctly rounded**, proven by an exhaustive test
Expand Down Expand Up @@ -115,9 +115,13 @@ loops are [go-asmgen](https://github.com/go-asmgen)-generated SIMD kernels: sum,
add/sub/mul/div, sqrt and the dot product (SSE2 or AVX2/FMA on amd64, NEON on
arm64, VSX on ppc64le, LASX on loong64, RVV on riscv64 and the vector facility
on s390x, the last three when the CPU has them), max/min too on
amd64, arm64 and s390x (z14 and later), and a **panel-packed, cache-blocked GEMM** with an SIMD-FMA
micro-kernel (NEON 4×8; AVX2/FMA 6×8 chosen at run time, SSE2 fallback; VSX
8×8; LASX 8×8; RVV 6×8; s390x 4×8), the OpenBLAS/BLIS structure. The 32-bit
amd64, arm64 and s390x (z14 and later). The **panel-packed, cache-blocked GEMM**
with its SIMD-FMA micro-kernels (NEON 4×8; AVX2/FMA 6×8 chosen at run time, SSE2
fallback; VSX 8×8; LASX 8×8; RVV 6×8; s390x 4×8; the OpenBLAS/BLIS structure)
no longer lives here: `MatMul`, `Dot` and the linalg products call
[go-blas/blas](https://github.com/go-blas/blas) (`DgemmStrided`/`SgemmStrided`,
which read a transposed or sliced view through its strides without copying), the
one implementation of that kernel. The 32-bit
targets run the same pure-Go code those kernels are tested against. `Exp` and `Log` are ports of Arm's optimized-routines
(≤ 0.51 ULP measured, against up to 0.88 and 0.72 for Go's
`math.Exp`/`math.Log` on arm64),
Expand Down
28 changes: 22 additions & 6 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,19 @@ bounds checks do not cover, so they are held to the following:

- **Every wrapper checks the reach of its kernel.** Before an assembly call, the
Go wrapper indexes the last element of every operand the kernel reads or
writes (`internal/kernels/{reduce,gemm}_<arch>.go`). A caller's mistake is an
index-out-of-range panic in Go, never an access outside a slice.
writes (`internal/kernels/{reduce,dot}_<arch>.go`). A caller's mistake is an
index-out-of-range panic in Go, never an access outside a slice. The GEMM
micro-kernels, their wrappers and their fence tests live in
[go-blas/blas](https://github.com/go-blas/blas), which owns that audit; the
entries below about GEMM kernels (v0.26.0 to v0.38.0) are history of code now
there.
`TestWrappersRefuseShortOperands` passes each wrapper an operand one element
short and requires the panic.
- **The kernels stay inside their slices.** `TestKernelsStayInsideTheirSlices`
and `TestGemmMicroStaysInsideItsOperands` (`internal/kernels/fence_test.go`,
Linux and macOS) map each operand flush against a `PROT_NONE` page, once
ending at it and once starting after one, for every length from 1 to 70 and
every GEMM depth from 1 to 9. A read or write one element past either end is a
(`internal/kernels/fence_test.go`, Linux and macOS) maps each operand flush
against a `PROT_NONE` page, once ending at it and once starting after one, for
every length from 1 to 70 (go-blas/blas fences its GEMM micro-kernels at every
depth from 1 to 9 the same way). A read or write one element past either end is a
fault. The fence itself was checked by a kernel deliberately reading one
element too far, which faults on amd64 (4 KiB pages), arm64 (16 KiB), loong64
(16 KiB) and ppc64le (64 KiB). Prefetch instructions may point past the end:
Expand Down Expand Up @@ -66,6 +70,18 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-11-15 (unreleased).** The GEMM moved to
[go-blas/blas](https://github.com/go-blas/blas) v0.2.1 (the micro-kernels, their
wrappers, the packing and their fence tests, byte for byte; the audit of that
code is that repository's). What ndarray adds is `internal/blasmm`, which hands
an operand's row and column strides to `DgemmStrided`/`SgemmStrided`. go-blas
checks every stride and slice length (overflow-safe, `*ArgError` on refusal), so
a view ndarray builds wrongly is an error there, which `blasmm` turns into a
panic naming the package, not a read outside a slice. A zero or negative stride
on an axis longer than one (a broadcast or reversed view) is copied to a
contiguous matrix first. ndarray keeps the dot kernels (`dot_<arch>.go`), the
mat-vec and the thread pool, with their wrappers and fences.

**2026-11-12 (unreleased, #103).** The gather of `random`'s `Choice`
(`takeAxis`, random/arrays.go) now copies bytes through `bytesOf`, an
`unsafe.Slice` view of a slice's backing array of exactly `Len()·size` bytes.
Expand Down
1 change: 1 addition & 0 deletions bench/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ require (
)

require (
github.com/go-blas/blas v0.2.1 // indirect
github.com/go-fft/fft v0.29.0 // indirect
github.com/go-fleettools/fleettools v0.81.0 // indirect
github.com/go-simd/floats v0.17.0 // indirect
Expand Down
2 changes: 2 additions & 0 deletions bench/go.sum
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
github.com/go-blas/blas v0.2.1 h1:5uZfmvXVZn9CjE+I3DalALqWHdKz+rbBs8VffH3dAZo=
github.com/go-blas/blas v0.2.1/go.mod h1:WjhlqzQ+RA8tS473f18QtcJl5Q0GonPs3RJIxT5G7Bk=
github.com/go-fft/fft v0.29.0 h1:WtJb5K3dmbLWREvvZ50od0fxtjIBiXrg7mVybz/nPoA=
github.com/go-fft/fft v0.29.0/go.mod h1:GN5V4TWCjV05ECqH1ONfjFM6Z1m2shqCdssxKGqQ41s=
github.com/go-fleettools/fleettools v0.81.0 h1:UnMflmNw5b82Z+7Ums7UVaW5RrGuACCVmSwytwT1hxw=
Expand Down
28 changes: 28 additions & 0 deletions bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,34 @@ func benchMat(n int) *Array {
return a
}

// BenchmarkMatMulVariants covers what BenchmarkMatMul does not: float32
// operands, a transposed operand (read in place by the GEMM) and a strided one
// (a[:, ::2], which the GEMM entry copies first).
func BenchmarkMatMulVariants(b *testing.B) {
for _, n := range []int{64, 256, 1024} {
x, y := benchMat(n), benchMat(n)
x32, y32 := x.AsType(Float32), y.AsType(Float32)
wide, _ := FromData(make([]float64, 2*n*n), n, 2*n)
half, _ := wide.Slice(All(), Step(2))
xt := x.Transpose()
b.Run(fmt.Sprintf("f32/n=%d", n), func(b *testing.B) {
for i := 0; i < b.N; i++ {
_, _ = x32.MatMul(y32)
}
})
b.Run(fmt.Sprintf("transA/n=%d", n), func(b *testing.B) {
for i := 0; i < b.N; i++ {
_, _ = xt.MatMul(y)
}
})
b.Run(fmt.Sprintf("strided/n=%d", n), func(b *testing.B) {
for i := 0; i < b.N; i++ {
_, _ = half.MatMul(y)
}
})
}
}

func BenchmarkMatMul(b *testing.B) {
for _, n := range []int{64, 128, 256, 512, 1024} {
x, y := benchMat(n), benchMat(n)
Expand Down
3 changes: 2 additions & 1 deletion docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ ndarray/
internal/kernels/ portable scalar inner loops over flat []float64
(Add/Sub/Mul/Div, Map, comparisons, Maximum/Minimum,
Sum/Prod/Max/Min, Abs, {Sum,Prod,Max,Min}Axis over
[outer][axisLen][inner], MatMul GEMM, Dot1D)
[outer][axisLen][inner], mat-vec, Dot1D; the GEMM is
github.com/go-blas/blas, through internal/blasmm)
docs/plan-ndarray.md this roadmap
```

Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ module github.com/go-ndarray/ndarray
go 1.27.2

require (
github.com/go-blas/blas v0.2.1
github.com/go-fft/fft v0.29.0
github.com/go-simd/floats v0.17.0
)
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
github.com/go-blas/blas v0.2.1 h1:5uZfmvXVZn9CjE+I3DalALqWHdKz+rbBs8VffH3dAZo=
github.com/go-blas/blas v0.2.1/go.mod h1:WjhlqzQ+RA8tS473f18QtcJl5Q0GonPs3RJIxT5G7Bk=
github.com/go-fft/fft v0.29.0 h1:WtJb5K3dmbLWREvvZ50od0fxtjIBiXrg7mVybz/nPoA=
github.com/go-fft/fft v0.29.0/go.mod h1:GN5V4TWCjV05ECqH1ONfjFM6Z1m2shqCdssxKGqQ41s=
github.com/go-simd/floats v0.17.0 h1:Dv0ZmZ3k0IavvHwQgstpUz+TeIf/gGJ9HEGQ/UART8Q=
Expand Down
76 changes: 76 additions & 0 deletions internal/blasmm/blasmm.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
// Package blasmm is the row-major strided matrix product of ndarray on top of
// the general-stride GEMM of github.com/go-blas/blas (DgemmStrided and
// SgemmStrided), the one implementation of the packed, cache-blocked, multicore
// kernel.
//
// An operand is read through its row and column strides, as go-blas packs it,
// so a transposed or sliced view (a.T, a[:, ::2]) is never copied. go-blas
// takes strides of at least 1, and ndarray's views can hold others, which are
// normalised here: a stride of an axis of extent 1 is never used, so it is set
// to 1; an operand with a zero (broadcast) or negative (reversed) stride on an
// axis longer than 1 is copied to a contiguous matrix first, an O(rows*cols)
// pass beside the O(m*n*k) product. The result is written row-major, contiguous.
package blasmm

import "github.com/go-blas/blas"

// Mat is a strided matrix operand: element (i, j) is Data[Off+i*RS+j*CS].
type Mat[T float32 | float64] struct {
Data []T
Off int
RS, CS int
}

// HasGemm32 reports whether SgemmStrided runs on a float32 SIMD kernel here;
// without one the caller multiplies in float64 instead.
func HasGemm32() bool { return blas.HasSgemmKernel() }

// MatMul writes the m x k by k x n product a*b into dst (m*n elements, row
// major, overwritten).
func MatMul(dst []float64, a, b Mat[float64], m, k, n int) {
matMul(blas.DgemmStrided, dst, a, b, m, k, n)
}

// MatMul32 is MatMul in float32. It requires HasGemm32.
func MatMul32(dst []float32, a, b Mat[float32], m, k, n int) {
matMul(blas.SgemmStrided, dst, a, b, m, k, n)
}

type stridedGemm[T float32 | float64] func(m, n, k int, alpha T, a []T, rsa, csa int, b []T, rsb, csb int, beta T, c []T, rsc, csc int) error

func matMul[T float32 | float64](gemm stridedGemm[T], dst []T, a, b Mat[T], m, k, n int) {
if m == 0 || n == 0 {
return
}
ad, rsa, csa := operand(a, m, k)
bd, rsb, csb := operand(b, k, n)
// The operands are valid views (the callers checked their shapes), so a
// refusal here is a defect in this package, not an input error.
if err := gemm(m, n, k, 1, ad, rsa, csa, bd, rsb, csb, 0, dst[:m*n], n, 1); err != nil {
panic("ndarray: blasmm: " + err.Error())
}
}

// operand returns x (rows x cols) as a slice and strides go-blas accepts.
func operand[T float32 | float64](x Mat[T], rows, cols int) (d []T, rs, cs int) {
if rows == 0 || cols == 0 {
return nil, 1, 1
}
rs, cs = x.RS, x.CS
if rows == 1 {
rs = 1 // never used
}
if cols == 1 {
cs = 1
}
if rs >= 1 && cs >= 1 {
return x.Data[x.Off:], rs, cs
}
d = make([]T, rows*cols)
for i := 0; i < rows; i++ {
for j := 0; j < cols; j++ {
d[i*cols+j] = x.Data[x.Off+i*x.RS+j*x.CS]
}
}
return d, cols, 1
}
122 changes: 122 additions & 0 deletions internal/blasmm/blasmm_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
package blasmm

import "testing"

// view lays a rows x cols matrix of small integers out with the given strides
// in a buffer padded with a sentinel, from offset off; the products below are
// exact, so every comparison is equality.
func view[T float32 | float64](rows, cols, off, rs, cs int, seed int) (Mat[T], [][]T) {
hi := off // highest element index
for _, ext := range [][2]int{{rows - 1, rs}, {cols - 1, cs}} {
if ext[1] > 0 {
hi += ext[0] * ext[1]
}
}
buf := make([]T, hi+1)
for i := range buf {
buf[i] = 9999 // never an element, or a product would be off
}
want := make([][]T, rows)
for i := range want {
want[i] = make([]T, cols)
for j := range want[i] {
v := T((i*7+j*3+seed)%5 - 2)
want[i][j] = v
buf[off+i*rs+j*cs] = v // an aliased view (rs = 0) keeps the last write
}
}
for i := range want { // re-read: aliasing makes the matrix what the buffer says
for j := range want[i] {
want[i][j] = buf[off+i*rs+j*cs]
}
}
return Mat[T]{Data: buf, Off: off, RS: rs, CS: cs}, want
}

func product[T float32 | float64](a, b [][]T, m, k, n int) []T {
c := make([]T, m*n)
for i := 0; i < m; i++ {
for j := 0; j < n; j++ {
for p := 0; p < k; p++ {
c[i*n+j] += a[i][p] * b[p][j]
}
}
}
return c
}

func testMatMul[T float32 | float64](t *testing.T, mul func(dst []T, a, b Mat[T], m, k, n int)) {
type lay struct {
name string
off, rs, cs func(rows, cols int) int
}
layouts := []lay{
{"row-major", func(r, c int) int { return 0 }, func(r, c int) int { return c }, func(r, c int) int { return 1 }},
{"padded", func(r, c int) int { return 5 }, func(r, c int) int { return c + 3 }, func(r, c int) int { return 1 }},
{"transposed", func(r, c int) int { return 0 }, func(r, c int) int { return 1 }, func(r, c int) int { return r }},
{"every other column", func(r, c int) int { return 0 }, func(r, c int) int { return 2 * c }, func(r, c int) int { return 2 }},
{"every other row", func(r, c int) int { return 0 }, func(r, c int) int { return 2 * c }, func(r, c int) int { return 1 }},
{"reversed rows", func(r, c int) int { return (r - 1) * c }, func(r, c int) int { return -c }, func(r, c int) int { return 1 }},
{"reversed columns", func(r, c int) int { return c - 1 }, func(r, c int) int { return c }, func(r, c int) int { return -1 }},
{"broadcast rows", func(r, c int) int { return 0 }, func(r, c int) int { return 0 }, func(r, c int) int { return 1 }},
}
shapes := [][3]int{{1, 1, 1}, {1, 5, 4}, {4, 5, 1}, {3, 1, 6}, {7, 5, 3}, {13, 17, 19}, {40, 33, 50}}
for _, sh := range shapes {
m, k, n := sh[0], sh[1], sh[2]
for _, la := range layouts {
for _, lb := range layouts {
a, aw := view[T](m, k, la.off(m, k), la.rs(m, k), la.cs(m, k), 1)
b, bw := view[T](k, n, lb.off(k, n), lb.rs(k, n), lb.cs(k, n), 2)
got := make([]T, m*n)
for i := range got {
got[i] = 7777 // overwritten, not accumulated
}
mul(got, a, b, m, k, n)
want := product(aw, bw, m, k, n)
for i := range want {
if got[i] != want[i] {
t.Fatalf("%v a=%s b=%s: [%d] = %v, want %v", sh, la.name, lb.name, i, got[i], want[i])
}
}
}
}
}
// A stride of an axis of extent 1 is never used, whatever it is.
a := Mat[T]{Data: []T{2, 3}, RS: 0, CS: 1} // 1 x 2, rs unused
b := Mat[T]{Data: []T{4, 5}, RS: 1, CS: -3} // 2 x 1, cs unused
got := make([]T, 1)
mul(got, a, b, 1, 2, 1)
if got[0] != 23 {
t.Fatalf("extent-1 strides: %v, want 23", got[0])
}
// Empty products write nothing; k = 0 is the zero matrix.
mul(nil, Mat[T]{}, Mat[T]{}, 0, 3, 4)
mul(nil, Mat[T]{}, Mat[T]{}, 3, 3, 0)
z := []T{5, 5, 5, 5}
mul(z, Mat[T]{}, Mat[T]{}, 2, 0, 2)
for _, v := range z {
if v != 0 {
t.Fatalf("k = 0: %v, want zeros", z)
}
}
}

func TestMatMul(t *testing.T) { testMatMul(t, MatMul) }

func TestMatMul32(t *testing.T) {
t.Logf("HasGemm32() = %v; the float32 product is tested where there is a kernel", HasGemm32())
if HasGemm32() {
testMatMul(t, MatMul32)
}
}

// A refusal by go-blas is a defect of this package and panics with its name.
func TestMatMulPanicsOnADefect(t *testing.T) {
defer func() {
if recover() == nil {
t.Fatal("a destination too short was accepted")
}
}()
one := Mat[float64]{Data: []float64{1}, RS: 1, CS: 1}
MatMul(make([]float64, 2), one, one, 1, 1, 2) // B is 1 x 2 but holds one element
}
Loading
Loading