Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, and `Max`/`Min` on every 64-bit target, bit-identical to its
scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64,
rounded once)
scalar loops. Its `MatMul` runs on a native float32 GEMM on arm64 (NEON)
and amd64 with FMA (AVX2), accumulating in float32 like NumPy's sgemm, and
elsewhere on the float64 GEMM (accumulated in float64, rounded once)
([docs/design-dtypes.md](docs/design-dtypes.md)).
- **Creation** — `New`/`Zeros`/`Ones`/`Full`/`FromData`/`Arange`/`Linspace`/
`Eye`/`Identity`.
Expand Down Expand Up @@ -154,8 +155,8 @@ and `Max` by 92–96% ([docs/perf.md](docs/perf.md), *SIMD coverage*). It is a
> pair Go supports; v0.1.0 and v0.2.0 were also run on real amd64, arm64,
> ppc64le, riscv64 and loong64 hardware, v0.3.0 on real amd64, arm64, ppc64le
> and loong64. See **[docs/plan-ndarray.md](docs/plan-ndarray.md)**
> for the roadmap (float32 SIMD for max/min and the products; linear algebra
> decompositions).
> for the roadmap (a native float32 GEMM on the other four 64-bit targets;
> linear algebra decompositions).

## Why this module?

Expand Down
12 changes: 12 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,18 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-26 (v0.26.0).** The float32 GEMM kernels, `gemmMicro8x8S` (arm64)
and `gemmMicro6x16FMA` (amd64, behind the FMA probe), read `kc·MR` and `kc·NR`
packed floats and read and write one MR×NR tile of C. Their Go wrapper
bounds-checks all three reaches before the call. They are fenced (PROT_NONE
on both sides of each operand) and tested against a float32 oracle. A
sabotaged kernel failed every CI job that runs it. The edge tile now lives
in the worker's pooled A buffer: its capacity adds MR·NR elements, and
`gemmBand` slices the tile from the end. The operand views are the same
`Off/RS/CS` triples the float64 GEMM reads, built from the array's own
strides. A non-Float32 operand is converted to a fresh contiguous copy
first. Shapes are validated (overflow included) before any allocation.

**2026-10-26 (v0.25.0).** The loong64 extremum kernels (`maxLSX`, `minLSX`,
`max32LSX`, `min32LSX`) read the whole blocks of `a` behind the `useLASX`
gate. They write one 16-byte vector to an `out` array on the wrapper's stack,
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/dtype_bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ func BenchmarkDTypeMax(b *testing.B) {
// BenchmarkDTypeMatMul is a square product, 2·n³ flops per op.
func BenchmarkDTypeMatMul(b *testing.B) {
for _, dt := range []nd.DType{nd.Float64, nd.Float32} {
for _, n := range []int{64, 256} {
x, err := vec(n * n).AsType(dt).Reshape(n, n)
for _, n := range []int{64, 256, 1024} {
x, err := vec(n*n).AsType(dt).Reshape(n, n)
if err != nil {
b.Fatal(err)
}
Expand Down
55 changes: 53 additions & 2 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1189,6 +1189,55 @@ float32 is now within the widening of float64: 86.7 against 54.0 µs at 64²,
lanes per register and half the bytes, could go further; it is the next step
for products.

### A native float32 GEMM on amd64 and arm64 (2026-10-26)

The packed GEMM driver (packing, cache blocking, the parallel bands) is now
generic over float32 and float64. Each element type brings its micro-kernel,
its tile and its pack-buffer pools. float32 gets two kernels, generated by
go-asmgen:

- **arm64:** `gemmMicro8x8S`, a NEON 8×8 tile. It has 16 accumulators and
uses the by-element `FMLA Vd.4S, Vn.4S, Vm.S[i]`, encoded as a `WORD` and
checked against `go tool objdump`.
- **amd64:** `gemmMicro6x16FMA`, the BLIS Haswell sgemm shape: a 6×16 tile,
`VBROADCASTSS` and `VFMADD231PS` into 12 YMM accumulators. It runs only
where the CPU has FMA.

float32 `MatMul` (and `Dot`, which goes through it) now accumulates in
float32, as NumPy's sgemm does. Without a kernel (no FMA, or the other four
targets) it still widens to the float64 GEMM. The tests check it against a
float32 ikj oracle, bit for bit on integer data, in every block
configuration, on a transposed and offset B view. On random data they bound
it by k·eps32·Σ|ab|. A sabotaged kernel (a wrong A lane on arm64, a wrong A
row on amd64) failed every job that runs one.

Profiling the first version at 64² found `packA` at 33% of the time, ahead
of the micro-kernel. It gathered A column by column and paid its loop and
bounds checks every MR elements. A row-major A is now read one contiguous
row at a time, which also speeds up float64.

| `MatMul` (`abbench`, 10 rounds, 1 core) | 64² | 256² | 1024² |
|---|---:|---:|---:|
| Zen 3, float32 | **−61%** | **−56%** | **−55%** |
| Zen 3, float64 | −5.8% | −4.2% | ~ |
| Neoverse N1, float32 | **−64%** | **−60%** | **−57%** |
| Neoverse N1, float64 | ~ | ~ | ~ |

Against NumPy 2.5.4 (OpenBLAS, `OPENBLAS_NUM_THREADS=1`, the same core):

| | 64² | 256² | 1024² |
|---|---:|---:|---:|
| Zen 3 float32, ours / NumPy | 15.0 / 8.1 µs | 485 / 366 µs | 26.0 / 23.1 ms |
| Zen 3 float64 | 23.2 / 15.8 µs | 866 / 795 µs | 51.0 / 50.0 ms |
| N1 float32 | 30.4 / 16.2 µs | 1.19 / 0.94 ms | 68.2 / 60.6 ms |
| N1 float64 | 48.2 / 31.3 µs | 2.46 / 1.95 ms | 154 / 123 ms |

**Open defect: products are still slower than OpenBLAS on one core**, by
1.1–1.3× at 1024² and up to 1.9× at 64². The gap is widest where the work
around the kernel weighs most: packing, edge tiles, and the result's
allocation and zeroing. The kernels for ppc64le, s390x, riscv64 and loong64
are still to write.

**Open: the allocating operations.** `Add` is still 1.5–3.2× slower than
NumPy's `a+b`, for float64 as well. The kernel is no longer the difference,
as `AddInto` shows; the allocation is. It splits like this, measured
Expand Down Expand Up @@ -1224,12 +1273,14 @@ What does remove it:
`VFMADD231PD`, C and A prefetched) when go-asmgen's CPUID probe reports FMA
with the OS saving YMM state, else a pair of SSE2 6×4 `MULPD`+`ADDPD` tiles
over the same packing. Generated by go-asmgen and validated per-arch in CI;
the FMA kernel on real Cascade Lake and Zen 3 hosts.
the FMA kernel on real Cascade Lake and Zen 3 hosts. float32 products have
their own FMA kernel, `gemmMicro6x16FMA` (6×16, `VFMADD231PS`).
- **arm64 (NEON)** ships hand-vectorized `sum`, `dot`, **packed `sqrt`**
(`VFSQRT`), the **elementwise `add`/`sub`/`mul`/`div`** (`VFADD`/`VFSUB`/
`VFMUL`/`VFDIV`), **`max`/`min`** (`VFMAX`/`VFMIN`, four D2 accumulators from
8 elements up) and the **GEMM micro-kernel** (`gemmMicro4x8`: a 4×8 tile, 16 D2
accumulators, using the **by-lane FMLA** `FMLA Vd.2D, Vn.2D, Vm.D[i]`). Only
accumulators, using the **by-lane FMLA** `FMLA Vd.2D, Vn.2D, Vm.D[i]`), with
a float32 one, `gemmMicro8x8S` (8×8, `FMLA Vd.4S, Vn.4S, Vm.S[i]`). Only
the indexed-element FMLA is still a raw `WORD` (16 of them): Go's assembler
does not name it yet (pending as CL 764224).

Expand Down
6 changes: 4 additions & 2 deletions docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,8 +159,10 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
`Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go
for float32: nothing in `Max`/`Min` any more (v0.20.0 to v0.25.0
vectorised them on all six 64-bit targets, for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32
GEMM is still to write.
GEMM since v0.21.0, 77–90% faster than the triple loop, and since v0.26.0
on a native float32 GEMM on arm64 (NEON 8×8) and amd64 with FMA (6×16),
a further 53–61% faster. The float32 kernels for ppc64le, s390x, riscv64
and loong64 are still to write.
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
comparisons returning Bool masks, `Maximum`/`Minimum`, `Logical*`, `Power`,
`Where` and the per-axis reductions, in every dtype. Not done: a public
Expand Down
74 changes: 74 additions & 0 deletions internal/kernels/asmgen/amd64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ func main() {
f.Add(binKernel("mulSSE2", "MULPD", "MULSD"))
f.Add(binKernel("divSSE2", "DIVPD", "DIVSD"))
f.Add(gemmFMAKernel())
f.Add(gemm32FMAKernel())
f.Add(dotFMAKernel())
f.Add(gemmSSE2HalfKernel())
f.Add(amd64.FeatureProbe("hasFMA", amd64.FMA))
Expand Down Expand Up @@ -487,6 +488,79 @@ func gemmFMAKernel() *emit.Function {
return b.Func()
}

// gemm32NR is the float32 tile width on amd64: two YMM of eight floats.
const gemm32NR = 16

// gemm32FMAKernel builds gemmMicro6x16FMA(kc int, pa, pb, c *float32, ldc int),
// the float32 micro-kernel of the packed GEMM:
//
// for p in [0,kc): C[r][col] += pa[p*6+r] * pb[p*16+col] (r<6, col<16)
//
// gemmFMAKernel at twice the lanes, the BLIS Haswell sgemm shape: the 16-float
// B row in Y12,Y13, each of the six A values broadcast with VBROADCASTSS, and
// twelve VFMADD231PS into Y0..Y11 (row r in Y(2r), Y(2r+1)). Gated on hasFMA.
func gemm32FMAKernel() *emit.Function {
b := amd64.NewFunc("gemmMicro6x16FMA", gemmSig(), 0)
b.LoadArg("kc", "CX")
b.LoadArg("pa", "SI")
b.LoadArg("pb", "DI")
b.LoadArg("c", "DX")
b.LoadArg("ldc", "R8")
b.Raw("SHLQ $2, R8") // ldc -> bytes
// Prefetch the C tile: six rows of 64 bytes, each on one or two lines.
b.Raw("MOVQ DX, R9")
for r := 0; r < gemmMR; r++ {
b.Raw("PREFETCHT0 (R9)")
b.Raw("PREFETCHT0 63(R9)")
if r < gemmMR-1 {
b.Raw("ADDQ R8, R9")
}
}
for r := 0; r < 12; r++ {
b.Raw(fmt.Sprintf("VXORPS Y%d, Y%d, Y%d", r, r, r))
}
b.Raw("MOVQ CX, R10")
b.Raw("SHRQ $1, R10")
b.Raw("JZ stail")
b.Raw("skloop:")
b.Raw(fmt.Sprintf("PREFETCHT0 %d(SI)", 16*gemmMR*4))
fma32Step(b, 0)
fma32Step(b, 1)
b.Raw(fmt.Sprintf("ADDQ $%d, SI", 2*4*gemmMR))
b.Raw(fmt.Sprintf("ADDQ $%d, DI", 2*4*gemm32NR))
b.Raw("DECQ R10")
b.Raw("JNZ skloop")
b.Raw("stail:")
b.Raw("TESTQ $1, CX")
b.Raw("JZ sstore")
fma32Step(b, 0)
b.Raw("sstore:")
for r := 0; r < gemmMR; r++ {
b.Raw(fmt.Sprintf("VADDPS (DX), Y%d, Y15", 2*r))
b.Raw("VMOVUPS Y15, (DX)")
b.Raw(fmt.Sprintf("VADDPS 32(DX), Y%d, Y15", 2*r+1))
b.Raw("VMOVUPS Y15, 32(DX)")
if r < gemmMR-1 {
b.Raw("ADDQ R8, DX")
}
}
b.Raw("VZEROUPPER")
b.Ret()
return b.Func()
}

// fma32Step emits one k step of the 6x16 float32 FMA tile.
func fma32Step(b *amd64.Builder, step int) {
ao, bo := 4*gemmMR*step, 4*gemm32NR*step
b.Raw(fmt.Sprintf("VMOVUPS %d(DI), Y12", bo)) // B cols 0..7
b.Raw(fmt.Sprintf("VMOVUPS %d(DI), Y13", bo+32)) // B cols 8..15
for r := 0; r < gemmMR; r++ {
b.Raw(fmt.Sprintf("VBROADCASTSS %d(SI), Y14", ao+4*r))
b.Raw(fmt.Sprintf("VFMADD231PS Y12, Y14, Y%d", 2*r))
b.Raw(fmt.Sprintf("VFMADD231PS Y13, Y14, Y%d", 2*r+1))
}
}

// dotFMAKernel builds dotFMA(a, b *float64, n int) float64, the inner product
// with four YMM accumulators (16 doubles per iteration, one VFMADD231PD per 4
// doubles reading b straight from memory): two loads per FMA make the loop
Expand Down
74 changes: 71 additions & 3 deletions internal/kernels/asmgen/arm64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ func main() {
f.Add(extremum32Kernel("max32NEON", "VFMAX", "FMAXS"))
f.Add(extremum32Kernel("min32NEON", "VFMIN", "FMINS"))
f.Add(gemmKernel())
f.Add(gemm32Kernel())
f.Add(blocks8Kernel())
if err := os.WriteFile("sum_arm64.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
Expand Down Expand Up @@ -290,9 +291,9 @@ func extremum32Kernel(name, vop, sop string) *emit.Function {
}
b.Raw("SUB $16, R1").Raw("B block").
Raw("fold:").
Raw(vop+" V1.S4, V0.S4, V0.S4").
Raw(vop+" V3.S4, V2.S4, V2.S4").
Raw(vop+" V2.S4, V0.S4, V0.S4")
Raw(vop + " V1.S4, V0.S4, V0.S4").
Raw(vop + " V3.S4, V2.S4, V2.S4").
Raw(vop + " V2.S4, V0.S4, V0.S4")
for l := 0; l < 4; l++ {
b.Raw(fmt.Sprintf("VMOV V0.S[%d], R%d", l, 3+l))
}
Expand Down Expand Up @@ -470,6 +471,73 @@ func gemmKernel() *emit.Function {
return b.Func()
}

// fmlaElemS is fmlaElem for single precision:
//
// FMLA V{acc}.4S, V{breg}.4S, V{areg}.S[idx] => V{acc} += V{breg} * V{areg}[idx]
//
// The same Advanced SIMD "FMLA (by element)" encoding with sz=0 (ARM C7-2):
//
// 0|1|0|01111|1|0|L|M|Rm[3:0]|0001|H|0|Rn|Rd
//
// where the element index is H:L and, for sz=0, the register is M:Rm, so
// areg may be any of V0..V31. idx in {0,1,2,3}.
func fmlaElemS(acc, breg, areg, idx int) string {
const base = 0x4F801000 // Q=1, sz=0, opcode 0001: FMLA by element .4S
M := (areg >> 4) & 1
Rm := areg & 0xF
H, L := (idx>>1)&1, idx&1
word := base | (L << 21) | (M << 20) | (Rm << 16) | (H << 11) | (breg << 5) | acc
return fmt.Sprintf("WORD $0x%08X // FMLA V%d.4S, V%d.4S, V%d.S[%d]",
word, acc, breg, areg, idx)
}

// gemm32Kernel builds gemmMicro8x8S(kc int, pa, pb, c *float32, ldc int), the
// float32 micro-kernel of the packed GEMM: an 8-row x 8-col tile,
//
// for p in [0,kc): C[r][col] += pa[p*8+r] * pb[p*8+col] (r<8, col<8)
//
// the float64 kernel's structure at twice the lanes. The tile lives in V0..V15
// (row r in V(2r), V(2r+1), four floats each), the B row in V16,V17 and the
// eight A values in V18=[a0..a3], V19=[a4..a7], which the by-element FMLA
// (fmlaElemS) reads lane by lane. Sixteen independent accumulator chains cover
// the FMA latency of two 4-cycle pipes (Neoverse N1) with room to spare; each
// k step is two loads for sixteen FMAs. The closing C += acc is a VFADD.
func gemm32Kernel() *emit.Function {
b := arm64.NewFunc("gemmMicro8x8S", gemmSig(), 0)
b.LoadArg("kc", "R0").
LoadArg("pa", "R1").
LoadArg("pb", "R2").
LoadArg("c", "R3").
LoadArg("ldc", "R4").
Raw("LSL $2, R4, R4") // ldc -> bytes
for r := 0; r < 16; r++ {
b.Raw(fmt.Sprintf("VEOR V%d.B16, V%d.B16, V%d.B16", r, r, r))
}
b.Raw("CBZ R0, sstore").
Raw("skloop:").
Raw("VLD1.P 32(R2), [V16.S4, V17.S4]"). // B row: 8 floats
Raw("VLD1.P 32(R1), [V18.S4, V19.S4]") // A column: 8 floats
for r := 0; r < 8; r++ {
areg, lane := 18+r/4, r%4
b.Raw(fmlaElemS(2*r, 16, areg, lane))
b.Raw(fmlaElemS(2*r+1, 17, areg, lane))
}
b.Raw("SUB $1, R0").
Raw("CBNZ R0, skloop").
Raw("sstore:")
for r := 0; r < 8; r++ {
b.Raw("VLD1 (R3), [V16.S4, V17.S4]")
b.Raw(fmt.Sprintf("VFADD V%d.S4, V16.S4, V16.S4", 2*r))
b.Raw(fmt.Sprintf("VFADD V%d.S4, V17.S4, V17.S4", 2*r+1))
b.Raw("VST1 [V16.S4, V17.S4], (R3)")
if r < 7 {
b.Raw("ADD R4, R3, R3")
}
}
b.Ret()
return b.Func()
}

// sumKernel builds sumNEON(a *float64, n int) float64.
//
// The lane-parallel accumulation is VFMLA against a vector of 1.0: acc += src *
Expand Down
20 changes: 20 additions & 0 deletions internal/kernels/fence_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -135,3 +135,23 @@ func TestGemmMicroStaysInsideItsOperands(t *testing.T) {
run()
forEachFallback(run)
}

// TestGemmMicro32StaysInsideItsOperands fences the float32 GEMM kernel's
// panels and C tile, where there is one.
func TestGemmMicro32StaysInsideItsOperands(t *testing.T) {
g := gemm32
if g == nil {
t.Skip("no float32 GEMM kernel here")
}
for kc := 1; kc <= 9; kc++ {
for _, ldc := range []int{g.nr, g.nr + 3, 64} {
for _, pa := range fences32(t, kc*g.mr) {
for _, pb := range fences32(t, kc*g.nr) {
for _, dst := range fences32(t, (g.mr-1)*ldc+g.nr) {
g.micro(kc, pa, pb, dst, ldc)
}
}
}
}
}
}
Loading
Loading