Skip to content

Commit 59b9df3

Browse files
tannevaledclaude
andcommitted
perf(gemm): pack a row-major A row by row; docs for the float32 GEMM
packA gathered A column by column, paying its loop and bounds checks every MR elements. At 64² that took 33% of a float32 product, more than the micro-kernel. A row-major A (CS == 1) is now read one contiguous row at a time and scattered into its lane of the panel. packB goes back to element loops: copy and clear on NR-element runs are runtime calls. abbench, 10 rounds, one core, against main: Zen 3 float32 -61/-56/-55% (64²/256²/1024²), float64 -5.8/-4.2/~ N1 float32 -64/-60/-57%, float64 ~/~/~ perf.md gains the measurements and the NumPy 2.5.4 comparison (still 1.1-1.9x behind OpenBLAS: an open defect). README, plan and SECURITY are updated too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 7ff1b34 commit 59b9df3

5 files changed

Lines changed: 109 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,9 @@
1717
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
1818
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
1919
64-bit target, and `Max`/`Min` on every 64-bit target, bit-identical to its
20-
scalar loops; its `MatMul` runs on the float64 GEMM (accumulated in float64,
21-
rounded once)
20+
scalar loops. Its `MatMul` runs on a native float32 GEMM on arm64 (NEON)
21+
and amd64 with FMA (AVX2), accumulating in float32 like NumPy's sgemm, and
22+
elsewhere on the float64 GEMM (accumulated in float64, rounded once)
2223
([docs/design-dtypes.md](docs/design-dtypes.md)).
2324
- **Creation** — `New`/`Zeros`/`Ones`/`Full`/`FromData`/`Arange`/`Linspace`/
2425
`Eye`/`Identity`.
@@ -154,8 +155,8 @@ and `Max` by 92–96% ([docs/perf.md](docs/perf.md), *SIMD coverage*). It is a
154155
> pair Go supports; v0.1.0 and v0.2.0 were also run on real amd64, arm64,
155156
> ppc64le, riscv64 and loong64 hardware, v0.3.0 on real amd64, arm64, ppc64le
156157
> and loong64. See **[docs/plan-ndarray.md](docs/plan-ndarray.md)**
157-
> for the roadmap (float32 SIMD for max/min and the products; linear algebra
158-
> decompositions).
158+
> for the roadmap (a native float32 GEMM on the other four 64-bit targets;
159+
> linear algebra decompositions).
159160
160161
## Why this module?
161162

‎SECURITY.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,18 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.
6666

6767
## Audit log
6868

69+
**2026-10-26 (v0.26.0).** The float32 GEMM kernels, `gemmMicro8x8S` (arm64)
70+
and `gemmMicro6x16FMA` (amd64, behind the FMA probe), read `kc·MR` and `kc·NR`
71+
packed floats and read and write one MR×NR tile of C. Their Go wrapper
72+
bounds-checks all three reaches before the call. They are fenced (PROT_NONE
73+
on both sides of each operand) and tested against a float32 oracle. A
74+
sabotaged kernel failed every CI job that runs it. The edge tile now lives
75+
in the worker's pooled A buffer: its capacity adds MR·NR elements, and
76+
`gemmBand` slices the tile from the end. The operand views are the same
77+
`Off/RS/CS` triples the float64 GEMM reads, built from the array's own
78+
strides. A non-Float32 operand is converted to a fresh contiguous copy
79+
first. Shapes are validated (overflow included) before any allocation.
80+
6981
**2026-10-26 (v0.25.0).** The loong64 extremum kernels (`maxLSX`, `minLSX`,
7082
`max32LSX`, `min32LSX`) read the whole blocks of `a` behind the `useLASX`
7183
gate. They write one 16-byte vector to an `out` array on the wrapper's stack,

‎docs/perf.md‎

Lines changed: 53 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1189,6 +1189,55 @@ float32 is now within the widening of float64: 86.7 against 54.0 µs at 64²,
11891189
lanes per register and half the bytes, could go further; it is the next step
11901190
for products.
11911191

1192+
### A native float32 GEMM on amd64 and arm64 (2026-10-26)
1193+
1194+
The packed GEMM driver (packing, cache blocking, the parallel bands) is now
1195+
generic over float32 and float64. Each element type brings its micro-kernel,
1196+
its tile and its pack-buffer pools. float32 gets two kernels, generated by
1197+
go-asmgen:
1198+
1199+
- **arm64:** `gemmMicro8x8S`, a NEON 8×8 tile. It has 16 accumulators and
1200+
uses the by-element `FMLA Vd.4S, Vn.4S, Vm.S[i]`, encoded as a `WORD` and
1201+
checked against `go tool objdump`.
1202+
- **amd64:** `gemmMicro6x16FMA`, the BLIS Haswell sgemm shape: a 6×16 tile,
1203+
`VBROADCASTSS` and `VFMADD231PS` into 12 YMM accumulators. It runs only
1204+
where the CPU has FMA.
1205+
1206+
float32 `MatMul` (and `Dot`, which goes through it) now accumulates in
1207+
float32, as NumPy's sgemm does. Without a kernel (no FMA, or the other four
1208+
targets) it still widens to the float64 GEMM. The tests check it against a
1209+
float32 ikj oracle, bit for bit on integer data, in every block
1210+
configuration, on a transposed and offset B view. On random data they bound
1211+
it by k·eps32·Σ|ab|. A sabotaged kernel (a wrong A lane on arm64, a wrong A
1212+
row on amd64) failed every job that runs one.
1213+
1214+
Profiling the first version at 64² found `packA` at 33% of the time, ahead
1215+
of the micro-kernel. It gathered A column by column and paid its loop and
1216+
bounds checks every MR elements. A row-major A is now read one contiguous
1217+
row at a time, which also speeds up float64.
1218+
1219+
| `MatMul` (`abbench`, 10 rounds, 1 core) | 64² | 256² | 1024² |
1220+
|---|---:|---:|---:|
1221+
| Zen 3, float32 | **−61%** | **−56%** | **−55%** |
1222+
| Zen 3, float64 | −5.8% | −4.2% | ~ |
1223+
| Neoverse N1, float32 | **−64%** | **−60%** | **−57%** |
1224+
| Neoverse N1, float64 | ~ | ~ | ~ |
1225+
1226+
Against NumPy 2.5.4 (OpenBLAS, `OPENBLAS_NUM_THREADS=1`, the same core):
1227+
1228+
| | 64² | 256² | 1024² |
1229+
|---|---:|---:|---:|
1230+
| Zen 3 float32, ours / NumPy | 15.0 / 8.1 µs | 485 / 366 µs | 26.0 / 23.1 ms |
1231+
| Zen 3 float64 | 23.2 / 15.8 µs | 866 / 795 µs | 51.0 / 50.0 ms |
1232+
| N1 float32 | 30.4 / 16.2 µs | 1.19 / 0.94 ms | 68.2 / 60.6 ms |
1233+
| N1 float64 | 48.2 / 31.3 µs | 2.46 / 1.95 ms | 154 / 123 ms |
1234+
1235+
**Open defect: products are still slower than OpenBLAS on one core**, by
1236+
1.1–1.3× at 1024² and up to 1.9× at 64². The gap is widest where the work
1237+
around the kernel weighs most: packing, edge tiles, and the result's
1238+
allocation and zeroing. The kernels for ppc64le, s390x, riscv64 and loong64
1239+
are still to write.
1240+
11921241
**Open: the allocating operations.** `Add` is still 1.5–3.2× slower than
11931242
NumPy's `a+b`, for float64 as well. The kernel is no longer the difference,
11941243
as `AddInto` shows; the allocation is. It splits like this, measured
@@ -1224,12 +1273,14 @@ What does remove it:
12241273
`VFMADD231PD`, C and A prefetched) when go-asmgen's CPUID probe reports FMA
12251274
with the OS saving YMM state, else a pair of SSE2 6×4 `MULPD`+`ADDPD` tiles
12261275
over the same packing. Generated by go-asmgen and validated per-arch in CI;
1227-
the FMA kernel on real Cascade Lake and Zen 3 hosts.
1276+
the FMA kernel on real Cascade Lake and Zen 3 hosts. float32 products have
1277+
their own FMA kernel, `gemmMicro6x16FMA` (6×16, `VFMADD231PS`).
12281278
- **arm64 (NEON)** ships hand-vectorized `sum`, `dot`, **packed `sqrt`**
12291279
(`VFSQRT`), the **elementwise `add`/`sub`/`mul`/`div`** (`VFADD`/`VFSUB`/
12301280
`VFMUL`/`VFDIV`), **`max`/`min`** (`VFMAX`/`VFMIN`, four D2 accumulators from
12311281
8 elements up) and the **GEMM micro-kernel** (`gemmMicro4x8`: a 4×8 tile, 16 D2
1232-
accumulators, using the **by-lane FMLA** `FMLA Vd.2D, Vn.2D, Vm.D[i]`). Only
1282+
accumulators, using the **by-lane FMLA** `FMLA Vd.2D, Vn.2D, Vm.D[i]`), with
1283+
a float32 one, `gemmMicro8x8S` (8×8, `FMLA Vd.4S, Vn.4S, Vm.S[i]`). Only
12331284
the indexed-element FMLA is still a raw `WORD` (16 of them): Go's assembler
12341285
does not name it yet (pending as CL 764224).
12351286

‎docs/plan-ndarray.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,8 +159,10 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
159159
`Add`/`Sub`/`Mul`/`Div` (v0.18.0) and `Sqrt` (v0.19.0). Still pure Go
160160
for float32: nothing in `Max`/`Min` any more (v0.20.0 to v0.25.0
161161
vectorised them on all six 64-bit targets, for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
162-
GEMM since v0.21.0, 77–90% faster than the triple loop; a native float32
163-
GEMM is still to write.
162+
GEMM since v0.21.0, 77–90% faster than the triple loop, and since v0.26.0
163+
on a native float32 GEMM on arm64 (NEON 8×8) and amd64 with FMA (6×16),
164+
a further 53–61% faster. The float32 kernels for ppc64le, s390x, riscv64
165+
and loong64 are still to write.
164166
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
165167
comparisons returning Bool masks, `Maximum`/`Minimum`, `Logical*`, `Power`,
166168
`Where` and the per-axis reductions, in every dtype. Not done: a public

‎internal/kernels/gemm.go‎

Lines changed: 35 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -203,11 +203,17 @@ func (g *gemmKernel[T]) packB(b MatOf[T], pc, kc, jc, nc int, pb []T) {
203203
nr := min(NR, nc-jr)
204204
dst := pb[jr*kc:]
205205
if b.CS == 1 {
206+
// Element loops, not copy and clear: on runs this short (NR
207+
// elements) each of those is a runtime call.
206208
for p := 0; p < kc; p++ {
207209
srcRow := b.Data[b.Off+(pc+p)*b.RS+jc+jr:]
208210
d := dst[p*NR:]
209-
copy(d[:nr], srcRow[:nr])
210-
clear(d[nr:NR])
211+
for c := 0; c < nr; c++ {
212+
d[c] = srcRow[c]
213+
}
214+
for c := nr; c < NR; c++ {
215+
d[c] = 0
216+
}
211217
}
212218
continue
213219
}
@@ -217,9 +223,10 @@ func (g *gemmKernel[T]) packB(b MatOf[T], pc, kc, jc, nc int, pb []T) {
217223
dst[p*NR+c] = b.Data[src+p*b.RS]
218224
}
219225
}
220-
if nr < NR {
221-
for p := 0; p < kc; p++ {
222-
clear(dst[p*NR+nr : p*NR+NR])
226+
for p := 0; p < kc; p++ {
227+
d := dst[p*NR:]
228+
for c := nr; c < NR; c++ {
229+
d[c] = 0
223230
}
224231
}
225232
}
@@ -234,13 +241,34 @@ func (g *gemmKernel[T]) packA(a MatOf[T], ic, mc, pc, kc int, pa []T) {
234241
mr := min(MR, mc-ir)
235242
dst := pa[ir*kc:]
236243
src := a.Off + (ic+ir)*a.RS + pc*a.CS
244+
if a.CS == 1 {
245+
// Row-major A: read each row as one contiguous run and scatter it
246+
// into its lane of the panel. The column-wise walk below spends
247+
// its loop and bounds checks on every MR elements; at 64² it
248+
// took longer than the micro-kernel.
249+
for r := 0; r < mr; r++ {
250+
row := a.Data[src+r*a.RS : src+r*a.RS+kc]
251+
d := dst[r : r+(kc-1)*MR+1]
252+
for p, v := range row {
253+
d[p*MR] = v
254+
}
255+
}
256+
for r := mr; r < MR; r++ {
257+
for p := 0; p < kc; p++ {
258+
dst[p*MR+r] = 0
259+
}
260+
}
261+
continue
262+
}
237263
for p := 0; p < kc; p++ {
238-
d := dst[p*MR : p*MR+MR]
264+
d := dst[p*MR:]
239265
at := src + p*a.CS
240266
for r := 0; r < mr; r++ {
241267
d[r] = a.Data[at+r*a.RS]
242268
}
243-
clear(d[mr:])
269+
for r := mr; r < MR; r++ {
270+
d[r] = 0
271+
}
244272
}
245273
}
246274
}

0 commit comments

Comments
 (0)