Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,15 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-27 (v0.27.0).** The unpacked-A GEMM kernels (`gemmUA*` on amd64
and arm64) read A in place: `mr` rows `lda` elements apart, `kc` elements
each. Their Go wrapper bounds-checks the whole reach,
`a[(mr-1)*lda+kc-1]`, the packed B panel and the C tile before the call.
`gemmBand` takes this path only for a unit column stride and a positive row
stride, so a reversed or broadcast view is still packed. The kernels are
fenced for every tile height and two row strides. A sabotaged kernel on each
architecture failed CI.

**2026-10-26 (v0.26.0).** The float32 GEMM kernels, `gemmMicro8x8S` (arm64)
and `gemmMicro6x16FMA` (amd64, behind the FMA probe), read `kc·MR` and `kc·NR`
packed floats and read and write one MR×NR tile of C. Their Go wrapper
Expand Down
40 changes: 35 additions & 5 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1243,11 +1243,41 @@ element. Profiling had put it at 21% of a 64² float32 product.
| Neoverse N1, float32 | −16.4% | −8.9% | ~ |
| Neoverse N1, float64 | −9.5% | ~ | ~ |

**Open defect: products are still slower than OpenBLAS on one core**, by
1.1–1.3× at 1024² and up to 1.9× at 64². The gap is widest where the work
around the kernel weighs most: packing, edge tiles, and the result's
allocation and zeroing. The kernels for ppc64le, s390x, riscv64 and loong64
are still to write.
### Reading A in place (2026-10-27)

v0.27.0 no longer packs a row-major A (unit column stride, positive row
stride). Unpacked-A kernels read it in place: the "rv" gemmsup kernels of
BLIS (`kernels/haswell/3/sup/bli_gemmsup_rv_haswell_asm_s6x16m.c`). B stays
packed, so its last panel is still zero-padded. There is one kernel per tile
height, so a band's bottom rows never read past A's last row.

- **amd64 (FMA):** each A value is broadcast straight from its row through
a scaled index (`VBROADCASTSS (SI)(R8*2)`), as BLIS does.
- **arm64:** the same idea, but broadcast loads (`VLD1R`) made it 26–78%
slower on a Neoverse N1, apparently because they compete with the FMAs for
the vector pipes. What shipped instead loads 4 floats (2 doubles) of each
row with one plain vector load, and the next k steps read them by lane with
the by-element FMLA.

| `MatMul` vs v0.26.1 (`abbench`, 10 rounds) | 64² | 256² | 1024² |
|---|---:|---:|---:|
| Zen 3, float32 | −28% | −9.5% | −3.9% |
| Zen 3, float64 | −17% | −5.5% | ~ |
| Neoverse N1, float32 | −20% | −7.7% | ~ |
| Neoverse N1, float64 | −12% | −4.5% | ~ |

Against NumPy 2.5.4 (OpenBLAS, one core), the ratio ours / NumPy:

| | 64² | 256² | 1024² |
|---|---:|---:|---:|
| Zen 3 float32 | 1.18 | 1.13 | 1.04 |
| Zen 3 float64 | 1.16 | 1.03 | **0.97** |
| N1 float32 | 1.20 | 1.06 | 1.09 |
| N1 float64 | 1.18 | 1.11 | 1.18 |

**Open defect: products are still up to 1.2× slower than OpenBLAS on one
core.** The exception is float64 at 1024² on Zen 3, where we are 3% faster.
The kernels for ppc64le, s390x, riscv64 and loong64 are still to write.

**Open: the allocating operations.** `Add` is still 1.5–3.2× slower than
NumPy's `a+b`, for float64 as well. The kernel is no longer the difference,
Expand Down
89 changes: 89 additions & 0 deletions internal/kernels/asmgen/amd64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ func main() {
f.Add(binKernel("divSSE2", "DIVPD", "DIVSD"))
f.Add(gemmFMAKernel())
f.Add(gemm32FMAKernel())
for mr := 1; mr <= gemmMR; mr++ {
f.Add(gemm32UAKernel(mr))
f.Add(gemmUAKernel(mr, false))
}
f.Add(dotFMAKernel())
f.Add(gemmSSE2HalfKernel())
f.Add(amd64.FeatureProbe("hasFMA", amd64.FMA))
Expand Down Expand Up @@ -549,6 +553,91 @@ func gemm32FMAKernel() *emit.Function {
return b.Func()
}

// gemm32UAKernel builds gemmUA<mr>x16FMA(kc int, a *float32, lda int, pb, c
// *float32, ldc int): the 6x16 float32 tile with A read in place instead of
// packed, for mr rows (1..6):
//
// for p in [0,kc): C[r][col] += a[r*lda+p] * pb[p*16+col] (r<mr, col<16)
//
// It is the "rv" sup kernel of BLIS (bli_gemmsup_rv_haswell_asm_s6x16m.c):
// each A value is broadcast straight from its row, r*lda bytes past the
// k-step's column, through a scaled index (lda, 2lda, 3lda in R11, 4lda,
// 5lda in R12), and the column advances 4 bytes per step. A small product
// then pays no copy of A at all. B stays packed, so a partial last panel is
// still zero-padded; the kernels for mr < 6 cover the bottom rows of a block
// without reading past A's last row.
//
// gemmUAKernel(mr, false) is the float64 counterpart over the 6x8 packing,
// gemmUA<mr>x8FMA: the same loop in PD instructions, 8-byte steps.
func gemm32UAKernel(mr int) *emit.Function { return gemmUAKernel(mr, true) }

func gemmUAKernel(mr int, single bool) *emit.Function {
sig := amd64.Layout(
[]string{"kc", "a", "lda", "pb", "c", "ldc"},
[]amd64.Type{amd64.Int64, amd64.Ptr, amd64.Int64, amd64.Ptr, amd64.Ptr, amd64.Int64},
nil, nil,
)
name, size, shift, sfx, bcast := fmt.Sprintf("gemmUA%dx8FMA", mr), 8, 3, "PD", "VBROADCASTSD"
if single {
name, size, shift, sfx, bcast = fmt.Sprintf("gemmUA%dx16FMA", mr), 4, 2, "PS", "VBROADCASTSS"
}
b := amd64.NewFunc(name, sig, 0)
b.LoadArg("kc", "CX")
b.LoadArg("a", "SI")
b.LoadArg("lda", "R8")
b.LoadArg("pb", "DI")
b.LoadArg("c", "DX")
b.LoadArg("ldc", "R9")
b.Raw(fmt.Sprintf("SHLQ $%d, R8", shift)) // lda -> bytes
b.Raw(fmt.Sprintf("SHLQ $%d, R9", shift)) // ldc -> bytes
b.Raw("LEAQ (R8)(R8*2), R11") // 3*lda
b.Raw("LEAQ (R8)(R8*4), R12") // 5*lda
b.Raw("MOVQ DX, R10")
for r := 0; r < mr; r++ {
b.Raw("PREFETCHT0 (R10)")
b.Raw("PREFETCHT0 63(R10)")
if r < mr-1 {
b.Raw("ADDQ R9, R10")
}
}
for r := 0; r < 2*mr; r++ {
b.Raw(fmt.Sprintf("VXOR%s Y%d, Y%d, Y%d", sfx, r, r, r))
}
b.Raw("TESTQ CX, CX")
b.Raw("JZ uastore")
b.Raw("uakloop:")
b.Raw(fmt.Sprintf("VMOVU%s (DI), Y12", sfx))
b.Raw(fmt.Sprintf("VMOVU%s 32(DI), Y13", sfx))
for r := 0; r < mr; r++ {
b.Raw(fmt.Sprintf("%s %s, Y14", bcast, uaRow(r)))
b.Raw(fmt.Sprintf("VFMADD231%s Y12, Y14, Y%d", sfx, 2*r))
b.Raw(fmt.Sprintf("VFMADD231%s Y13, Y14, Y%d", sfx, 2*r+1))
}
b.Raw(fmt.Sprintf("ADDQ $%d, SI", size))
b.Raw("ADDQ $64, DI") // one packed B row: 16 floats or 8 doubles
b.Raw("DECQ CX")
b.Raw("JNZ uakloop")
b.Raw("uastore:")
for r := 0; r < mr; r++ {
b.Raw(fmt.Sprintf("VADD%s (DX), Y%d, Y15", sfx, 2*r))
b.Raw(fmt.Sprintf("VMOVU%s Y15, (DX)", sfx))
b.Raw(fmt.Sprintf("VADD%s 32(DX), Y%d, Y15", sfx, 2*r+1))
b.Raw(fmt.Sprintf("VMOVU%s Y15, 32(DX)", sfx))
if r < mr-1 {
b.Raw("ADDQ R9, DX")
}
}
b.Raw("VZEROUPPER")
b.Ret()
return b.Func()
}

// uaRow is the address of row r of the unpacked A column at SI, rows lda
// bytes apart (R8), 3*lda in R11 and 5*lda in R12.
func uaRow(r int) string {
return [...]string{"(SI)", "(SI)(R8*1)", "(SI)(R8*2)", "(SI)(R11*1)", "(SI)(R8*4)", "(SI)(R12*1)"}[r]
}

// fma32Step emits one k step of the 6x16 float32 FMA tile.
func fma32Step(b *amd64.Builder, step int) {
ao, bo := 4*gemmMR*step, 4*gemm32NR*step
Expand Down
117 changes: 117 additions & 0 deletions internal/kernels/asmgen/arm64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ package main
import (
"fmt"
"os"
"strings"

"github.com/go-asmgen/asmgen/arm64"
"github.com/go-asmgen/asmgen/emit"
Expand Down Expand Up @@ -58,6 +59,12 @@ func main() {
f.Add(extremum32Kernel("min32NEON", "VFMIN", "FMINS"))
f.Add(gemmKernel())
f.Add(gemm32Kernel())
for mr := 1; mr <= 8; mr++ {
f.Add(gemmUAKernel(mr, true))
if mr <= 4 {
f.Add(gemmUAKernel(mr, false))
}
}
f.Add(blocks8Kernel())
if err := os.WriteFile("sum_arm64.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
Expand Down Expand Up @@ -471,6 +478,116 @@ func gemmKernel() *emit.Function {
return b.Func()
}

// gemmUAKernel builds the unpacked-A tiles (the "rv" sup kernels of BLIS,
// bli_gemmsup_rv_*): A is read in place, mr rows lda elements apart, instead
// of from a packed panel; B stays packed.
//
// single: gemmUA<mr>x8S(kc int, a *float32, lda int, pb, c *float32, ldc int), mr <= 8
// double: gemmUA<mr>x8D(kc int, a *float64, lda int, pb, c *float64, ldc int), mr <= 4
//
// computing C[r][col] += a[r*lda+p] * pb[p*8+col] for r < mr. Each row has its
// own pointer (R5..R12). A row is contiguous along k, so one plain vector load
// brings U consecutive k values of it (U = 4 floats or 2 doubles) into V20+r,
// and the U k steps that follow take their A value from lane s of that vector
// with the by-element FMLA (fmlaElem/fmlaElemS), as the packed kernels do.
// The kc%U remaining steps load each row's value as a scalar into lane 0.
// A first version loaded every A value with a load-and-replicate (VLD1R) and
// used the vector FMLA: on a Neoverse N1 it was 26-78% slower than packing A,
// apparently because the replicating loads compete with the FMAs for the
// vector pipes.
// Accumulators: float32 row r in V(2r), V(2r+1); float64 row r in V(4r)..V(4r+3).
func gemmUAKernel(mr int, single bool) *emit.Function {
sig := arm64.Layout(
[]string{"kc", "a", "lda", "pb", "c", "ldc"},
[]arm64.Type{arm64.Int64, arm64.Ptr, arm64.Int64, arm64.Ptr, arm64.Ptr, arm64.Int64},
nil, nil,
)
name, shift, arr, vecs := fmt.Sprintf("gemmUA%dx8D", mr), 3, "D2", 4
if single {
name, shift, arr, vecs = fmt.Sprintf("gemmUA%dx8S", mr), 2, "S4", 2
}
size := 1 << shift
bregs := make([]string, vecs)
for v := range bregs {
bregs[v] = fmt.Sprintf("V%d.%s", 16+v, arr)
}
blist := "[" + strings.Join(bregs, ", ") + "]"
b := arm64.NewFunc(name, sig, 0)
b.LoadArg("kc", "R0").
LoadArg("a", "R1").
LoadArg("lda", "R2").
LoadArg("pb", "R13").
LoadArg("c", "R3").
LoadArg("ldc", "R4").
Raw(fmt.Sprintf("LSL $%d, R2, R2", shift)).
Raw(fmt.Sprintf("LSL $%d, R4, R4", shift))
for r := 0; r < mr; r++ { // row pointers R5.. = a + r*lda
if r == 0 {
b.Raw("MOVD R1, R5")
} else {
b.Raw(fmt.Sprintf("ADD R2, R%d, R%d", 4+r, 5+r))
}
}
for v := 0; v < mr*vecs; v++ {
b.Raw(fmt.Sprintf("VEOR V%d.B16, V%d.B16, V%d.B16", v, v, v))
}
u := 16 / size // k steps per A vector
fmla, scalar := fmlaElem, "FMOVD"
if single {
fmla, scalar = fmlaElemS, "FMOVS"
}
step := func(lane int) {
b.Raw(fmt.Sprintf("VLD1.P %d(R13), %s", 8*size, blist))
for r := 0; r < mr; r++ {
for v := 0; v < vecs; v++ {
b.Raw(fmla(vecs*r+v, 16+v, 20+r, lane))
}
}
}
b.Raw(fmt.Sprintf("LSR $%d, R0, R14", shift4(u))). // whole groups of u steps
Raw(fmt.Sprintf("AND $%d, R0, R15", u-1)). // remaining steps
Raw("CBZ R14, uatail").
Raw("uakloop:")
for r := 0; r < mr; r++ {
b.Raw(fmt.Sprintf("VLD1.P 16(R%d), [V%d.%s]", 5+r, 20+r, arr))
}
for s := 0; s < u; s++ {
step(s)
}
b.Raw("SUB $1, R14").
Raw("CBNZ R14, uakloop").
Raw("uatail:").
Raw("CBZ R15, uastore").
Raw("uatloop:")
for r := 0; r < mr; r++ {
b.Raw(fmt.Sprintf("%s.P %d(R%d), F%d", scalar, size, 5+r, 20+r))
}
step(0)
b.Raw("SUB $1, R15").
Raw("CBNZ R15, uatloop").
Raw("uastore:")
for r := 0; r < mr; r++ {
b.Raw(fmt.Sprintf("VLD1 (R3), %s", blist))
for v := 0; v < vecs; v++ {
b.Raw(fmt.Sprintf("VFADD V%d.%s, V%d.%s, V%d.%s", vecs*r+v, arr, 16+v, arr, 16+v, arr))
}
b.Raw(fmt.Sprintf("VST1 %s, (R3)", blist))
if r < mr-1 {
b.Raw("ADD R4, R3, R3")
}
}
b.Ret()
return b.Func()
}

// shift4 is log2 of u, 2 or 4.
func shift4(u int) int {
if u == 4 {
return 2
}
return 1
}

// fmlaElemS is fmlaElem for single precision:
//
// FMLA V{acc}.4S, V{breg}.4S, V{areg}.S[idx] => V{acc} += V{breg} * V{areg}[idx]
Expand Down
29 changes: 29 additions & 0 deletions internal/kernels/fence_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,23 @@ func TestGemmMicroStaysInsideItsOperands(t *testing.T) {
}
run()
forEachFallback(run)
if gemm64.microUA == nil {
return
}
for kc := 1; kc <= 9; kc++ {
for _, ldc := range []int{NR, NR + 3} {
pb := fences(t, kc*NR)[1]
for mr := 1; mr <= MR; mr++ {
for _, lda := range []int{kc, kc + 5} {
for _, a := range fences(t, (mr-1)*lda+kc) {
for _, dst := range fences(t, (mr-1)*ldc+NR) {
gemm64.microUA(kc, a, lda, pb, dst, ldc, mr)
}
}
}
}
}
}
}

// TestGemmMicro32StaysInsideItsOperands fences the float32 GEMM kernel's
Expand All @@ -150,6 +167,18 @@ func TestGemmMicro32StaysInsideItsOperands(t *testing.T) {
for _, dst := range fences32(t, (g.mr-1)*ldc+g.nr) {
g.micro(kc, pa, pb, dst, ldc)
}
if g.microUA == nil {
continue
}
for mr := 1; mr <= g.mr; mr++ {
for _, lda := range []int{kc, kc + 5} {
for _, a := range fences32(t, (mr-1)*lda+kc) {
for _, dst := range fences32(t, (mr-1)*ldc+g.nr) {
g.microUA(kc, a, lda, pb, dst, ldc, mr)
}
}
}
}
}
}
}
Expand Down
Loading
Loading