Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
scalars with `Scalar`). Float64 is the default and the most SIMD-accelerated
one. Float32 has SIMD `Sum`, `Add`/`Sub`/`Mul`/`Div` and `Sqrt` on every
64-bit target, and `Max`/`Min` on every 64-bit target, bit-identical to its
scalar loops. Its `MatMul` runs on a native float32 GEMM on arm64 (NEON)
and amd64 with FMA (AVX2), accumulating in float32 like NumPy's sgemm, and
scalar loops. Its `MatMul` runs on a native float32 GEMM on arm64 (NEON),
amd64 with FMA (AVX2) and ppc64le (VSX), accumulating in float32 like NumPy's sgemm, and
elsewhere on the float64 GEMM (accumulated in float64, rounded once)
([docs/design-dtypes.md](docs/design-dtypes.md)).
- **Creation** — `New`/`Zeros`/`Ones`/`Full`/`FromData`/`Arange`/`Linspace`/
Expand Down Expand Up @@ -155,7 +155,7 @@ and `Max` by 92–96% ([docs/perf.md](docs/perf.md), *SIMD coverage*). It is a
> pair Go supports; v0.1.0 and v0.2.0 were also run on real amd64, arm64,
> ppc64le, riscv64 and loong64 hardware, v0.3.0 on real amd64, arm64, ppc64le
> and loong64. See **[docs/plan-ndarray.md](docs/plan-ndarray.md)**
> for the roadmap (a native float32 GEMM on the other four 64-bit targets;
> for the roadmap (a native float32 GEMM on s390x, riscv64 and loong64;
> linear algebra decompositions).

## Why this module?
Expand Down
6 changes: 6 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ standard library's known vulnerabilities; `govulncheck ./...` reports none.

## Audit log

**2026-10-27 (v0.28.0).** The ppc64le float32 GEMM kernel
(`gemmMicro8x16VSX`) reads `kc·8` and `kc·16` packed floats and one 8×16 tile
of C, which its wrapper bounds-checks before the call. It also reads two
16-byte read-only mask constants. It is fenced and tested against the float32
oracle on a real POWER9, and a sabotaged kernel failed CI's ppc64le lane.

**2026-10-27 (v0.27.0).** The unpacked-A GEMM kernels (`gemmUA*` on amd64
and arm64) read A in place: `mr` rows `lda` elements apart, `kc` elements
each. Their Go wrapper bounds-checks the whole reach,
Expand Down
22 changes: 21 additions & 1 deletion docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -1275,9 +1275,29 @@ Against NumPy 2.5.4 (OpenBLAS, one core), the ratio ours / NumPy:
| N1 float32 | 1.20 | 1.06 | 1.09 |
| N1 float64 | 1.18 | 1.11 | 1.18 |

### ppc64le float32 GEMM (2026-10-27)

`gemmMicro8x16VSX` follows OpenBLAS's POWER9 sgemm
(`kernel/power/sgemm_macros_power9.S`, `LOAD8x16`) and uses no splats. Each
4-float A vector is rotated three ways (`XXSLDWI a, a, $v`), and every
rotation multiplies every B vector: 32 `XVMADDASP` per k step against 6 loads
and 6 rotations. The tile therefore accumulates in a rotated layout, and the
store untangles each 4×4 block with three `XXSEL`s per row. OpenBLAS rotates
B; rotating A costs half as many permutes, because A has two vectors per k
step and B four.

| float32 `MatMul` vs v0.27.0 (`abbench`, 8 rounds, POWER9) | 64² | 256² | 1024² |
|---|---:|---:|---:|
| widened to the float64 GEMM → native | **−59%** | **−42%** | **−37%** |

The NumPy 2.5.3 wheel on that host is no reference for products: x @ x takes
0.48 ms at 64² and 2.7 s at 1024², for float32 and float64 alike. That is
14–31× slower than us, consistent with a BLAS that has no POWER9 kernels. A
comparison with a tuned POWER9 OpenBLAS remains to be made.

**Open defect: products are still up to 1.2× slower than OpenBLAS on one
core.** The exception is float64 at 1024² on Zen 3, where we are 3% faster.
The kernels for ppc64le, s390x, riscv64 and loong64 are still to write.
The float32 kernels for s390x, riscv64 and loong64 are still to write.

**Open: the allocating operations.** `Add` is still 1.5–3.2× slower than
NumPy's `a+b`, for float64 as well. The kernel is no longer the difference,
Expand Down
5 changes: 3 additions & 2 deletions docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,8 +161,9 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
vectorised them on all six 64-bit targets, for float32 and float64), and the transcendental ufuncs. `MatMul` runs on the float64
GEMM since v0.21.0, 77–90% faster than the triple loop, and since v0.26.0
on a native float32 GEMM on arm64 (NEON 8×8) and amd64 with FMA (6×16),
a further 53–61% faster. The float32 kernels for ppc64le, s390x, riscv64
and loong64 are still to write.
a further 53–61% faster, and since v0.28.0 on ppc64le (VSX 8×16, 37–59%
faster on a POWER9). The float32 kernels for s390x, riscv64 and loong64
are still to write.
- **Phase 3 — broadcasting ufuncs — MOSTLY DONE.** Broadcasting arithmetic,
comparisons returning Bool masks, `Maximum`/`Minimum`, `Logical*`, `Power`,
`Where` and the per-axis reductions, in every dtype. Not done: a public
Expand Down
87 changes: 87 additions & 0 deletions internal/kernels/asmgen/ppc64le/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ func main() {
f.Add(extremumKernel("minVSX", 64, false, abs64, inf64))
f.Add(extremumKernel("max32VSX", 32, true, abs32, inf32))
f.Add(extremumKernel("min32VSX", 32, false, abs32, inf32))
f.Add(gemm32Kernel(f.Data("oddw", words(1, 3)), f.Data("highw", words(2, 3))))
if err := os.WriteFile("sum_ppc64le.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
Expand Down Expand Up @@ -442,6 +443,92 @@ func gemmKernel() *emit.Function {
return b.Func()
}

// gemm32Kernel builds gemmMicro8x16VSX(kc int, pa, pb, c *float32, ldc int):
//
// for p in [0,kc): C[r][col] += pa[p*8+r] * pb[p*16+col] (r<8, col<16)
//
// with the scheme of OpenBLAS's POWER9 sgemm (kernel/power/sgemm_macros_power9.S,
// LOAD8x16): no splats. Each 4-float A vector is rotated three ways, and the
// rotations multiply each B vector, so the tile is accumulated in a rotated
// layout that the store untangles once. OpenBLAS rotates B with xxperm and
// xxpermdi; this kernel rotates A, which has two vectors per k step against
// B's four, with the cyclic word shift XXSLDWI a, a, $v: rotation v holds
// a[(i+v)%4] in lane i (lanes numbered as LXVW4X loads them, memory order).
//
// Accumulator VS(16p+4k+v) lane i is then C[4p+(i+v)%4][4k+i]: A vector p
// (rows 4p..4p+3), B vector k (columns 4k..4k+3), rotation v. Row 4p+r of
// that 4x4 block takes lane i from rotation (r-i) mod 4, so it is three
// selects: x = lanes 0,1 from rotations r, r-1; y = lanes 2,3 from r-2, r-3;
// row = x's low half and y's high half (XXSEL against the odd-lanes and
// high-lanes masks, VS48 and VS49).
//
// Per k step: 2 A loads, 6 rotations, 4 B loads, 32 XVMADDASP.
func gemm32Kernel(odd, high string) *emit.Function {
name := "gemmMicro8x16VSX"
sig := ppc64.Layout([]string{"kc", "pa", "pb", "c", "ldc"},
[]ppc64.Type{ppc64.Int64, ppc64.Ptr, ppc64.Ptr, ppc64.Ptr, ppc64.Int64}, nil, nil)
b := ppc64.NewFunc(name, sig, 0)
b.LoadArg("kc", "R3").LoadArg("pa", "R4").LoadArg("pb", "R5").LoadArg("c", "R6").LoadArg("ldc", "R7")
b.Raw("SLD $2, R7")
offsets(b, false)
b.Raw("MOVD $%s+0(SB), R8", odd).Raw("LXVW4X (R8)(R0), VS48").
Raw("MOVD $%s+0(SB), R8", high).Raw("LXVW4X (R8)(R0), VS49")
for r := 0; r < 32; r++ {
b.Raw("XXLXOR VS%d, VS%d, VS%d", r, r, r)
}
b.Raw("CMP R3, $0").Raw("BEQ %sstore", name)
b.Label(name+"loop").
Raw("LXVW4X (R4)(R0), VS32").Raw("LXVW4X (R4)(R14), VS36")
for k := 0; k < 4; k++ {
b.Raw("LXVW4X (R5)(%s), VS%d", vidx(k), 40+k)
}
for p := 0; p < 2; p++ {
for v := 1; v < 4; v++ {
b.Raw("XXSLDWI VS%d, VS%d, $%d, VS%d", 32+4*p, 32+4*p, v, 32+4*p+v)
}
}
for p := 0; p < 2; p++ {
for k := 0; k < 4; k++ {
for v := 0; v < 4; v++ {
b.XVMADDASP(16*p+4*k+v, 32+4*p+v, 40+k)
}
}
}
b.Raw("ADD $32, R4").Raw("ADD $64, R5").Raw("ADD $-1, R3").
Raw("CMP R3, $0").Raw("BNE %sloop", name)
b.Label(name + "store")
acc := func(p, k, v int) int { return 16*p + 4*k + (v+4)%4 }
for p := 0; p < 2; p++ {
for r := 0; r < 4; r++ {
for k := 0; k < 4; k++ {
b.Raw("XXSEL VS%d, VS%d, VS48, VS44", acc(p, k, r), acc(p, k, r-1)).
Raw("XXSEL VS%d, VS%d, VS48, VS45", acc(p, k, r-2), acc(p, k, r-3)).
Raw("XXSEL VS44, VS45, VS49, VS46").
Raw("LXVW4X (R6)(%s), VS47", vidx(k))
b.XVADDSP(47, 47, 46)
b.Raw("STXVW4X VS47, (R6)(%s)", vidx(k))
}
if !(p == 1 && r == 3) {
b.Raw("ADD R7, R6")
}
}
}
b.Ret()
return b.Func()
}

// words is a 16-byte mask with all-ones in the given 4-byte lanes (memory
// order, as LXVW4X loads them) and zeros elsewhere.
func words(on ...int) []byte {
m := make([]byte, 16)
for _, w := range on {
for i := 0; i < 4; i++ {
m[4*w+i] = 0xff
}
}
return m
}

// blocks8Kernel builds blocks8VSX(r, a *float32, blocks int): for each of the
// blocks of eight float32 at a, in order, r[j] += a[8i+j] (NumPy's pairwise-sum
// block). The eight accumulators are split over two VSX registers, VS0 and VS1
Expand Down
2 changes: 1 addition & 1 deletion internal/kernels/gemm32_other.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build !amd64 && !arm64
//go:build !amd64 && !arm64 && !ppc64le

package kernels

Expand Down
16 changes: 16 additions & 0 deletions internal/kernels/gemm32_ppc64le.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package kernels

// gemm32 is the float32 GEMM on the VSX 8x16 kernel (gemmMicro8x16VSX in
// sum_ppc64le.s, generated by go-asmgen; see asmgen/ppc64le/gen.go).
var gemm32 = &gemmKernel[float32]{mr: 8, nr: 16, micro: gemmMicro32}

// gemmMicro32 adds the 8 x 16 tile sum_p pa[p*8..]*pb[p*16..] into the C block
// at dst[0:], whose rows are ldc apart. pa, pb hold at least kc*8 and kc*16
// packed (zero-padded) floats; dst has at least 7*ldc + 16 elements.
func gemmMicro32(kc int, pa, pb, dst []float32, ldc int) {
_, _, _ = pa[kc*8-1], pb[kc*16-1], dst[7*ldc+15] // the kernel's whole reach
gemmMicro8x16VSX(kc, &pa[0], &pb[0], &dst[0], ldc)
}

//go:noescape
func gemmMicro8x16VSX(kc int, pa, pb, c *float32, ldc int)
Loading
Loading