Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ The numeric inner loops are kept behind a narrow kernel API. Behind it,
**large** elementwise ops, reductions and matmul run **multicore** (across
`GOMAXPROCS`); the sum reduction uses a **go-asmgen SIMD kernel** (NEON on
arm64, SSE2 on amd64); and `MatMul` is a **panel-packed, cache-blocked GEMM**
with a go-asmgen SIMD-FMA micro-kernel (NEON 4×8 on arm64, SSE2 4×4 on amd64) —
with a go-asmgen SIMD-FMA micro-kernel (NEON 4×8 on arm64; AVX2/FMA 6×8 on
amd64, chosen at run time, with an SSE2 fallback) —
the OpenBLAS/BLIS structure. So go-ndarray **beats single-threaded NumPy** on
its core ops on large arrays (Add/Mul ~2×, Sum ~2.4×), and `MatMul` reaches
**tuned-BLAS parity at 1024²**: ≈0.99× vs multi-threaded OpenBLAS 0.3.29 on an
Expand Down
86 changes: 72 additions & 14 deletions docs/perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,22 +212,25 @@ round implements the full OpenBLAS/BLIS structure and ships it for **all** sizes
The micro-kernel then streams conflict-free memory **regardless of the source
stride** — this is the precise fix for the power-of-two L1 set-conflicts that
defeated the unpacked attempt. Edge tiles are zero-padded so the kernel always
sees a full MR×NR block; the ragged right/bottom borders take a scalar path.
sees a full MR×NR block; a ragged right/bottom tile runs the same micro-kernel
into a scratch tile, and only its valid corner is added to C.
- **Cache blocking** — an `NC` (columns) → `KC` (contraction) → `MC` (rows) loop
nest keeps the packed B panel L2-resident and each packed A panel in L1.
Defaults `MC=256, KC=256, NC=512` (tuned on this VM).
- **SIMD-FMA micro-kernel** (go-asmgen) — **NEON 4×8 on arm64** (16 D2
accumulators, the source of the win), **SSE2 4×4 on amd64** (8 XMM
accumulators; SSE2 has no FMA, so explicit MULPD+ADDPD), **scalar 4×4** on the
accumulators, the source of the win), **AVX2/FMA 6×8 on amd64** (12 YMM
accumulators, picked at run time by a CPUID probe; an SSE2 6×4 pair without
FMA is the fallback), **scalar 4×4** on the
four arches without vector-double asm — which still gain from contiguous packed
data + blocking + multicore.
- **Parallelism** — the M rows are split into MR-aligned bands, one per core,
each writing a disjoint region of C.

The arithmetic is the standard ikj order `dst[i][j] += a[i][p]·b[p][j]`; packing
only relocates the operands, so the result is **bit-for-bit identical** to the
scalar oracle and to NumPy's `A@B` (max abs diff **0.0**, verified at 128×128
against OpenBLAS).
The arithmetic is the ikj order `dst[i][j] += a[i][p]·b[p][j]`, summed per KC
block; for k ≤ KC that is the scalar oracle's order exactly, and above it the
block partials regroup the sum (a valid reordering, as in every blocked BLAS).
At 128×128 the result matched NumPy's `A@B` against OpenBLAS with max abs diff
**0.0**.

### Packed GEMM vs the PRIOR kernel (why it ships)

Expand Down Expand Up @@ -261,8 +264,8 @@ there the gap is *fixed per-call overhead*, not compute:
This is a small-matrix overhead ceiling. Closing it further would mean a
serial-vs-parallel crossover tuned per size and a cheaper pack for tiny panels;
it is not a Go-assembler or algorithm limit, and it does not affect the
large-matrix parity result. (`GOAMD64=v3`/AVX2-FMA would likewise lift the amd64
micro-kernel, which today is SSE2-only; the arm64 path already has its FMA.)
large-matrix parity result. (amd64 now has its AVX2/FMA kernel too; see the
real-hardware section below.)

### Footnote: vs reference BLAS

Expand Down Expand Up @@ -310,15 +313,70 @@ The SIMD kernels are also validated against the scalar oracle per-arch in CI:
| `MatMul` vs tuned BLAS (OpenBLAS) | **FIXED — parity at n=1024 (~0.99×, ~203 GFLOP/s); 0.97× at n=512** | by-lane FMLA micro-kernel (`FMLA Vd.2D,Vn.2D,Vm.D[i]` via `WORD`) closed the prior 0.76× gap. Only small n=256 trails (0.67×) on per-call overhead, not throughput — see above |
| other `Map` ufuncs (`Exp`, `Log`, `Sin`…) | NumPy ~parity (libm-bound) | a packed `VEXP`/`VLOG` is libm-accuracy work; the math, not the dispatch, dominates here |

## amd64 on real hardware (2026-10-04)

Every figure above was taken on Apple silicon. This round measured amd64 for
the first time, on an AMD EPYC 7773X (Zen 3, GCC Compile Farm cfarm421), with
go-ndarray and NumPy both pinned to the same 16 cores of one NUMA node
(`taskset -c 16-31`; Go's GOMAXPROCS follows the affinity mask). NumPy 2.5.3
with its bundled OpenBLAS 0.3.34 (DYNAMIC_ARCH, Haswell kernels on Zen 3),
once with `OPENBLAS_NUM_THREADS=1` and once with 16. go rows are the minimum
of three runs interleaved with the previous version, so drift on a shared host
hits both alike.

| op | before | after | gain | vs OpenBLAS 1 thread | vs OpenBLAS 16 threads |
|----|------:|------:|:--:|:--:|:--:|
| MatMul 1024² | 21.87 ms | **8.05 ms** | ×2.72 | 5.93× | 0.60× |
| MatMul 512² | 3.69 ms | 1.56 ms | ×2.36 | 3.89× | 0.66× |
| MatMul 256² | 0.661 ms | 0.558 ms | ×1.19 | 1.36× | 0.30× |
| MatMul 128² | 0.210 ms | 0.141 ms | ×1.49 | 0.73× | 0.36× |
| MatMul 1024×256·256×1024 | 5.72 ms | 2.71 ms | ×2.11 | 4.55× | 0.54× |
| Inner 512² (a·bᵀ) | 5.87 ms | **1.47 ms** | ×4.00 | 4.19× | **0.96×** |

Four changes, each measured on its own before the next:

1. **AVX2/FMA 6×8 micro-kernel** (the BLIS Haswell shape). In L1 it runs at
43 GFLOP/s on one Cascade Lake core, the FMA ports' peak at that clock; the
SSE2 4×4 it replaces had no FMA and half the vector width.
2. **Edge tiles through the micro-kernel.** With MR=6, every 256-row MC block
ends in a 4-row edge panel, and the scalar edge loop took **40%** of a 512²
product (profiled). A partial tile now runs the kernel into a scratch tile.
3. **Prefetch** of the C tile at kernel entry and of the A panel eight steps
ahead: serial 768² went from 27.5 to 34.7 GFLOP/s (+26%); 90% of the time was
already in the kernel, stalled on memory. A sweep of MC (48–252) and KC
(128–512) moved nothing (±3%), so the blocking is unchanged.
4. **B packed by all workers.** MatMulP packs the shared B block once per
(jc, pc) round, and one goroutine did it while the others waited: +30–70%
in parallel. Below 64 Ki elements the launch costs more than the copy (M4:
−8–15% at 96³–128²), so small blocks are still packed by one.

`Inner` gained the most because it used to materialise bᵀ before multiplying.
The GEMM now reads any 2-D view through its strides while packing (the copy it
makes anyway), so a transposed, sliced, reversed or broadcast operand costs no
extra pass.

**Still behind OpenBLAS with 16 threads** (0.54–0.66× on large products, 0.3×
below 256²): per-core throughput in the parallel run is about a third of the
serial kernel's, so the next lever is the parallel schedule (a persistent
worker pool instead of a goroutine launch per round, and the BLIS jr-loop
split for short m), then an AVX-512 kernel for the hosts that have it.

Correctness was checked on real hardware for five of the six 64-bit targets:
amd64 (Zen 3, FMA path), arm64 (cfarm185), ppc64le (cfarm120), riscv64 (cfarm94)
and loong64 (cfarm401). The s390x host (LinuxONE) did not answer; s390x stays
covered by the qemu lane in CI.

## SIMD coverage

- **amd64 (SSE2)** ships hand-vectorized `sum` (4-accumulator `ADDPD`), `sqrt`
(packed `SQRTPD`), `max`/`min` (`MAXPD`/`MINPD` + `CMPPD` NaN scan), the
**elementwise `add`/`sub`/`mul`/`div`** (packed `ADD/SUB/MUL/DIVPD`, 8
doubles/iter + scalar tail), and the **GEMM micro-kernel** (`gemmMicro4x4`: a
4×4 SSE2 `MULPD`+`ADDPD` tile, no FMA at the v1 baseline), generated by
go-asmgen and validated per-arch in CI (and cross-run under qemu-x86_64 — the
GEMM tile included).
doubles/iter + scalar tail), all at the GOAMD64=v1 baseline, and the **GEMM
micro-kernel**: `gemmMicro6x8FMA` (6×8, 12 YMM accumulators, `VBROADCASTSD` +
`VFMADD231PD`, C and A prefetched) when go-asmgen's CPUID probe reports FMA
with the OS saving YMM state, else a pair of SSE2 6×4 `MULPD`+`ADDPD` tiles
over the same packing. Generated by go-asmgen and validated per-arch in CI;
the FMA kernel on real Cascade Lake and Zen 3 hosts.
- **arm64 (NEON)** ships hand-vectorized `sum`, **packed `sqrt`** (`FSQRT V.2D`),
the **elementwise `add`/`sub`/`mul`** (via `VFMLA`/`VFMLS` against a `1.0`
vector — `add: b+a·1`, `sub: a−b·1`, `mul: 0+a·b`, each FMA exact so
Expand All @@ -344,5 +402,5 @@ per-arch job regenerates the committed `.s`, fails if it is stale, vets
(asmdecl), builds (cmd/asm encodes), and runs the bit/NaN-correctness suite. The
multicore path and the packed/cache-blocked GEMM driver are
architecture-independent; only the GEMM micro-kernel is per-arch (NEON 4×8 on
arm64, SSE2 4×4 on amd64, scalar 4×4 on the other four — all bit-identical to the
arm64, AVX2/FMA 6×8 or SSE2 on amd64, scalar 4×4 on the other four — all bit-identical to the
scalar ikj oracle, validated per-arch in CI incl. amd64 under qemu-x86_64).
3 changes: 2 additions & 1 deletion docs/plan-ndarray.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ suite (plus the per-arch CI jobs already wired in `.github/workflows/ci.yml`).
`sum` (4-accumulator), `sqrt` (packed `SQRTPD` / NEON `FSQRT V.2D`),
`max`/`min` (`MAXPD`/`MINPD` + NaN scan / four-accumulator builtin), the
elementwise `add`/`sub`/`mul`/`div`, and the **GEMM micro-kernel** (NEON 4×8
by-lane FMLA on arm64, SSE2 4×4 on amd64), generated by go-asmgen and
by-lane FMLA on arm64, AVX2/FMA 6×8 with an SSE2 fallback on amd64),
generated by go-asmgen and
validated per-arch in CI (bit-identical to the scalar oracle for the
elementwise/sqrt/max ops; tight-tolerance for the sum reduction, whose
lane-parallel grouping is a valid reordering — the same trade-off as
Expand Down
217 changes: 141 additions & 76 deletions internal/kernels/asmgen/amd64/gen.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ func main() {
f.Add(binKernel("subSSE2", "SUBPD", "SUBSD"))
f.Add(binKernel("mulSSE2", "MULPD", "MULSD"))
f.Add(binKernel("divSSE2", "DIVPD", "DIVSD"))
f.Add(gemmKernel())
f.Add(gemmFMAKernel())
f.Add(gemmSSE2HalfKernel())
f.Add(amd64.FeatureProbe("hasFMA", amd64.FMA))

if err := os.WriteFile("sum_amd64.s", []byte(f.String()), 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
Expand Down Expand Up @@ -290,97 +292,160 @@ func binKernel(name, vec, sca string) *emit.Function {
return b.Func()
}

// gemmKernel builds gemmMicro4x4(kc int, pa, pb, c *float64, ldc int): the SSE2
// 4x4 register-blocked GEMM micro-kernel for the packed GEMM.
//
// It computes a 4-row x 4-col tile of C from packed panels and ADDS it into C:
//
// for p in [0,kc): C[r][col] += pa[p*4+r] * pb[p*4+col] (r<4, col<4)
//
// pa is the packed A panel (MR=4 contiguous A values per k step), pb the packed
// B panel (NR=4 contiguous B values per k step) — both unit-stride, so the loads
// are conflict-free regardless of the source matrices' strides (the point of
// packing: it removes the power-of-two L1 set-conflicts that defeat an unpacked
// register-blocked kernel).
//
// amd64 baseline is SSE2 (GOAMD64=v1) which has NO fused multiply-add (FMA is
// FMA3/AVX2), so each step is an explicit MULPD then ADDPD. SSE2 *does* have a
// packed FP add, so the running accumulation is the ordinary dst += a*b, in the
// same ikj order as the scalar oracle — the tile is bit-identical to it.
//
// Register map: the 4x4 C tile lives in X0..X7 (4 rows x 2 XMM, 2 doubles each);
// the per-step B row in X8,X9 (4 doubles); the broadcast A value in X10; the two
// products in X11,X12.
func gemmKernel() *emit.Function {
sig := amd64.Layout(
// gemmMR, gemmNR are the micro-tile the packing in gemm.go produces on amd64:
// 6 rows of packed A per k step, 8 columns of packed B.
const (
gemmMR = 6
gemmNR = 8
)

func gemmSig() amd64.Signature {
return amd64.Layout(
[]string{"kc", "pa", "pb", "c", "ldc"},
[]amd64.Type{amd64.Int64, amd64.Ptr, amd64.Ptr, amd64.Ptr, amd64.Int64},
nil, nil,
)
b := amd64.NewFunc("gemmMicro4x4", sig, 0)
}

// gemmFMAKernel builds gemmMicro6x8FMA(kc int, pa, pb, c *float64, ldc int):
//
// for p in [0,kc): C[r][col] += pa[p*6+r] * pb[p*8+col] (r<6, col<8)
//
// the BLIS Haswell dgemm shape (6x8, row-major here). Each k step loads the
// 8-double B row into two YMM, broadcasts each of the 6 A values with
// VBROADCASTSD straight from the panel, and issues 12 VFMADD231PD into the 12
// accumulators Y0..Y11 (row r in Y(2r), Y(2r+1)). Twelve independent chains
// cover the FMA latency on both ports (Haswell: 5 cycles x 2 ports needs >= 10),
// and 8 loads per 12 FMAs stay under the 2-loads-per-cycle budget, so the loop
// is FMA-throughput bound — the property that makes it a real dgemm kernel.
//
// Fused: each step rounds a*b+c once, as tuned BLAS (and gc's own scalar code at
// GOAMD64=v3, and arm64's) do. The SSE2 fallback below does not fuse, so on
// non-integer data the two can differ in the last place; both are the ikj sum.
// The caller gates this kernel on hasFMA (CPUID FMA+AVX+OSXSAVE and XCR0 YMM).
func gemmFMAKernel() *emit.Function {
b := amd64.NewFunc("gemmMicro6x8FMA", gemmSig(), 0)
b.LoadArg("kc", "CX")
b.LoadArg("pa", "SI") // A panel
b.LoadArg("pb", "DI") // B panel
b.LoadArg("c", "DX") // C base
b.LoadArg("ldc", "R8")
b.Raw("SHLQ $3, R8") // ldc -> bytes
// Zero the eight C accumulators X0..X7.
b.Raw("XORPS X0, X0")
b.Raw("XORPS X1, X1")
b.Raw("XORPS X2, X2")
b.Raw("XORPS X3, X3")
b.Raw("XORPS X4, X4")
b.Raw("XORPS X5, X5")
b.Raw("XORPS X6, X6")
b.Raw("XORPS X7, X7")
b.Raw("TESTQ CX, CX")
b.Raw("JZ gstore")
b.Raw("gkloop:")
// B row: 4 contiguous doubles into X8 (cols 0,1) and X9 (cols 2,3).
b.Raw("MOVUPD (DI), X8")
b.Raw("MOVUPD 16(DI), X9")
b.Raw("ADDQ $32, DI")
// For each of the 4 A values, broadcast and fuse into that row's two XMM.
gemmRow(b, "X0", "X1", 0)
gemmRow(b, "X2", "X3", 8)
gemmRow(b, "X4", "X5", 16)
gemmRow(b, "X6", "X7", 24)
b.Raw("ADDQ $32, SI") // advance A panel by 4 doubles
b.Raw("DECQ CX")
b.Raw("JNZ gkloop")
// C += accumulators, row by row (ordinary packed ADDPD, bit-identical).
b.Raw("gstore:")
gemmStore(b, "X0", "X1", true) // row0 at (DX)
gemmStore(b, "X2", "X3", false)
gemmStore(b, "X4", "X5", false)
gemmStore(b, "X6", "X7", false)
// Prefetch the C tile now: its rows are ldc apart, so they are 6 different
// cache lines (12 when a row straddles two), usually not in L1, and the
// store at the end would otherwise stall on each of them in turn. Issued
// first, they arrive while the k loop runs. The last byte of each row is
// fetched too, for a row that does not start on a line boundary.
b.Raw("MOVQ DX, R9")
for r := 0; r < gemmMR; r++ {
b.Raw("PREFETCHT0 (R9)")
b.Raw("PREFETCHT0 63(R9)")
if r < gemmMR-1 {
b.Raw("ADDQ R8, R9")
}
}
for r := 0; r < 12; r++ {
b.Raw(fmt.Sprintf("VXORPD Y%d, Y%d, Y%d", r, r, r))
}
// Unrolled by two k steps, with a one-step tail. The A panel streams from
// L2 (the B micro-panel stays in L1 across the whole A block), so each pair
// of steps prefetches the A line 8 steps ahead.
b.Raw("MOVQ CX, R10")
b.Raw("SHRQ $1, R10")
b.Raw("JZ ftail")
b.Raw("fkloop:")
b.Raw(fmt.Sprintf("PREFETCHT0 %d(SI)", 8*gemmMR*8))
fmaStep(b, 0)
fmaStep(b, 1)
b.Raw(fmt.Sprintf("ADDQ $%d, SI", 2*8*gemmMR))
b.Raw(fmt.Sprintf("ADDQ $%d, DI", 2*8*gemmNR))
b.Raw("DECQ R10")
b.Raw("JNZ fkloop")
b.Raw("ftail:")
b.Raw("TESTQ $1, CX")
b.Raw("JZ fstore")
fmaStep(b, 0)
b.Raw("fstore:")
for r := 0; r < gemmMR; r++ {
b.Raw(fmt.Sprintf("VADDPD (DX), Y%d, Y15", 2*r))
b.Raw("VMOVUPD Y15, (DX)")
b.Raw(fmt.Sprintf("VADDPD 32(DX), Y%d, Y15", 2*r+1))
b.Raw("VMOVUPD Y15, 32(DX)")
if r < gemmMR-1 {
b.Raw("ADDQ R8, DX")
}
}
b.Raw("VZEROUPPER") // no AVX-SSE transition penalty in the Go code after
b.Ret()
return b.Func()
}

// gemmRow fuses one A value (at offset off in the A panel SI) into one C row's
// two XMM accumulators (lo cols 0,1 = accLo; hi cols 2,3 = accHi): broadcast
// a -> X10, X11 = Brow_lo*a, X12 = Brow_hi*a, then add into the accumulators.
func gemmRow(b *amd64.Builder, accLo, accHi string, off int) {
b.Raw(fmt.Sprintf("MOVDDUP %d(SI), X10", off)) // X10 = [a,a]
b.Raw("MOVAPS X8, X11")
b.Raw("MULPD X10, X11") // X11 = Brow_lo * a
b.Raw("ADDPD X11, " + accLo)
b.Raw("MOVAPS X9, X12")
b.Raw("MULPD X10, X12") // X12 = Brow_hi * a
b.Raw("ADDPD X12, " + accHi)
// fmaStep emits one k step of the 6x8 FMA tile, step reading the A and B
// values at that many steps past SI and DI.
func fmaStep(b *amd64.Builder, step int) {
ao, bo := 8*gemmMR*step, 8*gemmNR*step
b.Raw(fmt.Sprintf("VMOVUPD %d(DI), Y12", bo)) // B cols 0..3
b.Raw(fmt.Sprintf("VMOVUPD %d(DI), Y13", bo+32)) // B cols 4..7
for r := 0; r < gemmMR; r++ {
b.Raw(fmt.Sprintf("VBROADCASTSD %d(SI), Y14", ao+8*r))
b.Raw(fmt.Sprintf("VFMADD231PD Y12, Y14, Y%d", 2*r))
b.Raw(fmt.Sprintf("VFMADD231PD Y13, Y14, Y%d", 2*r+1))
}
}

// gemmStore adds one C row's two accumulators into memory at the current row
// pointer (DX), then advances DX by ldc bytes (R8) unless this is the last row.
func gemmStore(b *amd64.Builder, accLo, accHi string, first bool) {
b.Raw("MOVUPD (DX), X11")
b.Raw("ADDPD " + accLo + ", X11")
b.Raw("MOVUPD X11, (DX)")
b.Raw("MOVUPD 16(DX), X12")
b.Raw("ADDPD " + accHi + ", X12")
b.Raw("MOVUPD X12, 16(DX)")
b.Raw("ADDQ R8, DX")
// gemmSSE2HalfKernel builds gemmMicro6x4SSE2(kc int, pa, pb, c *float64, ldc
// int), the baseline (GOAMD64=v1, no FMA) fallback over the same 6x8 packing:
// it computes the 6-row x 4-col half of the tile whose B columns start at pb,
// so the Go side calls it twice (pb, c) and (pb+4, c+4). A whole 6x8 tile in
// XMM would need 24 accumulators; the half needs 12 (X0..X11, row r in X(2r),
// X(2r+1)) plus the B half-row X12,X13, the broadcast A X14 and a product X15.
// The B stride is still the full NR=8 doubles per k step.
//
// SSE2 has no fused multiply-add, so each step is MULPD then ADDPD: the
// running sum is the plain dst += a*b in ikj order, like the scalar oracle at
// GOAMD64=v1.
func gemmSSE2HalfKernel() *emit.Function {
b := amd64.NewFunc("gemmMicro6x4SSE2", gemmSig(), 0)
b.LoadArg("kc", "CX")
b.LoadArg("pa", "SI")
b.LoadArg("pb", "DI")
b.LoadArg("c", "DX")
b.LoadArg("ldc", "R8")
b.Raw("SHLQ $3, R8")
for r := 0; r < 12; r++ {
b.Raw(fmt.Sprintf("XORPS X%d, X%d", r, r))
}
b.Raw("TESTQ CX, CX")
b.Raw("JZ hstore")
b.Raw("hkloop:")
b.Raw("MOVUPD (DI), X12")
b.Raw("MOVUPD 16(DI), X13")
for r := 0; r < gemmMR; r++ {
b.Raw(fmt.Sprintf("MOVDDUP %d(SI), X14", 8*r)) // X14 = [a,a]
b.Raw("MOVAPS X12, X15")
b.Raw("MULPD X14, X15")
b.Raw(fmt.Sprintf("ADDPD X15, X%d", 2*r))
b.Raw("MULPD X13, X14") // the broadcast is spent: reuse it for the product
b.Raw(fmt.Sprintf("ADDPD X14, X%d", 2*r+1))
}
b.Raw(fmt.Sprintf("ADDQ $%d, SI", 8*gemmMR))
b.Raw(fmt.Sprintf("ADDQ $%d, DI", 8*gemmNR))
b.Raw("DECQ CX")
b.Raw("JNZ hkloop")
b.Raw("hstore:")
for r := 0; r < gemmMR; r++ {
b.Raw("MOVUPD (DX), X14")
b.Raw(fmt.Sprintf("ADDPD X%d, X14", 2*r))
b.Raw("MOVUPD X14, (DX)")
b.Raw("MOVUPD 16(DX), X15")
b.Raw(fmt.Sprintf("ADDPD X%d, X15", 2*r+1))
b.Raw("MOVUPD X15, 16(DX)")
if r < gemmMR-1 {
b.Raw("ADDQ R8, DX")
}
}
b.Ret()
return b.Func()
}

// nanScan emits: X9 = (reg unordered reg) ? all-ones : 0 ; X8 |= X9, i.e. it
Expand Down
2 changes: 1 addition & 1 deletion internal/kernels/asmgen/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ module github.com/go-ndarray/ndarray/internal/kernels/asmgen

go 1.26.4

require github.com/go-asmgen/asmgen v0.10.0
require github.com/go-asmgen/asmgen v0.11.0
Loading
Loading