Skip to content

[AMD NPU] Add a RyzenAI backend that runs the network on the AMD NPU - #1240

Open
Looong01 wants to merge 49 commits into
lightvector:masterfrom
Looong01:AMD_NPU
Open

[AMD NPU] Add a RyzenAI backend that runs the network on the AMD NPU#1240
Looong01 wants to merge 49 commits into
lightvector:masterfrom
Looong01:AMD_NPU

Conversation

@Looong01

@Looong01 Looong01 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

This adds -DUSE_BACKEND=RYZENAI, a backend that runs KataGo's neural net on
the NPU built into AMD Ryzen AI processors (XDNA1 as in Phoenix/Hawk Point,
XDNA2 as in Strix). It is a native XRT backend - not ONNX Runtime, not the
Ryzen AI Software SDK - and it reads ordinary .bin.gz models with no
conversion, quantization or export step.

What a user needs

The NPU driver. That is
all. There is no tuning phase, no per-model setup, and no Python at runtime.
The compiled NPU kernels are committed here and CMake copies them next to
katago.exe, so the build directory runs as-is.

Why it can ship precompiled kernels at all

This was the part that decided the architecture. An NPU kernel binary
(.xclbin) bakes in exactly one property of a matrix multiply: the reduction
dimension K, because the AIE cores' accumulation loop count is a compile-time
constant. M and N are not in the binary at all - they live entirely in the
DMA instruction stream, which ryzenaisequence.cpp builds at run time.

That is measured, not assumed: driving a K=512 binary with instruction
streams for M from 256 to 5888 and N from 64 to 2048 gives bit-comparable
results throughout, while a K mismatch is silently wrong (the hardware
reports success and returns garbage), so the loader keys strictly on K.

The consequence is that a one-dimensional grid of 34 K values, zero-padded up
to the next size when a model asks for something in between, covers any network

  • including ones that did not exist when the kernels were built. So the
    backend keeps the property that matters for a Go engine: an arbitrary
    .bin.gz just runs.

What is committed

  • cpp/neuralnet/ryzenai*.{h,cpp} - about 6k lines: XRT device handling,
    run-time instruction-stream generation, shape analysis, the dispatch/
    fallback router, and a plain C++ reference implementation of the whole
    forward pass that doubles as the numerical ground truth and the fallback
    for anything the NPU cannot take.
  • cpp/external/ryzenai_artifacts/ - 426 kernel binaries, 30 MB. The GEMM
    grid plus fused operators for attention, softmax, SwiGLU and BatchNorm+Mish.
  • python/ryzenai_kernels/ - the offline generator, needed only to add
    kernels for a geometry that has none yet. Not part of the build.

Measured

Ryzen AI 9 HX 370 (Strix, XDNA2), 19x19, katago benchmark with 16 search
threads, i.e. batched the way real play batches:

model evals/s
b10c384h6nbt 9.2
b10c512h8nbt 4.9
b11c768h12nbt 2.7
b28c512nbt 3.7
b40c768nbt 1.7

Every matrix multiply runs on the NPU - dense layers, 1x1 and 3x3 convolutions
(the latter as implicit GEMM), attention projections, QK^T, softmax, P*V, and
the FFN including its SwiGLU activation. What remains on the CPU is RMSNorm,
RoPE and the residual adds, which are bandwidth-bound and measured to be
slower on the NPU than off it: a dispatch costs about 0.73 ms fixed, against
roughly 16 ms of CPU time per evaluation for all the norms in a 10-block net.

Threading matters more than any config option here. The NPU pays that fixed
cost per dispatch and KataGo amortizes it by batching evaluations across search
threads: going from one search thread to sixteen measured 1.9x more evaluations
per second on the same machine.

Precision

The NPU computes in bfloat16, or block floating point on XDNA2 where that is
the default. A single BFP16 matmul carries around 3% relative error against
float32, but that is input quantization and the accumulator is float32 either
way, so it does not compound the way the per-matmul figure suggests: over a
whole evaluation it lands within about 1.3-2x of bf16's deviation from the CPU
reference, and the policy's top ten moves are identical for both formats on
every model tested.

The practical effect is that the engine can choose differently between two
moves it considers nearly equal - in one test position the CPU reference's own
top two policy values differed by 1.8e-4, below the noise floor of either
format. ryzenaiDtype = bf16 selects the more accurate format at roughly 10%
throughput, and is the only option on XDNA1.

Testing

  • Five networks (b10c384h6, b10c512h8, b11c768h12, b28c512nbt, b40c768nbt)
    play full games; kata-raw-nn output compared against the CPU reference
    path for each.
  • The instruction-stream generator is verified against 79 golden streams
    produced by the vendor toolchain, byte for byte, and then against hardware
    on shapes with no golden.
  • Fused operators are checked individually against the reference path, and
    each falls back silently and correctly when its geometry does not match.
  • analysis, benchmark, match and gtp configs all run.

Notes for review

  • Windows only for now. Nothing in the design is Windows-specific, but XRT's
    Linux NPU stack is not something I have hardware to test against.
  • XDNA1 artifacts are built and shipped but I have no XDNA1 device, so they
    are untested on hardware; the manifest records that.
  • The CMake changes for this backend are confined to the RYZENAI branches
    plus a Windows MSVC-environment bootstrap so that cmake -G Ninja works
    from an ordinary shell rather than only from a Developer Command Prompt.

Looong01 and others added 16 commits July 11, 2026 18:03
Unify Metal and CoreML backends with optimizations and fixes
Update README, cfg, setup_env.ps1 and gitignore
# Conflicts:
#	CONTRIBUTORS
#	README.md
#	cpp/CMakeLists.txt
#	cpp/book/book.cpp
#	cpp/book/book.h
#	cpp/command/analysis.cpp
#	cpp/command/benchmark.cpp
#	cpp/command/evalsgf.cpp
#	cpp/command/genbook.cpp
#	cpp/command/gtp.cpp
#	cpp/command/misc.cpp
#	cpp/command/runtests.cpp
#	cpp/command/startposes.cpp
#	cpp/command/writetrainingdata.cpp
#	cpp/dataio/sgf.cpp
#	cpp/dataio/sgf.h
#	cpp/dataio/trainingwrite.cpp
#	cpp/game/boardhistory.cpp
#	cpp/game/boardhistory.h
#	cpp/game/rules.cpp
#	cpp/game/rules.h
#	cpp/main.cpp
#	cpp/neuralnet/cudabackend.cpp
#	cpp/neuralnet/cudahelpers.cu
#	cpp/neuralnet/cudahelpers.h
#	cpp/neuralnet/desc.cpp
#	cpp/neuralnet/desc.h
#	cpp/neuralnet/nneval.cpp
#	cpp/neuralnet/nneval.h
#	cpp/neuralnet/nninputs.cpp
#	cpp/neuralnet/nninputs.h
#	cpp/neuralnet/onnxmodelbuilder.cpp
#	cpp/neuralnet/trtbackend.cpp
#	cpp/program/play.cpp
#	cpp/program/play.h
#	cpp/program/playutils.cpp
#	cpp/program/setup.cpp
#	cpp/search/patternbonustable.cpp
#	cpp/search/search.cpp
#	cpp/search/search.h
#	cpp/search/searchnnhelpers.cpp
#	cpp/search/searchparams.cpp
#	cpp/search/searchparams.h
#	cpp/tests/results/gtp/humansl.log
#	cpp/tests/results/gtp/humansl.stdout
#	cpp/tests/results/gtp/misc.txt.log
#	cpp/tests/results/gtp/misc.txt.stdout
#	cpp/tests/results/gtp/setparams.txt.log
#	cpp/tests/results/gtp/setparams.txt.stdout
#	cpp/tests/results/runOutputTests.txt
#	cpp/tests/testboardbasic.cpp
#	cpp/tests/testbook.cpp
#	cpp/tests/testmisc.cpp
#	cpp/tests/testnnevalcanary.cpp
#	cpp/tests/testnninputs.cpp
#	cpp/tests/testownership.cpp
#	cpp/tests/testpassalivesuicide.cpp
#	cpp/tests/testrules.cpp
#	cpp/tests/tests.h
#	cpp/tests/testscore.cpp
#	cpp/tests/testsearchcommon.cpp
#	cpp/tests/testsearchmisc.cpp
#	cpp/tests/testsearchnonn.cpp
#	cpp/tests/testsearchv3.cpp
#	cpp/tests/testsearchv8.cpp
#	cpp/tests/testsearchv9.cpp
#	cpp/tests/testsgf.cpp
#	cpp/tests/testsymmetries.cpp
#	cpp/tests/testtime.cpp
#	cpp/tests/testtrainingwrite.cpp
#	cpp/tests/tinymodel.cpp
#	docs/releasepackaging/README.txt
#	python/export_model_pytorch.py
#	python/katago/train/model_pytorch.py
The auto-fetch path from AMD_GPU covers vcpkg only when KATAGO_AUTO_FETCH_DEPS
is on; this keeps a plain 'cmake .. -G Ninja -DUSE_BACKEND=RYZENAI' working on a
machine whose only zlib came with conda.
This branch is the base for a pull request against lightvector/master, and
AMD_GPU carries two commits that master does not have yet:

  62cb293  Fix CDNA slowdown
  edd1b06  Fix slowdown and Compiling.md

Both are ROCm work and unrelated to the NPU backend, so carrying them here
would put unrelated changes in the pull request and conflict with whatever
shape they land in upstream. Reverting them leaves cudaandrocmbackend.inc and
analysis_example.cfg identical to master; Compiling.md and gtp_example.cfg keep
only this branch's own RyzenAI sections.
@Looong01 Looong01 changed the title Add a RyzenAI backend that runs the network on the AMD NPU [AMD NPU] Add a RyzenAI backend that runs the network on the AMD NPU Aug 18, 2026
@Looong01

Copy link
Copy Markdown
Contributor Author

Looong01 added 3 commits August 19, 2026 06:04
Pulls in AMD_GPU's ROCm MSVC-toolset-selection rewrite (probe every
installed toolset newest-first via a real compile, instead of hardcoding
the v143/14.3x-14.4x version family), plus the file(STRINGS)-list-corruption
fix and per-attempt env restore it depended on.

Compiling.md had a textual conflict in the same ROCm/Windows section
(AMD_NPU's TheRock-specific wording vs. AMD_GPU's toolset-selection
wording); resolved by keeping this branch's TheRock framing while taking
AMD_GPU's updated toolset-selection description.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant