[AMD NPU] Add a RyzenAI backend that runs the network on the AMD NPU - #1240
Open
Looong01 wants to merge 49 commits into
Open
[AMD NPU] Add a RyzenAI backend that runs the network on the AMD NPU#1240Looong01 wants to merge 49 commits into
Looong01 wants to merge 49 commits into
Conversation
Unify Metal and CoreML backends with optimizations and fixes
Update README, cfg, setup_env.ps1 and gitignore
# Conflicts: # CONTRIBUTORS # README.md # cpp/CMakeLists.txt # cpp/book/book.cpp # cpp/book/book.h # cpp/command/analysis.cpp # cpp/command/benchmark.cpp # cpp/command/evalsgf.cpp # cpp/command/genbook.cpp # cpp/command/gtp.cpp # cpp/command/misc.cpp # cpp/command/runtests.cpp # cpp/command/startposes.cpp # cpp/command/writetrainingdata.cpp # cpp/dataio/sgf.cpp # cpp/dataio/sgf.h # cpp/dataio/trainingwrite.cpp # cpp/game/boardhistory.cpp # cpp/game/boardhistory.h # cpp/game/rules.cpp # cpp/game/rules.h # cpp/main.cpp # cpp/neuralnet/cudabackend.cpp # cpp/neuralnet/cudahelpers.cu # cpp/neuralnet/cudahelpers.h # cpp/neuralnet/desc.cpp # cpp/neuralnet/desc.h # cpp/neuralnet/nneval.cpp # cpp/neuralnet/nneval.h # cpp/neuralnet/nninputs.cpp # cpp/neuralnet/nninputs.h # cpp/neuralnet/onnxmodelbuilder.cpp # cpp/neuralnet/trtbackend.cpp # cpp/program/play.cpp # cpp/program/play.h # cpp/program/playutils.cpp # cpp/program/setup.cpp # cpp/search/patternbonustable.cpp # cpp/search/search.cpp # cpp/search/search.h # cpp/search/searchnnhelpers.cpp # cpp/search/searchparams.cpp # cpp/search/searchparams.h # cpp/tests/results/gtp/humansl.log # cpp/tests/results/gtp/humansl.stdout # cpp/tests/results/gtp/misc.txt.log # cpp/tests/results/gtp/misc.txt.stdout # cpp/tests/results/gtp/setparams.txt.log # cpp/tests/results/gtp/setparams.txt.stdout # cpp/tests/results/runOutputTests.txt # cpp/tests/testboardbasic.cpp # cpp/tests/testbook.cpp # cpp/tests/testmisc.cpp # cpp/tests/testnnevalcanary.cpp # cpp/tests/testnninputs.cpp # cpp/tests/testownership.cpp # cpp/tests/testpassalivesuicide.cpp # cpp/tests/testrules.cpp # cpp/tests/tests.h # cpp/tests/testscore.cpp # cpp/tests/testsearchcommon.cpp # cpp/tests/testsearchmisc.cpp # cpp/tests/testsearchnonn.cpp # cpp/tests/testsearchv3.cpp # cpp/tests/testsearchv8.cpp # cpp/tests/testsearchv9.cpp # cpp/tests/testsgf.cpp # cpp/tests/testsymmetries.cpp # cpp/tests/testtime.cpp # cpp/tests/testtrainingwrite.cpp # cpp/tests/tinymodel.cpp # docs/releasepackaging/README.txt # python/export_model_pytorch.py # python/katago/train/model_pytorch.py
The auto-fetch path from AMD_GPU covers vcpkg only when KATAGO_AUTO_FETCH_DEPS is on; this keeps a plain 'cmake .. -G Ninja -DUSE_BACKEND=RYZENAI' working on a machine whose only zlib came with conda.
This branch is the base for a pull request against lightvector/master, and AMD_GPU carries two commits that master does not have yet: 62cb293 Fix CDNA slowdown edd1b06 Fix slowdown and Compiling.md Both are ROCm work and unrelated to the NPU backend, so carrying them here would put unrelated changes in the pull request and conflict with whatever shape they land in upstream. Reverting them leaves cudaandrocmbackend.inc and analysis_example.cfg identical to master; Compiling.md and gtp_example.cfg keep only this branch's own RyzenAI sections.
Contributor
Author
|
Binary release here: https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.17.2_amd_npu |
added 3 commits
August 19, 2026 06:04
… backend building on Windows
Pulls in AMD_GPU's ROCm MSVC-toolset-selection rewrite (probe every installed toolset newest-first via a real compile, instead of hardcoding the v143/14.3x-14.4x version family), plus the file(STRINGS)-list-corruption fix and per-attempt env restore it depended on. Compiling.md had a textual conflict in the same ROCm/Windows section (AMD_NPU's TheRock-specific wording vs. AMD_GPU's toolset-selection wording); resolved by keeping this branch's TheRock framing while taking AMD_GPU's updated toolset-selection description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
-DUSE_BACKEND=RYZENAI, a backend that runs KataGo's neural net onthe NPU built into AMD Ryzen AI processors (XDNA1 as in Phoenix/Hawk Point,
XDNA2 as in Strix). It is a native XRT backend - not ONNX Runtime, not the
Ryzen AI Software SDK - and it reads ordinary
.bin.gzmodels with noconversion, quantization or export step.
What a user needs
The NPU driver. That is
all. There is no tuning phase, no per-model setup, and no Python at runtime.
The compiled NPU kernels are committed here and CMake copies them next to
katago.exe, so the build directory runs as-is.Why it can ship precompiled kernels at all
This was the part that decided the architecture. An NPU kernel binary
(
.xclbin) bakes in exactly one property of a matrix multiply: the reductiondimension
K, because the AIE cores' accumulation loop count is a compile-timeconstant.
MandNare not in the binary at all - they live entirely in theDMA instruction stream, which
ryzenaisequence.cppbuilds at run time.That is measured, not assumed: driving a
K=512binary with instructionstreams for
Mfrom 256 to 5888 andNfrom 64 to 2048 gives bit-comparableresults throughout, while a
Kmismatch is silently wrong (the hardwarereports success and returns garbage), so the loader keys strictly on
K.The consequence is that a one-dimensional grid of 34
Kvalues, zero-padded upto the next size when a model asks for something in between, covers any network
backend keeps the property that matters for a Go engine: an arbitrary
.bin.gzjust runs.What is committed
cpp/neuralnet/ryzenai*.{h,cpp}- about 6k lines: XRT device handling,run-time instruction-stream generation, shape analysis, the dispatch/
fallback router, and a plain C++ reference implementation of the whole
forward pass that doubles as the numerical ground truth and the fallback
for anything the NPU cannot take.
cpp/external/ryzenai_artifacts/- 426 kernel binaries, 30 MB. The GEMMgrid plus fused operators for attention, softmax, SwiGLU and BatchNorm+Mish.
python/ryzenai_kernels/- the offline generator, needed only to addkernels for a geometry that has none yet. Not part of the build.
Measured
Ryzen AI 9 HX 370 (Strix, XDNA2), 19x19,
katago benchmarkwith 16 searchthreads, i.e. batched the way real play batches:
Every matrix multiply runs on the NPU - dense layers, 1x1 and 3x3 convolutions
(the latter as implicit GEMM), attention projections, QK^T, softmax, P*V, and
the FFN including its SwiGLU activation. What remains on the CPU is RMSNorm,
RoPE and the residual adds, which are bandwidth-bound and measured to be
slower on the NPU than off it: a dispatch costs about 0.73 ms fixed, against
roughly 16 ms of CPU time per evaluation for all the norms in a 10-block net.
Threading matters more than any config option here. The NPU pays that fixed
cost per dispatch and KataGo amortizes it by batching evaluations across search
threads: going from one search thread to sixteen measured 1.9x more evaluations
per second on the same machine.
Precision
The NPU computes in bfloat16, or block floating point on XDNA2 where that is
the default. A single BFP16 matmul carries around 3% relative error against
float32, but that is input quantization and the accumulator is float32 either
way, so it does not compound the way the per-matmul figure suggests: over a
whole evaluation it lands within about 1.3-2x of bf16's deviation from the CPU
reference, and the policy's top ten moves are identical for both formats on
every model tested.
The practical effect is that the engine can choose differently between two
moves it considers nearly equal - in one test position the CPU reference's own
top two policy values differed by 1.8e-4, below the noise floor of either
format.
ryzenaiDtype = bf16selects the more accurate format at roughly 10%throughput, and is the only option on XDNA1.
Testing
play full games;
kata-raw-nnoutput compared against the CPU referencepath for each.
produced by the vendor toolchain, byte for byte, and then against hardware
on shapes with no golden.
each falls back silently and correctly when its geometry does not match.
analysis,benchmark,matchandgtpconfigs all run.Notes for review
Linux NPU stack is not something I have hardware to test against.
are untested on hardware; the manifest records that.
RYZENAIbranchesplus a Windows MSVC-environment bootstrap so that
cmake -G Ninjaworksfrom an ordinary shell rather than only from a Developer Command Prompt.