Add MLX backend for Apple Silicon (GPU + ANE/CoreML dispatch, incl. transformer nets)

[ChinChangYang]

This PR adds a new neural-net backend (USE_BACKEND=MLX) targeting Apple
Silicon via Apple's MLX framework,
with two dispatch paths sharing a single backend:

• GPU — MLX/Metal with an F(2×2, 3×3) Winograd path and an adaptive
  per-shape tuner.
• ANE — CoreML on CPU + Apple Neural Engine, mirroring the Metal
  backend's gpuIdx = 100 convention. Usable standalone or muxed with
  the GPU path on the same model to overlap forward passes.

It implements the full nninterface.h contract (model load, batched
evaluation, FP16/FP32 paths) and reuses the existing CoreML conversion
pipeline shared with the Metal backend. Both dispatch paths support
the v15+ transformer trunk (attention incl. GQA, learnable RoPE, SiLU,
RMSNorm/batchnorm tips, SwiGLU FFN) in addition to convnets.

What's new

Backend (cpp/neuralnet/)

• mlxbackend.cpp — backend implementation.
  • GPU path: Variable board sizes via input masking (same
    nnXLen / nnYLen contract as other backends; the global
    COMPILE_MAX_BOARD_LEN bound still applies), FP16/FP32 selected
    by mlxUseFP16 (default auto → fp16). Mish runs FP16-safe; the
    code asserts on ACTIVATION_MISH_SCALE8 so out-of-range variants
    fail loudly rather than truncate silently.
  • ANE path: Selected per server thread via
    mlxDeviceToUseThread<N> = 100 (or the backend-agnostic
    deviceToUseThread<N>); shares the model+converter cache with
    Metal. Feeds the real spatial mask (channel 0 NHWC → buffer) so
    rectangular / sub-NN-frame boards predict correctly, transposes
    NHWC → NCHW into a per-batch staging buffer for the Swift
    MLMultiArray contract, and uses path-correct strides in the
    policy-optimism postprocessor for v12+ models.
  • Mux (GPU + ANE): Serializes ComputeHandle construction with
    a file-static mutex (CoreML converter is not concurrent), and
    eagerly evaluates FP16 weight casts so secondary MLX/GPU server
    threads don't trip MLX 0.31.2's thread-local command-encoder map.
• mlxwinograd.h — F(2×2, 3×3) Winograd transform with fused
  activation + residual add.
• mlxwinotuner.{cpp,h} — per-shape Winograd tuner with adaptive
  scoring (rotates the candidate set per shape, scores by median-time
  delta against a baked-default baseline). Logs the conv-3x3 shape
  distribution at model load. At model load it loads a valid cache or,
  on a miss, coarse-tunes once — it never re-tunes a valid cache and
  never sweeps the wide grid; explicit re-tuning is command-only via
  ./katago tuner (see Build / wiring). KATAGO_MLX_WINOTUNER=0
  skips tuning entirely and uses baked-default launch geometry, while
  KATAGO_MLX_WINOGRAD=0 falls 3×3 convs back to mx::conv2d.
• mlxtests.cpp — Winograd + tuner numeric-consistency tests, gated
  under runnnlayertests.

Transformer trunk support (both dispatch paths)

• GPU path (native MLX): transformer trunk/tip implemented directly
  in mlxbackend.cpp — scaled-dot-product attention with grouped-query
  attention (GQA), learnable RoPE, RMSNorm and batchnorm trunk tips,
  SwiGLU FFN, and the SiLU activation. Mirrors eigenbackend.cpp.
• ANE path (CoreML/katagocoreml): the same transformer trunk in
  the shared MIL converter, with FP16 accuracy precision tiers gated
  on actual transformer-block presence (recursing into nested-bottleneck
  blocks) so plain convnets stay pure FP16 on the FP16-only ANE. For
  transformer trunks: narrow (<256ch) build fully FP32; wider ones
  escalate the accuracy-sensitive non-spatial ops (matmuls + global
  pooling) to FP32; very wide (≥320ch) also escalate convs; RMSNorm
  reductions run FP32 in FP16 mode while attention scores/softmax and the
  per-head out-projection stay FP16. Mirrors the TensorRT precision split
  and the Metal/CoreML backend's tiers. The converter's public API is
  unchanged, so the MLX ANE call site needs no edits.

Build / wiring

• cpp/CMakeLists.txt — USE_BACKEND=MLX target; pulls in the
  Metal/Swift CoreML bridge so the ANE path links cleanly. MLX needs
  CMake 3.27; cmake_minimum_required stays at 3.18.2 so other
  backends keep building on older CMake. Links Homebrew's prebuilt
  libmlx.dylib; OSX deployment target is intentionally not pinned
  so the executable's minos matches the linked dylib.
• cpp/main.cpp, cpp/program/setup.cpp, cpp/command/benchmark.cpp
  — wire MLX into backend selection / benchmark.
• cpp/command/tune.cpp — wires the MLX winotuner into the ./katago tuner subcommand (previously OpenCL-only), mirroring OpenCL's
  command-only tuning. ./katago tuner -model <net> always re-runs the
  sweep and overwrites the cache the backend reads at model load; add
  -full for the wide candidate grid. Args: -output (default: shared
  MLX cache), -xsize / -ysize (default 19), -batchsize (default
  8), -testFP16 (true|false|auto, default auto → FP16, the
  precision the engine and the cache filename key use), -full. The
  OpenCL-only FP16 sub-knobs (storage / compute / tensorcores) have no
  MLX analog and are omitted.
• cpp/configs/{gtp,analysis,match,contribute}_example.cfg —
  document mlxUseFP16 (default auto → fp16) and the
  numNNServerThreadsPerModel / mlxDeviceToUseThread<N> dispatch
  knobs (GPU-only / ANE-only / mux), with the note that
  mlxUseFP16=false on an ANE thread falls back to CPU FP32.
• cpp/rungpuerrortest.sh — backend-agnostic
  deviceToUseThread0=100 for the ANE mode, so the same script
  drives whichever backend the binary was built with.
• Compiling.md — build instructions.

Conversion + ANE steady-state memory

The ANE path carries the on-device memory levers from #1202 (cut CoreML
conversion peak and ANE steady-state RSS, with byte-identical converter
output), cherry-picked onto this branch and wired into the MLX backend:

• Non-owning weight views — the converter's weight tensors are
  non-owning FloatViews into the parsed model instead of owning extra
  FP32 copies (derived/transposed tensors keep an owned buffer). The
  rvalue registerWeight / addConstOp overloads are = deleted so a
  temporary can't bind to the view path and dangle, and CoreML
  serialization is made deterministic for byte-stable output.
• Streaming parser — the KataGo model parser streams the gzip
  through a bounded ~1 MB refill buffer (RAII gzFile) instead of
  decompressing the whole file into memory, preserving the existing
  NaN/Inf weight validation.
• Release weights on the ANE-only path — ModelDesc::releaseWeights()
  frees the in-memory FP32 weight arrays (keeping scalar shape metadata),
  gated by a new ComputeContext::aneOnly flag that is true only when
  every configured device index is MLX_MUX_ANE. On the MLX backend it
  fires in convertAndCreateCoreMLOnlyHandleMLX after the model has been
  converted to CoreML on disk; it is safe because the ANE path re-reads
  the model from modelPath (not the in-memory arrays), the
  ComputeHandle ctor takes the ANE early-return before building any
  MLX/GPU model (the only weight-array consumer), only scalar dims are
  read afterward, and it runs under computeHandleMutex. The GPU/MLX
  path keeps its weights (aneOnly is false whenever any thread uses the
  GPU). releaseWeights() covers the transformer descriptors too
  (attention incl. ropeFreqs, FFN, RMSNorm tips).

Measured (19×19, MLX ANE path)

Resident-memory A/B (mlxDeviceToUse=100, cold conversion every run),
before vs after the levers. Peak RSS is the /usr/bin/time -l maximum
over load+convert; idle steady-state is ps-sampled after load+genmove.

Model	Peak RSS (load+convert), before → after	Idle steady-state RSS, before → after
Classical 40b (b40c256)	1.61 → 1.05 GB (−35%)	0.41 → 0.16 GB (−62%)
18b nbt (b18c384nbt)	0.94 → 0.63 GB (−33%)	0.64 → 0.32 GB (−49%)
Transformer (b10c384h6nbttflrs)	1.69 → 1.52 GB (−10%)	1.69 → 1.52 GB (−10%)

The peak win is largest on the classical 40b (the duplicate owning weight
copies + whole-gzip decompress the levers target dominate there); the
transformer gains least because most of its converter-side weight is
derived (RoPE tables, rotation matrix, per-head out-proj slices) which
keeps an owned buffer, and its absolute peak is dominated by ANE
compilation of the MIL graph downstream of these host-side levers.

The levers don't change inference numerics: testgpuerror ANE output is
bit-identical before vs after on a convnet and all three transformer nets
(release happens after conversion; the MLX/GPU path never invokes the
converter), and benchmark throughput is within run-to-run noise on both
the GPU and ANE paths.

How to build

cd cpp
cmake -G Ninja -DUSE_BACKEND=MLX
ninja

Requires CMake ≥ 3.27 and brew install mlx.

Validation

Cross-backend testgpuerror vs an Eigen FP32 reference, via
cpp/rungpuerrortest.sh (all 12 nets across 9×9 / 13×13 / 19×19 / 10×14 /
rectangular / rect-buffer / weird-settings). MLX FP16 (auto) vs Eigen
FP32 — winrate error per net (99%-ile / worst-case max across that
net's configs):

Network	Ver	GPU FP16 (99% / max)	ANE FP16 (99% / max)
run4-b6c96	v3	0.22% / 0.43%	— pre-v8
grun50-b6c96	v4	0.31% / 1.11%	— pre-v8
g103-b6c96	v5	0.28% / 1.31%	— pre-v8
g170e-b10c128	v8	0.60% / 2.35%	0.40% / 1.45%
kata1-b18c384nbt (s5832)	v11	0.45% / 1.03%	0.39% / 0.65%
kata1-b18c384nbt (s9996)	v14	0.40% / 1.36%	0.76% / 1.38%
kata1-b28c512nbt	v15	0.35% / 1.30%	0.56% / 2.40%
b18c384nbt-humanv0	SL	0.14% / 0.42%	0.22% / 0.52%
b5c192nbt-v16test	v16	0.11% / 0.18%	0.19% / 0.25%
b7c96h3tfrs · RoPE+RMSNorm	v17	0.59% / 2.49%	0.0003% / 0.0004% ᵗ
b4c256h4nbttflrs · SiLU+sp-RMSNorm	v17	0.20% / 0.30%	0.48% / 0.79%
b7c96h6kv3qk32v16tflrs · GQA+bnorm	v17	0.24% / 0.59%	0.0001% / 0.0003% ᵗ

Pass: GPU 37/37 configs, ANE 31/31 supported configs. FP32-vs-Eigen
sanity (numerical noise): GPU ≤ 0.00092%, ANE ≤ 0.00082%.
./katago runtests and ./katago runnnlayertests pass.

ᵗ 96ch transformers hit the converter's full-FP32 tier on ANE (FP16 == FP32);
the 256ch SiLU transformer uses the non-spatial-FP32 tier (real FP16 residual).

The 6 pre-v8 ANE configs (v3/v4/v5) are skipped — the CoreML converter
supports model versions 8–17 only. Run ANE testgpuerror serially or with a
per-process TMPDIR: the converter stages weights in a shared
$TMPDIR/katagocoreml_weights dir, so concurrent processes race.

GPU throughput: MLX vs Metal

Single-GPU throughput of the two Apple-Silicon GPU backends, both configured
GPU-only (no ANE; MLX gpuIdx 0, Metal default device 0), on an Apple
M3 Max. The two backends take different GPU strategies, so this compares
each backend's actual GPU-only behavior rather than an identical-precision
kernel:

• MLX GPU — F(2×2, 3×3) Winograd path in FP16 (mlxUseFP16 auto).
• Metal GPU — MPSGraph in FP32 (this backend's GPU path, incl. the
  transformer trunk, is FP32-only).

The Metal binary is built from feature/metal-transformer-silu-b10c384h6
(the branch whose Metal GPU path supports the v15+ transformer trunk). Each
net was benchmarked with benchmark -half-batch-size -t 8,16,32 -v 800 at
19×19, run sequentially — one backend / one net at a time. Cells are
visits/s at each thread count; the human-SL net used
humanSLProfile=rank_9d (same override on both backends). The three v17
transformer MLX rows were re-measured after the FP16-attention fix
(52240987, median of 3 runs); the Metal column and all other rows are
unchanged.

Network	Ver	MLX GPU · FP16 visits/s @ t=8 / 16 / 32	Metal GPU · FP32 visits/s @ t=8 / 16 / 32	MLX best ÷ Metal best
run4-b6c96	v3	3808 / 5797 / 6930	3112 / 3804 / 3232	1.82×
grun50-b6c96	v4	3986 / 6160 / 7771	3614 / 5357 / 6400	1.21×
g103-b6c96	v5	4166 / 6354 / 7860	3458 / 5691 / 6254	1.26×
g170e-b10c128	v8	2710 / 3773 / 4538	2479 / 2943 / 3796	1.20×
kata1-b18c384nbt (s5832)	v11	529 / 662 / 711	359 / 506 / 476	1.41×
kata1-b18c384nbt (s9996)	v14	548 / 666 / 712	359 / 466 / 447	1.53×
kata1-b28c512nbt	v15	221 / 236 / 221	138 / 164 / 164	1.44×
b18c384nbt-humanv0	SL	487 / 576 / 552	333 / 430 / 413	1.34×
b5c192nbt-v16test	v16	2066 / 3372 / 3969	1926 / 3009 / 3790	1.05×
b7c96h3tfrs · RoPE+RMSNorm	v17	1698 / 2418 / 2925	1532 / 1907 / 1551	1.53×
b4c256h4nbttflrs · SiLU	v17	1226 / 1693 / 1848	900 / 1175 / 1007	1.57×
b7c96h6kv3qk32v16tflrs · GQA	v17	1364 / 1817 / 2005	1119 / 1240 / 1275	1.57×

MLX-GPU is faster than Metal-GPU on every net (best-thread throughput
1.05×–1.82×), with the largest margins on the deep NBT nets
(b18c384nbt ~1.4–1.5×, b28c512nbt ~1.44×) and the v17 transformer nets
(~1.5–1.6×, now that the GPU attention path runs end-to-end in FP16) where
the FP16 Winograd path has the most to exploit. The FP16-vs-FP32 precision gap is part of that lead;
accuracy of each path vs an Eigen FP32 reference is in the Validation
table above.

Status

Draft — opening for early feedback on the backend's structure, the
tuner approach, the GPU/ANE dispatch wiring, and the transformer-trunk
support (incl. the ANE FP16 precision tiers) before promoting to
ready-for-review.