Reduce CoreML conversion peak and ANE steady-state memory

[ChinChangYang]

Summary

Draft. Cuts memory during the on-device KataGo→CoreML conversion and while running the ANE/CoreML path, with byte-identical converter output. Three changes (first commit):

• Non-owning weight views — the converter's weight tensors become views into the parsed model instead of owning extra FP32 copies (derived/transposed tensors keep an owned buffer), dropping redundant resident weight copies during conversion. CoreML serialization is made deterministic (SetSerializationDeterministic) so output is byte-stable.
• Streaming parser — the KataGo model parser streams the gzip through a bounded ~1 MB refill buffer instead of decompressing the whole file into memory, preserving the existing NaN/Inf weight validation.
• Release weights on ANE — ModelDesc::releaseWeights() frees the in-memory weight arrays (keeping scalar shape metadata). The Metal backend calls it on the ANE (CoreML) path after converting from the model file on disk, gated by a new ComputeContext::aneOnly flag so it only fires when every configured device is ANE — the GPU/MPSGraph path keeps its weights. Serialized under computeHandleMutex; only scalar dims are read afterward.

A follow-up commit (Enforce non-owning weight-view contract at compile time) hardens the first bullet: WeightEntry now stores a non-owning view, so its backing std::vector must outlive serialization. The rvalue overloads of addConstOp/registerWeight are = deleted so a temporary can never bind to the view path (it would dangle) — temporaries are forced through the owning addOwnedConstOp/registerOwnedWeight. No codegen change: every existing call site passes an lvalue model member, verified by a full METAL rebuild plus a negative-compile test confirming the deleted overload is selected for a temporary.

Merged: Metal GPU + CoreML/ANE transformer support (#1205)

This branch now merges #1205 (transformer-net support — SiLU, GQA, learnable RoPE, RMSNorm, SwiGLU — for the Metal GPU and CoreML/ANE paths) so the two efforts reconcile in one place. All four merge conflicts were in the katagocoreml converter and came from a single root: this PR's non-owning FloatView weight registration vs #1205's per-weight FP32 precision tiers (is_fp32). Both are kept (registerWeight stamps entry.is_fp32; WeightSerializer stores fp16 iff use_fp16 && !is_fp32; addConstOp keeps the emitConstOp refactor and passes is_fp32 = (m_weight_dtype == FLOAT32)).

The merge required one integration fix (commit Conform CoreML transformer derived consts to the owned-weight + FP32 contract). The transformer attention builder emits four function-local tensors — RoPE cos/sin tables, the rotation matrix R, and per-head out-projection weight slices. After this PR made WeightEntry::data a non-owning view, those locals must:

1. be registered through the owning addOwnedConstOp (with std::move) so the view doesn't dangle once the build function returns (serialization runs afterwards); and
2. carry the same per-weight FP32 marking as addConstOp, because emitConstOp declares each const's dtype as m_weight_dtype. Without (2) a derived const that lands in the attention/value-head FP32 sub-region of an FP16 model is declared FP32 but stored FP16, which CoreML/ANE rejects at load ("Metadata data type does not match requested type", BNNS error −14), crashing every FP16-ANE transformer.

addOwnedConstOp/registerOwnedWeight now thread is_fp32 = (m_weight_dtype == FLOAT32), mirroring addConstOp so stored dtype always matches the declared dtype. This also fixes the same latent mismatch for addLinearOp's transposed value-head weights. The streaming parser needed no change (the new parseTransformer* use the unchanged readFloats), and ModelDesc::releaseWeights() already covers the transformer descriptors (including ropeFreqs).

Measured (19×19, ANE path)

Resident-memory A/B on the CoreML/ANE path (metalDeviceToUseThread0=100), cold conversion every run, before vs after this PR's three memory levers. before is this branch with the levers removed (the #1205 transformer tip 39f82f6d — no non-owning views, no streaming parser, no aneOnly release); after is HEAD. Peak RSS is the /usr/bin/time -l maximum over load+convert; idle steady-state is ps-sampled after the model is loaded and released. Numbers are GB, reproduced across two runs (<0.1% spread).

Model	Peak RSS (load+convert), before → after	Idle steady-state RSS, before → after
Classical 40b (b40c256)	1.50 → 0.80 GB (−47%)	0.38 → 0.25 GB (−33%)
18b nbt (b18c384nbt)	0.88 → 0.48 GB (−45%)	0.59 → 0.20 GB (−66%)
Transformer (b10c384h6nbttflrs)	1.58 → 1.37 GB (−13%)	0.66 → 0.46 GB (−31%)

The peak win is largest on the classical 40b, where the conversion transients the levers target (duplicate owning weight copies + whole-gzip decompression) dominate; the idle win is largest on the 18b nbt, which has the most resident FP32 weight for the ANE releaseWeights to free.

Why the transformer's peak reduces less (−13%), and why its absolute peak is highest despite the smallest file. Both are expected and not a regression — verified by instrumented byte-accounting at serialization time:

• The peak levers (non-owning views + streaming parser) act on host RAM during parse/convert. The transformer gives them less to remove: (i) ~39% of its converter-side weight bytes are derived tensors — RoPE cos/sin tables (the bulk), the rotation matrix, and per-head out-projection slices — that are host-computed and therefore keep an owned buffer (addOwnedConstOp), so the non-owning-view lever can dedup ~0% of them, versus ~100% of a conv net's weights (the conv nets carry only ~0.07 MB of owned tensors total). (ii) Its 38 MB file makes the streaming parser's removed whole-decompress transient small (~42 MB, vs ~187 MB for the 40b). Combined, the levers free ~80 MB here vs ~365 MB on the 40b.
• The ~1.4 GB absolute peak is reached after the C++ converter returns (which itself peaks at only ~0.24 GB): it is CoreML/ANE compilation of the MIL graph into an ANE program. The transformer emits ~3.4× more MIL ops than the 40b (op-level attention: per-block RoPE, GQA per-head slice/concat, 361×361 softmax), and the ANE compiler's memory scales with graph complexity, not weight bytes — so the smallest model on disk has the largest compile-time footprint. This floor sits downstream of, and is unaffected by, all three (host-side) levers.

Cross-backend parity vs an Eigen reference is unchanged on both the GPU and ANE paths.

Test plan

• METAL build; ./katago runtests (All tests passed) + ./katago runnnlayertests (14 configurations)
• ./katago testgpuerror vs Eigen reference — GPU and ANE paths both within tolerance, no NaN
• ANE-path RSS A/B (above); GPU path unaffected (aneOnly false)
• Full rungpuerrortest.sh regression vs current Eigen FP32 references: Metal GPU 37/37 pass; CoreML/ANE 31/31 supported configs pass (the 6 "failures" are the expected pre-v8 SIGABRTs — run4/grun50/g103, nets predating CoreML-converter support). The two transformer nets the merge had broken on the FP16 ANE path (b7c96h3tfrs, b7c96h6kv3qk32v16 GQA) now load and match the reference to <0.0005% winrate; convnet ANE output is byte-identical and the GPU path is unchanged.

Note

The CI cache fix that was previously bundled here has been split into its own PR: #1204 (orthogonal fix for a stale-cache link failure on the build-macos-metal job). This PR is now the memory work plus the merged #1205 transformer support.

🤖 Generated with Claude Code

[ChinChangYang]

I reviewed and approved this.

[ChinChangYang]

Completed. Plan to set this PR to ready when #1205 is merged.