Acquire the GIL when releasing NumPy-backed Core ML inputs

[cdeil]

Summary

• acquire the Python GIL before releasing the NumPy owner retained by PybindCompatibleArray
• keep the existing zero-copy MLMultiArray input path and its lifetime guarantee
• add a macOS regression test that observes the GIL state at final NumPy-owner release
• run the potentially crashing prediction in a subprocess so a regression is reported as an ordinary test failure

Problem

PybindCompatibleArray creates an MLMultiArray over array.mutable_data() with deallocator:nil and retains the NumPy storage owner in the C++ member py::array m_array.

Core ML may retain that feature after synchronous predict() returns and release it later on a private runtime queue. On macOS 26.6.2 and macOS 27 beta, the observed queue was:

com.apple.coreml.MLE5ExecutionStream.resetQueue

When Objective-C destroys PybindCompatibleArray on that queue, the implicit C++ member destructor decrements the py::array reference without an attached Python thread state or the GIL. A debug pybind11 build reports the violation directly:

pybind11::handle::dec_ref() is being called while the GIL is either not held or invalid
The failing ... dec_ref() call was triggered on a numpy.ndarray object.

With a release wheel and Python 3.12 through 3.14, a one-operation output = input + 1 ML Program predicts correctly and then segfaults asynchronously. The native stack is:

_PyObject_Free
libcoremlpython
object_cxxDestructFromClass
-[MLFeatureValue dealloc]
-[MLE5InputPortBinder reset]
-[MLE5ExecutionStream _reset]

Conversion is not involved in the failure: loading a model package generated by a separate process has the same result.

Why Python 3.12+ exposes it

This is a binding lifetime bug rather than a CPython allocator bug. Python's C API requires an attached thread state and the GIL before accessing Python objects or changing their reference counts.

Python 3.11's _PyObject_Free happened to proceed directly to pymalloc_free. Python 3.12+ first obtains per-interpreter allocator state through _PyInterpreterState_GET(). The Core ML reset queue has no Python thread state, so the invalid foreign-thread decrement now reliably faults. The contract violation exists regardless of whether a particular Python/runtime combination happens to crash.

Related documentation:

• https://docs.python.org/3.13/c-api/init.html#thread-state-and-the-global-interpreter-lock
• https://peps.python.org/pep-0684/

Fix

The explicit dealloc method:

1. calls m_array.release() to detach the raw Python handle without decrementing it;
2. acquires/attaches the current Core ML queue thread with py::gil_scoped_acquire;
3. decrements the detached handle while the GIL is held.

After dealloc returns, the implicit C++ destruction sees an empty m_array, so there is no second Python reference-count operation.

This is deliberately scoped to ownership release. It preserves the existing zero-copy input behavior and keeps the NumPy allocation alive for as long as Core ML uses its data pointer; it neither copies tensor data nor assumes that Core ML will destroy features on the prediction thread.

Regression test

The test creates a one-operation FP32 ML Program and predicts from a NumPy array in a child process. A weakref callback records PyGILState_Check() when Core ML releases the last owner. It verifies both that:

• the NumPy owner is eventually released rather than leaked; and
• final release occurs while the GIL is held.

The subprocess is important because the unfixed behavior can abort or segfault the interpreter. Before this patch, the parent test reports the child process's signal and native pybind11 diagnostic instead of terminating the full pytest worker.

The test lives under coremltools.test, so it is included by the existing macOS package-test job. Its explicit GIL check also exercises Python 3.10 CI without depending on the Python 3.12 allocator crash.

Validation

Host: Apple Silicon M3, macOS 26.6.2 build 25G83, Xcode 26.6.

• Python 3.10.18 debug native build: regression test passes
• Python 3.13.11 debug native build: regression test passes
• same Python 3.13.11 build with only the fix removed: regression test fails; child exits -6 with pybind11's off-GIL dec_ref() assertion
• pre-generated-model reproducer: correct prediction and clean exit
• 500 consecutive predictions: clean completion and exit
• 20 fresh processes exiting immediately after prediction: 20/20 clean exits
• weakref lifetime probe: NumPy owner released after Core ML's delayed reset; no retained-owner leak
• remaining API tests available without the optional PyTorch test dependency pass

[TobyRoseman]

CI: https://gitlab.com/coremltools1/coremltools/-/pipelines/2789803969