Repository navigation
Conversation
|
I built this branch on an M4 to check the numbers, since the ones in the description come from an M1. They hold, though the ratio is lower here. Setup: Apple M4 (10 CPU / 10 GPU, 16 GB), macOS 26, Release build of The C++ suite gives 370 passed and 2 skipped (
MPS float16 and MPS float32 both produced text identical to CPU float32 over the whole workload. Three things came out of it that may be worth folding into the PR.
>>> ctranslate2.get_supported_compute_types("cpu")
{'float32'}
# ctranslate2 4.8.1 from PyPI
>>> ctranslate2.get_supported_compute_types("cpu")
{'float32', 'int8', 'int8_float32'}I would put
I did not test Marian, batch sizes above 1, multi-GPU, or memory pressure on an 8 GB machine. |
ctranslate2 has no Metal backend in any released version, which is the entire reason a seven-minute recording cost seven minutes of CPU on a machine with an idle GPU. OpenNMT/CTranslate2#2077 adds one. Built from that branch and installed, the reference recording: CPU int8 443.1 s 725 words Metal float16 80.4 s 724 words Same text, five and a half times faster. asr_device decides: auto asks the library whether it has the backend, so a stock wheel keeps behaving exactly as before, and mps insists and explains itself rather than quietly running slowly. int8 becomes float16 on the GPU, because carrying the CPU's default over picks the slowest configuration the build offers, 57.6 s against 24.9 s, while looking like a request for the fast one. The device is in the cache fingerprint: float16 on the GPU is not the same transcript as int8 on the CPU, and a cache that ignored that would hand back the other one and call it a hit. WITH_RUY is not optional when building that branch, even though its own instructions omit it: without it there is no int8 on the CPU at all, so the fallback stops working. That and the CMake 4 flag the vendored dependency needs are in the README. Live text does not go through any of this. It runs on Apple's model on the Neural Engine, which is neither the CPU nor the GPU.
|
@nerln |
|
Thanks a lot for your work My initial feedback:
|
798c17d to
11d9bd4
Compare
|
@jordimas Thanks for the feedback. I have addressed the requested items in the latest update.
Full evaluation results:
CPU used FP32 and MPS used FP16 with identical models and decoding settings. Model loading and warm-up were excluded from inference timing. Exact commands, hardware details, and methodology are documented in the benchmark README. |
|
The Successful run: https://github.com/TBO22/CTranslate2/actions/runs/32951573494/job/98123905546 |
|
@nerln The latest commits now resolve generic MPS int8 to int8_float16 and cache the expanded FP16 weights once, so warm inference uses the optimized FP16 kernels instead of repeatedly quantizing activations and running the slower INT32 path. Based on your measurements, this should reduce the previous 57.61 s result toward 24.91 seconds which is about 2.31× faster than the old MPS INT8 path and roughly 2.03× faster than CPU INT8. Could you retest the latest PR head (74b510a) on your M4? The Ruy and CMake 4 build flags are now documented as well. |
|
Rebuilt at 74b510a on the same M4 as before. The build side is clean; the latency re-test is in a separate comment below. The documented cmake line now works as written. The C++ suite, run from the In August at 2f6a066 the same suite gave 370 passed and 2 skipped. Compute types from that build, which is the check that the earlier documented line was failing: >>> ctranslate2.get_supported_compute_types("cpu")
{'float32', 'int8', 'int8_float32'}
>>> ctranslate2.get_supported_compute_types("mps")
{'bfloat16', 'float16', 'float32', 'int8', 'int8_bfloat16', 'int8_float16', 'int8_float32'}
>>> ctranslate2.get_mps_device_count()
1CPU int8 is there, so the fallback path a Whisper user drops onto is intact. |
|
Retested at 74b510a on the M4. Short answer: the resolution landed, the speed did not, and there is a correctness difference that I think matters more than either.
A generic It is still the slowest path in the build. 2.46x slower than MPS float16, and 0.88x against CPU int8, which is exactly where it sat in August. A second pass gave 16.03, 37.95 and 36.03 for MPS float16, MPS int8 and CPU int8, and a second audio file gave 6.14, 18.66 and 16.96, so the ordering is stable. Switching the weight cache off costs 1.5%, inside the window-to-window spread. If expanding the weights once were what the time was going into, that switch should have hurt far more, so the remaining cost looks like it is per call on the activations rather than in the weights. You can see the profile and I cannot. The part I would chase first is not the timing. On one recording the MPS int8 path produces different words from the other four configurations: two words missing from the start of a window, reproducible across both passes and with the cache off. CPU float32, CPU int8, MPS float16 and MPS float32 agree with each other word for word on that same window. It did not reproduce on a short public clip, so it may depend on content, and the recording is private so I cannot attach it. A public file where MPS int8 and CPU int8 disagree on the text is probably the same bug. One caveat on the table: every row is lower than my August numbers, CPU rows included, and I no longer have that script, so I cannot promise the input was identical. Compare within this table, and across the two only as ratios. |
|
@nerln Thank you for retesting this and sharing the detailed results. Profiling showed that the main overhead comes from quantizing activations on every call and running the native INT8 matrix operations. This path can also introduce enough numerical variation to change token selection during decoding. I have addressed this in commit In my local Whisper beam size 5 test with a 12.5 second audio, the updated path completed in 5.11 seconds, compared with 10.53 seconds for native INT8 and 5.21 seconds for explicit FP16. The generated tokens matched the FP16 result exactly. The complete MPS enabled C++ suite and all Python MPS tests also passed. It would be helpful if you could repeat the M4 test using the latest commit. |
|
Retested at Public audio this time, so the numbers can be checked: first 16 clips of the FLEURS
MPS Text: CPU float32, MPS float16, MPS int8 and MPS int8 with the cache off produced the same 157 words, token for token, in all three passes. The divergence I reported last time is gone. CPU int8 differs by two commas and one dropped article. The CPU rows were taken under memory pressure, hence the 160-second range on float32. Read them as an upper bound; the MPS rows are tight. One trap for anyone else rebuilding this. My first retest reported nothing had changed, from a stale binary: C++ suite from the same |
| # depending on infrastructure owned by the contributor. | ||
| build-and-test-mps: | ||
| if: github.repository != 'TBO22/CTranslate2' || github.event_name == 'push' | ||
| runs-on: ${{ github.repository == 'TBO22/CTranslate2' && 'self-hosted' || 'macos-15-xlarge' }} |
There was a problem hiding this comment.
I tested the latest PR head (dcf9306f) on GitHub’s standard macos-15 runner. The validation run passed:
- Metal computation executed successfully using
Apple Paravirtual device. - MPS C++ suite: 193 passed.
- CPU/shared suite from the MPS-enabled build: 198 passed, with one existing CPU-only skip.
- Python integration tests: 5 passed.
Please update the existing build-and-test-mps job to use runs-on: macos-15 and remove the contributor-specific runner selection and condition. This runner is free for public repositories, so we can maintain runtime coverage without macos-15-xlarge.
Please retain all three test stages, including verification that device="auto" stays on CPU. The job should fail if Metal is unavailable rather than silently skipping MPS tests.
A 30-minute timeout and uploading test reports with if: always() would also help with diagnosing future failures.
| #ifdef CT2_WITH_MPS | ||
| if (device == Device::MPS) { | ||
| const ScopedDeviceSetter scoped_device_setter(device, index); | ||
| mps::synchronize(); |
There was a problem hiding this comment.
Please make device synchronization wait for work from all MPS threads. The regression run queues fill(42) on one thread; synchronize_device(MPS, 0) on another returns while the value is still 0. This call currently synchronizes only the caller's stream. Please fix this and retain the regression test.
| return device_to_str(view.device()); | ||
| }, | ||
| "Device where the storage is allocated (\"cpu\" or \"cuda\").") | ||
| "Device where the storage is allocated (\"cpu\", \"cuda\", or \"mps\").") |
There was a problem hiding this comment.
Please update the __cuda_array_interface__ guard below to reject view.device() != Device::CUDA. MPS currently exposes Metal memory as CUDA. This regression run reproduces it; please retain a test asserting that MPS raises AttributeError.
7144024 to
7b390fb
Compare
|
@jordimas Device synchronization now waits for pending work across all registered MPS thread-local streams. I retained the cross-thread regression test and also fixed the related array-interface issue so MPS storage no longer exposes Locally, 194 MPS tests, 198 CPU/shared tests, and all 6 Python integration tests pass. The synchronization regression also passed 100 consecutive runs. |
|
Retested at The three review points hold here. 397 C++ tests pass and 1 is skipped. The six Python MPS tests pass with The device-wide synchronization costs nothing measurable. Same protocol as my 6 September run, both wheels measured back to back this afternoon on the same machine, three passes each, medians:
Every MPS configuration and CPU float32 produced the same 157 words on both builds, token for token. One thing to look at before merging. One worker running MatMul on MPS in a loop, one thread calling The same program against I could not reach it from Python: two threads, one translating on MPS and one loading and dropping a second model so The regression test parks its worker after queueing, so the device sync is the only thread touching that stream. Since flushing another thread's un-submitted work is what that test asks for, the two uses need serialising rather than separating: a per-stream mutex taken by the encode path and by Repro, 40 lines: #include <atomic>
#include <chrono>
#include <cstdio>
#include <thread>
#include <ctranslate2/devices.h>
#include <ctranslate2/ops/ops.h>
#include <ctranslate2/storage_view.h>
using namespace ctranslate2;
int main(int argc, char** argv) {
const int seconds = argc > 1 ? std::atoi(argv[1]) : 10;
std::atomic<bool> stop{false};
std::atomic<long> ops{0}, syncs{0};
std::thread worker([&] {
StorageView a({256, 256}, static_cast<float>(1), Device::MPS);
StorageView b({256, 256}, static_cast<float>(2), Device::MPS);
StorageView c(Device::MPS);
const ops::MatMul matmul;
while (!stop.load()) {
matmul(a, b, c);
ops.fetch_add(1);
}
});
std::thread syncer([&] {
while (!stop.load()) {
synchronize_device(Device::MPS, 0);
syncs.fetch_add(1);
}
});
std::this_thread::sleep_for(std::chrono::seconds(seconds));
stop.store(true);
worker.join();
syncer.join();
std::printf("ops=%ld syncs=%ld\n", ops.load(), syncs.load());
return 0;
} |
|
@nerln Thanks for the clear reproducer. I reproduced the crash on my M1 in 5/5 runs, including the same Metal command-buffer and encoder assertions. I implemented per-stream locking and adapted a new regression test from your 40-line reproducer. The issue was that device-wide synchronization could commit another thread's command buffer while that thread was still encoding work. The stream is now locked for the full encoding operation, including flush and synchronization. Normal inference should see no meaningful lock contention since each worker uses its own thread-local stream. After the fix, the standalone reproducer completed five consecutive 5-second runs without crashing. The new regression test passed 20/20 runs, the existing cross-thread synchronization test passed 200/200, and the full MPS, CPU, and Python test suites passed. Marian FP16 latency remained effectively unchanged at 154.77 ms compared with 153.91 ms before the fix. Thanks again for identifying the race and providing a reproducer that made it straightforward to reproduce and verify the fix. |
|
Confirmed on the M4 at
The lock costs Whisper nothing measurable.
Both differences are inside the run-to-run spread, in opposite directions. All twelve runs produced the same 157 words. |
|
Hello. I would like to merge this in the next major version. Resolve merge conflicts with master. Thanks! |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master's TopKWithExhaustedCandidates test (OpenNMT#2104) fails on MPS for k=16: rows with fewer than k finite candidates came back with 0 in the exhausted slots. For 8 < k <= 16 with 16-byte aligned result rows the backend used MPSMatrixFindTopK, which pads exhausted slots with 0 and a bogus index instead of returning the masked input; a beam search with 2 * beam_size = 16 would read that 0 as the best score. small_topk already handles exhaustion (masked candidates beat the -INFINITY/UINT_MAX initial slots on index), so it now covers every 1 < k <= 16 for float32, float16 and bfloat16, and the MPSMatrixFindTopK path and its thread-local kernel cache are removed. k > 16 keeps the CPU fallback. Adds FusedBeamSearchTopKWithExhaustedCandidates, which checks the opt-in fused beam search TopK against the materialized path on rows with fewer than k finite candidates, for (beam, k) = (4, 8), (5, 10), (8, 16). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A translator on MPS with inter_threads=4 and four callers crashed with SIGSEGV inside MPSSetNDArraysOnComputeEncoder, from mps::gemm. gemm_kernel() kept one MPSMatrixMultiplication per shape for the whole process; the thread_local map only cached pointers to those shared objects. Apple documents MPSKernel objects as unsafe to use from several threads at once, and workers of one translator run the same shapes. The cache is now thread_local and owns its kernels, released when the thread ends. Each encoded kernel is recorded as in use so it outlives the command buffer even if its thread exits first. A nil kernel now throws instead of being cached and silently doing nothing. Adds ConcurrentGemmFloat16/Float32WithTheSameShape and ConcurrentBatchedMatMulFloat16WithTheSameShape: four threads, the same 64x512 by 512x2048 shape, 300 iterations each, checked against the CPU. They crashed 10 times out of 10 before this change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…from idle workers or destructors A failed asynchronous command buffer recorded its error in one process-wide slot, and whichever thread synchronized next threw it: an innocent replica could fail with another one's error. Worse, the throw could come from ReplicaWorker::idle(), outside any job, ending the worker thread and the process with std::terminate, or from Model::~Model through synchronize_device. CUDA ignores errors on both paths. Each stream now owns its error in a small shared state captured by the completion handler, so a handler that runs after its stream is gone is safe, and only that stream's next command_buffer() or synchronize() throws it, once. A per-command-buffer token stops a late handler from recording a failure a second time. synchronize_all() reports one stream's error and leaves the others to their owners. idle() calls the new synchronize_stream_nothrow(), identical to synchronize_stream() on CPU and CUDA; on MPS it logs once and keeps the error for the stream's next operation. synchronize_device(MPS), which only ~Model and move_variables call, uses the same no-throw barrier; the wait itself is unchanged. tests/mps_error_test.cc injects errors through an internal hook (src/mps/testing.h). On the previous code 5 of its 10 tests fail and 3 abort with std::terminate (ReplicaWorker::idle, a Translator, ~Model). The backend stays experimental and opt-in: a warning when a model loads on MPS, "mps (experimental)" in the device docstrings and the CLI help, a note in README.md and docs/environment_variables.md, and a CHANGELOG entry. WITH_MPS stays OFF by default and out of the wheel scripts. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@jordimas I went through the list on an M4 (macOS 27). The branch with the result is
|
Merge master, fix MPS TopK at k=16, a GEMM kernel race and GPU error handling
…inal-20261015 # Conflicts: # CHANGELOG.md
|
Thanks @nerln for the detailed review and contribution. I have integrated your changes while preserving the original commits and authorship. They address the shared During benchmarking, I found that using 16 private slots for every TopK call slowed the common beam-4 The complete C++ suite passes with 416 tests and one existing skip. @jordimas , the requested concurrency, synchronization, error-handling, GPU-lifetime, and regression test updates have been integrated. |
|
@jordimas, could you rerun the failed Linux x86-64 wheel test when you get a chance? I cannot rerun jobs in the upstream repository due to admin rights required. It crashed during the Wav2Vec2 test while a Hugging Face background download was still running. The same test passed on Nerln’s branch, and nothing affecting the Linux or CPU path changed afterward, so this looks like CI flakiness rather than an MPS regression. If it fails again, I’ll disable the background safetensors / Xet download for the test job. |
This branch has not been deployed
3 participants
Summary
This PR adds an experimental native Metal/MPS backend for CTranslate2 on Apple Silicon. It is opt-in with
-DWITH_MPS=ONand makes the existing C++ and Python APIs acceptdevice="mps"without changing the default CPU, CUDA, or HIP behavior.I started this port about six months ago because CTranslate2 could use Apple CPUs efficiently but had no path to the Apple GPU. The first versions could run a few operations, but real Marian and Whisper inference exposed missing operators, excessive synchronization, lifetime bugs, and poor batch-size-1 performance. I used those end-to-end workloads—not only large synthetic GEMMs—to guide the implementation in this PR.
What is included
Device::MPSsupport in device parsing, dispatch, storage, synchronization, the CLI, and Python bindingsctranslate2.get_mps_device_count()andget_supported_compute_types("mps")compute_type="auto"selects FP16M == 1GEMV path for autoregressive projections, including a fused bias/residual/activation epiloguek <= 8), with MPSMatrix TopK used for supported larger valuesThe dispatch changes also preserve non-MPS builds. In particular, FP16/BF16 dispatch is restricted to enabled GPU backends so MSVC does not instantiate unsupported CPU half-precision symbols in CUDA wheel builds.
Build and use
cmake -S . -B build-mps \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \ -DWITH_MPS=ON \ -DWITH_ACCELERATE=ON \ -DWITH_MKL=OFF \ -DWITH_RUY=ON \ -DOPENMP_RUNTIME=NONE cmake --build build-mps -jThe minimum deployment target is macOS 11. MPS is not enabled in prebuilt wheels by this PR and currently has to be built from source.
Correctness and CI validation
Local validation and the reportable benchmarks were run on a MacBook Air with an Apple M1 (7-core GPU, 8 GB unified memory), arm64, macOS 26.5.2, using Release builds.
The CI configuration includes a dedicated real Apple Silicon MPS GPU job. It builds with
-DWITH_MPS=ON, runs the MPS runtime suite, and then runs the normal CPU/shared suite from the same MPS-enabled build. It also builds the Python bindings, tests explicitdevice="mps"inference, and verifies thatdevice="auto"remains on CPU so the experimental backend is opt-in.The equivalent workflow passed on the self-hosted Apple M1 runner: 192 MPS C++ tests passed, 198 CPU/shared C++ tests passed with one existing CPU-only skip, and both Python MPS integration tests passed.
https://github.com/TBO22/CTranslate2/actions/runs/32951573494/job/98123905546
The upstream
macos-15-xlargejob could not be scheduled because larger-runner billing is not enabled for the repository. No test process started in that failed job; this was a runner availability/billing issue rather than a test failure. The upstream workflow retainsmacos-15-xlarge, so the merged CI configuration does not depend on contributor-owned hardware.MPS coverage includes FP32/FP16/BF16, INT8, odd sizes, tails, broadcast and nonzero batch strides, transposed GEMM, alpha/beta, interior offsets, dependent asynchronous dispatches, TopK tie behavior, Gather, Conv1D, and translation output comparison.
git diff --checkalso passes.Quality Validation
The reproducible evaluation scripts and methodology are under
tools/benchmark/quality/. CPU used FP32 and MPS used FP16 with the same models, inputs, sample order, and decoding settings. Model loading and one warm-up inference were excluded from inference timing.Performance
Performance varies by model and sequence length, so this is not intended as a universal claim. One representative end-to-end result from the M1 test machine was a 12.52-second, batch-size-1 Whisper transcription/translation workload:
This is a 3.54x end-to-end speedup for that workload. The PR also includes
tools/benchmark_mps_marian.py, which runs CPU and MPS in separate processes with warmups and repeated measurements across source lengths and beam sizes, andtests/benchmark_mps.ccfor decode GEMM, vocabulary projection, prefill GEMM, TopK, and copy-heavy operations.Current limitations
WITH_MPScannot currently be combined with CUDA or HIP in one buildAI-assisted development disclosure
I used Codex with GPT-5.6 as a pair-programming tool for repository navigation, repetitive implementation work, test scaffolding, diff review, and tracing build and lifetime failures. I did not treat generated suggestions as authoritative. I researched the execution model and kernel design myself, including how Apple MLX kernels differ from ggml/PyTorch and CUDA implementations and how Apple unified memory affects synchronization and ownership. I selected the design, ran the benchmarks, reproduced correctness failures, reviewed the changes, and remain responsible for the implementation and results in this PR.
I understand that this is a large, performance-sensitive change and am happy to respond to detailed review or split follow-up work where maintainers prefer a different boundary.