Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .claude/skills/porting-4-cpp/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,16 @@ Pick the closest existing family by architecture pattern:
not the whole file. Load the closest family's `src/arch/<family>/` as a working
example.

Load weights the way every family does: `gguf_init_from_file` into
`ctx_meta`, bind the weight struct, `load_common::init_backends`, then
`load_common::load_weights` into a `load_common::WeightStorage` member that
the destructor resets before freeing the backends. Weights are memory-mapped
read-only wherever the device allows it, so never write a `ctx_meta` tensor
after load: fusions and promotions create new tensors in their own buffer and
repoint the weight slot (see `fuse_batch_norm`, cohere's
`fuse_encoder_q_bias`). A stray write crashes the process on CPU and Metal;
compare a default run against `--no-mmap` to confirm the outputs match.

Wire the new family into the arch dispatch in `src/` per the existing
family wiring. Build iteratively:

Expand Down
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,10 @@ General rules:
- Do not add new ggml operators, `ggml_type` values, or backend behavior without
CPU validation, benchmark or accuracy evidence where relevant, and an upstream
plan for ggml/llama.cpp when appropriate.
- Treat GGUF-backed weight tensors (a family's `ctx_meta`) as read-only after
load: they are memory-mapped from the file wherever the device allows it, so
a write crashes. Load-time transforms (fusions, promotions, packing) write
new tensors into their own buffer and repoint the weight slot.
- Do not mix functional changes with broad reformatting or unrelated cleanup.
- Keep comments concise. Explain non-obvious invariants, ABI contracts, tensor
layout, numerical choices, and portability traps. Do not preserve local task
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,14 @@ Input must be 16 kHz mono WAV. Use `ffmpeg` or `sox` to convert other formats:
ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav
```

On the CPU and on Metal (Apple silicon), model weights are memory-mapped from
the GGUF file rather than copied into process memory: they live in the page
cache, which the OS can reclaim and share between processes, and loading is
faster. Don't truncate or rewrite a model file in place while a program has it
loaded; write a new file and rename it over the old one. `--no-mmap`
(`transcribe_model_load_params::no_mmap`, or `TRANSCRIBE_NO_MMAP=1`) copies the
weights instead.

## Bindings

Official bindings wrap the C API for other languages:
Expand Down
7 changes: 7 additions & 0 deletions bindings/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,9 +150,16 @@ with transcribe_cpp.Model("model.gguf", device=device) as model:
print(model.device)
```

Weights are memory-mapped from the model file where the device can use host
memory (CPU, Metal on Apple silicon), so they live in reclaimable page cache
rather than process memory. Don't truncate or rewrite a model file in place
while it is loaded; write a new file and rename it over the old one.
`Model(no_mmap=True)` copies the weights instead.

| Variable | Effect |
|---|---|
| `TRANSCRIBE_BACKEND` | overrides the `"auto"` default; explicit `backend=` still wins |
| `TRANSCRIBE_NO_MMAP` | `1` copies weights instead of memory-mapping them, like `no_mmap=True` |
| `TRANSCRIBE_NATIVE_PROVIDER` | forces an installed native provider package, for example `cu12` |
| `TRANSCRIBE_LIBRARY` | loads exactly this shared library |

Expand Down
19 changes: 15 additions & 4 deletions bindings/python/src/transcribe_cpp/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1057,10 +1057,18 @@ class Model:
the escape hatch for machines whose best-ranked device misbehaves (e.g.
CI's paravirtualized Metal, a broken GPU driver) without touching code.
An explicit ``backend=`` argument always wins over the environment.

The weights are memory-mapped from the model file where the device can
use host memory (the CPU everywhere, Metal on Apple silicon): they stay
in reclaimable page cache instead of process memory. Do not truncate or
rewrite the file in place while the model is open (write a new file and
rename it over the old one). ``no_mmap=True``, or the
``TRANSCRIBE_NO_MMAP=1`` environment variable, copies them instead.
"""

def __init__(self, path: str | os.PathLike, *,
backend: Backend = "auto", device: BackendDevice | None = None):
backend: Backend = "auto", device: BackendDevice | None = None,
no_mmap: bool = False):
# Live sessions, tracked weakly: close() must free them before the
# model, because transcribe_model_free is only valid once every
# derived session is gone (use-after-free otherwise). Created before
Expand All @@ -1078,6 +1086,7 @@ def __init__(self, path: str | os.PathLike, *,
if not isinstance(device, BackendDevice) or device._handle is None:
raise TypeError("device must be a BackendDevice returned by backends()")
params.device = device._handle
params.no_mmap = bool(no_mmap)

handle = ctypes.c_void_p()
status = _lib.transcribe_model_load_file(
Expand Down Expand Up @@ -2260,6 +2269,7 @@ def transcribe(
*,
backend: Backend = "auto",
device: BackendDevice | None = None,
no_mmap: bool = False,
n_threads: int = 0,
kv_type: KVType = "auto",
n_ctx: int = 0,
Expand All @@ -2282,8 +2292,9 @@ def transcribe(
*model* may be a path (loaded and freed within this call) or an existing
Model (reused and left open). Loading a model is not free, so to transcribe
many clips keep a Model and call ``model.session().run(...)`` yourself; this
helper is for the one-shot case. ``backend`` / ``device`` apply only when
*model* is a path — they are ignored when an already-loaded Model is passed.
helper is for the one-shot case. ``backend`` / ``device`` / ``no_mmap``
apply only when *model* is a path — they are ignored when an already-loaded
Model is passed.
``family`` / ``spec_k_drafts`` and the prompting inputs (``vocabulary``,
``prompt``, ``prefix``) pass through to :meth:`Session.run`.
"""
Expand All @@ -2298,6 +2309,6 @@ def transcribe(
with model.session(**session_opts) as session:
return session.run(pcm, **run_opts)

with Model(model, backend=backend, device=device) as owned:
with Model(model, backend=backend, device=device, no_mmap=no_mmap) as owned:
with owned.session(**session_opts) as session:
return session.run(pcm, **run_opts)
6 changes: 3 additions & 3 deletions bindings/python/src/transcribe_cpp/_generated.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# Stable digest of the ABI surface below (structs, enums, macros, layout,
# prototypes). A native provider package echoes this back so the API
# package can reject an ABI-mismatched provider before dlopen.
PUBLIC_HEADER_HASH = "98540564cd329b85"
PUBLIC_HEADER_HASH = "f99ad4169374b8f3"

# === enum constants ===
TRANSCRIBE_OK = 0
Expand Down Expand Up @@ -228,7 +228,7 @@ class transcribe_whisper_chunk_trace(_c.Structure):
transcribe_ext._fields_ = [("size", _c.c_uint64), ("kind", _c.c_uint32)]
transcribe_backend_init_params._fields_ = [("struct_size", _c.c_uint64), ("artifact_dir", _c.c_char_p), ("allowed_backends", _c.c_uint32)]
transcribe_device_info._fields_ = [("struct_size", _c.c_uint64), ("name", _c.c_char_p), ("description", _c.c_char_p), ("kind", _c.c_char_p), ("device_id", _c.c_char_p), ("memory_total", _c.c_uint64), ("memory_free", _c.c_uint64), ("device_type", _c.c_int)]
transcribe_model_load_params._fields_ = [("struct_size", _c.c_uint64), ("backend", _c.c_int), ("device", _c.c_void_p)]
transcribe_model_load_params._fields_ = [("struct_size", _c.c_uint64), ("backend", _c.c_int), ("device", _c.c_void_p), ("no_mmap", _c.c_bool)]
transcribe_session_params._fields_ = [("struct_size", _c.c_uint64), ("n_threads", _c.c_int), ("kv_type", _c.c_int), ("n_ctx", _c.c_int32)]
transcribe_run_params._fields_ = [("struct_size", _c.c_uint64), ("task", _c.c_int), ("timestamps", _c.c_int), ("pnc", _c.c_int), ("itn", _c.c_int), ("diarize", _c.c_int), ("language", _c.c_char_p), ("target_language", _c.c_char_p), ("keep_special_tags", _c.c_bool), ("family", _c.POINTER(transcribe_ext)), ("spec_k_drafts", _c.c_int32), ("vocabulary", _c.POINTER(_c.c_char_p)), ("n_vocabulary", _c.c_int32), ("prompt", _c.c_char_p), ("prefix", _c.c_char_p)]
transcribe_capabilities._fields_ = [("struct_size", _c.c_uint64), ("native_sample_rate", _c.c_int32), ("n_languages", _c.c_int), ("languages", _c.POINTER(_c.c_char_p)), ("max_timestamp_kind", _c.c_int), ("supports_language_detect", _c.c_bool), ("supports_translate", _c.c_bool), ("supports_streaming", _c.c_bool), ("supports_spec_decode", _c.c_bool), ("max_audio_ms", _c.c_int64), ("n_translate_target_languages", _c.c_int), ("translate_target_languages", _c.POINTER(_c.c_char_p))]
Expand Down Expand Up @@ -301,7 +301,7 @@ class transcribe_whisper_chunk_trace(_c.Structure):
'transcribe_ext': {'size': 16, 'align': 8, 'offsets': {'size': 0, 'kind': 8}},
'transcribe_backend_init_params': {'size': 24, 'align': 8, 'offsets': {'struct_size': 0, 'artifact_dir': 8, 'allowed_backends': 16}},
'transcribe_device_info': {'size': 64, 'align': 8, 'offsets': {'struct_size': 0, 'name': 8, 'description': 16, 'kind': 24, 'device_id': 32, 'memory_total': 40, 'memory_free': 48, 'device_type': 56}},
'transcribe_model_load_params': {'size': 24, 'align': 8, 'offsets': {'struct_size': 0, 'backend': 8, 'device': 16}},
'transcribe_model_load_params': {'size': 32, 'align': 8, 'offsets': {'struct_size': 0, 'backend': 8, 'device': 16, 'no_mmap': 24}},
'transcribe_session_params': {'size': 24, 'align': 8, 'offsets': {'struct_size': 0, 'n_threads': 8, 'kv_type': 12, 'n_ctx': 16}},
'transcribe_run_params': {'size': 104, 'align': 8, 'offsets': {'struct_size': 0, 'task': 8, 'timestamps': 12, 'pnc': 16, 'itn': 20, 'diarize': 24, 'language': 32, 'target_language': 40, 'keep_special_tags': 48, 'family': 56, 'spec_k_drafts': 64, 'vocabulary': 72, 'n_vocabulary': 80, 'prompt': 88, 'prefix': 96}},
'transcribe_capabilities': {'size': 56, 'align': 8, 'offsets': {'struct_size': 0, 'native_sample_rate': 8, 'n_languages': 12, 'languages': 16, 'max_timestamp_kind': 24, 'supports_language_detect': 28, 'supports_translate': 29, 'supports_streaming': 30, 'supports_spec_decode': 31, 'max_audio_ms': 32, 'n_translate_target_languages': 40, 'translate_target_languages': 48}},
Expand Down
9 changes: 6 additions & 3 deletions bindings/rust/sys/src/transcribe_sys.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
// @generated by `cargo xtask bindgen` from include/transcribe/extensions.h
// DO NOT EDIT BY HAND. Regenerate: `cargo xtask bindgen`.
// Pinned to include/transcribe.abihash = 98540564cd329b85
// Pinned to include/transcribe.abihash = f99ad4169374b8f3

/// The public-ABI digest these bindings were generated against
/// (sha256/16 over the normalized FFI surface). The load-time version
/// gate and the CI drift check both anchor on this value.
pub const PUBLIC_HEADER_HASH: &str = "98540564cd329b85";
pub const PUBLIC_HEADER_HASH: &str = "f99ad4169374b8f3";

/* automatically generated by rust-bindgen 0.72.1 */

Expand Down Expand Up @@ -348,11 +348,12 @@ pub struct transcribe_model_load_params {
pub struct_size: u64,
pub backend: transcribe_backend_request,
pub device: transcribe_device_t,
pub no_mmap: bool,
}
#[allow(clippy::unnecessary_operation, clippy::identity_op)]
const _: () = {
["Size of transcribe_model_load_params"]
[::std::mem::size_of::<transcribe_model_load_params>() - 24usize];
[::std::mem::size_of::<transcribe_model_load_params>() - 32usize];
["Alignment of transcribe_model_load_params"]
[::std::mem::align_of::<transcribe_model_load_params>() - 8usize];
["Offset of field: transcribe_model_load_params::struct_size"]
Expand All @@ -361,6 +362,8 @@ const _: () = {
[::std::mem::offset_of!(transcribe_model_load_params, backend) - 8usize];
["Offset of field: transcribe_model_load_params::device"]
[::std::mem::offset_of!(transcribe_model_load_params, device) - 16usize];
["Offset of field: transcribe_model_load_params::no_mmap"]
[::std::mem::offset_of!(transcribe_model_load_params, no_mmap) - 24usize];
};
unsafe extern "C" {
pub fn transcribe_model_load_params_init(params: *mut transcribe_model_load_params);
Expand Down
9 changes: 9 additions & 0 deletions bindings/rust/transcribe-cpp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,15 @@ finish `init_backends()` or `init_backends_default()` before any thread
enumerates devices, queries backend availability, or loads a model; native
registry mutation is a startup-only operation and must not race those calls.

## Memory-mapped weights

Where the device can use host memory (the CPU, Metal on Apple silicon), model
weights are memory-mapped from the file and served from the page cache, which
the OS can reclaim, instead of being copied into process memory. Don't
truncate or rewrite a model file in place while it is loaded; write a new file
and rename it over the old one. Set `ModelOptions::no_mmap` (or the
`TRANSCRIBE_NO_MMAP=1` environment variable) to copy the weights instead.

## Packaging a distributable (`shared` / `dynamic-backends`)

With the **default static** build there is nothing to do — the native code is
Expand Down
9 changes: 9 additions & 0 deletions bindings/rust/transcribe-cpp/src/model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,21 @@ pub struct ModelOptions {
/// Exact process-local compute device. `None` applies the backend's
/// automatic policy; `Some` selects that device or fails without fallback.
pub device: Option<crate::Device>,
/// Copy the weights into memory instead of memory-mapping the model file.
/// Default `false`: where the device can use host memory (CPU, Metal on
/// Apple silicon) the weights are served from the file's page cache, which
/// the OS can reclaim. Don't truncate or rewrite a mapped model file in
/// place; write a new file and rename it over the old one. The
/// `TRANSCRIBE_NO_MMAP=1` environment variable forces `true`.
pub no_mmap: bool,
}

impl Default for ModelOptions {
fn default() -> Self {
ModelOptions {
backend: Backend::Auto,
device: None,
no_mmap: false,
}
}
}
Expand Down Expand Up @@ -154,6 +162,7 @@ impl Model {
.device
.as_ref()
.map_or(std::ptr::null_mut(), |device| device.handle);
params.no_mmap = options.no_mmap;

let mut out: *mut sys::transcribe_model = std::ptr::null_mut();
let status = unsafe { sys::transcribe_model_load_file(c_path.as_ptr(), &params, &mut out) };
Expand Down
2 changes: 2 additions & 0 deletions bindings/rust/transcribe-cpp/tests/device_select.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ fn every_primary_device_can_be_selected_exactly() {
&ModelOptions {
backend: Backend::Auto,
device: Some(device.clone()),
..Default::default()
},
);
match result {
Expand Down Expand Up @@ -53,6 +54,7 @@ fn explicit_backend_must_match_device() {
&ModelOptions {
backend: Backend::Cpu,
device: Some(gpu),
..Default::default()
},
)
.unwrap_err();
Expand Down
7 changes: 7 additions & 0 deletions bindings/swift/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,13 @@ Request a backend policy with `ModelOptions(backend:)`; probe availability with
`Transcribe.devices()` to `ModelOptions(device:)`; exact selection never falls
back to another primary device.

On the CPU and on Metal, model weights are memory-mapped from the file: they
are served from the page cache, which the OS can reclaim, and do not count
toward the app's memory footprint (the figure jetsam enforces on iOS). Don't
truncate or rewrite a model file in place while it is loaded; write a new file
and rename it over the old one. `ModelOptions(noMmap: true)` copies the weights
instead.

## Concurrency and lifetime

- `Model` is shareable. In 0.x, compute is serialized per model, so concurrent
Expand Down
2 changes: 1 addition & 1 deletion bindings/swift/Sources/TranscribeCpp/ABIHash.swift
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import CTranscribe
extension Transcribe {
/// sha256/16 of the normalized public FFI surface, pinned to the value in
/// include/transcribe.abihash at the time this binding was last reviewed.
public static let pinnedHeaderHash = "98540564cd329b85"
public static let pinnedHeaderHash = "f99ad4169374b8f3"

/// The public-ABI digest this binding was reviewed against (16 hex chars).
public static func headerHash() -> String { pinnedHeaderHash }
Expand Down
1 change: 1 addition & 0 deletions bindings/swift/Sources/TranscribeCpp/Model.swift
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ public final class Model: @unchecked Sendable {
transcribe_model_load_params_init(&params)
params.backend = options.backend.cValue
params.device = options.device?.handle
params.no_mmap = options.noMmap
var out: OpaquePointer?
let status = transcribe_model_load_file(path, &params, &out)
try TranscribeError.check(status, context: "loading \(path)")
Expand Down
10 changes: 9 additions & 1 deletion bindings/swift/Sources/TranscribeCpp/Options.swift
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,17 @@ public struct ModelOptions: Sendable {
public var backend: Backend
/// Exact process-local device. `nil` applies the backend's automatic policy.
public var device: Device?
public init(backend: Backend = .auto, device: Device? = nil) {
/// Copy the weights into memory instead of memory-mapping the model file.
/// Default `false`: on the CPU and on Metal the weights are served from the
/// file's page cache, which the OS can reclaim and which does not count
/// toward the app's memory footprint. Don't truncate or rewrite a mapped
/// model file in place; write a new file and rename it over the old one.
/// The `TRANSCRIBE_NO_MMAP=1` environment variable forces `true`.
public var noMmap: Bool
public init(backend: Backend = .auto, device: Device? = nil, noMmap: Bool = false) {
self.backend = backend
self.device = device
self.noMmap = noMmap
}
}

Expand Down
7 changes: 7 additions & 0 deletions bindings/typescript/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,13 @@ const exact = await TranscribeModel.load("model.gguf", { device: cpu });
loader/driver degrades silently to CPU; an explicit backend that cannot be
satisfied raises a `BackendError`.

Where the device can use host memory (CPU, Metal on Apple silicon), weights are
memory-mapped from the model file and served from the page cache, which the OS
can reclaim, instead of being copied into process memory. Don't truncate or
rewrite a model file in place while it is loaded; write a new file and rename
it over the old one. `{ noMmap: true }` (or `TRANSCRIBE_NO_MMAP=1`) copies the
weights instead.

## Thread-safety

Your JavaScript runs on one thread, but `run`/`feed`/`finalize` dispatch the
Expand Down
Loading
Loading