Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,15 @@ loaded; write a new file and rename it over the old one. `--no-mmap`
(`transcribe_model_load_params::no_mmap`, or `TRANSCRIBE_NO_MMAP=1`) copies the
weights instead.

Models whose FFN gate and up weights are stored back to back (every packed
variant: granite, canary-qwen, qwen3-asr, voxtral, voxtral-realtime, moss,
funasr-nano) are fused at load time by pointing one tensor at those bytes
instead of packing a second copy, so the fusion usually costs no extra weight
memory. It falls back to the copying path when the device layout does not allow
that (tensor storage padded or the mapping split, e.g. a file larger than the
device's max buffer size). `TRANSCRIBE_NO_GATE_UP_ALIAS=1` forces the copying
path.

## Bindings

Official bindings wrap the C API for other languages:
Expand Down
1 change: 1 addition & 0 deletions docs/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ of tests.
| --- | --- |
| `TRANSCRIBE_BACKENDS=<list>` | Restrict which backends may register: a comma-separated list of `cpu`, `metal`, `vulkan`, `cuda`, `rocm`, `other`, `all` (case-insensitive). An excluded backend never runs any code (no module load, no driver init). It can only narrow the mask a host passes to `transcribe_init_backends_ex()`, applies even if the host never calls it, and CPU is always kept. Unknown names are dropped and logged as an error naming what is still allowed, so a typo narrows rather than widens (`vulcan` means CPU-only). Unset or empty means `all`. Read once per process. |
| `TRANSCRIBE_NO_MMAP` | Copy model weights into memory instead of memory-mapping the model file, for every model load in the process (same as `transcribe_model_load_params::no_mmap`, CLI `--no-mmap`). Read at each load. |
| `TRANSCRIBE_NO_GATE_UP_ALIAS` | Disable the load-time gate+up fusion shortcut that points one packed FFN weight tensor at the existing adjacent gate and up weights instead of copying them into a new buffer (granite, canary-qwen, qwen3-asr, voxtral, voxtral-realtime, moss, funasr-nano). Forces the copying path, i.e. the pre-mmapping memory cost. Read at each load; for A/B measurement only. |
| `TRANSCRIBE_NO_FLASH` | Disable flash attention on encoder and decoder (forces the manual F32 path). |
| `TRANSCRIBE_FORCE_FLASH` | Force flash attention on. Wins over `TRANSCRIBE_NO_FLASH` if both are set. |
| `TRANSCRIBE_CONV_DIRECT_DW` / `TRANSCRIBE_CONV_NO_DIRECT_DW` | Force the depthwise-conv dispatch to the direct `conv_2d_dw` path / the im2col path, overriding the per-family backend default. |
Expand Down
195 changes: 186 additions & 9 deletions src/causal_lm/causal_lm.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <string>
#include <vector>

namespace transcribe::causal_lm {
Expand Down Expand Up @@ -836,6 +837,132 @@ void PackedGateUpHandles::free() {
}
}

namespace {

// Why a block cannot alias. Reported at INFO when a load falls back, because a
// silently lost memory win is exactly the regression this path can suffer from
// without a wrong answer anywhere.
enum class AliasSkip {
None,
DisabledByEnv,
NotInOneBuffer,
IsAView,
NotContiguous,
NotAdjacent,
SplitMapping,
PaddedStorage,
};

const char * alias_skip_name(AliasSkip why) {
switch (why) {
case AliasSkip::None:
return "aliased";
case AliasSkip::DisabledByEnv:
return "disabled by TRANSCRIBE_NO_GATE_UP_ALIAS";
case AliasSkip::NotInOneBuffer:
return "gate/up in different buffers";
case AliasSkip::IsAView:
return "gate/up is a view";
case AliasSkip::NotContiguous:
return "gate/up not contiguous";
case AliasSkip::NotAdjacent:
return "up does not follow gate";
case AliasSkip::SplitMapping:
return "mapping split into several views";
case AliasSkip::PaddedStorage:
return "padded tensor storage";
}
return "unknown";
}

// Whether a block's gate/up bytes are already in packed layout: gate_w's bytes
// immediately followed by up_w's inside one backend buffer, so a
// [hidden, 2*intermediate] tensor starting at gate_w->data is the fused tensor.
//
// That is the normal case. The converter writes `ffn.up.weight` straight after
// `ffn.gate.weight` (verified for granite 4.0/4.1/plus, canary-qwen, qwen3-asr,
// voxtral, voxtral-realtime, moss and funasr-nano, all quant types), and the
// loader keeps tensors either in that same order (a copied buffer) or in place
// (load_common's read-only mmap). Aliasing instead of copying means the packed
// path costs no weight memory at all: under mmap the fused tensor is the very
// pages the gate/up tensors already are.
//
// Deliberately conservative. Anything unexpected falls back to the copying
// path, which is what every model got before this existed, and says why.
AliasSkip gate_up_alias_skip(const GateUpEntry & e, size_t fused_bytes) {
ggml_backend_buffer_t buffer = e.gate_w->buffer;
if (buffer == nullptr || buffer != e.up_w->buffer) {
return AliasSkip::NotInOneBuffer;
}
if (e.gate_w->data == nullptr || e.up_w->data == nullptr) {
return AliasSkip::NotInOneBuffer;
}
if (e.gate_w->view_src != nullptr || e.up_w->view_src != nullptr) {
return AliasSkip::IsAView;
}
if (!ggml_is_contiguous(e.gate_w) || !ggml_is_contiguous(e.up_w)) {
return AliasSkip::NotContiguous;
}
if (static_cast<const char *>(e.gate_w->data) + ggml_nbytes(e.gate_w) != e.up_w->data) {
return AliasSkip::NotAdjacent;
}

// A device may keep several views of one buffer (Metal splits a mapping
// larger than its max buffer size into overlapping views, after rounding the
// length up to the page size) and a tensor has to sit inside a single one.
// The generic API does not expose view bounds, so only alias a buffer the
// device can hold as one view. Splitting rather than aliasing costs a copy;
// aliasing across a boundary would hand the kernel the wrong buffer, so the
// margin is deliberately generous.
constexpr size_t k_max_page_size = 16384; // largest page size we target
ggml_backend_buffer_type_t buft = ggml_backend_buffer_get_type(buffer);
const size_t max_buffer_size = ggml_backend_buft_get_max_size(buft);
if (max_buffer_size != 0 && ggml_backend_buffer_get_size(buffer) + k_max_page_size > max_buffer_size) {
return AliasSkip::SplitMapping;
}

// Some devices pad or reshape a tensor's storage (see load_common's mmap
// gating); the fused tensor is only valid if it occupies exactly its bytes.
if (ggml_backend_buffer_get_alloc_size(buffer, *e.gate_up_w_out) != fused_bytes) {
return AliasSkip::PaddedStorage;
}
return AliasSkip::None;
}

// One line per load, at INFO: the aliased count, what it saved, and - when some
// blocks were copied - why, so a lost win is visible without a debug build.
void log_alias_summary(const char * error_tag,
size_t n_aliased,
size_t n_entries,
size_t aliased_bytes,
const size_t (&skipped)[static_cast<size_t>(AliasSkip::PaddedStorage) + 1]) {
std::string why;
if (n_aliased < n_entries) {
for (size_t i = 1; i <= static_cast<size_t>(AliasSkip::PaddedStorage); ++i) {
if (skipped[i] == 0) {
continue;
}
if (!why.empty()) {
why += ", ";
}
char buf[96];
std::snprintf(buf, sizeof(buf), "%s x%zu", alias_skip_name(static_cast<AliasSkip>(i)), skipped[i]);
why += buf;
}
}

if (n_aliased == 0) {
log_msg(TRANSCRIBE_LOG_LEVEL_INFO, "%s: gate+up packed by copying (%zu blocks; %s)", error_tag, n_entries,
why.c_str());
return;
}
log_msg(TRANSCRIBE_LOG_LEVEL_INFO, "%s: gate+up aliased for %zu/%zu blocks (%.1f MiB not duplicated)%s%s",
error_tag, n_aliased, n_entries, static_cast<double>(aliased_bytes) / (1024.0 * 1024.0),
why.empty() ? "" : "; copied ", why.c_str());
}

} // namespace

transcribe_status pack_gate_up(ggml_backend_t backend,
int hidden,
int intermediate,
Expand Down Expand Up @@ -878,9 +1005,61 @@ transcribe_status pack_gate_up(ggml_backend_t backend,
log_msg(TRANSCRIBE_LOG_LEVEL_ERROR, "%s: pack_gate_up new_tensor_2d failed at %zu", error_tag, i);
return TRANSCRIBE_ERR_OOM;
}
// The fused tensor is the byte-concat of the two, so their bytes must add
// up to it. Each family's weights.cpp pins gate/up to
// [hidden, intermediate] with its own GET_LIN(...) shape check, which is
// what makes this true; verifying it here, before the alias/copy split,
// is what lets the aliased path trust the fused tensor to cover exactly
// those two and nothing else.
const size_t gate_bytes = ggml_nbytes(e.gate_w);
const size_t up_bytes = ggml_nbytes(e.up_w);
if (ggml_nbytes(t) != gate_bytes + up_bytes) {
log_msg(TRANSCRIBE_LOG_LEVEL_ERROR, "%s: pack_gate_up entry %zu size mismatch (%zu vs %zu + %zu)",
error_tag, i, ggml_nbytes(t), gate_bytes, up_bytes);
return TRANSCRIBE_ERR_GGUF;
}
*e.gate_up_w_out = t;
}

// Blocks whose weights are already laid out as gate||up point at those
// bytes instead of duplicating them. The rest are copied into a buffer of
// our own, as before.
constexpr size_t k_n_skip_reasons = static_cast<size_t>(AliasSkip::PaddedStorage) + 1;
const bool alias_allowed = !env::flag("TRANSCRIBE_NO_GATE_UP_ALIAS");
std::vector<size_t> copied;
size_t n_aliased = 0;
size_t aliased_bytes = 0;
size_t skipped[k_n_skip_reasons] = {};
copied.reserve(entries.size());

for (size_t i = 0; i < entries.size(); ++i) {
const GateUpEntry & e = entries[i];
ggml_tensor * gate_up = *e.gate_up_w_out;
const size_t fused_size = ggml_nbytes(gate_up);
const AliasSkip skip = alias_allowed ? gate_up_alias_skip(e, fused_size) : AliasSkip::DisabledByEnv;
if (skip != AliasSkip::None) {
++skipped[static_cast<size_t>(skip)];
copied.push_back(i);
continue;
}
// Same shape, dtype and byte layout as the copied tensor; only the
// storage differs. gate_w->data is inside gate_w->buffer, which ggml's
// own bounds assert in ggml_backend_tensor_alloc checks.
if (ggml_backend_tensor_alloc(e.gate_w->buffer, gate_up, e.gate_w->data) != GGML_STATUS_SUCCESS) {
log_msg(TRANSCRIBE_LOG_LEVEL_ERROR, "%s: pack_gate_up could not alias gate/up at block %zu", error_tag, i);
return TRANSCRIBE_ERR_BACKEND;
}
++n_aliased;
aliased_bytes += fused_size;
}

if (copied.empty()) {
log_alias_summary(error_tag, n_aliased, entries.size(), aliased_bytes, skipped);
return TRANSCRIBE_OK;
}

// alloc_ctx_tensors skips tensors that already have storage (the aliased
// ones) and returns NULL when none are left, hence the early return above.
out_handles.buffer = ggml_backend_alloc_ctx_tensors(out_handles.ctx, backend);
if (out_handles.buffer == nullptr) {
log_msg(TRANSCRIBE_LOG_LEVEL_ERROR, "%s: pack_gate_up backend buffer alloc failed", error_tag);
Expand All @@ -891,21 +1070,19 @@ transcribe_status pack_gate_up(ggml_backend_t backend,
// Fill: concat-along-dim-1 is gate's bytes followed by up's bytes
// (one CPU round-trip per block at load time).
std::vector<uint8_t> buf;
for (const auto & e : entries) {
ggml_tensor * gate_up = *e.gate_up_w_out;
const size_t gate_bytes = ggml_nbytes(e.gate_w);
const size_t up_bytes = ggml_nbytes(e.up_w);
if (ggml_nbytes(gate_up) != gate_bytes + up_bytes) {
log_msg(TRANSCRIBE_LOG_LEVEL_ERROR, "%s: pack_gate_up size mismatch (%zu vs %zu + %zu)", error_tag,
ggml_nbytes(gate_up), gate_bytes, up_bytes);
return TRANSCRIBE_ERR_GGUF;
}
for (const size_t i : copied) {
const GateUpEntry & e = entries[i];
ggml_tensor * gate_up = *e.gate_up_w_out;
const size_t gate_bytes = ggml_nbytes(e.gate_w);
const size_t up_bytes = ggml_nbytes(e.up_w);
buf.resize(std::max(gate_bytes, up_bytes));
ggml_backend_tensor_get(e.gate_w, buf.data(), 0, gate_bytes);
ggml_backend_tensor_set(gate_up, buf.data(), 0, gate_bytes);
ggml_backend_tensor_get(e.up_w, buf.data(), 0, up_bytes);
ggml_backend_tensor_set(gate_up, buf.data(), gate_bytes, up_bytes);
}

log_alias_summary(error_tag, n_aliased, entries.size(), aliased_bytes, skipped);
return TRANSCRIBE_OK;
}

Expand Down
25 changes: 18 additions & 7 deletions src/causal_lm/causal_lm.h
Original file line number Diff line number Diff line change
Expand Up @@ -294,13 +294,24 @@ struct PackedGateUpHandles {
void free();
};

// Allocate a packed context + a [hidden, 2·intermediate] tensor per block
// on `backend`, then copy gate's bytes followed by up's bytes into each.
// Compatible with row-wise quants (Q4/Q5/Q6/Q8) because concat-along-dim-1
// is byte-concat for those types. Writes `*entries[i].gate_up_w_out` to the
// new tensor and marks the buffer GGML_BACKEND_BUFFER_USAGE_WEIGHTS.
// Returns TRANSCRIBE_ERR_OOM on allocation failure, TRANSCRIBE_ERR_GGUF on a
// gate/up shape or type mismatch; `out_handles` is left in a state safe to free.
// Allocate a packed context + a [hidden, 2·intermediate] tensor per block,
// then give each one the bytes of gate followed by up. Compatible with
// row-wise quants (Q4/Q5/Q6/Q8) because concat-along-dim-1 is byte-concat for
// those types.
//
// Where the block's gate and up tensors are already adjacent in one backend
// buffer (the layout the converter writes, and the one both the copying and
// the mmap load paths preserve), the packed tensor points straight at gate's
// bytes and nothing is copied or duplicated. `out_handles.buffer` then stays
// null, and the packed tensors' storage belongs to the weights buffer: they
// must not outlive it. Otherwise the bytes are copied into a buffer owned by
// `out_handles`, as before. `TRANSCRIBE_NO_GATE_UP_ALIAS` forces the copy path.
//
// Writes `*entries[i].gate_up_w_out` to the new tensor; a buffer created here
// is marked GGML_BACKEND_BUFFER_USAGE_WEIGHTS. Returns TRANSCRIBE_ERR_OOM on
// allocation failure, TRANSCRIBE_ERR_GGUF on a gate/up shape or type mismatch,
// TRANSCRIBE_ERR_BACKEND if a device rejects an aliased tensor;
// `out_handles` is left in a state safe to free.
transcribe_status pack_gate_up(ggml_backend_t backend,
int hidden,
int intermediate,
Expand Down
17 changes: 17 additions & 0 deletions tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,23 @@ transcribe_apply_warnings(transcribe_conv_pw_promote_unit)
add_test(NAME transcribe_conv_pw_promote_unit
COMMAND transcribe_conv_pw_promote_unit)

# -----------------------------------------------------------------------------
# causal_lm::pack_gate_up: alias / copy / mismatch, no model needed
# -----------------------------------------------------------------------------

add_executable(transcribe_pack_gate_up_unit
pack_gate_up_unit.cpp)

target_link_libraries(transcribe_pack_gate_up_unit PRIVATE transcribe ggml)

target_include_directories(transcribe_pack_gate_up_unit PRIVATE
${CMAKE_SOURCE_DIR}/src)

transcribe_apply_warnings(transcribe_pack_gate_up_unit)

add_test(NAME transcribe_pack_gate_up_unit
COMMAND transcribe_pack_gate_up_unit)

# -----------------------------------------------------------------------------
# Dispatcher-level transcribe_run behavior
# -----------------------------------------------------------------------------
Expand Down
Loading
Loading