Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ of tests.
| `TRANSCRIBE_CONV_DIRECT_DW` / `TRANSCRIBE_CONV_NO_DIRECT_DW` | Force the depthwise-conv dispatch to the direct `conv_2d_dw` path / the im2col path, overriding the per-family backend default. |
| `TRANSCRIBE_CONV_DIRECT_PW` / `TRANSCRIBE_CONV_NO_DIRECT_PW` | Force the pointwise-conv dispatch to direct `mul_mat` / im2col, overriding the backend default. |
| `TRANSCRIBE_NO_REPETITION_GUARD` | Turn off the stop for greedy decodes that start repeating themselves, and the trim of a repeating tail at a budget stop (see [`input-limits.md`](input-limits.md)). For byte-exact reference parity; a looping decode then runs to its budget and keeps its repeats. Read once per process. |
| `TRANSCRIBE_VOCABULARY_BOOST=<x>` | Logit bonus per matched vocabulary token for the parakeet transducer decoders (default 8.0; see [`prompting.md`](prompting.md)). For tuning against a term-recall set; `0` keeps the automaton but adds nothing. |
| `TRANSCRIBE_DUMP_DIR=<dir>` | Enable the per-stage tensor dumper; writes `<name>.f32` + `<name>.json` per dumped tensor into `<dir>`. The basis for the numerical-comparison harness (`scripts/compare_tensors.py`). |
| `TRANSCRIBE_PERF_DEBUG` | Print a per-stage timing breakdown to stderr (DEBUG log) on the families that profile (`cohere`, `granite`, `canary`, `canary_qwen`, `moonshine`, `moonshine_streaming`, `moss`, `qwen3_asr`, `whisper`). For whisper, a value containing `cpu` or `all` additionally prints the CPU sub-section breakdown. |
| `TRANSCRIBE_VOXTRAL_REALTIME_STREAM_TIMING` | Print a per-component streaming wall-time breakdown at stream finalize (voxtral_realtime). |
Expand Down
18 changes: 18 additions & 0 deletions docs/prompting.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,28 @@ Probe `transcribe_model_supports()` for the matching feature bit.
| Canary 180m-flash / 1b-flash / 1b-v2 | | | | yes |
| Fun-ASR-Nano | yes | | | |
| MOSS-Transcribe-Diarize | yes | | | |
| Parakeet (TDT and RNN-T heads) | yes | | | |

Per-model formats and restrictions are in each model doc's **Prompting:**
note under [`models/`](models/).

## Parakeet vocabulary

Parakeet has no prompt slot, so its transducer decoders take vocabulary terms
as phrase boosting: each term is tokenized the way the model writes it, starting
at a word boundary, and the terms form an Aho-Corasick automaton over token
ids. The model alone decides between blank and a token on every greedy step.
When it emits a token, the vocabulary re-ranks the non-blank tokens. Once the
model has begun a term on its own, each further piece that extends the match
earns a logit bonus (8.0 by default, chosen on a held-out term set), a token
that breaks the match gives back what the unfinished match earned, and a
completed term keeps its bonus. Biasing therefore changes which word is
written, never whether one is, so it cannot invent speech on silence. The
reported token confidences stay the model's own, and with no terms decoding is
bit-exact. Terms with characters the tokenizer cannot write are skipped with a
WARN. Offline runs and batches take terms; streaming ignores them, and CTC
heads do not take them yet.

## Limits

- Over the prompt budget, `vocabulary` drops terms from the end of the list and
Expand Down
1 change: 1 addition & 0 deletions src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ add_library(transcribe
transcribe-model.cpp
transcribe-tokenizer.cpp
transcribe-prompting.cpp
transcribe-phrase-boost.cpp
transcribe-unicode.cpp
transcribe-unicode-data.cpp
transcribe-debug.cpp
Expand Down
70 changes: 56 additions & 14 deletions src/arch/parakeet/decoder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@

#include "decoder.h"

#include <limits>

#include "parakeet.h"
#include "transcribe-batch-util.h"
#include "transcribe-debug.h"
Expand Down Expand Up @@ -966,6 +968,31 @@ int argmax_range(const float * data, int n) {
return best_i;
}

// Token argmax, biased toward the vocabulary when there is one.
//
// The model alone decides between blank and a token: only when its own
// argmax emits something does the vocabulary re-rank the non-blank tokens.
// Biasing can then change which word is written but never whether one is,
// so it cannot invent speech on silence or noise. The model's own logits are
// left untouched for the confidence that follows.
static int biased_argmax(const float * token_logits,
int n_token_cls,
int blank_id,
const transcribe::PhraseBoost * boost,
int boost_state,
std::vector<float> & scratch) {
const int unbiased = argmax_range(token_logits, n_token_cls);
if (boost == nullptr || boost->empty() || unbiased == blank_id) {
return unbiased;
}
scratch.assign(token_logits, token_logits + n_token_cls);
boost->apply(boost_state, scratch.data(), n_token_cls, blank_id);
if (blank_id >= 0 && blank_id < n_token_cls) {
scratch[static_cast<size_t>(blank_id)] = -std::numeric_limits<float>::infinity();
}
return argmax_range(scratch.data(), n_token_cls);
}

// Entropy-based confidence over a token-logit slice. Mirrors the
// reference ParakeetTDT.decode_greedy path:
//
Expand Down Expand Up @@ -1015,12 +1042,13 @@ float token_confidence(const float * token_logits, int n_token_classes, std::vec
// Greedy decode driver
// ---------------------------------------------------------------------------

transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens) {
transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost) {
if (enc_out == nullptr || T_enc <= 0 || d_enc <= 0) {
return TRANSCRIBE_ERR_INVALID_ARG;
}
Expand Down Expand Up @@ -1097,6 +1125,10 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
// the unconditional path.
bool predictor_dirty = true;

// Vocabulary biasing state; untouched when no terms were given.
int boost_state = transcribe::PhraseBoost::root;
std::vector<float> boosted;

// See ParakeetHParams::tdt_global_symbol_budget.
const bool global_budget = w.tdt_global_symbol_budget;
int64_t steps_remaining = static_cast<int64_t>(w.tdt_max_symbols) * T_enc;
Expand Down Expand Up @@ -1131,7 +1163,7 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * token_logits = logits.data();
const float * duration_logits = logits.data() + n_token_cls;

const int pred_token = argmax_range(token_logits, n_token_cls);
const int pred_token = biased_argmax(token_logits, n_token_cls, blank_id, boost, boost_state, boosted);
const int decision = argmax_range(duration_logits, n_dur);
const int duration = w.tdt_durations[static_cast<size_t>(decision)];

Expand Down Expand Up @@ -1172,6 +1204,9 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
last_token = pred_token;
std::swap(state, next_state); // commit
predictor_dirty = true;
if (boost != nullptr) {
boost_state = boost->next(boost_state, pred_token);
}
}

if (global_budget) {
Expand Down Expand Up @@ -1249,12 +1284,13 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
// duration). Matches the reference dump points (`dec.embed.0`,
// `dec.lstm.{layer}.{h,c}.0`, `dec.joint.0`) emitted on iter 1.

transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens) {
transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost) {
if (enc_out == nullptr || T_enc <= 0 || d_enc <= 0) {
return TRANSCRIBE_ERR_INVALID_ARG;
}
Expand Down Expand Up @@ -1316,6 +1352,9 @@ transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
// RNN-T iters/token ratio.
bool predictor_dirty = true;

int boost_state = transcribe::PhraseBoost::root;
std::vector<float> boosted;

while (step < T_enc && iter < max_iters) {
++iter;

Expand All @@ -1342,7 +1381,7 @@ transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,

// RNNT joint output is just `n_token_cls` floats (no duration extras).
const float * token_logits = logits.data();
const int pred_token = argmax_range(token_logits, n_token_cls);
const int pred_token = biased_argmax(token_logits, n_token_cls, blank_id, boost, boost_state, boosted);

if (iter == 1 && transcribe::debug::enabled()) {
const long long s_h = H;
Expand Down Expand Up @@ -1377,6 +1416,9 @@ transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
last_token = pred_token;
std::swap(state, next_state);
predictor_dirty = true;
if (boost != nullptr) {
boost_state = boost->next(boost_state, pred_token);
}

new_symbols += 1;
if (w.tdt_max_symbols > 0 && new_symbols >= w.tdt_max_symbols) {
Expand Down
31 changes: 19 additions & 12 deletions src/arch/parakeet/decoder.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
#include "ggml-backend.h"
#include "ggml.h"
#include "transcribe.h"
#include "transcribe-phrase-boost.h"

#include <cstdint>
#include <string>
Expand Down Expand Up @@ -218,24 +219,30 @@ struct TdtToken {
//
// Returns TRANSCRIBE_OK on success; cannot fail except via invalid args
// (validated by the family driver before this is called).
transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens);
//
// `boost`, when non-null and non-empty, biases the token argmax toward the
// caller's vocabulary terms (transcribe_run_params::vocabulary); see
// transcribe-phrase-boost.h. Token confidences stay the model's own.
transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost = nullptr);

// Run RNNT greedy decode end-to-end. Same predictor + joint code as TDT,
// but the joint emits `vocab+1` logits (no duration extras) and the step
// rule is "blank → advance one frame, non-blank → emit + stay, capped by
// tdt_max_symbols". Per-emit duration_frames is fixed at 1. Same I/O
// contract and TdtToken result type as decode_tdt_greedy.
transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens);
transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost = nullptr);

// Streaming variant of RNN-T greedy decode. Consumes T_enc_new encoder
// frames (the chunk just produced) and APPENDS emitted tokens to
Expand Down
62 changes: 60 additions & 2 deletions src/arch/parakeet/model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
#include "transcribe-log.h"
#include "transcribe-mel.h"
#include "transcribe-meta.h"
#include "transcribe-prompting.h"
#include "transcribe/parakeet.h"
#include "weights.h"

Expand All @@ -40,6 +41,7 @@
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <ios>
Expand Down Expand Up @@ -587,6 +589,10 @@ transcribe_status load(Loader & loader, const transcribe_model_load_params * par
return st;
}

// Vocabulary terms bias the greedy transducer decoders (TDT and RNN-T);
// CTC heads do not take them yet.
transcribe::set_feature(m.get(), TRANSCRIBE_FEATURE_VOCABULARY, m->host_decoder.head_kind != HostHeadKind::CTC);

// Multitalker bundle post-load: fuse the diarizer conformer's BN (needs
// uploaded data), build its ceil-framed mel frontend, and publish the
// DIARIZATION capability — for a bundle it is a property of the file.
Expand Down Expand Up @@ -789,6 +795,56 @@ static double parakeet_ms_per_enc_frame(const ParakeetHParams & hp) {
static_cast<double>(hp.fe_sample_rate);
}

// Logit bonus per matched vocabulary token, in natural-log units. Chosen on
// the dev half of a term-recall set and checked against general WER; see
// docs/prompting.md. TRANSCRIBE_VOCABULARY_BOOST overrides it for research.
constexpr float kVocabularyBoost = 8.0f;

static float vocabulary_boost_score() {
if (const char * v = transcribe::env::str("TRANSCRIBE_VOCABULARY_BOOST")) {
char * end = nullptr;
const float value = std::strtof(v, &end);
if (end != v && value >= 0.0f) {
return value;
}
}
return kVocabularyBoost;
}

// The phrase-boost automaton for this run's vocabulary, or nullptr when the
// run has none. Terms are tokenized as the model writes them (with the word
// boundary marker), so a term matches only from the start of a word.
static const transcribe::PhraseBoost * vocabulary_boost(ParakeetSession * pc,
const ParakeetModel * pm,
const transcribe_run_params * params) {
std::vector<std::string> terms = transcribe::prompting::terms(params);
if (terms.empty() || pm->host_decoder.head_kind == HostHeadKind::CTC) {
return nullptr;
}
if (terms != pc->boost_terms || pc->boost.empty()) {
std::vector<std::vector<int32_t>> phrases;
for (const std::string & term : terms) {
std::vector<int32_t> ids;
if (pm->tok.encode_sentencepiece_bpe(term, ids, 0, -1, /*remove_extra_whitespaces=*/true) != TRANSCRIBE_OK) {
log_msg(TRANSCRIBE_LOG_LEVEL_WARN, "parakeet: this tokenizer cannot encode vocabulary terms; ignoring them");
return nullptr;
}
bool unknown = false;
for (const int32_t id : ids) {
unknown = unknown || pm->tok.is_unknown(id);
}
if (unknown) {
log_msg(TRANSCRIBE_LOG_LEVEL_WARN, "parakeet: a vocabulary term has characters the model cannot write; skipped");
continue;
}
phrases.push_back(std::move(ids));
}
pc->boost.build(phrases, vocabulary_boost_score());
pc->boost_terms = std::move(terms);
}
return pc->boost.empty() ? nullptr : &pc->boost;
}

// Host-side TDT/RNNT/CTC decode + public result-hierarchy build for one
// utterance's encoder output. `enc` is the row-major [T_enc, d_enc]
// activation for ONE utterance; utt_index >= 0 tags the per-utterance
Expand Down Expand Up @@ -827,10 +883,12 @@ static transcribe_status decode_and_populate(ParakeetSession * pc,
transcribe_status st = TRANSCRIBE_OK;
switch (pm->host_decoder.head_kind) {
case HostHeadKind::TDT:
st = decode_tdt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens);
st = decode_tdt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens,
vocabulary_boost(pc, pm, params));
break;
case HostHeadKind::RNNT:
st = decode_rnnt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens);
st = decode_rnnt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens,
vocabulary_boost(pc, pm, params));
break;
case HostHeadKind::CTC:
st = decode_ctc_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens);
Expand Down
3 changes: 3 additions & 0 deletions src/arch/parakeet/parakeet.h
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,9 @@ struct ParakeetSession final : public transcribe_session {
std::vector<float> pos_div_term;
std::vector<float> enc_host;
std::vector<TdtToken> raw_tokens;
// Vocabulary biasing, rebuilt only when the run's terms change.
std::vector<std::string> boost_terms;
transcribe::PhraseBoost boost;

// Multitalker bundle scratch: streaming-diarizer AOSC/FIFO state for
// the embedded sortformer forward (diarize=ON runs only).
Expand Down
Loading