Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude/skills/porting-2-oracle/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ reference WER, and provisional tolerances.
creates a skeleton following the contract in `scripts/lib/ref_dump.py`
(or in one of the existing dumpers).
- A committed reference checkpoint is available to the reference framework
(HF download or `$TRANSCRIBE_MODELS_DIR/<slug>/`).
(HF download, `$TRANSCRIBE_MODELS_DIR/<slug>/`, or
`$TRANSCRIBE_MODELS_DIR/<slug>@<commit>/` for a revision pinned to a commit hash).
- `uv` is installed.

## Oracle packet contents
Expand Down
6 changes: 5 additions & 1 deletion .claude/skills/porting-4-cpp/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,11 @@ each entry with explicit justification.** Recipe:

- `1.5 × observed < provisional`: C++ drift is well below the magnitude
budget — common on simple tensors. Keep the magnitude budget;
tighten only with user confirmation.
tighten only with user confirmation. A variant added to a family whose
finalized tolerance file already tightened to `max(1.5 × observed,
1e-6)` (such as `tests/tolerances/gigaam.json`) takes that recipe, so
the family's entries stay comparable; its `_comment` names the file
whose recipe it follows.
- `1.5 × observed ≈ provisional`: C++ within budget. Preserve provisional.
- `1.5 × observed > provisional`: C++ drift exceeds the magnitude
budget. Record the widening factor in `_comment` and name the
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/python-bindings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ on:
- "include/**"
- "src/**"
- "scripts/lib/**"
- "scripts/convert-gigaam.py"
- "scripts/dump_reference_gigaam_author.py"
- "scripts/test_convert_gigaam.py"
- "tests/**"
- "ggml/**"
- "CMakeLists.txt"
Expand All @@ -38,6 +41,9 @@ on:
- "include/**"
- "src/**"
- "scripts/lib/**"
- "scripts/convert-gigaam.py"
- "scripts/dump_reference_gigaam_author.py"
- "scripts/test_convert_gigaam.py"
- "tests/**"
- "ggml/**"
- "CMakeLists.txt"
Expand Down Expand Up @@ -82,6 +88,12 @@ jobs:
run: uv run --no-project bindings/python/_generate/check_version_sync.py
- name: GGUF writer layout
run: uv run scripts/lib/test_gguf_writer.py
- name: Hugging Face snapshot commit check
run: uv run scripts/lib/test_hf_source.py
- name: GigaAM variant sources and loader
run: uv run scripts/lib/test_gigaam_hf.py
- name: GigaAM converter profiles
run: uv run scripts/test_convert_gigaam.py

python-shared:
runs-on: blacksmith-2vcpu-ubuntu-2404
Expand Down
108 changes: 106 additions & 2 deletions docs/porting/families/gigaam.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,19 @@ ASR variants (`gigaam-v3-e2e-rnnt`, `gigaam-v3-rnnt`, `gigaam-v3-ctc`,
`gigaam-v3-e2e-ctc`) ready for HF upload. `v3_ssl` (encoder-only HuBERT
pretraining checkpoint) intentionally out of scope.

`gigaam-multilingual-large-ctc` (ai-sage/GigaAM-Multilingual, branch
`large_ctc`) converts and validates at F32 on the existing runtime with no
C++ change; WER, benchmarks, and publication are not done. See
[GigaAM Multilingual Large CTC](#gigaam-multilingual-large-ctc).

## Identity

- Family key: `gigaam`
- Upstream architecture string: `gigaam` (HF `model_type`); paper name "GigaAM" (Sber/ai-sage, InterSpeech 2025, arXiv:2506.01192)
- Hugging Face repo: `ai-sage/GigaAM-v3` (five sibling branches upstream; four ported)
- Also converted, not yet published: `gigaam-multilingual-large-ctc` from `ai-sage/GigaAM-Multilingual`, see [GigaAM Multilingual Large CTC](#gigaam-multilingual-large-ctc)
- License: MIT
- Variants in scope (one HF branch each):
- v3 variants in scope (one HF branch each):

| Variant | HF branch | Head | num_classes | Vocab style |
|---|---|---|---|---|
Expand All @@ -33,7 +39,7 @@ pretraining checkpoint) intentionally out of scope.
encoder-output emission CLI, so `v3_ssl` has no user-observable
capability here and is intentionally **out of scope** for this port.

Canonical bench samples: `ru` (model is Russian-only; English `jfk`/`dots` would decode to garbage).
Canonical bench samples for the v3 variants: `ru` (model is Russian-only; English `jfk`/`dots` would decode to garbage).

## References

Expand Down Expand Up @@ -220,3 +226,101 @@ them from the converter source.
The intake's hypothesis of shared encoder weights was wrong;
the converter emits a full GGUF per variant, not a shared encoder
GGUF with per-variant head shards.

## GigaAM Multilingual Large CTC

Variant `gigaam-multilingual-large-ctc` is the `large_ctc` branch of
[`ai-sage/GigaAM-Multilingual`](https://huggingface.co/ai-sage/GigaAM-Multilingual),
pinned at commit `3905cd51c3ed4e88c8edf33f3302969ba480a327` (the repo's
`main` branch is the smaller `ctc` model). It is the v3 encoder at 24
layers, d_model=1024, 16 heads, with a CTC head over 71 classes: a
70-entry charwise vocab (space, apostrophe, a-z, Russian а-я and ё, and
і ғ қ ң ү ұ һ ә ө) plus blank at id 70. Languages: ru, en, kk, ky, uz.
Offline only; there is no language input or language-detection output.

| Item | Path |
|---|---|
| Intake | `reports/porting/gigaam/gigaam-multilingual-large-ctc/intake.json` |
| Golden manifest | `tests/golden/gigaam/gigaam-multilingual-large-ctc.manifest.json` |
| Tolerances | `tests/tolerances/gigaam-multilingual-large-ctc.json` |
| Variant sources and snapshot loader | `scripts/lib/gigaam_hf.py` (tests: `scripts/lib/test_gigaam_hf.py`) |

### Commands

Conversion (writes `models/gigaam-multilingual-large-ctc/gigaam-multilingual-large-ctc-F32.gguf`,
2341378624 bytes, 824 tensors):

```bash
uv run --project scripts/envs/gigaam scripts/convert-gigaam.py ai-sage/GigaAM-Multilingual \
--repo-id gigaam-multilingual-large-ctc
```

The converter downloads the snapshot (2.3 GB) at the pinned commit into
`$TRANSCRIBE_MODELS_DIR/<slug>@<commit>/` or the Hugging Face cache, both
of which keep each commit in its own directory; with `HF_HUB_OFFLINE=1`
it uses that local copy without network access, and fails when the
pinned commit was never downloaded. Unlike converters that take
`lib.hf_source.resolve_model_dir`, it accepts no other repo id and no local directory, so `general.repo_url`
and the reference dumps' `hf_revision` name the source the weights came
from.

The repo and commit of every GigaAM variant are in `VARIANTS` of
`scripts/lib/gigaam_hf.py`, which the converter, the reference dumper,
and the WER runner read. `scripts/lib/test_gigaam_hf.py` fails when a
golden manifest names another source, and the dumper refuses the
`--revision` that `validate.py` passes from the manifest when it differs
from the pin. Change the pin and the manifest together.

Reference dumps and validation use the same pin:

```bash
uv run scripts/preflight.py --family gigaam --variant gigaam-multilingual-large-ctc --gate A
uv run scripts/preflight.py --family gigaam --variant gigaam-multilingual-large-ctc --gate B
uv run scripts/validate.py all --family gigaam --variant gigaam-multilingual-large-ctc
uv run scripts/quantize-all.py models/gigaam-multilingual-large-ctc/gigaam-multilingual-large-ctc-F32.gguf
TRANSCRIBE_GIGAAM_GGUF=$PWD/models/gigaam-multilingual-large-ctc/gigaam-multilingual-large-ctc-F16.gguf \
ctest --test-dir build -R gigaam
```

### Reference

The reference is the family's `gigaam` package pin (6e4b027c), as for v3.
That commit has no `load_model("multilingual_large_ctc")`, so
`scripts/lib/gigaam_hf.py` instantiates the package's `FeatureExtractor`,
`ConformerEncoder`, `CTCHead`, and `CTCGreedyDecoding` from the snapshot's
`config.json`, whose `_target_` entries name the same classes in
`modeling_gigaam`, and loads `pytorch_model.bin` with its `model.` key
prefix removed. The upstream commits from 6e4b027c to 7447938 change the
encoder only for padded batch inputs. The repo's own `modeling_gigaam.py`
on the same weights produces bit-identical `frontend.mel.out`, `enc.out`,
and `ctc.log_probs` on `samples/ru.wav`.

The v3 config lists `mel_scale: htk` and `mel_norm: null`; the
Multilingual config omits both. The package's `FeatureExtractor` passes
neither to torchaudio and defaults `center` to true, so the converter
reads the values the module sets (sample rate, FFT, window and hop
lengths, mel count, `center`, mel scale and norm, frequency range) from
it, `htk` with no norm for every variant, and refuses a config value that
disagrees. The rest of `stt.frontend.*` (periodic Hann window, no dither,
pre-emphasis, or normalization) is fixed by `FeatureExtractor`'s code.

### Capability Validation

| Capability | Mode | Command / test | Expected observable | Status |
|------------|------|----------------|---------------------|--------|
| Transcribe (ru, en, kk, ky, uz) | auto / no language hint | `build/bin/transcribe-cli -m <gguf> <wav>` | transcript equal to the reference's | PASS on `samples/ru.wav`, `samples/fleurs-ru.wav`, `samples/fleurs-en.wav`, and two FLEURS validation utterances each for kk, ky, uz (not in the repo): exact match for F32, F16, Q8_0 on CPU and Metal and Q6_K, Q5_K_M on Metal; Q4_K_M on Metal differs in one letter on one kk utterance. |
| Transcribe | explicit language hint | same with `--language kk` (or any of the five) | same transcript as without the hint | PASS. The hint has no effect on CTC decoding; a language outside `general.languages`, such as `--language de`, fails with `run: unsupported language`. |
| Translate, language detection, streaming | n/a | n/a | n/a | SKIP (not exposed upstream or by the runtime), as for v3. |
| Word timestamps | n/a | n/a | n/a | ACCEPTED GAP, as for v3: token timestamps only. |

Numerical validation at F32 on CPU, one thread, against the ru golden
case is within the per-variant tolerances, with an exact transcript; the
observed drift is in the `_comment` of the tolerance file. The other four
languages have no golden case: their numerical parity rests on the shared
graph and on the transcript checks above. The reference WER baseline, the
per-language WER sweep, and benchmarks have not been run;
`scripts/wer/run_reference_gigaam_author.py --variant multilingual_large_ctc`
runs the reference side.

`TRANSCRIBE_GIGAAM_GGUF=<gguf> ctest --test-dir build -R gigaam` passes
with the F32, F16, and Q8_0 GGUFs of this variant.
9 changes: 8 additions & 1 deletion docs/tools/conversion.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,14 @@ models/

This matches the raw-checkpoint layout under `$TRANSCRIBE_MODELS_DIR`
(also keyed by HF repo name), so `models/<slug>/` on the converted side
mirrors `$TRANSCRIBE_MODELS_DIR/<slug>/` on the source side.
mirrors `$TRANSCRIBE_MODELS_DIR/<slug>/` on the source side. A snapshot
pinned to a commit hash, full or abbreviated, sits beside it in
`$TRANSCRIBE_MODELS_DIR/<slug>@<commit>/`, so that an offline run reads only
the files of that commit. A pinned run (gigaam, ecapa_tdnn, or any
`--revision <hash>`) never reads `<slug>/`: online it downloads the commit
into `<slug>@<commit>/`, and offline without that directory it fails with an
error that names it. A `<slug>/` copy that only pinned runs used can be
deleted.

The `--repo-id` flag on each converter builds both the directory and
filename for you via `slug_from_repo_id()` + `gguf_name()` in
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# gigaam / gigaam-multilingual-large-ctc porting log

Surprises worth feeding back into the docs and tooling.

- **The package pin cannot load this variant by name.** `gigaam` at
6e4b027c has no `multilingual_large_ctc` key, and newer package commits
download from the author CDN. `scripts/lib/gigaam_hf.py` builds the same
package classes from the pinned Hugging Face snapshot instead, so the
family keeps one reference implementation and one package pin.
- **Bumping the package pin was considered and rejected for this port.**
Upstream added the `multilingual_large_ctc` key in 559d88d, but that
commit loads the checkpoint from the author CDN by MD5, not from the
Hugging Face commit the GGUF records, and 8555893 and 559d88d change
`gigaam/encoder.py`. A bump moves the reference of the four shipped v3
variants as well, so it needs their dumps, tolerances, and WER baselines
redone; that is a family decision for the maintainers, not part of
adding one variant.
- **The revision pins had three copies.** The dumper, the WER runner, and
the golden manifests each listed the commits, the WER runner with `main`
for every v3 variant, and `validate.py` did not pass the manifest's
`hf_revision` to the gigaam dumper. `scripts/lib/gigaam_hf.py` `VARIANTS`
is now the only table; a unit test compares it with the manifests, and
the dumper refuses a `--revision` that differs from it.
- **`$TRANSCRIBE_MODELS_DIR/<slug>/` is keyed by repo, not by commit.**
With the Hub unreachable, huggingface_hub 1.14 returns that directory
without checking which commit it holds, so a pinned load could read a
file of another commit or a file another tool saved there.
`lib.hf_source.download_snapshot` now downloads a pinned commit hash
into `<slug>@<commit>/`, for every converter that pins one; a checkout
in the old `<slug>/` is not reused and can be deleted.
- **`mel_scale` and `mel_norm` are not in the Multilingual config.** The v3
configs list `htk` and `null`, but `FeatureExtractor` never passes either
to torchaudio, and it defaults `center` to true where a config omits it.
The converter now reads the frontend values the module sets from the
instantiated module and refuses a config value that disagrees with it; the four v3
GGUFs are byte-identical to the previous converter's output.
- **The author CDN served an expired TLS certificate** during this port,
so `gigaam.load_model("v3_*")` failed and the v3 regression check could
not use the converter's own download path. It used the v3 weights from
the `ai-sage/GigaAM-v3` Hugging Face branches instead.
- **`scripts/intake.py inspect` has no `--revision`.** It reads the repo's
`main` branch, which for `ai-sage/GigaAM-Multilingual` is the 220M `ctc`
model, not `large_ctc`. The intake was written by hand from the pinned
commit.
- **`validate.py --mel-from-ref` is whisper-only**, although the gigaam
runtime honors `TRANSCRIBE_MEL_FROM_REF`. The frontend isolation
(subsample drift unchanged with the reference mel) was run by setting
`TRANSCRIBE_DUMP_DIR` and `TRANSCRIBE_MEL_FROM_REF` by hand and calling
`compare_tensors.py`.
- **The WER reference runner was v3-only.**
`scripts/wer/run_reference_gigaam_author.py` called `gigaam.load_model`
with a v3 key; it now loads through `scripts/lib/gigaam_hf.py` and
accepts `multilingual_large_ctc`. The WER baseline has not been run.
- **No C++ change.** The 24-layer, d_model=1024 encoder and the 71-class
CTC head run on the graph built for v3, which reads both from GGUF
metadata.
- **Q4_K_M changes one letter** on one kk FLEURS utterance
(`оцелоттар` for `ацелоттар`) on Metal; Q5_K_M and above match the
reference on all nine checked clips.
Loading