From d803dc43489285e10a11637918a5641fc6438b30 Mon Sep 17 00:00:00 2001 From: Vladimir Sitnikov Date: Sat, 7 Nov 2026 13:01:26 +0300 Subject: [PATCH] gigaam: convert GigaAM Multilingual Large CTC from its HF snapshot Previously, the converter was loading GigaAM models via gigaam.load_model(), but the package gigaam was not able to load Multilingual. The new scripts/lib/gigaam_hf.py takes config.json and the weights from ai-sage/GigaAM-Multilingual (branch large_ctc, pinned commit) and builds the model from the same package. The config.json for Multilingual does not have mel parameters, so the converter takes them from the loaded model, and refuses a config value that disagrees with it. VARIANTS in scripts/lib/gigaam_hf.py is the one table of the repo and commit of every GigaAM variant; the converter, the reference dumper, and the WER runner read it, and a unit test compares it with the golden manifests. With $TRANSCRIBE_MODELS_DIR set, a revision that is a commit hash now downloads into $TRANSCRIBE_MODELS_DIR/@/ instead of /, for every converter that pins one: offline, huggingface_hub returns a non-empty / without checking its commit. Part of #87. Assisted-by: Claude Opus 5.5 Assisted-by: GPT 6.1 sol --- .claude/skills/porting-2-oracle/SKILL.md | 3 +- .claude/skills/porting-4-cpp/SKILL.md | 6 +- .github/workflows/python-bindings.yml | 12 + docs/porting/families/gigaam.md | 108 ++++++- docs/tools/conversion.md | 9 +- .../_porting-log.md | 59 ++++ .../gigaam-multilingual-large-ctc/intake.json | 225 +++++++++++++++ scripts/convert-gigaam.py | 160 ++++++----- scripts/dump_reference_gigaam_author.py | 96 +++---- scripts/lib/gigaam_hf.py | 191 +++++++++++++ scripts/lib/hf_source.py | 62 +++- scripts/lib/test_gguf_writer.py | 4 + scripts/lib/test_gigaam_hf.py | 270 ++++++++++++++++++ scripts/lib/test_hf_source.py | 113 ++++++++ scripts/test_convert_gigaam.py | 130 +++++++++ scripts/validate.py | 5 +- scripts/wer/run_reference_gigaam_author.py | 40 +-- ...igaam-multilingual-large-ctc.manifest.json | 61 ++++ .../gigaam-multilingual-large-ctc.json | 78 +++++ 19 files changed, 1466 insertions(+), 166 deletions(-) create mode 100644 reports/porting/gigaam/gigaam-multilingual-large-ctc/_porting-log.md create mode 100644 reports/porting/gigaam/gigaam-multilingual-large-ctc/intake.json create mode 100644 scripts/lib/gigaam_hf.py create mode 100644 scripts/lib/test_gigaam_hf.py create mode 100644 scripts/lib/test_hf_source.py create mode 100644 scripts/test_convert_gigaam.py create mode 100644 tests/golden/gigaam/gigaam-multilingual-large-ctc.manifest.json create mode 100644 tests/tolerances/gigaam-multilingual-large-ctc.json diff --git a/.claude/skills/porting-2-oracle/SKILL.md b/.claude/skills/porting-2-oracle/SKILL.md index da3f0beb9..8d195bdd8 100644 --- a/.claude/skills/porting-2-oracle/SKILL.md +++ b/.claude/skills/porting-2-oracle/SKILL.md @@ -20,7 +20,8 @@ reference WER, and provisional tolerances. creates a skeleton following the contract in `scripts/lib/ref_dump.py` (or in one of the existing dumpers). - A committed reference checkpoint is available to the reference framework - (HF download or `$TRANSCRIBE_MODELS_DIR//`). + (HF download, `$TRANSCRIBE_MODELS_DIR//`, or + `$TRANSCRIBE_MODELS_DIR/@/` for a revision pinned to a commit hash). - `uv` is installed. ## Oracle packet contents diff --git a/.claude/skills/porting-4-cpp/SKILL.md b/.claude/skills/porting-4-cpp/SKILL.md index 175c1c224..5dc03276d 100644 --- a/.claude/skills/porting-4-cpp/SKILL.md +++ b/.claude/skills/porting-4-cpp/SKILL.md @@ -223,7 +223,11 @@ each entry with explicit justification.** Recipe: - `1.5 × observed < provisional`: C++ drift is well below the magnitude budget — common on simple tensors. Keep the magnitude budget; - tighten only with user confirmation. + tighten only with user confirmation. A variant added to a family whose + finalized tolerance file already tightened to `max(1.5 × observed, + 1e-6)` (such as `tests/tolerances/gigaam.json`) takes that recipe, so + the family's entries stay comparable; its `_comment` names the file + whose recipe it follows. - `1.5 × observed ≈ provisional`: C++ within budget. Preserve provisional. - `1.5 × observed > provisional`: C++ drift exceeds the magnitude budget. Record the widening factor in `_comment` and name the diff --git a/.github/workflows/python-bindings.yml b/.github/workflows/python-bindings.yml index e3cf3b784..0706f613a 100644 --- a/.github/workflows/python-bindings.yml +++ b/.github/workflows/python-bindings.yml @@ -26,6 +26,9 @@ on: - "include/**" - "src/**" - "scripts/lib/**" + - "scripts/convert-gigaam.py" + - "scripts/dump_reference_gigaam_author.py" + - "scripts/test_convert_gigaam.py" - "tests/**" - "ggml/**" - "CMakeLists.txt" @@ -38,6 +41,9 @@ on: - "include/**" - "src/**" - "scripts/lib/**" + - "scripts/convert-gigaam.py" + - "scripts/dump_reference_gigaam_author.py" + - "scripts/test_convert_gigaam.py" - "tests/**" - "ggml/**" - "CMakeLists.txt" @@ -82,6 +88,12 @@ jobs: run: uv run --no-project bindings/python/_generate/check_version_sync.py - name: GGUF writer layout run: uv run scripts/lib/test_gguf_writer.py + - name: Hugging Face snapshot commit check + run: uv run scripts/lib/test_hf_source.py + - name: GigaAM variant sources and loader + run: uv run scripts/lib/test_gigaam_hf.py + - name: GigaAM converter profiles + run: uv run scripts/test_convert_gigaam.py python-shared: runs-on: blacksmith-2vcpu-ubuntu-2404 diff --git a/docs/porting/families/gigaam.md b/docs/porting/families/gigaam.md index 242a7fa63..b15f18e26 100644 --- a/docs/porting/families/gigaam.md +++ b/docs/porting/families/gigaam.md @@ -6,13 +6,19 @@ ASR variants (`gigaam-v3-e2e-rnnt`, `gigaam-v3-rnnt`, `gigaam-v3-ctc`, `gigaam-v3-e2e-ctc`) ready for HF upload. `v3_ssl` (encoder-only HuBERT pretraining checkpoint) intentionally out of scope. +`gigaam-multilingual-large-ctc` (ai-sage/GigaAM-Multilingual, branch +`large_ctc`) converts and validates at F32 on the existing runtime with no +C++ change; WER, benchmarks, and publication are not done. See +[GigaAM Multilingual Large CTC](#gigaam-multilingual-large-ctc). + ## Identity - Family key: `gigaam` - Upstream architecture string: `gigaam` (HF `model_type`); paper name "GigaAM" (Sber/ai-sage, InterSpeech 2025, arXiv:2506.01192) - Hugging Face repo: `ai-sage/GigaAM-v3` (five sibling branches upstream; four ported) +- Also converted, not yet published: `gigaam-multilingual-large-ctc` from `ai-sage/GigaAM-Multilingual`, see [GigaAM Multilingual Large CTC](#gigaam-multilingual-large-ctc) - License: MIT -- Variants in scope (one HF branch each): +- v3 variants in scope (one HF branch each): | Variant | HF branch | Head | num_classes | Vocab style | |---|---|---|---|---| @@ -33,7 +39,7 @@ pretraining checkpoint) intentionally out of scope. encoder-output emission CLI, so `v3_ssl` has no user-observable capability here and is intentionally **out of scope** for this port. -Canonical bench samples: `ru` (model is Russian-only; English `jfk`/`dots` would decode to garbage). +Canonical bench samples for the v3 variants: `ru` (model is Russian-only; English `jfk`/`dots` would decode to garbage). ## References @@ -220,3 +226,101 @@ them from the converter source. The intake's hypothesis of shared encoder weights was wrong; the converter emits a full GGUF per variant, not a shared encoder GGUF with per-variant head shards. + +## GigaAM Multilingual Large CTC + +Variant `gigaam-multilingual-large-ctc` is the `large_ctc` branch of +[`ai-sage/GigaAM-Multilingual`](https://huggingface.co/ai-sage/GigaAM-Multilingual), +pinned at commit `3905cd51c3ed4e88c8edf33f3302969ba480a327` (the repo's +`main` branch is the smaller `ctc` model). It is the v3 encoder at 24 +layers, d_model=1024, 16 heads, with a CTC head over 71 classes: a +70-entry charwise vocab (space, apostrophe, a-z, Russian а-я and ё, and +і ғ қ ң ү ұ һ ә ө) plus blank at id 70. Languages: ru, en, kk, ky, uz. +Offline only; there is no language input or language-detection output. + +| Item | Path | +|---|---| +| Intake | `reports/porting/gigaam/gigaam-multilingual-large-ctc/intake.json` | +| Golden manifest | `tests/golden/gigaam/gigaam-multilingual-large-ctc.manifest.json` | +| Tolerances | `tests/tolerances/gigaam-multilingual-large-ctc.json` | +| Variant sources and snapshot loader | `scripts/lib/gigaam_hf.py` (tests: `scripts/lib/test_gigaam_hf.py`) | + +### Commands + +Conversion (writes `models/gigaam-multilingual-large-ctc/gigaam-multilingual-large-ctc-F32.gguf`, +2341378624 bytes, 824 tensors): + +```bash +uv run --project scripts/envs/gigaam scripts/convert-gigaam.py ai-sage/GigaAM-Multilingual \ + --repo-id gigaam-multilingual-large-ctc +``` + +The converter downloads the snapshot (2.3 GB) at the pinned commit into +`$TRANSCRIBE_MODELS_DIR/@/` or the Hugging Face cache, both +of which keep each commit in its own directory; with `HF_HUB_OFFLINE=1` +it uses that local copy without network access, and fails when the +pinned commit was never downloaded. Unlike converters that take +`lib.hf_source.resolve_model_dir`, it accepts no other repo id and no local directory, so `general.repo_url` +and the reference dumps' `hf_revision` name the source the weights came +from. + +The repo and commit of every GigaAM variant are in `VARIANTS` of +`scripts/lib/gigaam_hf.py`, which the converter, the reference dumper, +and the WER runner read. `scripts/lib/test_gigaam_hf.py` fails when a +golden manifest names another source, and the dumper refuses the +`--revision` that `validate.py` passes from the manifest when it differs +from the pin. Change the pin and the manifest together. + +Reference dumps and validation use the same pin: + +```bash +uv run scripts/preflight.py --family gigaam --variant gigaam-multilingual-large-ctc --gate A +uv run scripts/preflight.py --family gigaam --variant gigaam-multilingual-large-ctc --gate B +uv run scripts/validate.py all --family gigaam --variant gigaam-multilingual-large-ctc +uv run scripts/quantize-all.py models/gigaam-multilingual-large-ctc/gigaam-multilingual-large-ctc-F32.gguf +TRANSCRIBE_GIGAAM_GGUF=$PWD/models/gigaam-multilingual-large-ctc/gigaam-multilingual-large-ctc-F16.gguf \ + ctest --test-dir build -R gigaam +``` + +### Reference + +The reference is the family's `gigaam` package pin (6e4b027c), as for v3. +That commit has no `load_model("multilingual_large_ctc")`, so +`scripts/lib/gigaam_hf.py` instantiates the package's `FeatureExtractor`, +`ConformerEncoder`, `CTCHead`, and `CTCGreedyDecoding` from the snapshot's +`config.json`, whose `_target_` entries name the same classes in +`modeling_gigaam`, and loads `pytorch_model.bin` with its `model.` key +prefix removed. The upstream commits from 6e4b027c to 7447938 change the +encoder only for padded batch inputs. The repo's own `modeling_gigaam.py` +on the same weights produces bit-identical `frontend.mel.out`, `enc.out`, +and `ctc.log_probs` on `samples/ru.wav`. + +The v3 config lists `mel_scale: htk` and `mel_norm: null`; the +Multilingual config omits both. The package's `FeatureExtractor` passes +neither to torchaudio and defaults `center` to true, so the converter +reads the values the module sets (sample rate, FFT, window and hop +lengths, mel count, `center`, mel scale and norm, frequency range) from +it, `htk` with no norm for every variant, and refuses a config value that +disagrees. The rest of `stt.frontend.*` (periodic Hann window, no dither, +pre-emphasis, or normalization) is fixed by `FeatureExtractor`'s code. + +### Capability Validation + +| Capability | Mode | Command / test | Expected observable | Status | +|------------|------|----------------|---------------------|--------| +| Transcribe (ru, en, kk, ky, uz) | auto / no language hint | `build/bin/transcribe-cli -m ` | transcript equal to the reference's | PASS on `samples/ru.wav`, `samples/fleurs-ru.wav`, `samples/fleurs-en.wav`, and two FLEURS validation utterances each for kk, ky, uz (not in the repo): exact match for F32, F16, Q8_0 on CPU and Metal and Q6_K, Q5_K_M on Metal; Q4_K_M on Metal differs in one letter on one kk utterance. | +| Transcribe | explicit language hint | same with `--language kk` (or any of the five) | same transcript as without the hint | PASS. The hint has no effect on CTC decoding; a language outside `general.languages`, such as `--language de`, fails with `run: unsupported language`. | +| Translate, language detection, streaming | n/a | n/a | n/a | SKIP (not exposed upstream or by the runtime), as for v3. | +| Word timestamps | n/a | n/a | n/a | ACCEPTED GAP, as for v3: token timestamps only. | + +Numerical validation at F32 on CPU, one thread, against the ru golden +case is within the per-variant tolerances, with an exact transcript; the +observed drift is in the `_comment` of the tolerance file. The other four +languages have no golden case: their numerical parity rests on the shared +graph and on the transcript checks above. The reference WER baseline, the +per-language WER sweep, and benchmarks have not been run; +`scripts/wer/run_reference_gigaam_author.py --variant multilingual_large_ctc` +runs the reference side. + +`TRANSCRIBE_GIGAAM_GGUF= ctest --test-dir build -R gigaam` passes +with the F32, F16, and Q8_0 GGUFs of this variant. diff --git a/docs/tools/conversion.md b/docs/tools/conversion.md index fd953c29b..130eac377 100644 --- a/docs/tools/conversion.md +++ b/docs/tools/conversion.md @@ -93,7 +93,14 @@ models/ This matches the raw-checkpoint layout under `$TRANSCRIBE_MODELS_DIR` (also keyed by HF repo name), so `models//` on the converted side -mirrors `$TRANSCRIBE_MODELS_DIR//` on the source side. +mirrors `$TRANSCRIBE_MODELS_DIR//` on the source side. A snapshot +pinned to a commit hash, full or abbreviated, sits beside it in +`$TRANSCRIBE_MODELS_DIR/@/`, so that an offline run reads only +the files of that commit. A pinned run (gigaam, ecapa_tdnn, or any +`--revision `) never reads `/`: online it downloads the commit +into `@/`, and offline without that directory it fails with an +error that names it. A `/` copy that only pinned runs used can be +deleted. The `--repo-id` flag on each converter builds both the directory and filename for you via `slug_from_repo_id()` + `gguf_name()` in diff --git a/reports/porting/gigaam/gigaam-multilingual-large-ctc/_porting-log.md b/reports/porting/gigaam/gigaam-multilingual-large-ctc/_porting-log.md new file mode 100644 index 000000000..ab817eec6 --- /dev/null +++ b/reports/porting/gigaam/gigaam-multilingual-large-ctc/_porting-log.md @@ -0,0 +1,59 @@ +# gigaam / gigaam-multilingual-large-ctc porting log + +Surprises worth feeding back into the docs and tooling. + +- **The package pin cannot load this variant by name.** `gigaam` at + 6e4b027c has no `multilingual_large_ctc` key, and newer package commits + download from the author CDN. `scripts/lib/gigaam_hf.py` builds the same + package classes from the pinned Hugging Face snapshot instead, so the + family keeps one reference implementation and one package pin. +- **Bumping the package pin was considered and rejected for this port.** + Upstream added the `multilingual_large_ctc` key in 559d88d, but that + commit loads the checkpoint from the author CDN by MD5, not from the + Hugging Face commit the GGUF records, and 8555893 and 559d88d change + `gigaam/encoder.py`. A bump moves the reference of the four shipped v3 + variants as well, so it needs their dumps, tolerances, and WER baselines + redone; that is a family decision for the maintainers, not part of + adding one variant. +- **The revision pins had three copies.** The dumper, the WER runner, and + the golden manifests each listed the commits, the WER runner with `main` + for every v3 variant, and `validate.py` did not pass the manifest's + `hf_revision` to the gigaam dumper. `scripts/lib/gigaam_hf.py` `VARIANTS` + is now the only table; a unit test compares it with the manifests, and + the dumper refuses a `--revision` that differs from it. +- **`$TRANSCRIBE_MODELS_DIR//` is keyed by repo, not by commit.** + With the Hub unreachable, huggingface_hub 1.14 returns that directory + without checking which commit it holds, so a pinned load could read a + file of another commit or a file another tool saved there. + `lib.hf_source.download_snapshot` now downloads a pinned commit hash + into `@/`, for every converter that pins one; a checkout + in the old `/` is not reused and can be deleted. +- **`mel_scale` and `mel_norm` are not in the Multilingual config.** The v3 + configs list `htk` and `null`, but `FeatureExtractor` never passes either + to torchaudio, and it defaults `center` to true where a config omits it. + The converter now reads the frontend values the module sets from the + instantiated module and refuses a config value that disagrees with it; the four v3 + GGUFs are byte-identical to the previous converter's output. +- **The author CDN served an expired TLS certificate** during this port, + so `gigaam.load_model("v3_*")` failed and the v3 regression check could + not use the converter's own download path. It used the v3 weights from + the `ai-sage/GigaAM-v3` Hugging Face branches instead. +- **`scripts/intake.py inspect` has no `--revision`.** It reads the repo's + `main` branch, which for `ai-sage/GigaAM-Multilingual` is the 220M `ctc` + model, not `large_ctc`. The intake was written by hand from the pinned + commit. +- **`validate.py --mel-from-ref` is whisper-only**, although the gigaam + runtime honors `TRANSCRIBE_MEL_FROM_REF`. The frontend isolation + (subsample drift unchanged with the reference mel) was run by setting + `TRANSCRIBE_DUMP_DIR` and `TRANSCRIBE_MEL_FROM_REF` by hand and calling + `compare_tensors.py`. +- **The WER reference runner was v3-only.** + `scripts/wer/run_reference_gigaam_author.py` called `gigaam.load_model` + with a v3 key; it now loads through `scripts/lib/gigaam_hf.py` and + accepts `multilingual_large_ctc`. The WER baseline has not been run. +- **No C++ change.** The 24-layer, d_model=1024 encoder and the 71-class + CTC head run on the graph built for v3, which reads both from GGUF + metadata. +- **Q4_K_M changes one letter** on one kk FLEURS utterance + (`оцелоттар` for `ацелоттар`) on Metal; Q5_K_M and above match the + reference on all nine checked clips. diff --git a/reports/porting/gigaam/gigaam-multilingual-large-ctc/intake.json b/reports/porting/gigaam/gigaam-multilingual-large-ctc/intake.json new file mode 100644 index 000000000..5391ee809 --- /dev/null +++ b/reports/porting/gigaam/gigaam-multilingual-large-ctc/intake.json @@ -0,0 +1,225 @@ +{ + "schema_version": "transcribe-intake-v1", + "family": "gigaam", + "hf_repo": "ai-sage/GigaAM-Multilingual", + "hf_revision": "3905cd51c3ed4e88c8edf33f3302969ba480a327", + "sources": { + "config": { + "kind": "hf_file", + "path": "config.json", + "status": "found", + "detail": "Branch large_ctc (the repo's main branch is the 220M ctc model). Hydra-style config under cfg.model.cfg: same encoder classes as gigaam-v3 at 24 layers, d_model=1024, 16 heads; head=CTCHead with feat_in=1024, num_classes=71 (70 chars + 1 blank). decoding=CTCGreedyDecoding with an inline character vocabulary. model_name=multilingual_large_ctc; model_class=ctc." + }, + "preprocessor": { + "kind": "hf_file", + "path": "preprocessor_config.json|feature_extractor_config.json", + "status": "missing", + "detail": "Frontend declared inside config.json under cfg.model.cfg.preprocessor (features=64, n_fft=320, hop=160, win=320, center=false). Unlike the v3 configs it does not list mel_scale / mel_norm; gigaam's FeatureExtractor never passes them to torchaudio, so the frontend is the same HTK, unnormalized filterbank as v3." + }, + "tokenizer_config": { + "kind": "hf_file", + "path": "tokenizer_config.json", + "status": "missing", + "detail": "Charwise tokenizer; vocabulary is inline in config.json under cfg.model.cfg.decoding.vocabulary: 70 strings (space, apostrophe, a-z, 33 Russian letters including ё, and і ғ қ ң ү ұ һ ә ө for Kazakh, Kyrgyz and Uzbek Cyrillic)." + }, + "tokenizer_model": { + "kind": "hf_file", + "path": "tokenizer.model", + "status": "missing", + "detail": "Branch has no tokenizer.model file: `Tokenizer.charwise=True` path, as in gigaam-v3-ctc." + }, + "tokenizer_json": { + "kind": "hf_file", + "path": "tokenizer.json", + "status": "missing", + "detail": "Not used." + }, + "generation_config": { + "kind": "hf_file", + "path": "generation_config.json", + "status": "missing", + "detail": "Decoding is CTC greedy; no HF generation_config." + }, + "model_weights": { + "kind": "hf_file", + "path": "pytorch_model.bin", + "status": "found", + "detail": "PyTorch pickle, 2,341,592,643 bytes, LFS sha256 c3fabefb50b41f08f4d7ad44e02c26c37d242882704cdcca2ebd98e45eff73d1. All tensors float32, every key prefixed `model.` by the HF wrapper." + }, + "safetensors_metadata": { + "kind": "hf_api", + "path": "HfApi.get_safetensors_metadata", + "status": "missing", + "detail": "Repo ships PyTorch pickle, not safetensors." + }, + "model_card": { + "kind": "hf_api", + "path": "https://huggingface.co/ai-sage/GigaAM-Multilingual", + "status": "found", + "detail": "One card covers the ssl, ctc, large_ssl and large_ctc branches; WER table on Common Voice, FLEURS and internal sets for ru, en, kk, ky, uz." + }, + "reference_modeling_code": { + "kind": "reference_code", + "path": "https://github.com/salute-developers/GigaAM (gigaam package classes, built from the HF snapshot by scripts/lib/gigaam_hf.py)", + "status": "found", + "detail": "Author-first PyTorch implementation, MIT. The pinned package commit 6e4b027c predates gigaam.load_model('multilingual_large_ctc') (added in 559d88d), but its preprocessor, ConformerEncoder, CTCHead and CTCGreedyDecoding already cover this config; the commits from 6e4b027c to 7447938 change the encoder only for padded batch inputs. The HF repo also ships modeling_gigaam.py with the same classes." + } + }, + "variants": [ + { + "name": "gigaam-multilingual-large-ctc", + "memory_gb": 2.34, + "files": [ + "config.json", + "modeling_gigaam.py", + "pytorch_model.bin" + ] + } + ], + "config": { + "architecture_candidates": [ + "encoder-ctc" + ], + "key_fields": { + "model_type": "gigaam", + "model_class": "ctc", + "model_name": "multilingual_large_ctc", + "encoder.type": "ConformerEncoder", + "encoder.feat_in": 64, + "encoder.n_layers": 24, + "encoder.d_model": 1024, + "encoder.n_heads": 16, + "encoder.ff_expansion_factor": 4, + "encoder.self_attention_model": "rotary", + "encoder.pos_emb_max_len": 5000, + "encoder.subsampling": "conv1d", + "encoder.subs_kernel_size": 5, + "encoder.subsampling_factor": 4, + "encoder.conv_kernel_size": 5, + "encoder.conv_norm_type": "layer_norm", + "head.type": "CTCHead", + "head.feat_in": 1024, + "head.num_classes": 71, + "decoding": "CTCGreedyDecoding", + "tokenizer.kind": "charwise", + "tokenizer.vocab_size_plus_blank": 71, + "tokenizer.vocab_inline": "70 entries: space, apostrophe, a-z, а-я, ё, і, ғ, қ, ң, ү, ұ, һ, ә, ө." + }, + "varying_across_variants": [ + "encoder.n_layers", + "encoder.d_model", + "head.feat_in", + "head.num_classes", + "tokenizer.vocab_inline" + ] + }, + "dtype": { + "expected": "float32", + "source": "manual", + "evidence": "Every tensor in pytorch_model.bin is torch.float32 (824 tensors, 585,319,495 parameters outside the frontend buffers).", + "details": { + "config_declared": null, + "header_distribution": {} + }, + "expected_f32_tensors": [] + }, + "frontend": { + "sample_rate": 16000, + "n_mels": 64, + "hop_length": 160, + "fft_size": 320, + "window": "hann_periodic", + "normalization": "none", + "preemphasis": null, + "dither": null, + "center": false, + "padding_mode": null, + "mel_filterbank_norm": "htk" + }, + "tokenizer": { + "type": "other", + "vocab_size": 70, + "special_tokens": { + "blank": 70 + }, + "has_language_tokens": false, + "vocab_sha256": "c74f14b62f397a12ab3d1d2db0986ae1ae929f11a65465bd8ea2db06e8022f49" + }, + "capabilities": { + "languages": ["ru", "en", "kk", "ky", "uz"], + "language_detection": false, + "translation": false, + "timestamps": [], + "streaming": false, + "speaker_diarization": false + }, + "upstream_benchmarks": [ + { + "dataset": "FLEURS", + "language": "ru", + "metric": "wer", + "score": 3.0, + "score_unit": "percent", + "source": "https://huggingface.co/ai-sage/GigaAM-Multilingual", + "notes": "GigaAM Multilingual Large column. Utterances over 30 s and references with digits excluded; lowercased, punctuation removed, numerals to words; greedy decoding." + }, + { + "dataset": "FLEURS", + "language": "kk", + "metric": "wer", + "score": 4.4, + "score_unit": "percent", + "source": "https://huggingface.co/ai-sage/GigaAM-Multilingual", + "notes": "Same column and normalization as the ru row." + }, + { + "dataset": "FLEURS", + "language": "ky", + "metric": "wer", + "score": 5.5, + "score_unit": "percent", + "source": "https://huggingface.co/ai-sage/GigaAM-Multilingual", + "notes": "Same column and normalization as the ru row." + }, + { + "dataset": "FLEURS", + "language": "uz", + "metric": "wer", + "score": 7.3, + "score_unit": "percent", + "source": "https://huggingface.co/ai-sage/GigaAM-Multilingual", + "notes": "Same column and normalization as the ru row." + }, + { + "dataset": "FLEURS", + "language": "en", + "metric": "wer", + "score": 9.4, + "score_unit": "percent", + "source": "https://huggingface.co/ai-sage/GigaAM-Multilingual", + "notes": "Same column and normalization as the ru row." + } + ], + "reference_framework": "author_repo_gigaam", + "reference_rationale": "Same as the rest of the gigaam family: the salute-developers `gigaam` package (MIT) at the family pin 6e4b027c. Its classes are instantiated from the pinned HF snapshot (config.json + pytorch_model.bin at large_ctc 3905cd51) by scripts/lib/gigaam_hf.py, because the package's own download for this variant needs a newer package and a CDN that the family does not pin.", + "architecture_pattern": "encoder-ctc", + "known_risks": [ + "Encoder is the gigaam-v3 Conformer at 24 layers, d_model=1024; the C++ graph reads both from GGUF KV and has only been validated at 16 x 768 before this variant.", + "CTCHead is `nn.Conv1d(1024, num_classes=71, kernel_size=1)` + `log_softmax(-1)`. Blank id = 70 = num_classes - 1, appended after the inline vocab.", + "Charwise tokenizer: the inline vocab order is the CTC class order; the converter must keep it, including the non-ASCII Kazakh / Kyrgyz / Uzbek letters, and must not sort or deduplicate.", + "No language input and no language-detection output: the model picks characters from all five languages' alphabets per frame, so a --language hint has no effect on decoding.", + "Conv1d subsampling, SpecScaler log, center=false, HTK mel without area normalization, rotary attention: same gotchas as gigaam-v3.", + "Upstream evaluates WER on lowercased, punctuation-free text with numerals spelled out; the model emits no digits, casing or punctuation, so WER comparisons need the same normalization.", + "Only the ru golden case is in the manifest; numerical parity for the other four languages rests on the shared graph and is not dumped." + ], + "intake_gaps": [ + { + "field": "upstream_benchmarks", + "reason": "Published scores are context only. The Oracle reference WER baseline (Stage 2 Step 7) and the per-language WER sweep (Stage 7) have not been run for this variant." + }, + { + "field": "dtype.expected_f32_tensors", + "reason": "Converter emits every tensor at F32 (reference dtype); same as the v3 variants." + } + ] +} diff --git a/scripts/convert-gigaam.py b/scripts/convert-gigaam.py index 23b946357..d6eb47b3f 100644 --- a/scripts/convert-gigaam.py +++ b/scripts/convert-gigaam.py @@ -1,8 +1,10 @@ #!/usr/bin/env python3 """ -convert-gigaam.py - convert a salute-developers/GigaAM-v3 checkpoint into -a GGUF that transcribe.cpp's loader can ingest end-to-end. Loads via the -upstream `gigaam` package (gigaam.load_model). Source dtype is F32; use +convert-gigaam.py - convert a salute-developers GigaAM checkpoint into +a GGUF that transcribe.cpp's loader can ingest end-to-end. Models are +built from the upstream `gigaam` package: the v3 variants through +gigaam.load_model, GigaAM Multilingual Large CTC from its pinned Hugging +Face snapshot (scripts/lib/gigaam_hf.py). Source dtype is F32; use tools/transcribe-quantize for deployment quantization. Run through the per-family reference environment: @@ -12,10 +14,19 @@ --repo-id gigaam-v3-e2e-rnnt \\ --variant-key v3_e2e_rnnt -The HF repo argument is informational only — the `gigaam` package -downloads from its own URL list keyed by --variant-key. The --repo-id -controls the output slug; it must be one of the five GigaAM-v3 family -slugs in `VARIANT_PROFILES`. +For the v3 variants the HF repo argument is informational only — the +`gigaam` package downloads from its own URL list keyed by --variant-key. +GigaAM Multilingual Large CTC reads the weights from the HF repo at the +commit pinned in scripts/lib/gigaam_hf.py; the argument must be that repo +id. Unlike the converters that take a local directory, this one refuses +one: a directory carries no commit for the GGUF to record. + + uv run --project scripts/envs/gigaam \\ + scripts/convert-gigaam.py ai-sage/GigaAM-Multilingual \\ + --repo-id gigaam-multilingual-large-ctc + +The --repo-id controls the output slug; it must be one of the slugs in +`VARIANT_PROFILES`. The four ported variants share a 16-layer Conformer encoder with rotary attention. They differ in head + tokenizer: @@ -25,6 +36,11 @@ v3_rnnt -> RNN-T head, charwise 33 entries (lowercased Russian) v3_ctc -> CTC head, charwise 33 entries (lowercased Russian) +GigaAM Multilingual Large CTC (multilingual_large_ctc) uses the same +encoder blocks at 24 layers, d=1024, 16 heads, and a CTC head over a +70-entry charwise vocab (space, apostrophe, Latin and Cyrillic letters +for ru, en, kk, ky, uz). + The upstream `v3_ssl` (HuBERT-CTC pretraining checkpoint, encoder-only) is out of scope for this runtime — it has no head, no tokenizer, no decoding path, and transcribe.cpp has no encoder-output emission CLI. @@ -38,10 +54,10 @@ loader will define): general.architecture = "gigaam" - general.basename = "gigaam-v3" - general.size_label = "242M" - general.version = "v3" - general.languages = ["ru"] + general.basename = "gigaam-v3" | "gigaam-multilingual" + general.size_label = "242M" (v3 RNN-T) | "585M" (Multilingual Large) + general.version = "v3" (v3 variants only) + general.languages = ["ru"] | ["ru", "en", "kk", "ky", "uz"] stt.variant = profile["variant"] stt.gigaam.head_kind = "rnnt" | "ctc" @@ -101,6 +117,7 @@ safe_id, slug_from_repo_id, ) +from lib.gigaam_hf import frontend_params, load_model, variant_source # noqa: E402 REPO_ROOT = Path(__file__).resolve().parent.parent @@ -117,11 +134,16 @@ # Variant profiles # --------------------------------------------------------------------------- # -# All five GigaAM-v3 variants share the same Conformer encoder shape -# (16 layers, d=768, 16 heads). They differ only in head + tokenizer. +# The v3 variants share one Conformer encoder shape (16 layers, d=768, +# 16 heads) and differ only in head + tokenizer; Multilingual Large CTC +# has 24 layers, d=1024. The converter reads the shape from the model. # The slug (HF-style basename used as the output directory + GGUF base # name) is the key the converter dispatches on; --variant-key picks the -# actual `gigaam.load_model(...)` argument. +# scripts/lib/gigaam_hf.py VARIANTS entry, which says where the weights +# come from. +# basename / version / languages fill the general.* identity block. + +_V3_IDENTITY = {"basename": "gigaam-v3", "version": "v3", "languages": ["ru"]} VARIANT_PROFILES: dict[str, dict] = { "gigaam-v3-e2e-rnnt": { @@ -130,6 +152,7 @@ "head_kind": "rnnt", "tokenizer": "sentencepiece", "expected_num_classes": 1025, # vocab(1024) + 1 blank + **_V3_IDENTITY, }, "gigaam-v3-e2e-ctc": { "variant": "gigaam-v3-e2e-ctc", @@ -137,6 +160,7 @@ "head_kind": "ctc", "tokenizer": "sentencepiece", "expected_num_classes": 257, # vocab(256) + 1 blank + **_V3_IDENTITY, }, "gigaam-v3-rnnt": { "variant": "gigaam-v3-rnnt", @@ -144,6 +168,7 @@ "head_kind": "rnnt", "tokenizer": "charwise", "expected_num_classes": 34, # 33 chars + 1 blank + **_V3_IDENTITY, }, "gigaam-v3-ctc": { "variant": "gigaam-v3-ctc", @@ -151,43 +176,31 @@ "head_kind": "ctc", "tokenizer": "charwise", "expected_num_classes": 34, + **_V3_IDENTITY, + }, + "gigaam-multilingual-large-ctc": { + "variant": "gigaam-multilingual-large-ctc", + "variant_key": "multilingual_large_ctc", + "head_kind": "ctc", + "tokenizer": "charwise", + "expected_num_classes": 71, # 70 chars + 1 blank + "basename": "gigaam-multilingual", + "version": None, + "languages": ["ru", "en", "kk", "ky", "uz"], }, } -GENERAL_BASENAME = "gigaam-v3" -GENERAL_VERSION = "v3" -GENERAL_LANGUAGES = ["ru"] - - # Friendly general.name per variant slug (== profile["variant"]). VARIANT_DISPLAY_NAMES: dict[str, str] = { "gigaam-v3-ctc": "GigaAM v3 CTC", "gigaam-v3-e2e-ctc": "GigaAM v3 E2E-CTC", "gigaam-v3-rnnt": "GigaAM v3 RNN-T", "gigaam-v3-e2e-rnnt": "GigaAM v3 E2E-RNN-T", + "gigaam-multilingual-large-ctc": "GigaAM Multilingual Large CTC", } -# --------------------------------------------------------------------------- -# Model loading -# --------------------------------------------------------------------------- - - -def load_gigaam(variant_key: str): - """Load a GigaAM-v3 variant via the upstream `gigaam` package. - - Forces fp16_encoder=False (F32) and device='cpu' to match the - Stage 2 reference dumps. - """ - import gigaam - - print(f"Loading gigaam.load_model({variant_key!r}, fp16_encoder=False, device='cpu')") - model = gigaam.load_model(variant_key, fp16_encoder=False, device="cpu") - model.eval() - return model - - # --------------------------------------------------------------------------- # Tokenizer extraction # --------------------------------------------------------------------------- @@ -243,10 +256,11 @@ def extract_sp_tokenizer(sp, blank_piece: str = "") -> dict: def extract_charwise_tokenizer(vocab: list[str], blank_piece: str = "") -> dict: - """Emit a charwise GigaAM vocab (33 entries: space + 32 Cyrillic - letters) as a tokenizer.ggml payload. Scores are all zero (charwise - has no learned scores). Blank goes last and is the only CONTROL - token; the 33 chars are all NORMAL. + """Emit a charwise GigaAM vocab (v3: 33 entries, space + 32 Cyrillic + letters; Multilingual: 70 entries) as a tokenizer.ggml payload. + Scores are all zero (charwise has no learned scores). Blank goes last + and is the only CONTROL token; the vocab entries are all NORMAL and + keep their config.json order, which is the CTC class order. """ tokens: list[str] = list(vocab) scores: list[float] = [0.0] * len(vocab) @@ -281,14 +295,15 @@ def read_hparams(config: dict, model) -> dict: enc = config["encoder"] pre = config["preprocessor"] - sample_rate = int(pre["sample_rate"]) - n_fft = int(pre["n_fft"]) - hop_length = int(pre["hop_length"]) - win_length = int(pre["win_length"]) - n_mels = int(pre["features"]) - mel_scale = str(pre["mel_scale"]) - mel_norm_raw = pre.get("mel_norm") - center = bool(pre.get("center", False)) + fe = frontend_params(pre, model.preprocessor) + sample_rate = fe["sample_rate"] + n_fft = fe["n_fft"] + hop_length = fe["hop_length"] + win_length = fe["win_length"] + n_mels = fe["n_mels"] + center = fe["center"] + mel_scale = fe["mel_scale"] + mel_norm_raw = fe["mel_norm"] # SpecScaler clamp constants used by GigaAM's preprocessor: # log(clamp(x, 1e-9, 1e9)). The upper clamp is so large it never @@ -343,8 +358,8 @@ def read_hparams(config: dict, model) -> dict: "fe_normalize": "none", # no per-utterance/feature normalize "fe_dither": 0.0, "fe_pre_emphasis": 0.0, - "fe_f_min": 0.0, - "fe_f_max": float(sample_rate) / 2.0, + "fe_f_min": fe["f_min"], + "fe_f_max": fe["f_max"], "fe_center": center, # `mel_scale` encodes both the HTK frequency formula and the # absence of slaney area normalization (mel_norm=null). The C++ @@ -488,7 +503,7 @@ def tensor_to_fp32_numpy(t) -> np.ndarray: # --------------------------------------------------------------------------- -def convert(variant_key: str, slug: str, out_path: Path, repo_id: str | None = None) -> None: +def convert(variant_key: str, slug: str, out_path: Path, repo_id: str) -> None: from omegaconf import OmegaConf print(f"Output dtype: {REFERENCE_DTYPE_LABEL} (source/reference dtype)") @@ -508,7 +523,7 @@ def convert(variant_key: str, slug: str, out_path: Path, repo_id: str | None = N print(f"Variant: {profile['variant']} (head_kind={head_kind}, " f"tokenizer={profile['tokenizer']})") - model = load_gigaam(variant_key) + model = load_model(variant_key) config = OmegaConf.to_container(model.cfg, resolve=True) hp = read_hparams(config, model) @@ -609,11 +624,11 @@ def convert(variant_key: str, slug: str, out_path: Path, repo_id: str | None = N add_general_identity( writer, name=VARIANT_DISPLAY_NAMES[profile["variant"]], - basename=GENERAL_BASENAME, + basename=profile["basename"], size_label=size_label, - version=GENERAL_VERSION, + version=profile["version"], file_type=REFERENCE_FILE_TYPE, - languages=GENERAL_LANGUAGES, + languages=profile["languages"], author="Salute Developers", organization="ai-sage", license="mit", @@ -800,14 +815,17 @@ def add_combined(src_a: str, src_b: str, dst_name: str) -> None: def main(argv: list[str]) -> int: p = argparse.ArgumentParser( - description="Convert a GigaAM-v3 author-repo checkpoint to a GGUF.", + description="Convert a GigaAM author-repo checkpoint to a GGUF.", ) p.add_argument( "model", type=str, - help="HF repo id (informational; e.g. ai-sage/GigaAM-v3) — the " - "gigaam package downloads from its own URL list keyed by " - "--variant-key.", + help="HF repo id. Informational for v3 (e.g. ai-sage/GigaAM-v3): " + "the gigaam package downloads from its own URL list keyed by " + "--variant-key. For gigaam-multilingual-large-ctc, the source " + "of the weights: ai-sage/GigaAM-Multilingual, fetched at the " + "commit pinned in scripts/lib/gigaam_hf.py. A local directory " + "is refused, because the GGUF records that commit.", ) p.add_argument( "out_path", @@ -820,15 +838,17 @@ def main(argv: list[str]) -> int: "--repo-id", type=str, default=None, - help="One of the five gigaam-v3-* family slugs (drives the output " - "filename and selects the VARIANT_PROFILES entry).", + help="A VARIANT_PROFILES slug, e.g. gigaam-v3-ctc or " + "gigaam-multilingual-large-ctc (drives the output filename " + "and selects the profile).", ) p.add_argument( "--variant-key", type=str, default=None, - help="gigaam.load_model() key (v3_e2e_rnnt, v3_e2e_ctc, v3_rnnt, " - "v3_ctc). If omitted, derived from --repo-id.", + help="gigaam variant key (v3_e2e_rnnt, v3_e2e_ctc, v3_rnnt, " + "v3_ctc, multilingual_large_ctc). If omitted, derived from " + "--repo-id.", ) args = p.parse_args(argv[1:]) @@ -847,6 +867,16 @@ def main(argv: list[str]) -> int: profile = VARIANT_PROFILES[slug] variant_key = args.variant_key or profile["variant_key"] + source = variant_source(variant_key) + if source.from_snapshot and args.model != source.repo_id: + # The GGUF records the pinned repo and commit; another repo or a + # local directory would not be the source it names. + print( + f"error: {variant_key} converts from the Hugging Face repo " + f"{source.repo_id} at {source.revision}, got {args.model!r}", + file=sys.stderr, + ) + return 2 out_path = args.out_path if out_path is None: diff --git a/scripts/dump_reference_gigaam_author.py b/scripts/dump_reference_gigaam_author.py index 0f0c186c9..659a7a297 100755 --- a/scripts/dump_reference_gigaam_author.py +++ b/scripts/dump_reference_gigaam_author.py @@ -1,7 +1,9 @@ #!/usr/bin/env python3 """ dump_reference_gigaam_author.py - generate GigaAM reference tensors via -the salute-developers/GigaAM author package. +the salute-developers/GigaAM author package. The v3 variants load through +gigaam.load_model; multilingual_large_ctc is built by the same package from +its pinned Hugging Face snapshot (scripts/lib/gigaam_hf.py). Single subcommand `decode` runs the full forward pass (frontend + encoder + optional decode) and dumps every stage. The `encoder` alias @@ -17,12 +19,14 @@ --audio samples/ru.wav \\ --out build/validate/gigaam/gigaam-v3-e2e-rnnt/ru/decode/ref -Dump points (match tests/tolerances/gigaam.json): +Dump points (match tests/tolerances/gigaam.json, or the variant's own +tolerance file named in its golden manifest): frontend.mel.out log-mel features after FeatureExtractor (preprocessor) enc.subsample.out after pre_encode (StridingSubsampling, conv1d factor 4) enc.pos_emb rotary [cos; sin] stacked PE tensor - enc.block.{0,7,15}.out sparse ConformerLayer outputs (16 layers total) + enc.block.{0,7,15}.out sparse ConformerLayer outputs (first, middle, + last; {0,11,23} for the 24-layer Multilingual) enc.out final encoder output (post-transpose) For RNN-T heads (rnnt, e2e_rnnt): @@ -51,19 +55,10 @@ # the on-disk contract is identical across families. _THIS_DIR = Path(__file__).resolve().parent sys.path.insert(0, str(_THIS_DIR)) +from lib.gigaam_hf import VARIANTS, load_model, variant_source # noqa: E402 from lib.ref_dump import write_tensor, write_transcript # noqa: E402 -# Branch SHA pin per variant — locked at Stage 1 intake. -_VARIANT_REVISIONS = { - "v3_e2e_rnnt": "ec1dc1f01d0d627ab2c0d3acc1e235702300d95e", # main - "v3_e2e_ctc": "cec030b4c4f35d928e4a9044a3bdb29ebd499fac", - "v3_rnnt": "c7f128b8accdd9624df905e5c2d7b7a48c27c0d8", - "v3_ctc": "15ef3b5a88da78f93134b3cb7f015c70aefa8946", -} -_HF_REPO = "ai-sage/GigaAM-v3" - - def configure_torch(args: argparse.Namespace) -> None: import torch @@ -119,32 +114,12 @@ def hook_module(module, hook: CaptureHook): def load_reference(args: argparse.Namespace): - """Load a GigaAM-v3 variant via gigaam.load_model. + """Load a GigaAM variant in F32 on `args.device` via lib.gigaam_hf. - Forces fp16_encoder=False (the package default is True; F32 is what - our intake declares and what every other reference dumper uses). - Forces device=cpu so the autocast branch in GigaAMASR.forward is - skipped — the cuda path enters torch.autocast(float16) regardless of - fp16_encoder. + Use device=cpu for reference dumps: on cuda, GigaAMASR.forward enters + torch.autocast(float16) whatever fp16_encoder is. """ - import gigaam - - if args.variant not in _VARIANT_REVISIONS: - raise SystemExit( - f"unknown variant {args.variant!r}; expected one of " - f"{sorted(_VARIANT_REVISIONS)}" - ) - print( - f"Loading gigaam.load_model({args.variant!r}, fp16_encoder=False, " - f"device={args.device!r})" - ) - model = gigaam.load_model( - args.variant, - fp16_encoder=False, - device=args.device, - ) - model.eval() - return model + return load_model(args.variant, args.device) def is_rnnt_variant(variant: str) -> bool: @@ -156,6 +131,16 @@ def is_ctc_variant(variant: str) -> bool: def cmd_decode(args: argparse.Namespace) -> int: + try: + pinned = variant_source(args.variant, args.revision) + except ValueError as e: + raise SystemExit(f"error: {e}") from None + if pinned.from_snapshot and args.model not in (None, pinned.repo_id): + raise SystemExit( + f"error: {args.variant} loads from the Hugging Face repo " + f"{pinned.repo_id} at {pinned.revision}, got --model {args.model!r}" + ) + import torch configure_torch(args) @@ -180,8 +165,8 @@ def cmd_decode(args: argparse.Namespace) -> int: source: dict[str, Any] = { "kind": "gigaam-author", "framework": "author_repo_gigaam", - "hf_repo": _HF_REPO, - "hf_revision": _VARIANT_REVISIONS[args.variant], + "hf_repo": pinned.repo_id, + "hf_revision": pinned.revision, "variant": args.variant, "device": args.device, "torch_threads": args.torch_threads, @@ -218,7 +203,8 @@ def dump(name: str, t, stage: str) -> None: layers = list(encoder.layers) n_layers = len(layers) - # Sparse layer dumps: first, mid, last. For 16-layer Conformer that's 0/7/15. + # Sparse layer dumps: first, mid, last. For 16-layer Conformer that's + # 0/7/15, for 24 layers 0/11/23. layer_idxs = sorted({0, n_layers // 2 - 1, n_layers - 1}) layer_hooks: dict[int, CaptureHook] = {} for i in layer_idxs: @@ -319,9 +305,9 @@ def dump(name: str, t, stage: str) -> None: def add_common_args(p: argparse.ArgumentParser) -> None: p.add_argument( "--variant", required=True, - choices=sorted(_VARIANT_REVISIONS), - help="GigaAM-v3 variant key (v3_e2e_rnnt, v3_e2e_ctc, v3_rnnt, " - "v3_ctc)", + choices=sorted(VARIANTS), + help="GigaAM variant key (v3_e2e_rnnt, v3_e2e_ctc, v3_rnnt, " + "v3_ctc, multilingual_large_ctc)", ) p.add_argument("--audio", required=True, help="16 kHz mono WAV path") p.add_argument("--out", required=True, help="Output directory for dumps") @@ -329,19 +315,27 @@ def add_common_args(p: argparse.ArgumentParser) -> None: p.add_argument("--torch-threads", type=int, default=1, help="torch.set_num_threads (0 = unchanged)") # Standard dumper CLI shape compatibility (validate.py always passes - # --model and optionally --language). GigaAM loads via - # gigaam.load_model(variant_key), so --model is accepted but the - # value is informational only. --language is a no-op (monolingual ru). - p.add_argument("--model", default=_HF_REPO, - help="HF repo id (informational; load path is " - "gigaam.load_model(variant_key))") + # --model and optionally --language). The v3 variants load via + # gigaam.load_model(variant_key), so for them --model is + # informational only. --language is a no-op: no GigaAM head takes a + # language input. + p.add_argument("--model", default=None, + help="HF repo id. Informational for v3 (load path is " + "gigaam.load_model(variant_key)); for " + "multilingual_large_ctc, the repo fetched at the " + "pinned commit (default: the variant's repo)") p.add_argument("--language", default=None, - help="Language hint (no-op; GigaAM is monolingual ru)") + help="Language hint (no-op; GigaAM heads take no language input)") + p.add_argument("--revision", default=None, + help="Commit the caller expects the variant to come from, " + "e.g. the golden manifest's hf_revision. The dump is " + "refused when it differs from the pin in " + "scripts/lib/gigaam_hf.py; the pin is what loads.") def main(argv: list[str] | None = None) -> int: p = argparse.ArgumentParser( - description="GigaAM-v3 reference dumper (salute-developers/GigaAM)." + description="GigaAM reference dumper (salute-developers/GigaAM)." ) sub = p.add_subparsers(dest="cmd", required=True) diff --git a/scripts/lib/gigaam_hf.py b/scripts/lib/gigaam_hf.py new file mode 100644 index 000000000..e75ab7e53 --- /dev/null +++ b/scripts/lib/gigaam_hf.py @@ -0,0 +1,191 @@ +"""Load a GigaAM variant from its pinned source, and read its mel frontend. + +`VARIANTS` is the one table of where each GigaAM variant comes from. The +converter, the reference dumper and the WER runner read it, and the golden +manifests under tests/golden/gigaam/ must name the same repo and commit. + +Two loaders build the model from the pinned `gigaam` package: + +- The v3 variants load through `gigaam.load_model`, which downloads the + checkpoint from the package's own URL list. Their pinned commit is the + ai-sage/GigaAM-v3 branch that publishes the same weights; it is recorded + in reference dumps and WER reports, and nothing downloads it. +- GigaAM Multilingual Large CTC has no `gigaam.load_model` key in the + pinned package (salute-developers/GigaAM 6e4b027c; the key arrived + upstream in 559d88d). Its Hugging Face repo publishes `config.json`, + which names `modeling_gigaam.` targets, and `pytorch_model.bin`, + which holds the weights under a `model.` prefix. This module reads both + at the pinned commit and instantiates the same classes from the pinned + package. +""" + +from __future__ import annotations + +import copy +import json +from dataclasses import dataclass +from pathlib import Path + + +@dataclass(frozen=True) +class VariantSource: + """Where a GigaAM variant's weights come from.""" + + repo_id: str + revision: str + # True: built from the Hugging Face snapshot at `revision`. + # False: loaded by gigaam.load_model; repo_id and revision are recorded only. + from_snapshot: bool + + +# gigaam variant key -> source. A branch name follows each commit. +VARIANTS: dict[str, VariantSource] = { + "v3_e2e_rnnt": VariantSource( + "ai-sage/GigaAM-v3", "ec1dc1f01d0d627ab2c0d3acc1e235702300d95e", False), # main + "v3_e2e_ctc": VariantSource( + "ai-sage/GigaAM-v3", "cec030b4c4f35d928e4a9044a3bdb29ebd499fac", False), # e2e_ctc + "v3_rnnt": VariantSource( + "ai-sage/GigaAM-v3", "c7f128b8accdd9624df905e5c2d7b7a48c27c0d8", False), # rnnt + "v3_ctc": VariantSource( + "ai-sage/GigaAM-v3", "15ef3b5a88da78f93134b3cb7f015c70aefa8946", False), # ctc + "multilingual_large_ctc": VariantSource( + "ai-sage/GigaAM-Multilingual", "3905cd51c3ed4e88c8edf33f3302969ba480a327", True), # large_ctc +} + +# Class named in config.json -> module that defines it in the gigaam package. +_AUTHOR_MODULES: dict[str, str] = { + "FeatureExtractor": "gigaam.preprocess", + "ConformerEncoder": "gigaam.encoder", + "CTCHead": "gigaam.decoder", + "CTCGreedyDecoding": "gigaam.decoding", +} + + +def variant_source(variant_key: str, revision: str | None = None) -> VariantSource: + """Return the pinned source of `variant_key`. + + `revision` is the commit a caller expects, such as a golden manifest's + `source_model.hf_revision`. Raises ValueError when it differs from the + pin, so a manifest bumped without this table fails instead of dumping + the old commit. + """ + try: + source = VARIANTS[variant_key] + except KeyError: + raise ValueError( + f"unknown GigaAM variant {variant_key!r}; expected one of {sorted(VARIANTS)}" + ) from None + if revision is not None and revision != source.revision: + raise ValueError( + f"{variant_key}: revision {revision} differs from the commit pinned in " + f"scripts/lib/gigaam_hf.py VARIANTS, {source.revision}. Change the pin " + f"and the golden manifest together." + ) + return source + + +def author_model_config(hf_config: dict) -> dict: + """Return the model cfg of a Hugging Face config.json, retargeted at the + gigaam package. + + Raises ValueError when a section names a class this module does not map, + so a new head or decoder type fails here instead of loading the wrong + module. + """ + cfg = copy.deepcopy(hf_config["cfg"]["model"]["cfg"]) + for section in ("preprocessor", "encoder", "head", "decoding"): + target = cfg[section]["_target_"] + module, _, name = target.rpartition(".") + if module != "modeling_gigaam" or name not in _AUTHOR_MODULES: + raise ValueError(f"unsupported {section} target {target!r} in config.json") + cfg[section]["_target_"] = f"{_AUTHOR_MODULES[name]}.{name}" + return cfg + + +def snapshot_dir(variant_key: str) -> Path: + """Return the snapshot directory of a `from_snapshot` variant. + + The snapshot comes from lib.hf_source.download_snapshot at the pinned + commit, which keeps each commit in its own directory and raises + LocalEntryNotFoundError offline when the pinned one is absent. + """ + from lib.hf_source import download_snapshot + + source = variant_source(variant_key) + return download_snapshot(source.repo_id, source.revision) + + +def load_snapshot_model(variant_key: str): + """Load a GigaAM ASR model in F32 on CPU from its Hugging Face snapshot.""" + import torch + from gigaam import GigaAMASR + from omegaconf import OmegaConf + + path = snapshot_dir(variant_key) + print(f"Loading {variant_key} from {path}") + hf_config = json.loads((path / "config.json").read_text(encoding="utf-8")) + asr = GigaAMASR(OmegaConf.create(author_model_config(hf_config))) + state_dict = torch.load(path / "pytorch_model.bin", map_location="cpu", weights_only=True) + # The Hugging Face wrapper prefixes every key with `model.`; the strict + # load_state_dict refuses any key left unmatched. + asr.load_state_dict({k.removeprefix("model."): v for k, v in state_dict.items()}) + return asr.float().cpu().eval() + + +def load_model(variant_key: str, device: str = "cpu"): + """Load a GigaAM variant in F32 and eval mode on `device`. + + A `from_snapshot` variant comes from its pinned Hugging Face repo and + commit. The v3 variants load with fp16_encoder=False, since the package + defaults to an F16 encoder. + """ + if variant_source(variant_key).from_snapshot: + return load_snapshot_model(variant_key).to(device) + + import gigaam + + print(f"Loading gigaam.load_model({variant_key!r}, fp16_encoder=False, device={device!r})") + return gigaam.load_model(variant_key, fp16_encoder=False, device=device).eval() + + +def frontend_params(preprocessor_cfg: dict, preprocessor) -> dict: + """Return the log-mel frontend parameters of a gigaam FeatureExtractor. + + The values come from `preprocessor`, the instantiated module, because + FeatureExtractor defaults some of them and passes no mel scale, + filterbank norm, or frequency range to torchaudio's MelSpectrogram. + Raises ValueError when `preprocessor_cfg` sets one of the keys below to + a value the module does not use (`features` for n_mels), so a GGUF never + describes a frontend the reference does not run; other config keys are + not checked. + + Keys: sample_rate, n_fft, win_length, hop_length, n_mels, center, + mel_scale, mel_norm, f_min, f_max (Hz). + """ + mel = preprocessor.featurizer[0] + params = { + "sample_rate": int(mel.sample_rate), + "n_fft": int(mel.n_fft), + "win_length": int(mel.win_length), + "hop_length": int(mel.hop_length), + "n_mels": int(mel.n_mels), + "center": bool(mel.spectrogram.center), + "mel_scale": str(mel.mel_scale.mel_scale), + "mel_norm": mel.mel_scale.norm, + "f_min": float(mel.f_min), + "f_max": float(mel.mel_scale.f_max), + } + # config.json key -> params key, where the two differ. + cfg_keys = {"features": "n_mels"} + mismatches = [ + f"{key}={preprocessor_cfg[key]!r} (module: {params[cfg_keys.get(key, key)]!r})" + for key in ("sample_rate", "n_fft", "win_length", "hop_length", "features", + "center", "mel_scale", "mel_norm", "f_min", "f_max") + if key in preprocessor_cfg and preprocessor_cfg[key] != params[cfg_keys.get(key, key)] + ] + if mismatches: + raise ValueError( + "preprocessor config disagrees with the FeatureExtractor it builds: " + + ", ".join(mismatches) + ) + return params diff --git a/scripts/lib/hf_source.py b/scripts/lib/hf_source.py index 60f6a0bef..cc9872cde 100644 --- a/scripts/lib/hf_source.py +++ b/scripts/lib/hf_source.py @@ -2,10 +2,11 @@ Every HF-sourced converter needs the same three things: decide whether a `--model` argument is a repo id or a local path, download a snapshot into -`$TRANSCRIBE_MODELS_DIR//` (falling back to the HF cache), and hand back -a local directory. This module is the single copy of that logic; before it, -`_download_snapshot` / `_looks_like_repo_id` / `hf_resolve` were duplicated -near-verbatim across a dozen converters. +`$TRANSCRIBE_MODELS_DIR//` (`@/` for a pinned commit, +falling back to the HF cache), and hand back a local directory. This +module is the single copy of that logic; before it, `_download_snapshot` / +`_looks_like_repo_id` / `hf_resolve` were duplicated near-verbatim across a +dozen converters. NeMo-sourced families (parakeet, canary) resolve checkpoints through `ASRModel.from_pretrained` instead and do not use this module. @@ -14,6 +15,7 @@ from __future__ import annotations import os +import re from pathlib import Path from lib.gguf_common import slug_from_repo_id @@ -32,30 +34,62 @@ def download_snapshot(repo_id: str, revision: str | None = None) -> Path: """Download an HF snapshot and return its local directory. When `$TRANSCRIBE_MODELS_DIR` is set the snapshot lands in - `//`; otherwise `snapshot_download` uses the default HF - cache and returns whatever local path it resolves to. Call this after the - caller has already decided `repo_id` is a repo (see `looks_like_repo_id`). + `//`, or in `/@/` when + `revision` is a commit hash, full or abbreviated; otherwise + `snapshot_download` uses the default HF cache and returns whatever local + path it resolves to. Call this after the caller has already decided + `repo_id` is a repo (see `looks_like_repo_id`). + + When the Hub is unreachable or HF_HUB_OFFLINE=1 is set, + `snapshot_download` returns a non-empty local directory without checking + what it holds. A directory per commit holds only the files + huggingface_hub downloaded at that commit, so a pinned load never reads + a file of another commit or one another tool saved beside the snapshot, + and an absent commit raises LocalEntryNotFoundError that names the + directory. The commit directories sit beside `/`, not in it, so an + unpinned load never mistakes them for its snapshot. A branch or tag + named like a hash (7 to 40 lowercase hex characters) is taken for a + commit and gets such a directory too, which holds files of more than + one commit once that ref moves. """ from huggingface_hub import snapshot_download + from huggingface_hub.errors import LocalEntryNotFoundError slug = slug_from_repo_id(repo_id) models_root = os.environ.get("TRANSCRIBE_MODELS_DIR") - local_dir = Path(models_root) / slug if models_root else None - if local_dir is not None: + local_dir = None + per_commit = bool(revision and _COMMIT_HASH.fullmatch(revision)) + if models_root: + local_dir = Path(models_root) / (f"{slug}@{revision}" if per_commit else slug) local_dir.mkdir(parents=True, exist_ok=True) if revision: print(f"Downloading {repo_id}@{revision} from Hugging Face...") else: print(f"Downloading {repo_id} from Hugging Face " f"(no revision pin; reproducibility depends on upstream)...") - resolved = snapshot_download( - repo_id=repo_id, - revision=revision, - local_dir=str(local_dir) if local_dir is not None else None, - ) + try: + resolved = snapshot_download( + repo_id=repo_id, + revision=revision, + local_dir=str(local_dir) if local_dir is not None else None, + ) + except LocalEntryNotFoundError as e: + if local_dir is None or not per_commit: + raise + raise LocalEntryNotFoundError( + f"{repo_id}@{revision} is not in {local_dir}, and HF_HUB_OFFLINE is " + f"set or the Hugging Face Hub cannot be reached. A pinned commit is read " + f"only from its own directory, not from {local_dir.parent / slug}/. Unset " + f"HF_HUB_OFFLINE if it is set, or restore network access, and rerun to " + f"download it there." + ) from e return Path(resolved) +# A full or abbreviated commit hash, as the Hub accepts in `revision`. +_COMMIT_HASH = re.compile(r"[0-9a-f]{7,40}") + + def resolve_model_dir(model_arg: str, revision: str | None = None) -> Path: """Return a local directory for `model_arg`, downloading if needed. diff --git a/scripts/lib/test_gguf_writer.py b/scripts/lib/test_gguf_writer.py index 72a4aa72d..042b7d13e 100755 --- a/scripts/lib/test_gguf_writer.py +++ b/scripts/lib/test_gguf_writer.py @@ -21,6 +21,7 @@ import sys import tempfile +import traceback from pathlib import Path import numpy as np @@ -127,6 +128,9 @@ def _main() -> int: except AssertionError as e: failed += 1 print(f"FAIL {t.__name__}: {e}") + except Exception: + failed += 1 + print(f"ERROR {t.__name__}:\n{traceback.format_exc()}") print(f"\n{len(tests) - failed}/{len(tests)} passed") return 1 if failed else 0 diff --git a/scripts/lib/test_gigaam_hf.py b/scripts/lib/test_gigaam_hf.py new file mode 100644 index 000000000..74382e2f7 --- /dev/null +++ b/scripts/lib/test_gigaam_hf.py @@ -0,0 +1,270 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = [ +# "gguf>=0.10", +# "huggingface-hub>=0.30", +# "numpy", +# ] +# /// +"""Unit tests for lib.gigaam_hf, the pinned source table and loader of the +GigaAM variants. + +Covers the parts that need neither torch nor the weights: the source table +against the golden manifests, the config retargeting, the snapshot fetch at +the pinned commit, the frontend read from a stand-in FeatureExtractor, and +the reference dumper's refusal of a revision or repo other than the pin. + +Run standalone (exit-code driven): uv run scripts/lib/test_gigaam_hf.py +Or under pytest: pytest scripts/lib/test_gigaam_hf.py +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import sys +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + +# Read by huggingface_hub at import; lib.hf_source imports it lazily. +os.environ["HF_HUB_OFFLINE"] = "1" + +from huggingface_hub.errors import LocalEntryNotFoundError # noqa: E402 + +_SCRIPTS = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(_SCRIPTS)) +from lib.gigaam_hf import ( # noqa: E402 + VARIANTS, + VariantSource, + author_model_config, + frontend_params, + snapshot_dir, + variant_source, +) + +_GOLDEN = _SCRIPTS.parent / "tests" / "golden" / "gigaam" +_ML_COMMIT = "3905cd51c3ed4e88c8edf33f3302969ba480a327" + + +def _hf_config(head_target: str = "modeling_gigaam.CTCHead") -> dict: + """config.json in the shape of the ai-sage GigaAM repos, trimmed to the + fields the loader reads or must carry through.""" + return { + "model_type": "gigaam", + "cfg": { + "model": { + "_target_": "modeling_gigaam.GigaAMASR", + "cfg": { + "model_name": "multilingual_large_ctc", + "preprocessor": {"_target_": "modeling_gigaam.FeatureExtractor", "features": 64}, + "encoder": {"_target_": "modeling_gigaam.ConformerEncoder", "n_layers": 24}, + "head": {"_target_": head_target, "num_classes": 71}, + "decoding": { + "_target_": "modeling_gigaam.CTCGreedyDecoding", + "vocabulary": [" ", "'", "a", "\u0451", "\u049b"], + }, + }, + } + }, + } + + +class SourceTableTest(unittest.TestCase): + def test_every_golden_manifest_names_the_pinned_source_of_its_variant(self): + manifests = sorted(_GOLDEN.glob("*.manifest.json")) + self.assertTrue(manifests, f"no manifests under {_GOLDEN}") + named = {} + for path in manifests: + m = json.loads(path.read_text(encoding="utf-8")) + key = m["reference"]["variant_key"] + src = m["source_model"] + named[key] = (src["hf_repo"], src["hf_revision"]) + pinned = {k: (v.repo_id, v.revision) for k, v in VARIANTS.items()} + self.assertEqual(named, pinned) + + def test_only_multilingual_large_ctc_is_built_from_its_snapshot(self): + from_snapshot = sorted(k for k, v in VARIANTS.items() if v.from_snapshot) + self.assertEqual(from_snapshot, ["multilingual_large_ctc"]) + + def test_the_pinned_revision_is_accepted(self): + self.assertEqual( + variant_source("multilingual_large_ctc", _ML_COMMIT), + VariantSource("ai-sage/GigaAM-Multilingual", _ML_COMMIT, True), + ) + + def test_a_revision_other_than_the_pin_is_refused(self): + other = "2f8a57144e6ec3adfd32fe0484d9ea9913305bc8" + with self.assertRaisesRegex(ValueError, f"{other} differs from .* {_ML_COMMIT}"): + variant_source("multilingual_large_ctc", other) + + def test_an_unknown_variant_is_refused(self): + with self.assertRaisesRegex(ValueError, "'v3_ssl'"): + variant_source("v3_ssl") + + +class AuthorModelConfigTest(unittest.TestCase): + def test_ctc_config_targets_the_gigaam_package_modules(self): + cfg = author_model_config(_hf_config()) + targets = {s: cfg[s]["_target_"] for s in ("preprocessor", "encoder", "head", "decoding")} + self.assertEqual(targets, { + "preprocessor": "gigaam.preprocess.FeatureExtractor", + "encoder": "gigaam.encoder.ConformerEncoder", + "head": "gigaam.decoder.CTCHead", + "decoding": "gigaam.decoding.CTCGreedyDecoding", + }) + self.assertEqual(cfg["decoding"]["vocabulary"], [" ", "'", "a", "\u0451", "\u049b"]) + + def test_retargeting_leaves_the_input_config_unchanged(self): + hf = _hf_config() + author_model_config(hf) + self.assertEqual(hf["cfg"]["model"]["cfg"]["head"]["_target_"], "modeling_gigaam.CTCHead") + + def test_a_head_class_without_a_mapping_is_refused(self): + with self.assertRaisesRegex(ValueError, "'modeling_gigaam.RNNTHead'"): + author_model_config(_hf_config(head_target="modeling_gigaam.RNNTHead")) + + def test_a_target_outside_modeling_gigaam_is_refused(self): + with self.assertRaisesRegex(ValueError, "'other_module.CTCHead'"): + author_model_config(_hf_config(head_target="other_module.CTCHead")) + + +def _models_dir_with(root: Path, commit: str) -> Path: + """Return a TRANSCRIBE_MODELS_DIR holding config.json of GigaAM-Multilingual + downloaded at `commit`, in the directory download_snapshot keeps for it.""" + (root / f"GigaAM-Multilingual@{commit}").mkdir(parents=True) + (root / f"GigaAM-Multilingual@{commit}" / "config.json").write_text("{}", encoding="utf-8") + return root + + +class SnapshotDirTest(unittest.TestCase): + def test_a_local_copy_of_the_pinned_commit_is_used(self): + with tempfile.TemporaryDirectory() as d: + models = _models_dir_with(Path(d), _ML_COMMIT) + with mock.patch.dict(os.environ, {"TRANSCRIBE_MODELS_DIR": str(models)}): + got = snapshot_dir("multilingual_large_ctc") + self.assertEqual(got, (models / f"GigaAM-Multilingual@{_ML_COMMIT}").resolve()) + + def test_a_local_copy_of_another_commit_is_not_used(self): + other = "2f8a57144e6ec3adfd32fe0484d9ea9913305bc8" + with tempfile.TemporaryDirectory() as d: + models = _models_dir_with(Path(d), other) + with mock.patch.dict(os.environ, {"TRANSCRIBE_MODELS_DIR": str(models)}): + with self.assertRaises(LocalEntryNotFoundError): + snapshot_dir("multilingual_large_ctc") + + +def _feature_extractor(*, center: bool = False, mel_scale: str = "htk", norm=None, f_max: float = 8000.0): + """Stand-in for gigaam's FeatureExtractor: the torchaudio MelSpectrogram + attributes frontend_params reads, at the values of the GigaAM + checkpoints.""" + # MelSpectrogram keeps f_max=None; MelScale holds the resolved value. + mel = SimpleNamespace( + sample_rate=16000, n_fft=320, win_length=320, hop_length=160, n_mels=64, + f_min=0.0, f_max=None, + spectrogram=SimpleNamespace(center=center), + mel_scale=SimpleNamespace(mel_scale=mel_scale, norm=norm, f_max=f_max), + ) + return SimpleNamespace(featurizer=[mel, SimpleNamespace()]) + + +# config.json preprocessor of the ai-sage/GigaAM-v3 branches. +_V3_PREPROCESSOR = { + "sample_rate": 16000, "features": 64, "win_length": 320, "hop_length": 160, + "mel_scale": "htk", "n_fft": 320, "mel_norm": None, "center": False, +} +# The ai-sage/GigaAM-Multilingual large_ctc preprocessor lists no mel scale or norm. +_ML_PREPROCESSOR = { + "sample_rate": 16000, "features": 64, "win_length": 320, "hop_length": 160, + "n_fft": 320, "center": False, +} +_EXPECTED_FRONTEND = { + "sample_rate": 16000, "n_fft": 320, "win_length": 320, "hop_length": 160, + "n_mels": 64, "center": False, "mel_scale": "htk", "mel_norm": None, + "f_min": 0.0, "f_max": 8000.0, +} + + +class FrontendParamsTest(unittest.TestCase): + def test_a_config_that_lists_every_value_reads_them_from_the_module(self): + self.assertEqual(frontend_params(_V3_PREPROCESSOR, _feature_extractor()), _EXPECTED_FRONTEND) + + def test_a_config_without_mel_scale_takes_the_module_scale(self): + self.assertEqual(frontend_params(_ML_PREPROCESSOR, _feature_extractor()), _EXPECTED_FRONTEND) + + def test_a_config_mel_scale_the_module_does_not_use_is_refused(self): + cfg = {**_V3_PREPROCESSOR, "mel_scale": "slaney"} + with self.assertRaisesRegex(ValueError, r"mel_scale='slaney' \(module: 'htk'\)"): + frontend_params(cfg, _feature_extractor()) + + def test_a_config_without_center_takes_the_module_value(self): + cfg = {k: v for k, v in _ML_PREPROCESSOR.items() if k != "center"} + got = frontend_params(cfg, _feature_extractor(center=True)) + self.assertEqual(got["center"], True) + + def test_the_upper_mel_frequency_comes_from_the_module(self): + got = frontend_params(_ML_PREPROCESSOR, _feature_extractor(f_max=7600.0)) + self.assertEqual((got["f_min"], got["f_max"]), (0.0, 7600.0)) + + def test_a_config_f_max_the_module_does_not_use_is_refused(self): + with self.assertRaisesRegex(ValueError, r"f_max=7600.0 \(module: 8000.0\)"): + frontend_params({**_ML_PREPROCESSOR, "f_max": 7600.0}, _feature_extractor()) + + def test_a_config_mel_count_the_module_does_not_use_is_refused(self): + with self.assertRaisesRegex(ValueError, r"features=80 \(module: 64\)"): + frontend_params({**_ML_PREPROCESSOR, "features": 80}, _feature_extractor()) + + def test_a_config_mel_norm_the_module_does_not_use_is_refused(self): + with self.assertRaisesRegex(ValueError, r"mel_norm='slaney' \(module: None\)"): + frontend_params({**_V3_PREPROCESSOR, "mel_norm": "slaney"}, _feature_extractor()) + + def test_a_config_center_the_module_does_not_use_is_refused(self): + with self.assertRaisesRegex(ValueError, r"center=False \(module: True\)"): + frontend_params(_ML_PREPROCESSOR, _feature_extractor(center=True)) + + +def _dumper(): + """Import dump_reference_gigaam_author.py, which has no package name.""" + spec = importlib.util.spec_from_file_location( + "dump_reference_gigaam_author", _SCRIPTS / "dump_reference_gigaam_author.py") + dumper = importlib.util.module_from_spec(spec) + spec.loader.exec_module(dumper) + return dumper + + +class _Loaded(Exception): + """Raised in place of loading the model, once every check has passed.""" + + +class DumperSourceTest(unittest.TestCase): + def decode(self, *extra: str) -> None: + dumper = _dumper() + with tempfile.TemporaryDirectory() as d, \ + mock.patch.object(dumper, "load_reference", side_effect=_Loaded), \ + mock.patch.dict(sys.modules, {"torch": mock.MagicMock()}): + dumper.main(["decode", "--variant", "multilingual_large_ctc", + "--audio", str(Path(d) / "a.wav"), "--out", d, *extra]) + + def test_the_pinned_repo_and_revision_reach_the_load(self): + for extra in ((), ("--model", "ai-sage/GigaAM-Multilingual", "--revision", _ML_COMMIT)): + with self.subTest(extra=extra), self.assertRaises(_Loaded): + self.decode(*extra) + + def test_a_revision_other_than_the_pin_stops_the_dump_before_loading(self): + other = "2f8a57144e6ec3adfd32fe0484d9ea9913305bc8" + with self.assertRaisesRegex(SystemExit, f"error: .*{other} differs from .* {_ML_COMMIT}"): + self.decode("--revision", other) + + def test_a_repo_other_than_the_pinned_one_stops_the_dump_before_loading(self): + with self.assertRaisesRegex( + SystemExit, "ai-sage/GigaAM-Multilingual at 3905cd51.*got --model 'ai-sage/GigaAM-v3'" + ): + self.decode("--model", "ai-sage/GigaAM-v3") + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/lib/test_hf_source.py b/scripts/lib/test_hf_source.py new file mode 100644 index 000000000..b84556160 --- /dev/null +++ b/scripts/lib/test_hf_source.py @@ -0,0 +1,113 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = [ +# "gguf>=0.10", +# "huggingface-hub>=0.30", +# "numpy", +# ] +# /// +"""Unit tests for lib.hf_source.download_snapshot with $TRANSCRIBE_MODELS_DIR +set and the Hub unreachable, where huggingface_hub returns an existing +non-empty local directory without checking what it holds. + +Run standalone (exit-code driven): uv run scripts/lib/test_hf_source.py +Or under pytest: pytest scripts/lib/test_hf_source.py +""" + +from __future__ import annotations + +import os +import re +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +# Read by huggingface_hub at import; lib.hf_source imports it lazily. +os.environ["HF_HUB_OFFLINE"] = "1" + +from huggingface_hub.errors import LocalEntryNotFoundError # noqa: E402 + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # repo scripts/ +from lib.hf_source import download_snapshot # noqa: E402 + +_PINNED = "1111111111111111111111111111111111111111" +_OTHER = "2222222222222222222222222222222222222222" + + +def _add_file(directory: Path, name: str, record_commit: str | None = None) -> None: + """Write `name` into `directory`, with a huggingface_hub download record + for `record_commit` unless it is None.""" + path = directory / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("x", encoding="utf-8") + if record_commit is not None: + meta = directory / ".cache" / "huggingface" / "download" / f"{name}.metadata" + meta.parent.mkdir(parents=True, exist_ok=True) + meta.write_text(f"{record_commit}\netag\n{path.stat().st_mtime}\n", encoding="utf-8") + + +class DownloadSnapshotLocalDirTest(unittest.TestCase): + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.models = Path(tmp.name) + self.slug_dir = self.models / "model" + env = mock.patch.dict(os.environ, {"TRANSCRIBE_MODELS_DIR": str(self.models)}) + env.start() + self.addCleanup(env.stop) + + def commit_dir(self, revision: str) -> Path: + return self.models / f"model@{revision}" + + def test_a_pinned_commit_is_read_from_its_own_directory(self): + _add_file(self.commit_dir(_PINNED), "config.json", _PINNED) + self.assertEqual(download_snapshot("org/model", _PINNED), self.commit_dir(_PINNED).resolve()) + + def test_files_in_the_repo_directory_are_not_a_snapshot_of_a_pinned_commit(self): + # A recorded config.json with an unrecorded checkpoint and a converted + # GGUF, as an earlier layout or another tool leaves in /. + _add_file(self.slug_dir, "config.json", _PINNED) + _add_file(self.slug_dir, "pytorch_model.bin") + _add_file(self.slug_dir, "model-F32.gguf") + with self.assertRaisesRegex( + LocalEntryNotFoundError, + f"is not in {re.escape(str(self.commit_dir(_PINNED)))}, .* Unset HF_HUB_OFFLINE", + ): + download_snapshot("org/model", _PINNED) + + def test_the_directory_of_another_commit_is_not_used(self): + _add_file(self.commit_dir(_OTHER), "config.json", _OTHER) + with self.assertRaisesRegex(LocalEntryNotFoundError, + f"is not in {re.escape(str(self.commit_dir(_PINNED)))}"): + download_snapshot("org/model", _PINNED) + + def test_an_abbreviated_commit_is_not_read_from_the_repo_directory(self): + _add_file(self.slug_dir, "config.json", _OTHER) + with self.assertRaisesRegex(LocalEntryNotFoundError, + f"is not in {re.escape(str(self.commit_dir('e37978b')))}"): + download_snapshot("org/model", "e37978b") + + def test_a_branch_revision_reads_the_repo_directory(self): + _add_file(self.slug_dir, "config.json", _OTHER) + self.assertEqual(download_snapshot("org/model", "main"), self.slug_dir.resolve()) + + def test_a_branch_revision_does_not_take_a_commit_directory_for_its_snapshot(self): + _add_file(self.commit_dir(_PINNED), "config.json", _PINNED) + with self.assertRaises(LocalEntryNotFoundError) as caught: + download_snapshot("org/model", "main") + self.assertNotIn("pinned commit", str(caught.exception)) + + def test_six_hex_characters_are_read_as_a_branch(self): + _add_file(self.slug_dir, "config.json", _OTHER) + self.assertEqual(download_snapshot("org/model", "abcdef"), self.slug_dir.resolve()) + + def test_forty_one_hex_characters_are_read_as_a_branch(self): + _add_file(self.slug_dir, "config.json", _OTHER) + self.assertEqual(download_snapshot("org/model", _PINNED + "1"), self.slug_dir.resolve()) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_convert_gigaam.py b/scripts/test_convert_gigaam.py new file mode 100644 index 000000000..e0fc92b90 --- /dev/null +++ b/scripts/test_convert_gigaam.py @@ -0,0 +1,130 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = [ +# "gguf>=0.10", +# "numpy", +# ] +# /// +"""Unit tests for convert-gigaam.py: the VARIANT_PROFILES table, the source +of the general.* identity block each GigaAM GGUF carries, the refusal of a +source other than the pinned repo, and the frontend values read_hparams +takes from the FeatureExtractor. + +Run standalone (exit-code driven): uv run scripts/test_convert_gigaam.py +Or under pytest: pytest scripts/test_convert_gigaam.py +""" + +from __future__ import annotations + +import importlib.util +import io +import json +import tempfile +import unittest +from contextlib import redirect_stderr +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + +_SCRIPTS = Path(__file__).resolve().parent +_GOLDEN = _SCRIPTS.parent / "tests" / "golden" / "gigaam" + +_spec = importlib.util.spec_from_file_location("convert_gigaam", _SCRIPTS / "convert-gigaam.py") +convert_gigaam = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(convert_gigaam) + + +class VariantProfilesTest(unittest.TestCase): + def test_each_slug_has_its_general_identity(self): + identity = { + slug: (p["basename"], p["version"], p["languages"]) + for slug, p in convert_gigaam.VARIANT_PROFILES.items() + } + self.assertEqual(identity, { + "gigaam-v3-e2e-rnnt": ("gigaam-v3", "v3", ["ru"]), + "gigaam-v3-e2e-ctc": ("gigaam-v3", "v3", ["ru"]), + "gigaam-v3-rnnt": ("gigaam-v3", "v3", ["ru"]), + "gigaam-v3-ctc": ("gigaam-v3", "v3", ["ru"]), + "gigaam-multilingual-large-ctc": ( + "gigaam-multilingual", None, ["ru", "en", "kk", "ky", "uz"]), + }) + + def test_every_profile_matches_the_variant_key_and_languages_of_its_golden_manifest(self): + manifests = {} + for path in sorted(_GOLDEN.glob("*.manifest.json")): + m = json.loads(path.read_text(encoding="utf-8")) + manifests[m["variant"]] = (m["reference"]["variant_key"], m["capabilities"]["languages"]) + profiles = { + slug: (p["variant_key"], p["languages"]) + for slug, p in convert_gigaam.VARIANT_PROFILES.items() + } + self.assertEqual(profiles, manifests) + + +class MainTest(unittest.TestCase): + def run_main(self, model: str, slug: str) -> tuple[int, str, list]: + """Run main() with convert() replaced, and return the exit code, + stderr, and the arguments convert() was called with.""" + err = io.StringIO() + with tempfile.TemporaryDirectory() as d, redirect_stderr(err), \ + mock.patch.object(convert_gigaam, "convert") as convert: + code = convert_gigaam.main( + ["convert-gigaam.py", model, str(Path(d) / "out.gguf"), "--repo-id", slug]) + return code, err.getvalue().strip(), [c.args[:2] for c in convert.call_args_list] + + def test_the_pinned_repo_is_converted(self): + self.assertEqual( + self.run_main("ai-sage/GigaAM-Multilingual", "gigaam-multilingual-large-ctc"), + (0, "", [("multilingual_large_ctc", "gigaam-multilingual-large-ctc")]), + ) + + def test_a_v3_variant_takes_any_model_argument(self): + self.assertEqual( + self.run_main("any/informational-id", "gigaam-v3-ctc"), + (0, "", [("v3_ctc", "gigaam-v3-ctc")]), + ) + + def test_another_repo_or_a_local_directory_is_refused_before_converting(self): + for model in ("ai-sage/GigaAM-v3", tempfile.gettempdir()): + with self.subTest(model=model): + self.assertEqual( + self.run_main(model, "gigaam-multilingual-large-ctc"), + (2, "error: multilingual_large_ctc converts from the Hugging Face repo " + "ai-sage/GigaAM-Multilingual at 3905cd51c3ed4e88c8edf33f3302969ba480a327, " + f"got {model!r}", []), + ) + + +class ReadHparamsTest(unittest.TestCase): + def test_the_frontend_values_come_from_the_feature_extractor(self): + # A CTC config with the keys read_hparams reads, and the Multilingual + # preprocessor, which lists no center, mel scale, norm, or range. + config = { + "preprocessor": {"sample_rate": 16000, "features": 64, "win_length": 320, + "hop_length": 160, "n_fft": 320}, + "encoder": {"n_layers": 24, "d_model": 1024, "n_heads": 16, + "ff_expansion_factor": 4, "conv_kernel_size": 5, + "subsampling_factor": 4, "subs_kernel_size": 5, + "pos_emb_max_len": 5000, "feat_in": 64, + "self_attention_model": "rotary", "conv_norm_type": "layer_norm"}, + "head": {"_target_": "gigaam.decoder.CTCHead", "feat_in": 1024, "num_classes": 71}, + } + # Stand-in for FeatureExtractor: torchaudio's MelSpectrogram keeps + # f_max=None and its MelScale holds the resolved value. + mel = SimpleNamespace( + sample_rate=16000, n_fft=320, win_length=320, hop_length=160, n_mels=64, + f_min=20.0, f_max=None, + spectrogram=SimpleNamespace(center=True), + mel_scale=SimpleNamespace(mel_scale="htk", norm=None, f_max=7600.0), + ) + model = SimpleNamespace(preprocessor=SimpleNamespace(featurizer=[mel, SimpleNamespace()])) + hp = convert_gigaam.read_hparams(config, model) + frontend = {k: hp[k] for k in ("fe_center", "fe_f_min", "fe_f_max", "fe_mel_norm")} + self.assertEqual(frontend, { + "fe_center": True, "fe_f_min": 20.0, "fe_f_max": 7600.0, "fe_mel_norm": "htk", + }) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/validate.py b/scripts/validate.py index 80132f015..4d19bc67d 100644 --- a/scripts/validate.py +++ b/scripts/validate.py @@ -411,9 +411,10 @@ def cmd_ref(args: argparse.Namespace) -> int: # Pin the HF revision when the manifest declares one and the # family's dumper accepts --revision. The dumper itself ignores - # --revision when --model resolves to a local directory. + # --revision when --model resolves to a local directory. The gigaam + # dumper loads its own pinned commit and refuses a different one. hf_revision = (manifest.get("source_model") or {}).get("hf_revision") - if hf_revision and args.family in ("qwen3_asr", "granite_nar", "granite5_ctc", "ecapa_tdnn"): + if hf_revision and args.family in ("qwen3_asr", "granite_nar", "granite5_ctc", "ecapa_tdnn", "gigaam"): common_args += ["--revision", str(hf_revision)] # Forward any manifest-declared dumper args verbatim. Used today diff --git a/scripts/wer/run_reference_gigaam_author.py b/scripts/wer/run_reference_gigaam_author.py index e7cbfe70c..c085b8456 100644 --- a/scripts/wer/run_reference_gigaam_author.py +++ b/scripts/wer/run_reference_gigaam_author.py @@ -2,13 +2,12 @@ """ run_reference_gigaam_author.py — GigaAM author-repo batch transcribe. -Loads a GigaAM-v3 variant via `gigaam.load_model(variant, fp16_encoder=False)` -once and runs greedy decode over a WER manifest. Writes run.py-compatible -JSONL so scripts/wer/score.py can score the output the same way it scores -the C++ port's report. +Loads a GigaAM variant once through scripts/lib/gigaam_hf.py, the loader +the reference dumper uses, and runs greedy decode over a WER manifest. +Writes run.py-compatible JSONL so scripts/wer/score.py can score the +output the same way it scores the C++ port's report. -Mirrors the dump-time setup (scripts/dump_reference_gigaam_author.py): -forces fp16_encoder=False so the encoder runs in fp32 on CPU. +The model runs in F32 on CPU, as in the reference dumps. Usage (from repo root): @@ -27,13 +26,8 @@ import time from pathlib import Path -# Same per-variant revision pins as the dumper. -_VARIANT_REVISIONS = { - "v3_e2e_rnnt": "ec1dc1f01d0d627ab2c0d3acc1e235702300d95e", - "v3_e2e_ctc": "ec1dc1f01d0d627ab2c0d3acc1e235702300d95e", - "v3_rnnt": "ec1dc1f01d0d627ab2c0d3acc1e235702300d95e", - "v3_ctc": "ec1dc1f01d0d627ab2c0d3acc1e235702300d95e", -} +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # repo scripts/ +from lib.gigaam_hf import VARIANTS, load_model # noqa: E402 def main() -> int: @@ -46,8 +40,8 @@ def main() -> int: p.add_argument("--out", type=Path, required=True, help="Output JSONL path (run.py-compatible).") p.add_argument("--variant", required=True, - choices=sorted(_VARIANT_REVISIONS.keys()), - help="GigaAM variant key passed to gigaam.load_model.") + choices=sorted(VARIANTS), + help="GigaAM variant key of scripts/lib/gigaam_hf.py VARIANTS.") p.add_argument("--torch-threads", type=int, default=1, help="torch intra-op threads for deterministic runs.") p.add_argument("--limit", type=int, default=0, @@ -64,20 +58,8 @@ def main() -> int: if args.torch_threads > 0: torch.set_num_threads(args.torch_threads) - import gigaam - - print( - f"loading: gigaam.load_model({args.variant!r}, " - f"fp16_encoder=False, device='cpu')", - flush=True, - ) t0 = time.monotonic() - model = gigaam.load_model( - args.variant, - fp16_encoder=False, - device="cpu", - ) - model.eval() + model = load_model(args.variant) load_ms = (time.monotonic() - t0) * 1000 with open(args.manifest) as f: @@ -98,7 +80,7 @@ def main() -> int: "load_ms": round(load_ms, 1), "framework": "gigaam-author", "model": args.variant, - "model_revision": _VARIANT_REVISIONS[args.variant], + "model_revision": VARIANTS[args.variant].revision, }) + "\n") fout.flush() diff --git a/tests/golden/gigaam/gigaam-multilingual-large-ctc.manifest.json b/tests/golden/gigaam/gigaam-multilingual-large-ctc.manifest.json new file mode 100644 index 000000000..80fd30503 --- /dev/null +++ b/tests/golden/gigaam/gigaam-multilingual-large-ctc.manifest.json @@ -0,0 +1,61 @@ +{ + "schema": "transcribe-golden-manifest-v1", + "family": "gigaam", + "variant": "gigaam-multilingual-large-ctc", + "source_model": { + "hf_repo": "ai-sage/GigaAM-Multilingual", + "hf_revision": "3905cd51c3ed4e88c8edf33f3302969ba480a327" + }, + "reference": { + "kind": "author_repo_gigaam", + "source": "https://github.com/salute-developers/GigaAM", + "revision": "6e4b027c6fb554e09e8b9059b757a175295ab879", + "entrypoint": "scripts/dump_reference_gigaam_author.py", + "variant_key": "multilingual_large_ctc", + "dump_args": ["--variant", "multilingual_large_ctc"] + }, + "expected_dtype": "float32", + "dtype_source": "manual", + "frontend": { + "sample_rate": 16000, + "n_mels": 64, + "hop_length": 160, + "fft_size": 320, + "win_length": 320, + "window": "hann_periodic", + "normalization": "none", + "center": false, + "mel_filterbank_norm": "htk", + "log_scale": "log(clamp(x, 1e-9, 1e9))" + }, + "tokenizer_summary": { + "type": "other", + "vocab_size": 70, + "special_tokens": { + "blank": 70 + }, + "_charwise_vocab_size": 70 + }, + "capabilities": { + "languages": [ + "ru", + "en", + "kk", + "ky", + "uz" + ], + "language_detection": false, + "translation": false, + "timestamps": [], + "streaming": false, + "speaker_diarization": false + }, + "head_type": "ctc", + "tolerance_file": "tests/tolerances/gigaam-multilingual-large-ctc.json", + "cases": [ + { + "audio": "ru", + "language": "ru" + } + ] +} diff --git a/tests/tolerances/gigaam-multilingual-large-ctc.json b/tests/tolerances/gigaam-multilingual-large-ctc.json new file mode 100644 index 000000000..659d1798a --- /dev/null +++ b/tests/tolerances/gigaam-multilingual-large-ctc.json @@ -0,0 +1,78 @@ +{ + "_comment": [ + "GigaAM Multilingual Large CTC per-tensor tolerances for compare_tensors.py.", + "Separate from gigaam.json because this variant dumps blocks 0/11/23 of", + "a 24-layer, d_model=1024 encoder instead of blocks 0/7/15.", + "", + "CORRECTNESS REGIME", + "- Reference: salute-developers/GigaAM @ 6e4b027c (author repo) classes,", + " built by scripts/lib/gigaam_hf.py from ai-sage/GigaAM-Multilingual", + " @ 3905cd51 (branch large_ctc), F32, CPU, one torch thread.", + " Dumper: scripts/dump_reference_gigaam_author.py.", + " Bridge: the repo's own modeling_gigaam.py on the same weights gives", + " bit-identical frontend.mel.out, enc.out and ctc.log_probs on ru.wav.", + "- C++: ggml CPU fp32 compute, weights stored as F32 in GGUF,", + " production C++ mel frontend, --backend cpu --threads 1.", + "- Case: samples/ru.wav (manifest case `ru`).", + "", + "ENTRY SOURCING (finalized)", + "- Recipe: tests/tolerances/gigaam.json's, max(1.5 x observed, 1e-6),", + " rounded up to two significant digits. Observed drift is 8 to 100 times", + " below the Stage 2 magnitude budget (1e-4 x p99_abs, 1e-5 x rms), so", + " the recipe tightens every entry and widens none. Observed max_abs /", + " mean_abs of the finalizing run; two earlier runs agree to the two", + " significant digits they recorded:", + " frontend.mel.out 1.631e-04 / 9.997e-07", + " enc.subsample.out 9.766e-04 / 2.966e-05", + " enc.block.0.out 7.248e-05 / 2.345e-06", + " enc.block.11.out 4.101e-05 / 1.655e-06", + " enc.block.23.out 1.383e-05 / 9.583e-07", + " enc.out 1.383e-05 / 9.583e-07", + " ctc.logits.raw 6.199e-05 / 3.444e-06", + " ctc.log_probs 9.441e-05 / 9.125e-06", + "", + "DRIFT SOURCES (named)", + "- frontend.mel.out: FFT and mel-filterbank reduction order, as in", + " gigaam.json; largest at low-power bins where log() amplifies it.", + "- enc.subsample.out: conv1d reduction order over 1024 channels on", + " activations up to ~474. Injecting the reference mel", + " (TRANSCRIBE_MEL_FROM_REF) leaves max_abs at 9.8e-04, so the frontend", + " does not cause it.", + "- enc.block.*: attention, GLU and LayerNorm accumulation; drift shrinks", + " from block 0 to block 23 as each LayerNorm renormalizes the residual.", + "- ctc.logits.raw / ctc.log_probs: one 1x1 conv over d_model=1024,", + " bounded by enc.out drift." + ], + "frontend.mel.out": { + "max_abs": 0.00025, + "mean_abs": 1.5e-06 + }, + "enc.subsample.out": { + "max_abs": 0.0015, + "mean_abs": 4.5e-05 + }, + "enc.block.0.out": { + "max_abs": 0.00011, + "mean_abs": 3.6e-06 + }, + "enc.block.11.out": { + "max_abs": 6.2e-05, + "mean_abs": 2.5e-06 + }, + "enc.block.23.out": { + "max_abs": 2.1e-05, + "mean_abs": 1.5e-06 + }, + "enc.out": { + "max_abs": 2.1e-05, + "mean_abs": 1.5e-06 + }, + "ctc.logits.raw": { + "max_abs": 9.3e-05, + "mean_abs": 5.2e-06 + }, + "ctc.log_probs": { + "max_abs": 0.00015, + "mean_abs": 1.4e-05 + } +}