Repository navigation
Wasm tokenizers: HF and Tekken tokenizers fail to load #25102
Description
🐛 Describe the bug
Summary: the Wasm tokenizer bindings (extension/wasm/tokenizers) can't load HF or Tekken tokenizers. That includes the repo's own test files.
What happens
- Built as the README says, in node:
| Class | Test file (extension/llm/tokenizers/test/resources/) |
Result |
|---|---|---|
HFTokenizer |
test_hf_tokenizer.json |
throws a raw C++ exception (a number) |
Tekken |
test_tekken.json |
Failed to load tokenizer |
Tiktoken |
test_tiktoken_tokenizer.model |
loads |
SPTokenizer |
test_sentencepiece.model |
loads |
Two causes
- No exception catching. The tokenizers library uses try/catch in normal operation (for example
HFTokenizerprobing optional JSON keys and{"String": ...}patterns). Emscripten doesn't run catch blocks unless the code is built with-fexceptions, and nothing sets it. So the throw escapes to JS. - No lookahead regex engine. HF and Tekken patterns use lookahead, which needs
regex_lookahead. The top-level CMake setsSUPPORT_REGEX_LOOKAHEADonly for the LLM runners, not forEXECUTORCH_BUILD_TOKENIZERS_WASM. Log:RE2 could not compile the pattern and no fallback regex engine is linked.
Repro
const mod = await loadTokenizers(); // tokenizers_wasm_lib built per the README
const t = new mod.HFTokenizer();
t.load(new Uint8Array(fs.readFileSync('extension/llm/tokenizers/test/resources/test_hf_tokenizer.json')));
// -> throws a number (C++ exception pointer)Expected: all four load, as they do natively.
Fix: PR coming next.
Versions
- ExecuTorch
mainat d3369c4 (also in v1.6.0). Emscripten 4.0.10, node. - Found while running ExecuTorch's LLM runner in the browser from Flutter.
collect_env output
Collecting environment information...
PyTorch version: 2.14.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 27.2.1 (arm64)
GCC version: Could not collect
Clang version: 21.0.0 (clang-2100.4.1.1)
CMake version: version 4.5.0
Libc version: N/A
Python version: 3.10.11 (v3.10.11:7d4cc5aa85, Apr 4 2023, 19:05:19) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-27.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A
CPU:
Apple M2 Pro
Versions of relevant libraries:
[pip3] executorch==1.5.0
[pip3] flake8==6.1.0
[pip3] flake8-breakpoint==1.1.0
[pip3] flake8-bugbear==24.4.26
[pip3] flake8-comprehensions==3.14.0
[pip3] flake8-plugin-utils==1.3.3
[pip3] flake8-pyi==23.5.0
[pip3] mypy==1.14.1
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.2.6
[pip3] pytorch_tokenizers==1.5.0
[pip3] torch==2.14.0
[pip3] torchao==0.18.0+git5f2baf9d5
[pip3] torchvision==0.29.0
[conda] Could not collect
Activity