[abdelaziz-mahdy]

Summary

Fixes #25102.

Bug: the Wasm tokenizer bindings can't load HF or Tekken tokenizers. That includes test_hf_tokenizer.json and test_tekken.json from this repo.

Two causes, two fixes

Cause	Fix
The tokenizers library uses try/catch in normal operation, but Emscripten runs no catch blocks without -fexceptions.	extension/wasm/tokenizers/CMakeLists.txt compiles tokenizers and regex_lookahead with -fexceptions, and passes it on to the final link through tokenizers_wasm.
HF and Tekken patterns use lookahead, which needs regex_lookahead. It was only enabled for the LLM runners.	The top-level CMakeLists.txt also sets SUPPORT_REGEX_LOOKAHEAD for EXECUTORCH_BUILD_TOKENIZERS_WASM.

Native builds don't change: both settings apply only to the Wasm tokenizers build.

Test plan

Built tokenizers_wasm_lib exactly as the README describes (Emscripten 4.0.10, Release), then loaded the repo's test tokenizers in node and encoded "Hello world":

Build	HFTokenizer	Tekken	Tiktoken / SPTokenizer
main	C++ exception	load fails	load
-fexceptions only	fails (lookahead)	fails (lookahead)	load
lookahead only	C++ exception	loads	load
this PR	loads	loads	load

So both changes are needed.

Repro script (node)

const fs = require('fs');
const loadTokenizers = require(process.argv[2]); // tokenizers_wasm_lib.js
const res = 'extension/llm/tokenizers/test/resources/';
(async () => {
  const mod = await loadTokenizers();
  for (const [cls, file] of [['HFTokenizer', 'test_hf_tokenizer.json'], ['Tiktoken', 'test_tiktoken_tokenizer.model'],
                             ['SPTokenizer', 'test_sentencepiece.model'], ['Tekken', 'test_tekken.json']]) {
    const t = new mod[cls]();
    try {
      t.load(new Uint8Array(fs.readFileSync(res + file)));
      console.log(cls, 'loaded', Array.from(t.encode('Hello world', 0, 0), String));
    } catch (e) {
      console.log(cls, 'FAILED', typeof e === 'number' ? 'C++ exception' : e.message);
    }
  }
})();

The Wasm tokenizers have no CI test today. I can add a jest test like extension/wasm/test has for the runtime, if you'd like it in this PR.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25103

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 29 Awaiting Approval

As of commit 74fb21b with merge base d3369c4 ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build Aarch64 Linux CUDA Wheels (gh)
• Build Aarch64 Linux Native Wheels (gh)
• Build Aarch64 Linux Wheels (gh)
• Build documentation (gh)
• Build Linux CUDA Wheels (gh)
• Build Linux Native Wheels (gh)
• Build Linux Wheels (gh)
• Build macOS Wheels (gh)
• Build Presets (gh)
• Build Windows CUDA Wheels (gh)
• Build Windows Wheels (gh)
• Cadence Build & Test (gh)
• Core AI (gh)
• Lint (gh)
• periodic (gh)
• pull (gh)
• Test ARM Backend (gh)
• Test CoreML Backend (gh)
• Test Cortex-M Backend (gh)
• Test CPU Backend (gh)
• Test Native Engines (gh)
• Test NXP Backend (gh)
• Test QNN Backend (gh)
• Test QNN Windows MSVC build (gh)
• Test ROCm AOTI (gh)
• Test Vulkan Backend (gh)
• Test Vulkan Backend (specialized runners) (gh)
• Test WebGPU Backend (gh)
• Test XNNPACK Backend (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.