Add nemo-text-processing-lightning: compiled tagger, 4-9x normalization - #478

Closed

ryanleary wants to merge 23 commits into

NVIDIA:mainfrom

ryanleary:pr6-nemo-fst

Commits on Aug 19, 2026

Replace setup.py packaging with pyproject.toml and add uv
ryanleary
committed
7f2796a
Move the package under src/
ryanleary
committed
c2ebfd8
Add nemo-fst, a uv workspace member for lookahead composition
ryanleary
committed
25aeb42
Add integration and benchmark tests, drop milestone labels from comments
ryanleary
committed
a625389
Sweep input length in the end-to-end benchmark
ryanleary
committed
2b015b4
Benchmark across density and length, measure the tagging share
ryanleary
committed
49e0044
Build OpenFst statically, package wheels, and wire in an opt-in fast tagger
ryanleary
committed
c78a2fb
Use the fast tagger automatically, and record both readings where ties exist
ryanleary
committed
c1377d3
Build on macOS, and say what to do when OpenFst is missing
ryanleary
committed
ee0b774
Default to OpenFst 1.8.4, and patch 1.8.3 for Clang
ryanleary
committed
ac8ad4b
Require OpenFst 1.8.4, and probe the linker for symbol-hiding flags
ryanleary
committed
44475e7
Give nemo-fst a dev group so uv run finds pytest
ryanleary
committed
13fec86
Document which environment gives a full nemo-fst test run
ryanleary
committed
383c1b4
Test what can be tested without pynini
ryanleary
committed
21e6be7

Commits on Aug 20, 2026

Add toy grammars so the oracle-free tests need no grammar cache
ryanleary
committed
b45d919
Check the export table and dependencies on macOS too
ryanleary
committed
c12de06
Fix the linker probe, and verify the export table at build time
ryanleary
committed
828c5c7
Record macOS as verified
ryanleary
committed
140f356
Ship the files the sdist needs to build itself
ryanleary
committed
79daeb7
Cache grammars by default, and build them on demand
ryanleary
committed
4ec93db
Support every language's tagger, and scope the default cache by version
ryanleary
committed
219460c
Rename the runtime package to nemo-text-processing-lightning
ryanleary
committed
eb3324a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot]
committed
816aad2