Skip to content

Pull requests: huggingface/tokenizers.js

New pull request New
5 Open 20 Closed
5 Open 20 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix kimi linear tokenizer regex
#35 opened Jul 20, 2026 by xenova Collaborator Loading…
15
Improve Unigram tokenization performance for long inputs
#34 opened Jul 16, 2026 by bepsvpt Loading…
Bump the actions group across 1 directory with 3 updates dependencies Pull requests that update a dependency file github_actions Pull requests that update GitHub Actions code
#33 opened Jul 14, 2026 by dependabot Bot Loading…
Add offset tracking to Encoding
#30 opened Jun 11, 2026 by anidoesdev Loading…
3
feat: add offset mapping support for WordPiece tokenizers
#17 opened Dec 11, 2025 by dkhokhlov Loading…
1
1
ProTip! Adding no:label will show everything without a label.