Skip to content

fix(core): accept split tokenizer files for Sherpa Qwen - #1056

Merged
H-Chris233 merged 1 commit into
Open-Less:betafrom
H-Chris233:codex/fix-beta1-sherpa-tokenizer-layout
Sep 10, 2026
Merged

fix(core): accept split tokenizer files for Sherpa Qwen#1056
H-Chris233 merged 1 commit into
Open-Less:betafrom
H-Chris233:codex/fix-beta1-sherpa-tokenizer-layout

Conversation

@H-Chris233

Copy link
Copy Markdown
Collaborator

Summary

  • accept both tokenizer/tokenizer.json and the supported tokenizer/vocab.json + tokenizer/merges.txt layout for Sherpa Qwen archives
  • reuse the same completeness rule for installed-state checks, legacy migration, and post-extraction validation
  • add regression coverage for Beta.1 upgrades and fresh archive extraction

Root cause

The Core ModelStore migration narrowed the previous Sherpa Qwen tokenizer contract to tokenizer/tokenizer.json only. Existing models and the release archive can use the split BPE files, so Beta.1 rejected valid models before runtime loading.

Tests

  • cargo test -p openless-core (795 passed, 1 ignored)
  • cargo test -p openless-core model_store::tests:: (23 passed)
  • rustfmt --edition 2021 --check crates/openless-core/src/model_store.rs
  • git diff --check upstream/beta...HEAD

@H-Chris233
H-Chris233 merged commit 615d6ac into Open-Less:beta Sep 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant