Skip to content

feat: support configurable FTS5 tokenizer for better CJK search #1961

Description

@7889545

Description

The current FTS5 tables use the built-in tokenizer, which works well for Latin scripts but has limitations for Chinese text:

  1. 2-char Chinese words (早报, 你好, 配置) fall below the trigram window (3-char minimum) and cannot be matched by FTS5
  2. Mixed ASCII+CJK tokens (C盘, API配置) fail to split correctly in
  3. No semantic segmentation — trigram treats text as character sequences, missing word boundaries (e.g., 早报 = early+report, not random bigrams)

Current Workaround

We modified to relax minimum token length:

// Changed from TRIGRAM_MIN(3) to allow 1-char CJK
if (r.length >= 1) expanded.push(r);

This works short-term but introduces noise (高频单字 like 的/是/了 enter FTS).

Proposed Solution

Add support for configurable FTS5 tokenizers, specifically:

  1. jieba-based tokenizer — Chinese word segmentation before FTS5 insertion
  2. Config option — storage.ftsTokenizer: 'trigram' | 'jieba' | 'custom'
  3. Write-time segmentation — Pre-segment text with jieba before inserting into FTS5 tables
  4. Query-time segmentation — Apply same segmentation to search queries in prepareFtsMatch()

Implementation Sketch

// config.yaml
storage:
  ftsTokenizer: jieba  // or 'trigram' (default)

// keyword.ts — prepareFtsMatch()
if (config.storage.ftsTokenizer === 'jieba') {
  tokens = jieba.cut(query);  // word-level segmentation
} else {
  tokens = trigramExtract(query);  // current behavior
}

Related Issues

Environment

  • MemOS: v2.0.20
  • OS: Windows 11
  • Language: TypeScript (memos-local-plugin)
  • Agent: Hermes Agent

Willingness to Implement

  • I'm willing to submit a PR if the maintainers are interested in this direction

Activity

  1. added
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    and removed
    types:enhancementNew feature or improvement | 新功能或改进
    on Jul 7, 2026
  2. removed
    area:databasegraph_db + vector_db | 图数据库与向量数据库
    on Jul 13, 2026
  3. Memtensor-AI commented on Jul 13, 2026

    @Memtensor-AI
    Collaborator

    Resolved by #2040. The merged fix is already included in main and released in v2.0.23, so closing this issue as completed.

  4. angarchie commented on Aug 3, 2026

    @angarchie
    Collaborator

    Hey @7889545! Thanks for the contribution — already solved the issue, thanks for helping us to improve our project! 📈

    Btw if you enjoyed contributing and want to stay connected with the project, we run a Discord server for contributors, a chill space to discuss features, ask questions, or pick up new issues  👉🏻 MemOS Discord Server (Contributor Exclusive)

    Thanks again for shipping this, and happy MemOSing 🧠✨

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:pluginOpenClaw & Hermesstatus:in-progressSomeone or AI is working on it | 人工或 AI 正在处理

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions