Skip to content

Fix #2461: [Bug] Content between embedder limit and chat_window_max_tokens is silently stor - #2606

Merged
endxxxx merged 5 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2461-20261008063231554
Nov 18, 2026
Merged

endxxxx merged 5 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2461-20261008063231554

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

Restores the proposal from #2462 as an open PR against main; its former base branch, dev-v2.0.36, is no longer available. This proposal still needs its implementation revised before merging.

Source head: 243ef50068787b1d00a6a45599d3d3b60b3f9481. No new source-code changes are introduced by restoring this proposal. Earlier discussion and evidence remain in #2462; validation claims in the original description refer to the earlier snapshot.

Outstanding review scope

reader仅按chat_window_max_tokens拆分、不结合embedder预算,fallback还会吞掉单项embedding失败,问题范围仍值得修复。但当前UniversalAPI/Ark/Ollama已经调用BaseEmbedder截断,所以旧描述“该区间内容必然无向量”不能直接成立。应按真实embedder配置重做预算/尾部覆盖和失败契约验证。


Original proposal

Description

Fixed the silent no-vector write for memory items whose token count falls in (embedder per-item limit, chat_window_max_tokens] (issue #2461). Such an item was previously neither split (the splitter only fired above chat_window_max_tokens, hard-coded to 1024 and never passed by either api/config.py mem_reader assembly site) nor embeddable (it exceeded the embedder's limit). /product/add still returned 200, but the item was persisted with metadata.embedding=None and vector_sync != "success", and search_by_embedding filters on vector_sync == "success" — so the memory was permanently invisible to semantic search. A secondary cause: _split_large_memory_item trusted chunker output and never re-checked the token budget per chunk, and the sentence chunker (chonkie) does not split CJK punctuation (it can also raise on an incompatible installed version), in which case the except branch returned the over-budget item unchanged.

The fix adds an explicit, env-configurable chat_window_max_tokens field on BaseMemReaderConfig (MEM_READER_CHAT_WINDOW_MAX_TOKENS, default 1024 so existing deployments are unchanged), wired into all three mem_reader assembly sites via APIConfig.get_chat_window_max_tokens() with fault-tolerant parsing. The effective split budget is now min(chat_window_max_tokens, embedder per-item limit). A new _chunk_within_budget helper re-checks every chunker-produced chunk and hard-splits any that still exceeds the budget — preferring a punctuation boundary (CJK and Latin terminators) and falling back to a binary-search character window — and a chunker that raises or yields nothing now hard-splits instead of passing the over-budget item through. _concat_multi_modal_memories uses the effective budget for both the split trigger and the window accumulator, plus a guard that emits over-budget items directly rather than aggregating them into a larger window. Finally, the per-item embedding retry (and the base reader's _make_memory_item path) truncates to budget before calling the embedder, so one over-limit item can no longer leave a whole batch without vectors.

Verification: a new 18-case regression suite (tests/mem_reader/test_embed_budget_split.py) passes, covering budget derivation, punctuation-free CJK splitting within budget, boundary preference, empty/failed chunker fallback, contiguous re-indexing, the 700-token in-window case, and the mixed-batch embedding fallback. Affected suites (mem_reader, configs, api, memories, mem_cube) report 341 passed with 4 pre-existing failures confirmed on the untouched baseline (markitdown extra not installed; unrelated kv-cache API). ruff check and ruff format both clean. An end-to-end reproduction using the real SentenceChunker and tiktoken counter with a 512-token embedder limit turns the issue's 780-token Chinese paragraph into chunks of 512 + 268 tokens, all embedded, while the same script's pre-fix branch reproduces the reported embedded=[False].

Out of scope: backfilling vectors for memories already stored without one.

Related Issue (Required): Fixes #2461

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Not run; documentation-only change.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@WeiminLee please review this PR.

Reviewer Checklist

autodev-bot and others added 5 commits October 8, 2026 16:04
Items whose token count fell in (embedder limit, chat_window_max_tokens]
were neither split nor embeddable, so /product/add returned 200 while the
item was stored with metadata.embedding=None and vector_sync != "success",
permanently invisible to semantic search (issue MemTensor#2461).

- expose chat_window_max_tokens as an explicit, env-configurable field on
  BaseMemReaderConfig (MEM_READER_CHAT_WINDOW_MAX_TOKENS), default 1024
- derive the split budget as min(window, embedder per-item limit)
- re-check chunker output and hard-split any over-budget chunk, preferring a
  punctuation boundary and falling back to a binary-search character window;
  a chunker that fails or yields nothing now hard-splits instead of passing
  the over-budget item through
- truncate to budget before the per-item embedding retry so one over-limit
  item cannot leave a whole batch without vectors
Addresses the review findings on the MemTensor#2461 fix:

- multi_modal_struct: the window loop and the overlap trim still called
  `_count_tokens` directly, so a reader built without a live tokenizer made
  its budget decision with `_count_tokens_safe` and then raised
  `AttributeError` a few lines later, dropping the whole scene. Both call
  sites now use the safe counter.
- multi_modal_struct: `_chunk_within_budget` wrapped the per-chunk token
  check and hard split in the same `try` as the chunker, so a tokenizer
  error was reported as "Chunker failed" and the already-validated chunks
  were thrown away. The guard now covers only `chunker.chunk(text)`.
- simple_struct: `_find_hard_split_index` could return `best + 1`, letting
  a terminator just past the budget be kept on the left side. That made the
  reported prefix `budget + 1` tokens; `_truncate_to_budget` used it
  directly, so the last guard before `embedder.embed([payload])` could still
  hand the provider an over-limit payload — the exact rejection this change
  set out to remove. The scan now stays inside the fitting prefix.
- simple_struct: drop the unreachable range guard in the scan loop.
@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 18, 2026 07:08
@endxxxx
endxxxx merged commit 39ddfa0 into MemTensor:dev-v2.0.37 Nov 18, 2026
12 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants