Skip to content

fix(chunkers): make SimpleTextSplitter fallback URL-safe (#2200) - #2228

Merged
CarltonXiang merged 7 commits into
mainfrom
dev-v2.0.29
Aug 12, 2026
Merged

CarltonXiang merged 7 commits into
mainfrom
dev-v2.0.29

Conversation

@hijzy

@hijzy hijzy commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Make SimpleTextSplitter inherit BaseChunker so the fallback has protect_urls and restore_urls, matching the existing downstream class layout.
  • Keep URL placeholders atomic at chunk boundaries and stop after the final chunk so overlap does not emit repeated shrinking tails.
  • Prevent silent text corruption when user input or a URL contains placeholder-like text such as URL_0. A collision-free namespace is selected for those inputs, and restoration uses one regex pass so restored URLs are not processed again.
  • Reject invalid chunk_size and chunk_overlap combinations that would lose text or create excessive duplicate chunks.
  • Move the expanded fallback regression suite to tests/chunkers/test_simple_chunker_fallback.py so it does not collide with an existing downstream test of the same component.

Related issues: #2115, #2200

Compatibility

  • Normal input keeps the existing short placeholder form and therefore preserves the established chunk-boundary behavior.
  • No new dependency is introduced.
  • A squash/cherry-pick simulation of the final diff onto the current downstream baseline completed without conflicts in the chunker source or tests.

How Has This Been Tested?

  • make format: passed; Ruff reported 616 files unchanged.
  • Full tests/chunkers suite: 24 passed.
  • Forced fallback integration smoke test with langchain_text_splitters unavailable: passed, URL remained intact and no placeholder leaked.
  • Downstream merged-tree tests: 23 passed, including the existing downstream SimpleTextSplitter tests and this PR regression suite.
  • URL-heavy benchmark after the single-pass restoration change: 10,000 URLs in 0.048s and 20,000 URLs in 0.169s on the local test machine.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (keeps URL protection shared through BaseChunker and aligns the fallback hierarchy)

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • Documentation change is not applicable; this fixes internal fallback behavior without changing the public API.
  • Related issues are linked above.
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

Reviewer Checklist

  • Required checks passed
  • Tests have been provided

* fix(chunkers): make SimpleTextSplitter fallback URL-safe (#2115)

`SimpleTextSplitter._simple_split_text()` called `self.protect_urls` and
`self.restore_urls`, methods defined only on `BaseChunker`. Since
`SimpleTextSplitter` does not inherit `BaseChunker`, every call raised
`AttributeError: 'SimpleTextSplitter' object has no attribute
'protect_urls'`. The multi-modal file-parsing pipeline swallowed the
error and fell back to returning the whole text as a single chunk,
producing ~5.8k noisy log rows on ACK where langchain_text_splitters is
missing and the fallback branch is actually exercised.

Extract the URL protect/restore helpers into a small `URLProtectionMixin`
in `chunkers/base.py`; have both `BaseChunker` and `SimpleTextSplitter`
inherit it. This preserves BaseChunker's public API (mixin methods are
inherited transparently), keeps SimpleTextSplitter's constructor and
return type unchanged, and shares a single URL regex between the two
paths.

Add regression tests in tests/chunkers/test_simple_chunker.py covering
short/long input, empty input, no-URL text, and parametrised
(chunk_size, overlap) combinations to ensure the fallback never raises
again.

* refactor(chunkers): expose URL placeholder prefix as class constant

Address OCR review on #2116: the placeholder-leak assertion in
tests/chunkers/test_simple_chunker.py hardcoded the string `'__URL_'`,
which duplicates an implementation detail of
`URLProtectionMixin.protect_urls` (formatted as
`f'__URL_{len(url_map)}__'`). If the prefix ever changed in `base.py`,
the assertion would silently keep passing while no longer catching real
placeholder leaks.

Expose the prefix as a class-level constant
`URLProtectionMixin._URL_PLACEHOLDER_PREFIX = "__URL_"`, use it inside
`protect_urls`, and import it in the test so the two paths stay in sync
automatically.

No behavior change: placeholders keep the same textual form
(`__URL_<n>__`), so both the current base chunker and the fallback
`SimpleTextSplitter` produce identical output to before.

* fix(chunkers): keep URL placeholders atomic at split boundaries

---------

Co-authored-by: MemOS AutoDev <autodev@memtensor.local>
@Memtensor-AI Memtensor-AI added area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 6, 2026
@Memtensor-AI
Memtensor-AI requested a review from WeiminLee August 6, 2026 11:10
@Memtensor-AI

Memtensor-AI commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2228
Task: 79d8da615a6ed8fa
Base: main
Head: dev-v2.0.29

🔍 OpenCodeReview found 1 issue(s) in this PR.

⚠️ 3 warning(s) occurred during review.


1. tests/embedders/test_cache.py (L211-L216)

Using assert inside the subprocess script to validate the ImportError message is unsafe: Python strips assert statements when run with the -O (optimized) flag. If the child process runs in optimized mode, this check silently disappears and the script exits 0 even if the error message contains no mention of cachetools, making the test a false positive.

Replace with an explicit conditional raise:

except ImportError as exc:
    if "cachetools" not in str(exc):
        raise AssertionError(f"Expected 'cachetools' in error message, got: {exc}")
💡 Suggested Change

Before:

        try:
            CachingEmbedder(object())
        except ImportError as exc:
            assert "cachetools" in str(exc)
        else:
            raise AssertionError("CachingEmbedder should require cachetools when instantiated")

After:

        try:
            CachingEmbedder(object())
        except ImportError as exc:
            if "cachetools" not in str(exc):
                raise AssertionError(f"Expected 'cachetools' in error message, got: {exc}")
        else:
            raise AssertionError("CachingEmbedder should require cachetools when instantiated")

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (9/9 executed). memos_python_core/changed-repo-python: 9/9. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-477e71d1dfcfc2e0-20260806193914: 103/104 passed, 1 failed — these do NOT affect the PR verdict; review the branch manually.

Branch: dev-v2.0.29

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 6, 2026
@Memtensor-AI Memtensor-AI added status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 and removed status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 labels Aug 6, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (21/21 executed). memos_python_core/changed-repo-python: 21/21. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-64dcd4c62d980dd4-20260806204020: 65/65 passed — these do NOT affect the PR verdict; review the branch manually.

Branch: dev-v2.0.29

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 6, 2026
@Memtensor-AI Memtensor-AI added status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 and removed status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 labels Aug 11, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (21/21 executed). memos_python_core/changed-repo-python: 21/21. Duration: 5s

Branch: dev-v2.0.29

@Memtensor-AI Memtensor-AI added area:model llm + embedder + reranker and removed area:core MOS 编排层 / 框架底座 / 跨模块问题 labels Aug 11, 2026
@Memtensor-AI
Memtensor-AI requested a review from endxxxx August 11, 2026 11:24
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (31/31 executed). memos_python_core/changed-repo-python: 31/31. Duration: 9s

Branch: dev-v2.0.29

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 11, 2026
@MatthewZhuang
MatthewZhuang self-requested a review August 11, 2026 12:16
hijzy and others added 2 commits August 12, 2026 16:29
## Description

Update the package version to v2.0.29 in `pyproject.toml` and
`src/memos/__init__.py`. No runtime behavior or dependencies change.

Related Issue: N/A

## Type of change

- [x] Refactor (does not change functionality, e.g. code style
improvements, linting)

## How Has This Been Tested?

- `make format` could not run because Poetry is not installed in the
current environment.
- No unit tests were added because this is a version-only metadata
change.

## Checklist

- [x] I have performed a self-review of my own code | 我已自行检查了自己的代码
- [x] I have commented my code in hard-to-understand areas | N/A:
metadata-only change
- [x] I have added tests that prove my fix is effective or that my
feature works | N/A: metadata-only change
- [x] I have created related documentation issue/PR in MemOS-Docs (if
applicable) | N/A
- [x] I have linked the issue to this PR (if applicable) | N/A
- [ ] I have mentioned the person who will review this PR | Reviewer not
yet assigned

## Reviewer Checklist

- [ ] Made sure checks passed
- [ ] Tests have been provided
@Memtensor-AI Memtensor-AI added area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 and removed status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 labels Aug 12, 2026
@Memtensor-AI
Memtensor-AI requested a review from endxxxx August 12, 2026 08:41
@CarltonXiang
CarltonXiang merged commit e86b5d8 into main Aug 12, 2026
32 of 34 checks passed
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: NO TEST SCOPE

Automated tests were not run because the changed files do not map to an executable test scope.

Details: No executable test scope maps to the changed files. Automated tests were not run; add env.yaml source_mapping + execution for this path family, then rerun. Changed files: (none detected)
Manual review or env.yaml source_mapping/execution coverage is required before merge.

Branch: dev-v2.0.29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core MOS 编排层 / 框架底座 / 跨模块问题 area:model llm + embedder + reranker status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants