Skip to content

fix(chunkers): make SimpleTextSplitter fallback URL-safe - #2200

Merged
MatthewZhuang merged 3 commits into
dev-v2.0.29from
fix/2115-simple-text-splitter-dev-v2.0.29
Aug 3, 2026
Merged

MatthewZhuang merged 3 commits into
dev-v2.0.29from
fix/2115-simple-text-splitter-dev-v2.0.29

Conversation

@MatthewZhuang

Copy link
Copy Markdown
Collaborator

Description

Fixes the dependency-free SimpleTextSplitter fallback used by the multi-modal file parsing pipeline.

The fallback called protect_urls() and restore_urls() without inheriting their implementation, causing AttributeError whenever the optional LangChain-backed chunkers were unavailable. This change:

  • extracts the shared URL helpers into URLProtectionMixin;
  • makes BaseChunker and SimpleTextSplitter reuse the same implementation;
  • keeps URL placeholders atomic when a raw chunk boundary crosses them;
  • prevents the final overlap from being emitted repeatedly as shrinking suffix chunks;
  • adds focused regression coverage for fallback, URL, boundary, and overlap behavior.

This is the clean replacement for #2116, based directly on dev-v2.0.29 with only 3 related files changed.

Related Issue (Required): Fixes #2115

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test
  • Test Script Or Test Steps

Local verification:

  • ruff format --check on all changed Python files
  • ruff check on all changed Python files
  • git diff --check
  • 9 focused SimpleTextSplitter test cases via a lightweight harness loading the actual changed modules
  • 264 URL-boundary property smoke cases across chunk sizes, overlaps, and URL positions

The repository's full pytest environment was not available locally because the base interpreter lacked concurrent-log-handler; GitHub CI should run the complete matrix.

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective
  • I have linked the issue to this PR
  • I have mentioned the person who will review this PR

@WeiminLee please review this clean replacement for #2116.

Reviewer Checklist

MemOS AutoDev and others added 3 commits August 3, 2026 20:19
`SimpleTextSplitter._simple_split_text()` called `self.protect_urls` and
`self.restore_urls`, methods defined only on `BaseChunker`. Since
`SimpleTextSplitter` does not inherit `BaseChunker`, every call raised
`AttributeError: 'SimpleTextSplitter' object has no attribute
'protect_urls'`. The multi-modal file-parsing pipeline swallowed the
error and fell back to returning the whole text as a single chunk,
producing ~5.8k noisy log rows on ACK where langchain_text_splitters is
missing and the fallback branch is actually exercised.

Extract the URL protect/restore helpers into a small `URLProtectionMixin`
in `chunkers/base.py`; have both `BaseChunker` and `SimpleTextSplitter`
inherit it. This preserves BaseChunker's public API (mixin methods are
inherited transparently), keeps SimpleTextSplitter's constructor and
return type unchanged, and shares a single URL regex between the two
paths.

Add regression tests in tests/chunkers/test_simple_chunker.py covering
short/long input, empty input, no-URL text, and parametrised
(chunk_size, overlap) combinations to ensure the fallback never raises
again.
Address OCR review on #2116: the placeholder-leak assertion in
tests/chunkers/test_simple_chunker.py hardcoded the string `'__URL_'`,
which duplicates an implementation detail of
`URLProtectionMixin.protect_urls` (formatted as
`f'__URL_{len(url_map)}__'`). If the prefix ever changed in `base.py`,
the assertion would silently keep passing while no longer catching real
placeholder leaks.

Expose the prefix as a class-level constant
`URLProtectionMixin._URL_PLACEHOLDER_PREFIX = "__URL_"`, use it inside
`protect_urls`, and import it in the test so the two paths stay in sync
automatically.

No behavior change: placeholders keep the same textual form
(`__URL_<n>__`), so both the current base chunker and the fallback
`SimpleTextSplitter` produce identical output to before.
@Memtensor-AI Memtensor-AI added area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 3, 2026
@Memtensor-AI
Memtensor-AI requested a review from WeiminLee August 3, 2026 12:30
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2200
Task: 9f2181dc08e1834a
Base: dev-v2.0.29
Head: fix/2115-simple-text-splitter-dev-v2.0.29

🔍 OpenCodeReview found 3 issue(s) in this PR.


1. src/memos/chunkers/simple_chunker.py (L42-L45)

When a chunk boundary falls inside a URL placeholder and placeholder_start - start <= chunk_overlap, _align_end_to_placeholder unconditionally returns placeholder_end, which can exceed start + chunk_size with no upper bound. For a long URL whose placeholder is, say, 40 characters wide, the emitted chunk can silently overshoot chunk_size by up to len(placeholder) - 1 characters. More critically, if a very long URL placeholder straddles a chunk boundary and placeholder_end itself is still in the middle of the text, the next chunk's next_start = max(start + 1, placeholder_end - chunk_overlap) may again land inside the same placeholder — triggering the same expansion on the next iteration and causing start to creep forward one character at a time through the placeholder, producing O(placeholder_length) near-duplicate micro-chunks until the placeholder is finally cleared.

Consider bounding the expansion or, at minimum, advancing next_start past placeholder_end when the end was adjusted to it:

💡 Suggested Change

Before:

            if placeholder_start < end < placeholder_end:
                if placeholder_start - start > chunk_overlap:
                    return placeholder_start
                return placeholder_end

After:

            if placeholder_start < end < placeholder_end:
                if placeholder_start - start > chunk_overlap:
                    return placeholder_start
                # Extend to include the full placeholder; the caller should
                # advance start past placeholder_end to avoid re-entering it.
                return placeholder_end

2. tests/chunkers/test_simple_chunker.py (L79)

The expected value ["X" * 100, "X" * 70] was verified against the current implementation by tracing the algorithm:

  • Iteration 1: start=0, end=100. No separator found in "X"*150, so end stays 100. Chunk = "X"*100. next_start = max(1, 100-20) = 80.
  • Iteration 2: start=80, end=min(80+100,150)=150 (last chunk, no separator search). Chunk = "X"*70. Break.

The assertion is correct and deterministic. No issue here — this is a valid regression guard.

💡 Suggested Change

Before:

    assert chunks == ["X" * 100, "X" * 70]

After:

    assert chunks == ["X" * 100, "X" * 70]  # verified against _simple_split_text: start=80 on second pass

3. tests/chunkers/test_simple_chunker.py (L70)

The upper bound < 10 is far too loose to be a meaningful regression guard. For a ~240-character input ('A'*95 + url(24 chars) + ' ' + 'B'*120) with chunk_size=100 and chunk_overlap=20, a correct implementation produces at most 3–4 chunks. A degenerate implementation that produces 9 near-empty chunks would still pass this assertion, defeating the stated purpose of verifying "sufficient progress".

Consider tightening the bound to reflect the expected chunk count for this specific input (e.g., <= 4).

💡 Suggested Change

Before:

    assert len(chunks) < 10, "splitter made insufficient progress around the URL boundary"

After:

    assert len(chunks) <= 4, "splitter made insufficient progress around the URL boundary"

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (9/9 executed). memos_python_core/changed-repo-python: 9/9. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-9f2181dc08e1834a-20260803203805: 70/72 passed, 2 failed — these do NOT affect the PR verdict; review the branch manually.

Branch: fix/2115-simple-text-splitter-dev-v2.0.29

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 3, 2026
@MatthewZhuang
MatthewZhuang merged commit 5990b76 into dev-v2.0.29 Aug 3, 2026
19 checks passed
@MatthewZhuang
MatthewZhuang deleted the fix/2115-simple-text-splitter-dev-v2.0.29 branch August 3, 2026 13:14
CarltonXiang added a commit that referenced this pull request Aug 12, 2026
## Summary

- Make SimpleTextSplitter inherit BaseChunker so the fallback has
protect_urls and restore_urls, matching the existing downstream class
layout.
- Keep URL placeholders atomic at chunk boundaries and stop after the
final chunk so overlap does not emit repeated shrinking tails.
- Prevent silent text corruption when user input or a URL contains
placeholder-like text such as __URL_0__. A collision-free namespace is
selected for those inputs, and restoration uses one regex pass so
restored URLs are not processed again.
- Reject invalid chunk_size and chunk_overlap combinations that would
lose text or create excessive duplicate chunks.
- Move the expanded fallback regression suite to
tests/chunkers/test_simple_chunker_fallback.py so it does not collide
with an existing downstream test of the same component.

Related issues: #2115, #2200

## Compatibility

- Normal input keeps the existing short placeholder form and therefore
preserves the established chunk-boundary behavior.
- No new dependency is introduced.
- A squash/cherry-pick simulation of the final diff onto the current
downstream baseline completed without conflicts in the chunker source or
tests.

## How Has This Been Tested?

- make format: passed; Ruff reported 616 files unchanged.
- Full tests/chunkers suite: 24 passed.
- Forced fallback integration smoke test with langchain_text_splitters
unavailable: passed, URL remained intact and no placeholder leaked.
- Downstream merged-tree tests: 23 passed, including the existing
downstream SimpleTextSplitter tests and this PR regression suite.
- URL-heavy benchmark after the single-pass restoration change: 10,000
URLs in 0.048s and 20,000 URLs in 0.169s on the local test machine.

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)
- [x] Refactor (keeps URL protection shared through BaseChunker and
aligns the fallback hierarchy)

## Checklist

- [x] I have performed a self-review of my own code | 我已自行检查了自己的代码
- [x] I have commented my code in hard-to-understand areas |
我已在难以理解的地方对代码进行了注释
- [x] I have added tests that prove my fix is effective or that my
feature works | 我已添加测试以证明我的修复有效或功能正常
- [x] Documentation change is not applicable; this fixes internal
fallback behavior without changing the public API.
- [x] Related issues are linked above.
- [ ] I have mentioned the person who will review this PR | 我已提及将审查此 PR
的人

## Reviewer Checklist

- [ ] Required checks passed
- [x] Tests have been provided
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core MOS 编排层 / 框架底座 / 跨模块问题 status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants