Repository navigation
fix(chunkers): make SimpleTextSplitter fallback URL-safe - #2200
Conversation
`SimpleTextSplitter._simple_split_text()` called `self.protect_urls` and `self.restore_urls`, methods defined only on `BaseChunker`. Since `SimpleTextSplitter` does not inherit `BaseChunker`, every call raised `AttributeError: 'SimpleTextSplitter' object has no attribute 'protect_urls'`. The multi-modal file-parsing pipeline swallowed the error and fell back to returning the whole text as a single chunk, producing ~5.8k noisy log rows on ACK where langchain_text_splitters is missing and the fallback branch is actually exercised. Extract the URL protect/restore helpers into a small `URLProtectionMixin` in `chunkers/base.py`; have both `BaseChunker` and `SimpleTextSplitter` inherit it. This preserves BaseChunker's public API (mixin methods are inherited transparently), keeps SimpleTextSplitter's constructor and return type unchanged, and shares a single URL regex between the two paths. Add regression tests in tests/chunkers/test_simple_chunker.py covering short/long input, empty input, no-URL text, and parametrised (chunk_size, overlap) combinations to ensure the fallback never raises again.
Address OCR review on #2116: the placeholder-leak assertion in tests/chunkers/test_simple_chunker.py hardcoded the string `'__URL_'`, which duplicates an implementation detail of `URLProtectionMixin.protect_urls` (formatted as `f'__URL_{len(url_map)}__'`). If the prefix ever changed in `base.py`, the assertion would silently keep passing while no longer catching real placeholder leaks. Expose the prefix as a class-level constant `URLProtectionMixin._URL_PLACEHOLDER_PREFIX = "__URL_"`, use it inside `protect_urls`, and import it in the test so the two paths stay in sync automatically. No behavior change: placeholders keep the same textual form (`__URL_<n>__`), so both the current base chunker and the fallback `SimpleTextSplitter` produce identical output to before.
🤖 Open Code ReviewTarget: PR #2200 🔍 OpenCodeReview found 3 issue(s) in this PR. 1.
|
✅ Automated Test Results: PASSEDAll tests passed (9/9 executed). memos_python_core/changed-repo-python: 9/9. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-9f2181dc08e1834a-20260803203805: 70/72 passed, 2 failed — these do NOT affect the PR verdict; review the branch manually. Branch: |
## Summary - Make SimpleTextSplitter inherit BaseChunker so the fallback has protect_urls and restore_urls, matching the existing downstream class layout. - Keep URL placeholders atomic at chunk boundaries and stop after the final chunk so overlap does not emit repeated shrinking tails. - Prevent silent text corruption when user input or a URL contains placeholder-like text such as __URL_0__. A collision-free namespace is selected for those inputs, and restoration uses one regex pass so restored URLs are not processed again. - Reject invalid chunk_size and chunk_overlap combinations that would lose text or create excessive duplicate chunks. - Move the expanded fallback regression suite to tests/chunkers/test_simple_chunker_fallback.py so it does not collide with an existing downstream test of the same component. Related issues: #2115, #2200 ## Compatibility - Normal input keeps the existing short placeholder form and therefore preserves the established chunk-boundary behavior. - No new dependency is introduced. - A squash/cherry-pick simulation of the final diff onto the current downstream baseline completed without conflicts in the chunker source or tests. ## How Has This Been Tested? - make format: passed; Ruff reported 616 files unchanged. - Full tests/chunkers suite: 24 passed. - Forced fallback integration smoke test with langchain_text_splitters unavailable: passed, URL remained intact and no placeholder leaked. - Downstream merged-tree tests: 23 passed, including the existing downstream SimpleTextSplitter tests and this PR regression suite. - URL-heavy benchmark after the single-pass restoration change: 10,000 URLs in 0.048s and 20,000 URLs in 0.169s on the local test machine. ## Type of change - [x] Bug fix (non-breaking change which fixes an issue) - [x] Refactor (keeps URL protection shared through BaseChunker and aligns the fallback hierarchy) ## Checklist - [x] I have performed a self-review of my own code | 我已自行检查了自己的代码 - [x] I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释 - [x] I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常 - [x] Documentation change is not applicable; this fixes internal fallback behavior without changing the public API. - [x] Related issues are linked above. - [ ] I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人 ## Reviewer Checklist - [ ] Required checks passed - [x] Tests have been provided
Description
Fixes the dependency-free
SimpleTextSplitterfallback used by the multi-modal file parsing pipeline.The fallback called
protect_urls()andrestore_urls()without inheriting their implementation, causingAttributeErrorwhenever the optional LangChain-backed chunkers were unavailable. This change:URLProtectionMixin;BaseChunkerandSimpleTextSplitterreuse the same implementation;This is the clean replacement for #2116, based directly on
dev-v2.0.29with only 3 related files changed.Related Issue (Required): Fixes #2115
Type of change
How Has This Been Tested?
Local verification:
ruff format --checkon all changed Python filesruff checkon all changed Python filesgit diff --checkSimpleTextSplittertest cases via a lightweight harness loading the actual changed modulesThe repository's full pytest environment was not available locally because the base interpreter lacked
concurrent-log-handler; GitHub CI should run the complete matrix.Checklist
@WeiminLee please review this clean replacement for #2116.
Reviewer Checklist
SimpleTextSplitterfallback 在文件解析链路中报错:缺少protect_urls#2115