Repository navigation
Description
Problem
A complete URL in the input disappears after CharacterTextChunker splits the text. Internal placeholder fragments appear in its place.
Here is a simple example for illustration.
Input (Chinese text meaning “Installation docs: [URL] Please check.”):
安装文档:https://example.com/install 请查看。
Actual output with chunk_size=10, chunk_overlap=0:
安装文档:__URL
_0__
请查看
。
Expected: The complete URL remains in at least one chunk, without generated placeholder fragments. SimpleTextSplitter preserves it with the same input and settings:
安装文档:
https://example.com/install
请查看。
Reproduction
from memos.chunkers.charactertext_chunker import CharacterTextChunker
splitter = CharacterTextChunker(chunk_size=10, chunk_overlap=0)
print(splitter.chunk("安装文档:https://example.com/install 请查看。"))Environment: MemOS commit a7367d07e55db61099f7b4e2c1108bc5831a24f3, Python 3.13.5, langchain-text-splitters 1.0.0, macOS.
This component-level reproduction uses unchanged chunker source and the real LangChain splitter. The local harness bypassed package-level startup and substituted standard logging. The snippet shows normal imports for a configured development environment. These are non-default chunk settings; this does not establish URL loss under the file parser's default settings or through the full API.
Cause and Related Fix
MemOS replaces the URL with __URL_0__ before splitting. LangChain splits that marker, so MemOS cannot restore the URL afterward.
PR #2228 already protects placeholder boundaries in SimpleTextSplitter, but does not cover CharacterTextChunker. This path needs equivalent URL-integrity protection and a regression test.
Possible Approaches
- Reuse the URL-safe fallback for URL-containing inputs. Use
SimpleTextSplitterwith the same size and overlap for those inputs, keeping the existing path for text without URLs. This reuses existing protection, but changes chunk boundaries for URL-containing text. - Keep recursive splitting and make placeholders indivisible. Adjust the MemOS wrapper so chunk boundaries and overlap cannot cut through generated placeholders. This aims to retain the current splitting strategy, but requires more boundary handling.
Either approach needs a policy for URLs longer than chunk_size: preserving the full URL may produce an oversized chunk. These are proposed directions for discussion.
Activity
fix(chunker): preserve URLs in CharacterTextChunker (#2510) (#2515)
github-actions commentedon Nov 18, 2026
This issue has been automatically marked as stale due to inactivity.