Skip to content

[Bug] CharacterTextChunker loses URLs when splitting their placeholders #2510

Open

Description

@DURUII

Problem

A complete URL in the input disappears after CharacterTextChunker splits the text. Internal placeholder fragments appear in its place.

Here is a simple example for illustration.

Input (Chinese text meaning “Installation docs: [URL] Please check.”):

安装文档:https://example.com/install 请查看。

Actual output with chunk_size=10, chunk_overlap=0:

安装文档:__URL
_0__
请查看
。

Expected: The complete URL remains in at least one chunk, without generated placeholder fragments. SimpleTextSplitter preserves it with the same input and settings:

安装文档:
https://example.com/install
请查看。

Reproduction

from memos.chunkers.charactertext_chunker import CharacterTextChunker

splitter = CharacterTextChunker(chunk_size=10, chunk_overlap=0)
print(splitter.chunk("安装文档:https://example.com/install 请查看。"))

Environment: MemOS commit a7367d07e55db61099f7b4e2c1108bc5831a24f3, Python 3.13.5, langchain-text-splitters 1.0.0, macOS.

This component-level reproduction uses unchanged chunker source and the real LangChain splitter. The local harness bypassed package-level startup and substituted standard logging. The snippet shows normal imports for a configured development environment. These are non-default chunk settings; this does not establish URL loss under the file parser's default settings or through the full API.

Cause and Related Fix

MemOS replaces the URL with __URL_0__ before splitting. LangChain splits that marker, so MemOS cannot restore the URL afterward.

PR #2228 already protects placeholder boundaries in SimpleTextSplitter, but does not cover CharacterTextChunker. This path needs equivalent URL-integrity protection and a regression test.

Possible Approaches

  • Reuse the URL-safe fallback for URL-containing inputs. Use SimpleTextSplitter with the same size and overlap for those inputs, keeping the existing path for text without URLs. This reuses existing protection, but changes chunk boundaries for URL-containing text.
  • Keep recursive splitting and make placeholders indivisible. Adjust the MemOS wrapper so chunk boundaries and overlap cannot cut through generated placeholders. This aims to retain the current splitting strategy, but requires more boundary handling.

Either approach needs a policy for URLs longer than chunk_size: preserving the full URL may produce an oversized chunk. These are proposed directions for discussion.

Activity

  1. added
    area:coreMOS 编排层 / 框架底座 / 跨模块问题
    status:needs-triageNeeds initial triage | 需要初步判断 & 问题复现
    on Oct 17, 2026
  2. github-actions commented on Nov 18, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  3. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:coreMOS 编排层 / 框架底座 / 跨模块问题status:needs-triageNeeds initial triage | 需要初步判断 & 问题复现status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions