Skip to content

fix: SimpleTextSplitter fallback 在文件解析链路中报错:缺少 protect_urls #2115

Closed

Description

@Lilili-04

Pre-submission checklist | 提交前检查

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
  • This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题

Bug Description | 问题描述

在处理会进入多模态文件解析链路的 /product/add 请求时,MemOS 可能在文本切分阶段出现如下错误:

[FileContentParser] Error splitting text: 'SimpleTextSplitter' object has no attribute 'protect_urls'

问题发生在 fallback 文本切分器路径中:当正常的 CharacterTextChunker / MarkdownChunker 初始化失败时,代码会退回使用 SimpleTextSplitter。但是 SimpleTextSplitter 内部调用了 self.protect_urls(text) 和 self.restore_urls(...),而这两个方法实际定义在 BaseChunker 中,SimpleTextSplitter 既没有继承 BaseChunker,也没有自己实现这两个方法。

相关代码路径:

  • src/memos/mem_reader/read_multi_modal/base.py
    • _split_text() 调用 get_text_splitter(),随后执行 splitter.chunk(text)。
  • src/memos/mem_reader/read_multi_modal/utils.py
    • 如果 CharacterTextChunker / MarkdownChunker 初始化失败,get_text_splitter() 会返回 SimpleTextSplitter。
  • src/memos/chunkers/simple_chunker.py
    • SimpleTextSplitter._simple_split_text() 调用了 self.protect_urls(text) 和 self.restore_urls(...)。
  • src/memos/chunkers/base.py
    • protect_urls() 和 restore_urls() 定义在 BaseChunker 中。

因此,一旦 fallback 路径被真正触发,当前实现就会因为缺少方法而无法正常工作。

How to Reproduce | 如何重现

是在运行环境中让 langchain_text_splitters 不可用,或者让 CharacterTextChunker / MarkdownChunker 初始化失败,从而触发文件解析链路中的 SimpleTextSplitter fallback。

最小复现代码:

from memos.chunkers.simple_chunker import SimpleTextSplitter

splitter = SimpleTextSplitter(chunk_size=512, chunk_overlap=128)
splitter.chunk("This is a test document with a URL: https://example.com/path/to/resource")

期望结果:

文本可以被正常切分,并且 URL 不会被错误切断。

实际结果:

AttributeError: 'SimpleTextSplitter' object has no attribute 'protect_urls'

Environment | 环境信息

该问题是在 MemOS 从 ECS 迁移到 ACK 后观察到的。

  • 部署目标:ACK
  • API 路径:/product/add
  • 解析链路:memos.mem_reader.read_multi_modal
  • ACK 日志中观察到的出现次数:约 5,813 次
  • 观察时间范围:2026-07-12 15:09:14 到 2026-07-14 15:14:50
  • ECS 对比:ECS 日志中未观察到该错误

该错误分布在多个 ACK Pod IP 上,因此更像是镜像、依赖或运行时代码版本一致性问题,而不是单个请求或单台机器异常。

示例日志:

2026-07-14 14:06:27,820 | ... | path=/product/add | env=prod | user_type=normal | user_name=*** | memos.mem_reader.read_multi_modal.base - ERROR - base.py:275 - _split_text - [FileContentParser] Error splitting text: 'SimpleTextSplitter' object has no attribute 'protect_urls'

Additional Context | 其他信息

No response

Willingness to Implement | 实现意愿

  • I'm willing to implement this myself | 我愿意自己解决
  • I would like someone else to implement this | 我希望其他人来解决

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    area:coreMOS 编排层 / 框架底座 / 跨模块问题
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Jul 16, 2026
  2. Memtensor-AI commented on Jul 16, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: e062b03201f62d58
    Working branch: bugfix/autodev-2115-20260716073650486
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  3. self-assigned this
    on Jul 16, 2026
  4. Memtensor-AI commented on Jul 16, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task e062b03201f62d58 completed.

    Summary: Fixes issue #2115: SimpleTextSplitter fallback in the multi-modal file-parsing pipeline was raising AttributeError: 'SimpleTextSplitter' object has no attribute 'protect_urls'. Root cause: SimpleTextSplitter._simple_split_text() calls self.protect_urls / self.restore_urls, but those helpers live on BaseChunker and SimpleTextSplitter does not inherit from it. On ACK (where langchain_text_splitters is missing and the fallback is actually exercised) this produced ~5.8k noisy log rows and silently degraded chunking to a single-chunk pass-through.

    The fix extracts protect_urls / restore_urls into a small URLProtectionMixin in src/memos/chunkers/base.py; BaseChunker now inherits it (behavioural no-op — same public API), and SimpleTextSplitter in src/memos/chunkers/simple_chunker.py also inherits it, so the two paths share one URL regex and one placeholder scheme. SimpleTextSplitter's constructor signature and chunk() -> list[str] return type are unchanged; no factory / config / OpenAPI changes are required.

    Verification: 7 new regression tests in tests/chunkers/test_simple_chunker.py (short/long input with URL, empty / whitespace input, no-URL text, parametrised chunk sizes) — all 7 failed before the fix with the exact AttributeError, all 7 pass after. Broader pytest tests/chunkers/ tests/mem_reader/ → 63 pass; the 2 unrelated failures (test_doc_local_file, test_doc_mixed) fail identically on the pristine branch with the change stashed and are caused by the sandbox missing markitdown[all]. ruff check + ruff format --check are green on all 3 changed files.

    Working branch pushed to origin; opsp task + verification report archived to memos-autodev-specs main. Ready for reviewer @WeiminLee.

    Base branch: main
    Branch: bugfix/autodev-2115-20260716073650486
    Commit: 74d8598
    PR: #2116
    Assigned to: @NydiaChung
    Reviewers: @WeiminLee

  5. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jul 16, 2026
  6. Memtensor-AI commented on Aug 12, 2026

    @Memtensor-AI
    Collaborator

    Resolved by #2200, #2228. The merged fix is already included in main and released in v2.0.29, so closing this issue as completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审area:coreMOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions