Repository navigation
Closed
Labels
ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审AI-created PR is ready for review | AI 生成的 PR 已等待评审area:coreMOS 编排层 / 框架底座 / 跨模块问题MOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理Someone or AI is working on it | 人工或 AI 正在处理types:bugSomething isn't working | 功能异常Something isn't working | 功能异常
Description
Pre-submission checklist | 提交前检查
- I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
- I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
- This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题
Bug Description | 问题描述
在处理会进入多模态文件解析链路的 /product/add 请求时,MemOS 可能在文本切分阶段出现如下错误:
[FileContentParser] Error splitting text: 'SimpleTextSplitter' object has no attribute 'protect_urls'
问题发生在 fallback 文本切分器路径中:当正常的 CharacterTextChunker / MarkdownChunker 初始化失败时,代码会退回使用 SimpleTextSplitter。但是 SimpleTextSplitter 内部调用了 self.protect_urls(text) 和 self.restore_urls(...),而这两个方法实际定义在 BaseChunker 中,SimpleTextSplitter 既没有继承 BaseChunker,也没有自己实现这两个方法。
相关代码路径:
src/memos/mem_reader/read_multi_modal/base.py_split_text()调用get_text_splitter(),随后执行splitter.chunk(text)。
src/memos/mem_reader/read_multi_modal/utils.py- 如果
CharacterTextChunker/MarkdownChunker初始化失败,get_text_splitter()会返回SimpleTextSplitter。
- 如果
src/memos/chunkers/simple_chunker.pySimpleTextSplitter._simple_split_text()调用了self.protect_urls(text)和self.restore_urls(...)。
src/memos/chunkers/base.pyprotect_urls()和restore_urls()定义在BaseChunker中。
因此,一旦 fallback 路径被真正触发,当前实现就会因为缺少方法而无法正常工作。
How to Reproduce | 如何重现
是在运行环境中让 langchain_text_splitters 不可用,或者让 CharacterTextChunker / MarkdownChunker 初始化失败,从而触发文件解析链路中的 SimpleTextSplitter fallback。
最小复现代码:
from memos.chunkers.simple_chunker import SimpleTextSplitter
splitter = SimpleTextSplitter(chunk_size=512, chunk_overlap=128)
splitter.chunk("This is a test document with a URL: https://example.com/path/to/resource")期望结果:
文本可以被正常切分,并且 URL 不会被错误切断。
实际结果:
AttributeError: 'SimpleTextSplitter' object has no attribute 'protect_urls'
Environment | 环境信息
该问题是在 MemOS 从 ECS 迁移到 ACK 后观察到的。
- 部署目标:ACK
- API 路径:
/product/add - 解析链路:
memos.mem_reader.read_multi_modal - ACK 日志中观察到的出现次数:约 5,813 次
- 观察时间范围:
2026-07-12 15:09:14到2026-07-14 15:14:50 - ECS 对比:ECS 日志中未观察到该错误
该错误分布在多个 ACK Pod IP 上,因此更像是镜像、依赖或运行时代码版本一致性问题,而不是单个请求或单台机器异常。
示例日志:
2026-07-14 14:06:27,820 | ... | path=/product/add | env=prod | user_type=normal | user_name=*** | memos.mem_reader.read_multi_modal.base - ERROR - base.py:275 - _split_text - [FileContentParser] Error splitting text: 'SimpleTextSplitter' object has no attribute 'protect_urls'
Additional Context | 其他信息
No response
Willingness to Implement | 实现意愿
- I'm willing to implement this myself | 我愿意自己解决
- I would like someone else to implement this | 我希望其他人来解决
Metadata
Metadata
Assignees
Labels
ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审AI-created PR is ready for review | AI 生成的 PR 已等待评审area:coreMOS 编排层 / 框架底座 / 跨模块问题MOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理Someone or AI is working on it | 人工或 AI 正在处理types:bugSomething isn't working | 功能异常Something isn't working | 功能异常
Activity
Memtensor-AI commentedon Jul 16, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
e062b03201f62d58Working branch:
bugfix/autodev-2115-20260716073650486Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(chunkers): make SimpleTextSplitter fallback URL-safe (MemTensor#2115
Memtensor-AI commentedon Jul 16, 2026
✅ AutoDev task
e062b03201f62d58completed.Summary: Fixes issue #2115:
SimpleTextSplitterfallback in the multi-modal file-parsing pipeline was raisingAttributeError: 'SimpleTextSplitter' object has no attribute 'protect_urls'. Root cause:SimpleTextSplitter._simple_split_text()callsself.protect_urls/self.restore_urls, but those helpers live onBaseChunkerandSimpleTextSplitterdoes not inherit from it. On ACK (wherelangchain_text_splittersis missing and the fallback is actually exercised) this produced ~5.8k noisy log rows and silently degraded chunking to a single-chunk pass-through.The fix extracts
protect_urls/restore_urlsinto a smallURLProtectionMixininsrc/memos/chunkers/base.py;BaseChunkernow inherits it (behavioural no-op — same public API), andSimpleTextSplitterinsrc/memos/chunkers/simple_chunker.pyalso inherits it, so the two paths share one URL regex and one placeholder scheme.SimpleTextSplitter's constructor signature andchunk() -> list[str]return type are unchanged; no factory / config / OpenAPI changes are required.Verification: 7 new regression tests in
tests/chunkers/test_simple_chunker.py(short/long input with URL, empty / whitespace input, no-URL text, parametrised chunk sizes) — all 7 failed before the fix with the exact AttributeError, all 7 pass after. Broaderpytest tests/chunkers/ tests/mem_reader/→ 63 pass; the 2 unrelated failures (test_doc_local_file,test_doc_mixed) fail identically on the pristine branch with the change stashed and are caused by the sandbox missingmarkitdown[all].ruff check+ruff format --checkare green on all 3 changed files.Working branch pushed to origin; opsp task + verification report archived to memos-autodev-specs main. Ready for reviewer @WeiminLee.
Base branch:
mainBranch:
bugfix/autodev-2115-20260716073650486Commit:
74d8598PR: #2116
Assigned to: @NydiaChung
Reviewers: @WeiminLee
fix(chunkers): make SimpleTextSplitter fallback URL-safe (#2200)
fix(chunkers): make SimpleTextSplitter fallback URL-safe (#2200) (#2228)
Memtensor-AI commentedon Aug 12, 2026
Resolved by #2200, #2228. The merged fix is already included in
mainand released in v2.0.29, so closing this issue as completed.