Repository navigation
Description
[Bug] sources array duplicates the full original message N times (N = chunk count)
Pre-submission checklist
- I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
- I have read the project documentation and confirmed this issue doesn't already exist
- This issue is specific to MemOS and not a general software issue
MemOS Version
Latest main (self-hosted, deployed 2026-09; src/memos/mem_reader/multi_modal_struct.py line numbers from current main)
Bug Description | 问题描述
中文
当写入一条较长的消息时,MemOS 会先按 chunker 分段(multi_modal_struct.py:138),
然后为每个分块建立一条新记忆项。问题在于:每一个分块产物都被挂上了同一份完整的 sources 列表
(multi_modal_struct.py:169:sources=item.metadata.sources or [])。
结果是:一条消息被分成 N 段时,每一条产出记忆的 sources 数组里都会出现 N 份内容逐字相同的原文副本。
SourceMessage 的 docstring 自述其用途是 "memory provenance / traceability …
Capture the minimal, reproducible origin context",但实际填入的是整条原始消息,
且被复制 N 份。
English
When a long message is ingested, MemOS first splits it into chunks (multi_modal_struct.py:138)
and then creates one memory item per chunk. The problem: every chunk item is attached
the same complete sources list (multi_modal_struct.py:169:
sources=item.metadata.sources or []).
Consequently, if a message is split into N chunks, each resulting memory carries N byte-identical
copies of the full original message in its sources array — even though SourceMessage's own
docstring states it should capture the "minimal, reproducible origin context".
Steps to Reproduce | 复现步骤
- Send a >1,400-character message via
POST /product/add(or the MCPadd_memorytool —
the MCP gateway builds{"user_id":..., "messages":[{"role":"user","content":...}], "writable_cubes":[...]}and forwards it):
curl -X POST http://localhost:8000/product/add \
-H 'Content-Type: application/json' \
-d '{"user_id":"test","messages":[{"role":"user","content":"<1456-char message>"}],"writable_cube_ids":["test"]}'- Wait for async extraction to finish (~30 s), then inspect the produced memory nodes in Neo4j:
MATCH (n:Memory) WHERE n.user_id = 'test'
RETURN n.id, n.memory, size(n.sources) AS nsrc
ORDER BY n.created_at DESC LIMIT 20- Observe
size(n.sources)> 1, and every element has identicalcontent.
Expected Behavior | 预期行为
sources should carry one origin context (or, at most, the chunk that actually produced
that memory). It should not be duplicated per chunk.
Actual Behavior | 实际行为
Every chunk-produced memory gets the full sources array. Measured:
- Controlled reproduction: one 1,456-character message → split into 4 chunks →
15 memory items produced, all withsourceslength = 4 and all 4 elements
byte-identical (length[1456, 1456, 1456, 1456], identical MD5). - Production database (a personal MemOS instance, 320 memories): 89 / 320 = 27.8%
of memories havesourceslength 2–4, and in 89/89 cases the contents are byte-identical.
An independent re-measurement by another agent on a later snapshot: 99 / 356 = 27%. - Threshold: a 1,102-character message did not trigger (sources=1);
1,456 characters triggered (4 chunks). Exact threshold not yet pinned down
(likely token-based, sincechunk_sizedefaults to 512 tokens in
src/memos/configs/chunker.py:14, whileSplitteruses 256).
Suspected Root Cause | 疑似根因
# src/memos/mem_reader/multi_modal_struct.py
# :138 chunk the incoming text
chunks = self.chunker.chunk(item_text)
chunk_total = len(chunks)
def _create_chunk_item(chunk_idx: int, chunk):
...
# :169 every chunk copies the WHOLE sources list
split_item = self._make_memory_item(
value=chunk_text,
...
sources=item.metadata.sources or [], # <-- duplicated N times
...
)
# :179 run per chunk
futures = [executor.submit(_create_chunk_item, i, c) for i, c in enumerate(chunks)]The same pattern also appears in prefer_text_memory/extractor.py
(_process_single_chunk_explicit / _process_single_chunk_implicit, lines ~118–175),
where sources is inherited from basic_info per chunk.
Impact | 影响
- Storage bloat: a memory's
sourcespayload grows N× (N = chunk count); for a
long document in an ingestion pipeline this can dominate the row size. - Provenance / dedup tooling breaks: any tool that counts or iterates
sourcesto
attribute a memory to its origin (or dedups by source hash) gets N× inflated counts.
We hit exactly this while building a memory-governance report. - Retrieval context bloat:
advanced_searcher.py:173renders
[f"({one.chat_time}) {one.content}" for one in mem.metadata.sources],
so the duplicated full text may be re-injected into the prompt N times.
Suggested Fix | 建议修法
Either of:
- Attach
sourcesonly to the chunk that actually produced the memory
(slice bychunk_index), or - If the full origin context is genuinely wanted, store it once on the parent /
episode node and have chunk items reference it (e.g. byparent_idor
internal_info.chunk_index+ a parent pointer), keeping per-itemsourcesminimal —
which is what theSourceMessagedocstring already promises.
Related (not duplicates) | 相关但不重复
- 删除记忆接口返回成功但记忆未实际删除 #2331 — delete API returns success but memory is not actually deleted (open)
- L3 world model: structure bloats with paraphrase duplicates — entry-level dedupe in mergeEntries is string-exact only #2409 — L3 world model structure bloats with paraphrase duplicates (different layer/mechanism)
Environment | 环境
- MemOS: self-hosted (WSL2 Ubuntu 24.04), Neo4j + Qdrant + local Ollama
- MemOS Server API on
:8000, MCP gateway (fastmcp) on:8002 - Ingested via MCP tool
add_memory→POST /product/add - Install method: source checkout (git clone)
Additional request (same ingest path) | 附带请求
The MCP gateway examples/mem_mcp/simple_fastmcp_serve.py exposes only:
def add_memory(memory_content: str, user_id: str, cube_id: str | None = None):so callers cannot pass tags / memory_type / key / background even though the
Memory node schema supports those fields. Adding optional passthrough parameters would
make structured tagging possible without abusing the message body. (Happy to open this
as a separate feature request if preferred.)
Activity
Memtensor-AI commentedon Oct 3, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
c72466b93711d6d9Working branch:
bugfix/autodev-2453-20261003090152992Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix: stop sources duplication across chunked memory items (MemTensor#…
Memtensor-AI commentedon Oct 3, 2026
✅ AutoDev task
c72466b93711d6d9completed.Summary: Fixes issue #2453: sources array duplicated per chunk in MultiModalStructMemReader.
Root cause:
_split_large_memory_itemattached the parentsourceslist to every chunk (multi_modal_struct.py:169), and_build_window_from_itemsthen concatenated them without dedup, so a message split into N chunks produced memory items whosesourceslist carried N byte-identical copies of the original message. The reporter measured 27% of production memories affected.Fix is two-layer: (1) attach
sourcesonly to chunk 0 (the "owning" chunk) with an additionalinternal_info["source_chunk_index"] = 0marker; (2) dedupall_sourcesin_build_window_from_itemsby a stable (type, role, message_id, chat_time, doc_path, content) signature for defense-in-depth. Lineage keys (ingest_batch_id/chunk_index/chunk_total) are preserved on every chunk.Testing: 8 new regression cases in tests/mem_reader/test_sources_dedup.py (4 reproduced the bug on pristine HEAD; all 8 pass after the fix). Full tests/mem_reader/ suite: 71 pass. Pre-existing failures from a missing
markitdowndependency and a torch/transformersDynamicCachecompat issue intests/memories/activation/test_kv.pyare unrelated (verified on pristine HEAD). Ruff check + format: clean.Base branch:
mainBranch:
bugfix/autodev-2453-20261003090152992Commit:
e79304ePR: #2454
Assigned to: @syzsunshine219
Reviewers: @WeiminLee
fix(mem_reader): pass sources=[] explicitly when creating split chunk…
fix(mem_reader): stop duplicating parent sources on every split chunk (…
fix(mem_reader): pass sources=[] explicitly when creating split chunk…
github-actions commentedon Nov 3, 2026
This issue has been automatically marked as stale due to inactivity.
fix(mem_reader): stop duplicating parent sources on every split chunk (…
fix(mem_reader): pass sources=[] explicitly when creating split chunk…
github-actions commentedon Nov 11, 2026
This issue has been automatically closed due to inactivity.
Fix #2453: [Bug] sources array duplicates the full original message N…