Skip to content

[Bug] sources array duplicates the full original message N times (N = chunk count) | sources 数组按分块数复制整条原文 #2453

Closed as not planned

Description

@ywhhbco

[Bug] sources array duplicates the full original message N times (N = chunk count)

Pre-submission checklist

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist
  • This issue is specific to MemOS and not a general software issue

MemOS Version

Latest main (self-hosted, deployed 2026-09; src/memos/mem_reader/multi_modal_struct.py line numbers from current main)

Bug Description | 问题描述

中文

当写入一条较长的消息时,MemOS 会先按 chunker 分段(multi_modal_struct.py:138),
然后为每个分块建立一条新记忆项。问题在于:每一个分块产物都被挂上了同一份完整的 sources 列表
(multi_modal_struct.py:169:sources=item.metadata.sources or [])。

结果是:一条消息被分成 N 段时,每一条产出记忆的 sources 数组里都会出现 N 份内容逐字相同的原文副本。
SourceMessage 的 docstring 自述其用途是 "memory provenance / traceability …
Capture the minimal, reproducible origin context",但实际填入的是整条原始消息,
且被复制 N 份。

English

When a long message is ingested, MemOS first splits it into chunks (multi_modal_struct.py:138)
and then creates one memory item per chunk. The problem: every chunk item is attached
the same complete sources list
(multi_modal_struct.py:169:
sources=item.metadata.sources or []).

Consequently, if a message is split into N chunks, each resulting memory carries N byte-identical
copies of the full original message in its sources array
— even though SourceMessage's own
docstring states it should capture the "minimal, reproducible origin context".

Steps to Reproduce | 复现步骤

  1. Send a >1,400-character message via POST /product/add (or the MCP add_memory tool —
    the MCP gateway builds {"user_id":..., "messages":[{"role":"user","content":...}], "writable_cubes":[...]} and forwards it):
curl -X POST http://localhost:8000/product/add \
  -H 'Content-Type: application/json' \
  -d '{"user_id":"test","messages":[{"role":"user","content":"<1456-char message>"}],"writable_cube_ids":["test"]}'
  1. Wait for async extraction to finish (~30 s), then inspect the produced memory nodes in Neo4j:
MATCH (n:Memory) WHERE n.user_id = 'test'
RETURN n.id, n.memory, size(n.sources) AS nsrc
ORDER BY n.created_at DESC LIMIT 20
  1. Observe size(n.sources) > 1, and every element has identical content.

Expected Behavior | 预期行为

sources should carry one origin context (or, at most, the chunk that actually produced
that memory). It should not be duplicated per chunk.

Actual Behavior | 实际行为

Every chunk-produced memory gets the full sources array. Measured:

  • Controlled reproduction: one 1,456-character message → split into 4 chunks →
    15 memory items produced, all with sources length = 4 and all 4 elements
    byte-identical
    (length [1456, 1456, 1456, 1456], identical MD5).
  • Production database (a personal MemOS instance, 320 memories): 89 / 320 = 27.8%
    of memories have sources length 2–4, and in 89/89 cases the contents are byte-identical.
    An independent re-measurement by another agent on a later snapshot: 99 / 356 = 27%.
  • Threshold: a 1,102-character message did not trigger (sources=1);
    1,456 characters triggered (4 chunks). Exact threshold not yet pinned down
    (likely token-based, since chunk_size defaults to 512 tokens in
    src/memos/configs/chunker.py:14, while Splitter uses 256).

Suspected Root Cause | 疑似根因

# src/memos/mem_reader/multi_modal_struct.py

# :138  chunk the incoming text
chunks = self.chunker.chunk(item_text)
chunk_total = len(chunks)

def _create_chunk_item(chunk_idx: int, chunk):
    ...
    # :169  every chunk copies the WHOLE sources list
    split_item = self._make_memory_item(
        value=chunk_text,
        ...
        sources=item.metadata.sources or [],   # <-- duplicated N times
        ...
    )

# :179  run per chunk
futures = [executor.submit(_create_chunk_item, i, c) for i, c in enumerate(chunks)]

The same pattern also appears in prefer_text_memory/extractor.py
(_process_single_chunk_explicit / _process_single_chunk_implicit, lines ~118–175),
where sources is inherited from basic_info per chunk.

Impact | 影响

  • Storage bloat: a memory's sources payload grows N× (N = chunk count); for a
    long document in an ingestion pipeline this can dominate the row size.
  • Provenance / dedup tooling breaks: any tool that counts or iterates sources to
    attribute a memory to its origin (or dedups by source hash) gets N× inflated counts.
    We hit exactly this while building a memory-governance report.
  • Retrieval context bloat: advanced_searcher.py:173 renders
    [f"({one.chat_time}) {one.content}" for one in mem.metadata.sources],
    so the duplicated full text may be re-injected into the prompt N times.

Suggested Fix | 建议修法

Either of:

  1. Attach sources only to the chunk that actually produced the memory
    (slice by chunk_index), or
  2. If the full origin context is genuinely wanted, store it once on the parent /
    episode node and have chunk items reference it (e.g. by parent_id or
    internal_info.chunk_index + a parent pointer), keeping per-item sources minimal —
    which is what the SourceMessage docstring already promises.

Related (not duplicates) | 相关但不重复

Environment | 环境

  • MemOS: self-hosted (WSL2 Ubuntu 24.04), Neo4j + Qdrant + local Ollama
  • MemOS Server API on :8000, MCP gateway (fastmcp) on :8002
  • Ingested via MCP tool add_memory → POST /product/add
  • Install method: source checkout (git clone)

Additional request (same ingest path) | 附带请求

The MCP gateway examples/mem_mcp/simple_fastmcp_serve.py exposes only:

def add_memory(memory_content: str, user_id: str, cube_id: str | None = None):

so callers cannot pass tags / memory_type / key / background even though the
Memory node schema supports those fields. Adding optional passthrough parameters would
make structured tagging possible without abusing the message body. (Happy to open this
as a separate feature request if preferred.)

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    area:coreMOS 编排层 / 框架底座 / 跨模块问题
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Oct 3, 2026
  2. self-assigned this
    on Oct 3, 2026
  3. Memtensor-AI commented on Oct 3, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: c72466b93711d6d9
    Working branch: bugfix/autodev-2453-20261003090152992
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. Memtensor-AI commented on Oct 3, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task c72466b93711d6d9 completed.

    Summary: Fixes issue #2453: sources array duplicated per chunk in MultiModalStructMemReader.

    Root cause: _split_large_memory_item attached the parent sources list to every chunk (multi_modal_struct.py:169), and _build_window_from_items then concatenated them without dedup, so a message split into N chunks produced memory items whose sources list carried N byte-identical copies of the original message. The reporter measured 27% of production memories affected.

    Fix is two-layer: (1) attach sources only to chunk 0 (the "owning" chunk) with an additional internal_info["source_chunk_index"] = 0 marker; (2) dedup all_sources in _build_window_from_items by a stable (type, role, message_id, chat_time, doc_path, content) signature for defense-in-depth. Lineage keys (ingest_batch_id / chunk_index / chunk_total) are preserved on every chunk.

    Testing: 8 new regression cases in tests/mem_reader/test_sources_dedup.py (4 reproduced the bug on pristine HEAD; all 8 pass after the fix). Full tests/mem_reader/ suite: 71 pass. Pre-existing failures from a missing markitdown dependency and a torch/transformers DynamicCache compat issue in tests/memories/activation/test_kv.py are unrelated (verified on pristine HEAD). Ruff check + format: clean.

    Base branch: main
    Branch: bugfix/autodev-2453-20261003090152992
    Commit: e79304e
    PR: #2454
    Assigned to: @syzsunshine219
    Reviewers: @WeiminLee

  5. added
    ai:testingAI agent is running tests | AI 正在运行测试
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Oct 3, 2026
  6. github-actions commented on Nov 3, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  7. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Nov 3, 2026
  8. github-actions commented on Nov 11, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

  9. added a commit that references this issue on Nov 18, 2026
    d024285
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务ai:testingAI agent is running tests | AI 正在运行测试area:coreMOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions