Skip to content

SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebinding to str #2433

Closed as not planned

Description

@simpleqt

Bug Description

SentenceChunker.chunk in src/memos/chunkers/sentence_chunker.py builds a proper Chunk object and then immediately overwrites it with the restored-URL string:

chunk = Chunk(text=c.text, token_count=c.token_count, sentences=c.sentences)
chunk = self.restore_urls(chunk.text, url_map)   # rebinds `chunk` to a bare str
chunks.append(chunk)

So the method always returns list[str] despite declaring list[str] | list[Chunk]: token_count and sentences are silently discarded, and callers that use the documented Chunk API (.text / .token_count / .sentences) get an AttributeError. The existing test masks this by accepting both return types.

Expected Behavior

The method should return actual Chunk objects with the URL-restored text plus the preserved token_count and sentences.

Environment

MemOS main, python 3.14.

Additional Context

Fix + red/green-verified test ready; PR to follow referencing this issue.

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    area:coreMOS 编排层 / 框架底座 / 跨模块问题
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Sep 28, 2026
  2. self-assigned this
    on Sep 28, 2026
  3. Memtensor-AI commented on Sep 28, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 8692b21ce1690c04
    Working branch: bugfix/autodev-2433-20260928165809486
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. Memtensor-AI commented on Sep 28, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 8692b21ce1690c04 completed.

    Summary: Fixes #2433. SentenceChunker.chunk in src/memos/chunkers/sentence_chunker.py was building a Chunk(text, token_count, sentences) and then rebinding the local chunk variable to the plain string returned by restore_urls, so the method returned list[str] and silently discarded token_count / sentences. Callers using the documented Chunk API (.text, .token_count, .sentences) raised AttributeError.

    The fix restores URLs on the raw text first, then constructs the Chunk with the restored text and the preserved chonkie metadata, and narrows the return annotation to list[Chunk] to match BaseChunker.chunk. Callers were audited: prefer_text_memory/spliter.py and mem_reader/simple_struct.py already read chunk.text (they had been silently broken); mem_reader/multi_modal_struct.py already tolerates both shapes; mem_reader/read_multi_modal/base.py uses SimpleTextSplitter, not the sentence chunker, so it is unaffected.

    tests/chunkers/test_sentence_chunker.py was rewritten to drop the if isinstance(chunks[0], str) mask that had been hiding the bug. It now unconditionally asserts Chunk type plus preserved token_count / sentences, and adds test_sentence_chunker_restores_urls_on_chunk_text to verify that URLs are restored inside chunk.text while metadata still survives. Both new tests fail on the unpatched code (verified TDD red) and pass after the fix; full pytest tests/chunkers/ -q reports 25 passed; ruff check and ruff format --check on the two touched files are clean.

    Base branch: main
    Branch: bugfix/autodev-2433-20260928165809486
    Commit: 6ae2753
    PR: #2435
    Assigned to: @syzsunshine219
    Reviewers: @WeiminLee

  5. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:failedAI task failed | AI 任务失败
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Sep 28, 2026
  6. github-actions commented on Oct 29, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  7. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Oct 29, 2026
  8. github-actions commented on Nov 6, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

  9. added a commit that references this issue on Nov 18, 2026
    9b6dcf3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:failedAI task failed | AI 任务失败area:coreMOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions