You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebinding to str #2433
SentenceChunker.chunk in src/memos/chunkers/sentence_chunker.py builds a proper Chunk object and then immediately overwrites it with the restored-URL string:
chunk=Chunk(text=c.text, token_count=c.token_count, sentences=c.sentences)
chunk=self.restore_urls(chunk.text, url_map) # rebinds `chunk` to a bare strchunks.append(chunk)
So the method always returns list[str] despite declaring list[str] | list[Chunk]: token_count and sentences are silently discarded, and callers that use the documented Chunk API (.text / .token_count / .sentences) get an AttributeError. The existing test masks this by accepting both return types.
Expected Behavior
The method should return actual Chunk objects with the URL-restored text plus the preserved token_count and sentences.
Environment
MemOS main, python 3.14.
Additional Context
Fix + red/green-verified test ready; PR to follow referencing this issue.
Summary: Fixes #2433. SentenceChunker.chunk in src/memos/chunkers/sentence_chunker.py was building a Chunk(text, token_count, sentences) and then rebinding the local chunk variable to the plain string returned by restore_urls, so the method returned list[str] and silently discarded token_count / sentences. Callers using the documented Chunk API (.text, .token_count, .sentences) raised AttributeError.
The fix restores URLs on the raw text first, then constructs the Chunk with the restored text and the preserved chonkie metadata, and narrows the return annotation to list[Chunk] to match BaseChunker.chunk. Callers were audited: prefer_text_memory/spliter.py and mem_reader/simple_struct.py already read chunk.text (they had been silently broken); mem_reader/multi_modal_struct.py already tolerates both shapes; mem_reader/read_multi_modal/base.py uses SimpleTextSplitter, not the sentence chunker, so it is unaffected.
tests/chunkers/test_sentence_chunker.py was rewritten to drop the if isinstance(chunks[0], str) mask that had been hiding the bug. It now unconditionally asserts Chunk type plus preserved token_count / sentences, and adds test_sentence_chunker_restores_urls_on_chunk_text to verify that URLs are restored inside chunk.text while metadata still survives. Both new tests fail on the unpatched code (verified TDD red) and pass after the fix; full pytest tests/chunkers/ -q reports 25 passed; ruff check and ruff format --check on the two touched files are clean.
Base branch:main Branch:bugfix/autodev-2433-20260928165809486 Commit:6ae2753 PR:#2435 Assigned to:@syzsunshine219 Reviewers:@WeiminLee
Activity
Memtensor-AI commentedon Sep 28, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
8692b21ce1690c04Working branch:
bugfix/autodev-2433-20260928165809486Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(chunkers): return Chunk objects from SentenceChunker.chunk (MemTe…
Memtensor-AI commentedon Sep 28, 2026
✅ AutoDev task
8692b21ce1690c04completed.Summary: Fixes #2433.
SentenceChunker.chunkinsrc/memos/chunkers/sentence_chunker.pywas building aChunk(text, token_count, sentences)and then rebinding the localchunkvariable to the plain string returned byrestore_urls, so the method returnedlist[str]and silently discardedtoken_count/sentences. Callers using the documentedChunkAPI (.text,.token_count,.sentences) raisedAttributeError.The fix restores URLs on the raw text first, then constructs the
Chunkwith the restored text and the preserved chonkie metadata, and narrows the return annotation tolist[Chunk]to matchBaseChunker.chunk. Callers were audited:prefer_text_memory/spliter.pyandmem_reader/simple_struct.pyalready readchunk.text(they had been silently broken);mem_reader/multi_modal_struct.pyalready tolerates both shapes;mem_reader/read_multi_modal/base.pyusesSimpleTextSplitter, not the sentence chunker, so it is unaffected.tests/chunkers/test_sentence_chunker.pywas rewritten to drop theif isinstance(chunks[0], str)mask that had been hiding the bug. It now unconditionally assertsChunktype plus preservedtoken_count/sentences, and addstest_sentence_chunker_restores_urls_on_chunk_textto verify that URLs are restored insidechunk.textwhile metadata still survives. Both new tests fail on the unpatched code (verified TDD red) and pass after the fix; fullpytest tests/chunkers/ -qreports 25 passed;ruff checkandruff format --checkon the two touched files are clean.Base branch:
mainBranch:
bugfix/autodev-2433-20260928165809486Commit:
6ae2753PR: #2435
Assigned to: @syzsunshine219
Reviewers: @WeiminLee
github-actions commentedon Oct 29, 2026
This issue has been automatically marked as stale due to inactivity.
github-actions commentedon Nov 6, 2026
This issue has been automatically closed due to inactivity.
Fix #2433: SentenceChunker.chunk discards Chunk objects (token_count/…