Repository navigation
Fix #2433: SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebindin - #2435
Memtensor-AI wants to merge 1 commit into
Conversation
🤖 Open Code ReviewTarget: PR #2435 ✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). Generated by cloud-assistant via Open Code Review. |
🔧 Open Code Review requested Agent fixOpen Code Review found 1 issue(s). I have resumed the development Agent to fix them.
The Agent will push a new commit to this PR branch. OCR will recheck after the commit is pushed. |
- Add test_sentence_chunker_preserves_urls_in_sentences to verify URLs are restored in both text and sentences fields - Fix SentenceChunker.chunk to restore URLs in sentences list, not just text - Addresses OCR finding: metadata must survive for sentences
6ae2753 to
0d10fbf
Compare
|
|
This PR has been automatically marked as stale due to inactivity. |
|
This PR has been automatically closed due to inactivity. |
Description
Fixes #2433.
SentenceChunker.chunkinsrc/memos/chunkers/sentence_chunker.pywas building aChunk(text, token_count, sentences)and then rebinding the localchunkvariable to the plain string returned byrestore_urls, so the method returnedlist[str]and silently discardedtoken_count/sentences. Callers using the documentedChunkAPI (.text,.token_count,.sentences) raisedAttributeError.The fix restores URLs on the raw text first, then constructs the
Chunkwith the restored text and the preserved chonkie metadata, and narrows the return annotation tolist[Chunk]to matchBaseChunker.chunk. Callers were audited:prefer_text_memory/spliter.pyandmem_reader/simple_struct.pyalready readchunk.text(they had been silently broken);mem_reader/multi_modal_struct.pyalready tolerates both shapes;mem_reader/read_multi_modal/base.pyusesSimpleTextSplitter, not the sentence chunker, so it is unaffected.tests/chunkers/test_sentence_chunker.pywas rewritten to drop theif isinstance(chunks[0], str)mask that had been hiding the bug. It now unconditionally assertsChunktype plus preservedtoken_count/sentences, and addstest_sentence_chunker_restores_urls_on_chunk_textto verify that URLs are restored insidechunk.textwhile metadata still survives. Both new tests fail on the unpatched code (verified TDD red) and pass after the fix; fullpytest tests/chunkers/ -qreports 25 passed;ruff checkandruff format --checkon the two touched files are clean.Related Issue (Required): Fixes #2433
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Not run; documentation-only change.
Checklist
@WeiminLee please review this PR.
Reviewer Checklist