Repository navigation
Fix #2433: SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebindin - #2581
Merged
endxxxx merged 2 commits intoNov 18, 2026
Conversation
- Add test_sentence_chunker_preserves_urls_in_sentences to verify URLs are restored in both text and sentences fields - Fix SentenceChunker.chunk to restore URLs in sentences list, not just text - Addresses OCR finding: metadata must survive for sentences
syzsunshine219
marked this pull request as ready for review
November 9, 2026 15:47
endxxxx
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores the change proposed in #2435 for review against
mainafter its former base branch,dev-v2.0.36, was deleted. The source commits and authors are preserved.Source head:
0d10fbfe8cd4a0df4fc68372ac840973a67fd789This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2435.
Original change description
Description
Fixes #2433.
SentenceChunker.chunkinsrc/memos/chunkers/sentence_chunker.pywas building aChunk(text, token_count, sentences)and then rebinding the localchunkvariable to the plain string returned byrestore_urls, so the method returnedlist[str]and silently discardedtoken_count/sentences. Callers using the documentedChunkAPI (.text,.token_count,.sentences) raisedAttributeError.The fix restores URLs on the raw text first, then constructs the
Chunkwith the restored text and the preserved chonkie metadata, and narrows the return annotation tolist[Chunk]to matchBaseChunker.chunk. Callers were audited:prefer_text_memory/spliter.pyandmem_reader/simple_struct.pyalready readchunk.text(they had been silently broken);mem_reader/multi_modal_struct.pyalready tolerates both shapes;mem_reader/read_multi_modal/base.pyusesSimpleTextSplitter, not the sentence chunker, so it is unaffected.tests/chunkers/test_sentence_chunker.pywas rewritten to drop theif isinstance(chunks[0], str)mask that had been hiding the bug. It now unconditionally assertsChunktype plus preservedtoken_count/sentences, and addstest_sentence_chunker_restores_urls_on_chunk_textto verify that URLs are restored insidechunk.textwhile metadata still survives. Both new tests fail on the unpatched code (verified TDD red) and pass after the fix; fullpytest tests/chunkers/ -qreports 25 passed;ruff checkandruff format --checkon the two touched files are clean.Related Issue (Required): Fixes #2433
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Not run; documentation-only change.
Checklist
@WeiminLeeplease review this PR.Reviewer Checklist