Skip to content

Fix #2433: SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebindin - #2581

Merged
endxxxx merged 2 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2433-20260928165809486
Nov 18, 2026
Merged

endxxxx merged 2 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2433-20260928165809486

Conversation

@syzsunshine219

@syzsunshine219 syzsunshine219 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Restores the change proposed in #2435 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: 0d10fbfe8cd4a0df4fc68372ac840973a67fd789

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2435.


Original change description

Description

Fixes #2433. SentenceChunker.chunk in src/memos/chunkers/sentence_chunker.py was building a Chunk(text, token_count, sentences) and then rebinding the local chunk variable to the plain string returned by restore_urls, so the method returned list[str] and silently discarded token_count / sentences. Callers using the documented Chunk API (.text, .token_count, .sentences) raised AttributeError.

The fix restores URLs on the raw text first, then constructs the Chunk with the restored text and the preserved chonkie metadata, and narrows the return annotation to list[Chunk] to match BaseChunker.chunk. Callers were audited: prefer_text_memory/spliter.py and mem_reader/simple_struct.py already read chunk.text (they had been silently broken); mem_reader/multi_modal_struct.py already tolerates both shapes; mem_reader/read_multi_modal/base.py uses SimpleTextSplitter, not the sentence chunker, so it is unaffected.

tests/chunkers/test_sentence_chunker.py was rewritten to drop the if isinstance(chunks[0], str) mask that had been hiding the bug. It now unconditionally asserts Chunk type plus preserved token_count / sentences, and adds test_sentence_chunker_restores_urls_on_chunk_text to verify that URLs are restored inside chunk.text while metadata still survives. Both new tests fail on the unpatched code (verified TDD red) and pass after the fix; full pytest tests/chunkers/ -q reports 25 passed; ruff check and ruff format --check on the two touched files are clean.

Related Issue (Required): Fixes #2433

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Not run; documentation-only change.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@WeiminLee please review this PR.

Reviewer Checklist

- Add test_sentence_chunker_preserves_urls_in_sentences to verify URLs are restored in both text and sentences fields
- Fix SentenceChunker.chunk to restore URLs in sentences list, not just text
- Addresses OCR finding: metadata must survive for sentences
@syzsunshine219
syzsunshine219 marked this pull request as ready for review November 9, 2026 15:47
@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 18, 2026 03:22
@endxxxx
endxxxx merged commit 9b6dcf3 into MemTensor:dev-v2.0.37 Nov 18, 2026
22 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebinding to str

2 participants