Skip to content

SentenceChunker.chunk: return Chunk objects instead of overwriting them with str - #2434

Closed
simpleqt wants to merge 1 commit into
MemTensor:mainfrom
simpleqt:sq/sentence-chunker-objects
Closed

simpleqt wants to merge 1 commit into
MemTensor:mainfrom
simpleqt:sq/sentence-chunker-objects

Conversation

@simpleqt

Copy link
Copy Markdown

Fixes #2433

Summary

SentenceChunker.chunk in src/memos/chunkers/sentence_chunker.py builds a proper Chunk and then immediately overwrites it with the restored-URL string:

chunk = Chunk(text=c.text, token_count=c.token_count, sentences=c.sentences)
chunk = self.restore_urls(chunk.text, url_map)   # rebinds `chunk` to a bare str
chunks.append(chunk)

So the method always returns list[str] despite declaring list[str] | list[Chunk]: token_count and sentences are silently discarded, and callers using the documented Chunk API (.text / .token_count / .sentences) get an AttributeError. The existing test_sentence_chunker.py masks this by accepting both return types.

Fix

Restore the URLs into the Chunk's text and append the object:

chunks.append(
    Chunk(
        text=self.restore_urls(c.text, url_map),
        token_count=c.token_count,
        sentences=c.sentences,
    )
)

Regression test

Added tests/chunkers/test_sentence_chunker_chunk_objects.py:

  • red/green verified: fails against the old implementation (chunks[0] is a str) and passes with the fix
  • asserts the returned item is a Chunk with restored URL text plus preserved token_count and sentences
  • the full tests/chunkers/ suite (25 tests) passes with it, and the test's package stub only fills in missing modules so it does not shadow the real package for other tests

ruff check (repo-pinned 0.11.8) passes on both touched files.

AI Disclosure

  • Tool(s): Claude (ZCode CLI)
  • Used for: debugging assistance, code suggestions, and drafting this PR description. All changes human-reviewed.

chunk() built a Chunk and then rebound it to the bare restored-URL
string, so the method always returned list[str] while discarding
token_count and sentences. Restore the URLs into the Chunk's text and
return the object.

Fixes MemTensor#2433
Copilot AI lite review requested due to automatic review settings September 28, 2026 17:05
@Memtensor-AI Memtensor-AI added area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Sep 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2434
Task: 30972e28f3a96d19
Base: main
Head: sq/sentence-chunker-objects

🔍 OpenCodeReview found 1 issue(s) in this PR.


1. tests/chunkers/test_sentence_chunker_chunk_objects.py (L77-L79)

The test expects sentences to contain the placeholder URL __URL_0__ while text contains the restored URL. However, examining the production code in sentence_chunker.py shows that restore_urls() is only applied to c.text, not to c.sentences. This means c.sentences will retain placeholder URLs from the backend chunker. The test assertion encodes this behavior as correct, but it creates an inconsistency: the text field shows the real URL while the sentences field shows placeholders. If users iterate over sentences expecting them to match the content in text, they'll encounter placeholder strings instead of actual URLs. Consider whether sentences should also have URLs restored, or document this intentional design choice.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: INCONCLUSIVE

Automated tests inconclusive (auto-generated test defect); treated as non-blocking. Manual review recommended. Details: The test fails at fixture setup because it tries to patch 'chonkie.SentenceChunker' but the 'chonkie' module is not installed in the test environment, causing a ModuleNotFoundError before any test logic runs.

Branch: sq/sentence-chunker-objects

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically marked as stale due to inactivity.

@github-actions github-actions Bot added the status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt. label Oct 29, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically closed due to inactivity.

@github-actions github-actions Bot closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SentenceChunker.chunk discards Chunk objects (token_count/sentences) by rebinding to str

4 participants