Repository navigation
Fix #2446: split_continuous_references corrupts plain bracketed lists (non-reference text) - #2583
Open
syzsunshine219 wants to merge 2 commits into
Open
syzsunshine219 wants to merge 2 commits into
syzsunshine219 wants to merge 2 commits into
Conversation
added 2 commits
October 1, 2026 00:09
… bracketed prose split_continuous_references previously rewrote any single [...] block that contained a comma into pseudo reference tags, so ordinary prose like "The set is [apple, banana] here" was corrupted into "The set is [apple][banana] here". Add a shape guard: only split when every comma-separated item in the block matches <int>:<id> (the actual reference tag shape, e.g. [1:92ff35fb, 4:bfe6f044]); otherwise return the text unchanged. Adds tests/mem_os/test_reference_utils.py covering the regression case, mixed / non-reference items, and boundary conditions (empty, multiple brackets, single item). Fixes MemTensor#2446
The previous two-step str.replace approach in split_continuous_references was broken for blocks that mixed ", " and bare "," separators: Input: "[1:aa, 2:bb,3:cc]" content_between_brackets = "1:aa, 2:bb,3:cc" After the first replace, text became "[1:aa][2:bb,3:cc]"; the second replace searched for the original substring, could no longer find it, and the bare comma between 2:bb and 3:cc was never split. Rebuild the bracketed block in a single pass by joining the already- computed items with "][" and stripping surrounding whitespace, so all separator styles collapse identically. Adds two regression tests for mixed separator styles (PR MemTensor#2450 review).
syzsunshine219
marked this pull request as ready for review
November 9, 2026 15:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores the change proposed in #2450 for review against
mainafter its former base branch,dev-v2.0.36, was deleted. The source commits and authors are preserved.Source head:
907c95e7a14a56feb5b724e16e00b299e699a314This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2450.
Original change description
Description
Fix #2446: split_continuous_references now guards against plain bracketed prose. Previously the function rewrote any single [...] block containing a comma into split tags, so "The set is [apple, banana] here" was corrupted into "The set is [apple][banana] here". Added a shape guard that only rewrites blocks whose every comma-separated item matches the reference tag shape
<int>:<id>(e.g.1:92ff35fb); otherwise the block is returned unchanged.Changes:
src/memos/mem_os/utils/reference_utils.pygains a module-level_REFERENCE_ITEM_REregex and a validation step before the split; docstring updated to describe the new condition. New test filetests/mem_os/test_reference_utils.pycovers the regression case from the issue, mixed-content brackets (e.g.[1:92ff35fb, banana]), numeric-only lists ([1, 2, 3]), non-integer prefixes ([a:1, b:2]), and the original happy path ([1:92ff35fb, 4:bfe6f044]→ split, with and without space after comma).Verification:
pytest tests/mem_os/test_reference_utils.py -q→ 11 passed.pytest tests/mem_os/ -q→ 47 passed.ruff check+ruff format --check→ clean. Task file archived to memos-autodev-specs main.Related Issue (Required): Fixes #2446
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Automated tests are pending.
Checklist
@WeiminLeeplease review this PR.Reviewer Checklist