Skip to content

Fix #2420: clean_json_response corrupts JSON values containing code fences - #2422

Closed
Memtensor-AI wants to merge 2 commits into
MemTensor:dev-v2.0.36from
Memtensor-AI:bugfix/autodev-2420-20260928074010713
Closed

Memtensor-AI wants to merge 2 commits into
MemTensor:dev-v2.0.36from
Memtensor-AI:bugfix/autodev-2420-20260928074010713

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fix issue #2420 — clean_json_response was globally stripping every occurrence via str.replace, corrupting JSON string values that legitimately contained fenced code blocks (e.g. {"text": "py\ncode```"}) before json.loads. Both chat_handler and suggestion_handler feed LLM output through this helper, so any suggestion or chat metadata carrying a markdown snippet was silently mangled.

The fix rewrites clean_json_response in src/memos/mem_os/utils/format_utils.py to recognise only the outermost fence: an anchored regex ^[^\n`]*\n matches an opening fence line (with optional info string like "json"), paired with a trailing at the very end. When both are present, only those delimiters are sliced off; interior fences are never candidates for removal. The None-input ValueError guard from #1525 is preserved.

Tests: added 4 red-then-green regression cases in tests/mem_os/test_format_utils.py covering wrapped/unwrapped payloads with interior py blocks and literal json snippets (including the exact reproduction from the issue). Full test run: 12/12 in test_format_utils.py green; 40/40 across tests/mem_os/ green; ruff check + format --check both clean. Commit 95dbd64 pushed to bugfix/autodev-2420-20260928074010713.

Related Issue (Required): Fixes #2420

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Automated tests are pending.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@WeiminLee please review this PR.

Reviewer Checklist

The old implementation stripped every ``` occurrence globally, corrupting
JSON string values that legitimately contained fenced code blocks (e.g.
{"text": "```py\ncode```"}) before json.loads. Both chat_handler and
suggestion_handler pipe LLM output through this helper, so any payload
carrying a markdown snippet came back mangled.

Recognise only the outermost fence: anchor a regex at the start of the
trimmed response for the opening line (``` + optional info string + \n),
pair it with a trailing ``` at the very end, and slice off exactly those
delimiters. Interior fences are never candidates for removal.

Adds 4 red-then-green regression tests covering wrapped/unwrapped
payloads with interior ```py blocks and literal ```json snippets.

Refs: MemTensor#2420
@Memtensor-AI Memtensor-AI added ai:generated Generated or modified by AI | 由 AI 生成或修改 area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Sep 28, 2026
@Memtensor-AI

Memtensor-AI commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2422
Task: d7dd51fe8ad9ac03
Base: dev-v2.0.36
Head: bugfix/autodev-2420-20260928074010713
Head SHA: 46ed407efc3e7d391de86282b645ccac137d35d1

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🔧 Open Code Review requested Agent fix

Open Code Review found 1 issue(s). I have resumed the development Agent to fix them.

  • Task: d7dd51fe8ad9ac03
  • Fix attempt: 1/2
  • Finding delta: 0 repeated / 1 new / 0 likely resolved

The Agent will push a new commit to this PR branch. OCR will recheck after the commit is pushed.

Tighten clean_json_response so the closing markdown fence must appear on
its own line ('\n```' at end). The prior endswith('```') check was
too permissive: any trailing triple-backtick — including one that
belongs to a JSON string value — could be interpreted as a closing
fence and stripped, corrupting the payload.

Adds a regression test covering '```json\n{"a": "x```"}\n```'
to lock in that interior code fences survive when the wrapper uses a
proper on-its-own-line closing fence.

Addresses Open Code Review finding on PR MemTensor#2422.
@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically marked as stale due to inactivity.

@github-actions github-actions Bot added the status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt. label Oct 31, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically closed due to inactivity.

@github-actions github-actions Bot closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 area:core MOS 编排层 / 框架底座 / 跨模块问题 status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants