Skip to content

fix(mem_reader): raise LLMResponseParseError instead of returning {} on malformed JSON (#2456) - #2607

Closed
syzsunshine219 wants to merge 2 commits into
MemTensor:dev-v2.0.37from
RerankerGuo:fix/issue-2456-malformed-json-detection
Closed

syzsunshine219 wants to merge 2 commits into
MemTensor:dev-v2.0.37from
RerankerGuo:fix/issue-2456-malformed-json-detection

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

Restores the proposal from #2487 as an open PR against main; its former base branch, dev-v2.0.36, is no longer available. This proposal still needs its implementation revised before merging.

Source head: 4ee17945c75bde27fbca09754344bcf9c06ccdcc. No new source-code changes are introduced by restoring this proposal. Earlier discussion and evidence remain in #2487; validation claims in the original description refer to the earlier snapshot.

Outstanding review scope

当前parse_json_result确实把解析失败返回{},但单独增加LLMResponseParseError不能保证错误透传:SimpleStruct._safe_parse会catch后返回None,MultiModal._get_llm_response会catch后生成fallback。因此需跨调用链重做失败契约和无原文泄漏日志,不能直接恢复原小补丁当作端到端修复。


Original proposal

Summary

parse_json_result silently returned {} on every parse failure, so a malformed LLM extraction was reported as an empty (but successful) /product/add response. The HTTP layer returned {"code": 200, "data": []} for an extraction that actually failed, hiding the wasted tokens and never alerting operators that a request had silently dropped.

Root cause

src/memos/mem_reader/utils.py::parse_json_result had three silent return paths:

i = s.find("{")
if i == -1:
    return {}                                  # ← empty/no JSON object

# ... cheap-close attempt
return {}                                      # ← unrecoverable parse error

Both branches are reached when the LLM emits invalid JSON or non-JSON chatter. Downstream, every /product/add caller (multi_modal_struct, simple_struct, strategy_struct, file_content_parser) checks the empty {} against truthiness and either short-circuits to a "no items" log line or falls into an unintended code path — never raising, never alerting.

Fix

  • Add LLMResponseParseError(ParserError) to memos.exceptions so callers can distinguish "no memories produced" from "could not parse the response".
  • parse_json_result now raises on empty input, on missing JSON object, and on every unrecoverable JSONDecodeError. The chained __cause__ preserves the underlying decode error for debugging.
  • Successful parses (including a literal {}) still return the parsed dict.
  • The cheap-close and backslash-escape repair attempts remain; they only succeed when valid JSON comes out.

Caller audit

Every in-tree caller of parse_json_result already wraps the call in try/except Exception, so the new exception is caught and routed to the existing fallback instead of the silent-{} path:

caller site behavior with new raise
mem_reader/multi_modal_struct.py :504 synthetic-memory-list fallback (was: silent {} → "no items prepared" → 200 OK with empty data)
mem_reader/multi_modal_struct.py :600 returns None from merge
mem_reader/simple_struct.py :118 (and _safe_parse at :269) returns None, downstream fallback fires
mem_reader/strategy_struct.py :55 synthetic-memory-list fallback
mem_reader/read_multi_modal/file_content_parser.py :94 returns {}
memories/textual/tree_text_memory/retrieve/task_goal_parser.py :117 existing retry loop raises ValueError after N attempts
memories/textual/tree_text_memory/retrieve/searcher.py :1384 returns [query]

textual/general.py and textual/tree_text_memory/.../retrieve_utils.py define their own parse_json_result with a different body — those are not affected by this change.

Tests

  • New tests/mem_reader/test_parse_json_result_strict.py — 10 red/green cases:
    • Success: parsed object, literal {}, markdown-fence stripping
    • Failure (raises): trailing-comma reproducer from the issue, no JSON object, unclosed fence, empty input, None input, prose-only "I cannot output JSON" symptom, exception-chain preservation
  • Updated tests/mem_reader/test_simple_structure.py::test_parse_json_result_failure to expect the new raise
  • Updated the docstring on tests/mem_reader/test_simple_struct_fallback.py::test_process_chat_data_fine_yields_node_when_llm_unparseable to describe the new exception path; the test still passes because _safe_parse catches the new exception and returns None, so the symptom-level chat fallback (which is the actual user-facing contract for fix: /product/add returns 200 success but stores no memory (No add/update items prepared) #1493) continues to emit a salvageable UserMemory item.

Verification

PYTHONPATH=src pytest tests/mem_reader/test_parse_json_result_strict.py -v
============================== 10 passed in 0.12s ==============================

ruff format --check  # all touched files pass
ruff check           # only pre-existing BLE001 errors remain; this change introduces 0 new findings

The full existing test_simple_structure.py suite pulls in the heavy LLM/embedder stack and was not exercised end-to-end here, but every individual test case the PR touches was verified against the new behavior in isolation.

Out of scope

  • Adds no new dependencies.
  • Does not modify the public API surface of memos or the OpenAPI contract.
  • Does not touch DB schemas, migrations, or graph models.
  • Does not touch Makefile / pyproject.toml / .pre-commit-config.yaml / .github/workflows/.

Fixes #2456.

RerankerGuo and others added 2 commits November 4, 2026 21:57
…on malformed JSON (MemTensor#2456)

parse_json_result silently returned {} on every parse failure, so a malformed LLM extraction was reported as an empty (but successful) /product/add response. The HTTP layer returned {"code": 200, "data": []} for an extraction that actually failed, hiding the wasted tokens and never alerting operators that a request had silently dropped.

This makes parse_json_result raise LLMResponseParseError (a new ParserError subclass in memos.exceptions) on any non-recoverable parse failure, so callers can distinguish "the model produced no memories" from "the model produced output we couldn't parse". Every in-tree caller already wraps parse_json_result in a try/except (multi_modal_struct, simple_struct, strategy_struct, file_content_parser, searcher, task_goal_parser), so the exception is caught and surfaces as the synthetic-memory-list fallback instead of the silent {} return path. The literal {} JSON value still returns {} so successful empty extractions are preserved.

The fix keeps test_simple_struct_fallback.py::test_process_chat_data_fine_yields_node_when_llm_unparseable passing: _safe_parse catches the new exception and returns None, which still triggers the chat fallback that emits a salvageable UserMemory item.

- New exception LLMResponseParseError(ParserError) in memos.exceptions
- parse_json_result now raises on empty input, missing JSON object, every unrecoverable JSONDecodeError, while still returning the dict when one is found (including a literal {})
- New tests/mem_reader/test_parse_json_result_strict.py covers the trailing-comma reproducer from the issue, the prose-only 'I cannot output JSON' symptom, empty/None input, no JSON object, and the exception-chain contract (the chained cause preserves the underlying JSONDecodeError for debugging)
- Updated test_simple_structure.py::test_parse_json_result_failure and the docstring on test_simple_struct_fallback.py to match the new contract
The test module injected stub packages for memos/memos.log/memos.exceptions/
memos.mem_reader at import time. Those stubs stayed in sys.modules for the
whole pytest session and broke every later test that imported the real
package: memos.log lost _LOGGING_CONFIGURED_PID/summarize_* helpers, the
stub root package shadowed real submodules (memos.chunkers, memos.mem_user),
and server_router tests saw 'module memos has no attribute log'. The stub
loader also resolved src/... paths against the current working directory,
so collecting the file from any other directory raised FileNotFoundError
(autotest 'collection' failures).

Replace the stub machinery with normal top-level imports — pyproject.toml
already sets pythonpath=src and sibling tests import the same modules the
same way. Also assert the exception chain directly via
exc_info.value.__cause__ being a json.JSONDecodeError, which the current
or-clause ('Failed' in message) could never fail.
@Memtensor-AI Memtensor-AI added area:memory 记忆存储、检索、更新、召回逻辑 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Nov 9, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2607
Task: b24b2e83d923813e
Base: main
Head: fix/issue-2456-malformed-json-detection
Head SHA: 4ee17945c75bde27fbca09754344bcf9c06ccdcc

🔍 OpenCodeReview found 1 issue(s) in this PR.


1. src/memos/mem_reader/utils.py (L51-L52)

Raising on an empty/whitespace response silently changes downstream behaviour in the multi-modal reader. Previously parse_json_result("") returned {}, so MultiModalStructMemReader._get_llm_response (multi_modal_struct.py:502-519) returned an empty dict and _process_string_fine (line 702) produced zero memories. Now the LLMResponseParseError is swallowed by that broad except Exception, which builds a "salvage" item with value = mem_str — i.e. an LLM refusal/truncation/empty reply now persists the whole chunk as a single memory. The same happens for chatter-only replies via the if i == -1 raise below. Since every caller (simple_struct._safe_parse, file_content_parser, multi_modal_struct) catches except Exception, the intended "no memories vs. unparseable" distinction is also lost. Consider keeping {} (or a separate, explicit empty-result signal) for the empty-response case and letting the callers decide, or updating those call sites so a parse failure does not fabricate a memory item.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

Required tests passed (19/19 executed, 0 skipped). Duration: 9s [advisory, non-gating] AI-generated tests on branch test/auto-gen-b24b2e83d923813e-20261110013053: 102/102 passed — these do NOT affect the PR verdict; review the branch manually.

Branch: fix/issue-2456-malformed-json-detection

Tested source: 4ee17945c75bde27fbca09754344bcf9c06ccdcc
Tested base: 3a16e5fe3048a209f722806f4123fa2e533588fd
Tested commit: e5106b541f4975ce34bd3becea5c6a891d93b70e
These results apply only to this immutable source/base snapshot.

@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 18, 2026 06:33
@endxxxx

endxxxx commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Superseded by #2633, targeting dev-v2.0.37. The replacement preserves both original commits and adds parse-error propagation through reader fallback and thread boundaries. Sync add now fails before storage or scheduling on malformed extraction; valid empty results still succeed. Async extraction retains fast memories and records a failed task.

The replacement's memory, reader, scheduler and API tests completed with 475 passed and 1 existing skip. Closing this PR to consolidate review in #2633. Thank you for the original contribution.

@endxxxx endxxxx closed this Nov 19, 2026
endxxxx added a commit that referenced this pull request Nov 19, 2026
)

## Description

Ranked search results could share nested metadata with the original
candidates, and malformed LLM extraction responses could be converted
into successful empty writes by reader fallback and worker exception
handlers.

Selected searched metadata is now copied deeply without Pydantic
revalidation. Parsing failures propagate through simple and multimodal
extraction, document workers and public reader boundaries. Synchronous
/product/add returns HTTP 500 before storage or scheduling on a parse
failure, while valid {} and {"memory list": []} remain successful empty
results. Async fine-extraction failures retain the existing fast
memories and emit a failed task event.

This PR supersedes #2600 and #2607 and targets dev-v2.0.37. Original
contributions from @yiwang-one and @RerankerGuo are preserved in
separate cherry-picked commits, followed by the metadata-isolation and
write-path fixes. Maintainer edits remained disabled on the original
PRs.

Related issue #2456.

## Type of change

- [x] Bug fix

## How Has This Been Tested?

- [x] Unit tests and regression tests, using synthetic data and mocked
model/storage boundaries
- [x] Test steps completed with Python 3.12
- [x] Automated API tests exercise the real AddHandler, SingleCubeView
and SimpleStruct reader through POST /product/add

Commands and actual results

    make format
    All checks passed!
    668 files left unchanged

poetry run pytest tests/memories/ tests/mem_reader/ tests/mem_scheduler/
tests/api/ -q -ra --tb=short
    475 passed, 1 skipped, 8 warnings in 11.13s

The skipped scheduler configuration test documents that the current
implementation accepts all components being None. Existing warnings
cover Pydantic serialization, deprecated UTC timestamp calls and the
intentional thread-exception fixture.

Regression coverage includes nested metadata independence across
repeated searches, unrecoverable JSON versus legitimate empty results,
both reader worker boundaries, failed async task reporting and
preservation of fast memories.

Both follow-up commits passed the repository pre-commit hooks.

## Checklist

- [x] Self-review completed
- [x] Comments explain metadata ownership and parse-failure behavior
- [x] Regression tests cover the changed behavior
- [x] Documentation scope reviewed for these internal fixes
- [x] Related issue and superseded PRs linked
- [x] Original contributors mentioned above for review context

## Reviewer Checklist

- [x] Issue #2456 and superseded PRs linked
- [x] Local format, pre-commit and relevant pytest checks passed
- [x] Tests provided
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:memory 记忆存储、检索、更新、召回逻辑 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Malformed extraction JSON silently becomes an empty successful /product/add response

5 participants