Repository navigation
fix(mem_reader): raise LLMResponseParseError instead of returning {} on malformed JSON (#2456) - #2607
syzsunshine219 wants to merge 2 commits into
Conversation
…on malformed JSON (MemTensor#2456) parse_json_result silently returned {} on every parse failure, so a malformed LLM extraction was reported as an empty (but successful) /product/add response. The HTTP layer returned {"code": 200, "data": []} for an extraction that actually failed, hiding the wasted tokens and never alerting operators that a request had silently dropped. This makes parse_json_result raise LLMResponseParseError (a new ParserError subclass in memos.exceptions) on any non-recoverable parse failure, so callers can distinguish "the model produced no memories" from "the model produced output we couldn't parse". Every in-tree caller already wraps parse_json_result in a try/except (multi_modal_struct, simple_struct, strategy_struct, file_content_parser, searcher, task_goal_parser), so the exception is caught and surfaces as the synthetic-memory-list fallback instead of the silent {} return path. The literal {} JSON value still returns {} so successful empty extractions are preserved. The fix keeps test_simple_struct_fallback.py::test_process_chat_data_fine_yields_node_when_llm_unparseable passing: _safe_parse catches the new exception and returns None, which still triggers the chat fallback that emits a salvageable UserMemory item. - New exception LLMResponseParseError(ParserError) in memos.exceptions - parse_json_result now raises on empty input, missing JSON object, every unrecoverable JSONDecodeError, while still returning the dict when one is found (including a literal {}) - New tests/mem_reader/test_parse_json_result_strict.py covers the trailing-comma reproducer from the issue, the prose-only 'I cannot output JSON' symptom, empty/None input, no JSON object, and the exception-chain contract (the chained cause preserves the underlying JSONDecodeError for debugging) - Updated test_simple_structure.py::test_parse_json_result_failure and the docstring on test_simple_struct_fallback.py to match the new contract
The test module injected stub packages for memos/memos.log/memos.exceptions/
memos.mem_reader at import time. Those stubs stayed in sys.modules for the
whole pytest session and broke every later test that imported the real
package: memos.log lost _LOGGING_CONFIGURED_PID/summarize_* helpers, the
stub root package shadowed real submodules (memos.chunkers, memos.mem_user),
and server_router tests saw 'module memos has no attribute log'. The stub
loader also resolved src/... paths against the current working directory,
so collecting the file from any other directory raised FileNotFoundError
(autotest 'collection' failures).
Replace the stub machinery with normal top-level imports — pyproject.toml
already sets pythonpath=src and sibling tests import the same modules the
same way. Also assert the exception chain directly via
exc_info.value.__cause__ being a json.JSONDecodeError, which the current
or-clause ('Failed' in message) could never fail.
🤖 Open Code ReviewTarget: PR #2607 🔍 OpenCodeReview found 1 issue(s) in this PR. 1.
|
✅ Automated Test Results: PASSEDRequired tests passed (19/19 executed, 0 skipped). Duration: 9s [advisory, non-gating] AI-generated tests on branch test/auto-gen-b24b2e83d923813e-20261110013053: 102/102 passed — these do NOT affect the PR verdict; review the branch manually. Branch: Tested source: |
|
Superseded by #2633, targeting dev-v2.0.37. The replacement preserves both original commits and adds parse-error propagation through reader fallback and thread boundaries. Sync add now fails before storage or scheduling on malformed extraction; valid empty results still succeed. Async extraction retains fast memories and records a failed task. The replacement's memory, reader, scheduler and API tests completed with 475 passed and 1 existing skip. Closing this PR to consolidate review in #2633. Thank you for the original contribution. |
) ## Description Ranked search results could share nested metadata with the original candidates, and malformed LLM extraction responses could be converted into successful empty writes by reader fallback and worker exception handlers. Selected searched metadata is now copied deeply without Pydantic revalidation. Parsing failures propagate through simple and multimodal extraction, document workers and public reader boundaries. Synchronous /product/add returns HTTP 500 before storage or scheduling on a parse failure, while valid {} and {"memory list": []} remain successful empty results. Async fine-extraction failures retain the existing fast memories and emit a failed task event. This PR supersedes #2600 and #2607 and targets dev-v2.0.37. Original contributions from @yiwang-one and @RerankerGuo are preserved in separate cherry-picked commits, followed by the metadata-isolation and write-path fixes. Maintainer edits remained disabled on the original PRs. Related issue #2456. ## Type of change - [x] Bug fix ## How Has This Been Tested? - [x] Unit tests and regression tests, using synthetic data and mocked model/storage boundaries - [x] Test steps completed with Python 3.12 - [x] Automated API tests exercise the real AddHandler, SingleCubeView and SimpleStruct reader through POST /product/add Commands and actual results make format All checks passed! 668 files left unchanged poetry run pytest tests/memories/ tests/mem_reader/ tests/mem_scheduler/ tests/api/ -q -ra --tb=short 475 passed, 1 skipped, 8 warnings in 11.13s The skipped scheduler configuration test documents that the current implementation accepts all components being None. Existing warnings cover Pydantic serialization, deprecated UTC timestamp calls and the intentional thread-exception fixture. Regression coverage includes nested metadata independence across repeated searches, unrecoverable JSON versus legitimate empty results, both reader worker boundaries, failed async task reporting and preservation of fast memories. Both follow-up commits passed the repository pre-commit hooks. ## Checklist - [x] Self-review completed - [x] Comments explain metadata ownership and parse-failure behavior - [x] Regression tests cover the changed behavior - [x] Documentation scope reviewed for these internal fixes - [x] Related issue and superseded PRs linked - [x] Original contributors mentioned above for review context ## Reviewer Checklist - [x] Issue #2456 and superseded PRs linked - [x] Local format, pre-commit and relevant pytest checks passed - [x] Tests provided
Restores the proposal from #2487 as an open PR against
main; its former base branch,dev-v2.0.36, is no longer available. This proposal still needs its implementation revised before merging.Source head:
4ee17945c75bde27fbca09754344bcf9c06ccdcc. No new source-code changes are introduced by restoring this proposal. Earlier discussion and evidence remain in #2487; validation claims in the original description refer to the earlier snapshot.Outstanding review scope
当前parse_json_result确实把解析失败返回{},但单独增加LLMResponseParseError不能保证错误透传:SimpleStruct._safe_parse会catch后返回None,MultiModal._get_llm_response会catch后生成fallback。因此需跨调用链重做失败契约和无原文泄漏日志,不能直接恢复原小补丁当作端到端修复。
Original proposal
Summary
parse_json_resultsilently returned{}on every parse failure, so a malformed LLM extraction was reported as an empty (but successful)/product/addresponse. The HTTP layer returned{"code": 200, "data": []}for an extraction that actually failed, hiding the wasted tokens and never alerting operators that a request had silently dropped.Root cause
src/memos/mem_reader/utils.py::parse_json_resulthad three silent return paths:Both branches are reached when the LLM emits invalid JSON or non-JSON chatter. Downstream, every
/product/addcaller (multi_modal_struct,simple_struct,strategy_struct,file_content_parser) checks the empty{}against truthiness and either short-circuits to a "no items" log line or falls into an unintended code path — never raising, never alerting.Fix
LLMResponseParseError(ParserError)tomemos.exceptionsso callers can distinguish "no memories produced" from "could not parse the response".parse_json_resultnow raises on empty input, on missing JSON object, and on every unrecoverableJSONDecodeError. The chained__cause__preserves the underlying decode error for debugging.{}) still return the parsed dict.Caller audit
Every in-tree caller of
parse_json_resultalready wraps the call intry/except Exception, so the new exception is caught and routed to the existing fallback instead of the silent-{}path:mem_reader/multi_modal_struct.py:504{}→ "no items prepared" →200 OKwith emptydata)mem_reader/multi_modal_struct.py:600Nonefrom mergemem_reader/simple_struct.py:118(and_safe_parseat:269)None, downstream fallback firesmem_reader/strategy_struct.py:55mem_reader/read_multi_modal/file_content_parser.py:94{}memories/textual/tree_text_memory/retrieve/task_goal_parser.py:117ValueErrorafter N attemptsmemories/textual/tree_text_memory/retrieve/searcher.py:1384[query]textual/general.pyandtextual/tree_text_memory/.../retrieve_utils.pydefine their ownparse_json_resultwith a different body — those are not affected by this change.Tests
tests/mem_reader/test_parse_json_result_strict.py— 10 red/green cases:{}, markdown-fence strippingNoneinput, prose-only "I cannot output JSON" symptom, exception-chain preservationtests/mem_reader/test_simple_structure.py::test_parse_json_result_failureto expect the new raisetests/mem_reader/test_simple_struct_fallback.py::test_process_chat_data_fine_yields_node_when_llm_unparseableto describe the new exception path; the test still passes because_safe_parsecatches the new exception and returnsNone, so the symptom-level chat fallback (which is the actual user-facing contract for fix: /product/add returns 200 success but stores no memory (No add/update items prepared) #1493) continues to emit a salvageableUserMemoryitem.Verification
The full existing
test_simple_structure.pysuite pulls in the heavy LLM/embedder stack and was not exercised end-to-end here, but every individual test case the PR touches was verified against the new behavior in isolation.Out of scope
memosor the OpenAPI contract.Makefile/pyproject.toml/.pre-commit-config.yaml/.github/workflows/.Fixes #2456.