Skip to content

Malformed extraction JSON silently becomes an empty successful /product/add response #2456

Closed as not planned

Description

@kristhianmanue1

Summary

A malformed LLM extraction response can be converted into an empty mapping by parse_json_result, after which synchronous /product/add reports success with data: [] and no memories are stored.

Related: #1355, fixed by #1977. This appears to be a different failure path: that fix corrected the fallback key in SimpleStructMemReader; this report concerns a parser failure returning {} without raising, so an exception-based fallback is not invoked.

Version and environment

  • Upstream main inspected: a7367d07e55db61099f7b4e2c1108bc5831a24f3 (also our checkout base).
  • Self-hosted Python server, Python 3.11 on macOS.
  • LLM: glm-5.3-flash, through an OpenAI-compatible Z.AI endpoint.
  • Neo4j Community 5.26.6, Qdrant 1.15.3, local Ollama embeddings.
  • /product/add with async_mode: "sync", explicitly created test cube.

Scope of reproduction: the parser failure below was reproduced from the unmodified upstream function at the pinned commit. The end-to-end API observation came from a local checkout with unrelated keyword-search/feedback and debug-logging changes; we have not reproduced the entire HTTP path on a pristine deployment. At the time of the failure, mem_reader/utils.py was unmodified. The isolated parser reproduction needs no LLM, database or server port.

Minimal deterministic reproduction

Run against the unmodified parser at the commit above, in an environment with MemOS dependencies installed:

from memos.mem_reader.utils import parse_json_result

valid = '{"memory list": [{"value": "synthetic evidence"}]}'
malformed = '{"memory list": [{"value": "synthetic evidence"}],}'

assert parse_json_result(valid) == {
    "memory list": [{"value": "synthetic evidence"}]
}
assert parse_json_result(malformed) == {}

The trailing comma is invalid JSON. The concern is not that strict parsing rejects it; the concern is that rejection becomes indistinguishable from a successful empty extraction downstream.

Observed HTTP behavior

In a bounded document-ingestion test:

  1. Created a new cube and submitted the document through synchronous /product/add.
  2. The extraction provider returned finish_reason: "stop", with a trailing comma before the final closing brace. This was not a reported output truncation.
  3. The parser logged Failed to decode JSON: Expecting property name enclosed in double quotes and returned {}.
  4. The downstream log reported No add/update items prepared.
  5. The API returned:
{"code": 200, "message": "Memory added successfully", "data": []}
  1. /product/get_all and subsequent searches in that cube returned zero memories. Later chat requests had no supporting context.

The failed extraction consumed 13,545 provider-reported tokens. We did not retry it inside that frozen evaluation. No source document, private logs, credentials or user identifiers are included here.

Relevant code

Expected behavior

Please distinguish these outcomes in the extraction/add flow:

  • Extraction failed because the model output could not be parsed or validated.
  • Extraction succeeded and legitimately found no memories.
  • Memories were successfully persisted.

A valid empty extraction should remain supported. An invalid extraction should not be represented solely as “Memory added successfully” with an empty list. A structured diagnostic or propagated extraction failure would let callers handle this without assuming persistence succeeded.

Local mitigation and validation limits

We tested a narrow local mitigation that removes commas immediately before } or ] only outside quoted strings, reparses with json.loads, and emits an explicit repair warning. It does not try to invent missing values or repair arbitrary malformed output.

  • Four regression tests cover repair reporting, punctuation/escaped quotes inside strings, unchanged valid JSON, and rejection of a missing value.
  • Two of these tests failed before the patch; all four passed afterward.
  • The selected complete tests/mem_reader run passed: 75 passed, with three warnings; formatting/lint passed in the validation copy.
  • Replaying the original malformed final response locally recovered five memory objects, with no model calls or database writes.
  • A later live run on the patched service stored five memories and supported both searches. However, its fresh model response was already valid JSON, so that live success is not causal evidence that the repair branch handled the original failure.

This mitigation does not solve the general empty-success ambiguity, and syntactic validity does not establish semantic fidelity. We are reporting the failure mode rather than proposing that every empty extraction must be treated as an error.

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    area:coreMOS 编排层 / 框架底座 / 跨模块问题
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Oct 3, 2026
  2. self-assigned this
    on Oct 3, 2026
  3. Memtensor-AI commented on Oct 3, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: f2cc0bcd220b1a33
    Working branch: bugfix/autodev-2456-20261003133440133
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. Memtensor-AI commented on Oct 3, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task f2cc0bcd220b1a33 completed.

    Summary: Fix issue #2456: malformed extraction JSON with a trailing comma before } or ] (common output from GLM / Z.AI / Kimi style models) was silently converted to {} by parse_json_result, which downstream surfaced as POST /product/add returning {"code":200,"message":"Memory added successfully","data":[]} with zero memories persisted. The multi_modal_struct._process_string_fine exception-based fallback never fired because no exception was raised. Related to #1355 (fixed by #1977) but a distinct failure path — that fix only corrected SimpleStructMemReader's fallback key.

    The fix adds a narrow last-chance repair in src/memos/mem_reader/utils.py::parse_json_result: a _strip_trailing_commas scanner walks the payload character-by-character, tracks whether it is inside a quoted string (respecting backslash-escaped quotes), and removes commas that sit immediately before } or ] outside strings. On successful repair it emits a [JSONParse] Repaired malformed JSON... warning so the previously-silent failure mode is observable; on genuine failure it still returns {} to preserve the historical contract for the 15 call sites. Zero signature change, zero API contract change.

    Testing: 11 new regression cases in tests/mem_reader/test_parse_json_result.py cover the exact repro from the bug body, trailing commas before }/]/newlines/fenced code blocks, safety cases (literal ,} inside strings must survive, escaped quotes must not confuse the scanner), and the preserved {} return for genuinely malformed JSON. All 11 pass. Adjacent suites (test_simple_structure, test_simple_struct_fallback) still green: 20/20. Full tests/mem_reader/ run: 80 passed, 2 pre-existing environment failures (test_doc_local_file, test_doc_mixed) requiring the optional markitdown extra — confirmed reproducible on untouched dev-v2.0.36 HEAD. Ruff format + check clean. Archive synced and pushed to memos-autodev-specs main (commit 33ce785).

    Base branch: main
    Branch: bugfix/autodev-2456-20261003133440133
    Commit: 2ea35df
    PR: #2457
    Assigned to: @syzsunshine219
    Reviewers: @WeiminLee

  5. added
    ai:testingAI agent is running tests | AI 正在运行测试
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Oct 3, 2026
  6. added a commit that references this issue on Oct 3, 2026
  7. github-actions commented on Nov 3, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  8. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Nov 3, 2026
  9. github-actions commented on Nov 11, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务ai:testingAI agent is running tests | AI 正在运行测试area:coreMOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions