Skip to content

fix(memory): decode legacy JSON info metadata - #2602

Merged
endxxxx merged 3 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2288-20260827081609558
Nov 19, 2026
Merged

endxxxx merged 3 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2288-20260827081609558

Conversation

@syzsunshine219

@syzsunshine219 syzsunshine219 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Historical Neo4j records can contain info as a JSON object string. Building a TextualMemoryItem from those records fails dict validation, even though the same representation is already supported for internal_info by #2517.

Extend the existing metadata validator to info and report the relevant field name for encoded non-object values. Existing dictionaries, None, missing info and modern flattened metadata remain compatible. Malformed JSON, encoded arrays/scalars and double-encoded strings raise ValidationError.

This PR is based on dev-v2.0.37 and retains the original #2289 contribution history. The final diff concentrates the remaining fix at the metadata validation boundary. Its tests exercise the native Neo4j single-node and Community single/batch parsers through TextualMemoryItem construction, including embedding alignment and strict handling of corrupt records.

Fixes #2288.

Type of change

  • Bug fix

How Has This Been Tested?

  • Unit tests cover both metadata fields across all four metadata model classes
  • Regression cases cover historical object strings, whitespace, nested Unicode data, existing dictionaries, None, missing fields, flattened data and invalid JSON
  • Existing API, memory and graph-store regression suites completed

Actual verification output

make format
All checks passed!

poetry run pytest tests/graph_dbs/ tests/memories/ tests/api/ -q -ra --tb=short
449 passed, 9 skipped, 6 warnings in 9.39s

The nine skips are opt-in database integration tests with services unconfigured. The Neo4j parser regressions use mocked vector clients. This run exercises the parser/model boundary with synthetic legacy records.

Before the validator change, the focused regression suite produced 12 failures and 82 passing controls. Repository pre-commit hooks passed on the follow-up commit.

Checklist

  • Self-review completed
  • Existing validator comment explains the storage representation
  • Regression tests provided
  • Documentation applicability reviewed for the metadata compatibility fix
  • Related issue and original proposal linked
  • Review context provided for @MatthewZhuang and @CarltonXiang

Reviewer Checklist

MemOS AutoDev and others added 3 commits August 27, 2026 16:53
…emTensor#2288)

`_sanitize_neo4j_value` serializes every dict-typed metadata value into
a JSON string on write, but `_parse_node` / `_parse_nodes` only reversed
the `sources` transform. `internal_info` and `info` therefore came back
as `str`, breaking `TextualMemoryItem`'s `dict | None` validation during
recall on document-ingest / Dream-enabled deployments.

Adds a small `_deserialize_dict_field` helper (symmetric with the
write-side sanitizer) and wires it into both `Neo4jGraphDB._parse_node`
and `Neo4jCommunityGraphDB._parse_node` / `_parse_nodes` for the two
known dict-typed fields. JSON-object-shaped strings deserialize to
dicts, malformed strings fall back to None so pydantic still accepts
them, all other values pass through unchanged. Write path and other
graph backends untouched.

Adds 16 regression tests including an end-to-end guard that feeds the
parser output into `TreeNodeTextualMemoryMetadata` to close the loop
with the failure point.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address two OCR findings on MemTensor#2289:

1. src/memos/graph_dbs/neo4j.py: `_deserialize_dict_field` silently
   dropped malformed dict-shaped strings to None, making the event
   indistinguishable from a legitimately-None field. Emit a
   `logger.warning` before returning None so operators can trace the
   corruption. Uses the module-level logger already declared at line 13.

2. tests/graph_dbs/test_neo4j_internal_info_roundtrip.py:
   `TestDeserializeDictField._helper` is a stable module-level lookup
   with no `self` use — promote it to `@staticmethod` for idiomatic
   pytest style.

Verified: pytest tests/graph_dbs/ (47 passed, 3 pre-existing skips) +
ruff check/format clean on both files.
@endxxxx endxxxx changed the title Fix #2288: Bug: internal_info / info stored as JSON string by Neo4j sanitizer but never des fix(memory): decode legacy JSON info metadata Nov 19, 2026
@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 19, 2026 11:58
@endxxxx
endxxxx merged commit a0fe4d0 into MemTensor:dev-v2.0.37 Nov 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: internal_info / info stored as JSON string by Neo4j sanitizer but never deserialized on read — breaks TextualMemoryItem validation on recall

2 participants