Repository navigation
Fix #2288: Bug: internal_info / info stored as JSON string by Neo4j sanitizer but never des - #2289
Memtensor-AI wants to merge 2 commits into
Conversation
…emTensor#2288) `_sanitize_neo4j_value` serializes every dict-typed metadata value into a JSON string on write, but `_parse_node` / `_parse_nodes` only reversed the `sources` transform. `internal_info` and `info` therefore came back as `str`, breaking `TextualMemoryItem`'s `dict | None` validation during recall on document-ingest / Dream-enabled deployments. Adds a small `_deserialize_dict_field` helper (symmetric with the write-side sanitizer) and wires it into both `Neo4jGraphDB._parse_node` and `Neo4jCommunityGraphDB._parse_node` / `_parse_nodes` for the two known dict-typed fields. JSON-object-shaped strings deserialize to dicts, malformed strings fall back to None so pydantic still accepts them, all other values pass through unchanged. Write path and other graph backends untouched. Adds 16 regression tests including an end-to-end guard that feeds the parser output into `TreeNodeTextualMemoryMetadata` to close the loop with the failure point. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Open Code ReviewTarget: PR #2289 🔍 OpenCodeReview found 1 issue(s) in this PR. 1.
|
🔧 Open Code Review requested Agent fixOpen Code Review found 2 issue(s). I have resumed the development Agent to fix them.
The Agent will push a new commit to this PR branch. OCR will recheck after the commit is pushed. |
Address two OCR findings on MemTensor#2289: 1. src/memos/graph_dbs/neo4j.py: `_deserialize_dict_field` silently dropped malformed dict-shaped strings to None, making the event indistinguishable from a legitimately-None field. Emit a `logger.warning` before returning None so operators can trace the corruption. Uses the module-level logger already declared at line 13. 2. tests/graph_dbs/test_neo4j_internal_info_roundtrip.py: `TestDeserializeDictField._helper` is a stable module-level lookup with no `self` use — promote it to `@staticmethod` for idiomatic pytest style. Verified: pytest tests/graph_dbs/ (47 passed, 3 pre-existing skips) + ruff check/format clean on both files.
## Description Neo4j-backed recall can return an empty result after `TextualMemoryItem` rejects a JSON-encoded `metadata.internal_info`. The native ingest reader can also skip the same nodes when constructing `TreeNodeTextualMemoryMetadata`. The chunking path creates `internal_info` as a dictionary. Neo4j serializes dictionary properties as JSON, but its read conversion does not restore this field before Pydantic validation. A short conversation with `internal_info=None` does not exercise the failure. Add a before-validator on `TextualMemoryMetadata.internal_info`, shared by the search and reader metadata models. Decode string values once and require a JSON object. Existing dictionaries and `None` keep their behavior; malformed JSON, encoded arrays/scalars/null, and double-encoded strings remain validation errors. No storage migration, write-path change, request/response route change, new dependency, LLM request, or retrieval-policy change is required. Related issue: #2288. Earlier PR #2289 was closed without merging; the failure remains reproducible on current `main` (`a7367d0`). This change deliberately retains validation errors for malformed values instead of silently replacing them with `None`. ## Type of change - [x] Bug fix ## How Has This Been Tested? - Before the fix: new regression file has **5 failed, 15 passed**, reproducing the JSON-string validation error. - After the fix: new regression file has **20 passed**. - In an isolated Python 3.12 / Pydantic 2.12.4 backend container, with networking disabled, no credentials and no business volumes: ```sh PYTHONPATH=src python -B -m pytest -p no:cacheprovider -o filterwarnings= -o addopts= -q tests/memories/textual/ ``` **80 passed**. Coverage includes all four metadata subclasses, search `from_dict`, direct reader metadata construction, unchanged source/session/content fields, repeated storage JSON round trips, existing dictionary/None values, and rejection of malformed or non-object JSON. The image does not include pytest-asyncio, producing an existing unknown `asyncio_mode` configuration warning; one existing naive-memory test also reports a Pydantic serialization warning. - `make format`: **passed** (`All checks passed!`; `631 files left unchanged`), using the repository-pinned Ruff 0.11.8. - `pre-commit run --files src/memos/memories/textual/item.py tests/memories/textual/test_item.py`: **passed** (all applicable hooks). ## Checklist - [x] Self-reviewed the diff; only the metadata validator and its regression tests change. - [x] Added an explanatory comment at the storage/model boundary. - [x] Added tests that fail before the fix and pass afterward. - [x] Linked the existing issue. No public API or documentation change is required. Maintainer review and upstream CI are pending; the full repository suite was not run locally.
* fix(graph_dbs): deserialize internal_info / info on Neo4j read path (#2288) `_sanitize_neo4j_value` serializes every dict-typed metadata value into a JSON string on write, but `_parse_node` / `_parse_nodes` only reversed the `sources` transform. `internal_info` and `info` therefore came back as `str`, breaking `TextualMemoryItem`'s `dict | None` validation during recall on document-ingest / Dream-enabled deployments. Adds a small `_deserialize_dict_field` helper (symmetric with the write-side sanitizer) and wires it into both `Neo4jGraphDB._parse_node` and `Neo4jCommunityGraphDB._parse_node` / `_parse_nodes` for the two known dict-typed fields. JSON-object-shaped strings deserialize to dicts, malformed strings fall back to None so pydantic still accepts them, all other values pass through unchanged. Write path and other graph backends untouched. Adds 16 regression tests including an end-to-end guard that feeds the parser output into `TreeNodeTextualMemoryMetadata` to close the loop with the failure point. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(graph_dbs): log dropped internal_info deserialize + tidy test helper Address two OCR findings on #2289: 1. src/memos/graph_dbs/neo4j.py: `_deserialize_dict_field` silently dropped malformed dict-shaped strings to None, making the event indistinguishable from a legitimately-None field. Emit a `logger.warning` before returning None so operators can trace the corruption. Uses the module-level logger already declared at line 13. 2. tests/graph_dbs/test_neo4j_internal_info_roundtrip.py: `TestDeserializeDictField._helper` is a stable module-level lookup with no `self` use — promote it to `@staticmethod` for idiomatic pytest style. Verified: pytest tests/graph_dbs/ (47 passed, 3 pre-existing skips) + ruff check/format clean on both files. --------- Co-authored-by: MemOS AutoDev <autodev@memos.local> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Wenqiang Wei <46308778+endxxxx@users.noreply.github.com>
Description
Fixes #2288:
Neo4jGraphDB/Neo4jCommunityGraphDBsanitize every dict-typed metadata value into a JSON string on write via_sanitize_neo4j_value, but the read path only reversed thesourcestransform.internal_infoandinfotherefore came back asstr, breakingTextualMemoryItem'sdict | Nonepydantic validation during recall on document-ingest / Dream-enabled deployments.Adds a small
_deserialize_dict_fieldhelper insrc/memos/graph_dbs/neo4j.py(symmetric with the write-side sanitizer, module constant_DICT_METADATA_FIELDS = ("internal_info", "info")), and wires it intoNeo4jGraphDB._parse_node,Neo4jCommunityGraphDB._parse_node, andNeo4jCommunityGraphDB._parse_nodes. JSON-object-shaped strings deserialize todict; malformed strings fall back toNoneso pydantic still validates;dict/None/ non-object strings pass through unchanged. The write path and other graph backends (Postgres, PolarDB, Nebular) are intentionally untouched — the boundary contract fix mirrors the analogous Postgres work in #2229 / #2270 without expanding scope.Coverage: 16 new tests in
tests/graph_dbs/test_neo4j_internal_info_roundtrip.py— 7 helper unit tests, 8_parse_node/_parse_nodesregressions for both editions, and one end-to-end guard that feeds the parser output intoTreeNodeTextualMemoryMetadata(**metadata)and asserts pydantic validation succeeds, reproducing and closing the exact failure mode from the issue.python3 -m pytest tests/graph_dbs/ tests/memories/textual/→ 107 passed, 3 skipped (Neo4j integration cases behind an opt-in flag). Ruff check + ruff format both clean.Related Issue (Required): Fixes #2288
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Not run; documentation-only change.
Checklist
@wustzdy please review this PR.
Reviewer Checklist