Skip to content

Fix #2288: Bug: internal_info / info stored as JSON string by Neo4j sanitizer but never des - #2289

Closed
Memtensor-AI wants to merge 2 commits into
MemTensor:dev-v2.0.32from
Memtensor-AI:bugfix/autodev-2288-20260827081609558
Closed

Memtensor-AI wants to merge 2 commits into
MemTensor:dev-v2.0.32from
Memtensor-AI:bugfix/autodev-2288-20260827081609558

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fixes #2288: Neo4jGraphDB / Neo4jCommunityGraphDB sanitize every dict-typed metadata value into a JSON string on write via _sanitize_neo4j_value, but the read path only reversed the sources transform. internal_info and info therefore came back as str, breaking TextualMemoryItem's dict | None pydantic validation during recall on document-ingest / Dream-enabled deployments.

Adds a small _deserialize_dict_field helper in src/memos/graph_dbs/neo4j.py (symmetric with the write-side sanitizer, module constant _DICT_METADATA_FIELDS = ("internal_info", "info")), and wires it into Neo4jGraphDB._parse_node, Neo4jCommunityGraphDB._parse_node, and Neo4jCommunityGraphDB._parse_nodes. JSON-object-shaped strings deserialize to dict; malformed strings fall back to None so pydantic still validates; dict / None / non-object strings pass through unchanged. The write path and other graph backends (Postgres, PolarDB, Nebular) are intentionally untouched — the boundary contract fix mirrors the analogous Postgres work in #2229 / #2270 without expanding scope.

Coverage: 16 new tests in tests/graph_dbs/test_neo4j_internal_info_roundtrip.py — 7 helper unit tests, 8 _parse_node / _parse_nodes regressions for both editions, and one end-to-end guard that feeds the parser output into TreeNodeTextualMemoryMetadata(**metadata) and asserts pydantic validation succeeds, reproducing and closing the exact failure mode from the issue. python3 -m pytest tests/graph_dbs/ tests/memories/textual/ → 107 passed, 3 skipped (Neo4j integration cases behind an opt-in flag). Ruff check + ruff format both clean.

Related Issue (Required): Fixes #2288

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Not run; documentation-only change.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@wustzdy please review this PR.

Reviewer Checklist

…emTensor#2288)

`_sanitize_neo4j_value` serializes every dict-typed metadata value into
a JSON string on write, but `_parse_node` / `_parse_nodes` only reversed
the `sources` transform. `internal_info` and `info` therefore came back
as `str`, breaking `TextualMemoryItem`'s `dict | None` validation during
recall on document-ingest / Dream-enabled deployments.

Adds a small `_deserialize_dict_field` helper (symmetric with the
write-side sanitizer) and wires it into both `Neo4jGraphDB._parse_node`
and `Neo4jCommunityGraphDB._parse_node` / `_parse_nodes` for the two
known dict-typed fields. JSON-object-shaped strings deserialize to
dicts, malformed strings fall back to None so pydantic still accepts
them, all other values pass through unchanged. Write path and other
graph backends untouched.

Adds 16 regression tests including an end-to-end guard that feeds the
parser output into `TreeNodeTextualMemoryMetadata` to close the loop
with the failure point.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Memtensor-AI Memtensor-AI added ai:generated Generated or modified by AI | 由 AI 生成或修改 area:database graph_db + vector_db | 图数据库与向量数据库 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 27, 2026
@Memtensor-AI
Memtensor-AI requested a review from wustzdy August 27, 2026 08:59
@Memtensor-AI

Memtensor-AI commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2289
Task: 9d183872ea1d90db
Base: dev-v2.0.32
Head: bugfix/autodev-2288-20260827081609558
Head SHA: edcdb1d1e146546423e44ae38b262081aa5fc2e1

🔍 OpenCodeReview found 1 issue(s) in this PR.

⚠️ 1 warning(s) occurred during review.


1. tests/graph_dbs/test_neo4j_internal_info_roundtrip.py (L39-L43)

Finding #1 is NOT fixed. The _helper() static method still re-imports _deserialize_dict_field on every individual test invocation. A single import at module or class level (e.g., from memos.graph_dbs.neo4j import _deserialize_dict_field at the top of the file or as a class attribute) would be cleaner and more idiomatic.

Suggested fix:

# At module level alongside other imports
from memos.graph_dbs.neo4j import _deserialize_dict_field

class TestDeserializeDictField:
    # Remove _helper() and call _deserialize_dict_field directly in each test
    def test_none_passthrough(self):
        assert _deserialize_dict_field(None) is None

🧹 Filtered 1 low-confidence OCR finding(s) before posting/fix-loop (existing_code_mismatch: 1).

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🔧 Open Code Review requested Agent fix

Open Code Review found 2 issue(s). I have resumed the development Agent to fix them.

  • Task: 9d183872ea1d90db
  • Fix attempt: 1/2
  • Finding delta: 0 repeated / 2 new / 0 likely resolved

The Agent will push a new commit to this PR branch. OCR will recheck after the commit is pushed.

Address two OCR findings on MemTensor#2289:

1. src/memos/graph_dbs/neo4j.py: `_deserialize_dict_field` silently
   dropped malformed dict-shaped strings to None, making the event
   indistinguishable from a legitimately-None field. Emit a
   `logger.warning` before returning None so operators can trace the
   corruption. Uses the module-level logger already declared at line 13.

2. tests/graph_dbs/test_neo4j_internal_info_roundtrip.py:
   `TestDeserializeDictField._helper` is a stable module-level lookup
   with no `self` use — promote it to `@staticmethod` for idiomatic
   pytest style.

Verified: pytest tests/graph_dbs/ (47 passed, 3 pre-existing skips) +
ruff check/format clean on both files.
@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 27, 2026
@syzsunshine219
syzsunshine219 deleted the branch MemTensor:dev-v2.0.32 August 28, 2026 07:00
endxxxx added a commit that referenced this pull request Oct 24, 2026
## Description

Neo4j-backed recall can return an empty result after `TextualMemoryItem`
rejects a JSON-encoded `metadata.internal_info`. The native ingest
reader can also skip the same nodes when constructing
`TreeNodeTextualMemoryMetadata`.

The chunking path creates `internal_info` as a dictionary. Neo4j
serializes dictionary properties as JSON, but its read conversion does
not restore this field before Pydantic validation. A short conversation
with `internal_info=None` does not exercise the failure.

Add a before-validator on `TextualMemoryMetadata.internal_info`, shared
by the search and reader metadata models. Decode string values once and
require a JSON object. Existing dictionaries and `None` keep their
behavior; malformed JSON, encoded arrays/scalars/null, and
double-encoded strings remain validation errors. No storage migration,
write-path change, request/response route change, new dependency, LLM
request, or retrieval-policy change is required.

Related issue: #2288. Earlier PR #2289 was closed without merging; the
failure remains reproducible on current `main` (`a7367d0`). This change
deliberately retains validation errors for malformed values instead of
silently replacing them with `None`.

## Type of change

- [x] Bug fix

## How Has This Been Tested?

- Before the fix: new regression file has **5 failed, 15 passed**,
reproducing the JSON-string validation error.
- After the fix: new regression file has **20 passed**.
- In an isolated Python 3.12 / Pydantic 2.12.4 backend container, with
networking disabled, no credentials and no business volumes:

  ```sh
PYTHONPATH=src python -B -m pytest -p no:cacheprovider -o
filterwarnings= -o addopts= -q tests/memories/textual/
  ```

**80 passed**. Coverage includes all four metadata subclasses, search
`from_dict`, direct reader metadata construction, unchanged
source/session/content fields, repeated storage JSON round trips,
existing dictionary/None values, and rejection of malformed or
non-object JSON.

The image does not include pytest-asyncio, producing an existing unknown
`asyncio_mode` configuration warning; one existing naive-memory test
also reports a Pydantic serialization warning.

- `make format`: **passed** (`All checks passed!`; `631 files left
unchanged`), using the repository-pinned Ruff 0.11.8.
- `pre-commit run --files src/memos/memories/textual/item.py
tests/memories/textual/test_item.py`: **passed** (all applicable hooks).

## Checklist

- [x] Self-reviewed the diff; only the metadata validator and its
regression tests change.
- [x] Added an explanatory comment at the storage/model boundary.
- [x] Added tests that fail before the fix and pass afterward.
- [x] Linked the existing issue. No public API or documentation change
is required.

Maintainer review and upstream CI are pending; the full repository suite
was not run locally.
endxxxx added a commit that referenced this pull request Nov 19, 2026
* fix(graph_dbs): deserialize internal_info / info on Neo4j read path (#2288)

`_sanitize_neo4j_value` serializes every dict-typed metadata value into
a JSON string on write, but `_parse_node` / `_parse_nodes` only reversed
the `sources` transform. `internal_info` and `info` therefore came back
as `str`, breaking `TextualMemoryItem`'s `dict | None` validation during
recall on document-ingest / Dream-enabled deployments.

Adds a small `_deserialize_dict_field` helper (symmetric with the
write-side sanitizer) and wires it into both `Neo4jGraphDB._parse_node`
and `Neo4jCommunityGraphDB._parse_node` / `_parse_nodes` for the two
known dict-typed fields. JSON-object-shaped strings deserialize to
dicts, malformed strings fall back to None so pydantic still accepts
them, all other values pass through unchanged. Write path and other
graph backends untouched.

Adds 16 regression tests including an end-to-end guard that feeds the
parser output into `TreeNodeTextualMemoryMetadata` to close the loop
with the failure point.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(graph_dbs): log dropped internal_info deserialize + tidy test helper

Address two OCR findings on #2289:

1. src/memos/graph_dbs/neo4j.py: `_deserialize_dict_field` silently
   dropped malformed dict-shaped strings to None, making the event
   indistinguishable from a legitimately-None field. Emit a
   `logger.warning` before returning None so operators can trace the
   corruption. Uses the module-level logger already declared at line 13.

2. tests/graph_dbs/test_neo4j_internal_info_roundtrip.py:
   `TestDeserializeDictField._helper` is a stable module-level lookup
   with no `self` use — promote it to `@staticmethod` for idiomatic
   pytest style.

Verified: pytest tests/graph_dbs/ (47 passed, 3 pre-existing skips) +
ruff check/format clean on both files.

---------

Co-authored-by: MemOS AutoDev <autodev@memos.local>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wenqiang Wei <46308778+endxxxx@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 area:database graph_db + vector_db | 图数据库与向量数据库 status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants