Skip to content

fix: normalize Postgres embedding strings for reorganizer - #2269

Open
Kwizii wants to merge 2 commits into
MemTensor:mainfrom
Kwizii:fix/postgres-embedding-normalize
Open

fix: normalize Postgres embedding strings for reorganizer#2269
Kwizii wants to merge 2 commits into
MemTensor:mainfrom
Kwizii:fix/postgres-embedding-normalize

Conversation

@Kwizii

@Kwizii Kwizii commented Aug 22, 2026

Copy link
Copy Markdown

Description

PostgresGraphDB._parse_row() and _prepare_node_metadata() assumed metadata["embedding"] is already a list[float]. When include_embedding=True, psycopg2/pgvector often returns the vector column as a JSON string (for example '[-0.047..., ...]').

The reorganizer loads nodes with get_node(..., include_embedding=True) and constructs GraphDBNode(**raw_node). Pydantic then rejects the string embedding, so the reorganizer consumer logs a traceback and skips reorganize work.

This change adds _normalize_embedding_value() to coerce list/tuple/JSON-string embeddings into list[float], and uses it in both metadata preparation and row parsing.

Related Issue (Required): Fixes #2270

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test

Added regression tests in tests/graph_dbs/test_postgres_embedding_parse.py:

  • _normalize_embedding_value() parses JSON string embeddings
  • _prepare_node_metadata() normalizes string embeddings in metadata
  • _parse_row(..., include_embedding=True) normalizes pgvector string column values

Reproduce with:

pytest tests/graph_dbs/test_postgres_embedding_parse.py -v

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

@Memtensor-AI Memtensor-AI added area:database graph_db + vector_db | 图数据库与向量数据库 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 22, 2026
@Memtensor-AI
Memtensor-AI requested a review from wustzdy August 22, 2026 15:07
@Kwizii Kwizii closed this Aug 22, 2026
@Memtensor-AI

Memtensor-AI commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2269
Task: 455b594769bdf0dd
Base: main
Head: fix/postgres-embedding-normalize

🔍 OpenCodeReview found 2 issue(s) in this PR.


1. src/memos/graph_dbs/postgres.py (L462-L469)

When include_embedding=True but len(row) <= 5 (the guard on line 462 fails), row[5] was never assigned to result["metadata"]["embedding"]. However, the block below still runs and its elif branch will silently delete any "embedding" key that was already present in the props JSON blob — data that wasn't fetched from the embedding column at all.

Consider restructuring so the normalization only applies when the embedding was actually fetched:

if include_embedding and len(row) > 5:
    normalized = _normalize_embedding_value(row[5])
    if normalized is not None:
        result["metadata"]["embedding"] = normalized
    else:
        result["metadata"].pop("embedding", None)

This removes the separate assignment on line 463 and the separate block on lines 464–469, avoiding the accidental deletion.

💡 Suggested Change

Before:

        if include_embedding and len(row) > 5:
            result["metadata"]["embedding"] = row[5]
        if include_embedding:
            normalized = _normalize_embedding_value(result["metadata"].get("embedding"))
            if normalized is not None:
                result["metadata"]["embedding"] = normalized
            elif "embedding" in result["metadata"]:
                del result["metadata"]["embedding"]

After:

        if include_embedding and len(row) > 5:
            normalized = _normalize_embedding_value(row[5])
            if normalized is not None:
                result["metadata"]["embedding"] = normalized
            else:
                result["metadata"].pop("embedding", None)

2. src/memos/graph_dbs/postgres.py (L33-L38)

If embedding is a type not handled by any branch (e.g., a numpy.ndarray returned by some pgvector adapters, a memoryview, or a Decimal), all three isinstance checks fail, the try exits without returning, and None is returned at line 51. In _parse_row, this causes the elif branch to silently delete the embedding from the result with no log or warning, making it very hard to diagnose why embeddings disappear for certain driver versions.

Consider adding a fallback that attempts conversion via list() for iterable types, or at minimum logs a warning before returning None:

    # fallback: try generic iterable
    try:
        return [float(x) for x in embedding]
    except (ValueError, TypeError):
        logger.warning(
            "_normalize_embedding_value: unrecognised embedding type %s, dropping value",
            type(embedding).__name__,
        )
        return None

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Command failed: git clone --depth 1 --branch fix/postgres-embedding-normalize git@github.com:Kwizii/MemOS.git /data/test-workspaces/8d1d826a3e671a05/repo
Cloning into '/data/test-workspaces/8d1d826a3e671a05/repo'...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Branch: fix/postgres-embedding-normalize

@Kwizii Kwizii reopened this Aug 22, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Command failed: git clone --depth 1 --branch fix/postgres-embedding-normalize git@github.com:Kwizii/MemOS.git /data/test-workspaces/455b594769bdf0dd/repo
Cloning into '/data/test-workspaces/455b594769bdf0dd/repo'...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Branch: fix/postgres-embedding-normalize

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:database graph_db + vector_db | 图数据库与向量数据库 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Postgres reorganizer fails when pgvector embedding is returned as string

3 participants