Skip to content

fix: support pagination and filters in Postgres export_graph - #2268

Open
Kwizii wants to merge 1 commit into
MemTensor:mainfrom
Kwizii:fix/postgres-export-graph-pagination
Open

fix: support pagination and filters in Postgres export_graph#2268
Kwizii wants to merge 1 commit into
MemTensor:mainfrom
Kwizii:fix/postgres-export-graph-pagination

Conversation

@Kwizii

@Kwizii Kwizii commented Aug 22, 2026

Copy link
Copy Markdown

Description

PostgresGraphDB.export_graph accepted page, page_size, memory_type, status and filter only through **kwargs and silently ignored them. Its SQL had no LIMIT/OFFSET and no filtering beyond user_name, and total_nodes was set to the length of the returned page.

As a result, when the graph backend is postgres, every consumer of POST /get_memory (which passes pagination and tag filters down to text_mem.get_allexport_graph) received the full memory list on every page, with an incorrect total — pagination and tag filtering were both broken.

This aligns the Postgres implementation with the existing Neo4j implementation (Neo4jGraphDB.export_graph):

  • translate page/page_size into LIMIT/OFFSET (invalid values normalized like neo4j: page < 1 → 1, page_size < 1 → 10)
  • filter memory_type via properties->>'memory_type' = ANY(...)
  • default status behavior excludes deleted; an explicit status list uses ANY(...)
  • reuse the existing _build_filter_where_clause so the and/or/contains filter DSL works
  • count total_nodes with a separate COUNT(*) query using the same filters, before pagination

Edges: resolved via subqueries over the full filtered node set (not page-local ids), so the edge set is complete and consistent across pages. Edges are deliberately not paginated; every response includes all edges connected to any node matching the filters. This is what /get_memory consumers rely on (nodes + total_nodes); documented in the docstring.

Related Issue (Required): Fixes #2271

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test

New unit tests in tests/graph_dbs/test_postgres_export_graph.py mock the connection/cursor and verify:

  • page=2&page_size=6 produces LIMIT 6 OFFSET 6 and total_nodes comes from the COUNT query (9), not from the returned rows (3)
  • no pagination → no LIMIT/OFFSET
  • memory_type=[...] and default deleted-status exclusion appear in the WHERE clause
  • tag filter {"tags": {"contains": ...}} reaches SQL as properties->'tags' @> %s::jsonb
  • invalid pagination inputs are normalized
  • edges resolve via subqueries over the full filtered node set, with filter params applied to both endpoint subqueries

Reproduce with:

pytest tests/graph_dbs/test_postgres_export_graph.py -v

All 37 tests in tests/graph_dbs pass (3 skipped, unrelated).

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

@Memtensor-AI Memtensor-AI added area:database graph_db + vector_db | 图数据库与向量数据库 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 22, 2026
@Memtensor-AI
Memtensor-AI requested a review from wustzdy August 22, 2026 09:18
@Memtensor-AI

Memtensor-AI commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2268
Task: 626590e71c0929b0
Base: main
Head: fix/postgres-export-graph-pagination

🔍 OpenCodeReview found 2 issue(s) in this PR.


1. tests/graph_dbs/test_postgres_export_graph.py (L32-L38)

_current is never initialized in __init__, and self.results.pop(0) is called unconditionally on every execute call without guard. Two concrete failure modes:

  1. If fetchone() or fetchall() is called before the first execute() (e.g., the implementation changes query order or adds a pre-flight query), an AttributeError: '_FakeCursor' object has no attribute '_current' is raised instead of a clear test failure.
  2. If the implementation issues more queries than the number of pre-loaded results, pop(0) raises IndexError: pop from empty list — again with no descriptive failure message pointing to the broken assertion.

Suggestion: initialise _current to None and guard the pop:

def __init__(self, results: list[Any]):
    self.results = list(results)
    self.executed: list[tuple[str, list]] = []
    self._current: Any = None

def execute(self, query: str, params=None):
    self.executed.append((query, list(params or [])))
    if not self.results:
        raise RuntimeError(
            f"_FakeCursor ran out of pre-loaded results on query:\n{query}"
        )
    self._current = self.results.pop(0)

This converts silent crashes into actionable error messages when the implementation's query count diverges from the test fixture.


2. src/memos/graph_dbs/postgres.py (L1139-L1145)

When only one of page or page_size is supplied, use_pagination silently becomes False and the method returns all rows without pagination or any warning. A caller who passes page=2 while forgetting page_size receives the full dataset rather than an error or an empty/clamped result — a silent correctness failure that is hard to debug.

Consider raising a ValueError when exactly one of the two arguments is provided, or defaulting the missing argument to a sensible value.

💡 Suggested Change

Before:

        use_pagination = page is not None and page_size is not None
        if use_pagination:
            if page < 1:
                page = 1
            if page_size < 1:
                page_size = 10
            offset = (page - 1) * page_size

After:

        if (page is None) != (page_size is None):
            raise ValueError(
                "Both 'page' and 'page_size' must be provided together, or both omitted."
            )
        use_pagination = page is not None and page_size is not None
        if use_pagination:
            if page < 1:
                page = 1
            if page_size < 1:
                page_size = 10
            offset = (page - 1) * page_size

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Command failed: git clone --depth 1 --branch fix/postgres-export-graph-pagination git@github.com:Kwizii/MemOS.git /data/test-workspaces/452e50135ab6d4b4/repo
Cloning into '/data/test-workspaces/452e50135ab6d4b4/repo'...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Branch: fix/postgres-export-graph-pagination

@Kwizii
Kwizii force-pushed the fix/postgres-export-graph-pagination branch from bca1f91 to 75c9d64 Compare August 22, 2026 09:59
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Command failed: git clone --depth 1 --branch fix/postgres-export-graph-pagination git@github.com:Kwizii/MemOS.git /data/test-workspaces/adf1ca54cdb22b89/repo
Cloning into '/data/test-workspaces/adf1ca54cdb22b89/repo'...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Branch: fix/postgres-export-graph-pagination

@Kwizii
Kwizii force-pushed the fix/postgres-export-graph-pagination branch from 75c9d64 to e49f3e9 Compare August 22, 2026 10:10
export_graph silently ignored page/page_size/memory_type/status/filter
(they were swallowed by **kwargs), so callers always received the full
node list with total_nodes equal to the page length. This broke
pagination and tag filtering for every consumer of POST /get_memory
when the graph backend is postgres.

- translate page/page_size into LIMIT/OFFSET (normalized like neo4j)
- filter memory_type via properties->>'memory_type' = ANY(...)
- default status filter excludes 'deleted'; explicit status list uses ANY
- reuse _build_filter_where_clause so and/or/contains filter DSL works
- count total_nodes with a separate COUNT query before pagination
- resolve edges via subqueries over the full filtered node set (not
  page-local ids) so the edge set is complete and consistent across pages
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Command failed: git clone --depth 1 --branch fix/postgres-export-graph-pagination git@github.com:Kwizii/MemOS.git /data/test-workspaces/626590e71c0929b0/repo
Cloning into '/data/test-workspaces/626590e71c0929b0/repo'...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Branch: fix/postgres-export-graph-pagination

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:database graph_db + vector_db | 图数据库与向量数据库 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Postgres export_graph ignores pagination and filters

3 participants