Skip to content

feat(graph-db): add openGauss backend and memory integration - #2646

Open
syzsunshine219 wants to merge 23 commits into
mainfrom
codex/maintainer-edit-pr-2610-20261119
Open

syzsunshine219 wants to merge 23 commits into
mainfrom
codex/maintainer-edit-pr-2610-20261119

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

The source branch for the change proposed in #2610 now lives in MemTensor/MemOS, so repository writers can contribute fixes through the normal upstream branch permissions.

The code and commit authors are unchanged. This PR retains target branch main and exact source head ad10805219524785580074010aac852d8249b76e from #2610.

Previous discussion and review remain in #2610. Review and checks must be re-evaluated for this replacement PR; no previous approval or new test result is asserted by this migration.

Previous PR description and implementation details

Restores the proposal from #2564 as an open PR against main; its former base branch, dev-v2.0.36, is no longer available. This proposal requires maintainer review of the feature or design direction before merging.

Source head: ad10805219524785580074010aac852d8249b76e. No new source-code changes are introduced by restoring this proposal. Earlier discussion and evidence remain in #2564; validation claims in the original description refer to the earlier snapshot.

Outstanding review scope

当前main的配置工厂与实现工厂都没有注册openGauss,属于新增存储后端而非已确认遗漏的既有功能修复。需要确认维护负责人、部署依赖和后端支持计划,再决定是否恢复Draft。


Original proposal

Description

This PR adds openGauss backend support to MemOS as part of internal feature development, targeting dev-v2.0.36.

It integrates AGE graph storage, native vector retrieval, and BM25 full-text search with the existing memory components, API, and Scheduler.

Related Issue / Task: N/A — internally planned feature development with no associated issue or task number.

Main changes

  • Add openGauss configuration validation, factory registration, environment settings, and connection pooling.
  • Implement scoped node and relationship operations, graph traversal, metadata filtering, vector search, BM25 search, and SQL LIKE lookup.
  • Support memory counts, capacity management, snapshot export, atomic import, and scoped clearing.
  • Add optional Dashboard range selection through all_cubes and exclude_cube_id, with documentation and OpenAPI schema updates.
  • Improve resource ownership and shutdown ordering across API, Cube, Tree memory, and Scheduler components.
  • Preserve Cube identity during search and distinct identities for working and long-term memory pairs.
  • Propagate required storage failures through search and Feedback instead of treating failures as successful empty results.

Dependencies

  • openGauss Enterprise with AGE and native DataVec/BM25 capabilities.
  • psycopg2-binary >=2.9.13,<3.0.0, included in the optional tree-mem and all extras.

Enable the backend with GRAPH_DB_BACKEND=opengauss and configure the corresponding OPENGAUSS_* connection settings. The database and required extensions must be provisioned before use.

Behavior changes and limitations

  • Empty Dashboard categories return empty lists instead of synthetic empty Cube buckets.
  • Non-openGauss backends reject explicit non-null values for the new Dashboard range fields.
  • Required storage errors propagate to callers. Multi-step Feedback updates are not a single transaction.
  • Database-level semantic deduplication, conflict detection, and node merging remain unsupported.
  • Full service E2E, Java client integration, real-model validation, and production-scale performance testing remain follow-up work.

Type of change

  • New feature
  • Bug fix
  • Refactor
  • Documentation update

How Has This Been Tested?

Latest recorded local validation:

  • Full repository suite with real openGauss integration enabled: 2771 passed, 9 skipped, 45 warnings.
  • Unit and integration tests cover scoped storage, retrieval, rollback, backup/restore, Dashboard responses, memory identities, Feedback failures, and concurrent shutdown.
  • A separate local HTTP smoke test passed with real openGauss and controlled model assembly, covering Dashboard results, graceful shutdown, and database session cleanup.
  • Ruff checks and formatting passed.

Integration tests use openGauss 7.0.0 with AGE 1.0.0. Test fixtures provision disposable databases and roles; application operations run with an ordinary database role.

To reproduce, configure MEMOS_OPENGAUSS_TEST_CONFIG with a JSON OpenGaussGraphDBConfig for a test administrator, then run:

poetry install --extras all --with dev --with test
make format
make test

OpenAPI was exported using the repository's component fixtures. Standard make openapi remains blocked in the local environment by missing InternetRetriever model configuration.

These results describe local validation. Hosted CI and full service E2E are not claimed as complete.

  • Unit Test
  • Test Script Or Test Steps
  • Pipeline Automated API Test

Checklist

  • I have performed a self-review of the final PR diff
  • I have commented code in hard-to-understand areas
  • I have added tests covering the new functionality and fixes
  • I have included backend setup and behavior documentation in this PR
  • I have assigned or mentioned the reviewer

Issue/task linkage: N/A — no associated issue or internal task number.

Reviewer Checklist

  • Confirmed the implementation scope and behavior changes
  • Made sure required checks passed
  • Reviewed the provided tests and validation results

Add a strict OpenGaussGraphDBConfig for connection settings, logical
tenant scopes, and bounded connection-pool configuration.

Validate required credentials, port and timeout boundaries, positive
embedding dimensions, and ordered pool limits. Require a tenant scope
for shared databases and preserve explicitly supplied scope values.

Add 105 parameter cases covering valid and invalid configuration,
JSON/YAML round trips, hidden password repr, driver-independent imports,
and existing backend configuration selection.

Validation:
- make format
- poetry run pytest tests/configs tests/graph_dbs -q
  (237 passed; 3 existing opt-in Neo4j integration cases not enabled)
- New configuration class: 26 statements and 6 branch exits covered,
  with zero missing statements or branch exits
- pre-commit checks for both changed files passed
Add an internal node store with scoped business identities, stable native
graphids, atomic batch upserts, ordered reads and row-locked partial updates.
Reject incompatible layouts without deleting data.

Prepare ordinary-role AGE sessions through a bounded psycopg2 pool. Preserve
driver causes, roll back failures and close owned connections deterministically.
Check server session cleanup within a fixed deadline after client close.

Add GraphDBError and declare psycopg2-binary in tree-mem/all extras.
Keep the incomplete backend out of the public factories.

Validate 107 unit cases and 29 live openGauss integration cases.
Config/graph regression: 373 passed, 3 existing skips.
Full suite: 1207 passed, 9 existing skips, with live openGauss enabled
and Hugging Face restricted to the local cache.
Pool/store statement and branch coverage: 100%, with no new exclusions.
Add native directed typed relationships, scoped edge/neighbor queries,
and atomic deletion of a node and all its incident edges.
Serialize graph mutations to preserve edge uniqueness and prevent deletion races.

Delete incident edges with AGE and node label rows with SQL in one transaction,
preserving trigger and permission failure rollback behavior. Bind business
values and validate relationship identifiers before borrowing a connection.

Add 73 unit cases and 17 live integration cases for directions, scope,
self/parallel edges, concurrency, special values and rollback.
Cumulative config/graph regression: 463 passed, 3 existing skips.
Full suite: 1297 passed, 9 existing skips, with live openGauss enabled
and Hugging Face restricted to the local cache.
Pool/store statement and branch coverage: 100%, with no new exclusions.
Add directed shortest paths with deterministic tie breaking and a strict
five-hop bound. Return complete two-way neighborhood nodes and induced
edges, preserving native direction, parallel edge types and self-loops.

Rank active non-reasoning/non-working memories by unique tag overlap,
with scoped exclusions, deterministic ties and optional vector projection.
Reject malformed tag data and invalid inputs instead of returning false success.

Add 86 unit cases and 21 live integration cases for traversal, projections,
scope, special strings, tag filtering and permission failure recovery.
Cumulative config/graph regression: 570 passed, 3 existing skips.
Full suite: 1404 passed, 9 existing skips, with live openGauss enabled
and Hugging Face restricted to the local cache.
Pool/store statement and branch coverage: 100%, with no new exclusions.
Consolidate the connection pool, AGE node/graph operations and filters in
OpenGaussGraphDB, following the existing PostgreSQL and PolarDB layout.
Create the pool in __init__ and keep filter parameters local to each call.

Include scoped metadata/type queries, atomic conditional deletion and
soft-delete/restore visibility tests. Preserve AGE fractional-number
round trips and the existing transaction, isolation and error contracts.

Keep unfinished BaseGraphDB methods abstract and factory registration
pending. Replace the private connection and storage modules with
opengauss.py; test-only concrete support raises for every unfinished
capability.

Validation: 398 storage/graph unit tests and 125 real openGauss integration
tests; configs/graph_dbs 760 passed, 3 existing skips; full suite 1594 passed,
9 existing skips. opengauss.py has 100% statement and branch coverage.
Formatting, applicable hooks and lock checks passed; temporary resources 0.
Add DataVec cosine retrieval with normalized scores, scoped filters,
thresholds, projection and compatible Cube completion. Validate HNSW
indexes through 2000 dimensions and use exact search up to 16000 dimensions.
Replace underfilled vector candidates with one identically filtered query.

Add native BM25 OR retrieval with raw scores and TAAT candidate scoring,
including requests above 200 results. Validate the owned text index and
recover its verified failed-write state with one REINDEX and one read retry.
Keep unrelated errors explicit and never replay writes.

Validate real GraphMemoryRetriever completion, filtering, rollback,
concurrent recovery, index permissions, cancellation and timeouts. Keep
factory registration pending the remaining maintenance and consumer APIs.

Validation: make format; applicable hooks; poetry check --lock;
configs/graph_dbs/retriever: 966 passed, 3 existing skips;
make test: 1794 passed, 9 existing skips, including all 170 openGauss IT;
backend statement
and branch coverage: 100% (869 statements, 358 branch exits).

This message describes the complete current workspace delta from 86ef67b.
Add scoped grouping, memory counts, existence checks, and oldest-memory
eviction to OpenGaussGraphDB. Count all statuses and keep numeric AGE groups
consistent with transaction-local sort aggregation.

Lock a fixed candidate set, validate every timestamp, then remove incident
edges and nodes in batches of at most 500 within one transaction. Preserve
concurrent inserts for a later cleanup and propagate transaction failures
without retrying writes.

Cover ordinary-role database behavior, public vector/BM25 visibility,
multibatch rollback, concurrent writers, cancellation, connection replacement,
commit/return failures, and real MemoryManager/Reorganizer boundaries.

Centralize transactional cursors and optional UTC/advisory setup. Compose
named SQL fragments with collision checks and reusable bindings, and separate
grouping validation, query construction, and result decoding.

Validate with 741 backend unit tests and 226 real openGauss integration tests.
The full suite passes 2040 tests with 9 existing skips; the complete openGauss
module has 100% statement and branch coverage.

Factory registration and the eight remaining abstract methods stay pending.
Select strictly activated memories of the requested type and Cube without
same-Cube PARENT adjacency in one SQL snapshot. Preserve deterministic order,
optional embeddings, scoped graph identities and existing transaction errors.

Request embeddings from Reorganizer and verify real partitioning, summaries
and PARENT writes with controlled model services. Keep the remaining seven
abstract methods pending and factory wiring unchanged.

Validation: make format; enabled openGauss make test: 2115 passed, 9 existing
skips. Full opengauss.py statements/branches and modified optimize_structure
function reach 100%. Temporary databases, roles and test/application sessions
are zero after the run.
Add explicit cross-Cube reads with preserved node and edge ownership, global
pagination, filtering and stable ordering. Keep ordinary exports scoped to
the configured Cube. Counts, nodes and induced edges share one read-only
REPEATABLE READ transaction; later borrowers return to read-write defaults.

Validation: 40 Red to Green tests; 1407 passed / 3 existing skips in affected
configuration, graph and textual-memory regression with real openGauss.
Full opengauss.py: 1060 statements and 446 branch exits covered, no omissions.
make format and git diff --check pass.
Validate graph input before borrowing, preserve existing node identities and
merge edges in one transaction. Reject missing endpoints without partial
imports. Clear a fixed locked Cube set in 500-node blocks and retain shared
schema and other Cubes; drop_database clears only its configured scope.

Validation: 43 Red to Green tests plus commit uncertainty, cancellation,
physical scope and concurrency cases. Related real-database regression:
1456 passed, 3 existing skips. Full opengauss.py: 1123 statements and 482
branch exits covered. make format and git diff --check pass.
Resolve unique Cube ownership and reject ambiguous IDs. Add atomic scoped
soft deletion, recovery and batch hard deletion while preserving other node
properties, vectors and relations.

Validation: 38 initial Red/Green cases, 3 concurrent transaction cases;
1497 passed and 3 existing skips in affected regression. The full backend
has 1172/1172 statements and 496/496 branches covered; make format passes.
Implement scoped one-hop outgoing context queries. Explicitly reject semantic
deduplication, conflict detection and merging with named capability errors;
keep application MMR independent. All tests now use the production class.

Validation: 6 initial Red cases; 8 targeted Green cases including structure.
Full make test with real openGauss: 2246 passed, 9 unchanged existing skips.
Backend coverage: 1183/1183 statements and 496/496 branches; changed
Reorganizer function: 48/48 statements and 24/24 branches. make format and
git diff --check pass. Temporary databases, roles and sessions: zero.
Exercise batch hard deletion under permission denial, last-block timeout
and actual driver cancellation. Verify full rollback, preserved deletion
records, replacement connections and search visibility after recovery.

Validation: 30 targeted real-database tests pass; final make test reports
2249 passed with the same 9 existing skips. Full backend statement and
branch coverage remains 100%; make format and git diff --check pass.
Restore read-only component mappings and leave close state with resource owners. Quiesce recovery before draining current and retired executors; reject self-close before entering terminal state. Preserve concurrent close results, retryable cleanup, explicit Cube initialization, and diagnostic logs.
Preserve primary IDs and derive stable Working IDs with working_binding. Submit each pair in one batch transaction on openGauss, propagate write failures, and preserve input result order across both add paths. Reuse binding cleanup rules and protect historical self-bindings. Cover rollback, retries, concurrent writes, capacity cleanup, and default Feedback with real database tests.
@syzsunshine219 syzsunshine219 added area:api 云服务 / FastAPI / OpenAPI / MCP area:core MOS 编排层 / 框架底座 / 跨模块问题 area:database graph_db + vector_db | 图数据库与向量数据库 area:docs 文档、示例 area:memcube GeneralMemCube / cube 生命周期 / cube 配置 area:memory 记忆存储、检索、更新、召回逻辑 area:scheduler 调度模块 labels Nov 19, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Nov 19, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2646
Task: ef5ac1e8da3b188d
Base: main
Head: codex/maintainer-edit-pr-2610-20261119
Head SHA: ad10805219524785580074010aac852d8249b76e

🔍 OpenCodeReview found 11 issue(s) in this PR.

⚠️ Review warnings (12)

  • completed primary review; remaining group budget is insufficient for another measured pass and finalization
  • completed primary review; remaining group budget is insufficient for another measured pass and finalization
  • completed primary review; remaining group budget is insufficient for another measured pass and finalization
  • completed primary review; remaining group budget is insufficient for another measured pass and finalization
  • semantic groups split by file count and complete diff byte size; all selected files remain in coverage
  • bounded analysis finalized with explicit task_done after reviewing all group files
  • bounded analysis finalized with explicit task_done after reviewing all group files
  • bounded analysis finalized with explicit task_done after reviewing all group files
  • bounded analysis finalized with explicit task_done after reviewing all group files
  • bounded analysis finalized with explicit task_done after reviewing all group files
  • … 2 additional warning(s) omitted.

1. src/memos/api/lifecycle.py (L38-L39)

shutdown_components now raises MemOSError on any close failure, but the two production call sites in the ASGI lifespan handlers (server_api.py:36 and server_api_ext.py:68) invoke it without a try/except. Previously failures were only logged; now they propagate out of the lifespan shutdown block, so Starlette sends lifespan.shutdown.failed and uvicorn logs the traceback / reports a failed application shutdown instead of a clean drain. In-process callers (e.g. component_init.init_server) already wrap this call, so either wrap the lifespan call sites the same way or provide a non-raising variant here and let callers decide whether to surface the error.


2. src/memos/graph_dbs/opengauss.py (L1266-L1270)

get_neighbors_by_tag loads every row of the scope (SELECT ... WHERE scope = ... with no LIMIT/WHERE on tags) and then JSON-decodes each properties blob in Python to compute the tag overlap. This is an O(N) scan plus N json.loads calls per call, so it degrades linearly with cube size and also fails the whole query if a single stored payload is malformed (_decode_node/_node_tags raise GraphDBError). If the AGE build exposes tag containment (@> on the tags array, as used by _build_filter_condition(..., "contains")), pushing the overlap/status/reasoning exclusions into SQL and limiting the decoded rows would keep this bounded; otherwise consider at least a comment documenting the intentional full scan.


3. src/memos/graph_dbs/opengauss.py (L791-L795)

The BM25 recovery path is keyed on an exact, full-text match of a third-party engine message. Any wording, punctuation, locale or build change in openGauss (or a different index name/parameterisation) silently disables the REINDEX recovery and the user gets the raw failure instead. Consider matching on a stable part of the condition, e.g. cause.pgcode == "42809" plus a substring/regex such as re.search(r"reindex or recreate bm25 index", message_primary or "", re.I), and log when a 42809 arrives with an unrecognised message so the mismatch is observable.


4. src/memos/graph_dbs/opengauss.py (L1418-L1423)

delete_node_by_prams runs one _cypher(...) (which does PREPARE/EXECUTE/DEALLOCATE) plus one DELETE per matched row, so deleting e.g. by file_ids costs 2 round trips per node while holding the graph advisory lock. The file already has the batched _delete_locked_nodes(cursor, scope, ids) helper (500-id batches) used by clear/remove_oldest_memory; grouping the selected rows by scope and reusing that helper would avoid the per-row overhead and keep lock hold times short.


5. src/memos/graph_dbs/opengauss.py (L1706)

_text_field_expr only guards that the stored value is a string (left(agtype_out(...),1) = '"'), then it is cast to timestamptz unconditionally here and in the temporal branch of _build_filter_condition. A row whose created_at/updated_at/delete_time is a string but not a valid timestamp makes the whole statement fail (GraphDBError), and such values are reachable because import_graph accepts arbitrary node metadata. Note the numeric comparison branch already guards with agtype_out(...) ~ '^[-.0-9]'; a similar validity predicate (or agtype_out(...) ~ '^"\d{4}-\d\d-\d\d') before the cast would make export/trim degrade per-row instead of aborting every export until the bad row is fixed.


6. src/memos/mem_scheduler/task_schedule_modules/handlers/feedback_handler.py (L35-L36)

Re-raising here aborts the rest of the per-message loop, and the dispatcher acks every message of the batch in its finally block (dispatcher._create_task_wrapper), so the unprocessed messages are dropped rather than retried. Previously a single failing feedback item only skipped itself. Consider processing the remaining messages (e.g., collect the first error, continue the loop, then re-raise) if batch-level failure signalling is required while preserving per-item progress.


7. src/memos/mem_scheduler/monitors/dispatcher_monitor.py (L263-L265)

Recovery bookkeeping escapes the critical section: the closing/paused check runs under _pool_lock (lines 253-260), but the retired-executor mutation below happens after the lock is released, while the second retirement path (line 278) does it under the lock. SchedulerDispatcher._close_executors iterates [self.dispatcher_executor, *self._retired_executors], so an executor appended after that snapshot is never shutdown(wait=True)-ed, i.e. its in-flight handlers are not drained before components/graph are closed — the invariant this change is introducing. Retiring the old executor in the same guarded block as the _recovery_paused/closing check keeps the check and the retirement atomic (and avoids depending on the dispatcher's private _retired_executors/_mark_worker from outside).

💡 Suggested Change

Before:

            old_executor = pool_info["executor"]
            if old_executor not in self.dispatcher._retired_executors:
                self.dispatcher._retired_executors.append(old_executor)

After:

            with self._pool_lock:
                if (
                    self._close_state.closing.is_set()
                    or self._recovery_paused
                    or self._restart_in_progress
                ):
                    return
                self._restart_in_progress = True
                old_executor = pool_info["executor"]
                if old_executor not in self.dispatcher._retired_executors:
                    self.dispatcher._retired_executors.append(old_executor)

8. src/memos/mem_scheduler/base_scheduler.py (L353-L363)

self._running = False / self._stop_event.set() are issued twice verbatim, and the with self._initialization_lock: pass block is described as "Release it before joining workers" although the lock is not held here — the statement actually acquires it as a barrier to wait for an in-flight initialize_modules(). The duplication plus the inaccurate comment make the intended ordering impossible to follow; signal once up front, then keep an explicitly-commented barrier before touching the monitor.

💡 Suggested Change

Before:

self._running = False
        self._stop_event.set()
        if self.dispatcher_monitor is not None:
            self.dispatcher_monitor.begin_close()
        # Only initialization/start owns this lock. Release it before joining
        # workers, which may need the same lock to observe the closing state.
        with self._initialization_lock:
            pass
        self._running = False
        self._stop_event.set()
        # An in-flight initialize_modules may have just created the monitor.

After:

        self._running = False
        self._stop_event.set()
        if self.dispatcher_monitor is not None:
            self.dispatcher_monitor.begin_close()
        # Barrier: wait for an in-flight initialize_modules()/start() to finish
        # (it may have just created the monitor) before joining workers.
        with self._initialization_lock:
            pass
        # An in-flight initialize_modules may have just created the monitor.

9. src/memos/multi_mem_cube/single_cube.py (L220-L221)

The new except (GraphDBError, ValueError): raise sits on a try that wraps the whole _fast_search/_fine_search/_mix_search pipeline, not just the graph call. That pipeline constructs pydantic models (e.g. TextualMemoryItem(**item_neighbor) in _fast_search), and pydantic.ValidationError subclasses ValueError, so a single malformed graph node now escapes and fails the entire search (propagating through CompositeCubeView.search_memories to the API) instead of degrading to an empty result. If only the new "unsupported Cube range" error needs to surface, prefer raising a dedicated exception (e.g. a ConfigurationError/new error type) from TreeTextMemory.get and re-raising only that plus GraphDBError here.


10. src/memos/mem_feedback/feedback.py (L1206-L1211)

Same concern as in the search path: this try covers LLM judgement/embedding, make_mem_item, and _vec_query (which does TextualMemoryItem(**item)), and pydantic.ValidationError is a subclass of ValueError. A single malformed retrieved memory item therefore now aborts the whole process_feedback_core run instead of being logged and skipped (previously returned {"record": {"add": [], "update": []}}). Consider narrowing the propagated type to GraphDBError (plus an explicit non-pydantic validation error) rather than all ValueErrors.


11. src/memos/mem_cube/general.py (L58-L60)

close() releases the underlying backends but leaves self._text_mem / _act_mem / _para_mem / _pref_mem pointing at the now-closed objects, and CloseState.ensure_open() is never consulted by GeneralMemCube (load/dump/property accessors keep working on closed backends). Combined with the accessor warnings only firing for None, a cube closed elsewhere (e.g. on init/load rollback) will still hand out closed memory objects and fail later with confusing driver errors. Clear the references (or gate load/dump on self._close_state.ensure_open()) when closing.

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:api 云服务 / FastAPI / OpenAPI / MCP area:core MOS 编排层 / 框架底座 / 跨模块问题 area:database graph_db + vector_db | 图数据库与向量数据库 area:docs 文档、示例 area:memcube GeneralMemCube / cube 生命周期 / cube 配置 area:memory 记忆存储、检索、更新、召回逻辑 area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants