Skip to content

feat(graph-db): add openGauss backend and memory integration - #2610

Open
syzsunshine219 wants to merge 23 commits into
MemTensor:mainfrom
SleepySuke:feat/opengauss-backend
Open

syzsunshine219 wants to merge 23 commits into
MemTensor:mainfrom
SleepySuke:feat/opengauss-backend

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

Restores the proposal from #2564 as an open PR against main; its former base branch, dev-v2.0.36, is no longer available. This proposal requires maintainer review of the feature or design direction before merging.

Source head: ad10805219524785580074010aac852d8249b76e. No new source-code changes are introduced by restoring this proposal. Earlier discussion and evidence remain in #2564; validation claims in the original description refer to the earlier snapshot.

Outstanding review scope

当前main的配置工厂与实现工厂都没有注册openGauss,属于新增存储后端而非已确认遗漏的既有功能修复。需要确认维护负责人、部署依赖和后端支持计划,再决定是否恢复Draft。


Original proposal

Description

This PR adds openGauss backend support to MemOS as part of internal feature development, targeting dev-v2.0.36.

It integrates AGE graph storage, native vector retrieval, and BM25 full-text search with the existing memory components, API, and Scheduler.

Related Issue / Task: N/A — internally planned feature development with no associated issue or task number.

Main changes

  • Add openGauss configuration validation, factory registration, environment settings, and connection pooling.
  • Implement scoped node and relationship operations, graph traversal, metadata filtering, vector search, BM25 search, and SQL LIKE lookup.
  • Support memory counts, capacity management, snapshot export, atomic import, and scoped clearing.
  • Add optional Dashboard range selection through all_cubes and exclude_cube_id, with documentation and OpenAPI schema updates.
  • Improve resource ownership and shutdown ordering across API, Cube, Tree memory, and Scheduler components.
  • Preserve Cube identity during search and distinct identities for working and long-term memory pairs.
  • Propagate required storage failures through search and Feedback instead of treating failures as successful empty results.

Dependencies

  • openGauss Enterprise with AGE and native DataVec/BM25 capabilities.
  • psycopg2-binary >=2.9.13,<3.0.0, included in the optional tree-mem and all extras.

Enable the backend with GRAPH_DB_BACKEND=opengauss and configure the corresponding OPENGAUSS_* connection settings. The database and required extensions must be provisioned before use.

Behavior changes and limitations

  • Empty Dashboard categories return empty lists instead of synthetic empty Cube buckets.
  • Non-openGauss backends reject explicit non-null values for the new Dashboard range fields.
  • Required storage errors propagate to callers. Multi-step Feedback updates are not a single transaction.
  • Database-level semantic deduplication, conflict detection, and node merging remain unsupported.
  • Full service E2E, Java client integration, real-model validation, and production-scale performance testing remain follow-up work.

Type of change

  • New feature
  • Bug fix
  • Refactor
  • Documentation update

How Has This Been Tested?

Latest recorded local validation:

  • Full repository suite with real openGauss integration enabled: 2771 passed, 9 skipped, 45 warnings.
  • Unit and integration tests cover scoped storage, retrieval, rollback, backup/restore, Dashboard responses, memory identities, Feedback failures, and concurrent shutdown.
  • A separate local HTTP smoke test passed with real openGauss and controlled model assembly, covering Dashboard results, graceful shutdown, and database session cleanup.
  • Ruff checks and formatting passed.

Integration tests use openGauss 7.0.0 with AGE 1.0.0. Test fixtures provision disposable databases and roles; application operations run with an ordinary database role.

To reproduce, configure MEMOS_OPENGAUSS_TEST_CONFIG with a JSON OpenGaussGraphDBConfig for a test administrator, then run:

poetry install --extras all --with dev --with test
make format
make test

OpenAPI was exported using the repository's component fixtures. Standard make openapi remains blocked in the local environment by missing InternetRetriever model configuration.

These results describe local validation. Hosted CI and full service E2E are not claimed as complete.

  • Unit Test
  • Test Script Or Test Steps
  • Pipeline Automated API Test

Checklist

  • I have performed a self-review of the final PR diff
  • I have commented code in hard-to-understand areas
  • I have added tests covering the new functionality and fixes
  • I have included backend setup and behavior documentation in this PR
  • I have assigned or mentioned the reviewer

Issue/task linkage: N/A — no associated issue or internal task number.

Reviewer Checklist

  • Confirmed the implementation scope and behavior changes
  • Made sure required checks passed
  • Reviewed the provided tests and validation results

Add a strict OpenGaussGraphDBConfig for connection settings, logical
tenant scopes, and bounded connection-pool configuration.

Validate required credentials, port and timeout boundaries, positive
embedding dimensions, and ordered pool limits. Require a tenant scope
for shared databases and preserve explicitly supplied scope values.

Add 105 parameter cases covering valid and invalid configuration,
JSON/YAML round trips, hidden password repr, driver-independent imports,
and existing backend configuration selection.

Validation:
- make format
- poetry run pytest tests/configs tests/graph_dbs -q
  (237 passed; 3 existing opt-in Neo4j integration cases not enabled)
- New configuration class: 26 statements and 6 branch exits covered,
  with zero missing statements or branch exits
- pre-commit checks for both changed files passed
Add an internal node store with scoped business identities, stable native
graphids, atomic batch upserts, ordered reads and row-locked partial updates.
Reject incompatible layouts without deleting data.

Prepare ordinary-role AGE sessions through a bounded psycopg2 pool. Preserve
driver causes, roll back failures and close owned connections deterministically.
Check server session cleanup within a fixed deadline after client close.

Add GraphDBError and declare psycopg2-binary in tree-mem/all extras.
Keep the incomplete backend out of the public factories.

Validate 107 unit cases and 29 live openGauss integration cases.
Config/graph regression: 373 passed, 3 existing skips.
Full suite: 1207 passed, 9 existing skips, with live openGauss enabled
and Hugging Face restricted to the local cache.
Pool/store statement and branch coverage: 100%, with no new exclusions.
Add native directed typed relationships, scoped edge/neighbor queries,
and atomic deletion of a node and all its incident edges.
Serialize graph mutations to preserve edge uniqueness and prevent deletion races.

Delete incident edges with AGE and node label rows with SQL in one transaction,
preserving trigger and permission failure rollback behavior. Bind business
values and validate relationship identifiers before borrowing a connection.

Add 73 unit cases and 17 live integration cases for directions, scope,
self/parallel edges, concurrency, special values and rollback.
Cumulative config/graph regression: 463 passed, 3 existing skips.
Full suite: 1297 passed, 9 existing skips, with live openGauss enabled
and Hugging Face restricted to the local cache.
Pool/store statement and branch coverage: 100%, with no new exclusions.
Add directed shortest paths with deterministic tie breaking and a strict
five-hop bound. Return complete two-way neighborhood nodes and induced
edges, preserving native direction, parallel edge types and self-loops.

Rank active non-reasoning/non-working memories by unique tag overlap,
with scoped exclusions, deterministic ties and optional vector projection.
Reject malformed tag data and invalid inputs instead of returning false success.

Add 86 unit cases and 21 live integration cases for traversal, projections,
scope, special strings, tag filtering and permission failure recovery.
Cumulative config/graph regression: 570 passed, 3 existing skips.
Full suite: 1404 passed, 9 existing skips, with live openGauss enabled
and Hugging Face restricted to the local cache.
Pool/store statement and branch coverage: 100%, with no new exclusions.
Consolidate the connection pool, AGE node/graph operations and filters in
OpenGaussGraphDB, following the existing PostgreSQL and PolarDB layout.
Create the pool in __init__ and keep filter parameters local to each call.

Include scoped metadata/type queries, atomic conditional deletion and
soft-delete/restore visibility tests. Preserve AGE fractional-number
round trips and the existing transaction, isolation and error contracts.

Keep unfinished BaseGraphDB methods abstract and factory registration
pending. Replace the private connection and storage modules with
opengauss.py; test-only concrete support raises for every unfinished
capability.

Validation: 398 storage/graph unit tests and 125 real openGauss integration
tests; configs/graph_dbs 760 passed, 3 existing skips; full suite 1594 passed,
9 existing skips. opengauss.py has 100% statement and branch coverage.
Formatting, applicable hooks and lock checks passed; temporary resources 0.
Add DataVec cosine retrieval with normalized scores, scoped filters,
thresholds, projection and compatible Cube completion. Validate HNSW
indexes through 2000 dimensions and use exact search up to 16000 dimensions.
Replace underfilled vector candidates with one identically filtered query.

Add native BM25 OR retrieval with raw scores and TAAT candidate scoring,
including requests above 200 results. Validate the owned text index and
recover its verified failed-write state with one REINDEX and one read retry.
Keep unrelated errors explicit and never replay writes.

Validate real GraphMemoryRetriever completion, filtering, rollback,
concurrent recovery, index permissions, cancellation and timeouts. Keep
factory registration pending the remaining maintenance and consumer APIs.

Validation: make format; applicable hooks; poetry check --lock;
configs/graph_dbs/retriever: 966 passed, 3 existing skips;
make test: 1794 passed, 9 existing skips, including all 170 openGauss IT;
backend statement
and branch coverage: 100% (869 statements, 358 branch exits).

This message describes the complete current workspace delta from 86ef67b.
Add scoped grouping, memory counts, existence checks, and oldest-memory
eviction to OpenGaussGraphDB. Count all statuses and keep numeric AGE groups
consistent with transaction-local sort aggregation.

Lock a fixed candidate set, validate every timestamp, then remove incident
edges and nodes in batches of at most 500 within one transaction. Preserve
concurrent inserts for a later cleanup and propagate transaction failures
without retrying writes.

Cover ordinary-role database behavior, public vector/BM25 visibility,
multibatch rollback, concurrent writers, cancellation, connection replacement,
commit/return failures, and real MemoryManager/Reorganizer boundaries.

Centralize transactional cursors and optional UTC/advisory setup. Compose
named SQL fragments with collision checks and reusable bindings, and separate
grouping validation, query construction, and result decoding.

Validate with 741 backend unit tests and 226 real openGauss integration tests.
The full suite passes 2040 tests with 9 existing skips; the complete openGauss
module has 100% statement and branch coverage.

Factory registration and the eight remaining abstract methods stay pending.
Select strictly activated memories of the requested type and Cube without
same-Cube PARENT adjacency in one SQL snapshot. Preserve deterministic order,
optional embeddings, scoped graph identities and existing transaction errors.

Request embeddings from Reorganizer and verify real partitioning, summaries
and PARENT writes with controlled model services. Keep the remaining seven
abstract methods pending and factory wiring unchanged.

Validation: make format; enabled openGauss make test: 2115 passed, 9 existing
skips. Full opengauss.py statements/branches and modified optimize_structure
function reach 100%. Temporary databases, roles and test/application sessions
are zero after the run.
Add explicit cross-Cube reads with preserved node and edge ownership, global
pagination, filtering and stable ordering. Keep ordinary exports scoped to
the configured Cube. Counts, nodes and induced edges share one read-only
REPEATABLE READ transaction; later borrowers return to read-write defaults.

Validation: 40 Red to Green tests; 1407 passed / 3 existing skips in affected
configuration, graph and textual-memory regression with real openGauss.
Full opengauss.py: 1060 statements and 446 branch exits covered, no omissions.
make format and git diff --check pass.
Validate graph input before borrowing, preserve existing node identities and
merge edges in one transaction. Reject missing endpoints without partial
imports. Clear a fixed locked Cube set in 500-node blocks and retain shared
schema and other Cubes; drop_database clears only its configured scope.

Validation: 43 Red to Green tests plus commit uncertainty, cancellation,
physical scope and concurrency cases. Related real-database regression:
1456 passed, 3 existing skips. Full opengauss.py: 1123 statements and 482
branch exits covered. make format and git diff --check pass.
Resolve unique Cube ownership and reject ambiguous IDs. Add atomic scoped
soft deletion, recovery and batch hard deletion while preserving other node
properties, vectors and relations.

Validation: 38 initial Red/Green cases, 3 concurrent transaction cases;
1497 passed and 3 existing skips in affected regression. The full backend
has 1172/1172 statements and 496/496 branches covered; make format passes.
Implement scoped one-hop outgoing context queries. Explicitly reject semantic
deduplication, conflict detection and merging with named capability errors;
keep application MMR independent. All tests now use the production class.

Validation: 6 initial Red cases; 8 targeted Green cases including structure.
Full make test with real openGauss: 2246 passed, 9 unchanged existing skips.
Backend coverage: 1183/1183 statements and 496/496 branches; changed
Reorganizer function: 48/48 statements and 24/24 branches. make format and
git diff --check pass. Temporary databases, roles and sessions: zero.
Exercise batch hard deletion under permission denial, last-block timeout
and actual driver cancellation. Verify full rollback, preserved deletion
records, replacement connections and search visibility after recovery.

Validation: 30 targeted real-database tests pass; final make test reports
2249 passed with the same 9 existing skips. Full backend statement and
branch coverage remains 100%; make format and git diff --check pass.
Restore read-only component mappings and leave close state with resource owners. Quiesce recovery before draining current and retired executors; reject self-close before entering terminal state. Preserve concurrent close results, retryable cleanup, explicit Cube initialization, and diagnostic logs.
Preserve primary IDs and derive stable Working IDs with working_binding. Submit each pair in one batch transaction on openGauss, propagate write failures, and preserve input result order across both add paths. Reuse binding cleanup rules and protect historical self-bindings. Cover rollback, retries, concurrent writes, capacity cleanup, and default Feedback with real database tests.
@Memtensor-AI Memtensor-AI added area:api 云服务 / FastAPI / OpenAPI / MCP area:core MOS 编排层 / 框架底座 / 跨模块问题 area:database graph_db + vector_db | 图数据库与向量数据库 area:docs 文档、示例 area:memcube GeneralMemCube / cube 生命周期 / cube 配置 area:memory 记忆存储、检索、更新、召回逻辑 area:scheduler 调度模块 labels Nov 9, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Nov 9, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2610
Task: 7db9ff84751d9edb
Base: main
Head: feat/opengauss-backend
Head SHA: ad10805219524785580074010aac852d8249b76e

🔍 OpenCodeReview found 23 issue(s) in this PR.

⚠️ Review warnings (2)

  • bounded analysis finalized with explicit task_done after reviewing all group files
  • bounded analysis finalized with explicit task_done after reviewing all group files

1. docs/openapi.json (L1123-L1138)

APIADDRequest is the only request schema with no required array, and user_id is emitted as a plain non-nullable "type": "string" with no default. That combination is unusable for generated clients: they cannot omit the property (server-side it is dereferenced unconditionally) and cannot send null either, yet nothing marks it mandatory. The root cause is product_models.py declaring user_id: str = Field(None, ...) instead of a required field, so the generated contract silently advertises a mandatory identifier as optional. Fix the model (user_id: str = Field(..., description="User ID")) and regenerate with make openapi; the schema should then carry "required": ["user_id"] like the sibling request models.


2. docs/openapi.json (L767-L773)

The contract defines no components.securitySchemes and no security requirement on any operation, while destructive/read operations such as /product/delete_memory, /product/delete_memory_by_record_id, /product/get_memory and /product/get_all are scoped solely by a caller-supplied user_id. This is faithful to the code today (server_router.py registers no auth dependency on its APIRouter), but the repo already ships memos/api/middleware/auth.py:verify_api_key, which admin_router.py uses — so the published spec tells SDK generators the data API needs no credentials and implies cross-user access is by design. Either wire the product routes to verify_api_key (and regenerate, which will emit the Authorization API-key scheme and per-operation security) or explicitly document these endpoints as internal/unauthenticated so consumers don't build clients that assume isolation.


3. docs/openapi.json (L1329-L1340)

The spec relies on the human-readable (Deprecated) prefix in description for legacy fields, but never emits the machine-readable "deprecated": true marker (grep for "deprecated" in this file returns nothing). The same applies to mem_cube_id, moscube, memory_content, doc_path, source, operation (APIADDRequest), mem_cube_id/moscube (APISearchRequest, ChatRequest, etc.) and conversation_id (DeleteMemoryRequest). Generated SDKs and docs tooling (OpenAPI codegen, Swagger UI, Redoc) only surface deprecation from that flag, so clients will keep using fields the server logs as deprecated. Add "deprecated": true to those property schemas (i.e. set deprecated=True on the corresponding Pydantic Field) and regenerate.

💡 Suggested Change

Before:

          "mem_cube_id": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "type": "null"
              }
            ],
            "title": "Mem Cube Id",
            "description": "(Deprecated) Target cube ID for this add request (optional for developer API)."
          },

After:

          "mem_cube_id": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "type": "null"
              }
            ],
            "title": "Mem Cube Id",
            "deprecated": true,
            "description": "(Deprecated) Target cube ID for this add request (optional for developer API)."
          },

4. docs/openapi.json (L478)

The chat operations only document 200 and 422, but the handlers return 503 Service Unavailable when the chat service is not initialized (see server_router.chat_stream, chat_complete, chat_stream_playground, chat_stream_business_user, which raise HTTPException(status_code=503, ...)). Declaring that response (e.g. "503": {"description": "Chat service unavailable"}) would let generated clients handle the degradation path instead of treating it as an unexpected error. Note /product/chat/complete also declares an empty 200 schema, so clients get no type information for the non-streaming completion payload.


5. src/memos/graph_dbs/opengauss.py (L2076-L2077)

datetime.fromisoformat() only accepts the Z UTC suffix starting in Python 3.11, but this project supports >=3.10. On 3.10, filtering created_at/updated_at/delete_time with the common 2024-01-01T00:00:00Z form raises ValueError and surfaces as "Invalid ISO date/time filter", while the same filter silently works on 3.11+. Normalize a trailing Z before parsing so filter behavior is version-independent.

💡 Suggested Change

Before:

                    instant = datetime.fromisoformat(value)
                    if instant.tzinfo is None:

After:

                    normalized = value[:-1] + "+00:00" if value.endswith("Z") else value
                    instant = datetime.fromisoformat(normalized)
                    if instant.tzinfo is None:

6. src/memos/graph_dbs/opengauss.py (L1418-L1419)

This loop issues a separate _cypher() call (which itself runs DEALLOCATE ALL / PREPARE / EXECUTE / DEALLOCATE) plus a single-row DELETE for every selected node. For a bulk delete by filter/file_ids over N rows that is roughly 6N round-trips, which will be very slow. _delete_locked_nodes() already batches incident-edge deletion in chunks of 500 and issues one DELETE per chunk; reuse that here (grouping ids by scope) instead of per-row statements.


7. src/memos/lifecycle.py (L43-L44)

The concurrent-close wait is unbounded. cleanup callbacks registered on CloseState do blocking I/O — e.g. Dispatcher._close_executors calls executor.shutdown(wait=True, cancel_futures=True) (a non-cancellable long-running task will block there), and graph/monitor closes join threads. If the owner's cleanup hangs, every other close caller blocks forever inside attempt.done.wait(), hanging process teardown with no way to report the failure. Since callers must not report success unless the resource is actually closed, a bounded wait that raises an explicit MemOSError on timeout keeps the same contract without the permanent hang.

💡 Suggested Change

Before:

        if not execute:
            attempt.done.wait()

After:

        if not execute:
            if not attempt.done.wait(_CLOSE_WAIT_TIMEOUT_SECONDS):
                raise MemOSError("Timed out waiting for the concurrent component close")

8. src/memos/lifecycle.py (L45-L46)

A waiter inspects only attempt.error and never re-checks self._closed. After a failed attempt, another caller is allowed to retry (execute is true once attempt.done is set) and may succeed; a waiter that was parked on the earlier failed attempt then still raises MemOSError("Concurrent component close failed") even though the component is now closed. Re-check _closed under the lock before raising so a successful concurrent retry is reported as success.

💡 Suggested Change

Before:

            if attempt.error is not None:
                raise MemOSError("Concurrent component close failed") from attempt.error

After:

            if attempt.error is not None:
                with self._lock:
                    if self._closed:
                        return
                raise MemOSError("Concurrent component close failed") from attempt.error

9. src/memos/mem_cube/general.py (L150-L153)

TreeTextMemory.dump already accepts include_embedding, and the memory instance already carries the configured value (self.include_embedding, fed by the include_embedding config / INCLUDE_EMBEDDING env), while TreeTextMemory.dump itself already special-cases OpenGaussGraphDB internally (the status option). Hard-coding include_embedding=True here silently overrides the user configuration for OpenGauss cubes and pushes a backend-specific switch up into the generic cube. If OpenGauss dumps must carry embeddings for a usable round-trip, consider expressing that through the existing flag (e.g. include_embedding=self.text_mem.include_embedding or <opengauss requirement>) so the configurable behaviour stays observable in one place. Non-blocking.


10. src/memos/memories/textual/tree.py (L115-L123)

close() releases the graph store but never stops the background work started by self.memory_manager. When config.reorganize is enabled, GraphStructureReorganizer.__init__ starts two non-daemon threads (_run_message_consumer_loop, _run_structure_organizer_loop) that keep issuing queries against this same graph_store. After this close the store is shut down while those threads are still alive for any caller of TreeTextMemory.close() (e.g. MemCube.close() in mem_cube/general.py), so they will loop on GraphDBError indefinitely and keep the process from exiting cleanly; stopping the reorganizer is currently only done by the ad-hoc MemOS.mem_reorganizer_off() path. Please drain the memory manager before closing the graph, e.g.:

        def release() -> None:
            if self._owns_graph_store:
                self.memory_manager.close()
                close = getattr(self.graph_store, "close", None)
                if callable(close):
                    close()
💡 Suggested Change

Before:

    def close(self) -> None:
        """Release the graph created by this memory; injected graphs are borrowed."""

        def release() -> None:
            close = getattr(self.graph_store, "close", None)
            if self._owns_graph_store and callable(close):
                close()

        self._close_state.close(release)

After:

    def close(self) -> None:
        """Release the graph created by this memory; injected graphs are borrowed."""

        def release() -> None:
            if not self._owns_graph_store:
                return
            self.memory_manager.close()
            close = getattr(self.graph_store, "close", None)
            if callable(close):
                close()

        self._close_state.close(release)

11. src/memos/memories/textual/tree.py (L417-L422)

all_cubes=False is the default/no-op value of the parameter, yet passing it explicitly makes get_all demand an OpenGauss backend and fail with ValueError even though no cross-Cube range was requested. Only forward all_cubes when it is truthy (or fall back to the plain per-Cube export otherwise) so callers that always pass all_cubes=False keep working on Neo4j/Postgres.

💡 Suggested Change

Before:

        scope = {}
        if all_cubes is not None or exclude_user_name is not None:
            if not isinstance(self.graph_store, OpenGaussGraphDB):
                raise ValueError("This graph backend does not support explicit Cube ranges")
            if all_cubes is not None:
                scope["all_cubes"] = all_cubes

After:

        scope = {}
        if all_cubes or exclude_user_name is not None:
            if not isinstance(self.graph_store, OpenGaussGraphDB):
                raise ValueError("This graph backend does not support explicit Cube ranges")
            if all_cubes:
                scope["all_cubes"] = all_cubes

12. src/memos/memories/textual/tree_text_memory/organize/manager.py (L133)

The fail-fast rewrite means the first failing future raises out of _add_memories_parallel before the remaining futures' results are retrieved, so:

  1. writes already committed by the other futures are silently discarded from the returned id list (callers lose track of nodes that do exist in the graph), and exceptions raised by those other futures are never observed;
  2. in add(), the raise happens before if mode == "sync": self._cleanup_working_memory(...), so a single failed write skips the WorkingMemory trim that previously always ran, letting the WorkingMemory backlog grow past memory_size["WorkingMemory"].

If propagation is required, at least drain the remaining futures and preserve the ids of the writes that succeeded, and make sure sync-mode cleanup still runs (e.g. via try/finally).


13. src/memos/memories/textual/tree_text_memory/organize/manager.py (L159-L160)

Previously each batch failure was logged and swallowed, so _submit_batches always returned and reorganizer.add_message(...) was still queued with all graph ids. Now future.result() re-raises before line if added_ids and self.is_reorganize:, so batches that already committed are left in the graph with no reorganize message queued for them (they will never be picked up by structure optimization unless a later add happens). Consider queuing the message for the successfully written ids in a finally/before re-raising, or document that partial batches intentionally skip reorganization.


14. src/memos/mem_scheduler/memory_manage_modules/enhancement_pipeline.py (L200)

The loop index i is unused in this comprehension (it is only needed when embedding the index as in _build_enhancement_prompt's REWRITE branch). Dropping enumerate keeps the re-indented block clean and avoids implying the index matters.

💡 Suggested Change

Before:

            text_memories = "\n".join([f"- {mem}" for i, mem in enumerate(memories)])

After:

            text_memories = "\n".join([f"- {mem}" for mem in memories])

15. src/memos/mem_scheduler/monitors/dispatcher_monitor.py (L346-L351)

start() mutates the shared shutdown state (_recovery_paused = False, _stop_event.clear()) before it verifies that it will actually start a thread. Because stop() sets those flags under _pool_lock and then joins outside the lock, a start() that acquires the lock in that window will clear _stop_event and re-enable recovery for a monitor that is being torn down: the old loop may keep running and stop()'s now unbounded join() can block forever. Move the state resets inside the "we are going to start" branch so a duplicate/crossed start() cannot resurrect a stopped monitor.

💡 Suggested Change

Before:

            self._close_state.ensure_open()
            self._recovery_paused = False
            self._stop_event.clear()
            if self._running:
                logger.warning("Dispatcher Monitor is already running")
                return False

After:

            self._close_state.ensure_open()
            if self._running:
                logger.warning("Dispatcher Monitor is already running")
                return False

            self._recovery_paused = False
            self._stop_event.clear()

16. src/memos/mem_scheduler/base_scheduler.py (L427-L430)

_lifecycle_entry() calls CloseState.ensure_open(), and CloseState.close() sets closing before any cleanup runs. From the moment scheduler.close() is entered, every self.mem_cube / self.feedback_server read raises MemOSError("Component is closing or closed") — even when the cube is already published.

That breaks the graceful drain this method is supposed to provide: _close_resources waits for running work in self.dispatcher.close() (executor.shutdown(wait=True)), but all handlers resolve the cube lazily inside the task (SchedulerHandlerContext(get_mem_cube=lambda: self.mem_cube, ...) in general_scheduler.py, used by add/answer/feedback/mem_read/query handlers, and self.mem_cube in optimized_scheduler.update_search_memories_to_redis). Those in-flight tasks will now fail with MemOSError mid-execution (and services.submit_messages follow-ups are rejected too) instead of completing.

Suggestion: only gate the lazy creation path on ensure_open() and keep serving an already-published cube while shutting down.

💡 Suggested Change

Before:

        with self._lifecycle_entry():
            if self.current_mem_cube is None and not self._borrowed_components:
                self._ensure_components()
            return self.current_mem_cube

After:

        cube = self.current_mem_cube
        if cube is not None or self._borrowed_components:
            return cube
        with self._lifecycle_entry():
            if self.current_mem_cube is None:
                self._ensure_components()
            return self.current_mem_cube

17. src/memos/mem_scheduler/base_scheduler.py (L383-L387)

thread.join() here has no timeout, so shutdown can block indefinitely. The loop only re-checks _stop_event between iterations; with enable_parallel_dispatch=False, SchedulerDispatcher.execute_task runs handlers synchronously on the consumer thread, so a single stalled handler (LLM/network call) makes close() hang forever, with no escalation path.

stop_consumer() intentionally uses join(timeout=5.0) + terminate() fallback for exactly this reason; this new final-close path drops all timeouts. Please bound the joins and fail (or log and continue) instead of blocking forever.

💡 Suggested Change

Before:

        for thread in (self._consumer_thread, self._monitor_thread):
            if thread is not None:
                if thread is threading.current_thread():
                    raise MemOSError("Scheduler cannot close from its consumer thread")
                thread.join()

After:

        for thread in (self._consumer_thread, self._monitor_thread):
            if thread is not None:
                if thread is threading.current_thread():
                    raise MemOSError("Scheduler cannot close from its consumer thread")
                thread.join(timeout=5.0)
                if thread.is_alive():
                    raise MemOSError(f"Scheduler {thread.name} has not exited")

18. src/memos/mem_scheduler/base_scheduler.py (L361-L365)

self._running = False / self._stop_event.set() are repeated verbatim after the lock barrier (and begin_close() is invoked twice). Only the monitor re-check after with self._initialization_lock is meaningful (an in-flight _initialize_modules may have just created the monitor); the duplicated resets add nothing and make the shutdown ordering harder to reason about.

💡 Suggested Change

Before:

        self._running = False
        self._stop_event.set()
        # An in-flight initialize_modules may have just created the monitor.
        if self.dispatcher_monitor is not None:
            self.dispatcher_monitor.begin_close()

After:

        # An in-flight initialize_modules may have just created the monitor.
        if self.dispatcher_monitor is not None:
            self.dispatcher_monitor.begin_close()

19. src/memos/mem_scheduler/monitors/dispatcher_monitor.py (L263-L265)

The monitor now mutates SchedulerDispatcher private state from another module (_retired_executors, _mark_worker) without going through the dispatcher's synchronization. SchedulerDispatcher._close_executors() drains [self.dispatcher_executor, *self._retired_executors] from a snapshot taken in the close thread, so a retired executor appended by the monitor while close is draining is silently missed, and the coupling makes the ownership contract ("dispatcher owns executors") implicit. Consider exposing a dispatcher method, e.g. dispatcher.install_executor(old, new) / dispatcher.retire_executor(old), that performs the retire + swap under the dispatcher's own lock so registration and final cleanup stay in one place.


20. src/memos/mem_scheduler/monitors/dispatcher_monitor.py (L277-L279)

In this abort path the freshly created new_executor is only handed to dispatcher._retired_executors, on the assumption that SchedulerDispatcher._close_executors() will still run afterwards. If the dispatcher has already completed its close (its _close_executors() snapshot was taken before this append), nothing will ever shut this executor down. Shut it down here before returning so the abort path is self-contained.

💡 Suggested Change

Before:

                if self._close_state.closing.is_set() or self._recovery_paused:
                    self.dispatcher._retired_executors.append(new_executor)
                    return

After:

                if self._close_state.closing.is_set() or self._recovery_paused:
                    new_executor.shutdown(wait=False, cancel_futures=True)
                    self.dispatcher._retired_executors.append(new_executor)
                    return

21. src/memos/mem_scheduler/base_scheduler.py (L367)

_close_resources releases RabbitMQ, the dispatcher monitor, consumers, the dispatcher and self.components, but never releases self.monitor (SchedulerGeneralMonitor), even though it owns per-user DB managers and exposes a close() that shuts them down (see monitors/general_monitor.close). After scheduler shutdown those DB connections stay open — the exact leak this drain path is meant to remove. Consider releasing it alongside the other owned resources.

💡 Suggested Change

Before:

        release("rabbitmq", self.rabbitmq_close)

After:

        release("rabbitmq", self.rabbitmq_close)
        release("general_monitor", self.monitor.close)

22. src/memos/mem_scheduler/base_scheduler.py (L420-L422)

_lifecycle_entry() is described as serializing component creation/start against shutdown, but it holds the scheduler-wide _initialization_lock for the entire body. Since submit_messages may run a LEVEL_1 handler inline (when enable_parallel_dispatch is False, dispatcher.execute_task calls wrapped_handler(msgs) synchronously), every concurrent mem_cube / feedback_server / submit_messages call from other threads blocks behind a potentially long LLM/DB handler. Consider acquiring the lock only for the guarded check and releasing it before the actual submission/dispatch.


23. src/memos/mem_scheduler/base_scheduler.py (L468-L471)

When _borrowed_components is True (e.g. init_mem_cube(mem_cube, searcher, feedback_server=None) called from the mem_cubes setter), the lazy fallback is now disabled and this getter returns None, although the declared return type is SimpleMemFeedback. mem_scheduler/task_schedule_modules/handlers/feedback_handler.py does self.scheduler_context.get_feedback_server().process_feedback(...), which would raise AttributeError on None. The previous implementation at least attempted init_components() here. Consider returning/handling the absent case explicitly.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: unsupported_lock/profile: changed poetry.lock requires its lock installer adapter
Branch: feat/opengauss-backend

Tested source: ad10805219524785580074010aac852d8249b76e
Tested base: 3a16e5fe3048a209f722806f4123fa2e533588fd
Tested commit: 064d922635c9775a6d2a8ffcdc181c96f0235647
These results apply only to this immutable source/base snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:api 云服务 / FastAPI / OpenAPI / MCP area:core MOS 编排层 / 框架底座 / 跨模块问题 area:database graph_db + vector_db | 图数据库与向量数据库 area:docs 文档、示例 area:memcube GeneralMemCube / cube 生命周期 / cube 配置 area:memory 记忆存储、检索、更新、召回逻辑 area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants