diff --git a/docs/cn/open_source/modules/memories/opengauss_graph_db.md b/docs/cn/open_source/modules/memories/opengauss_graph_db.md new file mode 100644 index 000000000..2836d87d8 --- /dev/null +++ b/docs/cn/open_source/modules/memories/opengauss_graph_db.md @@ -0,0 +1,80 @@ +--- +title: "openGauss 图存储" +desc: "介绍 MemOS 的 openGauss 图存储后端配置、Dashboard 查询范围、资源所有权、备份恢复及 Feedback 支持范围。" +--- + +设置 `GRAPH_DB_BACKEND=opengauss`,并配置 `OPENGAUSS_HOST`、`OPENGAUSS_USER`、`OPENGAUSS_PASSWORD` 和 `OPENGAUSS_DB`。树状记忆需启用 `tree-mem` 可选依赖组。目标数据库必须预先具备所需的 AGE 和 DataVec/BM25 能力;适配器不会创建物理数据库。 + +与 PostgreSQL、PolarDB 后端一样,需要在运行 MemOS 的 Python 环境中单独安装数据库驱动。`tree-mem` 和 `all` 均不会安装此驱动。已验证的驱动版本为 `psycopg2-binary==2.9.13`: + +```bash +python -m pip install "psycopg2-binary==2.9.13" +``` + +使用 Poetry 源码环境时,在 `make install` 后执行 `poetry run python -m pip install "psycopg2-binary==2.9.13"`。未安装驱动时仍可导入 MemOS;初始化 openGauss 后端会在打开连接前明确提示安装驱动。 + +可选配置包括:`OPENGAUSS_PORT`(默认 5432)、`OPENGAUSS_MIN_CONN`(默认 1)、`OPENGAUSS_MAX_CONN`(默认 20)、`OPENGAUSS_CONNECT_TIMEOUT`(默认 10 秒)、`OPENGAUSS_CONNECTION_WAIT_TIMEOUT`(默认 30 秒)和 `EMBEDDING_DIMENSION`(默认 1024)。请求中显式指定的 Cube ID 优先于配置中的默认范围。范围选择只用于查询过滤,不代表授予访问权限。 + +## Dashboard 查询范围 + +`POST /product/get_memory_dashboard` 接受两个可选且允许为 null 的字段。已有客户端可以继续省略这两个字段。 + +| 字段 | 含义 | +| --- | --- | +| `all_cubes: null` 或省略 | 查询传入的 `mem_cube_id`;未传入时查询配置中的默认 Cube。下文的旧版过滤控制字段仍可显式选择其他范围。 | +| `all_cubes: true` | 查询所有 Cube。不能同时传入外层 `mem_cube_id`,也不能与 `public` 单 Cube 选择同时使用。 | +| `all_cubes: false` | 查询指定 Cube;未指定时查询配置中的默认 Cube。 | +| `exclude_cube_id` | 从显式选择的全 Cube 范围中排除一个 Cube。仅传入此字段不会启用全 Cube 查询。值必须为非空白字符串;指定的 Cube 不存在时,不排除任何记录。 | + +例如,分页查询除 `public` 以外的所有范围: + +```json +{ + "all_cubes": true, + "exclude_cube_id": "public", + "page": 1, + "page_size": 20 +} +``` + +旧版过滤控制 `{"object_type":"public","mem_cube_id":"P"}` 表示选择 P,`{"object_type":"user","mem_cube_id":"P"}` 表示显式选择除 P 以外的所有 Cube。这些控制字段可以出现在过滤器根节点或 `and` 下,并对整个查询生效。`or` 下任何位置的范围控制字段都会被拒绝,因为将其提取出来会改变布尔表达式的含义;普通业务 `or` 过滤条件仍然保留。 + +允许重复传入相同的控制条件。外层 `mem_cube_id` 必须与 `public` 选择一致,不能与 `user` 选择同时使用。不完整或冲突的控制条件、未知的对象类型、空布尔节点、单 Cube 查询中的排除操作,以及冲突的排除 ID,都会返回参数错误。显式传入 `all_cubes=false` 与 `user` 选择也会产生冲突。 + +`user_id`、记忆类型、状态和业务过滤条件仍会进一步约束所选范围。分页时必须同时提供正数 `page` 和 `page_size`;同时省略这两个字段则返回所有匹配记录。四类记忆分别计算分页结果和总数。每个 Cube 分组的 `total_nodes` 表示当前页的记录数,`statistics` 表示分页前所有匹配记录的数量。 + +当当前页没有记录时,所有后端都会返回空的分类列表,即使显式指定了 Cube,也不会构造一个空 Cube 分组。当前页为空时,总数仍可能大于零。确实没有匹配记录时,响应结构如下: + +```json +{ + "code": 200, + "message": "Memories retrieved successfully", + "data": { + "text_mem": [], "tool_mem": [], "skill_mem": [], "pref_mem": [], + "statistics": { + "total_text_nodes": 0, "total_tool_nodes": 0, + "total_skill_nodes": 0, "total_preference_nodes": 0 + } + } +} +``` + +目前只有 openGauss 接受新增范围字段的非 null 值。对于其他适配器,省略字段或传入 null 时沿用原有范围规则,显式传入新的范围选择则会被拒绝。`all_cubes` 只接受 JSON 布尔值,不接受 `0`、`1` 或字符串。Java 客户端接入这些字段时,应使用可空的 `Boolean`/`String`,不要用基本类型布尔值默认的 `false` 代替“未指定”。 + +请求模型的类型或结构错误返回 HTTP 422,非法的范围或过滤条件组合返回 400,存储执行失败返回 500。错误响应包含 `data: null`;调用方不能将其视为查询成功但没有结果。普通 `/get_memory` 请求仍要求传入 `mem_cube_id`,不支持这两个新增范围字段。 + +## 资源所有权与当前支持范围 + +直接创建的 `TreeTextMemory` 和 `GeneralMemCube` 实例通过 `close()` 关闭自身拥有的资源。注入的图存储和记忆对象仍属于借用资源。API 会等待 Scheduler 完成对共享图的使用后再关闭图;独立初始化的 Scheduler 则关闭自身拥有的图。`Scheduler.stop()` 用于停止工作,`close()` 用于结束生命周期。最终关闭会拒绝新任务和懒初始化,等待正在执行的任务及恢复过程结束,报告清理失败,并允许重试尚未完成的清理。成功关闭后,重复调用不会再次执行清理。 + +为兼容已有实现,没有显式 `close` 方法的图适配器会被跳过;这不代表其资源已经释放。这些变更不提供完整的 Scheduler 重启支持。Tree 的组合 `drop` 流程、非默认 Feedback TFIDF 分支、实际 Java 集成、完整服务 HTTP E2E 和生产规模性能仍需单独验证。 + +`TreeTextMemory.dump` 导出一个 Cube 中所有状态的记忆,默认不包含向量;传入 `include_embedding=True` 可获得完整的逻辑备份。对于 openGauss Tree 记忆,`GeneralMemCube.dump` 会自动请求包含向量。只有写入、刷新、同步和关闭文件全部成功后,才会通过同目录原子替换发布文件。导出或写入失败时,原文件保持不变。该保证只覆盖单个记忆文件,不覆盖整个 Cube 目录。 + +显式执行加载时,文件缺失、不可读或 JSON 无效都会抛出 `memos.exceptions.MemoryError`。数据库异常保留其原始原因。恢复操作会原子地合并到同一个 Cube 中,保留备份文件中未出现的节点和边。导入不含嵌入向量的节点会清除该节点已有的向量。 + +搜索结果保留原始 Cube 身份,包括不同 Cube 中 ID 相同的记忆。并行 Cube 搜索按请求顺序合并结果,继续沿用已有 no/sim/MMR 选择和评分规则。必需的存储和输入错误会通过关键词搜索、API 处理器和调度 Feedback 向上传播;这些错误与查询成功但没有匹配结果不同。 + +默认 Feedback(`DB_IDX_READY=False`)支持通过 SQL LIKE 进行关键词替换。`search_by_keywords_like` 使用调用方传入的原始模式:`%` 匹配任意长度的字符序列,`_` 匹配单个字符,反斜杠用于转义这些字符。模式通过参数绑定传入,默认查询 activated 状态的记忆,应用已有 Cube/KB 范围和过滤条件,并按 Cube/ID 顺序返回全部匹配结果。空模式只匹配空文本;非法类型、NUL 和末尾未配对的转义字符会被拒绝。默认返回字段包含 `id`;跨 Cube 查询还包含 `user_name`。显式指定的字段可以包含 memory、embedding 和 metadata 字段。 + +Feedback 保留已有版本行为:关闭版本控制时,工作记忆原地更新,长期记忆先创建新节点,再归档旧节点。这些多次写入不属于同一个事务。存储错误会被报告,Feedback 包装层不会重试这类错误,因为连接失败可能导致某次写入是否已提交无法确定。此前已经成功的更新可能仍然保留。TFIDF 分支(`DB_IDX_READY=True`)仍不受支持。 diff --git a/docs/en/open_source/modules/memories/opengauss_graph_db.md b/docs/en/open_source/modules/memories/opengauss_graph_db.md new file mode 100644 index 000000000..b69cbb2f0 --- /dev/null +++ b/docs/en/open_source/modules/memories/opengauss_graph_db.md @@ -0,0 +1,99 @@ +# openGauss graph storage + +Select `GRAPH_DB_BACKEND=opengauss` and set `OPENGAUSS_HOST`, `OPENGAUSS_USER`, `OPENGAUSS_PASSWORD`, and `OPENGAUSS_DB`. Install the `tree-mem` extra for tree memory. The target database must already have the required AGE and DataVec/BM25 capabilities; the adapter does not create a physical database. + +As with the PostgreSQL and PolarDB backends, install the database driver separately in the Python environment running MemOS. Neither `tree-mem` nor `all` installs this driver. The verified driver version is `psycopg2-binary==2.9.13`: + +```bash +python -m pip install "psycopg2-binary==2.9.13" +``` + +For a Poetry checkout, run `poetry run python -m pip install "psycopg2-binary==2.9.13"` after `make install`. Without the driver, importing MemOS remains supported; initializing the openGauss backend raises an installation error before opening a connection. + +Optional settings are `OPENGAUSS_PORT` (5432), `OPENGAUSS_MIN_CONN` (1), `OPENGAUSS_MAX_CONN` (20), `OPENGAUSS_CONNECT_TIMEOUT` (10 seconds), `OPENGAUSS_CONNECTION_WAIT_TIMEOUT` (30 seconds), and `EMBEDDING_DIMENSION` (1024). Explicit request Cube IDs take precedence over the configured default scope. A scope selection is a query filter, not an authorization grant. + +## Dashboard ranges + +`POST /product/get_memory_dashboard` accepts two optional, nullable fields. Existing clients can omit both. + +| Field | Meaning | +| --- | --- | +| `all_cubes: null` or omitted | Select the supplied `mem_cube_id`, or the configured default Cube if omitted. Legacy filter controls below can explicitly select a different range. | +| `all_cubes: true` | Select all Cubes. Cannot be combined with an outer `mem_cube_id` or a `public` single-Cube selection. | +| `all_cubes: false` | Select the supplied Cube, or the configured default if none is supplied. | +| `exclude_cube_id` | Exclude exactly one Cube from an explicitly selected all-Cube range. This field alone does not enable all-Cube selection. Must be a nonblank string; an absent Cube excludes nothing. | + +For example, list all scopes except `public`, with pagination: + +```json +{ + "all_cubes": true, + "exclude_cube_id": "public", + "page": 1, + "page_size": 20 +} +``` + +The existing filter controls `{"object_type":"public","mem_cube_id":"P"}` select P, while `{"object_type":"user","mem_cube_id":"P"}` explicitly select all other Cubes. Controls can appear at the root or under `and` and apply globally. Scope controls anywhere beneath `or` are rejected because extracting them would change the Boolean meaning; ordinary business `or` filters are preserved. Repeated identical controls are accepted. An outer `mem_cube_id` must agree with a `public` selection and cannot accompany a `user` selection. Incomplete or conflicting controls, unknown object types, empty Boolean nodes, single-Cube exclusion, and conflicting exclusion IDs return parameter errors. Explicit `all_cubes=false` conflicts with a `user` selection. + +`user_id`, memory type, status and business filters still constrain the selected range. Supply both positive `page` and `page_size`, or omit both to return all matching records. Pagination and totals apply independently to each of the four memory categories. A bucket's `total_nodes` counts its records on the current page; `statistics` counts all matching records before pagination. + +All backends now return empty category lists when the page has no records, including explicitly selected Cubes. No empty Cube bucket is manufactured. An empty page can still have a positive total. A genuinely empty response has this shape: + +```json +{ + "code": 200, + "message": "Memories retrieved successfully", + "data": { + "text_mem": [], "tool_mem": [], "skill_mem": [], "pref_mem": [], + "statistics": { + "total_text_nodes": 0, "total_tool_nodes": 0, + "total_skill_nodes": 0, "total_preference_nodes": 0 + } + } +} +``` + +Only openGauss currently accepts non-null new range fields. Other adapters preserve their existing range rules for omitted/null fields and reject explicit new selections. `all_cubes` accepts JSON booleans, not `0`, `1`, or strings. Java clients adopting the fields should use nullable `Boolean`/`String`; do not replace an omitted choice with a primitive Boolean's default `false`. + +Expect HTTP 422 for request-model type/shape errors, 400 for invalid range/filter combinations, and 500 for storage execution failures. An error response has `data: null`; callers must not interpret it as an empty successful query. The ordinary `/get_memory` request still requires `mem_cube_id` and does not gain these range fields. + +## Resource ownership and current boundaries + +Direct `TreeTextMemory` and `GeneralMemCube` instances close their own resources through `close()`. Injected graphs and memories remain borrowed. The API closes its shared graph after its Scheduler has finished using it; independently initialized Schedulers close their own graph. `Scheduler.stop()` stops work, while `close()` ends the lifecycle. Final close rejects new work and lazy initialization, waits for running tasks and recovery, reports cleanup failures, and can retry incomplete cleanup. Successful repeated close is a no-op. + +Graph adapters without an explicit `close` are skipped for compatibility; that does not establish that their resources were released. This change does not add complete Scheduler restart support. The combined Tree `drop` workflow, non-default Feedback TFIDF, actual Java integration, full service HTTP E2E, and production-scale performance require separate validation. + +TreeTextMemory.dump exports every state from one Cube. Its default omits vectors; +pass `include_embedding=True` for a full logical backup. GeneralMemCube.dump +requests vectors for openGauss Tree memory automatically. Files are published +with a same-directory atomic replacement only after writing, flushing, syncing, +and closing succeeds. A failed export or write preserves the previous file. +This guarantee covers one memory file, not the Cube's entire directory. + +An explicit load fails on missing, unreadable, or invalid JSON files with +`memos.exceptions.MemoryError`. Database errors retain their cause. Restoration +is an atomic merge into the same Cube: it preserves nodes and edges absent from +the file. Importing a node without an embedding clears its existing embedding. + +Search results retain their original Cube identity, including equal memory IDs +in different Cubes. Parallel Cube searches merge in request order. The existing +no/sim/MMR selection and scoring rules remain in use. Required storage and input +errors propagate through keyword search, API handlers and scheduled Feedback; +these errors are distinct from a successful query with no matches. + +Default Feedback (`DB_IDX_READY=False`) supports SQL LIKE keyword replacement. +`search_by_keywords_like` accepts the caller's exact pattern: `%` matches any +sequence, `_` matches one character, and backslash escapes those characters. +It binds the pattern, defaults to activated memories, applies the usual Cube/KB +scope and filters, and returns all matches in Cube/ID order. Empty patterns only +match empty text; invalid types, NUL and an unpaired trailing escape are rejected. +The default projection contains `id`; multi-Cube projections also contain +`user_name`. Explicit fields can include memory, embedding and metadata fields. + +Feedback preserves the existing version behavior: with versioning off, working +memory updates in place and long-term memory creates a new node then archives +the old one. These multiple writes are not one transaction. Storage errors are +reported and are not retried by the Feedback wrapper, since a failed connection +can leave a write's commit outcome unknown. Successful earlier updates may remain. +The TFIDF branch (`DB_IDX_READY=True`) remains unsupported. diff --git a/docs/openapi.json b/docs/openapi.json new file mode 100644 index 000000000..8bc4e5106 --- /dev/null +++ b/docs/openapi.json @@ -0,0 +1,4855 @@ +{ + "openapi": "3.1.0", + "info": { + "title": "MemOS Server REST APIs", + "description": "A REST API for managing multiple users with MemOS Server.", + "version": "1.0.1" + }, + "paths": { + "/product/search": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Search memories", + "description": "Search memories for a specific user.\n\nThis endpoint uses the class-based SearchHandler for better code organization.", + "operationId": "search_memories_product_search_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/APISearchRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SearchResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/add": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Add memories", + "description": "Add memories for a specific user.\n\nThis endpoint uses the class-based AddHandler for better code organization.", + "operationId": "add_memories_product_add_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/APIADDRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/create_cube": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Create a new memory cube", + "description": "Create a new memory cube for a user.\n\nMemory cubes are containers that store different types of memories (textual, activation, parametric).\nEach cube can be owned by a user and shared with other users.\n\n**Note on cube_id vs mem_cube_id:**\nThese terms are used interchangeably throughout the API:\n- `cube_id` is the canonical identifier for a cube\n- `mem_cube_id` appears in many legacy endpoints and means the same thing\n- When using other endpoints (search, add, chat), you can reference this cube using either term\n\n**Semantic Clarification:**\n- **Single mem_cube_id** (deprecated): Used in older endpoints to identify a single cube.\n New code should use `readable_cube_ids` / `writable_cube_ids` lists instead.\n- **readable_cube_ids**: List of cube IDs the user can read from (used in search/chat)\n- **writable_cube_ids**: List of cube IDs the user can write to (used in add/chat)", + "operationId": "create_cube_product_create_cube_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CreateCubeRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CreateCubeResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/register_cube": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Register an existing memory cube", + "description": "Register an existing memory cube with the MOS system.\n\nThis method loads and registers a memory cube from a file path or creates a new one\nif the path doesn't exist. The cube becomes available for memory operations.\n\n**Note on cube_id vs mem_cube_id:**\nThese terms are used interchangeably throughout the API. The registered cube can then\nbe referenced by its cube_id/mem_cube_id in other endpoints.\n\n**Current Status:**\nThis endpoint validates the registration request. Full registration functionality\nrequires architectural integration with MOSCore, which will be completed in a future update.", + "operationId": "register_cube_product_register_cube_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/RegisterCubeRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/RegisterCubeResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/scheduler/allstatus": { + "get": { + "tags": [ + "Server API" + ], + "summary": "Get detailed scheduler status", + "description": "Get detailed scheduler status including running tasks and queue metrics.", + "operationId": "scheduler_allstatus_product_scheduler_allstatus_get", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AllStatusResponse" + } + } + } + } + } + } + }, + "/product/scheduler/status": { + "get": { + "tags": [ + "Server API" + ], + "summary": "Get scheduler running status", + "description": "Get scheduler running status.", + "operationId": "scheduler_status_product_scheduler_status_get", + "parameters": [ + { + "name": "user_id", + "in": "query", + "required": true, + "schema": { + "type": "string", + "description": "User ID", + "title": "User Id" + }, + "description": "User ID" + }, + { + "name": "task_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Optional Task ID to query a specific task", + "title": "Task Id" + }, + "description": "Optional Task ID to query a specific task" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/StatusResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/scheduler/task_queue_status": { + "get": { + "tags": [ + "Server API" + ], + "summary": "Get scheduler task queue status", + "description": "Get scheduler task queue backlog/pending status for a user.", + "operationId": "scheduler_task_queue_status_product_scheduler_task_queue_status_get", + "parameters": [ + { + "name": "user_id", + "in": "query", + "required": true, + "schema": { + "type": "string", + "description": "User ID whose queue status is requested", + "title": "User Id" + }, + "description": "User ID whose queue status is requested" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/TaskQueueResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/scheduler/wait": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Wait until scheduler is idle for a specific user", + "description": "Wait until scheduler is idle for a specific user.", + "operationId": "scheduler_wait_product_scheduler_wait_post", + "parameters": [ + { + "name": "user_name", + "in": "query", + "required": true, + "schema": { + "type": "string", + "title": "User Name" + } + }, + { + "name": "timeout_seconds", + "in": "query", + "required": false, + "schema": { + "type": "number", + "default": 120.0, + "title": "Timeout Seconds" + } + }, + { + "name": "poll_interval", + "in": "query", + "required": false, + "schema": { + "type": "number", + "default": 0.5, + "title": "Poll Interval" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": {} + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/scheduler/wait/stream": { + "get": { + "tags": [ + "Server API" + ], + "summary": "Stream scheduler progress for a user", + "description": "Stream scheduler progress via Server-Sent Events (SSE).", + "operationId": "scheduler_wait_stream_product_scheduler_wait_stream_get", + "parameters": [ + { + "name": "user_name", + "in": "query", + "required": true, + "schema": { + "type": "string", + "title": "User Name" + } + }, + { + "name": "timeout_seconds", + "in": "query", + "required": false, + "schema": { + "type": "number", + "default": 120.0, + "title": "Timeout Seconds" + } + }, + { + "name": "poll_interval", + "in": "query", + "required": false, + "schema": { + "type": "number", + "default": 0.5, + "title": "Poll Interval" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "text/event-stream": { + "schema": { + "type": "string" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/chat/complete": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Chat with MemOS (Complete Response)", + "description": "Chat with MemOS for a specific user. Returns complete response (non-streaming).\n\nThis endpoint uses the class-based ChatHandler.", + "operationId": "chat_complete_product_chat_complete_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/APIChatCompleteRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": {} + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/chat/stream": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Chat with MemOS", + "description": "Chat with MemOS for a specific user. Returns SSE stream.\n\nThis endpoint uses the class-based ChatHandler which internally\ncomposes SearchHandler and AddHandler for a clean architecture.", + "operationId": "chat_stream_product_chat_stream_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChatRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "text/event-stream": { + "schema": { + "type": "string" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/chat/stream/playground": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Chat with MemOS playground", + "description": "Chat with MemOS for a specific user. Returns SSE stream.\n\nThis endpoint uses the class-based ChatHandler which internally\ncomposes SearchHandler and AddHandler for a clean architecture.", + "operationId": "chat_stream_playground_product_chat_stream_playground_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChatPlaygroundRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "text/event-stream": { + "schema": { + "type": "string" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/suggestions": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Get suggestion queries", + "description": "Get suggestion queries for a specific user with language preference.", + "operationId": "get_suggestion_queries_product_suggestions_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SuggestionRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SuggestionResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/get_all": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Get all memories for user", + "description": "Get all memories or subgraph for a specific user.\n\nIf search_query is provided, returns a subgraph based on the query.\nOtherwise, returns all memories of the specified type.", + "operationId": "get_all_memories_product_get_all_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryPlaygroundRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/get_memory": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Get memories for user", + "operationId": "get_memories_product_get_memory_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/get_memory/{memory_id}": { + "get": { + "tags": [ + "Server API" + ], + "summary": "Get memory by id", + "operationId": "get_memory_by_id_product_get_memory__memory_id__get", + "parameters": [ + { + "name": "memory_id", + "in": "path", + "required": true, + "schema": { + "type": "string", + "title": "Memory Id" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/get_memory_by_ids": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Get memory by ids", + "operationId": "get_memory_by_ids_product_get_memory_by_ids_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Memory Ids" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/delete_memory": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Delete memories for user", + "operationId": "delete_memories_product_delete_memory_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/DeleteMemoryRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/DeleteMemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/feedback": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Feedback memories", + "description": "Feedback memories for a specific user.\n\nThis endpoint uses the class-based FeedbackHandler for better code organization.", + "operationId": "feedback_memories_product_feedback_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/APIFeedbackRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/get_user_names_by_memory_ids": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Get user names by memory ids", + "description": "Get user names by memory ids. Now unified to query from graph_db only.", + "operationId": "get_user_names_by_memory_ids_product_get_user_names_by_memory_ids_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetUserNamesByMemoryIdsRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetUserNamesByMemoryIdsResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/exist_mem_cube_id": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Check if mem cube id exists", + "description": "(inner) Check if mem cube id exists.", + "operationId": "exist_mem_cube_id_product_exist_mem_cube_id_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExistMemCubeIdRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExistMemCubeIdResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/chat/stream/business_user": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Chat with MemOS for business user", + "description": "(inner) Chat with MemOS for a specific business user. Returns SSE stream.", + "operationId": "chat_stream_business_user_product_chat_stream_business_user_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChatBusinessRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "text/event-stream": { + "schema": { + "type": "string" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/delete_memory_by_record_id": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Delete memory by record id", + "description": "(inner) Delete memory nodes by mem_cube_id (user_name) and delete_record_id. Record id is inner field, just for delete and recover memory, not for user to set.", + "operationId": "delete_memory_by_record_id_product_delete_memory_by_record_id_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/DeleteMemoryByRecordIdRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/DeleteMemoryByRecordIdResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/recover_memory_by_record_id": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Recover memory by record id", + "description": "(inner) Recover memory nodes by mem_cube_id (user_name) and delete_record_id. Record id is inner field, just for delete and recover memory, not for user to set.", + "operationId": "recover_memory_by_record_id_product_recover_memory_by_record_id_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/RecoverMemoryByRecordIdRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/RecoverMemoryByRecordIdResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/product/get_memory_dashboard": { + "post": { + "tags": [ + "Server API" + ], + "summary": "Get memories for dashboard", + "operationId": "get_memories_dashboard_product_get_memory_dashboard_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryDashboardRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetMemoryResponse" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/health": { + "get": { + "summary": "Health Check", + "description": "Container and load balancer health endpoint.", + "operationId": "health_check_health_get", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": {} + } + } + } + } + } + } + }, + "components": { + "schemas": { + "APIADDRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID. If not provided, a default session will be used." + }, + "task_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Task Id", + "description": "Task ID for monitoring async tasks" + }, + "manager_user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Manager User Id", + "description": "Manager User ID" + }, + "project_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Project Id", + "description": "Project ID" + }, + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "List of cube IDs user can write for multi-cube add" + }, + "async_mode": { + "type": "string", + "enum": [ + "async", + "sync" + ], + "title": "Async Mode", + "description": "Whether to add memory in async mode. Use 'async' to enqueue background add (non-blocking), or 'sync' to add memories in the current call. Default: 'async'.", + "default": "async" + }, + "mode": { + "anyOf": [ + { + "type": "string", + "enum": [ + "fast", + "fine" + ] + }, + { + "type": "null" + } + ], + "title": "Mode", + "description": "(Internal) Add mode used only when async_mode='sync'. If set to 'fast', the handler will use a fast add pipeline. Ignored when async_mode='async'." + }, + "custom_tags": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Custom Tags", + "description": "Custom tags for this add request, e.g. ['Travel', 'family']. These tags can be used as filters in search." + }, + "info": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Info", + "description": "Additional metadata for the add request. All keys can be used as filters in search. Example: {'agent_id': 'xxxxxx', 'app_id': 'xxxx', 'source_type': 'web', 'source_url': 'https://www.baidu.com', 'source_content': '西湖是杭州最著名的景点'}." + }, + "messages": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/File" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Messages", + "description": "List of messages to store. Supports: - system / user / assistant messages with 'content' and 'chat_time'; - tool messages including: * tool_description (name, description, parameters), * tool_input (call_id, name, argument), * raw tool messages where content is str or list[str], * tool_output with structured output items (input_text / input_image / input_file, etc.). Also supports pure input items when there is no dialog." + }, + "chat_history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Chat History", + "description": "Historical chat messages used internally by algorithms. If None, internal stored history will be used; if provided (even an empty list), this value will be used as-is." + }, + "is_feedback": { + "type": "boolean", + "title": "Is Feedback", + "description": "Whether this request represents user feedback. Default: False.", + "default": false + }, + "is_upload_skill": { + "type": "boolean", + "title": "Is Upload Skill", + "description": "Whether this request is an upload skill request. When True, the messages field should contain file items with zip file download URLs for pre-built skill packages. Default: False.", + "default": false + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "(Deprecated) Target cube ID for this add request (optional for developer API)." + }, + "memory_content": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Memory Content", + "description": "(Deprecated) Plain memory content to store. Prefer using `messages`." + }, + "doc_path": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Doc Path", + "description": "(Deprecated / internal) Path to document to store." + }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Source", + "description": "(Deprecated) Simple source tag of the memory. Prefer using `info.source_type` / `info.source_url`." + }, + "operation": { + "anyOf": [ + { + "items": { + "$ref": "#/components/schemas/PermissionDict" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Operation", + "description": "(Internal) Operation definitions for multi-cube write permissions." + } + }, + "type": "object", + "title": "APIADDRequest", + "description": "Request model for creating memories.", + "example": { + "async_mode": "async", + "messages": [ + { + "content": "I am learning ggplot2 in R.", + "role": "user" + } + ], + "user_id": "8736b16e-1d20-4163-980b-a5063c3facdc", + "writable_cube_ids": [ + "b32d0977-435d-4828-a86f-4f47f8b55bca" + ] + } + }, + "APIChatCompleteRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "query": { + "type": "string", + "title": "Query", + "description": "Chat query message" + }, + "readable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Readable Cube Ids", + "description": "List of cube IDs user can read for multi-cube chat" + }, + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "List of cube IDs user can write for multi-cube chat" + }, + "history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "History", + "description": "Chat history" + }, + "mode": { + "$ref": "#/components/schemas/SearchMode", + "description": "search mode: fast, fine, or mixture", + "default": "fast" + }, + "system_prompt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "System Prompt", + "description": "Base system prompt to use for chat" + }, + "top_k": { + "type": "integer", + "title": "Top K", + "description": "Number of results to return", + "default": 10 + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID for soft-filtering memories" + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to handle preference memory", + "default": true + }, + "pref_top_k": { + "type": "integer", + "title": "Pref Top K", + "description": "Number of preference results to return", + "default": 6 + }, + "model_name_or_path": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model Name Or Path", + "description": "Model name to use for chat" + }, + "max_tokens": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Max Tokens", + "description": "Max tokens to generate" + }, + "temperature": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Temperature", + "description": "Temperature for sampling" + }, + "top_p": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Top P", + "description": "Top-p (nucleus) sampling parameter" + }, + "add_message_on_answer": { + "type": "boolean", + "title": "Add Message On Answer", + "description": "Add dialogs to memory after chat", + "default": true + }, + "manager_user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Manager User Id", + "description": "Manager User ID" + }, + "project_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Project Id", + "description": "Project ID" + }, + "relativity": { + "type": "number", + "minimum": 0.0, + "title": "Relativity", + "description": "Relevance threshold for recalled memories. Only memories with metadata.relativity >= relativity will be returned. Use 0 to disable threshold filtering. Default: 0.45.", + "default": 0.45 + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied to recalled memories. Supports nested \"and\" / \"or\" conditions and comparison operators such as \"gt\" for fields like \"id\", \"created_at\", or other indexed metadata. Example: {\"and\": [{\"id\": \"uuid-xxx\"}, {\"created_at\": {\"gt\": \"2024-01-01\"}}]}" + }, + "internet_search": { + "type": "boolean", + "title": "Internet Search", + "description": "Whether to use internet search", + "default": false + }, + "threshold": { + "type": "number", + "title": "Threshold", + "description": "Threshold for filtering references", + "default": 0.5 + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "Cube ID to use for chat" + }, + "moscube": { + "type": "boolean", + "title": "Moscube", + "description": "(Deprecated) Whether to use legacy MemOSCube pipeline", + "default": false + } + }, + "type": "object", + "required": [ + "user_id", + "query" + ], + "title": "APIChatCompleteRequest", + "description": "Request model for chat operations." + }, + "APIFeedbackRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID for soft-filtering memories", + "default": "default_session" + }, + "task_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Task Id", + "description": "Task ID for monitoring async tasks" + }, + "history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "History", + "description": "Chat history" + }, + "retrieved_memory_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Retrieved Memory Ids", + "description": "Retrieved memory ids at last turn" + }, + "feedback_content": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Feedback Content", + "description": "Feedback content to process" + }, + "feedback_time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Feedback Time", + "description": "Feedback time" + }, + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "List of cube IDs user can write for multi-cube add" + }, + "async_mode": { + "type": "string", + "enum": [ + "sync", + "async" + ], + "title": "Async Mode", + "description": "feedback mode: sync or async", + "default": "async" + }, + "corrected_answer": { + "type": "boolean", + "title": "Corrected Answer", + "description": "Whether need return corrected answer", + "default": false + }, + "info": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Info", + "description": "Additional metadata for the add request. All keys can be used as filters in search. Example: {'agent_id': 'xxxxxx', 'app_id': 'xxxx', 'source_type': 'web', 'source_url': 'https://www.baidu.com', 'source_content': 'West Lake is the most famous scenic spot in Hangzhou'}." + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "(Deprecated) Single cube ID to search in. Prefer `readable_cube_ids` for multi-cube search." + } + }, + "type": "object", + "required": [ + "user_id", + "history", + "feedback_content" + ], + "title": "APIFeedbackRequest", + "description": "Request model for processing feedback info." + }, + "APISearchRequest": { + "properties": { + "query": { + "type": "string", + "title": "Query", + "description": "User search query" + }, + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "readable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Readable Cube Ids", + "description": "List of cube IDs that are readable for this request. Required for algorithm-facing API; optional for developer-facing API." + }, + "mode": { + "$ref": "#/components/schemas/SearchMode", + "description": "Search mode: fast, fine, or mixture.", + "default": "fast" + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID used as a soft signal to prioritize more relevant memories. Only used for weighting, not as a hard filter." + }, + "top_k": { + "type": "integer", + "minimum": 1.0, + "title": "Top K", + "description": "Number of textual memories to retrieve (top-K). Default: 10.", + "default": 10 + }, + "relativity": { + "type": "number", + "minimum": 0.0, + "title": "Relativity", + "description": "Relevance threshold for recalled memories. Only memories with metadata.relativity >= relativity will be returned. Use 0 to disable threshold filtering. Default: 0.45.", + "default": 0.45 + }, + "dedup": { + "anyOf": [ + { + "type": "string", + "enum": [ + "no", + "sim", + "mmr" + ] + }, + { + "type": "null" + } + ], + "title": "Dedup", + "description": "Optional dedup option for textual memories. Use 'no' for no dedup, 'sim' for similarity dedup, 'mmr' for MMR-based dedup. If None, default exact-text dedup is applied.", + "default": "mmr" + }, + "rerank": { + "type": "boolean", + "title": "Rerank", + "description": "Whether to apply the textual memory reranker during search. Set false to return retrieval-order candidates before post-search dedup/formatting.", + "default": true + }, + "pref_top_k": { + "type": "integer", + "minimum": 0.0, + "title": "Pref Top K", + "description": "Number of preference memories to retrieve (top-K). Default: 6.", + "default": 6 + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to retrieve preference memories along with general memories. If enabled, the system will automatically recall user preferences relevant to the query. Default: True.", + "default": true + }, + "search_tool_memory": { + "type": "boolean", + "title": "Search Tool Memory", + "description": "Whether to retrieve tool memories along with general memories. If enabled, the system will automatically recall tool memories relevant to the query. Default: True.", + "default": true + }, + "tool_mem_top_k": { + "type": "integer", + "minimum": 0.0, + "title": "Tool Mem Top K", + "description": "Number of tool memories to retrieve (top-K). Default: 6.", + "default": 6 + }, + "include_skill_memory": { + "type": "boolean", + "title": "Include Skill Memory", + "description": "Whether to retrieve skill memories along with general memories. If enabled, the system will automatically recall skill memories relevant to the query. Default: True.", + "default": true + }, + "skill_mem_top_k": { + "type": "integer", + "minimum": 0.0, + "title": "Skill Mem Top K", + "description": "Number of skill memories to retrieve (top-K). Default: 3.", + "default": 3 + }, + "context_format": { + "type": "string", + "title": "Context Format", + "description": "Optional search context format passed through to installed plugins.", + "default": "memory" + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied to memory search results. Supports nested \"and\" / \"or\" conditions and comparison operators such as \"gt\" for fields like \"id\", \"created_at\", or other indexed metadata. Example: {\"and\": [{\"id\": \"uuid-xxx\"}, {\"created_at\": {\"gt\": \"2024-01-01\"}}]}" + }, + "internet_search": { + "type": "boolean", + "title": "Internet Search", + "description": "Whether to enable internet search in addition to memory search. Primarily used by internal algorithms. Default: False.", + "default": false + }, + "threshold": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Threshold", + "description": "Internal similarity threshold for searching plaintext memories. If None, default thresholds will be applied." + }, + "search_memory_type": { + "type": "string", + "title": "Search Memory Type", + "description": "Memory category to search. Supported values include All, WorkingMemory, LongTermMemory, UserMemory, OuterMemory, ToolSchemaMemory, ToolTrajectoryMemory, RawFileMemory, AllSummaryMemory, SkillMemory, and PreferenceMemory. Default: All.", + "default": "All" + }, + "reference_time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Reference Time", + "description": "Optional reference time for time-sensitive search parsing. If omitted, search uses the current server time." + }, + "chat_history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Chat History", + "description": "Historical chat messages used internally by algorithms. If None, internal stored history may be used; if provided (even an empty list), this value will be used as-is." + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "(Deprecated) Single cube ID to search in. Prefer `readable_cube_ids` for multi-cube search." + }, + "moscube": { + "type": "boolean", + "title": "Moscube", + "description": "(Deprecated / internal) Whether to use legacy MemOSCube path.", + "default": false + }, + "operation": { + "anyOf": [ + { + "items": { + "$ref": "#/components/schemas/PermissionDict" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Operation", + "description": "(Internal) Operation definitions for multi-cube read permissions." + }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Source", + "description": "Origin of the search request. Plugin callers can use this value to route different search behavior without changing the query payload." + }, + "neighbor_discovery": { + "type": "boolean", + "title": "Neighbor Discovery", + "description": "Whether to enable neighbor discovery. If enabled, the system will automatically recall neighbor chunks relevant to the query. Default: False.", + "default": false + } + }, + "type": "object", + "required": [ + "query", + "user_id" + ], + "title": "APISearchRequest", + "description": "Request model for searching memories." + }, + "AllStatusResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "default": "Scheduler status summary retrieved successfully" + }, + "data": { + "anyOf": [ + { + "$ref": "#/components/schemas/AllStatusResponseData" + }, + { + "type": "null" + } + ], + "description": "Response data" + } + }, + "type": "object", + "title": "AllStatusResponse", + "description": "Response model for full scheduler status operations." + }, + "AllStatusResponseData": { + "properties": { + "scheduler_summary": { + "$ref": "#/components/schemas/TaskSummary", + "description": "Aggregated status for scheduler-managed tasks" + }, + "all_tasks_summary": { + "$ref": "#/components/schemas/TaskSummary", + "description": "Aggregated status for all tracked tasks" + } + }, + "type": "object", + "required": [ + "scheduler_summary", + "all_tasks_summary" + ], + "title": "AllStatusResponseData", + "description": "Aggregated scheduler status metrics." + }, + "Audio": { + "properties": { + "id": { + "type": "string", + "title": "Id" + } + }, + "type": "object", + "required": [ + "id" + ], + "title": "Audio" + }, + "ChatBusinessRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "query": { + "type": "string", + "title": "Query", + "description": "Chat query message" + }, + "readable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Readable Cube Ids", + "description": "List of cube IDs user can read for multi-cube chat" + }, + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "List of cube IDs user can write for multi-cube chat" + }, + "history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "History", + "description": "Chat history" + }, + "mode": { + "$ref": "#/components/schemas/SearchMode", + "description": "search mode: fast, fine, or mixture", + "default": "fast" + }, + "system_prompt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "System Prompt", + "description": "Base system prompt to use for chat" + }, + "top_k": { + "type": "integer", + "title": "Top K", + "description": "Number of results to return", + "default": 10 + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID for soft-filtering memories" + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to handle preference memory", + "default": true + }, + "pref_top_k": { + "type": "integer", + "title": "Pref Top K", + "description": "Number of preference results to return", + "default": 6 + }, + "model_name_or_path": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model Name Or Path", + "description": "Model name to use for chat" + }, + "max_tokens": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Max Tokens", + "description": "Max tokens to generate" + }, + "temperature": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Temperature", + "description": "Temperature for sampling" + }, + "top_p": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Top P", + "description": "Top-p (nucleus) sampling parameter" + }, + "add_message_on_answer": { + "type": "boolean", + "title": "Add Message On Answer", + "description": "Add dialogs to memory after chat", + "default": true + }, + "manager_user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Manager User Id", + "description": "Manager User ID" + }, + "project_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Project Id", + "description": "Project ID" + }, + "relativity": { + "type": "number", + "minimum": 0.0, + "title": "Relativity", + "description": "Relevance threshold for recalled memories. Only memories with metadata.relativity >= relativity will be returned. Use 0 to disable threshold filtering. Default: 0.45.", + "default": 0.45 + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied to recalled memories. Supports nested \"and\" / \"or\" conditions and comparison operators such as \"gt\" for fields like \"id\", \"created_at\", or other indexed metadata. Example: {\"and\": [{\"id\": \"uuid-xxx\"}, {\"created_at\": {\"gt\": \"2024-01-01\"}}]}" + }, + "internet_search": { + "type": "boolean", + "title": "Internet Search", + "description": "Whether to use internet search", + "default": false + }, + "threshold": { + "type": "number", + "title": "Threshold", + "description": "Threshold for filtering references", + "default": 0.5 + }, + "moscube": { + "type": "boolean", + "title": "Moscube", + "description": "(Deprecated) Whether to use legacy MemOSCube pipeline.", + "default": false + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "(Deprecated) Single cube ID to use for chat. Prefer `readable_cube_ids` / `writable_cube_ids` for multi-cube chat." + }, + "business_key": { + "type": "string", + "title": "Business Key", + "description": "Business User Key" + }, + "need_search": { + "type": "boolean", + "title": "Need Search", + "description": "Whether to need search before chat", + "default": false + } + }, + "type": "object", + "required": [ + "user_id", + "query", + "business_key" + ], + "title": "ChatBusinessRequest", + "description": "Request model for chat operations for business user." + }, + "ChatCompletionAssistantMessageParam": { + "properties": { + "role": { + "type": "string", + "const": "assistant", + "title": "Role" + }, + "audio": { + "anyOf": [ + { + "$ref": "#/components/schemas/Audio" + }, + { + "type": "null" + } + ] + }, + "content": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartRefusalParam" + } + ] + }, + "type": "array" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartRefusalParam" + }, + { + "type": "null" + } + ], + "title": "Content" + }, + "refusal": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Refusal" + }, + "tool_calls": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionMessageFunctionToolCallParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionMessageCustomToolCallParam" + } + ] + }, + "type": "array" + }, + { + "$ref": "#/components/schemas/ChatCompletionMessageFunctionToolCallParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionMessageCustomToolCallParam" + } + ], + "title": "Tool Calls" + }, + "chat_time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Chat Time" + }, + "message_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Message Id" + } + }, + "type": "object", + "required": [ + "role" + ], + "title": "ChatCompletionAssistantMessageParam" + }, + "ChatCompletionContentPartImageParam": { + "properties": { + "image_url": { + "$ref": "#/components/schemas/ImageURL" + }, + "type": { + "type": "string", + "const": "image_url", + "title": "Type" + } + }, + "type": "object", + "required": [ + "image_url", + "type" + ], + "title": "ChatCompletionContentPartImageParam" + }, + "ChatCompletionContentPartInputAudioParam": { + "properties": { + "input_audio": { + "$ref": "#/components/schemas/InputAudio" + }, + "type": { + "type": "string", + "const": "input_audio", + "title": "Type" + } + }, + "type": "object", + "required": [ + "input_audio", + "type" + ], + "title": "ChatCompletionContentPartInputAudioParam" + }, + "ChatCompletionContentPartRefusalParam": { + "properties": { + "refusal": { + "type": "string", + "title": "Refusal" + }, + "type": { + "type": "string", + "const": "refusal", + "title": "Type" + } + }, + "type": "object", + "required": [ + "refusal", + "type" + ], + "title": "ChatCompletionContentPartRefusalParam" + }, + "ChatCompletionContentPartTextParam": { + "properties": { + "text": { + "type": "string", + "title": "Text" + }, + "type": { + "type": "string", + "const": "text", + "title": "Type" + } + }, + "type": "object", + "required": [ + "text", + "type" + ], + "title": "ChatCompletionContentPartTextParam" + }, + "ChatCompletionMessageCustomToolCallParam": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "custom": { + "$ref": "#/components/schemas/Custom" + }, + "type": { + "type": "string", + "const": "custom", + "title": "Type" + } + }, + "type": "object", + "required": [ + "id", + "custom", + "type" + ], + "title": "ChatCompletionMessageCustomToolCallParam" + }, + "ChatCompletionMessageFunctionToolCallParam": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "function": { + "$ref": "#/components/schemas/Function" + }, + "type": { + "type": "string", + "const": "function", + "title": "Type" + } + }, + "type": "object", + "required": [ + "id", + "function", + "type" + ], + "title": "ChatCompletionMessageFunctionToolCallParam" + }, + "ChatCompletionSystemMessageParam": { + "properties": { + "content": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + "type": "array" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + } + ], + "title": "Content" + }, + "role": { + "type": "string", + "const": "system", + "title": "Role" + }, + "name": { + "type": "string", + "title": "Name" + }, + "chat_time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Chat Time" + }, + "message_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Message Id" + } + }, + "type": "object", + "required": [ + "content", + "role" + ], + "title": "ChatCompletionSystemMessageParam" + }, + "ChatCompletionToolMessageParam": { + "properties": { + "content": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartImageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartInputAudioParam" + }, + { + "$ref": "#/components/schemas/File" + } + ] + }, + "type": "array" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartImageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartInputAudioParam" + }, + { + "$ref": "#/components/schemas/File" + } + ], + "title": "Content" + }, + "role": { + "type": "string", + "const": "tool", + "title": "Role" + }, + "tool_call_id": { + "type": "string", + "title": "Tool Call Id" + }, + "chat_time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Chat Time" + }, + "message_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Message Id" + } + }, + "type": "object", + "required": [ + "content", + "role", + "tool_call_id" + ], + "title": "ChatCompletionToolMessageParam" + }, + "ChatCompletionUserMessageParam": { + "properties": { + "content": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartImageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartInputAudioParam" + }, + { + "$ref": "#/components/schemas/File" + } + ] + }, + "type": "array" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartImageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionContentPartInputAudioParam" + }, + { + "$ref": "#/components/schemas/File" + } + ], + "title": "Content" + }, + "role": { + "type": "string", + "const": "user", + "title": "Role" + }, + "name": { + "type": "string", + "title": "Name" + }, + "chat_time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Chat Time" + }, + "message_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Message Id" + } + }, + "type": "object", + "required": [ + "content", + "role" + ], + "title": "ChatCompletionUserMessageParam" + }, + "ChatPlaygroundRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "query": { + "type": "string", + "title": "Query", + "description": "Chat query message" + }, + "readable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Readable Cube Ids", + "description": "List of cube IDs user can read for multi-cube chat" + }, + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "List of cube IDs user can write for multi-cube chat" + }, + "history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "History", + "description": "Chat history" + }, + "mode": { + "$ref": "#/components/schemas/SearchMode", + "description": "search mode: fast, fine, or mixture", + "default": "fast" + }, + "system_prompt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "System Prompt", + "description": "Base system prompt to use for chat" + }, + "top_k": { + "type": "integer", + "title": "Top K", + "description": "Number of results to return", + "default": 10 + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID for soft-filtering memories" + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to handle preference memory", + "default": true + }, + "pref_top_k": { + "type": "integer", + "title": "Pref Top K", + "description": "Number of preference results to return", + "default": 6 + }, + "model_name_or_path": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model Name Or Path", + "description": "Model name to use for chat" + }, + "max_tokens": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Max Tokens", + "description": "Max tokens to generate" + }, + "temperature": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Temperature", + "description": "Temperature for sampling" + }, + "top_p": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Top P", + "description": "Top-p (nucleus) sampling parameter" + }, + "add_message_on_answer": { + "type": "boolean", + "title": "Add Message On Answer", + "description": "Add dialogs to memory after chat", + "default": true + }, + "manager_user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Manager User Id", + "description": "Manager User ID" + }, + "project_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Project Id", + "description": "Project ID" + }, + "relativity": { + "type": "number", + "minimum": 0.0, + "title": "Relativity", + "description": "Relevance threshold for recalled memories. Only memories with metadata.relativity >= relativity will be returned. Use 0 to disable threshold filtering. Default: 0.45.", + "default": 0.45 + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied to recalled memories. Supports nested \"and\" / \"or\" conditions and comparison operators such as \"gt\" for fields like \"id\", \"created_at\", or other indexed metadata. Example: {\"and\": [{\"id\": \"uuid-xxx\"}, {\"created_at\": {\"gt\": \"2024-01-01\"}}]}" + }, + "internet_search": { + "type": "boolean", + "title": "Internet Search", + "description": "Whether to use internet search", + "default": false + }, + "threshold": { + "type": "number", + "title": "Threshold", + "description": "Threshold for filtering references", + "default": 0.5 + }, + "moscube": { + "type": "boolean", + "title": "Moscube", + "description": "(Deprecated) Whether to use legacy MemOSCube pipeline.", + "default": false + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "(Deprecated) Single cube ID to use for chat. Prefer `readable_cube_ids` / `writable_cube_ids` for multi-cube chat." + }, + "beginner_guide_step": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Beginner Guide Step", + "description": "Whether to use beginner guide, option: [first, second]" + } + }, + "type": "object", + "required": [ + "user_id", + "query" + ], + "title": "ChatPlaygroundRequest", + "description": "Request model for chat operations in playground." + }, + "ChatRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "query": { + "type": "string", + "title": "Query", + "description": "Chat query message" + }, + "readable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Readable Cube Ids", + "description": "List of cube IDs user can read for multi-cube chat" + }, + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "List of cube IDs user can write for multi-cube chat" + }, + "history": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "History", + "description": "Chat history" + }, + "mode": { + "$ref": "#/components/schemas/SearchMode", + "description": "search mode: fast, fine, or mixture", + "default": "fast" + }, + "system_prompt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "System Prompt", + "description": "Base system prompt to use for chat" + }, + "top_k": { + "type": "integer", + "title": "Top K", + "description": "Number of results to return", + "default": 10 + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Session ID for soft-filtering memories" + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to handle preference memory", + "default": true + }, + "pref_top_k": { + "type": "integer", + "title": "Pref Top K", + "description": "Number of preference results to return", + "default": 6 + }, + "model_name_or_path": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model Name Or Path", + "description": "Model name to use for chat" + }, + "max_tokens": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Max Tokens", + "description": "Max tokens to generate" + }, + "temperature": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Temperature", + "description": "Temperature for sampling" + }, + "top_p": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Top P", + "description": "Top-p (nucleus) sampling parameter" + }, + "add_message_on_answer": { + "type": "boolean", + "title": "Add Message On Answer", + "description": "Add dialogs to memory after chat", + "default": true + }, + "manager_user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Manager User Id", + "description": "Manager User ID" + }, + "project_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Project Id", + "description": "Project ID" + }, + "relativity": { + "type": "number", + "minimum": 0.0, + "title": "Relativity", + "description": "Relevance threshold for recalled memories. Only memories with metadata.relativity >= relativity will be returned. Use 0 to disable threshold filtering. Default: 0.45.", + "default": 0.45 + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied to recalled memories. Supports nested \"and\" / \"or\" conditions and comparison operators such as \"gt\" for fields like \"id\", \"created_at\", or other indexed metadata. Example: {\"and\": [{\"id\": \"uuid-xxx\"}, {\"created_at\": {\"gt\": \"2024-01-01\"}}]}" + }, + "internet_search": { + "type": "boolean", + "title": "Internet Search", + "description": "Whether to use internet search", + "default": false + }, + "threshold": { + "type": "number", + "title": "Threshold", + "description": "Threshold for filtering references", + "default": 0.5 + }, + "moscube": { + "type": "boolean", + "title": "Moscube", + "description": "(Deprecated) Whether to use legacy MemOSCube pipeline.", + "default": false + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "(Deprecated) Single cube ID to use for chat. Prefer `readable_cube_ids` / `writable_cube_ids` for multi-cube chat." + } + }, + "type": "object", + "required": [ + "user_id", + "query" + ], + "title": "ChatRequest", + "description": "Request model for chat operations.\n\nThis model is used as the algorithm-facing chat interface, while also\nremaining backward compatible with older developer-facing APIs." + }, + "CreateCubeRequest": { + "properties": { + "cube_name": { + "type": "string", + "title": "Cube Name", + "description": "Human-readable name for the memory cube" + }, + "owner_id": { + "type": "string", + "title": "Owner Id", + "description": "User ID of the cube owner" + }, + "cube_path": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Cube Path", + "description": "File system path where cube data will be stored" + }, + "cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Cube Id", + "description": "Custom unique identifier for the cube. If not provided, one will be generated. Note: cube_id is also referred to as mem_cube_id throughout the API." + } + }, + "type": "object", + "required": [ + "cube_name", + "owner_id" + ], + "title": "CreateCubeRequest", + "description": "Request model for creating a new memory cube." + }, + "CreateCubeResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "default": "Cube created successfully" + }, + "data": { + "anyOf": [ + { + "$ref": "#/components/schemas/CreateCubeResponseData" + }, + { + "type": "null" + } + ], + "description": "Response data" + } + }, + "type": "object", + "title": "CreateCubeResponse", + "description": "Response model for cube creation." + }, + "CreateCubeResponseData": { + "properties": { + "cube_id": { + "type": "string", + "title": "Cube Id", + "description": "The created cube ID (also called mem_cube_id)" + }, + "cube_name": { + "type": "string", + "title": "Cube Name", + "description": "Name of the created cube" + }, + "owner_id": { + "type": "string", + "title": "Owner Id", + "description": "Owner user ID" + } + }, + "type": "object", + "required": [ + "cube_id", + "cube_name", + "owner_id" + ], + "title": "CreateCubeResponseData", + "description": "Data model for cube creation response." + }, + "Custom": { + "properties": { + "input": { + "type": "string", + "title": "Input" + }, + "name": { + "type": "string", + "title": "Name" + } + }, + "type": "object", + "required": [ + "input", + "name" + ], + "title": "Custom" + }, + "DeleteMemoryByRecordIdRequest": { + "properties": { + "mem_cube_id": { + "type": "string", + "title": "Mem Cube Id", + "description": "Mem cube ID" + }, + "record_id": { + "type": "string", + "title": "Record Id", + "description": "Record ID" + }, + "hard_delete": { + "type": "boolean", + "title": "Hard Delete", + "description": "Hard delete", + "default": false + } + }, + "type": "object", + "required": [ + "mem_cube_id", + "record_id" + ], + "title": "DeleteMemoryByRecordIdRequest", + "description": "Request model for deleting memory by record id." + }, + "DeleteMemoryByRecordIdResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "DeleteMemoryByRecordIdResponse", + "description": "Response model for deleting memory by record id." + }, + "DeleteMemoryRequest": { + "properties": { + "writable_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Writable Cube Ids", + "description": "Writable cube IDs" + }, + "memory_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Memory Ids", + "description": "Memory IDs" + }, + "file_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "File Ids", + "description": "File IDs" + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied when selecting memories for deletion." + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id", + "description": "Quick delete condition: remove memories for this user_id." + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id", + "description": "Quick delete condition: remove memories for this session_id." + }, + "conversation_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Conversation Id", + "description": "Alias of session_id for backward compatibility." + }, + "auto_cleanup_working": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Auto Cleanup Working", + "description": "(Internal) Whether to automatically delete related WorkingMemory nodes based on working_binding metadata when deleting by memory_ids.", + "default": false + } + }, + "type": "object", + "title": "DeleteMemoryRequest", + "description": "Request model for deleting memories." + }, + "DeleteMemoryResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "DeleteMemoryResponse", + "description": "Response model for deleting memories." + }, + "ExistMemCubeIdRequest": { + "properties": { + "mem_cube_id": { + "type": "string", + "title": "Mem Cube Id", + "description": "Mem cube ID" + } + }, + "type": "object", + "required": [ + "mem_cube_id" + ], + "title": "ExistMemCubeIdRequest", + "description": "Request model for checking if mem cube id exists." + }, + "ExistMemCubeIdResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": { + "type": "boolean" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "ExistMemCubeIdResponse", + "description": "Response model for checking if mem cube id exists." + }, + "File": { + "properties": { + "file": { + "$ref": "#/components/schemas/FileFile" + }, + "type": { + "type": "string", + "const": "file", + "title": "Type" + } + }, + "type": "object", + "required": [ + "file", + "type" + ], + "title": "File" + }, + "FileFile": { + "properties": { + "file_data": { + "type": "string", + "title": "File Data" + }, + "file_id": { + "type": "string", + "title": "File Id" + }, + "filename": { + "type": "string", + "title": "Filename" + } + }, + "type": "object", + "title": "FileFile" + }, + "Function": { + "properties": { + "arguments": { + "type": "string", + "title": "Arguments" + }, + "name": { + "type": "string", + "title": "Name" + } + }, + "type": "object", + "required": [ + "arguments", + "name" + ], + "title": "Function" + }, + "GetMemoryDashboardRequest": { + "properties": { + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "Cube ID" + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id", + "description": "User ID" + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to return preference memory", + "default": true + }, + "include_tool_memory": { + "type": "boolean", + "title": "Include Tool Memory", + "description": "Whether to return tool memory", + "default": true + }, + "include_skill_memory": { + "type": "boolean", + "title": "Include Skill Memory", + "description": "Whether to return skill memory", + "default": true + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied when retrieving memories." + }, + "page": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Page", + "description": "Page number (starts from 1). If None, exports all data without pagination." + }, + "page_size": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Page Size", + "description": "Number of items per page. If None, exports all data without pagination." + }, + "all_cubes": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "All Cubes", + "description": "Optional range: omitted/null or false selects one Cube (configured default when mem_cube_id is omitted); true selects all Cubes. Legacy user scope filters explicitly select all Cubes except their specified Cube. Supported by openGauss." + }, + "exclude_cube_id": { + "anyOf": [ + { + "type": "string", + "minLength": 1, + "pattern": "\\S" + }, + { + "type": "null" + } + ], + "title": "Exclude Cube Id", + "description": "Optionally exclude this exact Cube from the final all-Cube range. Supported by openGauss." + } + }, + "type": "object", + "title": "GetMemoryDashboardRequest", + "description": "Request model for getting memories for dashboard." + }, + "GetMemoryPlaygroundRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "memory_type": { + "type": "string", + "enum": [ + "text_mem", + "act_mem", + "param_mem", + "para_mem" + ], + "title": "Memory Type", + "description": "Memory type" + }, + "mem_cube_ids": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Ids", + "description": "Cube IDs" + }, + "search_query": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Search Query", + "description": "Search query" + }, + "search_type": { + "type": "string", + "enum": [ + "embedding", + "fulltext" + ], + "title": "Search Type", + "description": "Search type", + "default": "fulltext" + } + }, + "type": "object", + "required": [ + "user_id", + "memory_type" + ], + "title": "GetMemoryPlaygroundRequest", + "description": "Request model for getting memories." + }, + "GetMemoryRequest": { + "properties": { + "mem_cube_id": { + "type": "string", + "title": "Mem Cube Id", + "description": "Cube ID" + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id", + "description": "User ID" + }, + "include_preference": { + "type": "boolean", + "title": "Include Preference", + "description": "Whether to return preference memory", + "default": true + }, + "include_tool_memory": { + "type": "boolean", + "title": "Include Tool Memory", + "description": "Whether to return tool memory", + "default": true + }, + "include_skill_memory": { + "type": "boolean", + "title": "Include Skill Memory", + "description": "Whether to return skill memory", + "default": true + }, + "filter": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Filter", + "description": "Optional metadata filter applied when retrieving memories." + }, + "page": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Page", + "description": "Page number (starts from 1). If None, exports all data without pagination." + }, + "page_size": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Page Size", + "description": "Number of items per page. If None, exports all data without pagination." + } + }, + "type": "object", + "required": [ + "mem_cube_id" + ], + "title": "GetMemoryRequest", + "description": "Request model for getting memories." + }, + "GetMemoryResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "GetMemoryResponse", + "description": "Response model for getting memories." + }, + "GetUserNamesByMemoryIdsRequest": { + "properties": { + "memory_ids": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Memory Ids", + "description": "Memory IDs" + } + }, + "type": "object", + "required": [ + "memory_ids" + ], + "title": "GetUserNamesByMemoryIdsRequest", + "description": "Request model for getting user names by memory ids." + }, + "GetUserNamesByMemoryIdsResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "GetUserNamesByMemoryIdsResponse", + "description": "Response model for getting user names by memory ids." + }, + "HTTPValidationError": { + "properties": { + "detail": { + "items": { + "$ref": "#/components/schemas/ValidationError" + }, + "type": "array", + "title": "Detail" + } + }, + "type": "object", + "title": "HTTPValidationError" + }, + "ImageURL": { + "properties": { + "url": { + "type": "string", + "title": "Url" + }, + "detail": { + "type": "string", + "enum": [ + "auto", + "low", + "high" + ], + "title": "Detail" + }, + "image_id": { + "type": "string", + "title": "Image Id" + } + }, + "type": "object", + "required": [ + "url" + ], + "title": "ImageURL" + }, + "InputAudio": { + "properties": { + "data": { + "type": "string", + "title": "Data" + }, + "format": { + "type": "string", + "enum": [ + "wav", + "mp3" + ], + "title": "Format" + } + }, + "type": "object", + "required": [ + "data", + "format" + ], + "title": "InputAudio" + }, + "MemoryResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "items": {}, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "MemoryResponse", + "description": "Response model for memory operations." + }, + "PermissionDict": { + "properties": { + "permissions": { + "items": { + "type": "string", + "enum": [ + "read", + "write", + "delete", + "execute" + ] + }, + "type": "array", + "title": "Permissions" + }, + "mem_cube_id": { + "type": "string", + "title": "Mem Cube Id" + } + }, + "type": "object", + "title": "PermissionDict", + "description": "Typed dictionary for chat message dictionaries." + }, + "RecoverMemoryByRecordIdRequest": { + "properties": { + "mem_cube_id": { + "type": "string", + "title": "Mem Cube Id", + "description": "Mem cube ID" + }, + "delete_record_id": { + "type": "string", + "title": "Delete Record Id", + "description": "Delete record ID" + } + }, + "type": "object", + "required": [ + "mem_cube_id", + "delete_record_id" + ], + "title": "RecoverMemoryByRecordIdRequest", + "description": "Request model for recovering memory by record id." + }, + "RecoverMemoryByRecordIdResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "RecoverMemoryByRecordIdResponse", + "description": "Response model for recovering memory by record id." + }, + "RegisterCubeRequest": { + "properties": { + "mem_cube_name_or_path": { + "type": "string", + "title": "Mem Cube Name Or Path", + "description": "File path to the memory cube or name for a new cube" + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "Custom identifier for the cube (also called cube_id). If not provided, one will be generated." + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id", + "description": "User ID to associate with the cube. If not provided, uses default user" + } + }, + "type": "object", + "required": [ + "mem_cube_name_or_path" + ], + "title": "RegisterCubeRequest", + "description": "Request model for registering an existing memory cube.\n\nThis allows loading a cube from disk or creating a new one if the path doesn't exist." + }, + "RegisterCubeResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "default": "Cube registered successfully" + }, + "data": { + "anyOf": [ + { + "$ref": "#/components/schemas/RegisterCubeResponseData" + }, + { + "type": "null" + } + ], + "description": "Response data" + } + }, + "type": "object", + "title": "RegisterCubeResponse", + "description": "Response model for cube registration." + }, + "RegisterCubeResponseData": { + "properties": { + "cube_id": { + "type": "string", + "title": "Cube Id", + "description": "The registered cube ID (also called mem_cube_id)" + }, + "cube_name_or_path": { + "type": "string", + "title": "Cube Name Or Path", + "description": "Name or path of the registered cube" + } + }, + "type": "object", + "required": [ + "cube_id", + "cube_name_or_path" + ], + "title": "RegisterCubeResponseData", + "description": "Data model for cube registration response." + }, + "SearchMode": { + "type": "string", + "enum": [ + "fast", + "fine", + "mixture" + ], + "title": "SearchMode", + "description": "Enumeration for search modes." + }, + "SearchResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "SearchResponse", + "description": "Response model for search operations." + }, + "StatusResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "default": "Memory get status successfully" + }, + "data": { + "anyOf": [ + { + "items": { + "$ref": "#/components/schemas/StatusResponseItem" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "title": "StatusResponse", + "description": "Response model for scheduler status operations." + }, + "StatusResponseItem": { + "properties": { + "task_id": { + "type": "string", + "title": "Task Id", + "description": "The ID of the task" + }, + "status": { + "type": "string", + "enum": [ + "in_progress", + "completed", + "waiting", + "failed", + "cancelled" + ], + "title": "Status", + "description": "The current status of the task" + } + }, + "type": "object", + "required": [ + "task_id", + "status" + ], + "title": "StatusResponseItem", + "description": "Individual task status item." + }, + "SuggestionRequest": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID" + }, + "mem_cube_id": { + "type": "string", + "title": "Mem Cube Id", + "description": "Cube ID" + }, + "language": { + "type": "string", + "enum": [ + "zh", + "en" + ], + "title": "Language", + "description": "Language for suggestions", + "default": "zh" + }, + "message": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionSystemMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionUserMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionAssistantMessageParam" + }, + { + "$ref": "#/components/schemas/ChatCompletionToolMessageParam" + } + ] + }, + "type": "array" + }, + { + "items": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatCompletionContentPartTextParam" + }, + { + "$ref": "#/components/schemas/File" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Message", + "description": "List of messages to store." + } + }, + "type": "object", + "required": [ + "user_id", + "mem_cube_id" + ], + "title": "SuggestionRequest", + "description": "Request model for getting suggestion queries." + }, + "SuggestionResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "description": "Response message" + }, + "data": { + "anyOf": [ + { + "additionalProperties": { + "items": { + "type": "string" + }, + "type": "array" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "Response data" + } + }, + "type": "object", + "required": [ + "message" + ], + "title": "SuggestionResponse", + "description": "Response model for suggestion operations." + }, + "TaskQueueData": { + "properties": { + "user_id": { + "type": "string", + "title": "User Id", + "description": "User ID the query is scoped to" + }, + "user_name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Name", + "description": "User name if available" + }, + "mem_cube_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Mem Cube Id", + "description": "MemCube ID if a single cube is targeted; otherwise None" + }, + "stream_keys": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Stream Keys", + "description": "Matched Redis stream keys for this user" + }, + "users_count": { + "type": "integer", + "title": "Users Count", + "description": "Distinct users currently present in queue streams" + }, + "pending_tasks_count": { + "type": "integer", + "title": "Pending Tasks Count", + "description": "Count of pending (delivered, not acked) tasks" + }, + "remaining_tasks_count": { + "type": "integer", + "title": "Remaining Tasks Count", + "description": "Count of enqueued tasks (xlen)" + }, + "pending_tasks_detail": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Pending Tasks Detail", + "description": "Per-stream pending counts, formatted as '{stream_key}:{count}'" + }, + "remaining_tasks_detail": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Remaining Tasks Detail", + "description": "Per-stream remaining counts, formatted as '{stream_key}:{count}'" + } + }, + "type": "object", + "required": [ + "user_id", + "stream_keys", + "users_count", + "pending_tasks_count", + "remaining_tasks_count", + "pending_tasks_detail", + "remaining_tasks_detail" + ], + "title": "TaskQueueData", + "description": "Queue-level metrics for scheduler tasks." + }, + "TaskQueueResponse": { + "properties": { + "code": { + "type": "integer", + "title": "Code", + "description": "Response status code", + "default": 200 + }, + "message": { + "type": "string", + "title": "Message", + "default": "Scheduler task queue status retrieved successfully" + }, + "data": { + "anyOf": [ + { + "$ref": "#/components/schemas/TaskQueueData" + }, + { + "type": "null" + } + ], + "description": "Response data" + } + }, + "type": "object", + "title": "TaskQueueResponse", + "description": "Response model for scheduler task queue status." + }, + "TaskSummary": { + "properties": { + "waiting": { + "type": "integer", + "title": "Waiting", + "description": "Number of tasks waiting to run", + "default": 0 + }, + "in_progress": { + "type": "integer", + "title": "In Progress", + "description": "Number of tasks currently running", + "default": 0 + }, + "pending": { + "type": "integer", + "title": "Pending", + "description": "Number of tasks fetched by workers but not yet acknowledged", + "default": 0 + }, + "completed": { + "type": "integer", + "title": "Completed", + "description": "Number of tasks completed", + "default": 0 + }, + "failed": { + "type": "integer", + "title": "Failed", + "description": "Number of tasks failed", + "default": 0 + }, + "cancelled": { + "type": "integer", + "title": "Cancelled", + "description": "Number of tasks cancelled", + "default": 0 + }, + "total": { + "type": "integer", + "title": "Total", + "description": "Total number of tasks counted", + "default": 0 + } + }, + "type": "object", + "title": "TaskSummary", + "description": "Aggregated counts of tasks by status." + }, + "ValidationError": { + "properties": { + "loc": { + "items": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + } + ] + }, + "type": "array", + "title": "Location" + }, + "msg": { + "type": "string", + "title": "Message" + }, + "type": { + "type": "string", + "title": "Error Type" + } + }, + "type": "object", + "required": [ + "loc", + "msg", + "type" + ], + "title": "ValidationError" + } + } + } +} diff --git a/src/memos/api/config.py b/src/memos/api/config.py index f09e51335..2e08d1fde 100644 --- a/src/memos/api/config.py +++ b/src/memos/api/config.py @@ -926,6 +926,44 @@ def get_postgres_config(user_id: str | None = None) -> dict[str, Any]: "maxconn": int(os.getenv("POSTGRES_MAX_CONN", "20")), } + @staticmethod + def get_opengauss_config(user_id: str | None = None) -> dict[str, Any]: + """Read the selected openGauss connection without development credentials.""" + user_name = os.getenv("MEMOS_USER_NAME", "default") + if user_id: + user_name = f"memos_{user_id.replace('-', '')}" + return { + "host": os.getenv("OPENGAUSS_HOST", ""), + "port": int(os.getenv("OPENGAUSS_PORT", "5432")), + "user": os.getenv("OPENGAUSS_USER", ""), + "password": os.getenv("OPENGAUSS_PASSWORD", ""), + "db_name": os.getenv("OPENGAUSS_DB", ""), + "user_name": user_name, + "use_multi_db": False, + "embedding_dimension": int(os.getenv("EMBEDDING_DIMENSION", "1024")), + "minconn": int(os.getenv("OPENGAUSS_MIN_CONN", "1")), + "maxconn": int(os.getenv("OPENGAUSS_MAX_CONN", "20")), + "connect_timeout": int(os.getenv("OPENGAUSS_CONNECT_TIMEOUT", "10")), + "connection_wait_timeout": int(os.getenv("OPENGAUSS_CONNECTION_WAIT_TIMEOUT", "30")), + } + + @staticmethod + def get_graph_db_config(user_id: str | None = "default") -> dict[str, Any]: + """Select a graph backend before evaluating its environment configuration.""" + getters = { + "neo4j-community": APIConfig.get_neo4j_community_config, + "neo4j": APIConfig.get_neo4j_config, + "polardb": APIConfig.get_polardb_config, + "postgres": APIConfig.get_postgres_config, + "opengauss": APIConfig.get_opengauss_config, + } + backend = os.getenv( + "GRAPH_DB_BACKEND", os.getenv("NEO4J_BACKEND", "neo4j-community") + ).lower() + if backend not in getters: + raise ValueError(f"Invalid graph DB backend: {backend}") + return {"backend": backend, "config": getters[backend](user_id=user_id)} + @staticmethod def get_mysql_config() -> dict[str, Any]: """Get MySQL configuration.""" @@ -1215,73 +1253,53 @@ def create_user_config(user_name: str, user_id: str) -> tuple["MOSConfig", "Gene default_config = MOSConfig(**config_dict) - neo4j_community_config = APIConfig.get_neo4j_community_config(user_id) - neo4j_config = APIConfig.get_neo4j_config(user_id) - polardb_config = APIConfig.get_polardb_config(user_id) + graph_db_config = APIConfig.get_graph_db_config(user_id=user_id) internet_config = ( APIConfig.get_internet_config() if os.getenv("ENABLE_INTERNET", "false").lower() == "true" else None ) - postgres_config = APIConfig.get_postgres_config(user_id=user_id) - graph_db_backend_map = { - "neo4j-community": neo4j_community_config, - "neo4j": neo4j_config, - "polardb": polardb_config, - "postgres": postgres_config, - } - # Support both GRAPH_DB_BACKEND and legacy NEO4J_BACKEND env vars - graph_db_backend = os.getenv( - "GRAPH_DB_BACKEND", os.getenv("NEO4J_BACKEND", "neo4j-community") - ).lower() - if graph_db_backend in graph_db_backend_map: - # Create MemCube config - - default_cube_config = GeneralMemCubeConfig.model_validate( - { - "user_id": user_id, - "cube_id": f"{user_name}_default_cube", - "text_mem": { - "backend": "tree_text", - "config": { - "extractor_llm": {"backend": "openai", "config": openai_config}, - "dispatcher_llm": {"backend": "openai", "config": openai_config}, - "graph_db": { - "backend": graph_db_backend, - "config": graph_db_backend_map[graph_db_backend], - }, - "embedder": APIConfig.get_embedder_config(), - "internet_retriever": internet_config, - "reranker": APIConfig.get_reranker_config(), - "reorganize": os.getenv("MOS_ENABLE_REORGANIZE", "false").lower() - == "true", - "memory_size": { - "WorkingMemory": int(os.getenv("MOS_WORKING_MEMORY", 20)), - "LongTermMemory": int(os.getenv("MOS_LONGTERM_MEMORY", 1e6)), - "UserMemory": int(os.getenv("MOS_USER_MEMORY", 1e6)), - }, - "search_strategy": { - "fast_graph": bool(os.getenv("FAST_GRAPH", "false") == "true"), - "bm25": bool(os.getenv("BM25_CALL", "false") == "true"), - "cot": bool(os.getenv("VEC_COT_CALL", "false") == "true"), - "fulltext": bool(os.getenv("FULLTEXT_CALL", "false") == "true"), - }, - "include_embedding": bool( - os.getenv("INCLUDE_EMBEDDING", "false") == "true" - ), + # Create MemCube config + + default_cube_config = GeneralMemCubeConfig.model_validate( + { + "user_id": user_id, + "cube_id": f"{user_name}_default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": graph_db_config, + "embedder": APIConfig.get_embedder_config(), + "internet_retriever": internet_config, + "reranker": APIConfig.get_reranker_config(), + "reorganize": os.getenv("MOS_ENABLE_REORGANIZE", "false").lower() == "true", + "memory_size": { + "WorkingMemory": int(os.getenv("MOS_WORKING_MEMORY", 20)), + "LongTermMemory": int(os.getenv("MOS_LONGTERM_MEMORY", 1e6)), + "UserMemory": int(os.getenv("MOS_USER_MEMORY", 1e6)), }, + "search_strategy": { + "fast_graph": bool(os.getenv("FAST_GRAPH", "false") == "true"), + "bm25": bool(os.getenv("BM25_CALL", "false") == "true"), + "cot": bool(os.getenv("VEC_COT_CALL", "false") == "true"), + "fulltext": bool(os.getenv("FULLTEXT_CALL", "false") == "true"), + }, + "include_embedding": bool( + os.getenv("INCLUDE_EMBEDDING", "false") == "true" + ), }, - "act_mem": {} - if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" - else APIConfig.get_activation_vllm_config(), - "para_mem": {}, - "pref_mem": {} - if os.getenv("ENABLE_PREFERENCE_MEMORY", "false").lower() == "false" - else APIConfig.get_preference_memory_config(), - } - ) - else: - raise ValueError(f"Invalid graph DB backend: {graph_db_backend}") + }, + "act_mem": {} + if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" + else APIConfig.get_activation_vllm_config(), + "para_mem": {}, + "pref_mem": {} + if os.getenv("ENABLE_PREFERENCE_MEMORY", "false").lower() == "false" + else APIConfig.get_preference_memory_config(), + } + ) default_mem_cube = GeneralMemCube(default_cube_config) return default_config, default_mem_cube @@ -1298,69 +1316,49 @@ def get_default_cube_config() -> "GeneralMemCubeConfig | None": return None openai_config = APIConfig.get_openai_config() - neo4j_community_config = APIConfig.get_neo4j_community_config(user_id="default") - neo4j_config = APIConfig.get_neo4j_config(user_id="default") - polardb_config = APIConfig.get_polardb_config(user_id="default") - postgres_config = APIConfig.get_postgres_config(user_id="default") - graph_db_backend_map = { - "neo4j-community": neo4j_community_config, - "neo4j": neo4j_config, - "polardb": polardb_config, - "postgres": postgres_config, - } + graph_db_config = APIConfig.get_graph_db_config(user_id="default") internet_config = ( APIConfig.get_internet_config() if os.getenv("ENABLE_INTERNET", "false").lower() == "true" else None ) - # Support both GRAPH_DB_BACKEND and legacy NEO4J_BACKEND env vars - graph_db_backend = os.getenv( - "GRAPH_DB_BACKEND", os.getenv("NEO4J_BACKEND", "neo4j-community") - ).lower() - if graph_db_backend in graph_db_backend_map: - return GeneralMemCubeConfig.model_validate( - { - "user_id": "default", - "cube_id": "default_cube", - "text_mem": { - "backend": "tree_text", - "config": { - "extractor_llm": {"backend": "openai", "config": openai_config}, - "dispatcher_llm": {"backend": "openai", "config": openai_config}, - "graph_db": { - "backend": graph_db_backend, - "config": graph_db_backend_map[graph_db_backend], - }, - "embedder": APIConfig.get_embedder_config(), - "reranker": APIConfig.get_reranker_config(), - "reorganize": os.getenv("MOS_ENABLE_REORGANIZE", "false").lower() - == "true", - "internet_retriever": internet_config, - "memory_size": { - "WorkingMemory": int(os.getenv("MOS_WORKING_MEMORY", 20)), - "LongTermMemory": int(os.getenv("MOS_LONGTERM_MEMORY", 1e6)), - "UserMemory": int(os.getenv("MOS_USER_MEMORY", 1e6)), - }, - "search_strategy": { - "fast_graph": bool(os.getenv("FAST_GRAPH", "false") == "true"), - "bm25": bool(os.getenv("BM25_CALL", "false") == "true"), - "cot": bool(os.getenv("VEC_COT_CALL", "false") == "true"), - "fulltext": bool(os.getenv("FULLTEXT_CALL", "false") == "true"), - }, - "mode": os.getenv("ASYNC_MODE", "sync"), - "include_embedding": bool( - os.getenv("INCLUDE_EMBEDDING", "false") == "true" - ), + return GeneralMemCubeConfig.model_validate( + { + "user_id": "default", + "cube_id": "default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": graph_db_config, + "embedder": APIConfig.get_embedder_config(), + "reranker": APIConfig.get_reranker_config(), + "reorganize": os.getenv("MOS_ENABLE_REORGANIZE", "false").lower() == "true", + "internet_retriever": internet_config, + "memory_size": { + "WorkingMemory": int(os.getenv("MOS_WORKING_MEMORY", 20)), + "LongTermMemory": int(os.getenv("MOS_LONGTERM_MEMORY", 1e6)), + "UserMemory": int(os.getenv("MOS_USER_MEMORY", 1e6)), + }, + "search_strategy": { + "fast_graph": bool(os.getenv("FAST_GRAPH", "false") == "true"), + "bm25": bool(os.getenv("BM25_CALL", "false") == "true"), + "cot": bool(os.getenv("VEC_COT_CALL", "false") == "true"), + "fulltext": bool(os.getenv("FULLTEXT_CALL", "false") == "true"), }, + "mode": os.getenv("ASYNC_MODE", "sync"), + "include_embedding": bool( + os.getenv("INCLUDE_EMBEDDING", "false") == "true" + ), }, - "act_mem": {} - if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" - else APIConfig.get_activation_vllm_config(), - "para_mem": {}, - "pref_mem": {} - if os.getenv("ENABLE_PREFERENCE_MEMORY", "false").lower() == "false" - else APIConfig.get_preference_memory_config(), - } - ) - else: - raise ValueError(f"Invalid graph DB backend: {graph_db_backend}") + }, + "act_mem": {} + if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" + else APIConfig.get_activation_vllm_config(), + "para_mem": {}, + "pref_mem": {} + if os.getenv("ENABLE_PREFERENCE_MEMORY", "false").lower() == "false" + else APIConfig.get_preference_memory_config(), + } + ) diff --git a/src/memos/api/handlers/add_handler.py b/src/memos/api/handlers/add_handler.py index 4240545f6..ed1948c0f 100644 --- a/src/memos/api/handlers/add_handler.py +++ b/src/memos/api/handlers/add_handler.py @@ -5,6 +5,7 @@ using dependency injection for better modularity and testability. """ +from fastapi import HTTPException from pydantic import validate_call from memos.api.handlers.base_handler import BaseHandler, HandlerDependencies @@ -104,7 +105,8 @@ def _check_messages(messages: MessageList) -> None: data=[process_record], ) except Exception as e: - self.logger.warning(f"[ADDFeedbackHandler] Running error: {e}") + self.logger.exception("Feedback processing failed") + raise HTTPException(status_code=500, detail="Feedback processing failed") from e results = cube_view.add_memories(add_req) diff --git a/src/memos/api/handlers/component_init.py b/src/memos/api/handlers/component_init.py index 7e382c595..7ba0738e7 100644 --- a/src/memos/api/handlers/component_init.py +++ b/src/memos/api/handlers/component_init.py @@ -24,6 +24,7 @@ build_scheduler_llm_config, build_suggestion_llm_config, ) +from memos.api.lifecycle import shutdown_components from memos.configs.mem_scheduler import SchedulerConfigFactory from memos.embedders.factory import EmbedderFactory from memos.graph_dbs.factory import GraphStoreFactory @@ -175,178 +176,191 @@ def init_server() -> dict[str, Any]: # Create component instances graph_db = GraphStoreFactory.from_config(graph_db_config) - llm = LLMFactory.from_config(llm_config) - feedback_llm = LLMFactory.from_config(feedback_llm_config) - suggestion_llm = LLMFactory.from_config(suggestion_llm_config) - scheduler_llm = LLMFactory.from_config(scheduler_llm_config) - chat_llms = ( - _init_chat_llms(chat_llm_config) - if os.getenv("ENABLE_CHAT_API", "false") == "true" - else None - ) - playground_chat_llms = ( - _init_chat_llms(playground_chat_llm_config) - if os.getenv("ENABLE_CHAT_API", "false") == "true" and playground_chat_llm_config - else chat_llms - ) - embedder = EmbedderFactory.from_config(embedder_config) - - plugin_context = build_plugin_context( - graph_db=graph_db, - embedder=embedder, - llm=llm, - default_cube_config=default_cube_config, - nli_client_config=nli_client_config, - mem_reader_config=mem_reader_config, - reranker_config=reranker_config, - feedback_reranker_config=feedback_reranker_config, - internet_retriever_config=internet_retriever_config, - ) - plugin_manager.discover() - plugin_manager.init_components(plugin_context) - - # Pass graph_db to mem_reader for recall operations (deduplication, conflict detection) - mem_reader = MemReaderFactory.from_config( - mem_reader_config, - graph_db=graph_db, - ) - reranker = RerankerFactory.from_config(reranker_config) - feedback_reranker = RerankerFactory.from_config(feedback_reranker_config) - internet_retriever = InternetRetrieverFactory.from_config( - internet_retriever_config, embedder=embedder - ) - - # Initialize chat llms - - logger.debug("Core components instantiated") - - # Initialize memory manager - memory_manager = MemoryManager( - graph_db, - embedder, - llm, - memory_size=_get_default_memory_size(default_cube_config), - is_reorganize=getattr(default_cube_config.text_mem.config, "reorganize", False), - ) - - logger.debug("Memory manager initialized") - tokenizer = FastTokenizer() - # Initialize text memory - text_mem = SimpleTreeTextMemory( - llm=llm, - embedder=embedder, - mem_reader=mem_reader, - graph_db=graph_db, - reranker=reranker, - memory_manager=memory_manager, - config=default_cube_config.text_mem.config, - internet_retriever=internet_retriever, - tokenizer=tokenizer, - include_embedding=bool(os.getenv("INCLUDE_EMBEDDING", "false") == "true"), - ) - - logger.debug("Text memory initialized") - - # Create MemCube with pre-initialized memory instances - naive_mem_cube = NaiveMemCube( - text_mem=text_mem, - act_mem=None, - para_mem=None, - ) - - logger.debug("MemCube created") - - tree_mem: TreeTextMemory = naive_mem_cube.text_mem - searcher: Searcher = tree_mem.get_searcher( - manual_close_internet=os.getenv("ENABLE_INTERNET", "true").lower() == "false", - moscube=False, - process_llm=mem_reader.general_llm, - ) - logger.debug("Searcher created") - - # Set searcher to mem_reader - mem_reader.set_searcher(searcher) - - # Initialize feedback server - feedback_server = SimpleMemFeedback( - llm=feedback_llm, - embedder=embedder, - graph_store=graph_db, - memory_manager=memory_manager, - mem_reader=mem_reader, - searcher=searcher, - reranker=feedback_reranker, - pref_feedback=True, - ) - - # Initialize Scheduler - scheduler_config_dict = APIConfig.get_scheduler_config() - scheduler_config = SchedulerConfigFactory( - backend=scheduler_config_dict["backend"], - config=scheduler_config_dict["config"], - ) - mem_scheduler: OptimizedScheduler = SchedulerFactory.from_config(scheduler_config) - mem_scheduler.initialize_modules( - chat_llm=scheduler_llm, - process_llm=mem_reader.general_llm, - db_engine=BaseDBManager.create_default_sqlite_engine(), - mem_reader=mem_reader, - redis_client=redis_client, - ) - mem_scheduler.init_mem_cube( - mem_cube=naive_mem_cube, searcher=searcher, feedback_server=feedback_server - ) - logger.debug("Scheduler initialized") - - # Initialize SchedulerAPIModule - api_module = mem_scheduler.api_module - - # Plugins keep a reference to the original context dict, so updating the shared - # section here makes the runtime handles visible without adding a second init step. - plugin_context["shared"]["mem_scheduler"] = mem_scheduler - plugin_context["shared"]["submit_scheduler_messages"] = mem_scheduler.submit_messages - plugin_context["shared"]["api_module"] = api_module - - # Start scheduler if enabled - if os.getenv("API_SCHEDULER_ON", "true").lower() == "true": - mem_scheduler.start() - logger.info("Scheduler started") - - logger.info("MemOS server components initialized successfully") - - # Initialize online bot if enabled - online_bot = None - if dingding_enabled: - from memos.memos_tools.notification_service import get_online_bot_function - - online_bot = get_online_bot_function() if dingding_enabled else None - logger.info("DingDing bot is enabled") - - deepsearch_agent = DeepSearchMemAgent( - llm=llm, - memory_retriever=tree_mem, - ) - # Return all components as a dictionary for easy access and extension - return { - "graph_db": graph_db, - "mem_reader": mem_reader, - "llm": llm, - "suggestion_llm": suggestion_llm, - "scheduler_llm": scheduler_llm, - "chat_llms": chat_llms, - "playground_chat_llms": playground_chat_llms, - "embedder": embedder, - "reranker": reranker, - "internet_retriever": internet_retriever, - "memory_manager": memory_manager, - "default_cube_config": default_cube_config, - "mem_scheduler": mem_scheduler, - "naive_mem_cube": naive_mem_cube, - "searcher": searcher, - "api_module": api_module, - "text_mem": text_mem, - "online_bot": online_bot, - "feedback_server": feedback_server, - "redis_client": redis_client, - "deepsearch_agent": deepsearch_agent, - } + components = {"graph_db": graph_db} + try: + llm = LLMFactory.from_config(llm_config) + feedback_llm = LLMFactory.from_config(feedback_llm_config) + suggestion_llm = LLMFactory.from_config(suggestion_llm_config) + scheduler_llm = LLMFactory.from_config(scheduler_llm_config) + chat_llms = ( + _init_chat_llms(chat_llm_config) + if os.getenv("ENABLE_CHAT_API", "false") == "true" + else None + ) + playground_chat_llms = ( + _init_chat_llms(playground_chat_llm_config) + if os.getenv("ENABLE_CHAT_API", "false") == "true" and playground_chat_llm_config + else chat_llms + ) + embedder = EmbedderFactory.from_config(embedder_config) + + plugin_context = build_plugin_context( + graph_db=graph_db, + embedder=embedder, + llm=llm, + default_cube_config=default_cube_config, + nli_client_config=nli_client_config, + mem_reader_config=mem_reader_config, + reranker_config=reranker_config, + feedback_reranker_config=feedback_reranker_config, + internet_retriever_config=internet_retriever_config, + ) + plugin_manager.discover() + plugin_manager.init_components(plugin_context) + + # Pass graph_db to mem_reader for recall operations (deduplication, conflict detection) + mem_reader = MemReaderFactory.from_config( + mem_reader_config, + graph_db=graph_db, + ) + reranker = RerankerFactory.from_config(reranker_config) + feedback_reranker = RerankerFactory.from_config(feedback_reranker_config) + internet_retriever = InternetRetrieverFactory.from_config( + internet_retriever_config, embedder=embedder + ) + + # Initialize chat llms + + logger.debug("Core components instantiated") + + # Initialize memory manager + memory_manager = MemoryManager( + graph_db, + embedder, + llm, + memory_size=_get_default_memory_size(default_cube_config), + is_reorganize=getattr(default_cube_config.text_mem.config, "reorganize", False), + ) + + logger.debug("Memory manager initialized") + tokenizer = FastTokenizer() + # Initialize text memory + text_mem = SimpleTreeTextMemory( + llm=llm, + embedder=embedder, + mem_reader=mem_reader, + graph_db=graph_db, + reranker=reranker, + memory_manager=memory_manager, + config=default_cube_config.text_mem.config, + internet_retriever=internet_retriever, + tokenizer=tokenizer, + include_embedding=bool(os.getenv("INCLUDE_EMBEDDING", "false") == "true"), + ) + + logger.debug("Text memory initialized") + + # Create MemCube with pre-initialized memory instances + naive_mem_cube = NaiveMemCube( + text_mem=text_mem, + act_mem=None, + para_mem=None, + ) + + logger.debug("MemCube created") + + tree_mem: TreeTextMemory = naive_mem_cube.text_mem + searcher: Searcher = tree_mem.get_searcher( + manual_close_internet=os.getenv("ENABLE_INTERNET", "true").lower() == "false", + moscube=False, + process_llm=mem_reader.general_llm, + ) + logger.debug("Searcher created") + + # Set searcher to mem_reader + mem_reader.set_searcher(searcher) + + # Initialize feedback server + feedback_server = SimpleMemFeedback( + llm=feedback_llm, + embedder=embedder, + graph_store=graph_db, + memory_manager=memory_manager, + mem_reader=mem_reader, + searcher=searcher, + reranker=feedback_reranker, + pref_feedback=True, + ) + + # Initialize Scheduler + scheduler_config_dict = APIConfig.get_scheduler_config() + scheduler_config = SchedulerConfigFactory( + backend=scheduler_config_dict["backend"], + config=scheduler_config_dict["config"], + ) + mem_scheduler: OptimizedScheduler = SchedulerFactory.from_config(scheduler_config) + components["mem_scheduler"] = mem_scheduler + mem_scheduler.initialize_modules( + chat_llm=scheduler_llm, + process_llm=mem_reader.general_llm, + db_engine=BaseDBManager.create_default_sqlite_engine(), + mem_reader=mem_reader, + redis_client=redis_client, + ) + mem_scheduler.init_mem_cube( + mem_cube=naive_mem_cube, searcher=searcher, feedback_server=feedback_server + ) + logger.debug("Scheduler initialized") + + # Initialize SchedulerAPIModule + api_module = mem_scheduler.api_module + + # Plugins keep a reference to the original context dict, so updating the shared + # section here makes the runtime handles visible without adding a second init step. + plugin_context["shared"]["mem_scheduler"] = mem_scheduler + plugin_context["shared"]["submit_scheduler_messages"] = mem_scheduler.submit_messages + plugin_context["shared"]["api_module"] = api_module + + # Start scheduler if enabled + if os.getenv("API_SCHEDULER_ON", "true").lower() == "true": + mem_scheduler.start() + logger.info("Scheduler started") + + logger.info("MemOS server components initialized successfully") + + # Initialize online bot if enabled + online_bot = None + if dingding_enabled: + from memos.memos_tools.notification_service import get_online_bot_function + + online_bot = get_online_bot_function() if dingding_enabled else None + logger.info("DingDing bot is enabled") + + deepsearch_agent = DeepSearchMemAgent( + llm=llm, + memory_retriever=tree_mem, + ) + # Return all components as a dictionary for easy access and extension + components.update( + { + "graph_db": graph_db, + "mem_reader": mem_reader, + "llm": llm, + "suggestion_llm": suggestion_llm, + "scheduler_llm": scheduler_llm, + "chat_llms": chat_llms, + "playground_chat_llms": playground_chat_llms, + "embedder": embedder, + "reranker": reranker, + "internet_retriever": internet_retriever, + "memory_manager": memory_manager, + "default_cube_config": default_cube_config, + "mem_scheduler": mem_scheduler, + "naive_mem_cube": naive_mem_cube, + "searcher": searcher, + "api_module": api_module, + "text_mem": text_mem, + "online_bot": online_bot, + "feedback_server": feedback_server, + "redis_client": redis_client, + "deepsearch_agent": deepsearch_agent, + } + ) + return components + + except BaseException: + try: + shutdown_components(components) + except Exception: + logger.exception("Failed to release components after initialization failure") + raise diff --git a/src/memos/api/handlers/config_builders.py b/src/memos/api/handlers/config_builders.py index 1144b04d7..8eee76748 100644 --- a/src/memos/api/handlers/config_builders.py +++ b/src/memos/api/handlers/config_builders.py @@ -26,7 +26,7 @@ ) -def build_graph_db_config(user_id: str = "default") -> dict[str, Any]: +def build_graph_db_config(user_id: str | None = "default") -> GraphDBConfigFactory: """ Build graph database configuration. @@ -36,23 +36,7 @@ def build_graph_db_config(user_id: str = "default") -> dict[str, Any]: Returns: Validated graph database configuration dictionary """ - graph_db_backend_map = { - "neo4j-community": APIConfig.get_neo4j_community_config(user_id=user_id), - "neo4j": APIConfig.get_neo4j_config(user_id=user_id), - "polardb": APIConfig.get_polardb_config(user_id=user_id), - "postgres": APIConfig.get_postgres_config(user_id=user_id), - } - - # Support both GRAPH_DB_BACKEND and legacy NEO4J_BACKEND env vars - graph_db_backend = os.getenv( - "GRAPH_DB_BACKEND", os.getenv("NEO4J_BACKEND", "neo4j-community") - ).lower() - return GraphDBConfigFactory.model_validate( - { - "backend": graph_db_backend, - "config": graph_db_backend_map[graph_db_backend], - } - ) + return GraphDBConfigFactory.model_validate(APIConfig.get_graph_db_config(user_id)) def build_vec_db_config() -> dict[str, Any]: diff --git a/src/memos/api/handlers/feedback_handler.py b/src/memos/api/handlers/feedback_handler.py index 217bca7cd..02845404b 100644 --- a/src/memos/api/handlers/feedback_handler.py +++ b/src/memos/api/handlers/feedback_handler.py @@ -2,6 +2,8 @@ Feeback handler for memory add/update functionality. """ +from fastapi import HTTPException + from memos.api.handlers.base_handler import BaseHandler, HandlerDependencies from memos.api.product_models import APIFeedbackRequest, MemoryResponse from memos.log import get_logger @@ -42,7 +44,11 @@ def handle_feedback_memories(self, feedback_req: APIFeedbackRequest) -> MemoryRe """ cube_view = self._build_cube_view(feedback_req) - process_record = cube_view.feedback_memories(feedback_req) + try: + process_record = cube_view.feedback_memories(feedback_req) + except Exception as error: + self.logger.exception("Feedback processing failed") + raise HTTPException(status_code=500, detail="Feedback processing failed") from error self.logger.info(f"[FeedbackHandler] Final feedback results count={len(process_record)}") diff --git a/src/memos/api/handlers/formatters_handler.py b/src/memos/api/handlers/formatters_handler.py index ab205df2e..79fabc638 100644 --- a/src/memos/api/handlers/formatters_handler.py +++ b/src/memos/api/handlers/formatters_handler.py @@ -177,16 +177,14 @@ def rerank_knowledge_mem( """ Rerank knowledge memories and keep conversation memories. """ - memid2cubeid = {} - memories_list = [] + knowledge_mem = [] + conversation_mem = [] for memory_group in text_mem: cube_id = memory_group["cube_id"] - memories = memory_group["memories"] - memories_list.extend(memories) - for memory in memories: - memid2cubeid[memory["id"]] = cube_id + knowledge, conversation = separate_knowledge_and_conversation_mem(memory_group["memories"]) + knowledge_mem.extend((cube_id, memory) for memory in knowledge) + conversation_mem.extend((cube_id, memory) for memory in conversation) - knowledge_mem, conversation_mem = separate_knowledge_and_conversation_mem(memories_list) knowledge_mem_top_k = max(int(top_k * file_mem_proportion), int(top_k - len(conversation_mem))) # rerank set unuse reranked_knowledge_mem = knowledge_mem @@ -194,12 +192,12 @@ def rerank_knowledge_mem( # Sort by relativity in descending order reranked_knowledge_mem = sorted( reranked_knowledge_mem, - key=lambda item: item.get("metadata", {}).get("relativity", 0.0), + key=lambda pair: pair[1].get("metadata", {}).get("relativity", 0.0), reverse=True, ) # Combine the extracted memory with its original file source. is_chinese_query = detect_lang(query) == "zh" - for item in reranked_knowledge_mem: + for _, item in reranked_knowledge_mem: memory = item.get("memory", "") source_content = item["metadata"]["sources"][0]["content"] if is_chinese_query: @@ -209,18 +207,18 @@ def rerank_knowledge_mem( item["metadata"]["sources"] = [] if strip_conversation_sources: - for item in conversation_mem: + for _, item in conversation_mem: item.setdefault("metadata", {})["sources"] = [] # deduplicate: remove items with duplicate memory content original_count = len(reranked_knowledge_mem) seen_memories = set[Any]() deduplicated_knowledge_mem = [] - for item in reranked_knowledge_mem: + for cube_id, item in reranked_knowledge_mem: memory_content = item.get("memory", "") if memory_content and memory_content not in seen_memories: seen_memories.add(memory_content) - deduplicated_knowledge_mem.append(item) + deduplicated_knowledge_mem.append((cube_id, item)) deduplicated_count = len(deduplicated_knowledge_mem) logger.info( f"After filtering duplicate knowledge base text from sources, count changed from {original_count} to {deduplicated_count}" @@ -231,8 +229,7 @@ def rerank_knowledge_mem( cubeid2memories = {} text_mem_res = [] - for memory in reranked_knowledge_mem + conversation_mem[:conversation_mem_top_k]: - cube_id = memid2cubeid[memory["id"]] + for cube_id, memory in reranked_knowledge_mem + conversation_mem[:conversation_mem_top_k]: if cube_id not in cubeid2memories: cubeid2memories[cube_id] = [] cubeid2memories[cube_id].append(memory) diff --git a/src/memos/api/handlers/memory_handler.py b/src/memos/api/handlers/memory_handler.py index dc052fdbe..185f0719a 100644 --- a/src/memos/api/handlers/memory_handler.py +++ b/src/memos/api/handlers/memory_handler.py @@ -4,6 +4,7 @@ This module handles retrieving all memories or specific subgraphs based on queries. """ +from copy import deepcopy from typing import Any, Literal from memos.api.product_models import ( @@ -14,6 +15,7 @@ GetMemoryResponse, MemoryResponse, ) +from memos.graph_dbs.opengauss import OpenGaussGraphDB from memos.log import get_logger from memos.mem_cube.navie import NaiveMemCube from memos.mem_os.utils.format_utils import ( @@ -23,6 +25,7 @@ remove_embedding_recursive, sort_children_by_memory_type, ) +from memos.memories.textual.tree_text_memory.organize.manager import extract_working_binding_ids logger = get_logger(__name__) @@ -424,17 +427,7 @@ def handle_delete_memories(delete_mem_req: DeleteMemoryRequest, naive_mem_cube: memories = [] if memories: - import re - - pattern = re.compile(r"\[working_binding:([0-9a-fA-F-]{36})\]") - for mem in memories: - metadata = mem.get("metadata") or {} - bg = metadata.get("background") or "" - if not isinstance(bg, str): - continue - match = pattern.search(bg) - if match: - working_ids_to_delete.add(match.group(1)) + working_ids_to_delete = extract_working_binding_ids(memories) if delete_mem_req.memory_ids is not None: # Unified deletion from text_mem (includes preferences) @@ -443,7 +436,7 @@ def handle_delete_memories(delete_mem_req: DeleteMemoryRequest, naive_mem_cube: naive_mem_cube.text_mem.delete_by_filter( writable_cube_ids=delete_mem_req.writable_cube_ids, file_ids=delete_mem_req.file_ids ) - elif has_filter_mode: + else: # The validation above leaves only filter mode. merged_filter = _merge_delete_filter(delete_mem_req.filter, quick_constraints) naive_mem_cube.text_mem.delete_by_filter( writable_cube_ids=delete_mem_req.writable_cube_ids, @@ -479,168 +472,134 @@ def handle_delete_memories(delete_mem_req: DeleteMemoryRequest, naive_mem_cube: # ============================================================================= +def _dashboard_scope(request: GetMemoryDashboardRequest) -> dict[str, Any]: + """Extract global range controls while preserving nonempty business subtrees.""" + controls = {} + + def nonblank(value, name): + if not isinstance(value, str) or not value.strip() or "\0" in value: + raise ValueError(f"{name} must be a nonblank Cube selection string") + + def prune(condition, root=False, inside_or=False): + if root and (condition is None or condition == {}): + return None + if not isinstance(condition, dict) or not condition: + raise ValueError("Dashboard filter nodes must be nonempty objects") + remaining = {} + for key, value in condition.items(): + if key in ("and", "or"): + if not isinstance(value, list) or not value: + raise ValueError("Dashboard and/or filters require nonempty lists") + children = [prune(child, inside_or=inside_or or key == "or") for child in value] + children = [child for child in children if child is not None] + if children: + remaining[key] = children + elif key in ("object_type", "mem_cube_id"): + if inside_or: + raise ValueError("Dashboard scope controls are not allowed inside OR") + nonblank(value, key) + if key in controls and controls[key] != value: + raise ValueError("Conflicting Dashboard scope controls") + controls[key] = value + else: + remaining[key] = deepcopy(value) + return remaining or None + + business_filter = prune(request.filter, root=True) + if controls and set(controls) != {"object_type", "mem_cube_id"}: + raise ValueError("Dashboard scope requires object_type and mem_cube_id together") + kind = controls.get("object_type") + if kind is not None and kind not in ("public", "user"): + raise ValueError("Unknown Dashboard object_type") + owner = request.mem_cube_id + if owner is not None: + nonblank(owner, "mem_cube_id") + excluded = controls.get("mem_cube_id") if kind == "user" else None + if kind == "public": + if owner is not None and owner != controls["mem_cube_id"]: + raise ValueError("Conflicting Dashboard Cube selections") + owner = controls["mem_cube_id"] + elif kind == "user": + if owner is not None: + raise ValueError("User range conflicts with a single Cube selection") + all_cubes = kind == "user" + if request.all_cubes is True: + if request.mem_cube_id is not None or kind == "public": + raise ValueError("all_cubes=True conflicts with a single Cube selection") + all_cubes = True + elif request.all_cubes is False: + if kind == "user": + raise ValueError("all_cubes=False conflicts with user range selection") + all_cubes = False + if request.exclude_cube_id is not None: + nonblank(request.exclude_cube_id, "exclude_cube_id") + if excluded is not None and excluded != request.exclude_cube_id: + raise ValueError("Conflicting excluded Cubes") + excluded = request.exclude_cube_id + if excluded is not None and not all_cubes: + raise ValueError("Cube exclusion requires an all-Cube range") + return { + "user_name": owner, + "all_cubes": all_cubes, + "exclude_user_name": excluded, + "filter": business_filter, + } + + def handle_get_memories_dashboard( get_mem_req: GetMemoryDashboardRequest, naive_mem_cube: NaiveMemCube ) -> GetMemoryResponse: - results: dict[str, Any] = {"text_mem": [], "pref_mem": [], "tool_mem": [], "skill_mem": []} - # for statistics - total_text_nodes, total_tool_nodes, total_skill_nodes, total_preference_nodes = 0, 0, 0, 0 - total_tool_nodes = 0 - total_skill_nodes = 0 - total_preference_nodes = 0 - - text_memory_type = ["WorkingMemory", "LongTermMemory", "UserMemory", "OuterMemory"] - text_memories_info = naive_mem_cube.text_mem.get_all( - user_name=get_mem_req.mem_cube_id, - user_id=get_mem_req.user_id, - page=get_mem_req.page, - page_size=get_mem_req.page_size, - filter=get_mem_req.filter, - memory_type=text_memory_type, - ) - text_memories, total_text_nodes = text_memories_info["nodes"], text_memories_info["total_nodes"] - - # Group text memories by cube_id from metadata.user_name - text_mem_by_cube: dict[str, list] = {} - for memory in text_memories: - cube_id = memory.get("metadata", {}).get("user_name", get_mem_req.mem_cube_id) - if cube_id not in text_mem_by_cube: - text_mem_by_cube[cube_id] = [] - text_mem_by_cube[cube_id].append(memory) - - # If no memories found, create a default entry with the requested cube_id - if not text_mem_by_cube and get_mem_req.mem_cube_id: - text_mem_by_cube[get_mem_req.mem_cube_id] = [] - - results["text_mem"] = [ - { - "cube_id": cube_id, - "memories": memories, - "total_nodes": len(memories), - } - for cube_id, memories in text_mem_by_cube.items() + """List actual records within one resolved Dashboard range, with total counts.""" + memory = naive_mem_cube.text_mem + if isinstance(memory.graph_store, OpenGaussGraphDB): + scope = _dashboard_scope(get_mem_req) + else: + if get_mem_req.all_cubes is not None or get_mem_req.exclude_cube_id is not None: + raise ValueError("This graph backend does not support explicit Cube ranges") + scope = {"user_name": get_mem_req.mem_cube_id, "filter": get_mem_req.filter} + categories = [ + ( + "text_mem", + "total_text_nodes", + True, + ["WorkingMemory", "LongTermMemory", "UserMemory", "OuterMemory"], + ), + ( + "tool_mem", + "total_tool_nodes", + get_mem_req.include_tool_memory, + ["ToolSchemaMemory", "ToolTrajectoryMemory"], + ), + ("skill_mem", "total_skill_nodes", get_mem_req.include_skill_memory, ["SkillMemory"]), + ( + "pref_mem", + "total_preference_nodes", + get_mem_req.include_preference, + ["PreferenceMemory"], + ), ] - - if get_mem_req.include_tool_memory: - tool_memories_info = naive_mem_cube.text_mem.get_all( - user_name=get_mem_req.mem_cube_id, - user_id=get_mem_req.user_id, - page=get_mem_req.page, - page_size=get_mem_req.page_size, - filter=get_mem_req.filter, - memory_type=["ToolSchemaMemory", "ToolTrajectoryMemory"], - ) - tool_memories, total_tool_nodes = ( - tool_memories_info["nodes"], - tool_memories_info["total_nodes"], - ) - - # Group tool memories by cube_id from metadata.user_name - tool_mem_by_cube: dict[str, list] = {} - for memory in tool_memories: - cube_id = memory.get("metadata", {}).get("user_name", get_mem_req.mem_cube_id) - if cube_id not in tool_mem_by_cube: - tool_mem_by_cube[cube_id] = [] - tool_mem_by_cube[cube_id].append(memory) - - # If no memories found, create a default entry with the requested cube_id - if not tool_mem_by_cube and get_mem_req.mem_cube_id: - tool_mem_by_cube[get_mem_req.mem_cube_id] = [] - - results["tool_mem"] = [ - { - "cube_id": cube_id, - "memories": memories, - "total_nodes": len(memories), - } - for cube_id, memories in tool_mem_by_cube.items() - ] - - if get_mem_req.include_skill_memory: - skill_memories_info = naive_mem_cube.text_mem.get_all( - user_name=get_mem_req.mem_cube_id, + results = {} + statistics = {} + for category, total_key, enabled, memory_types in categories: + results[category] = [] + statistics[total_key] = 0 + if not enabled: + continue + page = memory.get_all( + **scope, user_id=get_mem_req.user_id, page=get_mem_req.page, page_size=get_mem_req.page_size, - filter=get_mem_req.filter, - memory_type=["SkillMemory"], - ) - skill_memories, total_skill_nodes = ( - skill_memories_info["nodes"], - skill_memories_info["total_nodes"], + memory_type=memory_types, ) - - # Group skill memories by cube_id from metadata.user_name - skill_mem_by_cube: dict[str, list] = {} - for memory in skill_memories: - cube_id = memory.get("metadata", {}).get("user_name", get_mem_req.mem_cube_id) - if cube_id not in skill_mem_by_cube: - skill_mem_by_cube[cube_id] = [] - skill_mem_by_cube[cube_id].append(memory) - - # If no memories found, create a default entry with the requested cube_id - if not skill_mem_by_cube and get_mem_req.mem_cube_id: - skill_mem_by_cube[get_mem_req.mem_cube_id] = [] - - results["skill_mem"] = [ - { - "cube_id": cube_id, - "memories": memories, - "total_nodes": len(memories), - } - for cube_id, memories in skill_mem_by_cube.items() + buckets = {} + for node in page["nodes"]: + owner = node.get("metadata", {}).get("user_name", get_mem_req.mem_cube_id) + buckets.setdefault(owner, []).append(node) + results[category] = [ + {"cube_id": owner, "memories": nodes, "total_nodes": len(nodes)} + for owner, nodes in buckets.items() ] - - if get_mem_req.include_preference: - pref_memories_info = naive_mem_cube.text_mem.get_all( - user_name=get_mem_req.mem_cube_id, - user_id=get_mem_req.user_id, - page=get_mem_req.page, - page_size=get_mem_req.page_size, - filter=get_mem_req.filter, - memory_type=["PreferenceMemory"], - ) - pref_memories, total_preference_nodes = ( - pref_memories_info["nodes"], - pref_memories_info["total_nodes"], - ) - - # Group preference memories by cube_id from metadata.user_name - pref_mem_by_cube: dict[str, list] = {} - for memory in pref_memories: - cube_id = memory.get("metadata", {}).get("user_name", get_mem_req.mem_cube_id) - if cube_id not in pref_mem_by_cube: - pref_mem_by_cube[cube_id] = [] - pref_mem_by_cube[cube_id].append(memory) - - # If no memories found, create a default entry with the requested cube_id - if not pref_mem_by_cube and get_mem_req.mem_cube_id: - pref_mem_by_cube[get_mem_req.mem_cube_id] = [] - - results["pref_mem"] = [ - { - "cube_id": cube_id, - "memories": memories, - "total_nodes": len(memories), - } - for cube_id, memories in pref_mem_by_cube.items() - ] - - # Filter to only keep text_mem, pref_mem, tool_mem, skill_mem - filtered_results = { - "text_mem": results.get("text_mem", []), - "pref_mem": results.get("pref_mem", []), - "tool_mem": results.get("tool_mem", []), - "skill_mem": results.get("skill_mem", []), - } - - # statistics - statistics = { - "total_text_nodes": total_text_nodes, - "total_tool_nodes": total_tool_nodes, - "total_skill_nodes": total_skill_nodes, - "total_preference_nodes": total_preference_nodes, - } - filtered_results["statistics"] = statistics - - return GetMemoryResponse(message="Memories retrieved successfully", data=filtered_results) + statistics[total_key] = page["total_nodes"] + results["statistics"] = statistics + return GetMemoryResponse(message="Memories retrieved successfully", data=results) diff --git a/src/memos/api/lifecycle.py b/src/memos/api/lifecycle.py index e05a2f777..dbfefe634 100644 --- a/src/memos/api/lifecycle.py +++ b/src/memos/api/lifecycle.py @@ -1,6 +1,7 @@ from collections.abc import Mapping from typing import Any +from memos.exceptions import MemOSError from memos.log import get_logger @@ -8,19 +9,31 @@ def shutdown_components(components: Mapping[str, Any] | None) -> None: - """Release long-lived API components before the logging system shuts down.""" + """Drain borrowers before the graph; each resource owns its close state.""" if not components: return - mem_scheduler = components.get("mem_scheduler") - if mem_scheduler is None: - return + errors = [] - for method_name in ("stop", "rabbitmq_close"): - method = getattr(mem_scheduler, method_name, None) - if not callable(method): - continue + def release(key: str, resource: Any, method_name: str) -> bool: + method = getattr(resource, method_name, None) try: - method() - except Exception: - logger.exception("Failed to run mem_scheduler.%s during API shutdown", method_name) + if callable(method): + method() + return True + except Exception as error: + logger.exception("Failed to close component %s", key) + errors.append(error) + return False + + scheduler = components.get("mem_scheduler") + if callable(getattr(scheduler, "close", None)): + drained = release("scheduler", scheduler, "close") + else: + drained = release("scheduler.stop", scheduler, "stop") + rabbitmq_drained = release("scheduler.rabbitmq", scheduler, "rabbitmq_close") + drained = drained and rabbitmq_drained + if drained: + release("graph", components.get("graph_db"), "close") + if errors: + raise MemOSError("Component shutdown failed; unreleased resources retained") from errors[0] diff --git a/src/memos/api/product_models.py b/src/memos/api/product_models.py index 1769b0ee6..790ca99fe 100644 --- a/src/memos/api/product_models.py +++ b/src/memos/api/product_models.py @@ -965,6 +965,17 @@ class GetMemoryDashboardRequest(GetMemoryRequest): """Request model for getting memories for dashboard.""" mem_cube_id: str | None = Field(None, description="Cube ID") + all_cubes: bool | None = Field( + None, + strict=True, + description="Optional range: omitted/null or false selects one Cube (configured default when mem_cube_id is omitted); true selects all Cubes. Legacy user scope filters explicitly select all Cubes except their specified Cube. Supported by openGauss.", + ) + exclude_cube_id: str | None = Field( + None, + min_length=1, + pattern=r"\S", + description="Optionally exclude this exact Cube from the final all-Cube range. Supported by openGauss.", + ) class DeleteMemoryRequest(BaseRequest): diff --git a/src/memos/api/routers/server_router.py b/src/memos/api/routers/server_router.py index 561b696d0..966c956cb 100644 --- a/src/memos/api/routers/server_router.py +++ b/src/memos/api/routers/server_router.py @@ -16,6 +16,7 @@ import socket from fastapi import APIRouter, HTTPException, Query +from fastapi.responses import StreamingResponse from memos.api import handlers from memos.api.handlers.add_handler import AddHandler @@ -247,7 +248,12 @@ def scheduler_wait( ) -@router.get("/scheduler/wait/stream", summary="Stream scheduler progress for a user") +@router.get( + "/scheduler/wait/stream", + summary="Stream scheduler progress for a user", + response_class=StreamingResponse, + responses={200: {"content": {"text/event-stream": {"schema": {"type": "string"}}}}}, +) def scheduler_wait_stream( user_name: str, timeout_seconds: float = 120.0, @@ -282,7 +288,12 @@ def chat_complete(chat_req: APIChatCompleteRequest): return chat_handler.handle_chat_complete(chat_req) -@router.post("/chat/stream", summary="Chat with MemOS") +@router.post( + "/chat/stream", + summary="Chat with MemOS", + response_class=StreamingResponse, + responses={200: {"content": {"text/event-stream": {"schema": {"type": "string"}}}}}, +) def chat_stream(chat_req: ChatRequest): """ Chat with MemOS for a specific user. Returns SSE stream. @@ -297,7 +308,12 @@ def chat_stream(chat_req: ChatRequest): return chat_handler.handle_chat_stream(chat_req) -@router.post("/chat/stream/playground", summary="Chat with MemOS playground") +@router.post( + "/chat/stream/playground", + summary="Chat with MemOS playground", + response_class=StreamingResponse, + responses={200: {"content": {"text/event-stream": {"schema": {"type": "string"}}}}}, +) def chat_stream_playground(chat_req: ChatPlaygroundRequest): """ Chat with MemOS for a specific user. Returns SSE stream. @@ -451,7 +467,12 @@ def exist_mem_cube_id(request: ExistMemCubeIdRequest): ) -@router.post("/chat/stream/business_user", summary="Chat with MemOS for business user") +@router.post( + "/chat/stream/business_user", + summary="Chat with MemOS for business user", + response_class=StreamingResponse, + responses={200: {"content": {"text/event-stream": {"schema": {"type": "string"}}}}}, +) def chat_stream_business_user(chat_req: ChatBusinessRequest): """(inner) Chat with MemOS for a specific business user. Returns SSE stream.""" if chat_handler is None: diff --git a/src/memos/configs/graph_db.py b/src/memos/configs/graph_db.py index 984c95060..75d317658 100644 --- a/src/memos/configs/graph_db.py +++ b/src/memos/configs/graph_db.py @@ -264,6 +264,56 @@ def validate_config(self): return self +class OpenGaussGraphDBConfig(BaseConfig): + """Connection and isolation settings for the openGauss enterprise AGE backend. + + Shared-database mode requires a logical scope. Physical isolation may omit + it; explicitly supplied scopes are preserved without normalization. + """ + + host: str = Field(..., min_length=1, description="Database host") + port: int = Field(default=5432, ge=1, le=65535, description="Database port") + user: str = Field(..., min_length=1, description="Database login user") + password: str = Field(..., min_length=1, repr=False, description="Database password") + db_name: str = Field(..., min_length=1, description="Target openGauss database") + user_name: str | None = Field( + default=None, + description="Logical tenant or Cube scope; required in shared-database mode", + ) + use_multi_db: bool = Field( + default=False, + description="Use physical database isolation instead of a required logical scope", + ) + embedding_dimension: int = Field( + default=1024, ge=1, le=16000, description="DataVec dimension; HNSW supports up to 2000" + ) + minconn: int = Field(default=1, ge=1, description="Minimum pooled connections") + maxconn: int = Field(default=20, ge=1, description="Maximum pooled connections") + connect_timeout: int = Field( + default=10, ge=1, le=3600, description="Database connection timeout in seconds" + ) + connection_wait_timeout: int = Field( + default=30, ge=1, le=3600, description="Maximum wait for a pooled connection in seconds" + ) + + @field_validator("host", "user", "password", "db_name", "user_name") + @classmethod + def validate_non_blank(cls, value: str | None) -> str | None: + """Reject blank values while preserving valid connection and scope strings.""" + if value is not None and not value.strip(): + raise ValueError("must not be empty or whitespace") + return value + + @model_validator(mode="after") + def validate_config(self) -> "OpenGaussGraphDBConfig": + """Require shared-database isolation and an ordered connection-pool range.""" + if not self.use_multi_db and self.user_name is None: + raise ValueError("In shared-database mode, `user_name` must be provided") + if self.minconn > self.maxconn: + raise ValueError("`minconn` must be <= `maxconn`") + return self + + class GraphDBConfigFactory(BaseModel): backend: str = Field(..., description="Backend for graph database") config: dict[str, Any] = Field(..., description="Configuration for the graph database backend") @@ -273,6 +323,7 @@ class GraphDBConfigFactory(BaseModel): "neo4j-community": Neo4jCommunityGraphDBConfig, "polardb": PolarDBGraphDBConfig, "postgres": PostgresGraphDBConfig, + "opengauss": OpenGaussGraphDBConfig, } @field_validator("backend") diff --git a/src/memos/configs/mem_scheduler.py b/src/memos/configs/mem_scheduler.py index 0f540d279..0a59611a1 100644 --- a/src/memos/configs/mem_scheduler.py +++ b/src/memos/configs/mem_scheduler.py @@ -2,7 +2,7 @@ import os from pathlib import Path -from typing import Any, ClassVar +from typing import Any, ClassVar, Literal from pydantic import ConfigDict, Field, field_validator, model_validator @@ -29,6 +29,11 @@ class BaseSchedulerConfig(BaseConfig): """Base configuration class for mem_scheduler.""" + scheduler_startup_mode: Literal["thread"] = Field( + default="thread", + description="Create independent process workers inside the child; runtime resources cannot be inherited.", + ) + top_k: int = Field( default=DEFAULT_TOP_K, description="Number of top candidates to consider in initial retrieval", diff --git a/src/memos/exceptions.py b/src/memos/exceptions.py index 511e68d8f..42beecdd4 100644 --- a/src/memos/exceptions.py +++ b/src/memos/exceptions.py @@ -21,6 +21,10 @@ class MemCubeError(MemOSError): ... class VectorDBError(MemOSError): ... +class GraphDBError(MemOSError): + """Graph storage, schema or connection lifecycle failure.""" + + class LLMError(MemOSError): ... diff --git a/src/memos/graph_dbs/factory.py b/src/memos/graph_dbs/factory.py index 93b5971ec..a111e0d10 100644 --- a/src/memos/graph_dbs/factory.py +++ b/src/memos/graph_dbs/factory.py @@ -4,6 +4,7 @@ from memos.graph_dbs.base import BaseGraphDB from memos.graph_dbs.neo4j import Neo4jGraphDB from memos.graph_dbs.neo4j_community import Neo4jCommunityGraphDB +from memos.graph_dbs.opengauss import OpenGaussGraphDB from memos.graph_dbs.polardb import PolarDBGraphDB from memos.graph_dbs.postgres import PostgresGraphDB @@ -16,6 +17,7 @@ class GraphStoreFactory(BaseGraphDB): "neo4j-community": Neo4jCommunityGraphDB, "polardb": PolarDBGraphDB, "postgres": PostgresGraphDB, + "opengauss": OpenGaussGraphDB, } @classmethod diff --git a/src/memos/graph_dbs/opengauss.py b/src/memos/graph_dbs/opengauss.py new file mode 100644 index 000000000..1ca7c24f1 --- /dev/null +++ b/src/memos/graph_dbs/opengauss.py @@ -0,0 +1,2234 @@ +"""openGauss enterprise AGE backend, following the existing GraphDB provider layout. + +The provider owns its connection pool, graph operations and filter builders. +Consumer capabilities are validated separately from factory registration. +""" + +import json +import math +import re +import threading + +from collections.abc import Iterator +from contextlib import contextmanager +from datetime import datetime, timezone +from typing import Any + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.dependency import require_python_package +from memos.exceptions import GraphDBError +from memos.graph_dbs.base import BaseGraphDB +from memos.log import get_logger + + +logger = get_logger(__name__) +_TABLE = '"memos_graph"."Memory"' +# This AGE build emits fractional floats as .8 / -.2. Protect quoted JSON +# strings while restoring the leading zero needed by a standard JSON parser. +_AGE_JSON_TOKEN = re.compile(r'"(?:[^"\\]|\\.)*"|(?(b:Memory) WHERE " + "(a.id = $id AND a.user_name = $scope) OR " + "(b.id = $id AND b.user_name = $scope) DELETE r" +) + + +_ARRAY_FIELDS = {"tags", "sources", "file_ids", "related_id", "message_ids"} +_ROOT_FIELDS = { + "id", + "key", + "tags", + "type", + "usage", + "memory", + "status", + "sources", + "user_id", + "graph_id", + "user_name", + "background", + "confidence", + "created_at", + "session_id", + "updated_at", + "memory_type", + "node_type", + "info", + "source", + "file_ids", + "project_id", + "manager_user_id", + "delete_time", + "delete_record_id", + "related_id", + "message_ids", +} +_OPERATORS = {"gt": ">", "gte": ">=", "lt": "<", "lte": "<="} + + +def _encode_filter_value(value: Any) -> str: + try: + encoded = json.dumps(value, ensure_ascii=False, allow_nan=False) + if json.loads(encoded) != value: + raise ValueError("Filter values must be JSON values with string object keys") + return encoded + except (TypeError, ValueError) as exc: + raise ValueError("Invalid JSON filter value") from exc + + +class OpenGaussGraphDB(BaseGraphDB): + """Scoped AGE graph backend with an owned pool and parameter-bound filters.""" + + @require_python_package( + import_name="psycopg2", + install_command="pip install psycopg2-binary", + install_link="https://pypi.org/project/psycopg2-binary/", + ) + def __init__(self, config: OpenGaussGraphDBConfig) -> None: + import psycopg2.pool + + self.config = config.model_copy(deep=True) + self.db_name = config.db_name + self.user_name = config.user_name + self._semaphore = threading.BoundedSemaphore(config.maxconn) + self._condition = threading.Condition() + self._active = 0 + self._closed = False + self._pool_closed = False + try: + self.connection_pool = psycopg2.pool.ThreadedConnectionPool( + config.minconn, + config.maxconn, + host=config.host, + port=config.port, + dbname=config.db_name, + user=config.user, + password=config.password, + connect_timeout=config.connect_timeout, + application_name="memos-opengauss", + ) + except psycopg2.Error as exc: + raise GraphDBError("Could not open the openGauss connection pool") from exc + + try: + self._init_schema() + except BaseException: + try: + self.close() + except GraphDBError: + logger.warning("openGauss pool cleanup failed after initialization failure") + raise + + def _ensure_open(self) -> None: + """Reject operations after close, including operations on empty inputs.""" + with self._condition: + if self._closed: + raise GraphDBError("The openGauss connection pool is closed") + + @contextmanager + def _get_connection(self, *, snapshot: bool = False) -> Iterator[Any]: + """Prepare one owned session, commit once, and always release admission.""" + import psycopg2 + + self._ensure_open() + if not self._semaphore.acquire(timeout=self.config.connection_wait_timeout): + raise GraphDBError("openGauss connection wait timed out") + with self._condition: + if self._closed: + self._semaphore.release() + raise GraphDBError("The openGauss connection pool is closed") + self._active += 1 + connection = None + failed = discard = False + try: + connection = self.connection_pool.getconn() + if connection.closed: + raise GraphDBError("The openGauss pool returned a closed connection") + connection.rollback() + connection.set_session( + isolation_level="REPEATABLE READ" if snapshot else "READ COMMITTED", + readonly=snapshot, + autocommit=False, + ) + with connection.cursor() as cursor: + cursor.execute("RESET ALL") + # Public type input loads AGE hooks without privileged LOAD. + cursor.execute("SELECT 'null'::ag_catalog.agtype") + cursor.execute("SET search_path = ag_catalog, public") + yield connection + connection.commit() + except BaseException as exc: + failed = True + discard = isinstance(exc, psycopg2.OperationalError | psycopg2.InterfaceError) + if connection is not None: + try: + connection.rollback() + except psycopg2.Error: + discard = True + logger.warning("openGauss rollback failed; discarding the connection") + if isinstance(exc, psycopg2.Error): + raise GraphDBError( + "openGauss transaction failed; the write was not retried" + ) from exc + raise + finally: + try: + if connection is not None: + self._return_connection(connection, discard, failed) + finally: + with self._condition: + self._active -= 1 + if self._closed and not self._active: + try: + self._close_pool() + except GraphDBError: + logger.warning( + "openGauss pool shutdown failed during transaction cleanup" + ) + self._condition.notify_all() + self._semaphore.release() + + @contextmanager + def _cursor( + self, *, utc: bool = False, lock: int | None = None, snapshot: bool = False + ) -> Iterator[Any]: + """Borrow one transactional cursor, optionally preparing UTC and a graph lock.""" + transaction = self._get_connection(snapshot=True) if snapshot else self._get_connection() + with transaction as connection, connection.cursor() as cursor: + if utc: + cursor.execute("SET LOCAL TIME ZONE 'UTC'") + if lock is not None: + cursor.execute("SELECT pg_advisory_xact_lock(621503, %(lock)s)", {"lock": lock}) + yield cursor + + def _return_connection(self, connection: Any, discard: bool, failed: bool) -> None: + import psycopg2 + + try: + self.connection_pool.putconn(connection, close=discard or bool(connection.closed)) + except psycopg2.Error as exc: + # A failed return may leave the driver's checkout bookkeeping dirty. + # Stop admission and drain owners before closing the entire pool. + with self._condition: + self._closed = True + try: + connection.close() + except psycopg2.Error: + logger.warning("openGauss connection close failed during pool cleanup") + if not failed: + raise GraphDBError( + "Could not return the openGauss connection; the transaction may already be committed" + ) from exc + logger.warning("openGauss connection return failed during error cleanup") + + def close(self) -> None: + """Reject new borrowers, wait for owners, then close; successful close is idempotent.""" + with self._condition: + self._closed = True + while self._active: + self._condition.wait() + self._close_pool() + + def _close_pool(self) -> None: + import psycopg2 + + if not self._pool_closed: + try: + self.connection_pool.closeall() + except psycopg2.Error as exc: + raise GraphDBError("Could not close the openGauss connection pool") from exc + self._pool_closed = True + + def _init_schema(self) -> None: + with self._cursor(lock=1) as cursor: + cursor.execute( + "SELECT oid FROM ag_catalog.ag_graph WHERE name = %(graph)s", + {"graph": "memos_graph"}, + ) + if cursor.fetchone() is None: + cursor.execute( + "SELECT ag_catalog.create_graph(%(graph)s)", {"graph": "memos_graph"} + ) + cursor.execute( + "SELECT l.kind, n.nspname, c.relname FROM ag_catalog.ag_label l " + "JOIN ag_catalog.ag_graph g ON g.oid = l.graph " + "JOIN pg_catalog.pg_class c ON c.oid = l.relation " + "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " + "WHERE g.name = %(graph)s AND l.name = %(label)s", + {"graph": "memos_graph", "label": "Memory"}, + ) + label = cursor.fetchone() + if label is None: + cursor.execute( + "SELECT ag_catalog.create_vlabel(%(graph)s, %(label)s)", + {"graph": "memos_graph", "label": "Memory"}, + ) + cursor.execute( + f"ALTER TABLE {_TABLE} ADD COLUMN scope text NOT NULL, " + "ADD COLUMN business_id text NOT NULL, ADD COLUMN memory text NOT NULL, " + f"ADD COLUMN embedding pg_catalog.vector({self.config.embedding_dimension}), " + "ADD UNIQUE(scope, business_id)" + ) + elif label != ("v", "memos_graph", "Memory"): + raise GraphDBError("The openGauss Memory label is incompatible") + cursor.execute( + "SELECT attname, pg_catalog.format_type(atttypid, atttypmod), attnotnull, " + "atthasdef, attnum FROM pg_catalog.pg_attribute " + "WHERE attrelid = %(table)s::regclass AND attnum > 0 AND NOT attisdropped ORDER BY attnum", + {"table": _TABLE}, + ) + expected = [ + ("id", "graphid", True, True, 1), + ("properties", "agtype", True, True, 2), + ("scope", "text", True, False, 3), + ("business_id", "text", True, False, 4), + ("memory", "text", True, False, 5), + ("embedding", f"vector({self.config.embedding_dimension})", False, False, 6), + ] + if cursor.fetchall() != expected: + raise GraphDBError("The openGauss Memory columns are incompatible") + cursor.execute( + "SELECT 1 FROM pg_catalog.pg_index WHERE indrelid = %(table)s::regclass " + "AND indisunique AND indisvalid AND indimmediate AND indpred IS NULL " + "AND indexprs IS NULL AND indkey::text = '3 4'", + {"table": _TABLE}, + ) + if cursor.fetchone() is None: + raise GraphDBError("The openGauss Memory unique constraint is incompatible") + self._init_vector_index(cursor) + self._init_fulltext_index(cursor) + + def _init_fulltext_index(self, cursor: Any, create: bool = True) -> None: + """Validate the owned native text index, creating it only during initialization.""" + inspect_index = ( + "SELECT tn.nspname, t.relname, am.amname, opc.opcname, i.indkey::text, " + "i.indisvalid, i.indisready, i.indpred, i.indexprs, idx.reloptions " + "FROM pg_catalog.pg_class idx " + "JOIN pg_catalog.pg_namespace n ON n.oid=idx.relnamespace " + "JOIN pg_catalog.pg_index i ON i.indexrelid=idx.oid " + "JOIN pg_catalog.pg_class t ON t.oid=i.indrelid " + "JOIN pg_catalog.pg_namespace tn ON tn.oid=t.relnamespace " + "JOIN pg_catalog.pg_am am ON am.oid=idx.relam " + "JOIN pg_catalog.pg_opclass opc ON opc.oid=i.indclass[0] " + "WHERE n.nspname='memos_graph' AND idx.relname='memory_fulltext_bm25'" + ) + cursor.execute(inspect_index) + index = cursor.fetchone() + if index is None and create: + cursor.execute(f"CREATE INDEX memory_fulltext_bm25 ON {_TABLE} USING bm25 (memory)") + cursor.execute(inspect_index) + index = cursor.fetchone() + if index != ( + "memos_graph", + "Memory", + "bm25", + "bm25_text_ops", + "5", + True, + True, + None, + None, + ["dict_path=DEFAULT"], + ): + raise GraphDBError("The openGauss fulltext index is missing or incompatible") + + def _rebuild_fulltext_index(self) -> None: + """Repair the known failed-write index state without replaying any write.""" + with self._cursor(lock=1) as cursor: + self._init_fulltext_index(cursor, create=False) + cursor.execute("REINDEX INDEX memos_graph.memory_fulltext_bm25") + + def _init_vector_index(self, cursor: Any) -> None: + """Keep full precision above the HNSW dimension limit using exact retrieval.""" + if self.config.embedding_dimension > 2000: + return + inspect_index = ( + "SELECT tn.nspname, t.relname, am.amname, opc.opcname, i.indkey::text, " + "i.indisvalid, i.indisready, i.indpred, i.indexprs " + "FROM pg_catalog.pg_class idx " + "JOIN pg_catalog.pg_namespace n ON n.oid=idx.relnamespace " + "JOIN pg_catalog.pg_index i ON i.indexrelid=idx.oid " + "JOIN pg_catalog.pg_class t ON t.oid=i.indrelid " + "JOIN pg_catalog.pg_namespace tn ON tn.oid=t.relnamespace " + "JOIN pg_catalog.pg_am am ON am.oid=idx.relam " + "JOIN pg_catalog.pg_opclass opc ON opc.oid=i.indclass[0] " + "WHERE n.nspname='memos_graph' AND idx.relname='memory_embedding_hnsw'" + ) + cursor.execute(inspect_index) + index = cursor.fetchone() + if index is None: + cursor.execute( + f"CREATE INDEX memory_embedding_hnsw ON {_TABLE} " + "USING hnsw (embedding pg_catalog.vector_cosine_ops)" + ) + cursor.execute(inspect_index) + index = cursor.fetchone() + if index != ( + "memos_graph", + "Memory", + "hnsw", + "vector_cosine_ops", + "6", + True, + True, + None, + None, + ): + raise GraphDBError("The openGauss vector index is incompatible") + + def _scope(self, user_name: str | None) -> str: + scope = self.config.user_name if user_name is None else user_name + if scope is None and self.config.use_multi_db: + return "" + if not isinstance(scope, str) or not scope.strip(): + raise ValueError("user_name must be a nonblank string") + return scope + + @staticmethod + def _validate_id(node_id: str) -> None: + if not isinstance(node_id, str) or not node_id.strip(): + raise ValueError("id must be a nonblank string") + + def _encode_node( + self, node_id: str, memory: str, metadata: dict[str, Any], scope: str + ) -> tuple[str, str, str, str | None, str]: + self._validate_id(node_id) + if not isinstance(memory, str): + raise ValueError("memory must be a string") + if not isinstance(metadata, dict): + raise ValueError("metadata must be a JSON object") + if "id" in metadata or "memory" in metadata: + raise ValueError("metadata cannot override id or memory") + if "user_name" in metadata and metadata["user_name"] != scope: + raise ValueError("metadata user_name must match the operation scope") + properties = {**metadata, "id": node_id, "memory": memory, "user_name": scope} + embedding = properties.pop("embedding", None) + if embedding is not None: + if not isinstance(embedding, list): + raise ValueError("embedding must be a list of finite numbers") + if embedding: + if len(embedding) != self.config.embedding_dimension: + raise ValueError("embedding must match embedding_dimension") + try: + if any( + type(value) not in (int, float) or not math.isfinite(value) + for value in embedding + ): + raise ValueError("embedding must contain finite numbers, excluding bool") + except OverflowError as exc: + raise ValueError( + "embedding numbers must be representable as finite floats" + ) from exc + try: + payload = json.dumps(properties, ensure_ascii=False, allow_nan=False) + except (TypeError, ValueError) as exc: + raise ValueError("metadata must contain valid JSON values") from exc + if json.loads(payload) != properties: + raise ValueError("metadata must contain JSON objects with string keys and JSON lists") + vector = json.dumps(embedding) if embedding else None + return scope, node_id, memory, vector, payload + + def add_node( + self, id: str, memory: str, metadata: dict[str, Any], user_name: str | None = None + ) -> None: + """Upsert one identity without replacing its native graphid.""" + self.add_nodes_batch([{"id": id, "memory": memory, "metadata": metadata}], user_name) + + def add_nodes_batch(self, nodes: list[dict[str, Any]], user_name: str | None = None) -> None: + """Prepare the entire batch before borrowing; SQL succeeds or rolls back together.""" + self._ensure_open() + scope = self._scope(user_name) + if not isinstance(nodes, list): + raise ValueError("nodes must be a list") + rows = [] + for node in nodes: + if not isinstance(node, dict) or "id" not in node or "memory" not in node: + raise ValueError("each node must contain id and memory") + rows.append( + self._encode_node(node["id"], node["memory"], node.get("metadata", {}), scope) + ) + if rows: + with self._cursor() as cursor: + for owner, node_id, memory, embedding, properties in rows: + cursor.execute( + _UPSERT, + { + "scope": owner, + "id": node_id, + "memory": memory, + "embedding": embedding, + "properties": properties, + }, + ) + + @staticmethod + def _decode_embedding(value: Any) -> list[float] | None: + """Decode the driver's vector value without hiding damaged data.""" + if value is None: + return None + try: + if isinstance(value, str): + value = json.loads(value) + if not isinstance(value, list | tuple) or any( + type(item) not in (int, float) or not math.isfinite(item) for item in value + ): + raise ValueError("Invalid vector values") + return list(value) + except (TypeError, ValueError, OverflowError) as exc: + raise GraphDBError("Invalid stored openGauss embedding") from exc + + @staticmethod + def _decode_node(row: tuple, scope: str, include_embedding: bool) -> dict[str, Any]: + node_id, memory, payload, vector = row + try: + payload = _AGE_JSON_TOKEN.sub( + lambda token: token[0] if token[0].startswith('"') else f"{token[1]}0.{token[2]}", + payload, + ) + properties = json.loads(payload) + except (TypeError, ValueError) as exc: + raise GraphDBError("Invalid AGE node properties") from exc + if not isinstance(properties, dict) or ( + properties.get("id"), + properties.get("memory"), + properties.get("user_name"), + ) != (node_id, memory, scope): + raise GraphDBError("Inconsistent AGE node properties and retrieval columns") + for key in ("id", "memory", "user_name"): + del properties[key] + if include_embedding: + properties["embedding"] = OpenGaussGraphDB._decode_embedding(vector) + return {"id": node_id, "memory": memory, "metadata": properties} + + def get_nodes( + self, + ids: list[str], + include_embedding: bool = False, + user_name: str | None = None, + cube_name: str | None = None, + ) -> list[dict[str, Any]]: + """Return existing identities once, in first-requested order, within one scope.""" + self._ensure_open() + scope = self._scope(cube_name if cube_name is not None else user_name) + if not isinstance(ids, list): + raise ValueError("ids must be a list") + for node_id in ids: + self._validate_id(node_id) + if not isinstance(include_embedding, bool): + raise ValueError("include_embedding must be a bool") + ids = list(dict.fromkeys(ids)) + if not ids: + return [] + with self._cursor() as cursor: + found = self._read_nodes(cursor, ids, scope, include_embedding) + return [found[node_id] for node_id in ids if node_id in found] + + def _read_nodes( + self, cursor: Any, ids: list[str], scope: str, include_embedding: bool = False + ) -> dict[str, dict[str, Any]]: + vector_column = "embedding::text" if include_embedding else "NULL" + cursor.execute( + f"SELECT business_id, memory, properties, {vector_column} FROM ONLY {_TABLE} " + "WHERE scope = %(scope)s AND business_id = ANY(%(ids)s)", + {"scope": scope, "ids": ids}, + ) + return { + row[0]: self._decode_node(row, scope, include_embedding) for row in cursor.fetchall() + } + + def get_node( + self, + id: str, + include_embedding: bool = False, + user_name: str | None = None, + cube_name: str | None = None, + ) -> dict[str, Any] | None: + """Read an identity; a missing node is None.""" + nodes = self.get_nodes([id], include_embedding, user_name, cube_name) + return nodes[0] if nodes else None + + def search_by_embedding( + self, + vector: list[float], + top_k: int = 5, + return_fields: list[str] | None = None, + *, + scope: str | None = None, + status: str | None = None, + threshold: float | None = None, + search_filter: dict | None = None, + filter: dict | None = None, + user_name: str | None = None, + cube_name: str | None = None, + knowledgebase_ids: list[str] | None = None, + include_embedding: bool = False, + ) -> list[dict]: + """Recall scoped DataVec candidates with normalized cosine scores. + + HNSW supports at most 2000 dimensions. An underfilled candidate query is + replaced by one exact, identically filtered query. Higher dimensions use + exact search directly. Separate reads retain READ COMMITTED semantics. + """ + self._ensure_open() + if not isinstance(vector, list) or len(vector) != self.config.embedding_dimension: + raise ValueError("vector must be a list matching embedding_dimension") + try: + if any(type(value) not in (int, float) or not math.isfinite(value) for value in vector): + raise ValueError("vector must contain finite numbers, excluding bool") + scale = max(abs(float(value)) for value in vector) + if not scale: + raise ValueError("Cosine query vector must be nonzero") + query_vector = json.dumps([value / scale for value in vector]) + except OverflowError as exc: + raise ValueError("vector numbers must be representable as finite floats") from exc + if type(top_k) is not int or top_k < 0: + raise ValueError("top_k must be a nonnegative integer") + if threshold is not None and ( + type(threshold) not in (int, float) or not 0 <= threshold <= 1 + ): + raise ValueError("threshold must be a finite number between zero and one") + if not isinstance(include_embedding, bool): + raise ValueError("include_embedding must be a bool") + if return_fields is not None and ( + not isinstance(return_fields, list) + or any( + not isinstance(field, str) or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", field) + for field in return_fields + ) + ): + raise ValueError("return_fields must contain valid field names") + fields = list(dict.fromkeys(return_fields or [])) + if include_embedding and "embedding" not in fields: + fields.append("embedding") + with_embedding = "embedding" in fields + scopes = self._filter_scopes( + cube_name if cube_name is not None else user_name, knowledgebase_ids + ) + if search_filter is not None and not isinstance(search_filter, dict): + raise ValueError("search_filter must be an object") + metadata = [{"field": key, "value": value} for key, value in (search_filter or {}).items()] + if scope is not None: + self._validate_id(scope) + metadata.append({"field": "memory_type", "value": scope}) + metadata_sql, metadata_params = self._build_metadata_conditions(metadata) + filter_sql, filter_params = self._build_filter_where_clause(filter) + status_sql, status_params = self._filter_status("activated" if status is None else status) + if top_k == 0: + return [] + vector_column = "embedding::text" if with_embedding else "NULL" + query = ( + f"SELECT business_id, memory, properties, {vector_column}, scope, " + f"embedding <=> %(vector)s::pg_catalog.vector FROM ONLY {_TABLE} " + "WHERE scope = ANY(%(scopes)s) AND embedding IS NOT NULL AND vector_norm(embedding) > 0 " + f"AND ({metadata_sql}) AND ({filter_sql}) AND ({status_sql}) " + "ORDER BY (embedding <=> %(vector)s::pg_catalog.vector)" + ) + query_params = self._merge_bindings( + metadata_params, + filter_params, + status_params, + {"vector": query_vector, "scopes": scopes, "top_k": top_k}, + ) + with self._cursor(utc=True) as cursor: + if self.config.embedding_dimension <= 2000: + cursor.execute("SET LOCAL hnsw_ef_search = 100") + cursor.execute(query + " LIMIT %(top_k)s", query_params) + rows = cursor.fetchall() + else: + rows = [] + if self.config.embedding_dimension > 2000 or len(rows) < top_k: + cursor.execute(query + " + 0, scope, business_id LIMIT %(top_k)s", query_params) + rows = cursor.fetchall() + ranked = [] + for row in rows: + try: + distance = float(row[5]) + if not 0 <= distance <= 2: + raise ValueError("Distance outside cosine range") + except (TypeError, ValueError, OverflowError) as exc: + raise GraphDBError("Invalid openGauss cosine distance") from exc + score = 1 - distance / 2 + node = self._decode_node(row[:4], row[4], with_embedding) + item = {"id": node["id"], "score": score} + for field in fields: + if field in ("id", "score"): + continue + if field == "user_name": + item[field] = row[4] + elif field == "memory": + item[field] = node["memory"] + else: + item[field] = node["metadata"].get(field) + if threshold is None or score >= threshold: + ranked.append((score, row[4], node["id"], item)) + ranked.sort(key=lambda hit: (-hit[0], hit[1], hit[2])) + return [hit[3] for hit in ranked[:top_k]] + + def _query_fulltext(self, query: str, params: dict[str, Any]) -> list[tuple]: + with self._cursor(utc=True) as cursor: + # TAAT scores all term candidates; SQL LIMIT can exceed bm25_topk's + # separate 200 maximum. Pin scoring independently of role defaults. + cursor.execute("SET LOCAL enable_bm25_taat = on") + cursor.execute("SET LOCAL bm25_k1 = 1.2") + cursor.execute("SET LOCAL bm25_b = 0.75") + cursor.execute("SET LOCAL enable_bm25_global_idf = off") + cursor.execute(query, params) + return cursor.fetchall() + + def search_by_fulltext( + self, + query_words: list[str], + top_k: int = 10, + scope: str | None = None, + status: str | None = None, + threshold: float | None = None, + search_filter: dict | None = None, + user_name: str | None = None, + filter: dict | None = None, + knowledgebase_ids: list[str] | None = None, + tsvector_field: str = "properties_tsvector_zh", + tsquery_config: str | None = "jiebacfg", + return_fields: list[str] | None = None, + *, + cube_name: str | None = None, + include_embedding: bool = False, + ) -> list[dict]: + """Return native BM25 OR matches and raw scores within the requested scopes. + + The legacy tsquery aliases select the native default tokenizer. Terms + are natural text, not executable tsquery expressions. One recognized + failed-write index error permits REINDEX and a fresh read, at most once. + """ + self._ensure_open() + if not isinstance(query_words, list) or any( + not isinstance(word, str) for word in query_words + ): + raise ValueError("query_words must be a list of strings") + terms = [] + for word in query_words: + term = word.strip() + if len(term) >= 2 and term[0] == term[-1] and term[0] in ("'", '"'): + quote = term[0] + term = term[1:-1] + if quote == "'": + term = term.replace("''", "'") + term = term.strip() + if term and term not in terms: + terms.append(term) + if type(top_k) is not int or top_k < 0: + raise ValueError("top_k must be a nonnegative integer") + if threshold is not None: + try: + valid_threshold = ( + type(threshold) in (int, float) and math.isfinite(threshold) and threshold >= 0 + ) + except OverflowError: + valid_threshold = False + if not valid_threshold: + raise ValueError("threshold must be a finite nonnegative number") + if tsvector_field not in ("properties_tsvector_zh", "memory") or tsquery_config not in ( + None, + "jiebacfg", + "jiebaqry", + ): + raise ValueError("Only native BM25 and the documented fulltext aliases are supported") + if not isinstance(include_embedding, bool): + raise ValueError("include_embedding must be a bool") + if return_fields is not None and ( + not isinstance(return_fields, list) + or any( + not isinstance(field, str) or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", field) + for field in return_fields + ) + ): + raise ValueError("return_fields must contain valid field names") + fields = list(dict.fromkeys(return_fields or [])) + if include_embedding and "embedding" not in fields: + fields.append("embedding") + with_embedding = "embedding" in fields + scopes = self._filter_scopes( + cube_name if cube_name is not None else user_name, knowledgebase_ids + ) + if search_filter is not None and not isinstance(search_filter, dict): + raise ValueError("search_filter must be an object") + metadata = [{"field": key, "value": value} for key, value in (search_filter or {}).items()] + if scope is not None: + self._validate_id(scope) + metadata.append({"field": "memory_type", "value": scope}) + metadata_sql, metadata_params = self._build_metadata_conditions(metadata) + filter_sql, filter_params = self._build_filter_where_clause(filter) + status_sql, status_params = self._filter_status("activated" if status is None else status) + if not terms or top_k == 0: + return [] + query_text = " ".join(terms) + vector_column = "embedding::text" if with_embedding else "NULL" + query = ( + f"SELECT business_id, memory, properties, {vector_column}, scope, memory <&> %(query)s " + f"FROM ONLY {_TABLE} WHERE scope = ANY(%(scopes)s) " + f"AND ({metadata_sql}) AND ({filter_sql}) AND ({status_sql}) " + "ORDER BY memory <&> %(query)s DESC LIMIT %(top_k)s" + ) + query_params = self._merge_bindings( + metadata_params, + filter_params, + status_params, + {"query": query_text, "scopes": scopes, "top_k": top_k}, + ) + try: + rows = self._query_fulltext(query, query_params) + except GraphDBError as exc: + import psycopg2 + + cause = exc.__cause__ + if ( + not isinstance(cause, psycopg2.Error) + or cause.pgcode != "42809" + or cause.diag.message_primary + != ( + "Last batch insert document failed, scanned score maybe affected, " + "please reindex or recreate bm25 index [memory_fulltext_bm25]." + ) + ): + raise + self._rebuild_fulltext_index() + rows = self._query_fulltext(query, query_params) + ranked = [] + for row in rows: + try: + score = float(row[5]) + if not math.isfinite(score) or score < 0: + raise ValueError("Invalid BM25 score") + except (TypeError, ValueError, OverflowError) as exc: + raise GraphDBError("Invalid openGauss BM25 score") from exc + node = self._decode_node(row[:4], row[4], with_embedding) + item = {"id": node["id"], "score": score} + for field in fields: + if field in ("id", "score"): + continue + if field == "user_name": + item[field] = row[4] + elif field == "memory": + item[field] = node["memory"] + else: + item[field] = node["metadata"].get(field) + if threshold is None or score >= threshold: + ranked.append((score, row[4], node["id"], item)) + ranked.sort(key=lambda hit: (-hit[0], hit[1], hit[2])) + return [hit[3] for hit in ranked[:top_k]] + + def search_by_keywords_like( + self, + query_word: str, + scope: str | None = None, + status: str | None = None, + search_filter: dict | None = None, + user_name: str | None = None, + filter: dict | None = None, + knowledgebase_ids: list[str] | None = None, + return_fields: list[str] | None = None, + **kwargs: Any, + ) -> list[dict[str, Any]]: + """Return every literal SQL LIKE match, ordered by Cube and business ID.""" + self._ensure_open() + if kwargs: + raise ValueError("Unknown search_by_keywords_like arguments") + if not isinstance(query_word, str) or "\0" in query_word: + raise ValueError("query_word must be a string without NUL") + if (len(query_word) - len(query_word.rstrip("\\"))) % 2: + raise ValueError("query_word cannot end with an unpaired LIKE escape") + if return_fields is not None and ( + not isinstance(return_fields, list) + or any( + not isinstance(field, str) or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", field) + for field in return_fields + ) + ): + raise ValueError("return_fields must contain valid field names") + fields = list(dict.fromkeys(return_fields or [])) + scopes = self._filter_scopes(user_name, knowledgebase_ids) + if len(scopes) > 1 and "user_name" not in fields: + fields.append("user_name") + if search_filter is not None and not isinstance(search_filter, dict): + raise ValueError("search_filter must be an object") + metadata = [{"field": key, "value": value} for key, value in (search_filter or {}).items()] + if scope is not None: + self._validate_id(scope) + metadata.append({"field": "memory_type", "value": scope}) + predicate, params = self._join_fragments( + [ + self._build_metadata_conditions(metadata), + self._build_filter_where_clause(filter), + self._filter_status("activated" if status is None else status), + ], + " AND ", + ) + with_embedding = "embedding" in fields + vector_column = "embedding::text" if with_embedding else "NULL" + with self._cursor() as cursor: + cursor.execute( + f"SELECT business_id, memory, properties, {vector_column}, scope FROM ONLY {_TABLE} " + "WHERE scope = ANY(%(scopes)s) AND memory LIKE %(memos_like)s " + f'AND {predicate} ORDER BY scope COLLATE "C", business_id COLLATE "C"', + self._merge_bindings(params, {"scopes": scopes, "memos_like": query_word}), + ) + results = [] + for row in cursor.fetchall(): + node = self._decode_node(row[:4], row[4], with_embedding) + item = {"id": node["id"]} + for field in fields: + if field == "id": + continue + if field == "user_name": + item[field] = row[4] + elif field == "memory": + item[field] = node["memory"] + else: + item[field] = node["metadata"].get(field) + results.append(item) + return results + + def update_node(self, id: str, fields: dict[str, Any], user_name: str | None = None) -> None: + """Merge fields under a row lock, retaining unspecified values and scope.""" + self._ensure_open() + scope = self._scope(user_name) + if not isinstance(fields, dict): + raise ValueError("fields must be a JSON object") + supplied = {key: value for key, value in fields.items() if key != "memory"} + self._encode_node(id, fields.get("memory", ""), supplied, scope) + if not fields: + return + with self._cursor() as cursor: + cursor.execute( + f"SELECT business_id, memory, properties, embedding::text FROM ONLY {_TABLE} " + "WHERE scope = %(scope)s AND business_id = %(id)s FOR UPDATE", + {"scope": scope, "id": id}, + ) + row = cursor.fetchone() + if row is None: + return + node = self._decode_node(row, scope, True) + metadata = {**node["metadata"], **supplied} + _, _, memory, vector, payload = self._encode_node( + id, fields.get("memory", node["memory"]), metadata, scope + ) + cursor.execute( + f"UPDATE {_TABLE} SET memory = %(memory)s, embedding = %(embedding)s::pg_catalog.vector, " + "properties = %(properties)s::ag_catalog.agtype WHERE scope = %(scope)s AND business_id = %(id)s", + { + "memory": memory, + "embedding": vector, + "properties": payload, + "scope": scope, + "id": id, + }, + ) + + @staticmethod + def _validate_edge_type(edge_type: str, allow_any: bool = True) -> None: + if ( + not isinstance(edge_type, str) + or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]{0,62}", edge_type) + or (edge_type == "ANY" and not allow_any) + ): + raise ValueError("Invalid relationship type") + + @staticmethod + def _cypher( + cursor: Any, body: str, parameters: dict[str, Any], columns: str = "result agtype" + ) -> list[tuple]: + # AGE requires a literal query body and a prepared agtype parameter. + # This connection belongs exclusively to the pool borrower. Clearing + # first also bounds statements left behind by an aborted transaction. + cursor.execute("DEALLOCATE ALL") + cursor.execute( + "PREPARE memos_age (ag_catalog.agtype) AS SELECT * FROM " + f"ag_catalog.cypher('memos_graph', $memos${body}$memos$, $1) AS ({columns})" + ) + cursor.execute( + "EXECUTE memos_age(%(parameters)s::ag_catalog.agtype)", + {"parameters": json.dumps(parameters, ensure_ascii=False)}, + ) + rows = cursor.fetchall() + cursor.execute("DEALLOCATE memos_age") + return rows + + @staticmethod + def _lock_graph_nodes(cursor: Any, ids: list[str], scope: str) -> set[str]: + # The caller holds the graph advisory lock before taking these row + # locks, which coordinate deletion with existing node upserts. + cursor.execute( + f"SELECT business_id FROM ONLY {_TABLE} " + "WHERE scope = %(scope)s AND business_id = ANY(%(ids)s) ORDER BY business_id FOR UPDATE", + {"scope": scope, "ids": sorted(set(ids))}, + ) + return {row[0] for row in cursor.fetchall()} + + def delete_node(self, id: str, user_name: str | None = None) -> None: + """Atomically hard-delete one scoped node and all its incident native edges.""" + self._ensure_open() + scope = self._scope(user_name) + self._validate_id(id) + with self._cursor(lock=2) as cursor: + if not self._lock_graph_nodes(cursor, [id], scope): + return + self._cypher( + cursor, + _DELETE_INCIDENT_EDGES, + {"id": id, "scope": scope}, + ) + # Native AGE node deletion bypasses SQL row triggers and DELETE + # privileges in this build. Keep node mutations on the SQL label + # path so projection, constraints and failure handling stay aligned. + cursor.execute( + _DELETE_NODE, + {"scope": scope, "id": id}, + ) + + def _change_edge( + self, source: str, target: str, edge_type: str, user_name: str | None, create: bool + ) -> None: + self._ensure_open() + scope = self._scope(user_name) + self._validate_id(source) + self._validate_id(target) + self._validate_edge_type(edge_type, allow_any=False) + endpoints = {source, target} + with self._cursor(lock=2) as cursor: + if self._lock_graph_nodes(cursor, list(endpoints), scope) != endpoints: + return + conditions = ( + "a.id = $source AND b.id = $target " + "AND a.user_name = $scope AND b.user_name = $scope " + ) + body = ( + f"MATCH (a:Memory), (b:Memory) WHERE {conditions}MERGE (a)-[:{edge_type}]->(b)" + if create + else f"MATCH (a:Memory)-[r]->(b:Memory) WHERE {conditions}" + "AND type(r) = $type DELETE r" + ) + parameters = {"source": source, "target": target, "scope": scope} + if not create: + parameters["type"] = edge_type + self._cypher(cursor, body, parameters) + + def add_edge( + self, source_id: str, target_id: str, type: str, user_name: str | None = None + ) -> None: + """Merge a directed typed edge; a missing scoped endpoint is a no-op.""" + self._change_edge(source_id, target_id, type, user_name, create=True) + + def delete_edge( + self, source_id: str, target_id: str, type: str, user_name: str | None = None + ) -> None: + """Remove only the given direction and type, retaining parallel edges.""" + self._change_edge(source_id, target_id, type, user_name, create=False) + + def get_edges( + self, id: str, type: str = "ANY", direction: str = "ANY", user_name: str | None = None + ) -> list[dict[str, str]]: + """Return sorted, unique from/to/type records, with both endpoints scoped.""" + self._ensure_open() + scope = self._scope(user_name) + self._validate_id(id) + self._validate_edge_type(type) + if direction not in ("OUTGOING", "INCOMING", "ANY"): + raise ValueError("Invalid edge direction") + endpoint = { + "OUTGOING": "a.id = $id", + "INCOMING": "b.id = $id", + "ANY": "(a.id = $id OR b.id = $id)", + }[direction] + condition = "" if type == "ANY" else " AND type(r) = $type" + with self._cursor() as cursor: + return self._read_edges( + cursor, + endpoint + condition, + {"id": id, "scope": scope, "type": type}, + ) + + def _read_edges( + self, cursor: Any, condition: str, parameters: dict[str, Any] + ) -> list[dict[str, str]]: + rows = self._cypher( + cursor, + "MATCH (a:Memory)-[r]->(b:Memory) WHERE " + f"{condition} AND a.user_name = $scope AND b.user_name = $scope " + "RETURN a.id, b.id, type(r)", + parameters, + "source agtype, target agtype, edge_type agtype", + ) + edges = set() + try: + for row in rows: + edge = tuple(json.loads(value) for value in row) + if len(edge) != 3 or not all(isinstance(value, str) for value in edge): + raise ValueError("Invalid edge fields") + edges.add(edge) + except (TypeError, ValueError) as exc: + raise GraphDBError("Invalid AGE edge projection") from exc + return [ + {"from": source, "to": target, "type": edge_type} + for source, target, edge_type in sorted(edges) + ] + + def edge_exists( + self, + source_id: str, + target_id: str, + type: str = "ANY", + direction: str = "OUTGOING", + user_name: str | None = None, + ) -> bool: + """Check adjacency in the requested direction without crossing scopes.""" + self._ensure_open() + self._validate_id(target_id) + return any( + (edge["to"] if edge["from"] == source_id else edge["from"]) == target_id + for edge in self.get_edges(source_id, type, direction, user_name) + ) + + def get_neighbors( + self, id: str, type: str, direction: str = "out", user_name: str | None = None + ) -> list[str]: + """Return unique sorted neighboring business IDs, including self-loops.""" + self._ensure_open() + if direction not in ("in", "out", "both"): + raise ValueError("Invalid neighbor direction") + edges = self.get_edges( + id, type, {"in": "INCOMING", "out": "OUTGOING", "both": "ANY"}[direction], user_name + ) + return sorted({edge["to"] if edge["from"] == id else edge["from"] for edge in edges}) + + def get_context_chain( + self, id: str, type: str = "FOLLOWS", user_name: str | None = None + ) -> list[str]: + """Return one-hop outgoing neighbors, matching the existing context contract.""" + return self.get_neighbors(id, type=type, direction="out", user_name=user_name) + + def deduplicate_nodes(self) -> None: + """Persistent semantic deduplication is outside this backend's capabilities.""" + self._ensure_open() + raise NotImplementedError("openGauss does not support deduplicate_nodes") + + def detect_conflicts(self) -> list[tuple[str, str]]: + """Semantic conflict detection is outside this backend's capabilities.""" + self._ensure_open() + raise NotImplementedError("openGauss does not support detect_conflicts") + + def merge_nodes(self, id1: str, id2: str) -> str: + """Semantic node merging is outside this backend's capabilities.""" + self._ensure_open() + raise NotImplementedError("openGauss does not support merge_nodes") + + @staticmethod + def _validate_depth(depth: int) -> None: + if type(depth) is not int or not 1 <= depth <= 5: + raise ValueError("depth must be an integer from 1 to 5") + + def get_path( + self, source_id: str, target_id: str, max_depth: int = 3, user_name: str | None = None + ) -> list[str]: + """Find the lexicographically first shortest directed path within 1-5 hops.""" + self._ensure_open() + scope = self._scope(user_name) + self._validate_id(source_id) + self._validate_id(target_id) + self._validate_depth(max_depth) + with self._cursor() as cursor: + endpoints = {source_id, target_id} + cursor.execute( + f"SELECT business_id FROM ONLY {_TABLE} WHERE scope = %(scope)s AND business_id = ANY(%(ids)s)", + {"scope": scope, "ids": sorted(endpoints)}, + ) + if {row[0] for row in cursor.fetchall()} != endpoints: + return [] + if source_id == target_id: + return [source_id] + paths = {source_id: [source_id]} + frontier = [source_id] + for _ in range(max_depth): + edges = self._read_edges(cursor, "a.id IN $ids", {"scope": scope, "ids": frontier}) + adjacency: dict[str, set[str]] = {} + for edge in edges: + adjacency.setdefault(edge["from"], set()).add(edge["to"]) + following = [] + # Discovery order preserves lexicographic order of full paths. + for source in frontier: + for target in sorted(adjacency.get(source, set())): + if target in paths: + continue + paths[target] = [*paths[source], target] + if target == target_id: + return paths[target] + following.append(target) + frontier = following + if not frontier: + break + return [] + + def get_subgraph( + self, + center_id: str, + depth: int = 2, + center_status: str | None = "activated", + user_name: str | None = None, + ) -> dict[str, Any]: + """Return full scoped nodes and the induced edges within a two-way radius.""" + self._ensure_open() + scope = self._scope(user_name) + self._validate_id(center_id) + self._validate_depth(depth) + if center_status is not None and not isinstance(center_status, str): + raise ValueError("center_status must be a string or None") + empty = {"core_node": None, "neighbors": [], "edges": []} + with self._cursor() as cursor: + core = self._read_nodes(cursor, [center_id], scope).get(center_id) + if core is None or ( + center_status and core["metadata"].get("status", "activated") != center_status + ): + return empty + reached = {center_id} + frontier = [center_id] + for _ in range(depth): + edges = self._read_edges( + cursor, + "(a.id IN $ids OR b.id IN $ids)", + {"scope": scope, "ids": frontier}, + ) + following = {edge[key] for edge in edges for key in ("from", "to")} - reached + if not following: + break + reached.update(following) + frontier = sorted(following) + neighbors = self._read_nodes(cursor, sorted(reached - {center_id}), scope) + present = {center_id, *neighbors} + edges = self._read_edges( + cursor, + "a.id IN $ids AND b.id IN $ids", + {"scope": scope, "ids": sorted(present)}, + ) + return { + "core_node": core, + "neighbors": [neighbors[node_id] for node_id in sorted(neighbors)], + "edges": [ + {"source": edge["from"], "target": edge["to"], "type": edge["type"]} + for edge in edges + ], + } + + @staticmethod + def _validate_string_list(values: list[str], name: str) -> None: + if not isinstance(values, list) or any( + not isinstance(value, str) or not value.strip() for value in values + ): + raise ValueError(f"{name} must be a list of nonblank strings") + + @staticmethod + def _node_tags(value: Any) -> set[str]: + if value is None: + return set() + try: + tags = json.loads(value) if isinstance(value, str) else value + if not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags): + raise ValueError("Invalid tag list") + return set(tags) + except (TypeError, ValueError) as exc: + raise GraphDBError("Invalid AGE node tags") from exc + + def get_neighbors_by_tag( + self, + tags: list[str], + exclude_ids: list[str], + top_k: int = 5, + min_overlap: int = 1, + include_embedding: bool = False, + user_name: str | None = None, + ) -> list[dict[str, Any]]: + """Rank active, non-reasoning/non-working scoped nodes by unique tag overlap.""" + self._ensure_open() + scope = self._scope(user_name) + self._validate_string_list(tags, "tags") + self._validate_string_list(exclude_ids, "exclude_ids") + if type(top_k) is not int or top_k < 0: + raise ValueError("top_k must be a nonnegative integer") + if type(min_overlap) is not int or min_overlap < 1: + raise ValueError("min_overlap must be a positive integer") + if not isinstance(include_embedding, bool): + raise ValueError("include_embedding must be a bool") + if not tags or top_k == 0: + return [] + requested = set(tags) + vector_column = "embedding::text" if include_embedding else "NULL" + with self._cursor() as cursor: + cursor.execute( + f"SELECT business_id, memory, properties, {vector_column} FROM ONLY {_TABLE} " + "WHERE scope = %(scope)s AND NOT (business_id = ANY(%(ids)s))", + {"scope": scope, "ids": exclude_ids}, + ) + ranked = [] + for row in cursor.fetchall(): + node = self._decode_node(row, scope, include_embedding) + metadata = node["metadata"] + if ( + metadata.get("status", "activated") != "activated" + or metadata.get("type") == "reasoning" + or metadata.get("node_type") == "reasoning" + or metadata.get("memory_type") == "WorkingMemory" + ): + continue + overlap = len(requested & self._node_tags(metadata.get("tags"))) + if overlap >= min_overlap: + ranked.append((overlap, node)) + ranked.sort(key=lambda item: (-item[0], item[1]["id"])) + return [node for _, node in ranked[:top_k]] + + def _filter_scopes( + self, user_name: str | None, knowledgebase_ids: list[str] | None + ) -> list[str]: + scopes = [self._scope(user_name)] + if knowledgebase_ids is not None: + self._validate_string_list(knowledgebase_ids, "knowledgebase_ids") + scopes.extend(knowledgebase_ids) + return sorted(set(scopes)) + + @staticmethod + def _filter_status(status: str | None) -> tuple[str, dict[str, Any]]: + if status is None: + return "TRUE", {} + if not isinstance(status, str) or not status.strip(): + raise ValueError("status must be a nonblank string or None") + return ( + "(CASE WHEN properties ? 'status'::text THEN " + "ag_catalog.agtype_access_operator(properties, '\"status\"'::ag_catalog.agtype) " + "ELSE '\"activated\"'::ag_catalog.agtype END) = %(memos_status)s::ag_catalog.agtype", + {"memos_status": json.dumps(status)}, + ) + + def get_by_metadata( + self, + filters: list[dict[str, Any]], + user_name: str | None = None, + filter: dict | None = None, + knowledgebase_ids: list[str] | None = None, + user_name_flag: bool = True, + status: str | None = None, + ) -> list[str]: + """Read matching IDs in explicit scopes, ordered by scope then business ID.""" + self._ensure_open() + scopes = self._filter_scopes(user_name, knowledgebase_ids) + if user_name_flag is not True: + raise ValueError("Disabling scope isolation is unsupported") + metadata_sql, metadata_params = self._build_metadata_conditions(filters) + filter_sql, filter_params = self._build_filter_where_clause(filter) + status_sql, status_params = self._filter_status(status) + params = self._merge_bindings( + metadata_params, filter_params, status_params, {"scopes": scopes} + ) + with self._cursor(utc=True) as cursor: + cursor.execute( + f"SELECT business_id FROM ONLY {_TABLE} WHERE scope = ANY(%(scopes)s) " + f"AND ({metadata_sql}) AND ({filter_sql}) AND ({status_sql}) " + "ORDER BY scope, business_id", + params, + ) + return [row[0] for row in cursor.fetchall()] + + def get_all_memory_items( + self, + scope: str, + user_name: str | None = None, + include_embedding: bool = False, + filter: dict | None = None, + knowledgebase_ids: list[str] | None = None, + status: str | None = None, + ) -> list[dict[str, Any]]: + """List a memory type without a hidden cap, including explicit Dream types.""" + self._ensure_open() + scopes = self._filter_scopes(user_name, knowledgebase_ids) + self._validate_id(scope) + if not isinstance(include_embedding, bool): + raise ValueError("include_embedding must be a bool") + type_sql, type_params = self._build_metadata_conditions( + [{"field": "memory_type", "value": scope}] + ) + filter_sql, filter_params = self._build_filter_where_clause(filter) + status_sql, status_params = self._filter_status(status) + params = self._merge_bindings(type_params, filter_params, status_params, {"scopes": scopes}) + vector_column = "embedding::text" if include_embedding else "NULL" + with self._cursor(utc=True) as cursor: + cursor.execute( + f"SELECT business_id, memory, properties, {vector_column}, scope FROM ONLY {_TABLE} " + f"WHERE scope = ANY(%(scopes)s) AND ({type_sql}) AND ({filter_sql}) AND ({status_sql}) " + "ORDER BY scope, business_id", + params, + ) + nodes = [] + for row in cursor.fetchall(): + node = self._decode_node(row[:4], row[4], include_embedding) + if len(scopes) > 1: + node["metadata"]["user_name"] = row[4] + nodes.append(node) + return nodes + + def delete_node_by_prams( + self, + writable_cube_ids: list[str] | None = None, + memory_ids: list[str] | None = None, + file_ids: list[str] | None = None, + filter: dict | None = None, + ) -> int: + """Atomically hard-delete one selector in explicit or default scopes.""" + self._ensure_open() + if writable_cube_ids is None: + scopes = [self._scope(None)] + else: + self._validate_string_list(writable_cube_ids, "writable_cube_ids") + scopes = sorted(set(writable_cube_ids)) + for name, values in (("memory_ids", memory_ids), ("file_ids", file_ids)): + if values is not None: + self._validate_string_list(values, name) + predicate, params = self._build_filter_where_clause(filter) + modes = sum(bool(value) for value in (memory_ids, file_ids, filter)) + if modes > 1: + raise ValueError("Select only one of memory_ids, file_ids or filter") + if memory_ids: + predicate, params = self._build_metadata_conditions( + [{"field": "id", "op": "in", "value": memory_ids}] + ) + elif file_ids: + predicate, params = self._build_metadata_conditions( + [{"field": "file_ids", "op": "contains", "value": file_ids}] + ) + if not modes or not scopes: + return 0 + params = self._merge_bindings(params, {"scopes": scopes}) + with self._cursor(utc=True, lock=2) as cursor: + cursor.execute( + f"SELECT scope, business_id FROM ONLY {_TABLE} WHERE scope = ANY(%(scopes)s) " + f"AND ({predicate}) ORDER BY scope, business_id FOR UPDATE", + params, + ) + selected = cursor.fetchall() + deleted = 0 + for owner, node_id in selected: + self._cypher( + cursor, + _DELETE_INCIDENT_EDGES, + {"id": node_id, "scope": owner}, + ) + cursor.execute( + _DELETE_NODE, + {"scope": owner, "id": node_id}, + ) + deleted += cursor.rowcount + return deleted + + def get_grouped_counts( + self, + group_fields: list[str], + where_clause: str | None = "", + params: dict[str, Any] | None = None, + user_name: str | None = None, + ) -> list[dict[str, Any]]: + """Group all scoped rows by AGE values, with an optional trusted SQL predicate. + + Callers must supply constant SQL and bind dynamic values through params. + This interface is not a sandbox for SQL supplied by end users. + """ + self._ensure_open() + scope = self._scope(user_name) + if "\0" in scope: + raise ValueError("user_name must not contain NUL") + predicate, params = self._validate_grouping(group_fields, where_clause, params) + query, bindings = self._build_grouping_query(group_fields, predicate, params, scope) + with self._cursor() as cursor: + # AGE's numeric equality merges 1 and 1.0, but this build's hash + # aggregation does not. Keep the workaround local to this read. + cursor.execute("SET LOCAL enable_hashagg = off") + cursor.execute(query, bindings) + return self._decode_grouped_rows(group_fields, cursor.fetchall()) + + @staticmethod + def _validate_grouping( + group_fields: list[str], where_clause: str | None, params: dict[str, Any] | None + ) -> tuple[str, dict[str, Any]]: + """Validate trusted predicates and scalar user bindings before borrowing.""" + if not isinstance(group_fields, list) or not group_fields: + raise ValueError("group_fields must be a nonempty list") + for field in group_fields: + if ( + not isinstance(field, str) + or not re.fullmatch(r"(?:info\.)?[A-Za-z_][A-Za-z0-9_]*", field) + or field == "count" + ): + raise ValueError("Invalid grouping field") + if len(set(group_fields)) != len(group_fields): + raise ValueError("Grouping fields must be unique") + if where_clause is not None and not isinstance(where_clause, str): + raise ValueError("where_clause must be a string or None") + predicate = re.sub(r"(?i)^WHERE\b", "", (where_clause or "").strip(), count=1).strip() + if any(token in predicate for token in ("\0", ";", "--", "/*")): + raise ValueError("Invalid SQL predicate") + if params is None: + params = {} + if not isinstance(params, dict): + raise ValueError("params must be a dict or None") + for key, value in params.items(): + if ( + not isinstance(key, str) + or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", key) + or key.startswith("memos_") + ): + raise ValueError("Invalid or reserved SQL parameter name") + if ( + type(value) not in (type(None), bool, int, float, str) + or (type(value) is float and not math.isfinite(value)) + or (type(value) is str and "\0" in value) + ): + raise ValueError("SQL parameters must be finite scalar values without NUL") + placeholder = re.compile(r"%%|%\(([A-Za-z_][A-Za-z0-9_]*)\)s") + keys = {match[1] for match in placeholder.finditer(predicate) if match[1] is not None} + if "%" in placeholder.sub("", predicate) or keys != set(params): + raise ValueError("SQL placeholders must exactly match named parameters") + return predicate, dict(params) + + def _build_grouping_query( + self, group_fields: list[str], predicate: str, params: dict[str, Any], scope: str + ) -> tuple[str, dict[str, Any]]: + """Build deterministic AGE value grouping with a separate outer scope.""" + bindings = self._merge_bindings(params, {"memos_scope": scope}) + expressions = [] + for index, field in enumerate(group_fields): + expression, field_params = self._field_expr( + field.split("."), prefix=f"memos_group_{index}" + ) + bindings = self._merge_bindings(bindings, field_params) + expressions.append(f"{expression} AS g{index}") + groups = ", ".join(f"g{i}" for i in range(len(group_fields))) + representatives = ", ".join( + f'MIN(ag_catalog.agtype_out(g{i})::text COLLATE "C") AS r{i}' + for i in range(len(group_fields)) + ) + order = ", ".join(f'r{i} COLLATE "C" ASC NULLS LAST' for i in range(len(group_fields))) + query = ( + f"SELECT {representatives}, COUNT(*) FROM " + f"(SELECT {', '.join(expressions)} FROM ONLY {_TABLE} " + "WHERE scope = %(memos_scope)s" + + (f" AND ({predicate})" if predicate else "") + + f") AS grouped GROUP BY {groups} ORDER BY {order}" + ) + return query, bindings + + @staticmethod + def _decode_grouped_rows(group_fields: list[str], rows: list[tuple]) -> list[dict[str, Any]]: + """Decode scalar/structured AGE representatives before the transaction commits.""" + result = [] + for row in rows: + record = {} + for field, payload in zip(group_fields, row[:-1], strict=True): + try: + record[field] = ( + None + if payload is None + else json.loads( + _AGE_JSON_TOKEN.sub( + lambda token: token[0] + if token[0].startswith('"') + else f"{token[1]}0.{token[2]}", + payload, + ) + ) + ) + # Python's JSON parser accepts NaN/Infinity extensions; + # AGE results must still be finite JSON, including nested values. + json.dumps(record[field], allow_nan=False) + except (TypeError, ValueError) as exc: + raise GraphDBError("Invalid AGE grouped value") from exc + record["count"] = row[-1] + result.append(record) + return result + + def _capacity_scope(self, memory_type: str, user_name: str | None) -> str: + """Validate the type and Cube before borrowing a connection.""" + self._ensure_open() + scope = self._scope(user_name) + if ( + not isinstance(memory_type, str) + or not memory_type.strip() + or "\0" in memory_type + or "\0" in scope + ): + raise ValueError("memory_type and user_name must be nonblank strings without NUL") + return scope + + def get_memory_count(self, memory_type: str, user_name: str | None = None) -> int: + """Count occupied rows of one type, including archived and deleted rows.""" + scope = self._capacity_scope(memory_type, user_name) + kind, params = self._build_equal_condition( + ["memory_type"], memory_type, prefix="memos_type" + ) + params = self._merge_bindings(params, {"scope": scope}) + with self._cursor() as cursor: + cursor.execute( + f"SELECT COUNT(*) FROM ONLY {_TABLE} WHERE scope = %(scope)s AND {kind}", + params, + ) + return cursor.fetchone()[0] + + def node_not_exist(self, scope: str, user_name: str | None = None) -> int: + """Return integer one only when this Cube has no rows of the given type.""" + return int(self.get_memory_count(scope, user_name) == 0) + + def get_structure_optimization_candidates( + self, + scope: str, + include_embedding: bool = False, + user_name: str | None = None, + ) -> list[dict[str, Any]]: + """Read activated nodes without same-Cube PARENT adjacency in one snapshot.""" + owner = self._capacity_scope(scope, user_name) + if not isinstance(include_embedding, bool): + raise ValueError("include_embedding must be a bool") + condition, params = self._join_fragments( + [ + self._build_equal_condition(["memory_type"], scope, prefix="memos_type"), + self._build_equal_condition(["status"], "activated", prefix="memos_status"), + ], + " AND ", + ) + vector = "m.embedding::text" if include_embedding else "NULL" + # The inherited edge table also works before a PARENT label exists. + # Its tableoid identifies the AGE label; native graphids join endpoints + # without conflating equal business IDs in different Cubes. + with self._cursor() as cursor: + cursor.execute( + f"SELECT m.business_id, m.memory, m.properties, {vector} FROM ONLY {_TABLE} m " + f"WHERE m.scope = %(scope)s AND {condition} AND NOT EXISTS (" + "SELECT 1 FROM memos_graph._ag_label_edge e " + "JOIN ag_catalog.ag_label l ON l.relation = e.tableoid " + "AND l.name = 'PARENT' AND l.kind = 'e' " + f"JOIN ONLY {_TABLE} n ON n.scope = m.scope AND " + "((e.start_id = m.id AND e.end_id = n.id) OR " + "(e.end_id = m.id AND e.start_id = n.id))) " + 'ORDER BY m.business_id COLLATE "C"', + self._merge_bindings(params, {"scope": owner}), + ) + return [self._decode_node(row, owner, include_embedding) for row in cursor.fetchall()] + + def _maintenance_scope(self, user_name: str | None) -> str: + self._ensure_open() + scope = self._scope(user_name) + if "\0" in scope: + raise ValueError("user_name cannot contain NUL") + return scope + + def _maintenance_strings(self, values: list[str], name: str) -> None: + self._validate_string_list(values, name) + if any("\0" in value for value in values): + raise ValueError(f"{name} cannot contain NUL") + + def export_graph( + self, + include_embedding: bool = False, + user_name: str | None = None, + user_id: str | None = None, + page: int | None = None, + page_size: int | None = None, + filter: dict | None = None, + memory_type: list[str] | None = None, + status: list[str] | None = None, + *, + all_cubes: bool = False, + exclude_user_name: str | None = None, + **kwargs: Any, + ) -> dict[str, Any]: + """Export a consistent filtered page; cross-Cube reads require explicit opt-in.""" + self._ensure_open() + if kwargs: + raise ValueError("Unknown export_graph arguments") + if not isinstance(include_embedding, bool) or not isinstance(all_cubes, bool): + raise ValueError("include_embedding and all_cubes must be bool") + if all_cubes and user_name is not None: + raise ValueError("all_cubes and user_name are mutually exclusive") + if (page is None) != (page_size is None) or any( + type(value) is not int or value <= 0 for value in (page, page_size) if value is not None + ): + raise ValueError("page and page_size must both be positive integers or omitted") + parts = [self._build_filter_where_clause(filter)] + if exclude_user_name is not None: + if not all_cubes: + raise ValueError("exclude_user_name requires all_cubes=True") + self._maintenance_scope(exclude_user_name) + parts.append( + ("scope <> %(memos_excluded_owner)s", {"memos_excluded_owner": exclude_user_name}) + ) + if not all_cubes: + parts.append( + ("scope = %(memos_owner)s", {"memos_owner": self._maintenance_scope(user_name)}) + ) + if user_id is not None: + self._maintenance_strings([user_id], "user_id") + parts.append(self._build_equal_condition(["user_id"], user_id, prefix="memos_user")) + if memory_type is not None: + self._maintenance_strings(memory_type, "memory_type") + if memory_type: + parts.append( + self._build_filter_condition( + "memory_type", "in", memory_type, prefix="memos_types" + ) + ) + if status is None: + deleted, bound = self._build_equal_condition( + ["status"], "deleted", prefix="memos_deleted" + ) + parts.append((f"({deleted}) IS NOT TRUE", bound)) + else: + self._maintenance_strings(status, "status") + if status: + statuses = [] + for index, value in enumerate(status): + expression, bound = self._filter_status(value) + key = f"memos_status_{index}" + statuses.append( + ( + expression.replace("%(memos_status)s", f"%({key})s"), + {key: bound["memos_status"]}, + ) + ) + expression, bound = self._join_fragments(statuses, " OR ") + parts.append((f"({expression})", bound)) + predicate, params = self._join_fragments(parts, " AND ") + instant, time_params = self._text_field_expr(["created_at"], prefix="memos_created") + vector = "embedding::text" if include_embedding else "NULL" + pagination = "" + page_params = {} + if page is not None: + pagination = " LIMIT %(memos_limit)s OFFSET %(memos_offset)s" + page_params = {"memos_limit": page_size, "memos_offset": (page - 1) * page_size} + with self._cursor(utc=True, snapshot=True) as cursor: + cursor.execute(f"SELECT COUNT(*) FROM ONLY {_TABLE} WHERE {predicate}", params) + total = cursor.fetchone()[0] + cursor.execute( + f"SELECT business_id, memory, properties, {vector}, scope, id::text " + f"FROM ONLY {_TABLE} WHERE {predicate} " + f"ORDER BY {instant}::timestamptz DESC NULLS LAST, " + 'business_id COLLATE "C" DESC, scope COLLATE "C"' + pagination, + self._merge_bindings(params, time_params, page_params), + ) + rows = cursor.fetchall() + nodes = [] + for row in rows: + node = self._decode_node(row[:4], row[4], include_embedding) + node["metadata"]["user_name"] = row[4] + nodes.append(node) + edges = self._export_edges(cursor, [row[5] for row in rows], all_cubes) + return {"nodes": nodes, "edges": edges, "total_nodes": total, "total_edges": len(edges)} + + def _export_edges( + self, cursor: Any, graphids: list[str], all_cubes: bool + ) -> list[dict[str, str]]: + if not graphids: + return [] + cursor.execute( + "SELECT a.business_id, b.business_id, l.name, a.scope " + "FROM memos_graph._ag_label_edge e " + "JOIN ag_catalog.ag_label l ON l.relation = e.tableoid AND l.kind = 'e' " + f"JOIN ONLY {_TABLE} a ON a.id = e.start_id " + f"JOIN ONLY {_TABLE} b ON b.id = e.end_id AND b.scope = a.scope " + "WHERE a.id::text = ANY(%(graphids)s) AND b.id::text = ANY(%(graphids)s)", + {"graphids": graphids}, + ) + edges = set() + for source, target, kind, owner in cursor.fetchall(): + if not all(isinstance(value, str) for value in (source, target, kind, owner)): + raise GraphDBError("Invalid stored openGauss edge") + edges.add((owner, source, target, kind)) + return [ + { + "source": source, + "target": target, + "type": kind, + **({"user_name": owner} if all_cubes else {}), + } + for owner, source, target, kind in sorted(edges) + ] + + @staticmethod + def _reject_nul(value: Any) -> None: + if isinstance(value, str): + if "\0" in value: + raise ValueError("Import strings cannot contain NUL") + elif isinstance(value, dict): + for key, item in value.items(): + OpenGaussGraphDB._reject_nul(key) + OpenGaussGraphDB._reject_nul(item) + elif isinstance(value, list): + for item in value: + OpenGaussGraphDB._reject_nul(item) + + def import_graph(self, data: dict[str, Any], user_name: str | None = None) -> None: + """Atomically upsert one Cube's nodes and merge edges with existing endpoints.""" + scope = self._maintenance_scope(user_name) + if ( + not isinstance(data, dict) + or not isinstance(data.get("nodes", []), list) + or not isinstance(data.get("edges", []), list) + ): + raise ValueError("Graph data must contain nodes and edges lists") + self._reject_nul(data) + nodes = {} + for node in data.get("nodes", []): + if not isinstance(node, dict) or "id" not in node or "memory" not in node: + raise ValueError("Each import node needs id and memory") + encoded = self._encode_node(node["id"], node["memory"], node.get("metadata", {}), scope) + if node["id"] in nodes: + raise ValueError("Duplicate import node ID") + nodes[node["id"]] = encoded + edges = set() + endpoints = set() + for edge in data.get("edges", []): + if not isinstance(edge, dict) or not {"source", "target", "type"} <= edge.keys(): + raise ValueError("Each import edge needs source, target and type") + self._validate_id(edge["source"]) + self._validate_id(edge["target"]) + self._validate_edge_type(edge["type"], allow_any=False) + if edge.get("user_name", scope) != scope: + raise ValueError("Import edge user_name must match the target Cube") + edges.add((edge["source"], edge["target"], edge["type"])) + endpoints.update((edge["source"], edge["target"])) + if not nodes and not edges: + return + with self._cursor(lock=2) as cursor: + existing = self._lock_graph_nodes(cursor, sorted(set(nodes) | endpoints), scope) + if endpoints - existing - nodes.keys(): + raise GraphDBError("An import edge endpoint does not exist in the target Cube") + for key in sorted(nodes): + owner, node_id, memory, embedding, properties = nodes[key] + cursor.execute( + _UPSERT, + { + "scope": owner, + "id": node_id, + "memory": memory, + "embedding": embedding, + "properties": properties, + }, + ) + for source, target, kind in sorted(edges): + self._cypher( + cursor, + "MATCH (a:Memory), (b:Memory) WHERE a.id=$source AND b.id=$target " + "AND a.user_name=$scope AND b.user_name=$scope " + f"MERGE (a)-[:{kind}]->(b)", + {"source": source, "target": target, "scope": scope}, + ) + + def _delete_locked_nodes(self, cursor: Any, scope: str, ids: list[str]) -> int: + deleted = 0 + for offset in range(0, len(ids), 500): + batch = ids[offset : offset + 500] + self._cypher( + cursor, + "MATCH (a:Memory)-[r]->(b:Memory) WHERE " + "(a.user_name = $scope AND a.id IN $ids) OR " + "(b.user_name = $scope AND b.id IN $ids) DELETE r", + {"scope": scope, "ids": batch}, + ) + cursor.execute( + f"DELETE FROM ONLY {_TABLE} WHERE scope = %(scope)s AND business_id = ANY(%(ids)s)", + {"scope": scope, "ids": batch}, + ) + deleted += cursor.rowcount + return deleted + + def clear(self, user_name: str | None = None) -> None: + """Clear a fixed locked set in one Cube, retaining graph schema and other Cubes.""" + scope = self._maintenance_scope(user_name) + with self._cursor(lock=2) as cursor: + cursor.execute( + f"SELECT business_id FROM ONLY {_TABLE} WHERE scope = %(scope)s ORDER BY business_id FOR UPDATE", + {"scope": scope}, + ) + self._delete_locked_nodes(cursor, scope, [row[0] for row in cursor.fetchall()]) + + def drop_database(self) -> None: + """Compatibility operation: clear the configured Cube, never drop the database.""" + self.clear() + + def get_user_names_by_memory_ids(self, memory_ids: list[str]) -> dict[str, str | None]: + """Resolve unique stored ownership; never arbitrarily choose among multiple Cubes.""" + self._ensure_open() + self._maintenance_strings(memory_ids, "memory_ids") + result = dict.fromkeys(memory_ids) + if not result: + return result + with self._cursor() as cursor: + cursor.execute( + f"SELECT business_id, scope FROM ONLY {_TABLE} WHERE business_id = ANY(%(ids)s)", + {"ids": list(result)}, + ) + for node_id, owner in cursor.fetchall(): + if result[node_id] is not None and result[node_id] != owner: + raise GraphDBError("A memory ID belongs to multiple Cubes") + result[node_id] = owner + return result + + def exist_user_name(self, user_name: str) -> dict[str, bool]: + """Report stored data presence, including archived and deleted memories.""" + self._ensure_open() + self._maintenance_strings([user_name], "user_name") + with self._cursor() as cursor: + cursor.execute( + f"SELECT EXISTS (SELECT 1 FROM ONLY {_TABLE} WHERE scope = %(scope)s)", + {"scope": user_name}, + ) + return {user_name: bool(cursor.fetchone()[0])} + + def delete_node_by_mem_cube_id( + self, mem_cube_id: str, delete_record_id: str, hard_delete: bool = False + ) -> int: + """Mark currently nondeleted rows, or hard-delete one already-deleted batch.""" + self._ensure_open() + if not isinstance(hard_delete, bool): + raise ValueError("hard_delete must be a bool") + return self._change_deletion_record( + mem_cube_id, delete_record_id, "hard" if hard_delete else "delete" + ) + + def recover_memory_by_mem_cube_id(self, mem_cube_id: str, delete_record_id: str) -> int: + """Reactivate only deleted rows carrying this Cube's deletion batch.""" + return self._change_deletion_record(mem_cube_id, delete_record_id, "recover") + + def _change_deletion_record(self, scope: str, record_id: str, action: str) -> int: + self._ensure_open() + self._maintenance_strings([scope, record_id], "Cube and deletion record") + deleted, params = self._build_equal_condition(["status"], "deleted", prefix="memos_deleted") + if action == "delete": + predicate = f"({deleted}) IS NOT TRUE" + else: + batch, bound = self._build_equal_condition( + ["delete_record_id"], record_id, prefix="memos_batch" + ) + predicate = f"{deleted} AND {batch}" + params = self._merge_bindings(params, bound) + with self._cursor(lock=2) as cursor: + cursor.execute( + f"SELECT business_id, memory, properties, NULL FROM ONLY {_TABLE} " + f"WHERE scope = %(scope)s AND ({predicate}) ORDER BY business_id FOR UPDATE", + self._merge_bindings(params, {"scope": scope}), + ) + rows = cursor.fetchall() + if action == "hard": + return self._delete_locked_nodes(cursor, scope, [row[0] for row in rows]) + changes = ( + { + "status": "deleted", + "delete_record_id": record_id, + "delete_time": datetime.now(timezone.utc).isoformat(), + } + if action == "delete" + else {"status": "activated", "delete_record_id": "", "delete_time": ""} + ) + affected = 0 + for row in rows: + node = self._decode_node(row, scope, False) + metadata = {**node["metadata"], **changes} + payload = self._encode_node(node["id"], node["memory"], metadata, scope)[4] + cursor.execute( + f"UPDATE ONLY {_TABLE} SET properties = %(properties)s::ag_catalog.agtype " + "WHERE scope = %(scope)s AND business_id = %(id)s", + {"scope": scope, "id": node["id"], "properties": payload}, + ) + affected += cursor.rowcount + return affected + + def remove_oldest_memory( + self, memory_type: str, keep_latest: int | str, user_name: str | None = None + ) -> None: + """Trim atomically; digit strings are limited to 64 characters, including zeros.""" + scope = self._capacity_scope(memory_type, user_name) + if isinstance(keep_latest, str) and len(keep_latest) > 64: + raise ValueError("keep_latest digit strings must contain at most 64 characters") + if isinstance(keep_latest, str) and re.fullmatch(r"[0-9]+", keep_latest): + keep_latest = int(keep_latest) + if type(keep_latest) is not int or keep_latest < 0: + raise ValueError("keep_latest must be a nonnegative integer or ASCII digit string") + kind, params = self._build_equal_condition( + ["memory_type"], memory_type, prefix="memos_type" + ) + params = self._merge_bindings(params, {"scope": scope}) + with self._cursor(lock=2) as cursor: + cursor.execute( + f"SELECT business_id FROM ONLY {_TABLE} WHERE scope = %(scope)s " + f"AND {kind} ORDER BY business_id FOR UPDATE", + params, + ) + locked = [row[0] for row in cursor.fetchall()] + if not locked: + return + cursor.execute("SET LOCAL TIME ZONE 'UTC'") + instant, time_params = self._text_field_expr(["updated_at"], prefix="memos_time") + cursor.execute( + f"SELECT business_id FROM ONLY {_TABLE} " + "WHERE scope = %(scope)s AND business_id = ANY(%(ids)s) " + f"ORDER BY {instant}::timestamptz DESC NULLS LAST, " + 'business_id COLLATE "C" DESC', + self._merge_bindings(time_params, {"scope": scope, "ids": locked}), + ) + # Fetch the full ranking even when keep=0 or keep>=len(S): every + # locked timestamp must be parsed before any deletion can occur. + ranked = [row[0] for row in cursor.fetchall()] + deleted = sorted(ranked[min(keep_latest, len(ranked)) :]) + for offset in range(0, len(deleted), 500): + batch = deleted[offset : offset + 500] + self._cypher( + cursor, + "MATCH (a:Memory)-[r]->(b:Memory) WHERE " + "(a.user_name = $scope AND a.id IN $ids) OR " + "(b.user_name = $scope AND b.id IN $ids) DELETE r", + {"scope": scope, "ids": batch}, + ) + cursor.execute( + f"DELETE FROM ONLY {_TABLE} WHERE scope = %(scope)s AND business_id = ANY(%(ids)s)", + {"scope": scope, "ids": batch}, + ) + + @staticmethod + def _merge_bindings(*parts: dict[str, Any]) -> dict[str, Any]: + """Compose independent fragments without silently replacing a parameter.""" + result = {} + for part in parts: + if result.keys() & part.keys(): + raise ValueError("Duplicate SQL parameter names") + result.update(part) + return result + + def _join_fragments( + self, parts: list[tuple[str, dict[str, Any]]], separator: str + ) -> tuple[str, dict[str, Any]]: + return separator.join(sql for sql, _ in parts), self._merge_bindings( + *(params for _, params in parts) + ) + + def _field_expr(self, path: list[str], *, prefix: str) -> tuple[str, dict[str, Any]]: + """Bind field paths; prefixes are internal names supplied by the query builder.""" + params = {f"{prefix}_root": _encode_filter_value(path[0])} + root = ( + f"ag_catalog.agtype_access_operator(properties, %({prefix}_root)s::ag_catalog.agtype)" + ) + if len(path) == 1: + return root, params + params[f"{prefix}_child"] = _encode_filter_value(path[1]) + # Serialized info strings are preserved by storage, but are not maps. + return ( + f"(CASE WHEN left(ag_catalog.agtype_out({root})::text, 1) = '{{' " + f"THEN ag_catalog.agtype_access_operator({root}, %({prefix}_child)s::ag_catalog.agtype) END)", + params, + ) + + def _text_field_expr(self, path: list[str], *, prefix: str) -> tuple[str, dict[str, Any]]: + value, params = self._field_expr(path, prefix=prefix) + return ( + f"(CASE WHEN left(ag_catalog.agtype_out({value})::text, 1) = '\"' " + f"THEN {value}::text END)", + params, + ) + + def _build_equal_condition( + self, path: list[str], value: Any, *, prefix: str + ) -> tuple[str, dict[str, Any]]: + # AGE access returns SQL NULL for both missing fields and JSON null. + # Containment distinguishes them without converting AGE maps to JSON. + if value is None: + nested = value + for key in reversed(path): + nested = {key: nested} + return f"(properties @> %({prefix}_value)s::ag_catalog.agtype)", { + f"{prefix}_value": _encode_filter_value(nested) + } + expr, params = self._field_expr(path, prefix=prefix) + return f"({expr} = %({prefix}_value)s::ag_catalog.agtype)", self._merge_bindings( + params, {f"{prefix}_value": _encode_filter_value(value)} + ) + + def _build_filter_condition( + self, field: str, op: str, value: Any, *, prefix: str + ) -> tuple[str, dict[str, Any]]: + if not isinstance(field, str) or not re.fullmatch( + r"(?:info\.)?[A-Za-z_][A-Za-z0-9_]*", field + ): + raise ValueError("Invalid filter field") + if not isinstance(op, str): + raise ValueError("Invalid filter operator") + path = field.split(".") + _encode_filter_value(value) + operator = _OPERATORS.get(op, op) + temporal = path[-1].endswith("_at") or path[-1] == "delete_time" + bound = f"%({prefix}_value)s" + if operator in {">", ">=", "<", "<="} or (operator == "=" and temporal): + if temporal: + try: + if not isinstance(value, str): + raise ValueError("Time filter must be an ISO date/time string") + instant = datetime.fromisoformat(value) + if instant.tzinfo is None: + instant = instant.replace(tzinfo=timezone.utc) + except ValueError as exc: + raise ValueError("Invalid ISO date/time filter") from exc + expr, params = self._text_field_expr(path, prefix=prefix) + return ( + f"({expr}::timestamptz {operator} {bound}::timestamptz)", + self._merge_bindings(params, {f"{prefix}_value": instant.isoformat()}), + ) + try: + if type(value) not in (int, float) or not math.isfinite(value): + raise ValueError("Numeric filter requires a finite number") + except OverflowError as exc: + raise ValueError("Numeric filter is too large") from exc + expr, params = self._field_expr(path, prefix=prefix) + return ( + f"((CASE WHEN ag_catalog.agtype_out({expr})::text ~ '^[-.0-9]' " + f"THEN ag_catalog.agtype_out({expr})::text::numeric END) {operator} {bound})", + self._merge_bindings(params, {f"{prefix}_value": value}), + ) + if operator == "=": + return self._build_equal_condition(path, value, prefix=prefix) + if operator in {"in", "contains", "all"}: + if operator == "in" and not isinstance(value, list): + raise ValueError("in requires a list") + values = value if isinstance(value, list) else [value] + if operator == "all": + expr, params = self._field_expr(path, prefix=prefix) + return f"({expr} @> {bound}::ag_catalog.agtype)", self._merge_bindings( + params, {f"{prefix}_value": _encode_filter_value(values)} + ) + if not values: + return "FALSE", {} + array = operator == "contains" or path[-1] in _ARRAY_FIELDS + parts = [] + for index, item in enumerate(values): + child = f"{prefix}_item_{index}" + if array: + expr, params = self._field_expr(path, prefix=child) + parts.append( + ( + f"{expr} @> %({child}_value)s::ag_catalog.agtype", + self._merge_bindings( + params, {f"{child}_value": _encode_filter_value([item])} + ), + ) + ) + else: + parts.append(self._build_equal_condition(path, item, prefix=child)) + sql, params = self._join_fragments(parts, " OR ") + return f"({sql})", params + if operator in {"like", "nolike", "starts_with", "ends_with"}: + if not isinstance(value, str): + raise ValueError("String filter requires a string") + escaped = value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + pattern = ("" if operator == "starts_with" else "%") + escaped + pattern += "" if operator == "ends_with" else "%" + expr, params = self._text_field_expr(path, prefix=prefix) + comparison = "NOT LIKE" if operator == "nolike" else "LIKE" + return f"({expr} {comparison} {bound})", self._merge_bindings( + params, {f"{prefix}_value": pattern} + ) + raise ValueError(f"Unsupported filter operator: {op}") + + def _build_metadata_conditions( + self, filters: list[dict[str, Any]], *, prefix: str = "memos_metadata" + ) -> tuple[str, dict[str, Any]]: + """Compile the existing field/op/value list as independent AND fragments.""" + if not isinstance(filters, list): + raise ValueError("filters must be a list") + parts = [] + for index, item in enumerate(filters): + if ( + not isinstance(item, dict) + or not {"field", "value"} <= item.keys() + or (item.keys() - {"field", "op", "value"}) + ): + raise ValueError("Invalid metadata filter") + parts.append( + self._build_filter_condition( + item["field"], item.get("op", "="), item["value"], prefix=f"{prefix}_{index}" + ) + ) + sql, params = self._join_fragments(parts, " AND ") + return sql or "TRUE", params + + @staticmethod + def parse_filter(filter_dict: dict | None = None) -> dict | None: + """Route business fields to info.*, as in PostgreSQL and PolarDB.""" + if filter_dict is None: + return None + if not isinstance(filter_dict, dict): + raise ValueError("filter must be an object") + normalized = {} + for key, value in filter_dict.items(): + if not isinstance(key, str): + raise ValueError("Invalid filter field") + if key in ("and", "or"): + if not isinstance(value, list) or not value: + raise ValueError("and/or requires a nonempty list") + normalized[key] = [OpenGaussGraphDB.parse_filter(item) for item in value] + else: + field = key if key in _ROOT_FIELDS or key.startswith("info.") else f"info.{key}" + normalized[field] = value + return normalized + + def _build_single_filter_condition( + self, condition_dict: dict[str, Any], *, prefix: str + ) -> tuple[str, dict[str, Any]]: + """Assign each field/operator occurrence its own binding namespace.""" + parts = [] + for index, (field, value) in enumerate(condition_dict.items()): + child = f"{prefix}_{index}" + if isinstance(value, dict): + if not value: + raise ValueError("Filter operators cannot be empty") + parts.extend( + self._build_filter_condition(field, op, val, prefix=f"{child}_{op_index}") + for op_index, (op, val) in enumerate(value.items()) + ) + else: + op = "all" if field in _ARRAY_FIELDS else "=" + parts.append(self._build_filter_condition(field, op, value, prefix=child)) + return self._join_fragments(parts, " AND ") + + def _build_filter_where_clause( + self, filter_dict: dict | None, *, prefix: str = "memos_filter", nested: bool = False + ) -> tuple[str, dict[str, Any]]: + """Build named predicates without sharing a mutable parameter accumulator.""" + condition = self.parse_filter(filter_dict) + if nested and not condition: + raise ValueError("Nested filter conditions cannot be empty") + if condition is None: + return "TRUE", {} + parts = [] + for index, (field, value) in enumerate(condition.items()): + child_prefix = f"{prefix}_{index}" + if field in ("and", "or"): + children = [ + self._build_filter_where_clause( + child, prefix=f"{child_prefix}_{child_index}", nested=True + ) + for child_index, child in enumerate(value) + ] + sql, params = self._join_fragments(children, f" {field.upper()} ") + parts.append((f"({sql})", params)) + else: + parts.append( + self._build_single_filter_condition({field: value}, prefix=child_prefix) + ) + sql, params = self._join_fragments(parts, " AND ") + return f"({sql})" if parts else "TRUE", params diff --git a/src/memos/lifecycle.py b/src/memos/lifecycle.py new file mode 100644 index 000000000..1d19e613e --- /dev/null +++ b/src/memos/lifecycle.py @@ -0,0 +1,57 @@ +"""Share the result of a terminal close attempt; resource owners handle draining.""" + +from collections.abc import Callable +from dataclasses import dataclass, field +from threading import Event, Lock, get_ident + +from memos.exceptions import MemOSError + + +@dataclass +class _CloseAttempt: + owner: int + done: Event = field(default_factory=Event) + error: BaseException | None = None + + +class CloseState: + """Keep concurrent close callers on one attempt without tracking operations.""" + + def __init__(self) -> None: + self._lock = Lock() + self.closing = Event() + self._closed = False + self._attempt: _CloseAttempt | None = None + + def ensure_open(self) -> None: + """Reject new work once final close has started, including after failure.""" + if self.closing.is_set(): + raise MemOSError("Component is closing or closed") + + def close(self, cleanup: Callable[[], None]) -> None: + """Run cleanup once per attempt; retry only after a completed failure.""" + with self._lock: + if self._closed: + return + attempt = self._attempt + execute = attempt is None or attempt.done.is_set() + if execute: + self.closing.set() + attempt = self._attempt = _CloseAttempt(get_ident()) + elif attempt.owner == get_ident(): + raise MemOSError("A close callback cannot close its own owner") + if not execute: + attempt.done.wait() + if attempt.error is not None: + raise MemOSError("Concurrent component close failed") from attempt.error + return + try: + cleanup() + except BaseException as error: + attempt.error = error + raise + else: + with self._lock: + self._closed = True + finally: + attempt.done.set() diff --git a/src/memos/mem_cube/general.py b/src/memos/mem_cube/general.py index 1238ae050..c68623f8d 100644 --- a/src/memos/mem_cube/general.py +++ b/src/memos/mem_cube/general.py @@ -6,6 +6,8 @@ from memos.configs.mem_cube import GeneralMemCubeConfig from memos.configs.utils import get_json_file_model_schema from memos.exceptions import ConfigurationError, MemCubeError +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.lifecycle import CloseState from memos.log import get_logger from memos.mem_cube.base import BaseMemCube from memos.mem_cube.utils import download_repo, merge_config_with_default @@ -13,6 +15,7 @@ from memos.memories.factory import MemoryFactory from memos.memories.parametric.base import BaseParaMemory from memos.memories.textual.base import BaseTextMemory +from memos.memories.textual.tree import TreeTextMemory logger = get_logger(__name__) @@ -24,28 +27,57 @@ class GeneralMemCube(BaseMemCube): def __init__(self, config: GeneralMemCubeConfig): """Initialize the MemCube with a configuration.""" self.config = config - time_start = time.time() - self._text_mem: BaseTextMemory | None = ( - MemoryFactory.from_config(config.text_mem) - if config.text_mem.backend != "uninitialized" - else None - ) - logger.info(f"init_text_mem in {time.time() - time_start} seconds") - self._act_mem: BaseActMemory | None = ( - MemoryFactory.from_config(config.act_mem) - if config.act_mem.backend != "uninitialized" - else None - ) - self._para_mem: BaseParaMemory | None = ( - MemoryFactory.from_config(config.para_mem) - if config.para_mem.backend != "uninitialized" - else None - ) - self._pref_mem: BaseTextMemory | None = ( - MemoryFactory.from_config(config.pref_mem) - if config.pref_mem.backend != "uninitialized" - else None - ) + self._close_state = CloseState() + self._owned_memories = [] + self._text_mem: BaseTextMemory | None = None + self._act_mem: BaseActMemory | None = None + self._para_mem: BaseParaMemory | None = None + self._pref_mem: BaseTextMemory | None = None + try: + time_start = time.time() + if config.text_mem.backend != "uninitialized": + self._text_mem = MemoryFactory.from_config(config.text_mem) + self._owned_memories.append(self._text_mem) + logger.info("init_text_mem in %s seconds", time.time() - time_start) + if config.act_mem.backend != "uninitialized": + self._act_mem = MemoryFactory.from_config(config.act_mem) + self._owned_memories.append(self._act_mem) + if config.para_mem.backend != "uninitialized": + self._para_mem = MemoryFactory.from_config(config.para_mem) + self._owned_memories.append(self._para_mem) + if config.pref_mem.backend != "uninitialized": + self._pref_mem = MemoryFactory.from_config(config.pref_mem) + self._owned_memories.append(self._pref_mem) + except BaseException: + try: + self.close() + except Exception: + logger.exception("Failed to close memories after Cube initialization failure") + raise + + def close(self) -> None: + """Close memories created by this Cube, including subsequently replaced ones.""" + self._close_state.close(self._close_owned_memories) + + def _close_owned_memories(self) -> None: + errors = [] + seen = set() + for memory in self._owned_memories[:]: + if id(memory) in seen: + continue + seen.add(id(memory)) + try: + close = getattr(memory, "close", None) + if callable(close): + close() + self._owned_memories = [ + owned for owned in self._owned_memories if owned is not memory + ] + except Exception as error: + errors.append(error) + logger.exception("Failed to close owned memory %s", type(memory).__name__) + if errors: + raise errors[0] def load( self, @@ -115,7 +147,12 @@ def dump( # Dump specified memory types if "text_mem" in memory_types and self.text_mem: - self.text_mem.dump(dir) + if isinstance(self.text_mem, TreeTextMemory) and isinstance( + self.text_mem.graph_store, OpenGaussGraphDB + ): + self.text_mem.dump(dir, include_embedding=True) + else: + self.text_mem.dump(dir) logger.info(f"Dumped text_mem to {dir}") if "act_mem" in memory_types and self.act_mem: @@ -158,7 +195,14 @@ def init_from_dir( config = merge_config_with_default(config, default_config) logger.info(f"Applied default config to cube {config.cube_id}") mem_cube = GeneralMemCube(config) - mem_cube.load(dir, memory_types) + try: + mem_cube.load(dir, memory_types) + except BaseException: + try: + mem_cube.close() + except Exception: + logger.exception("Failed to close Cube after load failure") + raise return mem_cube @staticmethod diff --git a/src/memos/mem_feedback/feedback.py b/src/memos/mem_feedback/feedback.py index 18ebb0b49..dbd6dd409 100644 --- a/src/memos/mem_feedback/feedback.py +++ b/src/memos/mem_feedback/feedback.py @@ -7,12 +7,13 @@ from datetime import datetime from typing import TYPE_CHECKING, Any, Literal -from tenacity import retry, stop_after_attempt, wait_random_exponential +from tenacity import retry, retry_if_not_exception_type, stop_after_attempt, wait_random_exponential from memos.configs.memory import MemFeedbackConfig from memos.context.context import ContextThreadPoolExecutor from memos.dependency import require_python_package from memos.embedders.factory import EmbedderFactory, OllamaEmbedder +from memos.exceptions import GraphDBError, LLMError, MemOSError from memos.graph_dbs.factory import GraphStoreFactory, PolarDBGraphDB from memos.llms.factory import AzureLLM, LLMFactory, OllamaLLM, OpenAILLM from memos.log import get_logger @@ -116,7 +117,11 @@ def _tokenize_chinese(self, text): def _embed_once(self, texts): return self.embedder.embed(texts) - @retry(stop=stop_after_attempt(3), wait=wait_random_exponential(multiplier=1, min=4, max=10)) + @retry( + stop=stop_after_attempt(3), + wait=wait_random_exponential(multiplier=1, min=4, max=10), + retry=retry_if_not_exception_type((GraphDBError, ValueError)), + ) def _retry_db_operation(self, operation): try: return operation() @@ -1062,11 +1067,14 @@ def process_keyword_replace( try: result = future.result() update_results.append(result) + except (GraphDBError, ValueError): + raise except Exception as e: - mem_id = future_to_info[future][0] + mem_id = future_to_info[future] logger.error( - f"[Feedback Core DB] Exception during update operation for memory {mem_id}: {e}" + "Feedback update failed for memory %s: %s", mem_id, e, exc_info=True ) + raise return {"record": {"add": [], "update": update_results}} @@ -1122,9 +1130,14 @@ def check_validity(item): if not chat_history: return self._pure_add(user_name, feedback_content, feedback_time, info) else: - raw_judge = self._feedback_judgement( - chat_history, feedback_content, feedback_time=feedback_time - ) + try: + raw_judge = self._feedback_judgement( + chat_history, feedback_content, feedback_time=feedback_time + ) + except ValueError as error: + raise LLMError("Feedback model produced invalid judgement data") from error + if not isinstance(raw_judge, list): + raise LLMError("Feedback model did not return a judgement list") valid_feedback = ( [item for item in raw_judge if check_validity(item)] if raw_judge else [] ) @@ -1158,26 +1171,29 @@ def check_validity(item): + "\nUser feedback: " + str(feedback_content) ) - mem_item = make_mem_item( - value, - user_id=user_id, - user_name=user_name, - session_id=session_id, - tags=tags, - key=key, - embedding=embedding, - sources=[ - { - "type": "feedback", - "role": "user", - "chat_time": feedback_time, - "content": feedback_content, - } - ], - background=background, - type="fine", - info=info, - ) + try: + mem_item = make_mem_item( + value, + user_id=user_id, + user_name=user_name, + session_id=session_id, + tags=tags, + key=key, + embedding=embedding, + sources=[ + { + "type": "feedback", + "role": "user", + "chat_time": feedback_time, + "content": feedback_content, + } + ], + background=background, + type="fine", + info=info, + ) + except ValueError as error: + raise LLMError("Feedback model produced invalid memory data") from error feedback_memories.append(mem_item) mem_record = self._feedback_memory( @@ -1196,11 +1212,11 @@ def check_validity(item): ) return mem_record + except (MemOSError, ValueError): + raise except Exception as e: - logger.error( - f"[0107 Feedback Core: process_feedback_core] Error for user {user_name}: {e}" - ) - return {"record": {"add": [], "update": []}} + logger.exception("Required feedback processing failed for user %s", user_name) + raise MemOSError("Feedback core processing failed") from e def process_feedback( self, @@ -1212,7 +1228,7 @@ def process_feedback( **kwargs, ): """ - Process feedback with different modes. + Process memory feedback and independently generate an optional answer. Args: user_name: cube_ids @@ -1221,16 +1237,23 @@ def process_feedback( **kwargs: Additional arguments including async_mode Returns: - Dict with answer and/or memory operation records + Dict with answer, answer_status (not_requested/generated/unavailable), + and actual memory operation records. An unavailable answer is not + display text; callers decide how to present or retry it. Core failures + propagate and may follow partial writes; do not blindly retry them. """ corrected_answer = kwargs.get("corrected_answer", False) with ContextThreadPoolExecutor(max_workers=2) as ex: - answer_future = ex.submit( - self._generate_answer, - chat_history, - feedback_content, - corrected_answer=corrected_answer, + answer_future = ( + ex.submit( + self._generate_answer, + chat_history, + feedback_content, + corrected_answer=True, + ) + if corrected_answer + else None ) core_future = ex.submit( self.process_feedback_core, @@ -1241,28 +1264,19 @@ def process_feedback( info, **kwargs, ) - _done, pending = concurrent.futures.wait([answer_future, core_future], timeout=30) - for fut in pending: - fut.cancel() - try: - answer = answer_future.result() - record = core_future.result() - task_id = kwargs.get("task_id", "default") - - logger.info( - f"[Feedback Core MemFeedback process] Feedback Completed : user {user_name} | task_id {task_id} | record {record}." - ) - - return {"answer": answer, "record": record["record"]} - except concurrent.futures.TimeoutError: - logger.error( - f"[Feedback Core MemFeedback process] Timeout in sync mode for {user_name}", - exc_info=True, - ) - return {"answer": "", "record": {"add": [], "update": []}} - except Exception as e: - logger.error( - f"[Feedback Core MemFeedback process] Error in concurrent tasks for {user_name}: {e}", - exc_info=True, - ) - return {"answer": "", "record": {"add": [], "update": []}} + answer = "" + answer_status = "not_requested" + if answer_future is not None: + answer_status = "unavailable" + try: + candidate = answer_future.result() + if isinstance(candidate, str) and candidate.strip(): + answer = candidate + answer_status = "generated" + except Exception: + logger.exception("Optional feedback answer generation failed") + # Always consume the core result, including when answer generation + # failed. Its exception must take precedence and its records are real. + record = core_future.result() + logger.info("Feedback completed for user %s, task %s", user_name, kwargs.get("task_id")) + return {"answer": answer, "answer_status": answer_status, "record": record["record"]} diff --git a/src/memos/mem_scheduler/base_mixins/queue_ops.py b/src/memos/mem_scheduler/base_mixins/queue_ops.py index e8d215dc6..282ae1cfb 100644 --- a/src/memos/mem_scheduler/base_mixins/queue_ops.py +++ b/src/memos/mem_scheduler/base_mixins/queue_ops.py @@ -1,6 +1,5 @@ from __future__ import annotations -import multiprocessing import time from contextlib import suppress @@ -15,6 +14,7 @@ get_current_trace_id, set_request_context, ) +from memos.exceptions import ConfigurationError from memos.log import get_logger from memos.mem_scheduler.schemas.general_schemas import STARTUP_BY_PROCESS from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem @@ -160,12 +160,12 @@ def submit_messages(self, messages: ScheduleMessageItem | list[ScheduleMessageIt self.memos_message_queue.submit_messages(messages=queued_msgs) def _message_consumer(self) -> None: - while self._running: + while self._running and not self._stop_event.is_set(): try: if self.enable_parallel_dispatch and self.dispatcher: running_tasks = self.dispatcher.get_running_task_count() if running_tasks >= self.dispatcher.max_workers: - time.sleep(self._consume_interval) + self._stop_event.wait(self._consume_interval) continue messages = self.memos_message_queue.get_messages(batch_size=self.consume_batch) @@ -215,8 +215,7 @@ def _message_consumer(self) -> None: set_request_context(prev_context) try: with suppress(Exception): - if messages: - self.dispatcher.on_messages_enqueued(messages) + self.dispatcher.on_messages_enqueued(messages) if len(messages) >= self.consume_batch: unique_labels = sorted({msg.label for msg in messages}) @@ -231,12 +230,12 @@ def _message_consumer(self) -> None: except Exception as e: logger.error("Error dispatching messages: %s", e) - time.sleep(self._consume_interval) + self._stop_event.wait(self._consume_interval) except Exception as e: if "No messages available in Redis queue" not in str(e): logger.error("Unexpected error in message consumer: %s", e, exc_info=True) - time.sleep(self._consume_interval) + self._stop_event.wait(self._consume_interval) def _monitor_loop(self): while self._running: @@ -261,7 +260,7 @@ def _monitor_loop(self): except Exception as e: logger.error("Error in metrics monitor loop: %s", e, exc_info=True) - time.sleep(15) + self._stop_event.wait(15) def start(self) -> None: if self.enable_parallel_dispatch: @@ -287,24 +286,25 @@ def start_consumer(self) -> None: logger.warning("Memory Scheduler consumer is already running") return - self._running = True - if self.scheduler_startup_mode == STARTUP_BY_PROCESS: - self._consumer_process = multiprocessing.Process( - target=self._message_consumer, - daemon=True, - name="MessageConsumerProcess", - ) - self._consumer_process.start() - logger.info("Message consumer process started") - else: - self._consumer_thread = ContextThread( - target=self._message_consumer, - daemon=True, - name="MessageConsumerThread", + raise ConfigurationError( + "Internal process startup is unsupported; construct the Scheduler inside an " + "independent worker using run_scheduler_worker and thread startup mode. " + "Use Redis for tasks submitted across processes." ) + self._running = True + self._consumer_thread = ContextThread( + target=self._message_consumer, + daemon=True, + name="MessageConsumerThread", + ) + try: self._consumer_thread.start() - logger.info("Message consumer thread started") + except BaseException: + self._consumer_thread = None + self._running = False + raise + logger.info("Message consumer thread started") def stop_consumer(self) -> None: if not self._running: @@ -312,6 +312,7 @@ def stop_consumer(self) -> None: return self._running = False + self._stop_event.set() if self.scheduler_startup_mode == STARTUP_BY_PROCESS and self._consumer_process: if self._consumer_process.is_alive(): @@ -326,14 +327,15 @@ def stop_consumer(self) -> None: logger.info("Consumer process terminated") else: logger.info("Consumer process stopped") - self._consumer_process = None + if not self._consumer_process.is_alive(): + self._consumer_process = None elif self._consumer_thread and self._consumer_thread.is_alive(): self._consumer_thread.join(timeout=5.0) if self._consumer_thread.is_alive(): logger.warning("Consumer thread did not stop gracefully") else: logger.info("Consumer thread stopped") - self._consumer_thread = None + self._consumer_thread = None logger.info("Memory Scheduler consumer stopped") @@ -347,14 +349,14 @@ def stop(self) -> None: if self._monitor_thread: self._monitor_thread.join(timeout=2.0) - if self.dispatcher: - logger.info("Shutting down dispatcher...") - self.dispatcher.shutdown() - if self.dispatcher_monitor: logger.info("Shutting down monitor...") self.dispatcher_monitor.stop() + if self.dispatcher: + logger.info("Shutting down dispatcher...") + self.dispatcher.shutdown() + @property def handlers(self) -> dict[str, Callable]: if not self.dispatcher: diff --git a/src/memos/mem_scheduler/base_scheduler.py b/src/memos/mem_scheduler/base_scheduler.py index 7c26336ed..175a5461e 100644 --- a/src/memos/mem_scheduler/base_scheduler.py +++ b/src/memos/mem_scheduler/base_scheduler.py @@ -3,11 +3,16 @@ import os import threading +from contextlib import contextmanager from pathlib import Path from typing import TYPE_CHECKING +from memos.api.lifecycle import shutdown_components from memos.configs.mem_scheduler import AuthConfig, BaseSchedulerConfig +from memos.exceptions import ConfigurationError, MemOSError +from memos.lifecycle import CloseState from memos.log import get_logger +from memos.mem_feedback.simple_feedback import SimpleMemFeedback from memos.mem_scheduler.base_mixins import ( BaseSchedulerMemoryMixin, BaseSchedulerQueueMixin, @@ -48,14 +53,18 @@ if TYPE_CHECKING: + from collections.abc import Iterator + import redis from sqlalchemy.engine import Engine from memos.llms.base import BaseLLM from memos.mem_cube.base import BaseMemCube - from memos.mem_feedback.simple_feedback import SimpleMemFeedback - from memos.mem_scheduler.schemas.message_schemas import ScheduleLogForWebItem + from memos.mem_scheduler.schemas.message_schemas import ( + ScheduleLogForWebItem, + ScheduleMessageItem, + ) from memos.memories.textual.item import TextualMemoryItem from memos.memories.textual.tree import TreeTextMemory from memos.memories.textual.tree_text_memory.retrieve.searcher import Searcher @@ -79,6 +88,14 @@ class BaseScheduler( def __init__(self, config: BaseSchedulerConfig): """Initialize the scheduler with the given configuration.""" super().__init__() + self._owner_pid = os.getpid() + self._close_state = CloseState() + self._initialization_lock = threading.RLock() + self._entry_local = threading.local() + self._stop_event = threading.Event() + self._closed_parts = set() + self.components = None + self._borrowed_components = False self.config = config # hyper-parameters @@ -207,6 +224,18 @@ def initialize_modules( mem_reader=None, redis_client: redis.Redis | None = None, ): + with self._lifecycle_entry(): + self._initialize_modules(chat_llm, process_llm, db_engine, mem_reader, redis_client) + self._close_state.ensure_open() + + def _initialize_modules( + self, + chat_llm: BaseLLM, + process_llm: BaseLLM | None, + db_engine: Engine | None, + mem_reader, + redis_client: redis.Redis | None, + ) -> None: if process_llm is None: process_llm = chat_llm @@ -281,19 +310,144 @@ def _cleanup_on_init_failure(self): except Exception as e: logger.warning(f"Error during cleanup: {e}") + def close(self) -> None: + """End this scheduler's lifecycle, draining users before its owned graph.""" + self._ensure_owner_process() + current = threading.current_thread() + owned_threads = [self._consumer_thread, self._monitor_thread, self._io_loop_thread] + if self.dispatcher_monitor is not None: + owned_threads.append(self.dispatcher_monitor._monitor_thread) + if ( + getattr(self._entry_local, "active", False) + or self.dispatcher.is_worker_thread() + or any(current is thread for thread in owned_threads) + ): + raise MemOSError("Scheduler close must run outside its workers and initialization") + self._close_state.close(self._close_resources) + + @contextmanager + def _lifecycle_entry(self) -> Iterator[None]: + """Serialize component creation/start with the owner's shutdown barrier.""" + self._ensure_owner_process() + with self._initialization_lock: + self._close_state.ensure_open() + nested = getattr(self._entry_local, "active", False) + self._entry_local.active = True + try: + yield + finally: + self._entry_local.active = nested + + def _ensure_owner_process(self) -> None: + if os.getpid() != self._owner_pid: + raise ConfigurationError( + "Scheduler cannot be reused after fork; construct it inside the worker process" + ) + + def _close_resources(self) -> None: + errors = [] + + def release(name, callback): + if name in self._closed_parts: + return True + try: + callback() + self._closed_parts.add(name) + return True + except Exception as error: + errors.append(error) + logger.exception("Failed to close scheduler resource %s", name) + return False + + self._running = False + self._stop_event.set() + if self.dispatcher_monitor is not None: + self.dispatcher_monitor.begin_close() + # Only initialization/start owns this lock. Release it before joining + # workers, which may need the same lock to observe the closing state. + with self._initialization_lock: + pass + self._running = False + self._stop_event.set() + # An in-flight initialize_modules may have just created the monitor. + if self.dispatcher_monitor is not None: + self.dispatcher_monitor.begin_close() + # RabbitMQ is independent; failure must not prevent draining DB users. + release("rabbitmq", self.rabbitmq_close) + monitor_stopped = self.dispatcher_monitor is None or release( + "monitor", self.dispatcher_monitor.close + ) + consumers_stopped = release("consumers", self._join_consumers) + tasks_stopped = False + if monitor_stopped and consumers_stopped: + tasks_stopped = release("dispatcher", self.dispatcher.close) + if tasks_stopped and self.components is not None: + release("components", lambda: shutdown_components(self.components)) + if errors: + raise MemOSError( + "Scheduler shutdown failed; unreleased resources retained" + ) from errors[0] + + def _join_consumers(self) -> None: + for thread in (self._consumer_thread, self._monitor_thread): + if thread is not None: + if thread is threading.current_thread(): + raise MemOSError("Scheduler cannot close from its consumer thread") + thread.join() + if self._consumer_process is not None: + self._consumer_process.join(timeout=5) + if self._consumer_process.is_alive(): + logger.warning("Consumer process did not stop gracefully, terminating") + self._consumer_process.terminate() + self._consumer_process.join(timeout=2) + if self._consumer_process.is_alive(): + raise MemOSError("Scheduler consumer process has not exited") + + def _ensure_components(self) -> None: + with self._initialization_lock: + self._close_state.ensure_open() + if self.components is None: + self.components = init_components() + # Retain ownership even when close won the race; never publish late. + self._close_state.ensure_open() + if self.current_mem_cube is None: + self.current_mem_cube = self.components["naive_mem_cube"] + if self._feedback_server is None: + self._feedback_server = self.components["feedback_server"] + self._feedback_cube = self.current_mem_cube + + def start(self) -> None: + with self._lifecycle_entry(): + super().start() + + def stop(self) -> None: + self._ensure_owner_process() + super().stop() + + def stop_consumer(self) -> None: + self._ensure_owner_process() + super().stop_consumer() + + def start_consumer(self) -> None: + with self._lifecycle_entry(): + self._stop_event.clear() + super().start_consumer() + + def start_background_monitor(self) -> None: + with self._lifecycle_entry(): + super().start_background_monitor() + + def submit_messages(self, messages: ScheduleMessageItem | list[ScheduleMessageItem]) -> None: + with self._lifecycle_entry(): + super().submit_messages(messages) + @property def mem_cube(self) -> BaseMemCube: """The memory cube associated with this MemChat.""" - if self.current_mem_cube is None: - logger.error("mem_cube is None when accessed", stack_info=True) - try: - self.components = init_components() - self.current_mem_cube: BaseMemCube = self.components["naive_mem_cube"] - except Exception: - logger.info( - "No environment available to initialize mem cube. Using fallback naive_mem_cube." - ) - return self.current_mem_cube + with self._lifecycle_entry(): + if self.current_mem_cube is None and not self._borrowed_components: + self._ensure_components() + return self.current_mem_cube @property def status_tracker(self) -> TaskStatusTracker | None: @@ -330,27 +484,54 @@ def status_tracker(self, value: TaskStatusTracker | None) -> None: @property def feedback_server(self) -> SimpleMemFeedback: - """The memory cube associated with this MemChat.""" - if self._feedback_server is None: - logger.error("feedback_server is None when accessed", stack_info=True) - try: - self.components = init_components() - self._feedback_server: SimpleMemFeedback = self.components["feedback_server"] - except Exception: - logger.info( - "No environment available to initialize feedback_server. Using fallback feedback_server." - ) - return self._feedback_server + """Feedback bound to the current Cube, borrowing its dependencies.""" + with self._lifecycle_entry(): + if self._feedback_server is None: + if self._borrowed_components: + feedback = self._feedback_for_borrowed_cube() + self._close_state.ensure_open() + self._feedback_server = feedback + self._feedback_cube = self.current_mem_cube + else: + self._ensure_components() + return self._feedback_server + + def _feedback_for_borrowed_cube(self) -> SimpleMemFeedback: + text_mem = getattr(self.current_mem_cube, "text_mem", None) + dependencies = { + "llm": getattr(text_mem, "extractor_llm", None), + "embedder": getattr(text_mem, "embedder", None), + "graph_store": getattr(text_mem, "graph_store", None), + "memory_manager": getattr(text_mem, "memory_manager", None), + "mem_reader": self.mem_reader, + "searcher": self.searcher, + "reranker": getattr(text_mem, "reranker", None), + } + missing = [name for name, value in dependencies.items() if value is None] + if missing: + raise ConfigurationError( + "Feedback requires an explicit feedback_server or current Cube dependencies: " + + ", ".join(missing) + ) + return SimpleMemFeedback(**dependencies) @feedback_server.setter - def feedback_server(self, value: SimpleMemFeedback) -> None: - self._feedback_server = value + def feedback_server(self, value: SimpleMemFeedback | None) -> None: + with self._lifecycle_entry(): + self._feedback_server = value + self._feedback_cube = None @mem_cube.setter def mem_cube(self, value: BaseMemCube) -> None: """The memory cube associated with this MemChat.""" - self.current_mem_cube = value - self.retriever.mem_cube = value + with self._lifecycle_entry(): + if self._feedback_cube is not None and self._feedback_cube is not value: + self._feedback_server = None + self._feedback_cube = None + self._borrowed_components = True + self.current_mem_cube = value + if self.retriever is not None: + self.retriever.mem_cube = value @property def mem_cubes(self) -> dict[str, BaseMemCube]: diff --git a/src/memos/mem_scheduler/general_modules/init_components_for_scheduler.py b/src/memos/mem_scheduler/general_modules/init_components_for_scheduler.py index 00ff972f3..399b0be8d 100644 --- a/src/memos/mem_scheduler/general_modules/init_components_for_scheduler.py +++ b/src/memos/mem_scheduler/general_modules/init_components_for_scheduler.py @@ -4,6 +4,7 @@ from typing import TYPE_CHECKING, Any from memos.api.config import APIConfig +from memos.api.lifecycle import shutdown_components from memos.configs.embedder import EmbedderConfigFactory from memos.configs.graph_db import GraphDBConfigFactory from memos.configs.internet_retriever import InternetRetrieverConfigFactory @@ -36,7 +37,7 @@ logger = get_logger(__name__) -def build_graph_db_config(user_id: str = "default") -> dict[str, Any]: +def build_graph_db_config(user_id: str | None = "default") -> GraphDBConfigFactory: """ Build graph database configuration. @@ -46,23 +47,7 @@ def build_graph_db_config(user_id: str = "default") -> dict[str, Any]: Returns: Validated graph database configuration dictionary """ - graph_db_backend_map = { - "neo4j-community": APIConfig.get_neo4j_community_config(user_id=user_id), - "neo4j": APIConfig.get_neo4j_config(user_id=user_id), - "polardb": APIConfig.get_polardb_config(user_id=user_id), - "postgres": APIConfig.get_postgres_config(user_id=user_id), - } - - # Support both GRAPH_DB_BACKEND and legacy NEO4J_BACKEND env vars - graph_db_backend = os.getenv( - "GRAPH_DB_BACKEND", os.getenv("NEO4J_BACKEND", "neo4j-community") - ).lower() - return GraphDBConfigFactory.model_validate( - { - "backend": graph_db_backend, - "config": graph_db_backend_map[graph_db_backend], - } - ) + return GraphDBConfigFactory.model_validate(APIConfig.get_graph_db_config(user_id)) def build_vec_db_config() -> dict[str, Any]: @@ -274,84 +259,93 @@ def init_components() -> dict[str, Any]: # Create component instances graph_db = GraphStoreFactory.from_config(graph_db_config) - llm = LLMFactory.from_config(llm_config) - feedback_llm = LLMFactory.from_config(feedback_llm_config) - embedder = EmbedderFactory.from_config(embedder_config) - - plugin_manager.discover() - plugin_context = build_plugin_context( - graph_db=graph_db, - embedder=embedder, - default_cube_config=default_cube_config, - nli_client_config=nli_client_config, - mem_reader_config=mem_reader_config, - reranker_config=reranker_config, - feedback_reranker_config=feedback_reranker_config, - internet_retriever_config=internet_retriever_config, - ) - plugin_manager.init_components(plugin_context) - - # Pass graph_db to mem_reader for recall operations (deduplication, conflict detection) - mem_reader = MemReaderFactory.from_config(mem_reader_config, graph_db=graph_db) - reranker = RerankerFactory.from_config(reranker_config) - feedback_reranker = RerankerFactory.from_config(feedback_reranker_config) - internet_retriever = InternetRetrieverFactory.from_config( - internet_retriever_config, embedder=embedder - ) - - # Initialize chat llms - logger.debug("Core components instantiated") - - # Initialize memory manager - memory_manager = MemoryManager( - graph_db, - embedder, - llm, - memory_size=_get_default_memory_size(default_cube_config), - is_reorganize=getattr(default_cube_config.text_mem.config, "reorganize", False), - ) - - logger.debug("Memory manager initialized") - - tokenizer = FastTokenizer() - # Initialize text memory - text_mem = SimpleTreeTextMemory( - llm=llm, - embedder=embedder, - mem_reader=mem_reader, - graph_db=graph_db, - reranker=reranker, - memory_manager=memory_manager, - config=default_cube_config.text_mem.config, - internet_retriever=internet_retriever, - tokenizer=tokenizer, - ) - - logger.debug("Text memory initialized") - - # Create MemCube with pre-initialized memory instances - naive_mem_cube = NaiveMemCube( - text_mem=text_mem, - act_mem=None, - para_mem=None, - ) - - tree_mem: SimpleTreeTextMemory = naive_mem_cube.text_mem - searcher: Searcher = tree_mem.get_searcher( - manual_close_internet=os.getenv("ENABLE_INTERNET", "true").lower() == "false", - moscube=False, - process_llm=mem_reader.general_llm, - ) - # Initialize feedback server - feedback_server = SimpleMemFeedback( - llm=feedback_llm, - embedder=embedder, - graph_store=graph_db, - memory_manager=memory_manager, - mem_reader=mem_reader, - searcher=searcher, - reranker=feedback_reranker, - pref_feedback=True, - ) - # Return all components as a dictionary for easy access and extension - return {"naive_mem_cube": naive_mem_cube, "feedback_server": feedback_server} + components = {"graph_db": graph_db} + try: + llm = LLMFactory.from_config(llm_config) + feedback_llm = LLMFactory.from_config(feedback_llm_config) + embedder = EmbedderFactory.from_config(embedder_config) + + plugin_manager.discover() + plugin_context = build_plugin_context( + graph_db=graph_db, + embedder=embedder, + default_cube_config=default_cube_config, + nli_client_config=nli_client_config, + mem_reader_config=mem_reader_config, + reranker_config=reranker_config, + feedback_reranker_config=feedback_reranker_config, + internet_retriever_config=internet_retriever_config, + ) + plugin_manager.init_components(plugin_context) + + # Pass graph_db to mem_reader for recall operations (deduplication, conflict detection) + mem_reader = MemReaderFactory.from_config(mem_reader_config, graph_db=graph_db) + reranker = RerankerFactory.from_config(reranker_config) + feedback_reranker = RerankerFactory.from_config(feedback_reranker_config) + internet_retriever = InternetRetrieverFactory.from_config( + internet_retriever_config, embedder=embedder + ) + + # Initialize chat llms + logger.debug("Core components instantiated") + + # Initialize memory manager + memory_manager = MemoryManager( + graph_db, + embedder, + llm, + memory_size=_get_default_memory_size(default_cube_config), + is_reorganize=getattr(default_cube_config.text_mem.config, "reorganize", False), + ) + + logger.debug("Memory manager initialized") + + tokenizer = FastTokenizer() + # Initialize text memory + text_mem = SimpleTreeTextMemory( + llm=llm, + embedder=embedder, + mem_reader=mem_reader, + graph_db=graph_db, + reranker=reranker, + memory_manager=memory_manager, + config=default_cube_config.text_mem.config, + internet_retriever=internet_retriever, + tokenizer=tokenizer, + ) + + logger.debug("Text memory initialized") + + # Create MemCube with pre-initialized memory instances + naive_mem_cube = NaiveMemCube( + text_mem=text_mem, + act_mem=None, + para_mem=None, + ) + + tree_mem: SimpleTreeTextMemory = naive_mem_cube.text_mem + searcher: Searcher = tree_mem.get_searcher( + manual_close_internet=os.getenv("ENABLE_INTERNET", "true").lower() == "false", + moscube=False, + process_llm=mem_reader.general_llm, + ) + # Initialize feedback server + feedback_server = SimpleMemFeedback( + llm=feedback_llm, + embedder=embedder, + graph_store=graph_db, + memory_manager=memory_manager, + mem_reader=mem_reader, + searcher=searcher, + reranker=feedback_reranker, + pref_feedback=True, + ) + # Return all components as a dictionary for easy access and extension + components.update(naive_mem_cube=naive_mem_cube, feedback_server=feedback_server) + return components + except BaseException: + try: + shutdown_components(components) + except Exception: + logger.exception("Failed to release components after initialization failure") + raise diff --git a/src/memos/mem_scheduler/memory_manage_modules/enhancement_pipeline.py b/src/memos/mem_scheduler/memory_manage_modules/enhancement_pipeline.py index 98125c13b..f49cad21a 100644 --- a/src/memos/mem_scheduler/memory_manage_modules/enhancement_pipeline.py +++ b/src/memos/mem_scheduler/memory_manage_modules/enhancement_pipeline.py @@ -4,6 +4,7 @@ from typing import TYPE_CHECKING +from memos.exceptions import GraphDBError from memos.log import get_logger from memos.mem_scheduler.schemas.general_schemas import ( DEFAULT_SCHEDULER_RETRIEVER_BATCH_SIZE, @@ -194,27 +195,35 @@ def _split_batches( return batches def recall_for_missing_memories(self, query: str, memories: list[str]) -> tuple[str, bool]: - text_memories = "\n".join([f"- {mem}" for i, mem in enumerate(memories)]) - - prompt = self.build_prompt( - template_name="enlarge_recall", - query=query, - memories_inline=text_memories, - ) - llm_response = self.process_llm.generate([{"role": "user", "content": prompt}]) - - json_result: dict = extract_json_obj(llm_response) - - logger.info( - "[recall_for_missing_memories] done | prompt=%s | llm_response=%s", - prompt, - llm_response, - ) - - hint = json_result.get("hint", "") - if len(hint) == 0: - return hint, False - return hint, json_result.get("trigger_recall", False) + # This optional model-only step may fail without invalidating retrieved data. + try: + text_memories = "\n".join([f"- {mem}" for i, mem in enumerate(memories)]) + prompt = self.build_prompt( + template_name="enlarge_recall", + query=query, + memories_inline=text_memories, + ) + llm_response = self.process_llm.generate([{"role": "user", "content": prompt}]) + json_result: dict = extract_json_obj(llm_response) + logger.info( + "[recall_for_missing_memories] done | prompt=%s | llm_response=%s", + prompt, + llm_response, + ) + hint = json_result.get("hint", "") + trigger = json_result.get("trigger_recall", False) + if not isinstance(hint, str) or not isinstance(trigger, bool): + raise ValueError("Invalid optional recall hint or trigger") + if len(hint) == 0: + return hint, False + return hint, trigger + except GraphDBError: + raise + except Exception: + logger.warning( + "Optional recall enhancement failed; keeping retrieved memories", exc_info=True + ) + return "", False def enhance_memories_with_query( self, diff --git a/src/memos/mem_scheduler/monitors/dispatcher_monitor.py b/src/memos/mem_scheduler/monitors/dispatcher_monitor.py index f30efa52f..24ab093fe 100644 --- a/src/memos/mem_scheduler/monitors/dispatcher_monitor.py +++ b/src/memos/mem_scheduler/monitors/dispatcher_monitor.py @@ -1,10 +1,11 @@ import threading -import time from time import perf_counter from memos.configs.mem_scheduler import BaseSchedulerConfig from memos.context.context import ContextThread, ContextThreadPoolExecutor +from memos.exceptions import MemOSError +from memos.lifecycle import CloseState from memos.log import get_logger from memos.mem_scheduler.general_modules.base import BaseSchedulerModule from memos.mem_scheduler.schemas.general_schemas import ( @@ -25,6 +26,9 @@ class SchedulerDispatcherMonitor(BaseSchedulerModule): def __init__(self, config: BaseSchedulerConfig): super().__init__() + self._close_state = CloseState() + self._stop_event = threading.Event() + self._recovery_paused = False self.config: BaseSchedulerConfig = config self.check_interval = self.config.get( @@ -119,8 +123,7 @@ def _monitor_loop(self) -> None: """Main monitoring loop that periodically checks all registered pools.""" logger.info(f"Starting monitor loop with {self.check_interval} second interval") - while self._running: - time.sleep(self.check_interval) + while not self._stop_event.wait(self.check_interval): try: self._check_pools_health() except Exception as e: @@ -247,53 +250,66 @@ def _restart_pool(self, name: str, pool_info: dict) -> None: name: Name of the pool to restart pool_info: Dictionary containing pool configuration """ - if self._restart_in_progress: - return - - self._restart_in_progress = True - logger.info(f"Attempting to restart thread pool '{name}'") - + with self._pool_lock: + if ( + self._close_state.closing.is_set() + or self._recovery_paused + or self._restart_in_progress + ): + return + self._restart_in_progress = True + logger.info("Attempting to restart thread pool %s", name) try: old_executor = pool_info["executor"] - self.dispatcher.shutdown() - - # Create new executor with same parameters + if old_executor not in self.dispatcher._retired_executors: + self.dispatcher._retired_executors.append(old_executor) + old_executor.shutdown(wait=False, cancel_futures=True) new_executor = ContextThreadPoolExecutor( max_workers=pool_info["max_workers"], - thread_name_prefix=self.dispatcher.thread_name_prefix, # pylint: disable=protected-access + thread_name_prefix=self.dispatcher.thread_name_prefix, + initializer=self.dispatcher._mark_worker, ) - self.unregister_pool(name=self.dispatcher_pool_name) - self.dispatcher.dispatcher_executor = new_executor - self.register_pool( - name=self.dispatcher_pool_name, - executor=self.dispatcher.dispatcher_executor, - max_workers=self.dispatcher.max_workers, - restart_on_failure=True, - ) - - # Replace in registry - start_time = perf_counter() + lock_started = perf_counter() with self._pool_lock: - pool_info["executor"] = new_executor - pool_info["failure_count"] = 0 - pool_info["healthy"] = True - pool_info["last_active"] = get_utc_now() - - elapsed_time = perf_counter() - start_time - if elapsed_time > 1: - logger.warning(f"Long lock wait: {elapsed_time:.3f}s") - - # Shutdown old executor - try: - old_executor.shutdown(wait=False) - except Exception as e: - logger.error(f"Error shutting down old executor: {e!s}", exc_info=True) - - logger.info(f"Successfully restarted thread pool '{name}'") - except Exception as e: - logger.error(f"Failed to restart pool '{name}': {e!s}", exc_info=True) + lock_wait = perf_counter() - lock_started + if lock_wait > 1: + logger.warning("Long lock wait: %.3fs", lock_wait) + if self._close_state.closing.is_set() or self._recovery_paused: + self.dispatcher._retired_executors.append(new_executor) + return + self.dispatcher.dispatcher_executor = new_executor + self.dispatcher.thread_manager.thread_pool_executor = new_executor + pool_info.update( + executor=new_executor, + failure_count=0, + healthy=True, + last_active=get_utc_now(), + ) + logger.info("Successfully restarted thread pool %s", name) + except Exception: + logger.exception("Failed to restart pool %s", name) finally: - self._restart_in_progress = False + with self._pool_lock: + self._restart_in_progress = False + + def begin_close(self) -> None: + """Block recovery publication before waiting for any in-flight work.""" + with self._pool_lock: + self._close_state.closing.set() + self._recovery_paused = True + self._running = False + self._stop_event.set() + + def close(self) -> None: + """End monitoring and recovery; executor ownership stays with dispatcher.""" + if self._monitor_thread is threading.current_thread(): + raise MemOSError("Monitor cannot close from its own thread") + self._close_state.close(self._quiesce) + + def _quiesce(self) -> None: + self.begin_close() + if self._monitor_thread is not None: + self._monitor_thread.join() def get_status(self, name: str | None = None) -> dict: """ @@ -326,30 +342,39 @@ def start(self) -> bool: Returns: bool: True if monitor started successfully, False if already running """ - if self._running: - logger.warning("Dispatcher Monitor is already running") - return False + with self._pool_lock: + self._close_state.ensure_open() + self._recovery_paused = False + self._stop_event.clear() + if self._running: + logger.warning("Dispatcher Monitor is already running") + return False - self._running = True - self._monitor_thread = ContextThread( - target=self._monitor_loop, name="threadpool_monitor", daemon=True - ) - self._monitor_thread.start() - logger.info("Dispatcher Monitor monitor started") - return True + self._running = True + self._monitor_thread = ContextThread( + target=self._monitor_loop, name="threadpool_monitor", daemon=True + ) + self._monitor_thread.start() + logger.info("Dispatcher Monitor monitor started") + return True def stop(self) -> None: """ Stop the monitoring thread and clean up all managed thread pools. Ensures proper shutdown of all monitored executors. """ + if self._monitor_thread is threading.current_thread(): + raise MemOSError("Monitor cannot stop from its own thread") if not self._running: return # Stop the monitoring loop - self._running = False + with self._pool_lock: + self._running = False + self._recovery_paused = True + self._stop_event.set() if self._monitor_thread and self._monitor_thread.is_alive(): - self._monitor_thread.join(timeout=5) + self._monitor_thread.join() # Shutdown all registered pools with self._pool_lock: diff --git a/src/memos/mem_scheduler/optimized_scheduler.py b/src/memos/mem_scheduler/optimized_scheduler.py index d6b566dfe..7553f9186 100644 --- a/src/memos/mem_scheduler/optimized_scheduler.py +++ b/src/memos/mem_scheduler/optimized_scheduler.py @@ -94,7 +94,8 @@ def submit_memory_history_async_task( ) # Submit async task - self.memos_message_queue.submit_messages([message]) + with self._lifecycle_entry(): + self.memos_message_queue.submit_messages([message]) logger.info(f"Submitted async fine search task for user {search_req.user_id}") return async_task_id diff --git a/src/memos/mem_scheduler/task_schedule_modules/base_handler.py b/src/memos/mem_scheduler/task_schedule_modules/base_handler.py index 603b038e1..498bf2c2c 100644 --- a/src/memos/mem_scheduler/task_schedule_modules/base_handler.py +++ b/src/memos/mem_scheduler/task_schedule_modules/base_handler.py @@ -3,6 +3,7 @@ from abc import abstractmethod from typing import TYPE_CHECKING +from memos.exceptions import GraphDBError from memos.log import get_logger from memos.mem_scheduler.utils.misc_utils import group_messages_by_user_and_mem_cube @@ -46,6 +47,8 @@ def process_grouped_messages( continue try: message_handler(user_id, mem_cube_id, batch) + except (GraphDBError, ValueError): + raise except Exception as e: self.handle_exception( e, f"Error processing batch for user {user_id}, mem_cube {mem_cube_id}" diff --git a/src/memos/mem_scheduler/task_schedule_modules/dispatcher.py b/src/memos/mem_scheduler/task_schedule_modules/dispatcher.py index 151a0f795..159b2bc30 100644 --- a/src/memos/mem_scheduler/task_schedule_modules/dispatcher.py +++ b/src/memos/mem_scheduler/task_schedule_modules/dispatcher.py @@ -13,6 +13,8 @@ generate_trace_id, set_request_context, ) +from memos.exceptions import MemOSError +from memos.lifecycle import CloseState from memos.log import get_logger from memos.mem_scheduler.general_modules.base import BaseSchedulerModule from memos.mem_scheduler.general_modules.task_threads import ThreadManager @@ -61,6 +63,10 @@ def __init__( ): super().__init__() self.config = config + self._close_state = CloseState() + self._worker_context = threading.local() + self._retired_executors = [] + self._closed_executors = set() # Main dispatcher thread pool self.max_workers = max_workers @@ -82,7 +88,9 @@ def __init__( self.thread_name_prefix = "dispatcher" if self.enable_parallel_dispatch: self.dispatcher_executor = ContextThreadPoolExecutor( - max_workers=self.max_workers, thread_name_prefix=self.thread_name_prefix + max_workers=self.max_workers, + thread_name_prefix=self.thread_name_prefix, + initializer=self._mark_worker, ) logger.info(f"Max works of dispatcher is set to {self.max_workers}") else: @@ -772,6 +780,35 @@ def run_multiple_tasks( logger.error(f"Multiple tasks execution failed: {e}", exc_info=True) raise + def _mark_worker(self) -> None: + """Identify this owner's workers before their first submitted task.""" + self._worker_context.active = True + + def is_worker_thread(self) -> bool: + """Include workers from retired pools, which keep the same owner marker.""" + return getattr(self._worker_context, "active", False) + + def close(self) -> None: + """Drain all current and retired executors after recovery is disabled.""" + if self.is_worker_thread(): + raise MemOSError("Dispatcher cannot close from its own worker") + self._close_state.close(self._close_executors) + + def _close_executors(self) -> None: + self._running = False + errors = [] + for executor in [self.dispatcher_executor, *self._retired_executors]: + if executor is None or executor in self._closed_executors: + continue + try: + executor.shutdown(wait=True, cancel_futures=True) + self._closed_executors.add(executor) + except Exception as error: + errors.append(error) + if errors: + raise MemOSError("Dispatcher executors have not all exited") from errors[0] + self._futures.clear() + def shutdown(self) -> None: """Gracefully shutdown the dispatcher.""" self._running = False diff --git a/src/memos/mem_scheduler/task_schedule_modules/handlers/feedback_handler.py b/src/memos/mem_scheduler/task_schedule_modules/handlers/feedback_handler.py index 445a50d8e..237d69c76 100644 --- a/src/memos/mem_scheduler/task_schedule_modules/handlers/feedback_handler.py +++ b/src/memos/mem_scheduler/task_schedule_modules/handlers/feedback_handler.py @@ -29,14 +29,12 @@ def batch_handler( self, user_id: str, mem_cube_id: str, batch: list[ScheduleMessageItem] ) -> None: for message in batch: - try: - self.process_single_feedback(message) - except Exception as e: - logger.error( - "Error processing feedbackMemory message: %s", - e, - exc_info=True, - ) + self.process_single_feedback(message) + + def handle_exception(self, e: Exception, message: str = "Error processing feedback") -> None: + """A required feedback failure must reach the dispatcher's failed task state.""" + logger.error("%s: %s", message, e, exc_info=True) + raise e def process_single_feedback(self, message: ScheduleMessageItem) -> None: mem_cube = self.scheduler_context.get_mem_cube() @@ -48,13 +46,9 @@ def process_single_feedback(self, message: ScheduleMessageItem) -> None: try: feedback_data = json.loads(content) if isinstance(content, str) else content if not isinstance(feedback_data, dict): - logger.error( - "Failed to decode feedback_data or it is not a dict: %s", feedback_data - ) - return - except json.JSONDecodeError: - logger.error("Invalid JSON content for feedback message: %s", content, exc_info=True) - return + raise ValueError("Feedback message content must be an object") + except json.JSONDecodeError as error: + raise ValueError("Feedback message content must be valid JSON") from error task_id = feedback_data.get("task_id") or message.task_id feedback_result = self.scheduler_context.get_feedback_server().process_feedback( diff --git a/src/memos/mem_scheduler/webservice_modules/rabbitmq_service.py b/src/memos/mem_scheduler/webservice_modules/rabbitmq_service.py index e39f4bbd9..165590e78 100644 --- a/src/memos/mem_scheduler/webservice_modules/rabbitmq_service.py +++ b/src/memos/mem_scheduler/webservice_modules/rabbitmq_service.py @@ -10,6 +10,7 @@ from memos.configs.mem_scheduler import AuthConfig, RabbitMQConfig from memos.context.context import ContextThread from memos.dependency import require_python_package +from memos.exceptions import MemOSError from memos.log import get_logger from memos.mem_scheduler.general_modules.base import BaseSchedulerModule from memos.mem_scheduler.general_modules.misc import AutoDroppingQueue @@ -448,38 +449,34 @@ def rabbit_reconnect(self): self.rabbitmq_channel = None self.initialize_rabbitmq() - def rabbitmq_close(self): - """Gracefully shutdown connection.""" + def rabbitmq_close(self) -> None: + """Stop the connection and surface incomplete cleanup for owner retry.""" + errors = [] with self._rabbitmq_lock: self._rabbitmq_stop_flag = True - - # Close channel if open + callbacks = [] if self.rabbitmq_channel and self.rabbitmq_channel.is_open: - try: - self.rabbitmq_channel.close() - except Exception as e: - logger.warning(f"Error closing channel: {e}") - - # Close connection if open + callbacks.append(self.rabbitmq_channel.close) if self.rabbitmq_connection: if self.rabbitmq_connection.is_open: - try: - self.rabbitmq_connection.close() - except Exception as e: - logger.warning(f"Error closing connection: {e}") - - # Stop IOLoop if running + callbacks.append(self.rabbitmq_connection.close) + callbacks.append(self.rabbitmq_connection.ioloop.stop) + for callback in callbacks: try: - self.rabbitmq_connection.ioloop.stop() - except Exception as e: - logger.warning(f"Error stopping IOLoop: {e}") - - # Wait for IOLoop thread to finish - if self._io_loop_thread and self._io_loop_thread.is_alive(): + callback() + except Exception as error: + errors.append(error) + logger.exception("RabbitMQ resource cleanup failed") + # Callbacks on the loop may need _rabbitmq_lock while it exits. + if self._io_loop_thread and self._io_loop_thread.is_alive(): + if self._io_loop_thread is threading.current_thread(): + errors.append(MemOSError("RabbitMQ cannot join its own IOLoop thread")) + else: self._io_loop_thread.join(timeout=5) if self._io_loop_thread.is_alive(): - logger.warning("IOLoop thread did not terminate cleanly") - + errors.append(MemOSError("RabbitMQ IOLoop thread has not exited")) + if errors: + raise MemOSError("RabbitMQ cleanup failed") from errors[0] logger.info("RabbitMQ connection closed") def _flush_cached_publish_messages(self): diff --git a/src/memos/mem_scheduler/worker.py b/src/memos/mem_scheduler/worker.py new file mode 100644 index 000000000..d90309e95 --- /dev/null +++ b/src/memos/mem_scheduler/worker.py @@ -0,0 +1,33 @@ +"""Process entry point: build every runtime dependency in the worker process.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + + +if TYPE_CHECKING: + from collections.abc import Callable + from multiprocessing.synchronize import Event + + from memos.mem_scheduler.base_scheduler import BaseScheduler + + +def run_scheduler_worker(factory: Callable[[], BaseScheduler], stop_event: Event) -> None: + """Run an independently assembled scheduler until a cooperative stop signal. + + Pass this function as a process target, not an existing Scheduler instance. + The caller chooses the multiprocessing context (spawn is recommended) and + supplies a picklable factory that creates/configures resources in the child. + The factory must clean up its own partial initialization if it raises. + + Use thread startup mode inside the worker and Redis for cross-process tasks. + A local scheduler queue remains local to its worker. The caller owns process + supervision; setting stop_event drains accepted work through close(), whereas + forcibly terminating a worker cannot guarantee application-level cleanup. + """ + scheduler = factory() + try: + scheduler.start() + stop_event.wait() + finally: + scheduler.close() diff --git a/src/memos/memories/textual/simple_tree.py b/src/memos/memories/textual/simple_tree.py index 2df819f3a..43a27b918 100644 --- a/src/memos/memories/textual/simple_tree.py +++ b/src/memos/memories/textual/simple_tree.py @@ -3,6 +3,7 @@ from memos.configs.memory import TreeTextMemoryConfig from memos.embedders.base import BaseEmbedder from memos.graph_dbs.base import BaseGraphDB +from memos.lifecycle import CloseState from memos.llms.base import BaseLLM from memos.log import get_logger from memos.mem_reader.base import BaseMemReader @@ -40,6 +41,8 @@ def __init__( include_embedding: bool = False, ): """Initialize memory with the given configuration.""" + self._close_state = CloseState() + self._owns_graph_store = False self.config: TreeTextMemoryConfig = config self.mode = self.config.mode logger.info(f"Tree mode is {self.mode}") diff --git a/src/memos/memories/textual/tree.py b/src/memos/memories/textual/tree.py index c9d74d43f..e150ff848 100644 --- a/src/memos/memories/textual/tree.py +++ b/src/memos/memories/textual/tree.py @@ -14,7 +14,10 @@ from memos.context.context import ContextThreadPoolExecutor from memos.dependency import require_python_package from memos.embedders.factory import EmbedderFactory, OllamaEmbedder +from memos.exceptions import MemoryError from memos.graph_dbs.factory import GraphStoreFactory, Neo4jGraphDB +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.lifecycle import CloseState from memos.llms.factory import AzureLLM, LLMFactory, OllamaLLM, OpenAILLM from memos.log import get_logger from memos.mem_reader.read_multi_modal.utils import detect_lang @@ -55,50 +58,69 @@ def __init__(self, config: TreeTextMemoryConfig): self.embedder: OllamaEmbedder = EmbedderFactory.from_config(config.embedder) self.graph_store: Neo4jGraphDB = GraphStoreFactory.from_config(config.graph_db) - self.search_strategy = config.search_strategy - self.bm25_retriever = ( - EnhancedBM25() if self.search_strategy and self.search_strategy["bm25"] else None - ) - - if config.reranker is None: - default_cfg = RerankerConfigFactory.model_validate( - { - "backend": "cosine_local", - "config": { - "level_weights": {"topic": 1.0, "concept": 1.0, "fact": 1.0}, - "level_field": "background", - }, - } - ) - self.reranker = RerankerFactory.from_config(default_cfg) - else: - self.reranker = RerankerFactory.from_config(config.reranker) - self.is_reorganize = config.reorganize - self.memory_manager: MemoryManager = MemoryManager( - self.graph_store, - self.embedder, - self.extractor_llm, - memory_size=config.memory_size - or { - "WorkingMemory": 20, - "LongTermMemory": 1500, - "UserMemory": 480, - }, - is_reorganize=self.is_reorganize, - ) - # Create internet retriever if configured - self.internet_retriever = None - if config.internet_retriever is not None: - self.internet_retriever = InternetRetrieverFactory.from_config( - config.internet_retriever, self.embedder + self._close_state = CloseState() + self._owns_graph_store = True + try: + self.search_strategy = config.search_strategy + self.bm25_retriever = ( + EnhancedBM25() if self.search_strategy and self.search_strategy["bm25"] else None ) - logger.info( - f"Internet retriever initialized with backend: {config.internet_retriever.backend}" + + if config.reranker is None: + default_cfg = RerankerConfigFactory.model_validate( + { + "backend": "cosine_local", + "config": { + "level_weights": {"topic": 1.0, "concept": 1.0, "fact": 1.0}, + "level_field": "background", + }, + } + ) + self.reranker = RerankerFactory.from_config(default_cfg) + else: + self.reranker = RerankerFactory.from_config(config.reranker) + self.is_reorganize = config.reorganize + self.memory_manager: MemoryManager = MemoryManager( + self.graph_store, + self.embedder, + self.extractor_llm, + memory_size=config.memory_size + or { + "WorkingMemory": 20, + "LongTermMemory": 1500, + "UserMemory": 480, + }, + is_reorganize=self.is_reorganize, ) - else: - logger.info("No internet retriever configured") - self.tokenizer = None - self.include_embedding = config.include_embedding or False + # Create internet retriever if configured + self.internet_retriever = None + if config.internet_retriever is not None: + self.internet_retriever = InternetRetrieverFactory.from_config( + config.internet_retriever, self.embedder + ) + logger.info( + f"Internet retriever initialized with backend: {config.internet_retriever.backend}" + ) + else: + logger.info("No internet retriever configured") + self.tokenizer = None + self.include_embedding = config.include_embedding or False + except BaseException: + try: + self.close() + except Exception: + logger.exception("Failed to close graph after text memory initialization failure") + raise + + def close(self) -> None: + """Release the graph created by this memory; injected graphs are borrowed.""" + + def release() -> None: + close = getattr(self.graph_store, "close", None) + if self._owns_graph_store and callable(close): + close() + + self._close_state.close(release) def add( self, @@ -384,11 +406,22 @@ def get_all( page_size: int | None = None, filter: dict | None = None, memory_type: list[str] | None = None, + *, + all_cubes: bool | None = None, + exclude_user_name: str | None = None, ) -> dict: """Get all memories. Returns: list[TextualMemoryItem]: List of all memories. """ + scope = {} + if all_cubes is not None or exclude_user_name is not None: + if not isinstance(self.graph_store, OpenGaussGraphDB): + raise ValueError("This graph backend does not support explicit Cube ranges") + if all_cubes is not None: + scope["all_cubes"] = all_cubes + if exclude_user_name is not None: + scope["exclude_user_name"] = exclude_user_name graph_output = self.graph_store.export_graph( user_name=user_name, user_id=user_id, @@ -396,6 +429,7 @@ def get_all( page_size=page_size, filter=filter, memory_type=memory_type, + **scope, ) return graph_output @@ -502,43 +536,47 @@ def _unlink_knowledge_relations( self.graph_store.update_node(related_id, updates, user_name=user_name) def load(self, dir: str, user_name: str | None = None) -> None: + """Read a graph file and merge it atomically into the selected Cube.""" + memory_file = os.path.join(dir, self.config.memory_filename) try: - memory_file = os.path.join(dir, self.config.memory_filename) - - if not os.path.exists(memory_file): - logger.warning(f"Memory file not found: {memory_file}") - return - with open(memory_file, encoding="utf-8") as f: memories = json.load(f) + except (OSError, UnicodeError, json.JSONDecodeError) as error: + raise MemoryError(f"Unable to read memory file: {memory_file}") from error - self.graph_store.import_graph(memories, user_name=user_name) - logger.info(f"Loaded {len(memories)} memories from {memory_file}") - - except FileNotFoundError: - logger.error(f"Memory file not found in directory: {dir}") - except json.JSONDecodeError as e: - logger.error(f"Error decoding JSON from memory file: {e}") - except Exception as e: - logger.error(f"An error occurred while loading memories: {e}") + self.graph_store.import_graph(memories, user_name=user_name) + logger.info("Loaded memories from %s", memory_file) def dump(self, dir: str, include_embedding: bool = False, user_name: str | None = None) -> None: - """Dump memories to os.path.join(dir, self.config.memory_filename)""" + """Publish a single-Cube graph file after it has been fully written.""" + options = {"status": []} if isinstance(self.graph_store, OpenGaussGraphDB) else {} + json_memories = self.graph_store.export_graph( + include_embedding=include_embedding, user_name=user_name, **options + ) + memory_file = os.path.join(dir, self.config.memory_filename) + temporary_path = None try: - json_memories = self.graph_store.export_graph( - include_embedding=include_embedding, user_name=user_name - ) - os.makedirs(dir, exist_ok=True) - memory_file = os.path.join(dir, self.config.memory_filename) - with open(memory_file, "w", encoding="utf-8") as f: + with tempfile.NamedTemporaryFile( + mode="w", encoding="utf-8", dir=dir, prefix=".memos-", suffix=".json", delete=False + ) as f: + temporary_path = f.name json.dump(json_memories, f, indent=4, ensure_ascii=False) - - logger.info(f"Dumped {len(json_memories.get('nodes'))} memories to {memory_file}") - - except Exception as e: - logger.error(f"An error occurred while dumping memories: {e}") - raise + f.flush() + os.fsync(f.fileno()) + os.replace(temporary_path, memory_file) + temporary_path = None + except (OSError, TypeError, ValueError) as error: + raise MemoryError(f"Unable to write memory file: {memory_file}") from error + finally: + if temporary_path is not None: + try: + os.unlink(temporary_path) + except FileNotFoundError: + pass + except OSError: + logger.exception("Unable to remove unpublished memory file %s", temporary_path) + logger.info("Dumped memories to %s", memory_file) def drop(self, keep_last_n: int = 30) -> None: """ diff --git a/src/memos/memories/textual/tree_text_memory/organize/manager.py b/src/memos/memories/textual/tree_text_memory/organize/manager.py index 1fe6a4811..5f3e0c9a1 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/manager.py +++ b/src/memos/memories/textual/tree_text_memory/organize/manager.py @@ -7,6 +7,7 @@ from memos.context.context import ContextThreadPoolExecutor from memos.embedders.factory import OllamaEmbedder +from memos.exceptions import GraphDBError from memos.graph_dbs.neo4j import Neo4jGraphDB from memos.llms.factory import AzureLLM, OllamaLLM, OpenAILLM from memos.log import get_logger @@ -20,34 +21,39 @@ logger = get_logger(__name__) -def extract_working_binding_ids(mem_items: list[TextualMemoryItem]) -> set[str]: - """ - Scan enhanced memory items for background hints like - "[working_binding:]" and collect those working memory IDs. - - We store the working<->long binding inside metadata.background when - initially adding memories in async mode, so we can later clean up - the temporary WorkingMemory nodes after mem_reader produces the - final LongTermMemory/UserMemory. - - Args: - mem_items: list of TextualMemoryItem we just added (enhanced memories) +def extract_working_binding_ids(mem_items: list[TextualMemoryItem | dict]) -> set[str]: + """Read structured bindings, with compatibility for legacy background hints. - Returns: - A set of working memory IDs (as strings) that should be deleted. + Historical self-bindings are not Working nodes and must never be deleted + as a side effect of cleaning up a separate memory. """ bindings: set[str] = set() pattern = re.compile(r"\[working_binding:([0-9a-fA-F-]{36})\]") for item in mem_items: + if isinstance(item, dict): + primary_id, metadata = item.get("id"), item.get("metadata") or {} + else: + primary_id, metadata = item.id, item.metadata + get_metadata = ( + metadata.get + if isinstance(metadata, dict) + else lambda key, metadata=metadata: getattr(metadata, key, None) + ) + binding = get_metadata("working_binding") + if not binding: + background = get_metadata("background") or "" + match = pattern.search(background) if isinstance(background, str) else None + binding = match.group(1) if match else None try: - bg = getattr(item.metadata, "background", "") or "" - except Exception: - bg = "" - if not isinstance(bg, str): + parsed = uuid.UUID(binding) if isinstance(binding, str) else None + except ValueError: continue - match = pattern.search(bg) - if match: - bindings.add(match.group(1)) + try: + primary = uuid.UUID(primary_id) if isinstance(primary_id, str) else None + except ValueError: + primary = None + if parsed is not None and parsed != primary: + bindings.add(binding) return bindings @@ -101,7 +107,7 @@ def add( user_name: Optional user name for the memories. mode: "sync" to cleanup and refresh after adding, "async" to skip. use_batch: If True, use batch database operations (more efficient for large batches). - If False, use parallel single-node operations (original behavior). + If False, submit each memory (including its Working pair) independently. Returns: List of added memory IDs. @@ -120,133 +126,91 @@ def add( def _add_memories_parallel( self, memories: list[TextualMemoryItem], user_name: str | None = None ) -> list[str]: - """ - Add memories using parallel single-node operations (original behavior). - """ - added_ids: list[str] = [] + """Submit memories independently while preserving their return order.""" with ContextThreadPoolExecutor(max_workers=10) as executor: - futures = {executor.submit(self._process_memory, m, user_name): m for m in memories} - for future in as_completed(futures, timeout=500): - try: - ids = future.result() - added_ids.extend(ids) - except Exception as e: - logger.exception("Memory processing error: ", exc_info=e) - logger.info(f"[MemoryManager: _add_memories_parallel] Added {len(added_ids)} memories") + futures = [executor.submit(self._process_memory, m, user_name) for m in memories] + # Keep correspondence with the input even when writes finish out of order. + added_ids = [node_id for future in futures for node_id in future.result()] + logger.info("[MemoryManager: _add_memories_parallel] Added %s memories", len(added_ids)) return added_ids def _add_memories_batch( self, memories: list[TextualMemoryItem], user_name: str | None = None, batch_size: int = 5 ) -> list[str]: - """ - Add memories using batch database operations (more efficient for large batches). - - Args: - memories: List of memory items to add. - user_name: Optional user name for the memories. - batch_size: Number of nodes to insert per batch. - - Returns: - List of added graph memory node IDs. - """ + """Write batches of input memories without splitting a Working/graph pair.""" + if batch_size < 1: + raise ValueError("batch_size must be positive") if not memories: return [] - - added_ids: list[str] = [] - working_nodes: list[dict] = [] - graph_nodes: list[dict] = [] - graph_node_ids: list[str] = [] - - for memory in memories: - working_id = memory.id if hasattr(memory, "id") else memory.id or str(uuid.uuid4()) - - if memory.metadata.memory_type in ( - "WorkingMemory", - "LongTermMemory", - "UserMemory", - "OuterMemory", - ): - working_metadata = memory.metadata.model_copy( - update={"memory_type": "WorkingMemory"} - ).model_dump(exclude_none=True) - working_metadata["updated_at"] = datetime.now().isoformat() - working_nodes.append( - { - "id": working_id, - "memory": memory.memory, - "metadata": working_metadata, - } - ) - if memory.metadata.memory_type in ( - "LongTermMemory", - "UserMemory", - "ToolSchemaMemory", - "ToolTrajectoryMemory", - "RawFileMemory", - "SkillMemory", - "PreferenceMemory", - ): - graph_node_id = ( - memory.id if hasattr(memory, "id") else memory.id or str(uuid.uuid4()) - ) - metadata_dict = memory.metadata.model_dump(exclude_none=True) - metadata_dict["updated_at"] = datetime.now().isoformat() - metadata_dict["working_binding"] = working_id - - # Add working_binding for fast mode - tags = metadata_dict.get("tags") or [] - if "mode:fast" in tags: - metadata_dict["is_fast"] = True # Temporal fix - prev_bg = metadata_dict.get("background", "") or "" - binding_line = f"[working_binding:{working_id}] direct built from raw inputs" - metadata_dict["background"] = ( - f"{prev_bg} || {binding_line}" if prev_bg else binding_line - ) - - graph_nodes.append( - { - "id": graph_node_id, - "memory": memory.memory, - "metadata": metadata_dict, - } - ) - graph_node_ids.append(graph_node_id) - added_ids.append(graph_node_id) - - def _submit_batches(nodes: list[dict], node_kind: str) -> None: - if not nodes: - return - - max_workers = min(8, max(1, len(nodes) // max(1, batch_size))) - with ContextThreadPoolExecutor(max_workers=max_workers) as executor: - futures: list[tuple[int, int, object]] = [] - for batch_index, i in enumerate(range(0, len(nodes), batch_size), start=1): - batch = nodes[i : i + batch_size] - fut = executor.submit( - self.graph_store.add_nodes_batch, batch, user_name=user_name - ) - futures.append((batch_index, len(batch), fut)) - - for idx, size, fut in futures: - try: - fut.result() - except Exception as e: - logger.exception( - f"Batch add {node_kind} nodes error (batch {idx}, size {size}): ", - exc_info=e, + prepared = [self._prepare_memory_nodes(memory) for memory in memories] + added_ids = [node_id for _, ids in prepared for node_id in ids] + with ContextThreadPoolExecutor(max_workers=min(8, len(memories))) as executor: + futures = [] + for offset in range(0, len(prepared), batch_size): + nodes = [ + node for group, _ in prepared[offset : offset + batch_size] for node in group + ] + if nodes: + futures.append( + executor.submit( + self.graph_store.add_nodes_batch, nodes, user_name=user_name ) - - # TODO: working id is same with item.id, need to fix, currently stop adding WorkingMemories here. - # here used to be: _submit_batches(working_nodes, "WorkingMemory") - _submit_batches(graph_nodes, "graph memory") - - if graph_node_ids and self.is_reorganize: + ) + for future in futures: + future.result() + if added_ids and self.is_reorganize: self.reorganizer.add_message( - QueueMessage(op="add", after_node=graph_node_ids, user_name=user_name) + QueueMessage(op="add", after_node=added_ids, user_name=user_name) ) - return added_ids + def _prepare_memory_nodes(self, memory: TextualMemoryItem) -> tuple[list[dict], list[str]]: + """Preserve the primary ID and derive a stable, separate Working identity.""" + kind = memory.metadata.memory_type + paired = kind in ("LongTermMemory", "UserMemory") + working_only = kind in ("WorkingMemory", "OuterMemory") + graph = paired or kind in ( + "ToolSchemaMemory", + "ToolTrajectoryMemory", + "RawFileMemory", + "SkillMemory", + "PreferenceMemory", + ) + if not working_only and not graph: + return [], [] + + metadata = memory.metadata.model_dump(exclude_none=True) + metadata["updated_at"] = datetime.now().isoformat() + metadata.pop("working_binding", None) + if "background" in metadata: + metadata["background"] = re.sub( + r"\[working_binding:[^\]]*\]", "", metadata["background"] + ) + nodes = [] + working_id = ( + str(uuid.uuid5(uuid.NAMESPACE_URL, "memos:working:" + memory.id)) + if paired + else memory.id + ) + if paired or working_only: + nodes.append( + { + "id": working_id, + "memory": memory.memory, + "metadata": {**metadata, "memory_type": "WorkingMemory"}, + } + ) + if graph: + if paired: + metadata["working_binding"] = working_id + if "mode:fast" in (metadata.get("tags") or []): + metadata["is_fast"] = True + hint = f"[working_binding:{working_id}] direct built from raw inputs" + background = metadata.get("background") or "" + metadata["background"] = f"{background} || {hint}" if background else hint + nodes.append({"id": memory.id, "memory": memory.memory, "metadata": metadata}) + return nodes, [memory.id] if graph else [] + def _cleanup_working_memory(self, user_name: str | None = None) -> None: """ Remove oldest WorkingMemory nodes to keep within size limit. @@ -257,6 +221,8 @@ def _cleanup_working_memory(self, user_name: str | None = None) -> None: keep_latest=self.memory_size["WorkingMemory"], user_name=user_name, ) + except (GraphDBError, ValueError): + raise except Exception: logger.warning(f"Remove WorkingMemory error: {traceback.format_exc()}") @@ -306,64 +272,15 @@ def _refresh_memory_size(self, user_name: str | None = None) -> None: } logger.info(f"[MemoryManager] Refreshed memory sizes: {self.current_memory_size}") - def _process_memory(self, memory: TextualMemoryItem, user_name: str | None = None): - """ - Process and add memory to different memory types. - - Behavior: - 1. Always create a WorkingMemory node from `memory` and get its node id. - 2. If `memory.metadata.memory_type` is "LongTermMemory" or "UserMemory", - also create a corresponding long/user node. - - In async mode, that long/user node's metadata will include - `working_binding` in `background` which records the WorkingMemory - node id created in step 1. - 3. Return ONLY the ids of the long/user nodes (NOT the working node id), - which preserves the previous external contract of `add()`. - """ - ids: list[str] = [] - futures = [] - - # TODO: working id is same with item.id, need to fix - working_id = memory.id if hasattr(memory, "id") else memory.id or str(uuid.uuid4()) - - with ContextThreadPoolExecutor(max_workers=2, thread_name_prefix="mem") as ex: - if memory.metadata.memory_type in ( - "WorkingMemory", - "LongTermMemory", - "UserMemory", - "OuterMemory", - ): - f_working = ex.submit( - self._add_memory_to_db, memory, "WorkingMemory", user_name, working_id - ) - futures.append(("working", f_working)) - - if memory.metadata.memory_type in ( - "LongTermMemory", - "UserMemory", - "ToolSchemaMemory", - "ToolTrajectoryMemory", - "RawFileMemory", - "SkillMemory", - "PreferenceMemory", - ): - f_graph = ex.submit( - self._add_to_graph_memory, - memory=memory, - memory_type=memory.metadata.memory_type, - user_name=user_name, - working_binding=working_id, - ) - futures.append(("long", f_graph)) - - for kind, fut in futures: - try: - res = fut.result() - if kind != "working" and isinstance(res, str) and res: - ids.append(res) - except Exception: - logger.warning("Parallel memory processing failed:\n%s", traceback.format_exc()) - + def _process_memory(self, memory: TextualMemoryItem, user_name: str | None = None) -> list[str]: + """Write a complete pair in one backend call; return only the graph ID.""" + nodes, ids = self._prepare_memory_nodes(memory) + if nodes: + self.graph_store.add_nodes_batch(nodes, user_name=user_name) + if ids and self.is_reorganize: + self.reorganizer.add_message( + QueueMessage(op="add", after_node=ids, user_name=user_name) + ) return ids def _add_memory_to_db( @@ -387,45 +304,6 @@ def _add_memory_to_db( self.graph_store.add_node(working_memory.id, working_memory.memory, metadata, user_name) return node_id - def _add_to_graph_memory( - self, - memory: TextualMemoryItem, - memory_type: str, - user_name: str | None = None, - working_binding: str | None = None, - ): - """ - Generalized method to add memory to a graph-based memory type (e.g., LongTermMemory, UserMemory). - """ - node_id = memory.id if hasattr(memory, "id") else str(uuid.uuid4()) - # Step 2: Add new node to graph - metadata_dict = memory.metadata.model_dump(exclude_none=True) - if working_binding: - metadata_dict["working_binding"] = working_binding - tags = metadata_dict.get("tags") or [] - if working_binding and ("mode:fast" in tags): - metadata_dict["is_fast"] = True # Temporal fix - prev_bg = metadata_dict.get("background", "") or "" - binding_line = f"[working_binding:{working_binding}] direct built from raw inputs" - if prev_bg: - metadata_dict["background"] = prev_bg + " || " + binding_line - else: - metadata_dict["background"] = binding_line - self.graph_store.add_node( - node_id, - memory.memory, - metadata_dict, - user_name=user_name, - ) - self.reorganizer.add_message( - QueueMessage( - op="add", - after_node=[node_id], - user_name=user_name, - ) - ) - return node_id - def _inherit_edges(self, from_id: str, to_id: str, user_name: str | None = None) -> None: """ Migrate all non-lineage edges from `from_id` to `to_id`, diff --git a/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py b/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py index b7fb6b1a0..a7795ee22 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py +++ b/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py @@ -257,7 +257,7 @@ def _check_deadline(where: str): if _check_deadline("[GraphStructureReorganize] Before loading candidates"): return raw_nodes = self.graph_store.get_structure_optimization_candidates( - scope, user_name=user_name + scope, include_embedding=True, user_name=user_name ) nodes = [GraphDBNode(**n) for n in raw_nodes] diff --git a/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py b/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py index b44d28218..995f0f890 100644 --- a/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py +++ b/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py @@ -5,6 +5,7 @@ from memos.context.context import ContextThreadPoolExecutor from memos.embedders.factory import OllamaEmbedder +from memos.exceptions import GraphDBError from memos.graph_dbs.factory import Neo4jGraphDB from memos.llms.factory import AzureLLM, OllamaLLM, OpenAILLM from memos.log import get_logger, summarize_textual_memories @@ -680,6 +681,8 @@ def _retrieve_from_keyword( user_name=user_name, tsquery_config="jiebaqry", ) + except (GraphDBError, ValueError): + raise except Exception: logger.warning( f"[PATH-KEYWORD] search_by_fulltext failed, scope={scope}, user_name={user_name}" diff --git a/src/memos/multi_mem_cube/composite_cube.py b/src/memos/multi_mem_cube/composite_cube.py index 11c5245ba..81fdd8227 100644 --- a/src/memos/multi_mem_cube/composite_cube.py +++ b/src/memos/multi_mem_cube/composite_cube.py @@ -1,6 +1,5 @@ from __future__ import annotations -from concurrent.futures import as_completed from dataclasses import dataclass from typing import TYPE_CHECKING, Any @@ -59,13 +58,16 @@ def _search_single_cube(view: SingleCubeView) -> dict[str, Any]: self.logger.info(f"[CompositeCubeView] fan-out search to cube={view.cube_id}") return view.search_memories(search_req) - # parallel search for each cube + # Submit together, then merge in the first requested order for stable ties. + ordered_views = {} + for view in self.cube_views: + ordered_views.setdefault(view.cube_id, view) with ContextThreadPoolExecutor(max_workers=2) as executor: - future_to_view = { - executor.submit(_search_single_cube, view): view for view in self.cube_views - } + futures = [ + executor.submit(_search_single_cube, view) for view in ordered_views.values() + ] - for future in as_completed(future_to_view): + for future in futures: cube_result = future.result() merged_results["text_mem"].extend(cube_result.get("text_mem", [])) merged_results["act_mem"].extend(cube_result.get("act_mem", [])) @@ -88,6 +90,9 @@ def feedback_memories(self, feedback_req: APIFeedbackRequest) -> list[dict[str, for view in self.cube_views: self.logger.info(f"[CompositeCubeView] fan-out add to cube={view.cube_id}") results = view.feedback_memories(feedback_req) - all_results.extend(results) + if isinstance(results, dict): + all_results.append(results) + else: + all_results.extend(results) return all_results diff --git a/src/memos/multi_mem_cube/single_cube.py b/src/memos/multi_mem_cube/single_cube.py index 0b9829b7f..f2090ff26 100644 --- a/src/memos/multi_mem_cube/single_cube.py +++ b/src/memos/multi_mem_cube/single_cube.py @@ -12,6 +12,7 @@ format_memory_item, post_process_textual_mem, ) +from memos.exceptions import GraphDBError from memos.log import get_logger, summarize_search_request, summarize_search_results from memos.mem_reader.utils import parse_keep_filter_response from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem @@ -133,7 +134,7 @@ def search_memories(self, search_req: APISearchRequest) -> dict[str, Any]: return memories_result @timed - def feedback_memories(self, feedback_req: APIFeedbackRequest) -> dict[str, Any]: + def feedback_memories(self, feedback_req: APIFeedbackRequest) -> dict[str, Any] | list: target_session_id = feedback_req.session_id or "default_session" if feedback_req.async_mode == "async": try: @@ -156,6 +157,7 @@ def feedback_memories(self, feedback_req: APIFeedbackRequest) -> dict[str, Any]: f"[SingleCubeView] cube={self.cube_id} Failed to submit FEEDBACK: {e}", exc_info=True, ) + raise return [] else: feedback_result = self.feedback_server.process_feedback( @@ -216,6 +218,8 @@ def _search_text( return [] return text_memories + except (GraphDBError, ValueError): + raise except Exception as e: self.logger.error("Error in search_text: %s; traceback: %s", e, traceback.format_exc()) return [] diff --git a/tests/api/test_component_ownership.py b/tests/api/test_component_ownership.py new file mode 100644 index 000000000..8cb3e475b --- /dev/null +++ b/tests/api/test_component_ownership.py @@ -0,0 +1,267 @@ +"""Owned API resources survive failures only until their close can be retried.""" + +from concurrent.futures import ThreadPoolExecutor +from threading import Event +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.api.handlers import component_init +from memos.api.lifecycle import shutdown_components +from memos.exceptions import GraphDBError, MemOSError +from memos.lifecycle import CloseState +from memos.mem_scheduler.general_modules import init_components_for_scheduler as standalone + + +@pytest.mark.parametrize("scheduler", [None, SimpleNamespace(), SimpleNamespace(close=None)]) +def test_api_closes_graph_without_active_scheduler(scheduler): + graph = Mock() + components = {"graph_db": graph, "mem_scheduler": scheduler} + shutdown_components(components) + shutdown_components(components) + assert graph.close.call_count == 2 + + +def test_api_waits_for_scheduler_and_retries_failed_cleanup(): + calls = [] + failure = GraphDBError("cannot stop") + scheduler = Mock(close=Mock(side_effect=[failure, None])) + graph = Mock(close=Mock(side_effect=lambda: calls.append("graph"))) + components = {"graph_db": graph, "mem_scheduler": scheduler} + with pytest.raises(MemOSError): + shutdown_components(components) + graph.close.assert_not_called() + shutdown_components(components) + assert calls == ["graph"] + assert scheduler.close.call_count == 2 + + +def test_api_retry_delegates_idempotence_after_scheduler_succeeded(): + graph = Mock(close=Mock(side_effect=[GraphDBError("pool"), None])) + scheduler = Mock() + components = {"graph_db": graph, "mem_scheduler": scheduler} + with pytest.raises(MemOSError): + shutdown_components(components) + shutdown_components(components) + assert scheduler.close.call_count == 2 + assert graph.close.call_count == 2 + + +@pytest.mark.parametrize("failure", [None, GraphDBError("close failed")]) +def test_concurrent_api_close_delegates_same_result_to_resource(failure, monkeypatch): + entered, release, joined = Event(), Event(), Event() + state = CloseState() + + def close(): + entered.set() + assert release.wait(3) + if failure: + raise failure + + cleanup = Mock(side_effect=close) + graph = Mock(close=Mock(side_effect=lambda: state.close(cleanup))) + components = {"graph_db": graph} + + def concurrent(): + shutdown_components(components) + + with ThreadPoolExecutor(2) as pool: + first = pool.submit(shutdown_components, components) + assert entered.wait(3) + original_wait = state._attempt.done.wait + + def observed_wait(*args, **kwargs): + joined.set() + return original_wait(*args, **kwargs) + + monkeypatch.setattr(state._attempt.done, "wait", observed_wait) + second = pool.submit(concurrent) + assert joined.wait(3) + assert not second.done() + release.set() + for future in (first, second): + if failure: + with pytest.raises(MemOSError): + future.result(3) + else: + future.result(3) + assert graph.close.call_count == 2 + cleanup.assert_called_once_with() + + +def prepare_components(monkeypatch, module): + """Mock model/services at their factories, keep the assembly function real.""" + config = SimpleNamespace(text_mem=SimpleNamespace(config=SimpleNamespace())) + monkeypatch.setattr(module.APIConfig, "get_default_cube_config", lambda: config) + monkeypatch.setattr(module.APIConfig, "is_dingding_bot_enabled", lambda: False) + monkeypatch.setattr( + module.APIConfig, "get_scheduler_config", lambda: {"backend": "optimized", "config": {}} + ) + for name in vars(module): + if name.startswith("build_") and name.endswith("_config"): + monkeypatch.setattr(module, name, Mock(return_value={})) + graph = Mock() + monkeypatch.setattr(module.GraphStoreFactory, "from_config", Mock(return_value=graph)) + for factory in ( + "LLMFactory", + "EmbedderFactory", + "MemReaderFactory", + "RerankerFactory", + "InternetRetrieverFactory", + ): + monkeypatch.setattr(getattr(module, factory), "from_config", Mock(return_value=Mock())) + for name in ( + "MemoryManager", + "FastTokenizer", + "SimpleTreeTextMemory", + "NaiveMemCube", + "SimpleMemFeedback", + ): + monkeypatch.setattr(module, name, Mock(return_value=Mock())) + monkeypatch.setattr(module, "build_plugin_context", Mock(return_value={"shared": {}})) + monkeypatch.setattr(module.plugin_manager, "discover", Mock()) + monkeypatch.setattr(module.plugin_manager, "init_components", Mock()) + monkeypatch.setenv("MEMSCHEDULER_USE_REDIS_QUEUE", "false") + monkeypatch.setenv("ENABLE_CHAT_API", "false") + if module is component_init: + monkeypatch.setattr(module, "SchedulerConfigFactory", Mock()) + monkeypatch.setattr(module.SchedulerFactory, "from_config", Mock(return_value=Mock())) + monkeypatch.setattr(module.BaseDBManager, "create_default_sqlite_engine", Mock()) + monkeypatch.setattr(module, "DeepSearchMemAgent", Mock()) + else: + monkeypatch.setattr( + "memos.mem_scheduler.orm_modules.api_redis_model.APIRedisDBManager.load_redis_engine_from_env", + Mock(return_value=None), + ) + return graph + + +@pytest.mark.parametrize("module", [component_init, standalone]) +@pytest.mark.parametrize( + "stage", + [ + "LLMFactory", + "EmbedderFactory", + "MemReaderFactory", + "RerankerFactory", + "InternetRetrieverFactory", + "MemoryManager", + "FastTokenizer", + "SimpleTreeTextMemory", + "NaiveMemCube", + "SimpleMemFeedback", + "plugin_manager", + ], +) +def test_assembly_failure_closes_owned_graph(monkeypatch, module, stage): + graph = prepare_components(monkeypatch, module) + error = ValueError("construction failed") + target = getattr(module, stage) + if stage.endswith("Factory"): + target.from_config.side_effect = error + elif stage == "plugin_manager": + target.init_components.side_effect = error + else: + target.side_effect = error + initialize = module.init_server if module is component_init else module.init_components + with pytest.raises(ValueError) as raised: + initialize() + assert raised.value is error + graph.close.assert_called_once_with() + + +@pytest.mark.parametrize("module", [component_init, standalone]) +def test_successful_assembly_retains_graph_ownership(monkeypatch, module): + graph = prepare_components(monkeypatch, module) + initialize = module.init_server if module is component_init else module.init_components + components = initialize() + assert components["graph_db"] is graph + graph.close.assert_not_called() + shutdown_components(components) + graph.close.assert_called_once_with() + + +@pytest.mark.parametrize("stage", ["initialize_modules", "init_mem_cube", "start"]) +def test_api_failed_scheduler_is_closed_before_graph(monkeypatch, stage): + graph = prepare_components(monkeypatch, component_init) + scheduler = component_init.SchedulerFactory.from_config.return_value + error = ValueError("scheduler failed") + getattr(scheduler, stage).side_effect = error + monkeypatch.setenv("API_SCHEDULER_ON", "true") + calls = [] + scheduler.close.side_effect = lambda: calls.append("scheduler") + graph.close.side_effect = lambda: calls.append("graph") + with pytest.raises(ValueError) as raised: + component_init.init_server() + assert raised.value is error + assert calls == ["scheduler", "graph"] + + +def test_cleanup_error_does_not_replace_construction_error(monkeypatch): + graph = prepare_components(monkeypatch, component_init) + original = ValueError("model failed") + component_init.LLMFactory.from_config.side_effect = original + graph.close.side_effect = GraphDBError("cleanup failed") + with pytest.raises(ValueError) as raised: + component_init.init_server() + assert raised.value is original + + +def test_legacy_graph_without_close_is_compatible(): + shutdown_components({"graph_db": SimpleNamespace()}) + shutdown_components(None) + shutdown_components({}) + + +@pytest.mark.parametrize("module", [component_init, standalone]) +@pytest.mark.parametrize("redis_result", [None, True, ValueError("unavailable")]) +def test_redis_initialization_does_not_bypass_graph_cleanup(monkeypatch, module, redis_result): + graph = prepare_components(monkeypatch, module) + monkeypatch.setenv("MEMSCHEDULER_USE_REDIS_QUEUE", "true") + redis = ( + Mock(side_effect=redis_result) + if isinstance(redis_result, Exception) + else Mock(return_value=redis_result) + ) + monkeypatch.setattr( + "memos.mem_scheduler.orm_modules.api_redis_model.APIRedisDBManager.load_redis_engine_from_env", + redis, + ) + initialize = module.init_server if module is component_init else module.init_components + components = initialize() + shutdown_components(components) + graph.close.assert_called_once_with() + + +@pytest.mark.parametrize("chat", [False, True]) +def test_api_optional_components_and_disabled_scheduler(monkeypatch, chat): + graph = prepare_components(monkeypatch, component_init) + monkeypatch.setenv("API_SCHEDULER_ON", "false") + monkeypatch.setenv("ENABLE_CHAT_API", str(chat).lower()) + monkeypatch.setattr( + component_init, + "build_chat_llm_config", + Mock(return_value=[{"config_class": {}, "support_models": ["example"]}]), + ) + monkeypatch.setattr(component_init.APIConfig, "is_dingding_bot_enabled", lambda: True) + bot = Mock() + monkeypatch.setattr( + "memos.memos_tools.notification_service.get_online_bot_function", lambda: bot + ) + components = component_init.init_server() + assert components["online_bot"] is bot + assert (components["chat_llms"] is not None) == chat + components["mem_scheduler"].start.assert_not_called() + shutdown_components(components) + graph.close.assert_called_once_with() + + +def test_standalone_preserves_original_error_when_cleanup_fails(monkeypatch): + graph = prepare_components(monkeypatch, standalone) + original = ValueError("model failed") + standalone.LLMFactory.from_config.side_effect = original + graph.close.side_effect = GraphDBError("cleanup failed") + with pytest.raises(ValueError) as raised: + standalone.init_components() + assert raised.value is original diff --git a/tests/api/test_dashboard_scope.py b/tests/api/test_dashboard_scope.py new file mode 100644 index 000000000..eb6001225 --- /dev/null +++ b/tests/api/test_dashboard_scope.py @@ -0,0 +1,227 @@ +"""OP-09 B1 scope selection, pruning and real-record-only response buckets.""" + +from copy import deepcopy +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from pydantic import ValidationError + +from memos.api.handlers.memory_handler import handle_get_memories_dashboard +from memos.api.product_models import GetMemoryDashboardRequest, GetMemoryRequest +from memos.exceptions import GraphDBError +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.memories.textual.tree import TreeTextMemory + + +PUBLIC = {"and": [{"object_type": "public"}, {"mem_cube_id": "P"}]} +USER = {"object_type": "user", "mem_cube_id": "P"} + + +def cube(open_gauss=True, nodes=None, total=0): + graph = OpenGaussGraphDB.__new__(OpenGaussGraphDB) if open_gauss else SimpleNamespace() + graph.export_graph = Mock(return_value={"nodes": nodes or [], "total_nodes": total}) + memory = TreeTextMemory.__new__(TreeTextMemory) + memory.graph_store = graph + return SimpleNamespace(text_mem=memory) + + +@pytest.mark.parametrize( + "fields,owner,all_cubes,excluded,business", + [ + ({}, None, False, None, None), + ({"all_cubes": None, "exclude_cube_id": None}, None, False, None, None), + ({"mem_cube_id": "A"}, "A", False, None, None), + ({"all_cubes": False}, None, False, None, None), + ({"all_cubes": False, "mem_cube_id": "A"}, "A", False, None, None), + ({"all_cubes": True}, None, True, None, None), + ({"all_cubes": True, "exclude_cube_id": "P"}, None, True, "P", None), + ({"filter": PUBLIC, "mem_cube_id": "P"}, "P", False, None, None), + ({"filter": PUBLIC, "all_cubes": False}, "P", False, None, None), + ({"filter": USER}, None, True, "P", None), + ({"filter": USER, "all_cubes": True}, None, True, "P", None), + ({"filter": USER, "exclude_cube_id": "P"}, None, True, "P", None), + ( + {"filter": {"and": [PUBLIC, PUBLIC, {"or": [{"tag": "x"}, {"tag": "y"}]}]}}, + "P", + False, + None, + {"and": [{"or": [{"tag": "x"}, {"tag": "y"}]}]}, + ), + ({"filter": {}}, None, False, None, None), + ({"filter": {"tag": {"in": ["x", "y"]}}}, None, False, None, {"tag": {"in": ["x", "y"]}}), + ], +) +def test_dashboard_resolves_once_and_preserves_request( + fields, owner, all_cubes, excluded, business +): + target = cube() + request = GetMemoryDashboardRequest(**fields, user_id="u", page=2, page_size=3) + before = deepcopy(request.model_dump()) + response = handle_get_memories_dashboard(request, target) + calls = target.text_mem.graph_store.export_graph.call_args_list + assert len(calls) == 4 + for call in calls: + assert call.kwargs["user_name"] == owner + assert call.kwargs["all_cubes"] is all_cubes + assert call.kwargs.get("exclude_user_name") == excluded + assert call.kwargs["filter"] == business + assert call.kwargs["user_id"] == "u" + assert (call.kwargs["page"], call.kwargs["page_size"]) == (2, 3) + assert request.model_dump() == before + assert response.data["text_mem"] == [] + + +@pytest.mark.parametrize( + "fields", + [ + {"exclude_cube_id": "P"}, + {"mem_cube_id": "A", "filter": PUBLIC}, + {"mem_cube_id": "A", "filter": USER}, + {"filter": {"or": [PUBLIC, {"tag": "x"}]}}, + {"filter": {"or": [{"and": [USER, {"tag": "x"}]}, {"tag": "y"}]}}, + {"all_cubes": True, "mem_cube_id": "A"}, + {"all_cubes": True, "filter": PUBLIC}, + {"all_cubes": False, "filter": USER}, + {"all_cubes": False, "exclude_cube_id": "P"}, + {"mem_cube_id": "A", "exclude_cube_id": "P"}, + {"filter": PUBLIC, "exclude_cube_id": "P"}, + {"filter": USER, "exclude_cube_id": "Q"}, + {"mem_cube_id": " "}, + *( + {"filter": condition} + for condition in [ + {"object_type": "public"}, + {"mem_cube_id": "P"}, + {"object_type": "unknown", "mem_cube_id": "P"}, + {"object_type": [], "mem_cube_id": "P"}, + {"object_type": "public", "mem_cube_id": " "}, + {"object_type": "public", "mem_cube_id": 3}, + {"and": [PUBLIC, USER]}, + {"and": [PUBLIC, {"mem_cube_id": "Q"}]}, + {"and": []}, + {"or": []}, + {"and": {}}, + {"or": [None]}, + {"or": [{}]}, + {"and": [PUBLIC, {}]}, + {"or": ["tag"]}, + ] + ), + ], +) +def test_invalid_scope_or_original_structure_never_calls_database(fields): + target = cube() + with pytest.raises(ValueError): + handle_get_memories_dashboard(GetMemoryDashboardRequest(**fields), target) + target.text_mem.graph_store.export_graph.assert_not_called() + + +@pytest.mark.parametrize("fields", [{}, {"all_cubes": None, "exclude_cube_id": None}]) +def test_legacy_requests_keep_original_parameters(fields): + target = cube(False) + request = GetMemoryDashboardRequest(mem_cube_id="A", filter=PUBLIC, **fields) + handle_get_memories_dashboard(request, target) + for call in target.text_mem.graph_store.export_graph.call_args_list: + assert call.kwargs["user_name"] == "A" + assert call.kwargs["filter"] == PUBLIC + assert "all_cubes" not in call.kwargs and "exclude_user_name" not in call.kwargs + + +@pytest.mark.parametrize( + "fields", [{"all_cubes": False}, {"all_cubes": True}, {"exclude_cube_id": "P"}] +) +def test_legacy_backends_reject_explicit_new_inputs(fields): + target = cube(False) + with pytest.raises(ValueError, match="support"): + handle_get_memories_dashboard(GetMemoryDashboardRequest(**fields), target) + target.text_mem.graph_store.export_graph.assert_not_called() + + +@pytest.mark.parametrize("open_gauss", [False, True]) +@pytest.mark.parametrize( + "fields", + [{}, {"mem_cube_id": "A"}, {"filter": PUBLIC}, {"filter": USER}, {"page": 5, "page_size": 2}], +) +def test_no_virtual_buckets_and_empty_page_keeps_statistics(open_gauss, fields): + target = cube(open_gauss, total=7) + response = handle_get_memories_dashboard(GetMemoryDashboardRequest(**fields), target) + for category in ("text_mem", "tool_mem", "skill_mem", "pref_mem"): + assert response.data[category] == [] + assert response.data["statistics"] == { + "total_text_nodes": 7, + "total_tool_nodes": 7, + "total_skill_nodes": 7, + "total_preference_nodes": 7, + } + + +def test_real_nodes_keep_cross_cube_identity_and_bucket_counts(): + nodes = [ + {"id": "same", "memory": owner, "metadata": {"user_name": owner}} + for owner in ("A", "B", "A") + ] + response = handle_get_memories_dashboard( + GetMemoryDashboardRequest(), cube(nodes=nodes, total=8) + ) + for category in ("text_mem", "tool_mem", "skill_mem", "pref_mem"): + assert response.data[category] == [ + {"cube_id": "A", "memories": [nodes[0], nodes[2]], "total_nodes": 2}, + {"cube_id": "B", "memories": [nodes[1]], "total_nodes": 1}, + ] + + +def test_disabled_categories_are_not_queried(): + target = cube() + request = GetMemoryDashboardRequest( + include_tool_memory=False, include_skill_memory=False, include_preference=False + ) + response = handle_get_memories_dashboard(request, target) + target.text_mem.graph_store.export_graph.assert_called_once() + assert all(value == 0 for value in response.data["statistics"].values()) + + +def test_pure_control_still_queries_and_storage_failure_is_visible(): + target = cube() + error = GraphDBError("storage failed") + target.text_mem.graph_store.export_graph.side_effect = error + with pytest.raises(GraphDBError) as raised: + handle_get_memories_dashboard(GetMemoryDashboardRequest(filter=PUBLIC), target) + assert raised.value is error + target.text_mem.graph_store.export_graph.assert_called_once() + + +@pytest.mark.parametrize("value", ["true", "false", 0, 1, [], {}]) +def test_new_boolean_is_strict(value): + with pytest.raises(ValidationError): + GetMemoryDashboardRequest(all_cubes=value) + + +@pytest.mark.parametrize("value", ["", " ", 3, False]) +def test_excluded_cube_must_be_nonblank_string(value): + with pytest.raises(ValidationError): + GetMemoryDashboardRequest(exclude_cube_id=value) + + +def test_optional_schema_and_ordinary_get_contract(): + schema = GetMemoryDashboardRequest.model_json_schema() + for name in ("all_cubes", "exclude_cube_id"): + assert name not in schema.get("required", []) + assert schema["properties"][name]["default"] is None + assert {"type": "null"} in schema["properties"][name]["anyOf"] + ordinary = GetMemoryRequest.model_json_schema() + assert "mem_cube_id" in ordinary["required"] + assert "all_cubes" not in ordinary["properties"] + + +def test_tree_defaults_are_single_cube_and_legacy_rejects_new_arguments(): + memory = cube().text_mem + memory.get_all() + assert "all_cubes" not in memory.graph_store.export_graph.call_args.kwargs + memory.get_all(exclude_user_name="P") + assert "all_cubes" not in memory.graph_store.export_graph.call_args.kwargs + assert memory.graph_store.export_graph.call_args.kwargs["exclude_user_name"] == "P" + legacy = cube(False).text_mem + with pytest.raises(ValueError, match="support"): + legacy.get_all(all_cubes=False) diff --git a/tests/api/test_feedback_failures.py b/tests/api/test_feedback_failures.py new file mode 100644 index 000000000..08389f9cf --- /dev/null +++ b/tests/api/test_feedback_failures.py @@ -0,0 +1,165 @@ +"""Feedback failures stay failures for HTTP consumers and direct cube users.""" + +import inspect + +from unittest.mock import Mock + +import pytest + +from fastapi import HTTPException +from test_server_router import client as _client +from test_server_router import mock_init_server as _mock_init_server + +from memos.api.handlers.add_handler import AddHandler +from memos.api.handlers.base_handler import HandlerDependencies +from memos.api.handlers.feedback_handler import FeedbackHandler +from memos.api.product_models import APIADDRequest, APIFeedbackRequest +from memos.exceptions import GraphDBError, MemOSError +from memos.multi_mem_cube.composite_cube import CompositeCubeView +from memos.multi_mem_cube.single_cube import SingleCubeView + + +client, mock_init_server = _client, _mock_init_server + + +@pytest.fixture +def components(): + dependencies = HandlerDependencies( + naive_mem_cube=Mock(), + mem_reader=Mock(), + mem_scheduler=Mock(), + feedback_server=Mock(), + searcher=Mock(), + reranker=Mock(), + ) + view = SingleCubeView( + cube_id="cube", + naive_mem_cube=dependencies.naive_mem_cube, + mem_reader=dependencies.mem_reader, + mem_scheduler=dependencies.mem_scheduler, + logger=Mock(), + feedback_server=dependencies.feedback_server, + searcher=None, + ) + view.add_memories = Mock(return_value=[]) + return dependencies, view + + +@pytest.mark.parametrize("mode", ["sync", "async"]) +@pytest.mark.parametrize("entry", ["add", "feedback"]) +def test_failed_feedback_returns_safe_error_without_ordinary_add(components, mode, entry): + dependencies, view = components + error = ( + GraphDBError("private driver diagnostic") + if mode == "sync" + else OSError("private queue diagnostic") + ) + dependencies.feedback_server.process_feedback.side_effect = error + dependencies.mem_scheduler.submit_messages.side_effect = error + handler = AddHandler(dependencies) if entry == "add" else FeedbackHandler(dependencies) + handler._build_cube_view = Mock(return_value=view) + request = ( + APIADDRequest( + user_id="u", + is_feedback=True, + async_mode=mode, + messages=[{"role": "user", "content": "feedback"}], + ) + if entry == "add" + else APIFeedbackRequest( + user_id="u", history=[], feedback_content="feedback", async_mode=mode + ) + ) + call = ( + inspect.unwrap(AddHandler.handle_add_memories) + if entry == "add" + else FeedbackHandler.handle_feedback_memories + ) + with pytest.raises(HTTPException) as caught: + call(handler, request) + assert caught.value.status_code == 500 + assert caught.value.detail == "Feedback processing failed" + assert caught.value.__cause__ is error + view.add_memories.assert_not_called() + + +def test_direct_cube_submission_failure_is_visible(components): + dependencies, view = components + error = MemOSError("queue rejected") + dependencies.mem_scheduler.submit_messages.side_effect = error + with pytest.raises(MemOSError) as caught: + view.feedback_memories( + APIFeedbackRequest( + user_id="u", history=[], feedback_content="feedback", async_mode="async" + ) + ) + assert caught.value is error + + +@pytest.mark.parametrize("mode", ["sync", "async"]) +def test_composite_preserves_full_feedback_results(components, mode): + _, view = components + first = {"answer": "one", "answer_status": "generated", "record": {"add": ["a"], "update": []}} + second = {"answer": "two", "answer_status": "generated", "record": {"add": [], "update": ["b"]}} + view.feedback_server.process_feedback.side_effect = [first, second] + composite = CompositeCubeView(cube_views=[view, view], logger=Mock()) + result = composite.feedback_memories( + APIFeedbackRequest(user_id="u", history=[], feedback_content="feedback", async_mode=mode) + ) + assert result == ([first, second] if mode == "sync" else []) + + +@pytest.mark.parametrize("entry", ["add", "feedback"]) +@pytest.mark.parametrize("mode", ["sync", "async"]) +@pytest.mark.parametrize("failed", [False, True]) +def test_real_routes_accept_only_successful_feedback( + client, monkeypatch, components, entry, mode, failed +): + from memos.api.routers import server_router + + dependencies, view = components + record = { + "answer": "", + "answer_status": "not_requested", + "record": {"add": ["stored"], "update": []}, + } + dependencies.feedback_server.process_feedback.return_value = record + if failed: + dependencies.feedback_server.process_feedback.side_effect = GraphDBError( + "private diagnostic" + ) + dependencies.mem_scheduler.submit_messages.side_effect = OSError("private diagnostic") + handler = AddHandler(dependencies) if entry == "add" else FeedbackHandler(dependencies) + handler._build_cube_view = Mock(return_value=view) + monkeypatch.setattr(server_router, entry + "_handler", handler) + payload = {"user_id": "u", "async_mode": mode} + if entry == "add": + payload.update(is_feedback=True, messages=[{"role": "user", "content": "feedback"}]) + else: + payload.update(history=[], feedback_content="feedback") + response = client.post("/product/" + entry, json=payload) + assert response.status_code == (500 if failed else 200) + assert response.json()["code"] == response.status_code + if failed: + assert response.json()["message"] == "Feedback processing failed" + assert "private" not in response.text + else: + assert response.json()["data"] == ([[]] if mode == "async" else [record]) + view.add_memories.assert_not_called() + + +@pytest.mark.parametrize( + "info", [None, {"application": "agent"}, {"memory_type": "ignored", "application": "agent"}] +) +def test_ordinary_add_and_info_filter_are_unchanged(components, info): + dependencies, view = components + handler = AddHandler(dependencies) + handler._build_cube_view = Mock(return_value=view) + request = APIADDRequest( + user_id="u", messages=[{"role": "user", "content": "ordinary"}], info=info + ) + view.add_memories.return_value = [{"id": "ordinary-memory"}] + result = inspect.unwrap(AddHandler.handle_add_memories)(handler, request) + assert result.data == [{"id": "ordinary-memory"}] + assert request.info == (None if info is None else {"application": "agent"}) + dependencies.feedback_server.process_feedback.assert_not_called() diff --git a/tests/api/test_lifecycle.py b/tests/api/test_lifecycle.py index b2f7f7064..a5672c91c 100644 --- a/tests/api/test_lifecycle.py +++ b/tests/api/test_lifecycle.py @@ -1,4 +1,7 @@ +import pytest + from memos.api.lifecycle import shutdown_components +from memos.exceptions import MemOSError class SchedulerStub: @@ -26,7 +29,8 @@ def test_shutdown_components_stops_scheduler_before_rabbitmq_close(): def test_shutdown_components_still_closes_rabbitmq_when_stop_fails(): scheduler = SchedulerStub(fail_stop=True) - shutdown_components({"mem_scheduler": scheduler}) + with pytest.raises(MemOSError, match="shutdown failed"): + shutdown_components({"mem_scheduler": scheduler}) assert scheduler.calls == ["stop", "rabbitmq_close"] diff --git a/tests/api/test_opengauss_config.py b/tests/api/test_opengauss_config.py new file mode 100644 index 000000000..4ef728afc --- /dev/null +++ b/tests/api/test_opengauss_config.py @@ -0,0 +1,147 @@ +"""OP-09 R09-01/02: independent configuration expectations.""" + +from unittest.mock import Mock + +import pytest + +from memos.api.config import APIConfig +from memos.api.handlers.config_builders import build_graph_db_config +from memos.configs.graph_db import GraphDBConfigFactory, OpenGaussGraphDBConfig +from memos.graph_dbs.factory import GraphStoreFactory +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.mem_scheduler.general_modules.init_components_for_scheduler import ( + build_graph_db_config as scheduler_config, +) + + +@pytest.fixture +def env(monkeypatch): + import os + + for key in tuple(os.environ): + if key.startswith(("OPENGAUSS_", "POSTGRES_", "POLAR_", "NEO4J_", "MOS_")): + monkeypatch.delenv(key) + for key, value in { + "GRAPH_DB_BACKEND": "opengauss", + "OPENGAUSS_HOST": "db.example", + "OPENGAUSS_USER": "app", + "OPENGAUSS_PASSWORD": "synthetic", + "OPENGAUSS_DB": "memos", + "EMBEDDING_DIMENSION": "1024", + "MEMOS_USER_NAME": "default", + "ENABLE_INTERNET": "false", + "ENABLE_ACTIVATION_MEMORY": "false", + "ENABLE_PREFERENCE_MEMORY": "false", + }.items(): + monkeypatch.setenv(key, value) + + +def test_factories_register_formal_class_without_losing_existing_backends(env): + assert set(GraphDBConfigFactory.backend_to_class) == { + "neo4j", + "neo4j-community", + "polardb", + "postgres", + "opengauss", + } + assert GraphStoreFactory.backend_to_class["opengauss"] is OpenGaussGraphDB + assert not OpenGaussGraphDB.__abstractmethods__ + assert isinstance(build_graph_db_config().config, OpenGaussGraphDBConfig) + + +@pytest.mark.parametrize("user,scope", [(None, "default"), ("u-1", "memos_u1"), ("", "default")]) +def test_open_gauss_defaults(env, user, scope): + assert APIConfig.get_opengauss_config(user) == { + "host": "db.example", + "port": 5432, + "user": "app", + "password": "synthetic", + "db_name": "memos", + "user_name": scope, + "use_multi_db": False, + "embedding_dimension": 1024, + "minconn": 1, + "maxconn": 20, + "connect_timeout": 10, + "connection_wait_timeout": 30, + } + + +def test_environment_overrides(env, monkeypatch): + for key, value in { + "OPENGAUSS_PORT": "55432", + "OPENGAUSS_MIN_CONN": "2", + "OPENGAUSS_MAX_CONN": "7", + "OPENGAUSS_CONNECT_TIMEOUT": "8", + "OPENGAUSS_CONNECTION_WAIT_TIMEOUT": "9", + "EMBEDDING_DIMENSION": "3", + "MEMOS_USER_NAME": "custom", + }.items(): + monkeypatch.setenv(key, value) + config = build_graph_db_config(None).config + assert (config.port, config.minconn, config.maxconn) == (55432, 2, 7) + assert (config.connect_timeout, config.connection_wait_timeout) == (8, 9) + assert (config.embedding_dimension, config.user_name) == (3, "custom") + + +@pytest.mark.parametrize("name", ["HOST", "USER", "PASSWORD", "DB"]) +@pytest.mark.parametrize("value", [None, "", " "]) +def test_required_connection_fields(env, monkeypatch, name, value): + key = "OPENGAUSS_" + name + if value is None: + monkeypatch.delenv(key) + else: + monkeypatch.setenv(key, value) + with pytest.raises(ValueError): + build_graph_db_config() + + +def test_all_configuration_entries_select_only_requested_backend(env, monkeypatch): + monkeypatch.setenv("POSTGRES_PORT", "broken") + for backend in ("neo4j", "neo4j_community", "polardb"): + monkeypatch.setattr(APIConfig, f"get_{backend}_config", Mock(side_effect=AssertionError)) + expected = build_graph_db_config() + assert scheduler_config() == expected + assert APIConfig.get_default_cube_config().text_mem.config.graph_db == expected + cube = Mock() + monkeypatch.setattr("memos.mem_cube.general.GeneralMemCube", cube) + APIConfig.create_user_config("tester", "default") + assert cube.call_args.args[0].text_mem.config.graph_db == expected + + +@pytest.mark.parametrize("backend", ["neo4j", "neo4j-community", "polardb", "postgres"]) +def test_legacy_selection_only_calls_selected_getter(env, monkeypatch, backend): + sentinel = {"selected": backend} + for name in ("neo4j", "neo4j-community", "polardb", "postgres", "opengauss"): + getter = ( + Mock(return_value=sentinel) if name == backend else Mock(side_effect=AssertionError) + ) + monkeypatch.setattr(APIConfig, "get_" + name.replace("-", "_") + "_config", getter) + monkeypatch.delenv("GRAPH_DB_BACKEND") + monkeypatch.setenv("NEO4J_BACKEND", backend.upper()) + assert APIConfig.get_graph_db_config("u") == {"backend": backend, "config": sentinel} + + +def test_precedence_default_and_invalid_backend(env, monkeypatch): + monkeypatch.setenv("NEO4J_BACKEND", "invalid") + assert APIConfig.get_graph_db_config()["backend"] == "opengauss" + monkeypatch.delenv("GRAPH_DB_BACKEND") + with pytest.raises(ValueError, match="backend"): + APIConfig.get_graph_db_config() + monkeypatch.delenv("NEO4J_BACKEND") + assert APIConfig.get_graph_db_config()["backend"] == "neo4j-community" + + +def test_disabled_default_cube_does_not_read_connection(env, monkeypatch): + monkeypatch.setenv("MOS_ENABLE_DEFAULT_CUBE_CONFIG", "false") + monkeypatch.delenv("OPENGAUSS_HOST") + assert APIConfig.get_default_cube_config() is None + + +def test_user_cube_keeps_scheduler_and_mysql_configuration(env, monkeypatch): + monkeypatch.setenv("MOS_ENABLE_SCHEDULER", "true") + monkeypatch.setenv("MOS_USER_MANAGER_BACKEND", "mysql") + monkeypatch.setattr("memos.mem_cube.general.GeneralMemCube", Mock()) + config, _ = APIConfig.create_user_config("tester", "default") + assert config.enable_mem_scheduler + assert config.user_manager.backend == "mysql" diff --git a/tests/api/test_search_identity.py b/tests/api/test_search_identity.py new file mode 100644 index 000000000..575ff9cd3 --- /dev/null +++ b/tests/api/test_search_identity.py @@ -0,0 +1,203 @@ +"""Cross-Cube identity, stable fan-out and required storage failure propagation.""" + +from copy import deepcopy +from threading import Event +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.api.handlers.formatters_handler import rerank_knowledge_mem +from memos.api.product_models import APISearchRequest +from memos.exceptions import GraphDBError +from memos.memories.textual.tree_text_memory.retrieve.searcher import Searcher +from memos.multi_mem_cube.composite_cube import CompositeCubeView +from memos.multi_mem_cube.single_cube import SingleCubeView +from memos.types.general_types import UserContext + + +def item(key, text, source=None, score=0.5): + return { + "id": key, + "memory": text, + "metadata": { + "memory_type": "LongTermMemory", + "relativity": score, + "sources": [] if source is None else [{"type": "file", "content": source}], + }, + } + + +@pytest.mark.parametrize("query,prefix", [("question", "Memory: "), ("是什么记忆", "记忆:")]) +@pytest.mark.parametrize("strip", [False, True]) +def test_formatter_preserves_original_cube_per_record(query, prefix, strip): + groups = [ + { + "cube_id": "A", + "memories": [item("same", "alpha", "source alpha", 0.9), item("chat", "chat A")], + }, + { + "cube_id": "B", + "memories": [item("same", "beta", "source beta", 0.8), item("chat", "chat B")], + }, + {"cube_id": "C", "memories": [item("duplicate", "alpha", "source alpha", 0.7)]}, + ] + result = rerank_knowledge_mem(None, query, groups, top_k=4, strip_conversation_sources=strip) + assert [(b["cube_id"], [m["id"] for m in b["memories"]]) for b in result] == [ + ("A", ["same", "chat"]), + ("B", ["same", "chat"]), + ] + assert result[0]["memories"][0]["memory"].startswith(prefix + "alpha") + assert result[1]["memories"][0]["metadata"]["relativity"] == 0.8 + assert result[0]["memories"][0]["metadata"]["sources"] == [] + + +def test_formatter_empty_and_quota(): + assert rerank_knowledge_mem(None, "query", [], 3) == [] + groups = [ + { + "cube_id": "A", + "memories": [ + item("one", "one", "file"), + item("two", "two", "file"), + item("three", "conversation"), + ], + } + ] + result = rerank_knowledge_mem(None, "query", groups, 2) + assert [m["id"] for m in result[0]["memories"]] == ["one", "three"] + + +def test_composite_preserves_deduplicated_request_order_when_completion_reverses(): + finished_b = Event() + + def search(owner): + if owner == "A": + assert finished_b.wait(3) + else: + finished_b.set() + return { + "text_mem": [{"cube_id": owner, "memories": [item("same", owner)]}], + "pref_note": owner, + } + + views = [ + SimpleNamespace( + cube_id=owner, search_memories=Mock(side_effect=lambda req, owner=owner: search(owner)) + ) + for owner in ["A", "B", "A"] + ] + result = CompositeCubeView(views, Mock()).search_memories(Mock()) + assert [b["cube_id"] for b in result["text_mem"]] == ["A", "B"] + assert result["pref_note"] == "A | B" + views[2].search_memories.assert_not_called() + assert CompositeCubeView([], Mock()).search_memories(Mock())["text_mem"] == [] + empty = SimpleNamespace(cube_id="E", search_memories=lambda req: {}) + assert CompositeCubeView([empty], Mock()).search_memories(Mock())["pref_note"] == "" + + +def view(): + return SingleCubeView("A", Mock(), Mock(), Mock(), Mock(), Mock()) + + +@pytest.mark.parametrize( + "mode,method", [("fast", "_fast_search"), ("fine", "_fine_search"), ("mixture", "_mix_search")] +) +@pytest.mark.parametrize("error_type", [GraphDBError, ValueError]) +def test_required_search_failure_reaches_composite(mode, method, error_type): + one = view() + failure = error_type("required Cube failed") + failure.__cause__ = OSError("database cause") + setattr(one, method, Mock(side_effect=failure)) + request = APISearchRequest(query="query", user_id="u", readable_cube_ids=["A"], mode=mode) + with pytest.raises(error_type) as caught: + CompositeCubeView([one], Mock()).search_memories(request) + assert caught.value is failure and caught.value.__cause__.args == ("database cause",) + + +@pytest.mark.parametrize( + "mode,method", [("fast", "_fast_search"), ("fine", "_fine_search"), ("mixture", "_mix_search")] +) +def test_optional_nonstorage_failure_and_success_keep_existing_behavior(mode, method): + one = view() + context = UserContext(user_id="u", mem_cube_id="A", session_id="s") + setattr(one, method, Mock(return_value=[item("one", "one")])) + assert one._search_text(Mock(), context, mode)[0]["id"] == "one" + getattr(one, method).side_effect = RuntimeError("optional external") + assert one._search_text(Mock(), context, mode) == [] + assert one._search_text(Mock(), context, "unknown") == [] + + +@pytest.mark.parametrize("failure", [GraphDBError("sql"), ValueError("filter")]) +def test_keyword_storage_failure_is_not_empty(failure): + searcher = Searcher.__new__(Searcher) + searcher.graph_store = Mock(search_by_fulltext=Mock(side_effect=failure)) + searcher._extract_weighted_keyword_terms = Mock(return_value=["alpha"]) + with pytest.raises(type(failure)) as caught: + searcher._retrieve_from_keyword("query", None, [[1, 0, 0]], 3, "All", user_name="A") + assert caught.value is failure + + +@pytest.mark.parametrize( + "kind,embedding,terms", + [ + ("WorkingMemory", [[1]], ["x"]), + ("All", [], ["x"]), + ("All", [[1]], []), + ("All", [[1]], ["", " "]), + ], +) +def test_keyword_early_empty(kind, embedding, terms): + s = Searcher.__new__(Searcher) + s.graph_store = Mock() + s._extract_weighted_keyword_terms = Mock(return_value=terms) + assert s._retrieve_from_keyword("q", None, embedding, 3, kind, user_name="A") == [] + s.graph_store.search_by_fulltext.assert_not_called() + + +@pytest.mark.parametrize("hits", [None, [], RuntimeError("optional backend")]) +def test_keyword_optional_failure_and_empty(hits): + s = Searcher.__new__(Searcher) + s.graph_store = Mock() + s.graph_store.search_by_fulltext.side_effect = hits if isinstance(hits, Exception) else None + s.graph_store.search_by_fulltext.return_value = hits + s._extract_weighted_keyword_terms = Mock(return_value=["x"]) + assert s._retrieve_from_keyword("q", None, [[1]], 3, "LongTermMemory", user_name="A") == [] + + +def test_keyword_merge_and_legacy_decode_shapes(monkeypatch): + s = Searcher.__new__(Searcher) + s.graph_store = Mock() + s._extract_weighted_keyword_terms = Mock(return_value=["o'clock"]) + s.graph_store.search_by_fulltext.side_effect = [ + [{"id": ""}, {"id": "one", "score": 3}, {"id": "two", "score": 2}, {"id": "gone"}], + [ + {"id": "one", "score": 1}, + {"id": "two", "score": 4}, + {"id": "none", "score": 1}, + {"id": "new", "score": 0.5}, + ], + ] + nodes = [ + {"id": "one", "memory": "one", "metadata": {"metadata": {}}}, + {"id": "two", "memory": "two", "metadata": {}}, + {"id": "none", "memory": "none", "metadata": None}, + {"id": "new", "memory": "new"}, + ] + s.graph_store.get_nodes.return_value = nodes + original = deepcopy(nodes) + monkeypatch.setattr( + "memos.memories.textual.tree_text_memory.retrieve.searcher.TextualMemoryItem.from_dict", + lambda n: n, + ) + s._maybe_rerank = Mock(side_effect=lambda *args, **kw: kw["graph_results"]) + result = s._retrieve_from_keyword("q", None, [[1]], 10, "All", user_name="A") + assert [n["id"] for n in result] == ["two", "one", "none", "new"] + assert result[0]["metadata"]["keyword_score"] == 4 + assert result[1]["metadata"]["metadata"]["keyword_score"] == 3 + assert nodes == original + assert s.graph_store.search_by_fulltext.call_args.kwargs["query_words"] == ["'o''clock'"] + s.graph_store.search_by_fulltext.side_effect = None + s.graph_store.search_by_fulltext.return_value = [{"id": "gone"}] + s.graph_store.get_nodes.return_value = None + assert s._retrieve_from_keyword("q", None, [[1]], 1, "All", user_name="A", rerank=False) == [] diff --git a/tests/api/test_server_router.py b/tests/api/test_server_router.py index 5906697d9..46ff7a184 100644 --- a/tests/api/test_server_router.py +++ b/tests/api/test_server_router.py @@ -66,6 +66,82 @@ def client(mock_init_server): return TestClient(mock_init_server) +@pytest.mark.parametrize( + "path,method", + [ + ("/scheduler/wait/stream", "get"), + ("/chat/stream", "post"), + ("/chat/stream/playground", "post"), + ("/chat/stream/business_user", "post"), + ], +) +def test_sse_openapi_declares_actual_response_media_type(mock_init_server, path, method): + paths = mock_init_server.openapi()["paths"] + key = next(key for key in paths if key.endswith(path)) + content = paths[key][method]["responses"]["200"]["content"] + assert set(content) == {"text/event-stream"} + + +@pytest.mark.parametrize( + "path,handler_name", + [ + ("/chat/stream", "handle_chat_stream"), + ("/chat/stream/playground", "handle_chat_stream_playground"), + ("/chat/stream/business_user", "handle_chat_stream_for_business_user"), + ], +) +@pytest.mark.parametrize("available", [False, True]) +def test_chat_stream_keeps_event_bytes_and_unavailable_error( + client, monkeypatch, path, handler_name, available +): + from fastapi.responses import StreamingResponse + + from memos.api.routers import server_router + + handler = Mock() + getattr(handler, handler_name).return_value = StreamingResponse( + iter(["data: fixture\n\n"]), media_type="text/event-stream" + ) + monkeypatch.setattr(server_router, "chat_handler", handler if available else None) + payload = {"user_id": "u", "query": "q"} + if path.endswith("business_user"): + payload["business_key"] = "fixture-business" + response = client.post("/product" + path, json=payload) + assert response.status_code == (200 if available else 503), response.text + if available: + assert response.headers["content-type"].startswith("text/event-stream") + assert response.text == "data: fixture\n\n" + getattr(handler, handler_name).assert_called_once() + else: + assert "not available" in response.json()["message"] + + +def test_scheduler_stream_keeps_event_bytes_and_parameters(client, monkeypatch): + from fastapi.responses import StreamingResponse + + from memos.api.routers import server_router + + handler = Mock( + return_value=StreamingResponse(iter(["data: done\n\n"]), media_type="text/event-stream") + ) + monkeypatch.setattr( + server_router.handlers.scheduler_handler, "handle_scheduler_wait_stream", handler + ) + response = client.get( + "/product/scheduler/wait/stream", + params={"user_name": "A", "timeout_seconds": 2, "poll_interval": 0.1}, + ) + assert response.status_code == 200 and response.text == "data: done\n\n" + assert response.headers["content-type"].startswith("text/event-stream") + assert handler.call_args.kwargs == { + "user_name": "A", + "timeout_seconds": 2, + "poll_interval": 0.1, + "status_tracker": server_router.status_tracker, + "instance_id": server_router.INSTANCE_ID, + } + + @pytest.fixture def mock_handlers(): """Mock all handlers used by server_router.""" diff --git a/tests/api/test_shutdown_mapping.py b/tests/api/test_shutdown_mapping.py new file mode 100644 index 000000000..f668f465a --- /dev/null +++ b/tests/api/test_shutdown_mapping.py @@ -0,0 +1,63 @@ +"""Shutdown reads a component Mapping; lifecycle state belongs to each resource.""" + +from types import MappingProxyType, SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.api.lifecycle import shutdown_components +from memos.exceptions import MemOSError + + +@pytest.mark.parametrize("readonly", [False, True]) +@pytest.mark.parametrize("scheduler_present", [False, True]) +def test_shutdown_does_not_modify_component_mapping(readonly, scheduler_present): + order = [] + graph = Mock(close=Mock(side_effect=lambda: order.append("graph"))) + scheduler = Mock(close=Mock(side_effect=lambda: order.append("scheduler"))) + components = {"graph_db": graph} + if scheduler_present: + components["mem_scheduler"] = scheduler + before = dict(components) + shutdown_components(MappingProxyType(components) if readonly else components) + assert components == before + assert order == (["scheduler", "graph"] if scheduler_present else ["graph"]) + + +def test_api_delegates_repeat_safety_to_resources(): + graph, scheduler = Mock(), Mock() + components = MappingProxyType({"graph_db": graph, "mem_scheduler": scheduler}) + shutdown_components(components) + shutdown_components(components) + assert scheduler.close.call_count == graph.close.call_count == 2 + + +def test_failed_scheduler_keeps_graph_open_then_retry_delegates(): + scheduler = Mock(close=Mock(side_effect=[MemOSError("busy"), None])) + graph = Mock() + components = MappingProxyType({"mem_scheduler": scheduler, "graph_db": graph}) + with pytest.raises(MemOSError): + shutdown_components(components) + graph.close.assert_not_called() + shutdown_components(components) + graph.close.assert_called_once() + + +def test_legacy_without_close_still_stops_before_graph(): + order = [] + scheduler = SimpleNamespace( + stop=lambda: order.append("stop"), rabbitmq_close=lambda: order.append("rabbitmq") + ) + graph = SimpleNamespace(close=lambda: order.append("graph")) + shutdown_components(MappingProxyType({"mem_scheduler": scheduler, "graph_db": graph})) + assert order == ["stop", "rabbitmq", "graph"] + + +def test_legacy_rabbitmq_failure_does_not_close_graph(): + from types import SimpleNamespace + + scheduler = SimpleNamespace(stop=Mock(), rabbitmq_close=Mock(side_effect=ValueError("drain"))) + graph = Mock() + with pytest.raises(MemOSError): + shutdown_components({"mem_scheduler": scheduler, "graph_db": graph}) + graph.close.assert_not_called() diff --git a/tests/api/test_working_binding_cleanup.py b/tests/api/test_working_binding_cleanup.py new file mode 100644 index 000000000..46f9b8dff --- /dev/null +++ b/tests/api/test_working_binding_cleanup.py @@ -0,0 +1,64 @@ +"""Delete cleanup uses the same binding rules as refinement and Feedback.""" + +import uuid + +from unittest.mock import Mock, call + +import pytest + +from memos.api.handlers.memory_handler import handle_delete_memories +from memos.api.product_models import DeleteMemoryRequest + + +@pytest.mark.parametrize("source", ["structured", "legacy", "self", "none"]) +def test_delete_resolves_only_separate_working_node(source): + primary, working = str(uuid.uuid4()), str(uuid.uuid4()) + metadata = {} + if source == "structured": + metadata = {"working_binding": working} + elif source == "legacy": + metadata = {"background": f"[working_binding:{working}]"} + elif source == "self": + metadata = {"background": f"[working_binding:{primary}]", "working_binding": primary} + cube = Mock() + cube.text_mem.get_by_ids.return_value = [{"id": primary, "metadata": metadata}] + result = handle_delete_memories( + DeleteMemoryRequest(memory_ids=[primary], auto_cleanup_working=True), cube + ) + assert result.data["status"] == "success" + expected = [call([primary])] + if source in {"structured", "legacy"}: + expected.append(call([working])) + assert cube.text_mem.delete_by_memory_ids.call_args_list == expected + + +@pytest.mark.parametrize("failure", ["read", "cleanup", "main", "empty"]) +def test_delete_cleanup_keeps_existing_failure_contract(failure): + primary, working = str(uuid.uuid4()), str(uuid.uuid4()) + cube = Mock() + cube.text_mem.get_by_ids.return_value = [ + {"id": primary, "metadata": {"working_binding": working}} + ] + if failure == "read": + cube.text_mem.get_by_ids.side_effect = ValueError("read") + elif failure == "cleanup": + cube.text_mem.delete_by_memory_ids.side_effect = [None, ValueError("cleanup")] + elif failure == "main": + cube.text_mem.delete_by_memory_ids.side_effect = ValueError("main") + else: + cube.text_mem.get_by_ids.return_value = [] + result = handle_delete_memories( + DeleteMemoryRequest(memory_ids=[primary], auto_cleanup_working=True), cube + ) + assert result.data["status"] == ("failure" if failure == "main" else "success") + + +def test_delete_file_mode_keeps_scope(): + cube = Mock() + result = handle_delete_memories( + DeleteMemoryRequest(file_ids=["file"], writable_cube_ids=["A"]), cube + ) + assert result.data["status"] == "success" + cube.text_mem.delete_by_filter.assert_called_once_with( + writable_cube_ids=["A"], file_ids=["file"] + ) diff --git a/tests/configs/test_opengauss.py b/tests/configs/test_opengauss.py new file mode 100644 index 000000000..bae1d7492 --- /dev/null +++ b/tests/configs/test_opengauss.py @@ -0,0 +1,326 @@ +"""openGauss configuration contract (CFG-02 through CFG-04, REG-01).""" + +import json +import os +import subprocess +import sys + +from copy import deepcopy + +import pytest + +from pydantic import ValidationError + +from memos.configs import graph_db +from tests.utils import check_config_base_class + + +@pytest.fixture +def config_class(): + return graph_db.OpenGaussGraphDBConfig + + +@pytest.fixture +def valid_config(): + return { + "host": "localhost", + "user": "memos", + "password": "synthetic-test-password", + "db_name": "memos_test", + "user_name": "cube-a", + } + + +def test_config_fields_and_defaults(config_class, valid_config): + check_config_base_class( + config_class, + required_fields=["host", "user", "password", "db_name"], + optional_fields=[ + "port", + "embedding_dimension", + "minconn", + "maxconn", + "connect_timeout", + "connection_wait_timeout", + "use_multi_db", + "user_name", + ], + ) + config = config_class(**valid_config) + assert config.model_dump(exclude=set(valid_config)) == { + "port": 5432, + "embedding_dimension": 1024, + "minconn": 1, + "maxconn": 20, + "connect_timeout": 10, + "connection_wait_timeout": 30, + "use_multi_db": False, + } + + +@pytest.mark.parametrize("field", ["host", "user", "password", "db_name"]) +def test_required_field_is_located(config_class, valid_config, field): + del valid_config[field] + with pytest.raises(ValidationError) as exc: + config_class(**valid_config) + assert (field,) in [error["loc"] for error in exc.value.errors()] + + +@pytest.mark.parametrize("field", ["host", "user", "password", "db_name", "user_name"]) +@pytest.mark.parametrize("value", ["", " \t\n"]) +def test_blank_identifiers_and_credentials_are_rejected(config_class, valid_config, field, value): + valid_config[field] = value + with pytest.raises(ValidationError, match=field): + config_class(**valid_config) + + +@pytest.mark.parametrize("field", ["host", "user", "password", "db_name", "user_name"]) +@pytest.mark.parametrize("value", [42, False, ["value"], {"value": "name"}]) +def test_string_fields_are_strict(config_class, valid_config, field, value): + valid_config[field] = value + with pytest.raises(ValidationError) as exc: + config_class(**valid_config) + assert (field,) in [error["loc"] for error in exc.value.errors()] + + +@pytest.mark.parametrize("field", ["host", "user", "password", "db_name"]) +def test_required_strings_reject_none(config_class, valid_config, field): + valid_config[field] = None + with pytest.raises(ValidationError) as exc: + config_class(**valid_config) + assert (field,) in [error["loc"] for error in exc.value.errors()] + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("port", 1), + ("port", 65535), + ("embedding_dimension", 1), + ("embedding_dimension", 16000), + ("minconn", 20), + ("maxconn", 1), + ("connect_timeout", 1), + ("connect_timeout", 3600), + ("connection_wait_timeout", 1), + ("connection_wait_timeout", 3600), + ], +) +def test_numeric_boundaries_are_inclusive(config_class, valid_config, field, value): + valid_config[field] = value + assert getattr(config_class(**valid_config), field) == value + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("port", 0), + ("port", 65536), + ("embedding_dimension", 0), + ("embedding_dimension", -1), + ("embedding_dimension", 16001), + ("minconn", 0), + ("maxconn", 0), + ("connect_timeout", 0), + ("connect_timeout", 3601), + ("connection_wait_timeout", 0), + ("connection_wait_timeout", 3601), + ], +) +def test_numeric_values_outside_contract_are_rejected(config_class, valid_config, field, value): + valid_config[field] = value + with pytest.raises(ValidationError) as exc: + config_class(**valid_config) + assert (field,) in [error["loc"] for error in exc.value.errors()] + + +@pytest.mark.parametrize( + "field", + [ + "port", + "embedding_dimension", + "minconn", + "maxconn", + "connect_timeout", + "connection_wait_timeout", + ], +) +@pytest.mark.parametrize("value", ["10", 1.5, True, None]) +def test_numeric_fields_do_not_coerce_invalid_types(config_class, valid_config, field, value): + valid_config[field] = value + with pytest.raises(ValidationError) as exc: + config_class(**valid_config) + assert (field,) in [error["loc"] for error in exc.value.errors()] + + +def test_pool_minimum_cannot_exceed_maximum(config_class, valid_config): + with pytest.raises(ValidationError, match="minconn.*maxconn"): + config_class(**valid_config, minconn=3, maxconn=2) + + +@pytest.mark.parametrize("use_multi_db", [False, True]) +def test_explicit_scope_is_preserved_in_either_mode(config_class, valid_config, use_multi_db): + valid_config["user_name"] = " 范围甲 " + config = config_class(**valid_config, use_multi_db=use_multi_db) + assert config.user_name == " 范围甲 " + assert config.use_multi_db is use_multi_db + + +@pytest.mark.parametrize("explicit_none", [False, True]) +def test_physical_isolation_allows_absent_logical_scope(config_class, valid_config, explicit_none): + del valid_config["user_name"] + if explicit_none: + valid_config["user_name"] = None + config = config_class(**valid_config, use_multi_db=True) + assert config.user_name is None + + +@pytest.mark.parametrize("scope", ["", " \t\n"]) +def test_physical_isolation_rejects_explicit_blank_scope(config_class, valid_config, scope): + valid_config["user_name"] = scope + with pytest.raises(ValidationError, match="user_name"): + config_class(**valid_config, use_multi_db=True) + + +@pytest.mark.parametrize("explicit_none", [False, True]) +def test_shared_database_requires_scope(config_class, valid_config, explicit_none): + del valid_config["user_name"] + if explicit_none: + valid_config["user_name"] = None + with pytest.raises(ValidationError, match="user_name"): + config_class(**valid_config) + + +@pytest.mark.parametrize("value", ["false", "true", 0, 1, None]) +def test_isolation_mode_is_a_strict_boolean(config_class, valid_config, value): + with pytest.raises(ValidationError) as exc: + config_class(**valid_config, use_multi_db=value) + assert ("use_multi_db",) in [error["loc"] for error in exc.value.errors()] + + +def test_unknown_options_are_rejected(config_class, valid_config): + with pytest.raises(ValidationError) as exc: + config_class(**valid_config, shard_count=400) + assert exc.value.errors()[0]["type"] == "extra_forbidden" + assert exc.value.errors()[0]["loc"] == ("shard_count",) + + +def test_password_is_hidden_from_repr(config_class, valid_config): + assert valid_config["password"] not in repr(config_class(**valid_config)) + + +def test_overrides_and_serialization_preserve_values(config_class, valid_config, tmp_path): + values = { + **valid_config, + "host": "db.example", + "port": 55432, + "embedding_dimension": 3, + "minconn": 2, + "maxconn": 4, + "connect_timeout": 5, + "connection_wait_timeout": 15, + "use_multi_db": True, + } + before = deepcopy(values) + config = config_class(**values) + assert config.model_dump() == before + assert values == before + for suffix in ("json", "yaml"): + path = tmp_path / f"config.{suffix}" + getattr(config, f"to_{suffix}_file")(str(path)) + restored = getattr(config_class, f"from_{suffix}_file")(str(path)) + assert restored.model_dump() == before + + +def test_config_import_needs_no_database_driver_or_connection(): + script = """ +import importlib.abc +import json +import sys + +class NoDatabaseDriver(importlib.abc.MetaPathFinder): + def find_spec(self, fullname, path=None, target=None): + if fullname == 'psycopg2' or fullname.startswith('psycopg2.'): + raise ModuleNotFoundError('driver intentionally unavailable') + +sys.meta_path.insert(0, NoDatabaseDriver()) +from memos.configs.graph_db import OpenGaussGraphDBConfig +config = OpenGaussGraphDBConfig(host='localhost', user='memos', password='synthetic', db_name='memos', user_name='cube-a') +assert 'psycopg2' not in sys.modules +print('CONFIG_RESULT:' + json.dumps({'port': config.port, 'scope': config.user_name})) +""" + result = subprocess.run( + [sys.executable, "-c", script], + env=os.environ.copy(), + capture_output=True, + text=True, + timeout=60, + check=True, + ) + results = [ + line.removeprefix("CONFIG_RESULT:") + for line in result.stdout.splitlines() + if line.startswith("CONFIG_RESULT:") + ] + assert len(results) == 1 + assert json.loads(results[0]) == {"port": 5432, "scope": "cube-a"} + + +@pytest.mark.parametrize( + ("backend", "values", "class_name"), + [ + ( + "neo4j", + { + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "synthetic", + "db_name": "neo4j", + }, + "Neo4jGraphDBConfig", + ), + ( + "neo4j-community", + { + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "synthetic", + "db_name": "neo4j", + "use_multi_db": False, + "user_name": "cube-a", + "vec_config": {"backend": "qdrant", "config": {"collection_name": "memos"}}, + }, + "Neo4jCommunityGraphDBConfig", + ), + ( + "polardb", + {"host": "localhost", "user": "memos", "password": "synthetic", "db_name": "memos"}, + "PolarDBGraphDBConfig", + ), + ( + "postgres", + { + "host": "localhost", + "user": "memos", + "password": "synthetic", + "db_name": "memos", + "user_name": "cube-a", + }, + "PostgresGraphDBConfig", + ), + ], +) +def test_existing_backend_configs_are_still_selected(backend, values, class_name): + config = graph_db.GraphDBConfigFactory(backend=backend, config=values) + assert type(config.config).__name__ == class_name + + +def test_factories_expose_the_concrete_opengauss_store(): + from memos.graph_dbs.factory import GraphStoreFactory + from memos.graph_dbs.opengauss import OpenGaussGraphDB + + expected = {"neo4j", "neo4j-community", "polardb", "postgres", "opengauss"} + assert set(graph_db.GraphDBConfigFactory.backend_to_class) == expected + assert set(GraphStoreFactory.backend_to_class) == expected + assert GraphStoreFactory.backend_to_class["opengauss"] is OpenGaussGraphDB + assert not OpenGaussGraphDB.__abstractmethods__ diff --git a/tests/graph_dbs/conftest.py b/tests/graph_dbs/conftest.py new file mode 100644 index 000000000..565d91969 --- /dev/null +++ b/tests/graph_dbs/conftest.py @@ -0,0 +1,39 @@ +"""Opt-in driver doubles for unit tests; real database fixtures never use them.""" + +import sys + +from types import ModuleType +from unittest.mock import MagicMock + +import pytest + + +@pytest.fixture +def psycopg2_stub(monkeypatch): + """Replace only the driver boundary, retaining the production pool lifecycle.""" + driver = ModuleType("psycopg2") + driver.__path__ = [] + driver.Error = type("Error", (Exception,), {}) + driver.DatabaseError = type("DatabaseError", (driver.Error,), {}) + driver.OperationalError = type("OperationalError", (driver.DatabaseError,), {}) + driver.ProgrammingError = type("ProgrammingError", (driver.DatabaseError,), {}) + driver.InterfaceError = type("InterfaceError", (driver.Error,), {}) + + pool = ModuleType("psycopg2.pool") + pool.PoolError = type("PoolError", (driver.Error,), {}) + pool.ThreadedConnectionPool = MagicMock() + driver.pool = pool + + errors = ModuleType("psycopg2.errors") + errors.QueryCanceled = type("QueryCanceled", (driver.OperationalError,), {}) + errors.DeadlockDetected = type("DeadlockDetected", (driver.OperationalError,), {}) + errors.InsufficientPrivilege = type("InsufficientPrivilege", (driver.ProgrammingError,), {}) + driver.errors = errors + + for name, module in ( + ("psycopg2", driver), + ("psycopg2.pool", pool), + ("psycopg2.errors", errors), + ): + monkeypatch.setitem(sys.modules, name, module) + return driver diff --git a/tests/graph_dbs/opengauss_test_support.py b/tests/graph_dbs/opengauss_test_support.py new file mode 100644 index 000000000..eb5461ce6 --- /dev/null +++ b/tests/graph_dbs/opengauss_test_support.py @@ -0,0 +1,6 @@ +"""Keep the historical fixture name while exercising the concrete production class.""" + +from memos.graph_dbs.opengauss import OpenGaussGraphDB + + +PartialOpenGaussGraphDB = OpenGaussGraphDB diff --git a/tests/graph_dbs/test_opengauss_candidates.py b/tests/graph_dbs/test_opengauss_candidates.py new file mode 100644 index 000000000..b8b6b0d8e --- /dev/null +++ b/tests/graph_dbs/test_opengauss_candidates.py @@ -0,0 +1,123 @@ +"""Structure candidates: strict state, one snapshot and explicit projection.""" + +import inspect +import json + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity import capacity as _capacity +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +capacity, pool_case = _capacity, _pool_case + + +@pytest.mark.parametrize("kind", [None, "", " ", 1, False, "LongTermMemory\0"]) +def test_invalid_type_before_borrow(capacity, kind): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.get_structure_optimization_candidates(kind) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("cube", ["", " ", 1, False, "A\0"]) +def test_invalid_cube_before_borrow(capacity, cube): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.get_structure_optimization_candidates("LongTermMemory", user_name=cube) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("embedding", [None, 0, 1, "true", []]) +def test_invalid_projection_before_borrow(capacity, embedding): + db, pool, _, _ = capacity + with pytest.raises(ValueError, match="include_embedding"): + db.get_structure_optimization_candidates("LongTermMemory", embedding) + pool.getconn.assert_not_called() + + +def test_closed_precedes_invalid_input(capacity): + db, pool, _, _ = capacity + db.close() + with pytest.raises(GraphDBError, match="closed"): + db.get_structure_optimization_candidates(None, None, "\0") + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("embedding", [False, True]) +def test_single_bound_query_projection_and_decode_before_commit(capacity, embedding): + db, pool, connection, cursor = capacity + cube, kind = " 中文 Cube ' ", " DreamDiary ' " + metadata = {"status": "activated", "memory_type": kind, "info": {"x": [True, None]}} + cursor.fetchall.return_value = [ + (key, "text", json.dumps({**metadata, "id": key, "memory": "text", "user_name": cube}), vec) + for key, vec in [("a", "[1,0,0]"), ("b", None)] + ] + assert db.get_structure_optimization_candidates(kind, embedding, cube) == [ + {"id": key, "memory": "text", "metadata": {**metadata, **extra}} + for key, extra in [ + ("a", {"embedding": [1, 0, 0]} if embedding else {}), + ("b", {"embedding": None} if embedding else {}), + ] + ] + reads = [c.args for c in cursor.execute.call_args_list if "FROM ONLY" in c.args[0]] + assert len(reads) == 1 + query, params = reads[0] + assert cube not in query and kind not in query + assert cube in params.values() and json.dumps(kind) in params.values() + assert ("m.embedding::text" in query) == embedding + assert "COALESCE" not in query and "LIMIT" not in query + pool.getconn.assert_called_once() + connection.commit.assert_called_once() + + +def test_empty_physical_scope_and_public_signature(capacity): + db, _, _, cursor = capacity + db.config.user_name = None + db.config.use_multi_db = True + assert db.get_structure_optimization_candidates("InsightMemory") == [] + assert "" in cursor.execute.call_args.args[1].values() + assert list(inspect.signature(db.get_structure_optimization_candidates).parameters) == [ + "scope", + "include_embedding", + "user_name", + ] + + +@pytest.mark.parametrize("payload,vector", [("{", None), ("[]", None), (None, "[NaN]")]) +def test_bad_row_fails_whole_call_and_rolls_back(capacity, payload, vector): + db, pool, connection, cursor = capacity + valid = json.dumps({"id": "a", "memory": "text", "user_name": "A"}) + cursor.fetchall.return_value = [ + ("a", "text", valid, None), + ("a", "text", payload or valid, vector), + ] + with pytest.raises(GraphDBError): + db.get_structure_optimization_candidates("LongTermMemory", True) + connection.commit.assert_not_called() + assert connection.rollback.call_count == 2 + pool.putconn.assert_called_once_with(connection, close=False) + + +@pytest.mark.parametrize("stage", ["cursor", "execute", "fetch", "commit", "putconn"]) +def test_driver_failure_preserves_cause_and_releases_slot(capacity, stage): + import psycopg2 + + db, pool, connection, cursor = capacity + error = psycopg2.ProgrammingError("synthetic candidate failure") + if stage == "cursor": + connection.cursor.side_effect = error + elif stage == "execute": + cursor.execute.side_effect = error + elif stage == "fetch": + cursor.fetchall.side_effect = error + elif stage == "commit": + connection.commit.side_effect = error + else: + pool.putconn.side_effect = error + with pytest.raises(GraphDBError) as caught: + db.get_structure_optimization_candidates("LongTermMemory") + assert caught.value.__cause__ is error + assert pool.getconn.call_count == 1 and connection.commit.call_count <= 1 + assert db._active == 0 and db._closed == (stage == "putconn") diff --git a/tests/graph_dbs/test_opengauss_candidates_consumers.py b/tests/graph_dbs/test_opengauss_candidates_consumers.py new file mode 100644 index 000000000..94de40ee9 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_candidates_consumers.py @@ -0,0 +1,171 @@ +"""Nonempty real Reorganizer -> AGE writes, controlling only model services.""" + +import json +import os +import re +import uuid + +from datetime import datetime +from unittest.mock import MagicMock + +import pytest + +from memos.memories.textual.tree_text_memory.organize.reorganizer import GraphStructureReorganizer +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) +GROUPS = [ + [f"00000000-0000-4000-8000-{i:012}" for i in range(start, start + 12)] for start in (1, 13) +] + + +def seed(db): + for index, keys in enumerate(GROUPS): + db.add_nodes_batch( + [ + { + "id": key, + "memory": f"group-{index} child {key}", + "metadata": { + "memory_type": "LongTermMemory", + "status": "activated", + "key": f"group-{index}", + "embedding": [1, 0, 0] if index == 0 else [0, 1, 0], + "created_at": "2000-01-01T00:00:00", + "updated_at": "2000-01-01T00:00:00", + }, + } + for key in keys + ] + ) + db.add_node( + GROUPS[0][0], + "foreign", + {"status": "activated", "memory_type": "LongTermMemory"}, + user_name="B", + ) + + +def controlled_models(db, action=None): + llm, embedder = MagicMock(), MagicMock() + + def generate(messages): + prompt = messages[0]["content"] + keys = re.findall(r"- ID: ([\w-]+) \|", prompt) + if keys: + assert keys in GROUPS # Requires actual embedding partition; all 24 fails. + if action is not None: + action() + return json.dumps({"clusters": [{"ids": keys}]}) + indices = [i for i in (0, 1) if f"group-{i}" in prompt] + assert len(indices) == 1 + i = indices[0] + return json.dumps( + { + "key": f"parent-{i}", + "value": f"summary-{i}", + "tags": [f"group-{i}"], + "summary": f"background-{i}", + } + ) + + llm.generate.side_effect = generate + embedder.embed.side_effect = lambda texts: [[0, 0, 1] for _ in texts] + instance = GraphStructureReorganizer(db, llm, embedder, False) + instance._is_optimizing = {"LongTermMemory": False, "UserMemory": False} + return instance, llm, embedder + + +def test_real_partition_summary_and_parent_edges(stores, monkeypatch): + db = stores(maxconn=4) + seed(db) + keys = [key for group in GROUPS for key in group] + before = db.get_nodes(keys, include_embedding=True) + foreign = db.get_node(keys[0], user_name="B") + instance, llm, embedder = controlled_models(db) + spy = MagicMock(wraps=db.get_structure_optimization_candidates) + monkeypatch.setattr(db, "get_structure_optimization_candidates", spy) + start = datetime.now() + assert instance.optimize_structure(user_name="A") is None + end = datetime.now() + spy.assert_called_once_with("LongTermMemory", include_embedding=True, user_name="A") + assert llm.generate.call_count == 4 and embedder.embed.call_count == 2 + assert db.get_memory_count("LongTermMemory") == 26 + assert db.get_nodes(keys, include_embedding=True) == before + assert db.get_node(keys[0], user_name="B") == foreign + parents = [] + for i, group in enumerate(GROUPS): + edges = db.get_edges(group[0], "PARENT", "INCOMING") + assert len(edges) == 1 + parent_id = edges[0]["from"] + assert str(uuid.UUID(parent_id)) == parent_id and parent_id not in keys + parents.append(parent_id) + parent = db.get_node(parent_id, include_embedding=True) + assert parent["memory"] == f"summary-{i}" + metadata = parent["metadata"] + for field in ("created_at", "updated_at"): + assert start <= datetime.fromisoformat(metadata.pop(field)) <= end + assert metadata == { + "status": "activated", + "memory_type": "LongTermMemory", + "key": f"parent-{i}", + "tags": [f"group-{i}"], + "background": f"background-{i}", + "embedding": [0, 0, 1], + "usage": [], + "confidence": 0.66, + "type": "topic", + "evolve_to": [], + "version": 1, + "history": [], + "file_ids": [], + "knowledge_material_ids": [], + "knowledge_summary_ids": [], + "sources": [ + {"type": "chat", "content": f"group-{i} child {key}", "node_id": key} + for key in group + ], + } + assert db.get_edges(parent_id) == [ + {"from": parent_id, "to": key, "type": "PARENT"} for key in group + ] + assert len(set(parents)) == 2 + assert db.get_structure_optimization_candidates("LongTermMemory") == [] + assert not instance._is_optimizing["LongTermMemory"] + + +@pytest.mark.parametrize("action", ["delete", "model-failure"]) +def test_after_candidate_snapshot_failure_boundary(stores, action, caplog): + db = stores(maxconn=4) + seed(db) + + # Each model call happens after the candidate read, using an independent connection. + def intervene(): + if action == "delete": + db.delete_node(GROUPS[0][0]) + else: + raise ValueError("synthetic model failure") + + instance, llm, embedder = controlled_models(db, intervene) + assert instance.optimize_structure(user_name="A") is None + assert not instance._is_optimizing["LongTermMemory"] + if action == "model-failure": + assert "Cluster processing failed" in caplog.text + assert db.get_memory_count("LongTermMemory") == 24 + assert db.get_edges(GROUPS[0][0]) == [] + embedder.embed.assert_not_called() + else: + assert db.get_node(GROUPS[0][0]) is None + assert db.get_memory_count("LongTermMemory") == 25 + # The already-read source stays in the summary; missing-endpoint add_edge is a no-op. + parent = db.get_edges(GROUPS[0][1], "PARENT", "INCOMING")[0]["from"] + assert len(db.get_edges(parent)) == 11 + assert [ + source["node_id"] for source in db.get_node(parent)["metadata"]["sources"] + ] == GROUPS[0] + assert db.get_node(GROUPS[0][0], user_name="B")["memory"] == "foreign" diff --git a/tests/graph_dbs/test_opengauss_candidates_integration.py b/tests/graph_dbs/test_opengauss_candidates_integration.py new file mode 100644 index 000000000..f5e6782d3 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_candidates_integration.py @@ -0,0 +1,204 @@ +"""Real AGE candidate sets, projection, isolation and cancellation.""" + +import os + +from concurrent.futures import ThreadPoolExecutor +from copy import deepcopy + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity_integration import observe, wait_until +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) +TYPE = "LongTermMemory" + + +def node(key, status="activated", kind=TYPE, vector=None): + return { + "id": key, + "memory": "fixture " + key, + "metadata": {"status": status, "memory_type": kind, "embedding": vector}, + } + + +@pytest.mark.parametrize("embedding", [False, True]) +def test_strict_status_and_type_full_projection_and_isolation(stores, embedding): + db = stores() + inputs = [node("same", vector=[1, 0, 0]), node("z"), node("missing")] + del inputs[-1]["metadata"]["status"] + inputs += [ + node(str(i), state) + for i, state in enumerate( + [None, "resolving", "archived", "deleted", "Activated", True, ["activated"]] + ) + ] + inputs += [node("other-type", kind="UserMemory"), node("null-type", kind=None)] + original = deepcopy(inputs) + db.add_nodes_batch(inputs) + db.add_node("same", "B same ID", node("same")["metadata"], user_name="B") + assert db.get_structure_optimization_candidates(TYPE, embedding) == [ + { + "id": key, + "memory": "fixture " + key, + "metadata": { + "status": "activated", + "memory_type": TYPE, + **({"embedding": vec} if embedding else {}), + }, + } + for key, vec in [("same", [1, 0, 0]), ("z", None)] + ] + assert db.get_structure_optimization_candidates(TYPE, user_name="B") == [ + { + "id": "same", + "memory": "B same ID", + "metadata": {"status": "activated", "memory_type": TYPE}, + } + ] + assert db.get_structure_optimization_candidates(TYPE, user_name="empty") == [] + assert inputs == original + assert db.get_memory_count(TYPE) == 10 # Candidate strictness does not change capacity. + + +def test_parent_directions_self_loop_neighbor_state_and_dirty_edges(stores, database): + db = stores() + admin, _ = database + keep = ["a-other-relation", "b-cross-out", "c-cross-in", "d-isolated"] + db.add_nodes_batch([node(key) for key in [*keep, "in", "out", "self"]]) + db.add_nodes_batch([node("deleted", "deleted", "UserMemory"), node("archived", "archived")]) + db.add_edge("deleted", "in", "PARENT") + db.add_edge("out", "archived", "PARENT") + db.add_edge("self", "self", "PARENT") + db.add_edge("a-other-relation", "archived", "FOLLOWS") + db.add_node("foreign", "B", node("foreign")["metadata"], user_name="B") + _query( + admin, + """SELECT * FROM ag_catalog.cypher('memos_graph', $$ + MATCH (a:Memory {id:'b-cross-out',user_name:'A'}), + (b:Memory {id:'foreign',user_name:'B'}), + (c:Memory {id:'c-cross-in',user_name:'A'}) + CREATE (a)-[:PARENT]->(b), (b)-[:PARENT]->(c) + $$) AS (result agtype)""", + ) + assert [n["id"] for n in db.get_structure_optimization_candidates(TYPE)] == keep + assert [n["id"] for n in db.get_structure_optimization_candidates(TYPE, user_name="B")] == [ + "foreign" + ] + + +def test_no_limit_special_strings_and_physical_scope(stores): + db = stores(use_multi_db=True, user_name=None) + keys = [f"id-{i:04}" for i in range(510)] + [" 中文 ' %_\\ "] + kind = " DreamDiary ' " + db.add_nodes_batch([node(key, kind=kind) for key in reversed(keys)]) + assert [n["id"] for n in db.get_structure_optimization_candidates(kind)] == sorted(keys) + cube = " Cube 中文 ' %_\\ " + db.add_node("explicit", "exact", node("explicit", kind=kind)["metadata"], user_name=cube) + assert [n["id"] for n in db.get_structure_optimization_candidates(kind, user_name=cube)] == [ + "explicit" + ] + + +@pytest.mark.parametrize("direction", ["INCOMING", "OUTGOING"]) +@pytest.mark.parametrize("kind", [TYPE, "UserMemory"]) +@pytest.mark.parametrize( + "state", ["activated", "resolving", "archived", "deleted", None, "missing"] +) +def test_parent_neighbor_status_and_type_never_restore_eligibility(stores, direction, kind, state): + db = stores() + neighbor = node("neighbor", state, kind) + if state == "missing": + del neighbor["metadata"]["status"] + db.add_nodes_batch([node("candidate"), neighbor]) + source, target = ( + ("neighbor", "candidate") if direction == "INCOMING" else ("candidate", "neighbor") + ) + db.add_edge(source, target, "PARENT") + assert db.get_structure_optimization_candidates(TYPE) == [] + + +def test_candidate_and_content_share_one_statement_snapshot(stores, monkeypatch): + db, writer = stores(), stores() + db.add_nodes_batch([node("a"), node("b")]) + queries = [] + + def interleave(stage, cursor, rows): + if stage == "execute" and "NOT EXISTS" in cursor.statement: + queries.append(cursor.statement) + writer.add_edge("a", "b", "PARENT") + writer.add_node("b", "concurrent change", {"memory_type": TYPE, "status": "deleted"}) + + with monkeypatch.context() as patch: + observe(db, patch, interleave) + assert db.get_structure_optimization_candidates(TYPE) == [ + { + "id": key, + "memory": "fixture " + key, + "metadata": {"status": "activated", "memory_type": TYPE}, + } + for key in ("a", "b") + ] + assert len(queries) == 1 + assert db.get_structure_optimization_candidates(TYPE) == [] + assert db.get_node("b")["memory"] == "concurrent change" + + +def test_inconsistent_stored_properties_fail_then_pool_recovers(stores, database): + db = stores() + admin, _ = database + db.add_nodes_batch([node("a"), node("z")]) + _query( + admin, + """UPDATE memos_graph."Memory" SET properties = + '{"id":"wrong","memory":"fixture z","user_name":"A","memory_type":"LongTermMemory","status":"activated"}'::agtype + WHERE business_id='z' """, + ) + with pytest.raises(GraphDBError, match="Inconsistent"): + db.get_structure_optimization_candidates(TYPE) + db.add_nodes_batch([node("z")]) + assert [n["id"] for n in db.get_structure_optimization_candidates(TYPE)] == ["a", "z"] + + +def test_real_cancel_replaces_connection_and_preserves_candidates(stores, database): + db = stores(maxconn=1) + admin, _ = database + db.add_nodes_batch([node("a"), node("b")]) + connection = db.connection_pool.getconn() + pid = _query(connection, "SELECT pg_backend_pid()")[0][0] + connection.rollback() + db.connection_pool.putconn(connection) + admin.autocommit = False + try: + _query(admin, 'LOCK TABLE memos_graph."Memory" IN ACCESS EXCLUSIVE MODE') + with ThreadPoolExecutor(max_workers=1) as executor: + operation = executor.submit(db.get_structure_optimization_candidates, TYPE) + try: + wait_until( + lambda: _query( + admin, "SELECT 1 FROM pg_locks WHERE pid=%s AND NOT granted", (pid,) + ) + ) + connection.cancel() + with pytest.raises(GraphDBError) as caught: + operation.result(timeout=8) + assert caught.value.__cause__.pgcode == "57014" + finally: + admin.rollback() + finally: + admin.rollback() + admin.autocommit = True + assert connection.closed + assert [n["id"] for n in db.get_structure_optimization_candidates(TYPE)] == ["a", "b"] + replacement = db.connection_pool.getconn() + try: + assert replacement is not connection + finally: + db.connection_pool.putconn(replacement) diff --git a/tests/graph_dbs/test_opengauss_capacity.py b/tests/graph_dbs/test_opengauss_capacity.py new file mode 100644 index 000000000..d00f0460b --- /dev/null +++ b/tests/graph_dbs/test_opengauss_capacity.py @@ -0,0 +1,431 @@ +"""Counts/capacity contracts: validation, bound SQL and transaction failures.""" + +import inspect +import json +import sys + +from copy import deepcopy +from datetime import datetime + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +pool_case = _pool_case + +CALLS = [ + ("get_grouped_counts", (["memory_type"],)), + ("get_memory_count", ("WorkingMemory",)), + ("node_not_exist", ("WorkingMemory",)), + ("remove_oldest_memory", ("WorkingMemory", 2)), +] + + +@pytest.fixture +def capacity(pool_case): + _, db, pool, connection, _ = pool_case + cursor = connection.cursor.return_value.__enter__.return_value + cursor.fetchall.return_value = [] + cursor.fetchone.return_value = (0,) + return db, pool, connection, cursor + + +@pytest.mark.parametrize("method,args", CALLS) +def test_closed_precedes_validation(capacity, method, args): + db, pool, _, _ = capacity + db.close() + with pytest.raises(GraphDBError, match="closed"): + getattr(db, method)(*args, user_name="\0") + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("method,args", CALLS) +@pytest.mark.parametrize("scope", ["", " ", 1, False, "A\0"]) +def test_invalid_scope_before_borrow(capacity, method, args, scope): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + getattr(db, method)(*args, user_name=scope) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("method,args", CALLS[1:]) +@pytest.mark.parametrize("kind", [None, "", " ", 1, False, "WorkingMemory\0"]) +def test_invalid_memory_type_before_borrow(capacity, method, args, kind): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + getattr(db, method)(kind, *args[1:]) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize( + "fields", + [ + None, + [], + (), + "memory_type", + [1], + [None], + [[]], + ["memory_type", "memory_type"], + ["count"], + ["a.b"], + ["info.a.b"], + ["类型"], + ["a;"], + ["a\0"], + ["1bad"], + ], +) +def test_invalid_group_fields_before_borrow(capacity, fields): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.get_grouped_counts(fields) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize( + "where,params", + [ + (1, None), + (False, None), + (";", None), + ("true -- comment", None), + ("true /* comment */", None), + ("\0", None), + ("x=%s", None), + ("x=%(x)d", {"x": 1}), + ("%", None), + ("%(bad-key)s", {"bad-key": 1}), + ("%(x)s", None), + ("", {"x": 1}), + ("", []), + ("", {1: 1}), + ("%(memos_scope)s", {"memos_scope": "B"}), + ("%(memos_x)s", {"memos_x": 1}), + ("%(x)s", {"x": []}), + ("%(x)s", {"x": {}}), + ("%(x)s", {"x": b"x"}), + ("%(x)s", {"x": datetime(2026, 1, 1)}), + ("%(x)s", {"x": object()}), + ("%(x)s", {"x": float("inf")}), + ("%(x)s", {"x": float("nan")}), + ("%(x)s", {"x": "a\0"}), + ], +) +def test_invalid_predicate_before_borrow(capacity, where, params): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.get_grouped_counts(["memory_type"], where, params) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("where", [None, "", " ", "WHERE", " wHeRe "]) +def test_empty_predicate_and_empty_groups(capacity, where): + db, _, connection, cursor = capacity + assert db.get_grouped_counts(["memory_type"], where) == [] + statement, params = cursor.execute.call_args.args + assert isinstance(params, dict) and params["memos_scope"] == "A" + assert "AND ()" not in statement + cursor.execute.assert_any_call("SET LOCAL enable_hashagg = off") + connection.commit.assert_called_once() + + +@pytest.mark.parametrize("value", [None, True, 1, 1.5, "bound'", 10**400]) +def test_named_bindings_keep_predicate_grouping_and_inputs(capacity, value): + db, _, _, cursor = capacity + fields = ["memory_type", "info.count", "unknown"] + params = {"x": value} + before = deepcopy((fields, params)) + predicate = "business_id = %(x)s OR scope = %(x)s OR memory LIKE '100%%'" + db.get_grouped_counts(fields, " wHeRe " + predicate, params, "B'") + statement, bound = cursor.execute.call_args.args + assert f"AND ({predicate})" in statement + assert bound["memos_scope"] == "B'" and bound["x"] == value + assert "B'" not in statement and "memory_type" not in statement + assert all(key == "x" or key.startswith("memos_") for key in bound) + assert "GROUP BY" in statement and 'COLLATE "C"' in statement + assert (fields, params) == before + + +def test_where_prefix_requires_word_boundary(capacity): + db, _, _, cursor = capacity + db.get_grouped_counts(["info.x"], "WHEREVER") + assert "AND (WHEREVER)" in cursor.execute.call_args.args[0] + + +def test_group_decoding_preserves_json_types_and_age_decimals(capacity): + db, _, _, cursor = capacity + cursor.fetchall.return_value = [ + ('"1"', 1), + ("1", 2), + ('[.8, -.8, "text .8"]', 1), + ("true", 1), + ('{"a": 1.0}', 1), + (None, 2), + ] + result = db.get_grouped_counts(["info.v"]) + assert result == [ + {"info.v": "1", "count": 1}, + {"info.v": 1, "count": 2}, + {"info.v": [0.8, -0.8, "text .8"], "count": 1}, + {"info.v": True, "count": 1}, + {"info.v": {"a": 1.0}, "count": 1}, + {"info.v": None, "count": 2}, + ] + assert [type(row["info.v"]) for row in result] == [str, int, list, bool, dict, type(None)] + assert all(type(row["count"]) is int for row in result) + + +@pytest.mark.parametrize("payload", ["{", "not-json", 12, "NaN", "Infinity", "-Infinity", "[NaN]"]) +def test_corrupt_group_result_rolls_back(capacity, payload): + db, pool, connection, cursor = capacity + cursor.fetchall.return_value = [(payload, 1)] + with pytest.raises(GraphDBError) as caught: + db.get_grouped_counts(["info.x"]) + assert isinstance(caught.value.__cause__, TypeError | ValueError) + connection.commit.assert_not_called() + assert connection.rollback.call_count == 2 + pool.putconn.assert_called_once_with(connection, close=False) + + +@pytest.mark.parametrize( + "method,raw,expected", + [ + ("get_memory_count", 3, 3), + ("get_memory_count", 0, 0), + ("node_not_exist", 3, 0), + ("node_not_exist", 0, 1), + ], +) +def test_counts_include_every_status_and_bind_exact_type(capacity, method, raw, expected): + db, _, _, cursor = capacity + cursor.fetchone.return_value = (raw,) + result = getattr(db, method)(" WorkingMemory ", user_name="B'") + assert result == expected and type(result) is int + statement, params = cursor.execute.call_args.args + assert "status" not in statement and "B'" not in statement + assert "B'" in params.values() and json.dumps(" WorkingMemory ") in params.values() + + +@pytest.mark.parametrize( + "keep", [-1, True, False, 1.5, None, "", " ", "+2", "-1", "٢", "2", "2 ", "0" * 5000] +) +def test_invalid_capacity_before_borrow(capacity, keep): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.remove_oldest_memory("WorkingMemory", keep) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("keep", ["0" * 65, "9" * 65, "0" * 5000]) +def test_capacity_string_limit_is_independent_of_interpreter(capacity, keep): + db, pool, _, _ = capacity + previous = sys.get_int_max_str_digits() + try: + sys.set_int_max_str_digits(0) + with pytest.raises(ValueError, match="64"): + db.remove_oldest_memory("WorkingMemory", keep) + finally: + sys.set_int_max_str_digits(previous) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize( + "keep", + ["0" * 64, "9" * 64, "02", 10**5000], + ids=["zeros", "digits", "leading-zero", "large-int"], +) +def test_capacity_text_boundary_and_unbounded_integer_remain_valid(capacity, keep): + db, pool, _, _ = capacity + db.remove_oldest_memory("WorkingMemory", keep) + pool.getconn.assert_called_once() + + +@pytest.mark.parametrize( + "keep,deleted", + [ + (0, ["w1", "w2", "w3"]), + ("0", ["w1", "w2", "w3"]), + ("02", ["w1"]), + (2, ["w1"]), + (3, []), + (10**100, []), + ], +) +def test_capacity_ranks_full_locked_set_before_slicing(capacity, keep, deleted): + db, _, connection, cursor = capacity + cursor.fetchall.side_effect = [[("w1",), ("w2",), ("w3",)], [("w3",), ("w2",), ("w1",)], []] + assert db.remove_oldest_memory("WorkingMemory", keep) is None + calls = [call.args for call in cursor.execute.call_args_list] + assert ("SELECT pg_advisory_xact_lock(621503, %(lock)s)", {"lock": 2}) in calls + assert ("SET LOCAL TIME ZONE 'UTC'",) in calls + ranked = [(s, p) for s, *rest in calls if "NULLS LAST" in s for p in rest] + assert len(ranked) == 1 + assert "ANY(%(ids)s)" in ranked[0][0] and "LIMIT" not in ranked[0][0] + assert ["w1", "w2", "w3"] in ranked[0][1].values() + deletes = [args for args in calls if args[0].startswith("DELETE FROM ONLY")] + assert [args[1]["ids"] for args in deletes] == ([deleted] if deleted else []) + connection.commit.assert_called_once() + + +def test_capacity_empty_set_and_physical_scope(capacity): + db, _, connection, cursor = capacity + db.config.user_name = None + db.config.use_multi_db = True + db.remove_oldest_memory("WorkingMemory", 0) + assert not any("NULLS LAST" in c.args[0] for c in cursor.execute.call_args_list) + assert cursor.execute.call_args.args[1]["scope"] == "" + connection.commit.assert_called_once() + + +def test_capacity_chunks_are_sorted_and_share_one_transaction(capacity): + db, _, connection, cursor = capacity + ids = [f"x{i:04}" for i in range(1200)] + cursor.fetchall.side_effect = [[(i,) for i in ids], [(i,) for i in reversed(ids)], [], [], []] + db.remove_oldest_memory("WorkingMemory", 100) + deletes = [ + c.args[1]["ids"] + for c in cursor.execute.call_args_list + if c.args[0].startswith("DELETE FROM ONLY") + ] + assert deletes == [ids[:500], ids[500:1000], ids[1000:1100]] + bodies = [c.args[0] for c in cursor.execute.call_args_list if c.args[0].startswith("PREPARE")] + assert len(bodies) == 3 and all("-[r]->" in b and "DELETE r" in b for b in bodies) + connection.commit.assert_called_once() + + +@pytest.mark.parametrize("method,args", CALLS) +@pytest.mark.parametrize("stage", ["cursor", "execute", "fetch", "commit", "putconn"]) +def test_method_failures_preserve_cause_and_never_retry(capacity, method, args, stage): + import psycopg2 + + db, pool, connection, cursor = capacity + error = psycopg2.ProgrammingError("synthetic failure") + if stage == "cursor": + connection.cursor.side_effect = error + elif stage == "execute": + cursor.execute.side_effect = error + elif stage == "fetch": + cursor.fetchall.side_effect = error + cursor.fetchone.side_effect = error + elif stage == "commit": + connection.commit.side_effect = error + else: + pool.putconn.side_effect = error + with pytest.raises(GraphDBError) as caught: + getattr(db, method)(*args) + assert caught.value.__cause__ is error + assert pool.getconn.call_count == 1 and connection.commit.call_count <= 1 + assert db._active == 0 + assert db._closed == (stage == "putconn") + + +def test_public_signatures(capacity): + db, _, _, _ = capacity + assert list(inspect.signature(db.get_grouped_counts).parameters) == [ + "group_fields", + "where_clause", + "params", + "user_name", + ] + assert list(inspect.signature(db.get_memory_count).parameters) == ["memory_type", "user_name"] + assert list(inspect.signature(db.node_not_exist).parameters) == ["scope", "user_name"] + assert list(inspect.signature(db.remove_oldest_memory).parameters) == [ + "memory_type", + "keep_latest", + "user_name", + ] + + +@pytest.mark.parametrize( + "stage", + [ + "lock", + "lock_fetch", + "rank", + "rank_fetch", + "deallocate_all", + "prepare", + "cypher", + "cypher_fetch", + "deallocate", + "delete1", + "delete2", + "delete3", + ], +) +def test_capacity_sql_failpoints_rollback_all_blocks(capacity, stage): + import psycopg2 + + db, pool, connection, cursor = capacity + error = psycopg2.ProgrammingError("capacity failpoint") + current = None + deletes = 0 + keys = [f"x{i:04}" for i in range(1200)] + + def execute(statement, params=None): + nonlocal current, deletes + current = statement + point = None + if "FOR UPDATE" in statement: + point = "lock" + elif "NULLS LAST" in statement: + point = "rank" + elif statement == "DEALLOCATE ALL": + point = "deallocate_all" + elif statement.startswith("PREPARE"): + point = "prepare" + elif statement.startswith("EXECUTE memos_age"): + point = "cypher" + elif statement == "DEALLOCATE memos_age": + point = "deallocate" + elif statement.startswith("DELETE FROM ONLY"): + deletes += 1 + point = f"delete{deletes}" + if point == stage: + raise error + + def fetch(): + if "FOR UPDATE" in current: + point, result = "lock_fetch", [(key,) for key in keys] + elif "NULLS LAST" in current: + point, result = "rank_fetch", [(key,) for key in reversed(keys)] + else: + point, result = "cypher_fetch", [] + if point == stage: + raise error + return result + + cursor.execute.side_effect = execute + cursor.fetchall.side_effect = fetch + with pytest.raises(GraphDBError) as caught: + db.remove_oldest_memory("WorkingMemory", 100) + assert caught.value.__cause__ is error + connection.commit.assert_not_called() + assert connection.rollback.call_count == 2 + pool.putconn.assert_called_once_with(connection, close=False) + assert pool.getconn.call_count == 1 + + +@pytest.mark.parametrize("method,args", CALLS) +@pytest.mark.parametrize("cleanup", ["none", "rollback", "putconn", "both"]) +def test_original_operation_error_survives_cleanup_failures(capacity, method, args, cleanup): + import psycopg2 + + db, pool, connection, cursor = capacity + error = psycopg2.errors.QueryCanceled("query canceled") + cursor.fetchall.side_effect = error + cursor.fetchone.side_effect = error + if cleanup in {"rollback", "both"}: + connection.rollback.side_effect = [None, psycopg2.OperationalError("rollback failed")] + if cleanup in {"putconn", "both"}: + pool.putconn.side_effect = psycopg2.InterfaceError("return failed") + with pytest.raises(GraphDBError) as caught: + getattr(db, method)(*args) + assert caught.value.__cause__ is error + connection.commit.assert_not_called() + pool.putconn.assert_called_once_with(connection, close=True) + assert db._closed == (cleanup in {"putconn", "both"}) diff --git a/tests/graph_dbs/test_opengauss_capacity_consumers.py b/tests/graph_dbs/test_opengauss_capacity_consumers.py new file mode 100644 index 000000000..9df4a86a2 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_capacity_consumers.py @@ -0,0 +1,258 @@ +"""Real MemoryManager/Reorganizer calls, with only model services replaced.""" + +import os +import uuid + +from collections import Counter +from datetime import datetime +from unittest.mock import MagicMock + +import pytest + +from memos.exceptions import GraphDBError +from memos.memories.textual.item import TextualMemoryItem, TreeNodeTextualMemoryMetadata +from memos.memories.textual.tree_text_memory.organize.manager import MemoryManager +from tests.graph_dbs import test_opengauss_capacity_integration as support + + +database, stores = support.database, support.stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) +M1, M2, M3 = [f"00000000-0000-4000-8000-{i:012}" for i in range(1, 4)] + + +def item(node_id=M3, memory="新增工作记忆"): + return TextualMemoryItem( + id=node_id, + memory=memory, + metadata=TreeNodeTextualMemoryMetadata( + memory_type="WorkingMemory", status="activated", embedding=[1, 0, 0] + ), + ) + + +def seed_old(db, count=2): + for i, node_id in enumerate([M1, M2, M3][:count], 1): + db.add_node( + node_id, + ["旧工作一", "旧工作二", "旧工作三"][i - 1], + { + "memory_type": "WorkingMemory", + "status": "activated", + "embedding": [1, 0, 0], + "updated_at": f"2000-01-0{i}T00:00:00Z", + }, + ) + + +@pytest.fixture +def manager(stores): + db = stores() + instance = MemoryManager( + db, MagicMock(), MagicMock(), memory_size={"WorkingMemory": 2}, is_reorganize=False + ) + instance.current_memory_size = {"WorkingMemory": 2} + try: + yield instance, db + finally: + instance.close() + + +def stored(db): + return db.get_nodes(support.ids(db), include_embedding=True) + + +def assert_generated(rows, texts, start, end): + assert end >= start > datetime(2000, 1, 3) + assert Counter(row["memory"] for row in rows) == Counter(texts) + assert len({row["id"] for row in rows}) == len(rows) + for row in rows: + assert str(uuid.UUID(row["id"])) == row["id"] + metadata = row["metadata"] + assert metadata["embedding"] == [1, 0, 0] + assert metadata["memory_type"] == "WorkingMemory" + assert metadata["status"] == "activated" + assert start <= datetime.fromisoformat(metadata["updated_at"]) <= end + + +def test_nonbatch_add_really_trims_new_working_memory(manager): + instance, db = manager + seed_old(db) + old = db.get_node(M2, include_embedding=True) + start = datetime.now() + assert instance.add([item()], user_name="A", mode="sync", use_batch=False) == [] + end = datetime.now() + assert support.ids(db) == [M2, M3] + assert db.get_node(M2, include_embedding=True) == old + assert_generated([db.get_node(M3, include_embedding=True)], ["新增工作记忆"], start, end) + assert instance.get_current_memory_size(user_name="A") == {"WorkingMemory": 2} + + +def test_default_batch_writes_and_trims_working_nodes(manager): + instance, db = manager + start = datetime.now() + assert instance.add([item(key) for key in [M1, M2, M3]], user_name="A", use_batch=True) == [] + end = datetime.now() + assert support.ids(db) == [M2, M3] + assert_generated(stored(db), ["新增工作记忆", "新增工作记忆"], start, end) + assert instance.get_current_memory_size(user_name="A") == {"WorkingMemory": 2} + + +def test_replace_preslices_two_inputs_then_trims_old_nodes(manager): + instance, db = manager + seed_old(db) + start = datetime.now() + assert ( + instance.replace_working_memory( + [item(memory=text) for text in ["替换一", "替换二", "替换三"]], user_name="A" + ) + is None + ) + end = datetime.now() + rows = stored(db) + assert not {M1, M2} & {row["id"] for row in rows} + assert_generated(rows, ["替换一", "替换二"], start, end) + assert instance.current_memory_size == {"WorkingMemory": 2} + + +@pytest.mark.parametrize( + "limit,cached,called", + [ + (2, 0, False), + (2, 1, True), + (2, 2, True), + (2, 3, True), + (3, 1, False), + (3, 2, True), + (20, 15, False), + (20, 16, True), + ], +) +def test_cleanup_uses_cached_threshold_and_forwards_limit( + manager, monkeypatch, limit, cached, called +): + instance, db = manager + instance.memory_size = {"WorkingMemory": limit} + instance.current_memory_size = {"WorkingMemory": cached} + spy = MagicMock(wraps=db.remove_oldest_memory) + monkeypatch.setattr(db, "remove_oldest_memory", spy) + instance._cleanup_memories_if_needed(user_name="A") + assert spy.call_count == int(called) + if called: + spy.assert_called_once_with(memory_type="WorkingMemory", keep_latest=limit, user_name="A") + assert stored(db) == [] + + +def test_cleanup_then_refresh_requires_second_call_with_stale_cache(manager): + instance, db = manager + support.seed(db, only_working=True) + instance.current_memory_size = {"WorkingMemory": 0} + instance.remove_and_refresh_memory(user_name="A") + assert support.ids(db) == support.WORKING + assert instance.current_memory_size == {"WorkingMemory": 5} + instance.remove_and_refresh_memory(user_name="A") + assert support.ids(db) == ["w3", "w5"] + assert instance.current_memory_size == {"WorkingMemory": 2} + + +def test_string_capacity_still_fails_in_consumers_list_slice(manager): + instance, db = manager + seed_old(db) + before = stored(db) + instance.memory_size = {"WorkingMemory": "2"} + with pytest.raises(TypeError): + instance.replace_working_memory([item()], user_name="A") + assert stored(db) == before + + +@pytest.mark.parametrize("has_node", [False, True]) +def test_reorganizer_skips_missing_or_archived_only_candidates(manager, monkeypatch, has_node): + instance, db = manager + if has_node: + db.add_node(M1, "archived node", {"memory_type": "LongTermMemory", "status": "archived"}) + reorganizer = instance.reorganizer + reorganizer._is_optimizing = {"LongTermMemory": False, "UserMemory": False} + missing = MagicMock(wraps=db.node_not_exist) + count = MagicMock(wraps=db.get_memory_count) + candidates = MagicMock(wraps=db.get_structure_optimization_candidates) + monkeypatch.setattr(db, "node_not_exist", missing) + monkeypatch.setattr(db, "get_memory_count", count) + monkeypatch.setattr(db, "get_structure_optimization_candidates", candidates) + if has_node: + assert reorganizer.optimize_structure(scope="LongTermMemory", user_name="A") is None + candidates.assert_called_once_with("LongTermMemory", include_embedding=True, user_name="A") + # node_not_exist also delegates to the real count method. + assert count.call_count == 2 + else: + assert reorganizer.optimize_structure(scope="LongTermMemory", user_name="A") is None + candidates.assert_not_called() + assert count.call_count == 1 + missing.assert_called_once_with("LongTermMemory", user_name="A") + assert reorganizer._is_optimizing == {"LongTermMemory": False, "UserMemory": False} + + +@pytest.mark.parametrize("operation", ["add", "replace", "refresh"]) +def test_consumer_cleanup_failure_preserves_separately_committed_writes( + manager, database, operation, caplog +): + instance, db = manager + admin, _ = database + seed_old(db, count=3 if operation == "refresh" else 2) + old = stored(db) + start = datetime.now() + with support.failing_delete(admin, node_id=M1): + if operation == "replace": + with pytest.raises(GraphDBError) as caught: + instance.replace_working_memory( + [item(memory=text) for text in ["替换一", "替换二", "替换三"]], user_name="A" + ) + assert caught.value.__cause__.pgcode == "P0001" + elif operation == "add": + with pytest.raises(GraphDBError) as caught: + instance.add([item()], user_name="A", mode="sync", use_batch=False) + assert caught.value.__cause__.pgcode == "P0001" + else: + assert instance.remove_and_refresh_memory(user_name="A") is None + end = datetime.now() + after = stored(db) + assert [row for row in after if row["id"] in {row["id"] for row in old}] == old + generated = [row for row in after if row["id"] not in {row["id"] for row in old}] + assert_generated( + generated, + {"add": ["新增工作记忆"], "replace": ["替换一", "替换二"], "refresh": []}[operation], + start, + end, + ) + assert len(after) == (4 if operation == "replace" else 3) + assert instance.current_memory_size == {"WorkingMemory": 3 if operation == "refresh" else 2} + if operation == "refresh": + assert "Remove WorkingMemory error" in caplog.text + + +def test_failed_count_refresh_preserves_cache_and_rows(manager, database): + from psycopg2 import sql + + instance, db = manager + admin, config = database + seed_old(db, count=3) + before = stored(db) + support._query( + admin, + sql.SQL("REVOKE SELECT ON {} FROM {}").format( + sql.SQL(support.TABLE), sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as caught: + instance.get_current_memory_size(user_name="A") + assert caught.value.__cause__.pgcode == "42501" + assert instance.current_memory_size == {"WorkingMemory": 2} + finally: + support._query( + admin, + sql.SQL("GRANT SELECT ON {} TO {}").format( + sql.SQL(support.TABLE), sql.Identifier(config.user) + ), + ) + assert stored(db) == before diff --git a/tests/graph_dbs/test_opengauss_capacity_integration.py b/tests/graph_dbs/test_opengauss_capacity_integration.py new file mode 100644 index 000000000..ecc68184a --- /dev/null +++ b/tests/graph_dbs/test_opengauss_capacity_integration.py @@ -0,0 +1,846 @@ +"""Public counts/capacity methods against AGE, DataVec and native BM25. + +Uses only the UUID database and ordinary application role owned by the existing +integration fixtures. Expected business results are fixed independently of SQL. +""" + +import math +import os +import sys +import threading +import time + +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_integration import ( + _query, +) +from tests.graph_dbs.test_opengauss_integration import ( + database as _database, +) +from tests.graph_dbs.test_opengauss_integration import ( + stores as _stores, +) + + +database, stores = _database, _stores + +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + +TABLE = '"memos_graph"."Memory"' +WORKING = ["w1", "w2", "w3", "w4", "w5"] +GROUPS = [ + {"memory_type": "InsightMemory", "count": 1}, + {"memory_type": "LongTermMemory", "count": 3}, + {"memory_type": "UserMemory", "count": 1}, + {"memory_type": "WorkingMemory", "count": 5}, +] + + +def seed(db, scope="A", only_working=False): + rows = [] + times = [ + "2026-01-01T00:00:00", + "2026-01-01T01:00:00", + "2026-01-01T09:00:00+08:00", + None, + "2026-01-01T02:00:00", + ] + for node_id, instant in zip(WORKING, times, strict=True): + metadata = {"memory_type": "WorkingMemory", "embedding": [1, 0, 0]} + if instant is not None: + metadata["updated_at"] = instant + if node_id in {"w1", "w2"}: + metadata["info"] = {"department": "研发部"} + rows.append({"id": node_id, "memory": "星港 回归", "metadata": metadata}) + if not only_working: + for node_id, kind, status in [ + ("l1", "LongTermMemory", "deleted"), + ("l2", "LongTermMemory", "activated"), + ("l3", "LongTermMemory", "archived"), + ("u1", "UserMemory", "activated"), + ("i1", "InsightMemory", "activated"), + ]: + metadata = {"memory_type": kind, "status": status, "updated_at": "2026-01-01T00:00:00Z"} + if node_id == "l2": + metadata["info"] = {"department": "研发部"} + rows.append({"id": node_id, "memory": "计数 样例", "metadata": metadata}) + db.add_nodes_batch(rows, user_name=scope) + if not only_working: + db.add_edge("w1", "l2", "RELATE", user_name=scope) + db.add_edge("w3", "w5", "FOLLOWS", user_name=scope) + + +def snapshot(admin, scope="A"): + return { + "nodes": _query( + admin, + f"SELECT business_id,id::text,memory,properties,embedding::text FROM ONLY {TABLE} WHERE scope=%s ORDER BY business_id", + (scope,), + ), + "edges": _query( + admin, + 'SELECT id::text,start_id::text,end_id::text,properties FROM "memos_graph"._ag_label_edge ORDER BY id', + ), + } + + +def ids(db, scope="A"): + return db.get_by_metadata([], user_name=scope, status=None) + + +def visible(db, expected, scope="A"): + vector = db.search_by_embedding( + [1, 0, 0], scope="WorkingMemory", user_name=scope, top_k=100, threshold=None + ) + text = db.search_by_fulltext( + ["星港"], scope="WorkingMemory", user_name=scope, top_k=100, threshold=None + ) + assert [row["id"] for row in vector] == expected + assert [row["id"] for row in text] == expected + assert all(row["score"] == pytest.approx(1, abs=1e-6) for row in vector) + assert all(math.isfinite(row["score"]) and row["score"] > 0 for row in text) + + +@contextmanager +def failing_delete(admin, scope="A", node_id="w1", action="RAISE EXCEPTION 'capacity injection'"): + from psycopg2 import sql + + _query( + admin, + sql.SQL( + "CREATE FUNCTION memos_graph.capacity_fail() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.scope={} AND OLD.business_id={} THEN " + ).format(sql.Literal(scope), sql.Literal(node_id)) + + sql.SQL(action + "; END IF; RETURN OLD; END $$"), + ) + _query( + admin, + f"CREATE TRIGGER capacity_fail BEFORE DELETE ON {TABLE} FOR EACH ROW EXECUTE PROCEDURE memos_graph.capacity_fail()", + ) + try: + yield + finally: + _query(admin, f"DROP TRIGGER capacity_fail ON {TABLE}") + _query(admin, "DROP FUNCTION memos_graph.capacity_fail()") + + +def test_grouped_counts_full_results_and_scoped_predicates(stores): + db = stores() + seed(db) + db.add_node( + "w1", + "星港 回归", + { + "memory_type": "WorkingMemory", + "embedding": [1, 0, 0], + "updated_at": "2026-01-01T00:00:00", + "info": {"department": "研发部"}, + }, + user_name="B", + ) + db.add_node( + "l1", + "计数 样例", + { + "memory_type": "LongTermMemory", + "status": "activated", + "updated_at": "2026-01-01T00:00:00Z", + }, + user_name="B", + ) + assert db.get_grouped_counts(["memory_type"]) == GROUPS + assert db.get_grouped_counts(["memory_type"], user_name="B") == [ + {"memory_type": "LongTermMemory", "count": 1}, + {"memory_type": "WorkingMemory", "count": 1}, + ] + assert db.get_grouped_counts(["memory_type"], user_name="K") == [] + assert db.get_grouped_counts(["memory_type", "status"]) == [ + {"memory_type": kind, "status": status, "count": count} + for kind, status, count in [ + ("InsightMemory", "activated", 1), + ("LongTermMemory", "activated", 1), + ("LongTermMemory", "archived", 1), + ("LongTermMemory", "deleted", 1), + ("UserMemory", "activated", 1), + ("WorkingMemory", None, 5), + ] + ] + assert db.get_grouped_counts(["memory_type", "info.department"]) == [ + {"memory_type": kind, "info.department": department, "count": count} + for kind, department, count in [ + ("InsightMemory", None, 1), + ("LongTermMemory", "研发部", 1), + ("LongTermMemory", None, 2), + ("UserMemory", None, 1), + ("WorkingMemory", "研发部", 2), + ("WorkingMemory", None, 3), + ] + ] + condition = "ag_catalog.agtype_access_operator(properties, '\"status\"'::ag_catalog.agtype) = %(status)s::ag_catalog.agtype" + expected = [GROUPS[0], {"memory_type": "LongTermMemory", "count": 1}, GROUPS[2]] + for predicate in [ + condition, + "WHERE " + condition, + condition + " OR " + condition, + condition + " OR scope='B'", + ]: + assert ( + db.get_grouped_counts(["memory_type"], predicate, {"status": '"activated"'}) == expected + ) + assert db.get_grouped_counts(["memory_type"], "scope=%(scope)s", {"scope": "B"}) == [] + assert db.get_grouped_counts(["memory_type"], "memory LIKE '%%'", {}) == GROUPS + for kind, expected_count in [("LongTermMemory", 3), ("WorkingMemory", 5), ("SkillMemory", 0)]: + count = db.get_memory_count(kind) + missing = db.node_not_exist(kind) + assert count == expected_count and type(count) is int + assert missing == int(expected_count == 0) and type(missing) is int + assert db.get_memory_count("WorkingMemory", user_name="B") == 1 + assert db.node_not_exist("UserMemory", user_name="B") == 1 + + +def test_grouped_age_values_numeric_equivalence_and_many_groups(stores): + db = stores() + values = [ + {}, + {"v": None}, + {"v": 1}, + {"v": 1.0}, + {"v": "1"}, + {"v": True}, + {"v": [1]}, + {"v": {"a": 1}}, + ] + db.add_nodes_batch( + [ + {"id": f"j{i}", "memory": "", "metadata": {"info": value}} + for i, value in enumerate(values) + ] + ) + result = db.get_grouped_counts(["info.v"]) + assert result == [ + {"info.v": value, "count": count} + for value, count in [("1", 1), (1, 2), ([1], 1), (True, 1), ({"a": 1}, 1), (None, 2)] + ] + assert [type(row["info.v"]) for row in result] == [str, int, list, bool, dict, type(None)] + db.add_nodes_batch( + [ + {"id": f"n{i}", "memory": "", "metadata": {"info": {"v": value}, "status": "p"}} + for i, value in enumerate([1, 1.0, [1], [1.0], {"a": 1}, {"a": 1.0}]) + ], + user_name="N", + ) + nested = db.get_grouped_counts(["info.v", "status"], user_name="N") + assert nested == [ + {"info.v": value, "status": "p", "count": 2} for value in [1, [1.0], {"a": 1.0}] + ] + assert type(nested[0]["info.v"]) is int + assert type(nested[1]["info.v"][0]) is float + assert type(nested[2]["info.v"]["a"]) is float + db.add_nodes_batch( + [ + {"id": f"g{i:03}", "memory": "", "metadata": {"info": {"bucket": f"g{i:03}"}}} + for i in range(105) + ], + user_name="G", + ) + assert db.get_grouped_counts(["info.bucket"], user_name="G") == [ + {"info.bucket": f"g{i:03}", "count": 1} for i in range(105) + ] + + +@pytest.mark.parametrize( + "keep,retained,edges", + [ + (0, [], 0), + (1, ["w5"], 0), + (2, ["w3", "w5"], 1), + (5, WORKING, 2), + (6, WORKING, 2), + ("02", ["w3", "w5"], 1), + ], +) +def test_capacity_retains_exact_nodes_edges_and_public_search( + stores, database, keep, retained, edges +): + db = stores() + admin, _ = database + seed(db) + db.add_node( + "w1", + "星港 回归", + { + "memory_type": "WorkingMemory", + "embedding": [1, 0, 0], + "updated_at": "2026-01-01T00:00:00", + "info": {"department": "研发部"}, + }, + user_name="B", + ) + before = snapshot(admin) + before_b = db.get_node("w1", include_embedding=True, user_name="B") + visible(db, WORKING) + visible(db, ["w1"], "B") + assert db.remove_oldest_memory("WorkingMemory", keep) is None + assert ids(db) == ["i1", "l1", "l2", "l3", "u1", *retained] + after = snapshot(admin) + assert after["nodes"] == [ + row for row in before["nodes"] if row[0] not in set(WORKING) - set(retained) + ] + assert len(after["edges"]) == edges + graphids = {row[1] for row in after["nodes"]} + assert after["edges"] == [ + edge for edge in before["edges"] if edge[1] in graphids and edge[2] in graphids + ] + assert [node["id"] for node in db.get_nodes(WORKING)] == retained + assert [ + node["id"] for node in db.get_all_memory_items("WorkingMemory", status=None) + ] == retained + visible(db, retained) + visible(db, ["w1"], "B") + assert db.get_node("w1", include_embedding=True, user_name="B") == before_b + + +@pytest.mark.parametrize("keep", [0, 2, 6, 7]) +def test_bad_time_is_checked_even_without_deletions(stores, database, keep): + db = stores() + admin, _ = database + seed(db) + db.add_node( + "w6", + "星港 回归", + {"memory_type": "WorkingMemory", "updated_at": "not-a-time", "embedding": [1, 0, 0]}, + ) + before = snapshot(admin) + with pytest.raises(GraphDBError) as caught: + db.remove_oldest_memory("WorkingMemory", keep) + assert caught.value.__cause__.pgcode == "22007" + assert snapshot(admin) == before + visible(db, [*WORKING, "w6"]) + + +def test_oversized_zero_string_cannot_delete_real_memories(stores, database): + db = stores() + admin, _ = database + seed(db) + seed(db, "B") + before_a, before_b = snapshot(admin), snapshot(admin, "B") + previous = sys.get_int_max_str_digits() + try: + sys.set_int_max_str_digits(0) + for keep in ("0" * 65, "0" * 5000): + with pytest.raises(ValueError, match="64"): + db.remove_oldest_memory("WorkingMemory", keep, user_name="A") + finally: + sys.set_int_max_str_digits(previous) + assert snapshot(admin) == before_a + assert snapshot(admin, "B") == before_b + db.remove_oldest_memory("WorkingMemory", "9" * 64, user_name="A") + assert snapshot(admin) == before_a + db.remove_oldest_memory("WorkingMemory", "0" * 64, user_name="A") + assert db.get_memory_count("WorkingMemory", "A") == 0 + assert db.get_memory_count("LongTermMemory", "A") == 3 + assert db.get_memory_count("WorkingMemory", "B") == 5 + + +def test_all_statuses_occupy_capacity_and_equal_time_uses_id(stores): + db = stores() + statuses = ["activated", "resolving", "archived", "deleted", "missing", None] + db.add_nodes_batch( + [ + { + "id": f"s{i}", + "memory": "状态样例", + "metadata": { + "memory_type": "WorkingMemory", + "updated_at": f"2026-01-0{i + 1}T00:00:00Z", + **({} if status == "missing" else {"status": status}), + }, + } + for i, status in enumerate(statuses) + ] + ) + assert db.get_memory_count("WorkingMemory") == 6 + assert db.node_not_exist("WorkingMemory") == 0 + assert db.get_grouped_counts(["status"]) == [ + {"status": value, "count": count} + for value, count in [ + ("activated", 1), + ("archived", 1), + ("deleted", 1), + ("resolving", 1), + (None, 2), + ] + ] + db.remove_oldest_memory("WorkingMemory", 3) + assert ids(db) == ["s3", "s4", "s5"] + assert db.get_memory_count("WorkingMemory") == 3 + assert db.node_not_exist("WorkingMemory") == 0 + assert db.get_grouped_counts(["status"]) == [ + {"status": "deleted", "count": 1}, + {"status": None, "count": 2}, + ] + seed(db, "L") + db.remove_oldest_memory("LongTermMemory", 1, user_name="L") + assert ids(db, "L") == ["i1", "l3", "u1", *WORKING] + + +@pytest.mark.parametrize("value", [None, 3, True, [], {}]) +def test_nonstring_time_is_oldest_and_null_ties_use_id(stores, value): + db = stores() + db.add_nodes_batch( + [ + { + "id": node_id, + "memory": "", + "metadata": {"memory_type": "WorkingMemory", "updated_at": value}, + } + for node_id in ["a", "b"] + ] + ) + db.remove_oldest_memory("WorkingMemory", 1) + assert ids(db) == ["b"] + + +def test_delete_failure_restores_edges_nodes_and_public_search(stores, database): + db = stores() + admin, _ = database + seed(db) + before = snapshot(admin) + visible(db, WORKING) + with failing_delete(admin), pytest.raises(GraphDBError) as caught: + db.remove_oldest_memory("WorkingMemory", 2) + assert caught.value.__cause__.pgcode == "P0001" + assert snapshot(admin) == before + visible(db, WORKING) + + +class ObservedCursor: + """Forward every database operation; hooks observe real rows or inject faults.""" + + def __init__(self, cursor, hook): + self.cursor = cursor + self.hook = hook + self.statement = None + self.params = None + + def __enter__(self): + self.cursor.__enter__() + return self + + def __exit__(self, *args): + return self.cursor.__exit__(*args) + + def execute(self, statement, params=None): + self.statement, self.params = statement, params + result = self.cursor.execute(statement, params) + self.hook("execute", self, None) + return result + + def fetchall(self): + rows = self.cursor.fetchall() + self.hook("fetchall", self, rows) + return rows + + def __getattr__(self, name): + return getattr(self.cursor, name) + + +class ObservedConnection: + def __init__(self, connection, hook): + self.connection = connection + self.hook = hook + + def cursor(self): + return ObservedCursor(self.connection.cursor(), self.hook) + + +def observe(db, monkeypatch, hook): + original = db._get_connection + + @contextmanager + def connection(): + with original() as raw: + yield ObservedConnection(raw, hook) + + monkeypatch.setattr(db, "_get_connection", connection) + + +def raw_setting(db, setting): + connection = db.connection_pool.getconn() + try: + with connection.cursor() as cursor: + cursor.execute("SHOW " + setting) + return cursor.fetchone()[0] + finally: + connection.rollback() + db.connection_pool.putconn(connection) + + +def test_groupaggregate_plan_and_setting_restored_on_success_and_failure( + stores, monkeypatch, record_property +): + db = stores(maxconn=1) + db.add_nodes_batch( + [ + {"id": "int", "memory": "", "metadata": {"info": {"v": 1}}}, + {"id": "float", "memory": "", "metadata": {"info": {"v": 1.0}}}, + ] + ) + assert raw_setting(db, "enable_hashagg") == "on" + plans = [] + + def hook(stage, cursor, rows): + if stage == "fetchall" and "GROUP BY" in cursor.statement: + with cursor.connection.cursor() as check: + check.execute("SHOW enable_hashagg") + assert check.fetchone() == ("off",) + check.execute("EXPLAIN " + cursor.statement, cursor.params) + plans.extend(row[0] for row in check.fetchall()) + + observe(db, monkeypatch, hook) + assert db.get_grouped_counts(["info.v"]) == [{"info.v": 1, "count": 2}] + assert any("GroupAggregate" in line for line in plans) + assert not any("HashAggregate" in line for line in plans) + record_property("group_plan", "\n".join(plans)) + assert raw_setting(db, "enable_hashagg") == "on" + with pytest.raises(GraphDBError) as caught: + db.get_grouped_counts(["info.v"], "nonexistent_column = 1") + assert caught.value.__cause__.pgcode == "42703" + assert raw_setting(db, "enable_hashagg") == "on" + + +def seed_large(db, admin): + db.add_nodes_batch( + [ + { + "id": f"x{i:04}", + "memory": "批量样例", + "metadata": { + "memory_type": "WorkingMemory", + "updated_at": "2026-01-01T00:00:00Z", + "embedding": [1, 0, 0], + }, + } + for i in range(1200) + ] + ) + # Native label already belongs to this test's graph. Bulk edge fixture + # avoids 1199 independent transactions while retaining real graph IDs. + db.add_edge("x0000", "x0000", "RELATE") + _query( + admin, + f"INSERT INTO memos_graph.\"RELATE\"(start_id,end_id,properties) SELECT a.id,b.id,'{{}}'::ag_catalog.agtype FROM ONLY {TABLE} a JOIN ONLY {TABLE} b ON b.scope=a.scope AND substring(b.business_id,2)::int=substring(a.business_id,2)::int+1 WHERE a.scope=%s", + ("A",), + ) + + +@pytest.mark.parametrize("failure", [None, "x0500", "x1000", "timeout"]) +def test_three_delete_blocks_are_atomic_and_timeout_replaces_session( + stores, database, monkeypatch, record_property, failure +): + db = stores(maxconn=1) + admin, _ = database + seed_large(db, admin) + before = snapshot(admin) + assert len(before["nodes"]) == 1200 and len(before["edges"]) == 1200 + assert db.get_memory_count("WorkingMemory") == 1200 + connection = db.connection_pool.getconn() + old_pid = _query(connection, "SELECT pg_backend_pid()")[0][0] + old_session = _query( + admin, "SELECT pid,backend_start FROM pg_stat_activity WHERE pid=%s", (old_pid,) + ) + connection.rollback() + db.connection_pool.putconn(connection) + blocks = [] + plans = [] + + def hook(stage, cursor, rows): + if ( + stage == "execute" + and cursor.statement == "SET LOCAL TIME ZONE 'UTC'" + and failure == "timeout" + ): + cursor.cursor.execute("SET LOCAL statement_timeout = '2000ms'") + if stage == "execute" and cursor.statement.startswith("DELETE FROM ONLY"): + blocks.append(len(cursor.params["ids"])) + if stage == "fetchall" and "NULLS LAST" in cursor.statement: + with cursor.connection.cursor() as check: + check.execute("EXPLAIN " + cursor.statement, cursor.params) + plans.extend(row[0] for row in check.fetchall()) + + observe(db, monkeypatch, hook) + start = time.monotonic() + if failure is None: + db.remove_oldest_memory("WorkingMemory", 100) + assert ids(db) == [f"x{i:04}" for i in range(1100, 1200)] + after = snapshot(admin) + assert after["nodes"] == before["nodes"][1100:] + retained_graphids = {row[1] for row in after["nodes"]} + assert after["edges"] == [ + edge + for edge in before["edges"] + if edge[1] in retained_graphids and edge[2] in retained_graphids + ] + assert len(after["edges"]) == 99 + assert blocks == [500, 500, 100] + assert db.get_memory_count("WorkingMemory") == 100 + else: + action = ( + "PERFORM pg_sleep(10)" + if failure == "timeout" + else "RAISE EXCEPTION 'capacity injection'" + ) + node_id = "x1000" if failure == "timeout" else failure + with ( + failing_delete(admin, node_id=node_id, action=action), + pytest.raises(GraphDBError) as caught, + ): + db.remove_oldest_memory("WorkingMemory", 100) + assert caught.value.__cause__.pgcode == ("57014" if failure == "timeout" else "P0001") + assert blocks == ([500] if failure == "x0500" else [500, 500]) + assert snapshot(admin) == before + assert db.get_memory_count("WorkingMemory") == 1200 + new = db.connection_pool.getconn() + try: + new_pid = _query(new, "SELECT pg_backend_pid()")[0][0] + new_session = _query( + admin, "SELECT pid,backend_start FROM pg_stat_activity WHERE pid=%s", (new_pid,) + ) + # openGauss can immediately reuse the server thread identifier. + # The closed driver object and new backend_start identify replacement. + assert old_session and new_session + assert (new_session != old_session) == (failure == "timeout") + assert (new is not connection) == (failure == "timeout") + assert bool(connection.closed) == (failure == "timeout") + finally: + new.rollback() + db.connection_pool.putconn(new) + record_property("elapsed_seconds", time.monotonic() - start) + record_property("ranking_plan", "\n".join(plans)) + + +def wait_until(predicate, timeout=8): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.01) + pytest.fail("Database synchronization deadline exceeded") + + +@pytest.mark.parametrize("mode", ["insert", "update", "upsert"]) +def test_fixed_locked_set_with_real_concurrent_writers(stores, database, monkeypatch, mode): + db, writer = stores(maxconn=1), stores(maxconn=1) + admin, _ = database + seed(db) + before = snapshot(admin) + locked, release = threading.Event(), threading.Event() + + def hook(stage, cursor, rows): + if stage == "fetchall" and "FOR UPDATE" in cursor.statement: + assert rows == [(key,) for key in WORKING] + locked.set() + assert release.wait(8), "Trim barrier timed out" + + observe(db, monkeypatch, hook) + # Record PID before SQL can block, using the already-owned idle session. + connection = writer.connection_pool.getconn() + pid = _query(connection, "SELECT pg_backend_pid()")[0][0] + connection.rollback() + writer.connection_pool.putconn(connection) + + def write(): + if mode == "update": + writer.update_node("w3", {"memory": "更新后正文"}) + else: + writer.add_node( + "w0" if mode == "insert" else "w1", + "重建后正文", + { + "memory_type": "WorkingMemory", + "updated_at": "2000-01-01T00:00:00Z", + "embedding": [1, 0, 0], + }, + ) + + with ThreadPoolExecutor(max_workers=2) as executor: + trim = executor.submit(db.remove_oldest_memory, "WorkingMemory", 2) + try: + assert locked.wait(8), "Trim did not lock its set" + writing = executor.submit(write) + if mode == "insert": + writing.result(timeout=8) + else: + wait_until( + lambda: _query( + admin, "SELECT 1 FROM pg_locks WHERE pid=%s AND NOT granted", (pid,) + ) + ) + assert not writing.done() + finally: + release.set() + trim.result(timeout=8) + writing.result(timeout=8) + expected = {"insert": ["w0", "w3", "w5"], "update": ["w3", "w5"], "upsert": ["w1", "w3", "w5"]}[ + mode + ] + assert ids(db) == ["i1", "l1", "l2", "l3", "u1", *expected] + after = snapshot(admin) + assert len(after["edges"]) == 1 + graphids = {row[1] for row in after["nodes"]} + assert after["edges"] == [ + edge for edge in before["edges"] if edge[1] in graphids and edge[2] in graphids + ] + old = {row[0]: row for row in before["nodes"]} + new = {row[0]: row for row in after["nodes"]} + if mode == "update": + assert new["w3"][1] == old["w3"][1] and new["w3"][2] == "更新后正文" + elif mode == "upsert": + assert new["w1"][1] != old["w1"][1] and new["w1"][2] == "重建后正文" + else: + # Remove the one-shot observer before the second, independent trim. + monkeypatch.undo() + db.remove_oldest_memory("WorkingMemory", 2) + assert ids(db) == ["i1", "l1", "l2", "l3", "u1", "w3", "w5"] + + +@pytest.mark.parametrize("stage", ["before_commit", "after_commit", "putconn"]) +def test_commit_acknowledgement_and_return_failures_have_distinct_side_effects( + stores, database, monkeypatch, stage +): + import psycopg2 + + db = stores(maxconn=1) + admin, _ = database + seed(db) + before = snapshot(admin) + pool = db.connection_pool + getconn, putconn = pool.getconn, pool.putconn + counts = {"borrow": 0, "commit": 0} + original = psycopg2.OperationalError("synthetic commit acknowledgement loss") + + class CommitConnection: + def __init__(self, raw): + self.raw = raw + + def __getattr__(self, name): + return getattr(self.raw, name) + + def commit(self): + counts["commit"] += 1 + if stage != "before_commit": + self.raw.commit() + if stage != "putconn": + raise original + + def borrow(): + counts["borrow"] += 1 + return CommitConnection(getconn()) + + def give_back(connection, close=False): + if stage == "putconn": + raise original + putconn(connection.raw, close=close) + + with monkeypatch.context() as patch: + patch.setattr(pool, "getconn", borrow) + patch.setattr(pool, "putconn", give_back) + with pytest.raises(GraphDBError) as caught: + db.remove_oldest_memory("WorkingMemory", 2) + assert caught.value.__cause__ is original + assert counts == {"borrow": 1, "commit": 1} + assert db._closed == (stage == "putconn") + after = snapshot(admin) + if stage == "before_commit": + assert after == before + visible(db, WORKING) + else: + assert after["nodes"] == [ + row for row in before["nodes"] if row[0] not in {"w1", "w2", "w4"} + ] + graphids = {row[1] for row in after["nodes"]} + assert after["edges"] == [ + edge for edge in before["edges"] if edge[1] in graphids and edge[2] in graphids + ] + assert len(after["edges"]) == 1 + check = stores(maxconn=1) if stage == "putconn" else db + visible(check, ["w3", "w5"]) + + +def test_real_query_cancel_rolls_back_and_discards_original_session(stores, database, monkeypatch): + db = stores(maxconn=1) + admin, _ = database + seed(db) + before = snapshot(admin) + connection = db.connection_pool.getconn() + pid = _query(connection, "SELECT pg_backend_pid()")[0][0] + connection.rollback() + db.connection_pool.putconn(connection) + old_session = _query( + admin, "SELECT pid,backend_start FROM pg_stat_activity WHERE pid=%s", (pid,) + ) + _query(admin, "SELECT pg_advisory_lock(621503, 9000)") + + def hook(stage, cursor, rows): + if stage == "execute" and cursor.statement == "SET LOCAL TIME ZONE 'UTC'": + cursor.cursor.execute("SET LOCAL statement_timeout = '10000ms'") + + observe(db, monkeypatch, hook) + try: + with ( + failing_delete(admin, action="PERFORM pg_advisory_xact_lock(621503, 9000)"), + ThreadPoolExecutor(max_workers=1) as executor, + ): + operation = executor.submit(db.remove_oldest_memory, "WorkingMemory", 2) + try: + wait_until( + lambda: _query( + admin, + "SELECT 1 FROM pg_locks WHERE pid=%s AND locktype='advisory' AND NOT granted", + (pid,), + ) + ) + connection.cancel() + with pytest.raises(GraphDBError) as caught: + operation.result(timeout=8) + assert caught.value.__cause__.pgcode == "57014" + finally: + _query(admin, "SELECT pg_advisory_unlock(621503, 9000)") + finally: + _query(admin, "SELECT pg_advisory_unlock_all()") + assert snapshot(admin) == before + assert connection.closed + visible(db, WORKING) + replacement = db.connection_pool.getconn() + try: + replacement_pid = _query(replacement, "SELECT pg_backend_pid()")[0][0] + new_session = _query( + admin, "SELECT pid,backend_start FROM pg_stat_activity WHERE pid=%s", (replacement_pid,) + ) + assert old_session and new_session and old_session != new_session + assert replacement is not connection + finally: + replacement.rollback() + db.connection_pool.putconn(replacement) + + +def test_physical_scope_and_unknown_group_fields_are_not_remapped(stores): + db = stores(user_name=None, use_multi_db=True) + db.add_node("p", "", {"memory_type": "WorkingMemory", "info": {"custom": "nested", "count": 2}}) + db.add_node("p", "", {"memory_type": "WorkingMemory"}, user_name="B") + assert db.get_grouped_counts(["custom", "info.custom", "info.count"]) == [ + {"custom": None, "info.custom": "nested", "info.count": 2, "count": 1} + ] + assert db.get_memory_count("WorkingMemory") == 1 + assert db.node_not_exist("WorkingMemory") == 0 + db.remove_oldest_memory("WorkingMemory", 0) + assert db.get_grouped_counts(["memory_type"]) == [] + assert db.node_not_exist("WorkingMemory") == 1 + assert db.get_memory_count("WorkingMemory", user_name="B") == 1 diff --git a/tests/graph_dbs/test_opengauss_connection.py b/tests/graph_dbs/test_opengauss_connection.py new file mode 100644 index 000000000..fd90f743c --- /dev/null +++ b/tests/graph_dbs/test_opengauss_connection.py @@ -0,0 +1,266 @@ +"""Connection ownership, bounded admission and failure preservation.""" + +import builtins +import importlib +import runpy +import threading + +from concurrent.futures import ThreadPoolExecutor +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def pool_case(monkeypatch, psycopg2_stub): + config = OpenGaussGraphDBConfig( + host="db", + user="memos", + password="synthetic-password", + db_name="test", + user_name="A", + minconn=1, + maxconn=2, + ) + raw = MagicMock() + conn = raw.getconn.return_value + conn.closed = 0 + factory = MagicMock(return_value=raw) + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", factory) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + pool = PartialOpenGaussGraphDB(config) + return psycopg2_stub, pool, raw, conn, factory + + +def test_pool_parameters_and_prepared_session(pool_case): + _, pool, raw, conn, factory = pool_case + args, kwargs = factory.call_args + assert args == (1, 2) + assert kwargs["connect_timeout"] == 10 + assert kwargs["dbname"] == "test" + assert kwargs["application_name"] == "memos-opengauss" + with pool._get_connection() as borrowed: + assert borrowed is conn + cur = conn.cursor.return_value.__enter__.return_value + cur.execute.assert_any_call("RESET ALL") + cur.execute.assert_any_call("SELECT 'null'::ag_catalog.agtype") + cur.execute.assert_any_call("SET search_path = ag_catalog, public") + conn.rollback.assert_called_once() + conn.set_session.assert_called_once_with( + isolation_level="READ COMMITTED", readonly=False, autocommit=False + ) + conn.commit.assert_called_once() + raw.putconn.assert_called_once_with(conn, close=False) + + +@pytest.mark.parametrize( + "stage", ["getconn", "rollback", "session", "prepare", "body", "fetch", "commit"] +) +def test_failures_are_not_replayed_and_do_not_leak_slots(pool_case, stage): + driver, pool, raw, conn, _ = pool_case + error = driver.OperationalError("synthetic-sensitive-detail") + cursor = conn.cursor.return_value.__enter__.return_value + if stage == "getconn": + raw.getconn.side_effect = error + elif stage == "rollback": + conn.rollback.side_effect = [error, None] + elif stage == "prepare": + cursor.execute.side_effect = error + elif stage == "session": + conn.set_session.side_effect = error + elif stage == "fetch": + cursor.fetchone.side_effect = error + elif stage == "commit": + conn.commit.side_effect = error + with pytest.raises(GraphDBError) as caught, pool._get_connection(): + if stage == "body": + raise error + if stage == "fetch": + cursor.fetchone() + assert caught.value.__cause__ is error + assert "synthetic-sensitive-detail" not in str(caught.value) + assert raw.getconn.call_count == 1 + assert pool._active == 0 + assert pool._semaphore.acquire(blocking=False) + pool._semaphore.release() + + +@pytest.mark.parametrize("error", [ValueError("bad input"), KeyboardInterrupt(), SystemExit(3)]) +def test_non_driver_exceptions_and_cancellation_preserve_identity(pool_case, error): + _, pool, raw, conn, _ = pool_case + with pytest.raises(type(error)) as caught, pool._get_connection(): + raise error + assert caught.value is error + assert conn.rollback.call_count == 2 + conn.commit.assert_not_called() + raw.putconn.assert_called_once_with(conn, close=False) + + +@pytest.mark.parametrize("put_fails", [False, True]) +@pytest.mark.parametrize("close_fails", [False, True]) +def test_cleanup_failures_keep_original_error(pool_case, put_fails, close_fails): + driver, pool, raw, conn, _ = pool_case + original = ValueError("original") + conn.rollback.side_effect = [None, driver.OperationalError("rollback")] + if put_fails: + raw.putconn.side_effect = driver.pool.PoolError("return") + if close_fails: + conn.close.side_effect = driver.Error("close") + with pytest.raises(ValueError) as caught, pool._get_connection(): + raise original + assert caught.value is original + raw.putconn.assert_called_once_with(conn, close=True) + assert pool._active == 0 + + +def test_return_failure_after_commit_is_observable(pool_case): + driver, pool, raw, conn, _ = pool_case + raw.putconn.side_effect = driver.pool.PoolError("return") + with pytest.raises(GraphDBError, match="return"), pool._get_connection(): + pass + conn.commit.assert_called_once() + conn.close.assert_called_once() + assert pool._active == 0 + + +def test_return_failure_stops_admission_and_drains_existing_owner(pool_case): + driver, pool, raw, conn, _ = pool_case + other = MagicMock() + other.closed = 0 + raw.getconn.side_effect = [conn, other] + raw.putconn.side_effect = [driver.pool.PoolError("return"), None] + with pool._get_connection(), pytest.raises(GraphDBError, match="closed"): + with pytest.raises(GraphDBError, match="return"), pool._get_connection(): + pass + raw.closeall.assert_not_called() + pool._ensure_open() + raw.closeall.assert_called_once() + pool.close() + raw.closeall.assert_called_once() + + +def test_shutdown_failure_during_return_cleanup_preserves_return_error(pool_case): + driver, pool, raw, _, _ = pool_case + raw.putconn.side_effect = driver.pool.PoolError("return") + raw.closeall.side_effect = driver.Error("shutdown") + with pytest.raises(GraphDBError, match="return"), pool._get_connection(): + pass + raw.closeall.assert_called_once() + with pytest.raises(GraphDBError, match="close"): + pool.close() + + +def test_closed_connection_is_discarded(pool_case): + driver, pool, raw, conn, _ = pool_case + conn.closed = 1 + with pytest.raises(GraphDBError, match="closed connection"), pool._get_connection(): + pytest.fail("must not yield a closed connection") + raw.putconn.assert_called_once_with(conn, close=True) + + +def test_wait_timeout_does_not_borrow(pool_case): + driver, pool, raw, _, _ = pool_case + pool._semaphore = MagicMock() + pool._semaphore.acquire.return_value = False + with pytest.raises(GraphDBError, match="wait timed out"), pool._get_connection(): + pytest.fail("must not yield") + raw.getconn.assert_not_called() + + +def test_close_during_admission_releases_slot(pool_case): + driver, pool, raw, _, _ = pool_case + gate = MagicMock() + + def close_before_admission(**kwargs): + pool._closed = True + return True + + gate.acquire.side_effect = close_before_admission + pool._semaphore = gate + with pytest.raises(GraphDBError, match="closed"), pool._get_connection(): + pytest.fail("must not yield") + gate.release.assert_called_once() + raw.getconn.assert_not_called() + + +def test_close_is_idempotent_and_blocks_new_borrows(pool_case): + driver, pool, raw, _, _ = pool_case + pool.close() + pool.close() + raw.closeall.assert_called_once() + with pytest.raises(GraphDBError, match="closed"), pool._get_connection(): + pytest.fail("must not yield") + + +def test_close_waits_for_owned_transaction(pool_case, monkeypatch): + _, pool, raw, _, _ = pool_case + waiting = threading.Event() + original_wait = pool._condition.wait + + def wait(): + waiting.set() + return original_wait() + + monkeypatch.setattr(pool._condition, "wait", wait) + with ThreadPoolExecutor(1) as executor: + with pool._get_connection(): + closing = executor.submit(pool.close) + assert waiting.wait(5) + raw.closeall.assert_not_called() + closing.result(timeout=5) + raw.closeall.assert_called_once() + + +def test_close_failure_can_be_observed_and_retried(pool_case): + driver, pool, raw, _, _ = pool_case + raw.closeall.side_effect = [driver.Error("close"), None] + with pytest.raises(GraphDBError, match="close"): + pool.close() + pool.close() + assert raw.closeall.call_count == 2 + + +def test_constructor_driver_failure_is_sanitized(pool_case, monkeypatch): + driver, pool, _, _, factory = pool_case + error = driver.OperationalError("synthetic-sensitive-detail") + factory.side_effect = error + with pytest.raises(GraphDBError) as caught: + PartialOpenGaussGraphDB(pool.config) + assert caught.value.__cause__ is error + assert "synthetic-sensitive-detail" not in str(caught.value) + + +def test_optional_driver_import_has_install_guidance(monkeypatch): + module = importlib.import_module("memos.graph_dbs.opengauss") + original = builtins.__import__ + original_import_module = importlib.import_module + attempts = [] + + def blocked(name, *args, **kwargs): + if name.startswith("psycopg2"): + attempts.append(name) + raise ImportError("driver deliberately unavailable") + return original(name, *args, **kwargs) + + def blocked_module(name, *args, **kwargs): + if name.startswith("psycopg2"): + raise ImportError("driver deliberately unavailable") + return original_import_module(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", blocked) + namespace = runpy.run_path(module.__file__) + assert not attempts + monkeypatch.setattr(importlib, "import_module", blocked_module) + config = OpenGaussGraphDBConfig( + host="db", + user="memos", + password="synthetic-password", + db_name="test", + user_name="A", + ) + with pytest.raises(ImportError, match="pip install psycopg2-binary"): + namespace["OpenGaussGraphDB"](config) diff --git a/tests/graph_dbs/test_opengauss_consumers_integration.py b/tests/graph_dbs/test_opengauss_consumers_integration.py new file mode 100644 index 000000000..67b2d1992 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_consumers_integration.py @@ -0,0 +1,192 @@ +"""Real factory/consumer ownership, using ordinary-role openGauss connections.""" + +import os +import time + +from concurrent.futures import ThreadPoolExecutor +from threading import Event +from unittest.mock import Mock + +import pytest + +from memos.api.config import APIConfig +from memos.api.handlers import component_init +from memos.api.handlers.config_builders import build_graph_db_config +from memos.api.lifecycle import shutdown_components +from memos.graph_dbs.factory import GraphStoreFactory +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.mem_cube.general import GeneralMemCube +from memos.mem_scheduler.base_scheduler import BaseScheduler +from memos.mem_scheduler.general_modules import init_components_for_scheduler as standalone +from memos.memories.textual.tree import TreeTextMemory +from tests.api.test_component_ownership import prepare_components +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database + + +database = _database +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +@pytest.fixture +def environment(monkeypatch, database): + admin, config = database + for name, value in { + "GRAPH_DB_BACKEND": "opengauss", + "OPENGAUSS_HOST": config.host, + "OPENGAUSS_PORT": str(config.port), + "OPENGAUSS_USER": config.user, + "OPENGAUSS_PASSWORD": config.password, + "OPENGAUSS_DB": config.db_name, + "EMBEDDING_DIMENSION": "3", + "OPENGAUSS_MIN_CONN": "1", + "OPENGAUSS_MAX_CONN": "2", + "MOS_ENABLE_DEFAULT_CUBE_CONFIG": "true", + "ENABLE_INTERNET": "false", + "ENABLE_ACTIVATION_MEMORY": "false", + "ENABLE_PREFERENCE_MEMORY": "false", + }.items(): + monkeypatch.setenv(name, value) + yield + deadline = time.monotonic() + 5 + while True: + count = _query( + admin, + "SELECT count(*) FROM pg_stat_activity WHERE datname=%s AND application_name='memos-opengauss'", + (config.db_name,), + ) + if count == [(0,)] or time.monotonic() >= deadline: + break + time.sleep(0.01) + assert count == [(0,)] + + +def assert_scope_round_trip(graph): + assert type(graph) is OpenGaussGraphDB + graph.add_node("same", "configured", {"memory_type": "LongTermMemory"}) + graph.add_node("same", "explicit", {"memory_type": "LongTermMemory"}, user_name="request-cube") + assert graph.get_node("same")["memory"] == "configured" + assert graph.get_node("same", user_name="request-cube")["memory"] == "explicit" + assert graph.config.user_name == "memos_default" + + +def model_factories(monkeypatch): + import memos.memories.textual.tree as module + + for name in ("LLMFactory", "EmbedderFactory", "RerankerFactory"): + monkeypatch.setattr(getattr(module, name), "from_config", Mock(return_value=Mock())) + monkeypatch.setattr(module, "MemoryManager", Mock()) + + +@pytest.mark.parametrize("entry", ["factory", "tree", "cube", "user_cube"]) +def test_owned_entries_close_real_pool_and_preserve_requested_scope( + environment, monkeypatch, entry +): + model_factories(monkeypatch) + if entry == "factory": + owner = graph = GraphStoreFactory.from_config(build_graph_db_config()) + elif entry == "tree": + owner = TreeTextMemory(APIConfig.get_default_cube_config().text_mem.config) + graph = owner.graph_store + else: + owner = ( + GeneralMemCube(APIConfig.get_default_cube_config()) + if entry == "cube" + else APIConfig.create_user_config("tester", "default")[1] + ) + graph = owner.text_mem.graph_store + try: + assert_scope_round_trip(graph) + connections = list(graph.connection_pool._pool) + finally: + owner.close() + owner.close() + assert all(connection.closed for connection in connections) + assert graph.connection_pool.closed + + +@pytest.mark.parametrize("module", [component_init, standalone]) +@pytest.mark.parametrize("failure", [False, True]) +def test_component_assembly_real_pool_failure_cleanup(environment, monkeypatch, module, failure): + create_graph = GraphStoreFactory.from_config + graph_config = build_graph_db_config() + prepare_components(monkeypatch, module) + opened = [] + + def create(_): + graph = create_graph(graph_config) + opened.append(graph) + return graph + + monkeypatch.setattr(module.GraphStoreFactory, "from_config", create) + original = ValueError("model unavailable") + if failure: + module.MemReaderFactory.from_config.side_effect = original + initialize = module.init_server if module is component_init else module.init_components + if failure: + with pytest.raises(ValueError) as raised: + initialize() + assert raised.value is original + else: + components = initialize() + try: + assert_scope_round_trip(components["graph_db"]) + finally: + shutdown_components(components) + assert len(opened) == 1 and opened[0].connection_pool.closed + + +def test_standalone_scheduler_owns_one_real_graph(environment, monkeypatch): + graph_config = build_graph_db_config() + create_graph = GraphStoreFactory.from_config + prepare_components(monkeypatch, standalone) + monkeypatch.setattr( + standalone.GraphStoreFactory, "from_config", lambda _: create_graph(graph_config) + ) + scheduler = BaseScheduler({"use_redis_queue": False}) + try: + cube, feedback = scheduler.mem_cube, scheduler.feedback_server + assert cube is scheduler.components["naive_mem_cube"] + assert feedback is scheduler.components["feedback_server"] + graph = scheduler.components["graph_db"] + assert_scope_round_trip(graph) + finally: + scheduler.close() + assert graph.connection_pool.closed + + +def test_scheduler_finishes_inflight_database_task_before_closing_pool(environment, monkeypatch): + graph = GraphStoreFactory.from_config(build_graph_db_config()) + components = {"graph_db": graph, "naive_mem_cube": Mock(), "feedback_server": Mock()} + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", lambda: components) + scheduler = BaseScheduler({"use_redis_queue": False}) + assert scheduler.mem_cube is components["naive_mem_cube"] + entered, release = Event(), Event() + connections = [] + + def task(): + with graph._get_connection() as connection: + connections.append(connection) + with connection.cursor() as cursor: + cursor.execute("SELECT 17") + assert cursor.fetchone() == (17,) + entered.set() + assert release.wait(5) + assert not graph._closed and not connection.closed + + running = scheduler.dispatcher.dispatcher_executor.submit(task) + try: + assert entered.wait(5) + with ThreadPoolExecutor(1) as pool: + closing = pool.submit(scheduler.close) + assert scheduler._close_state.closing.wait(5) + assert not closing.done() + release.set() + running.result(5) + closing.result(5) + finally: + release.set() + scheduler.close() + assert graph.connection_pool.closed and all(connection.closed for connection in connections) diff --git a/tests/graph_dbs/test_opengauss_context.py b/tests/graph_dbs/test_opengauss_context.py new file mode 100644 index 000000000..e6fce4fc8 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_context.py @@ -0,0 +1,48 @@ +"""Explicit one-hop context support and semantic capability refusals.""" + +from unittest.mock import MagicMock + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity import capacity as _capacity +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +capacity, pool_case = _capacity, _pool_case + + +def test_context_uses_outgoing_scoped_neighbors(capacity, monkeypatch): + db, _, _, _ = capacity + neighbors = MagicMock(return_value=["a", "b"]) + monkeypatch.setattr(db, "get_neighbors", neighbors) + assert db.get_context_chain("start", "FOLLOWS", user_name="B") == ["a", "b"] + neighbors.assert_called_once_with("start", type="FOLLOWS", direction="out", user_name="B") + + +@pytest.mark.parametrize( + "method,args", + [ + ("deduplicate_nodes", ()), + ("detect_conflicts", ()), + ("merge_nodes", ("a", "b")), + ], +) +def test_semantic_methods_refuse_explicitly_without_borrowing(capacity, method, args): + db, pool, _, _ = capacity + with pytest.raises(NotImplementedError, match=method): + getattr(db, method)(*args) + pool.getconn.assert_not_called() + db.close() + with pytest.raises(GraphDBError, match="closed"): + getattr(db, method)(*args) + + +def test_closed_context_and_invalid_type(capacity): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.get_context_chain("a", "bad-type") + pool.getconn.assert_not_called() + db.close() + with pytest.raises(GraphDBError, match="closed"): + db.get_context_chain("a") diff --git a/tests/graph_dbs/test_opengauss_context_integration.py b/tests/graph_dbs/test_opengauss_context_integration.py new file mode 100644 index 000000000..dd6262bdc --- /dev/null +++ b/tests/graph_dbs/test_opengauss_context_integration.py @@ -0,0 +1,38 @@ +"""Production one-hop context over real AGE relations.""" + +import os + +import pytest + +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +def test_context_is_one_hop_including_branch_self_loop_and_cycle(stores, database): + db = stores() + for key in ["a", "b", "c", "d", "isolated"]: + db.add_node(key, key, {"status": "deleted"}) + db.add_node("b", "B", {}, user_name="B") + for source, target in [("a", "b"), ("b", "c"), ("a", "d"), ("d", "a"), ("a", "a")]: + db.add_edge(source, target, "FOLLOWS") + db.add_edge("a", "c", "PARENT") + admin, _ = database + _query( + admin, + """SELECT * FROM ag_catalog.cypher('memos_graph', $$MATCH (a:Memory {id:'a',user_name:'A'}),(b:Memory {id:'b',user_name:'B'}) CREATE (a)-[:FOLLOWS]->(b)$$) AS (r agtype)""", + ) + assert db.get_context_chain("absent") == [] + assert db.get_context_chain("isolated") == [] + assert db.get_context_chain("a") == ["a", "b", "d"] + assert db.get_context_chain("b") == ["c"] + assert db.get_context_chain("a", "PARENT") == ["c"] + assert db.get_context_chain("a", "ANY") == ["a", "b", "c", "d"] + assert db.get_context_chain("b", user_name="B") == [] + assert db.export_graph(status=[])["total_edges"] == 6 diff --git a/tests/graph_dbs/test_opengauss_dashboard_integration.py b/tests/graph_dbs/test_opengauss_dashboard_integration.py new file mode 100644 index 000000000..84f6b7b03 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_dashboard_integration.py @@ -0,0 +1,193 @@ +"""Real Dashboard selection, pagination and HTTP contract (not service E2E).""" + +import os + +from contextlib import contextmanager +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from fastapi.testclient import TestClient + +from memos.api.handlers.config_builders import build_graph_db_config +from memos.api.handlers.memory_handler import handle_get_memories_dashboard +from memos.api.product_models import GetMemoryDashboardRequest +from memos.graph_dbs.factory import GraphStoreFactory +from memos.memories.textual.tree import TreeTextMemory +from tests.api.test_server_router import mock_init_server +from tests.graph_dbs.test_opengauss_consumers_integration import environment as _environment +from tests.graph_dbs.test_opengauss_integration import _drop_test_graph, _query +from tests.graph_dbs.test_opengauss_integration import database as _database + + +database, environment = _database, _environment +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) +KINDS = { + "text_mem": "LongTermMemory", + "tool_mem": "ToolTrajectoryMemory", + "skill_mem": "SkillMemory", + "pref_mem": "PreferenceMemory", +} +PUBLIC = {"and": [{"object_type": "public"}, {"mem_cube_id": "P"}]} +USER = {"object_type": "user", "mem_cube_id": "P"} + + +@pytest.fixture +def dashboard(environment, database, monkeypatch): + admin, _ = database + if _query(admin, "SELECT 1 FROM ag_catalog.ag_graph WHERE name='memos_graph'"): + _drop_test_graph(admin) + monkeypatch.setenv("MEMOS_USER_NAME", "A") + graph = GraphStoreFactory.from_config(build_graph_db_config(None)) + try: + for owner in ("A", "B", "P"): + for category, kind in KINDS.items(): + metadata = { + "memory_type": kind, + "status": "activated", + "created_at": "2026-01-01T00:00:00Z", + "user_id": "public" if owner == "P" else "u", + "info": {"tag": "y" if owner == "B" else "x"}, + } + graph.add_node(category, owner + ":" + category, metadata, user_name=owner) + graph.add_node( + "deleted", "excluded", {"memory_type": "LongTermMemory", "status": "deleted"} + ) + for owner in ("A", "P"): + graph.add_edge("text_mem", "tool_mem", "PARENT", user_name=owner) + memory = TreeTextMemory.__new__(TreeTextMemory) + memory.graph_store = graph + yield SimpleNamespace(text_mem=memory) + finally: + graph.close() + + +@pytest.mark.parametrize( + "fields,owners,total", + [ + ({}, ["A"], 1), + ({"all_cubes": None, "exclude_cube_id": None}, ["A"], 1), + ({"mem_cube_id": "A"}, ["A"], 1), + ({"all_cubes": False}, ["A"], 1), + ({"all_cubes": True}, ["A", "B", "P"], 3), + ({"all_cubes": True, "exclude_cube_id": "P"}, ["A", "B"], 2), + ({"all_cubes": True, "exclude_cube_id": "X"}, ["A", "B", "P"], 3), + ({"mem_cube_id": "P", "filter": PUBLIC}, ["P"], 1), + ({"filter": USER}, ["A", "B"], 2), + ({"all_cubes": True, "user_id": "u"}, ["A", "B"], 2), + ({"filter": {"tag": "x"}}, ["A"], 1), + ({"page": 2, "page_size": 2}, [], 1), + ({"all_cubes": True, "page": 2, "page_size": 2, "exclude_cube_id": "P"}, [], 2), + ({"mem_cube_id": "missing"}, [], 0), + ], +) +def test_dashboard_real_categories_scopes_pages_and_statistics(dashboard, fields, owners, total): + response = handle_get_memories_dashboard(GetMemoryDashboardRequest(**fields), dashboard) + for category in KINDS: + buckets = response.data[category] + assert [bucket["cube_id"] for bucket in buckets] == owners + for bucket in buckets: + assert bucket["total_nodes"] == 1 + assert [(node["id"], node["memory"]) for node in bucket["memories"]] == [ + (category, bucket["cube_id"] + ":" + category) + ] + assert set(response.data["statistics"].values()) == {total} + + +def test_exclusion_applies_to_induced_edges_and_preserves_data(dashboard): + graph = dashboard.text_mem.graph_store + result = graph.export_graph(all_cubes=True, exclude_user_name="P") + assert result["total_nodes"] == 8 + assert result["edges"] == [ + {"source": "text_mem", "target": "tool_mem", "type": "PARENT", "user_name": "A"} + ] + assert graph.export_graph(all_cubes=True)["total_nodes"] == 12 + + +@pytest.mark.parametrize( + "fields", + [ + {"exclude_cube_id": "P"}, + {"mem_cube_id": "A", "filter": PUBLIC}, + {"mem_cube_id": "A", "filter": USER}, + {"filter": {"or": [PUBLIC, {"tag": "x"}]}}, + {"filter": {"or": [{}]}}, + {"all_cubes": True, "mem_cube_id": "A"}, + {"page": 1}, + {"page": 0, "page_size": 1}, + {"filter": {"tag": {"unknown": "x"}}}, + ], +) +def test_invalid_input_does_not_borrow_real_connection(dashboard, monkeypatch, fields): + pool = dashboard.text_mem.graph_store.connection_pool + borrow = Mock(wraps=pool.getconn) + monkeypatch.setattr(pool, "getconn", borrow) + with pytest.raises(ValueError): + handle_get_memories_dashboard(GetMemoryDashboardRequest(**fields), dashboard) + borrow.assert_not_called() + + +@pytest.fixture +def client(dashboard, monkeypatch, tmp_path): + monkeypatch.setenv("FILE_LOCAL_PATH", str(tmp_path)) + with contextmanager(mock_init_server.__wrapped__)() as app: + import memos.api.routers.server_router as router + + monkeypatch.setattr(router, "naive_mem_cube", dashboard) + yield TestClient(app, raise_server_exceptions=False) + + +@pytest.mark.parametrize( + "payload,status", + [ + ({}, 200), + ({"all_cubes": None}, 200), + ({"all_cubes": False}, 200), + ({"all_cubes": True, "exclude_cube_id": "P"}, 200), + ({"mem_cube_id": "missing"}, 200), + ({"all_cubes": True, "page": 2, "page_size": 2, "exclude_cube_id": "P"}, 200), + ({"all_cubes": True, "mem_cube_id": "A"}, 400), + ({"filter": {"and": []}}, 400), + ({"all_cubes": "false"}, 422), + ({"exclude_cube_id": " "}, 422), + ], +) +def test_http_optional_contract_errors_and_no_virtual_buckets(client, payload, status): + response = client.post("/product/get_memory_dashboard", json=payload) + assert response.status_code == status + body = response.json() + assert body["code"] == status + if status != 200: + assert body["data"] is None + elif payload.get("mem_cube_id") == "missing" or payload.get("page") == 2: + assert all(body["data"][category] == [] for category in KINDS) + assert set(body["data"]["statistics"].values()) == ( + {2} if payload.get("page") == 2 else {0} + ) + + +def test_real_storage_failure_is_http_500_and_connection_recovers(client, dashboard, database): + from psycopg2 import sql + + admin, config = database + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + response = client.post("/product/get_memory_dashboard", json={"all_cubes": True}) + assert response.status_code == 500 + assert response.json()["data"] is None + finally: + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert dashboard.text_mem.graph_store.get_node("text_mem")["memory"] == "A:text_mem" diff --git a/tests/graph_dbs/test_opengauss_dependencies.py b/tests/graph_dbs/test_opengauss_dependencies.py new file mode 100644 index 000000000..dcf956129 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_dependencies.py @@ -0,0 +1,93 @@ +"""Optional driver boundaries must hold even when psycopg2 is not installed.""" + +import os +import subprocess +import sys +import textwrap + +from pathlib import Path + +import pytest + + +def run_without_driver(body): + prefix = """ +import importlib.abc +import sys +class MissingDriver(importlib.abc.MetaPathFinder): + attempts = [] + def find_spec(self, fullname, path=None, target=None): + if fullname == "psycopg2" or fullname.startswith("psycopg2."): + self.attempts.append(fullname) + raise ModuleNotFoundError("No module named " + repr(fullname), name=fullname) +missing = MissingDriver() +sys.meta_path.insert(0, missing) +""" + environment = dict(os.environ) + environment.pop("MEMOS_OPENGAUSS_TEST_CONFIG", None) + return subprocess.run( + [sys.executable, "-c", prefix + textwrap.dedent(body)], + cwd=Path(__file__).resolve().parents[2], + env=environment, + capture_output=True, + text=True, + timeout=30, + ) + + +@pytest.mark.parametrize("initialize", [False, True]) +def test_driver_is_loaded_only_when_backend_is_initialized(initialize): + result = run_without_driver( + f""" + from memos.graph_dbs.factory import GraphStoreFactory + from memos.graph_dbs.opengauss import OpenGaussGraphDB + assert not missing.attempts, missing.attempts + assert GraphStoreFactory.backend_to_class["opengauss"] is OpenGaussGraphDB + if {initialize!r}: + try: + OpenGaussGraphDB(None) + except ImportError as error: + assert "pip install psycopg2-binary" in str(error) + assert "https://pypi.org/project/psycopg2-binary/" in str(error) + else: + raise AssertionError("Missing driver was not rejected") + """ + ) + assert result.returncode == 0, result.stdout + result.stderr + + +def test_integration_modules_collect_without_optional_driver(): + result = run_without_driver( + """ + from pathlib import Path + import pytest + paths = sorted(str(path) for path in Path("tests/graph_dbs").glob("test_opengauss*.py")) + status = pytest.main(["--collect-only", "-q", "-rs", *paths]) + assert not missing.attempts, missing.attempts + raise SystemExit(status) + """ + ) + assert result.returncode == 0, result.stdout + result.stderr + assert "tests collected" in result.stdout + assert "could not import" not in result.stdout + result.stderr + + +def test_enabled_integration_requires_real_driver(): + result = run_without_driver( + """ + import json + import os + from tests.graph_dbs.test_opengauss_integration import database + os.environ["MEMOS_OPENGAUSS_TEST_CONFIG"] = json.dumps({ + "host": "db", "user": "fixture", "password": "synthetic-password", + "db_name": "test", "user_name": "A", "embedding_dimension": 3, + }) + try: + next(database.__wrapped__()) + except ImportError as error: + assert "pip install psycopg2-binary" in str(error) + else: + raise AssertionError("Enabled integration must fail when its driver is missing") + """ + ) + assert result.returncode == 0, result.stdout + result.stderr diff --git a/tests/graph_dbs/test_opengauss_export.py b/tests/graph_dbs/test_opengauss_export.py new file mode 100644 index 000000000..10f7619a8 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_export.py @@ -0,0 +1,171 @@ +"""Export validation, projections, snapshot selection and failure handling.""" + +import json + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity import capacity as _capacity +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +capacity, pool_case = _capacity, _pool_case + + +@pytest.mark.parametrize( + "kwargs", + [ + {"include_embedding": 1}, + {"all_cubes": None}, + {"all_cubes": True, "user_name": "A"}, + {"user_name": ""}, + {"user_name": "A\0"}, + {"user_id": ""}, + {"user_id": 1}, + {"user_id": "u\0"}, + {"unknown": True}, + {"page": 1}, + {"page_size": 2}, + {"page": 0, "page_size": 2}, + {"page": True, "page_size": 2}, + {"page": 1, "page_size": 0}, + {"page": 1, "page_size": True}, + {"memory_type": "LongTermMemory"}, + {"memory_type": [1]}, + {"memory_type": ["x\0"]}, + {"status": [""]}, + {"status": ["x\0"]}, + {"status": 1}, + {"exclude_user_name": "P"}, + {"all_cubes": True, "exclude_user_name": ""}, + {"all_cubes": True, "exclude_user_name": " "}, + {"all_cubes": True, "exclude_user_name": 1}, + {"all_cubes": True, "exclude_user_name": "P\0"}, + ], +) +def test_invalid_export_never_borrows(capacity, kwargs): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.export_graph(**kwargs) + pool.getconn.assert_not_called() + + +def test_closed_export_precedes_validation(capacity): + db, pool, _, _ = capacity + db.close() + with pytest.raises(GraphDBError, match="closed"): + db.export_graph(page=-1) + pool.getconn.assert_not_called() + + +def test_excluded_cube_is_bound_in_count_and_page_predicates(capacity): + db, _, _, cursor = capacity + cursor.fetchone.return_value = (0,) + cursor.fetchall.return_value = [] + excluded = "P' OR true --" + assert db.export_graph(all_cubes=True, exclude_user_name=excluded)["nodes"] == [] + queries = [ + call + for call in cursor.execute.call_args_list + if call.args[0].startswith(("SELECT COUNT(*)", "SELECT business_id,")) + ] + assert len(queries) == 2 + for query in queries: + assert excluded not in query.args[0] + assert "scope <> %(memos_excluded_owner)s" in query.args[0] + assert query.args[1]["memos_excluded_owner"] == excluded + + +@pytest.mark.parametrize("embedding,all_cubes", [(False, False), (True, True)]) +def test_snapshot_projection_and_bound_input(capacity, embedding, all_cubes): + db, pool, connection, cursor = capacity + cursor.fetchone.return_value = (1,) + cursor.fetchall.side_effect = [ + [ + ( + "same", + "text", + json.dumps({"id": "same", "memory": "text", "user_name": "A"}), + "[1,0,0]", + "A", + "123", + ) + ], + [("same", "same", "PARENT", "A")], + ] + result = db.export_graph( + include_embedding=embedding, + all_cubes=all_cubes, + user_id=" u' ", + status=[], + memory_type=["DreamDiary"], + page=1, + page_size=2, + ) + assert result == { + "nodes": [ + { + "id": "same", + "memory": "text", + "metadata": {"user_name": "A", **({"embedding": [1, 0, 0]} if embedding else {})}, + } + ], + "edges": [ + { + "source": "same", + "target": "same", + "type": "PARENT", + **({"user_name": "A"} if all_cubes else {}), + } + ], + "total_nodes": 1, + "total_edges": 1, + } + connection.set_session.assert_called_once_with( + isolation_level="REPEATABLE READ", readonly=True, autocommit=False + ) + assert all(" u' " not in call.args[0] for call in cursor.execute.call_args_list) + pool.getconn.assert_called_once() + connection.commit.assert_called_once() + + +@pytest.mark.parametrize("stage", ["execute", "fetch", "decode", "edge", "commit", "putconn"]) +def test_export_errors_preserve_transaction_semantics(capacity, stage): + import psycopg2 + + db, pool, connection, cursor = capacity + cursor.fetchone.return_value = (1,) + row = ("a", "text", '{"id":"a","memory":"text","user_name":"A"}', None, "A", "123") + cursor.fetchall.side_effect = [[row], [("a", "a", "PARENT", "A")]] + error = psycopg2.ProgrammingError("export injected failure") + if stage == "execute": + cursor.execute.side_effect = error + elif stage == "fetch": + cursor.fetchall.side_effect = error + elif stage == "decode": + cursor.fetchall.side_effect = [[(*row[:2], "{", *row[3:])]] + elif stage == "edge": + cursor.fetchall.side_effect = [[row], [("a", "a", None, "A")]] + elif stage == "commit": + connection.commit.side_effect = error + else: + pool.putconn.side_effect = error + with pytest.raises(GraphDBError) as caught: + db.export_graph() + if stage not in {"decode", "edge"}: + assert caught.value.__cause__ is error + assert db._active == 0 and pool.getconn.call_count == 1 + if stage not in {"commit", "putconn"}: + connection.commit.assert_not_called() + + +def test_snapshot_cursor_returns_to_default_session(capacity): + db, _, connection, cursor = capacity + with db._cursor(snapshot=True): + pass + with db._cursor(): + pass + assert [c.kwargs for c in connection.set_session.call_args_list] == [ + {"isolation_level": "REPEATABLE READ", "readonly": True, "autocommit": False}, + {"isolation_level": "READ COMMITTED", "readonly": False, "autocommit": False}, + ] diff --git a/tests/graph_dbs/test_opengauss_export_integration.py b/tests/graph_dbs/test_opengauss_export_integration.py new file mode 100644 index 000000000..9a8649948 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_export_integration.py @@ -0,0 +1,178 @@ +"""Real consistent graph export and global Cube pagination.""" + +import os + +from contextlib import contextmanager +from copy import deepcopy + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity_integration import ObservedConnection +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +def seed(db): + for owner, key, state, instant in [ + ("A", "same", "activated", "2026-01-02T08:00:00+08:00"), + ("B", "same", "activated", "2026-01-02T00:00:00Z"), + ("A", "old", "deleted", "2026-01-01T00:00:00Z"), + ("A", "null", None, None), + ("A", "missing", "missing", 10), + ]: + meta = { + "status": state, + "memory_type": "LongTermMemory", + "created_at": instant, + "user_id": "u", + "embedding": [1, 0, 0], + } + if state == "missing": + del meta["status"] + db.add_node(key, owner + " " + key, meta, user_name=owner) + db.add_edge("same", "old", "PARENT") + db.add_edge("same", "same", "FOLLOWS", user_name="B") + + +def identities(result): + return [(n["metadata"]["user_name"], n["id"]) for n in result["nodes"]] + + +def test_global_pages_counts_scope_and_induced_edges(stores): + db = stores() + seed(db) + assert identities(db.export_graph()) == [("A", "same"), ("A", "null"), ("A", "missing")] + assert identities(db.export_graph(user_name="B")) == [("B", "same")] + result = db.export_graph(all_cubes=True, status=[], include_embedding=True) + assert identities(result) == [ + ("A", "same"), + ("B", "same"), + ("A", "old"), + ("A", "null"), + ("A", "missing"), + ] + assert result["total_nodes"] == 5 and result["total_edges"] == 2 + assert result["edges"] == [ + {"source": "same", "target": "old", "type": "PARENT", "user_name": "A"}, + {"source": "same", "target": "same", "type": "FOLLOWS", "user_name": "B"}, + ] + assert all(n["metadata"]["embedding"] == [1, 0, 0] for n in result["nodes"]) + pages = [db.export_graph(all_cubes=True, status=[], page=p, page_size=2) for p in range(1, 5)] + assert [identities(p) for p in pages] == [ + [("A", "same"), ("B", "same")], + [("A", "old"), ("A", "null")], + [("A", "missing")], + [], + ] + assert [p["total_nodes"] for p in pages] == [5] * 4 + assert [p["total_edges"] for p in pages] == [1, 0, 0, 0] + assert all("embedding" not in n["metadata"] for p in pages for n in p["nodes"]) + + +@pytest.mark.parametrize( + "status,expected", + [ + (None, ["same", "null", "missing"]), + ([], ["same", "old", "null", "missing"]), + (["activated"], ["same", "missing"]), + (["deleted", "activated"], ["same", "old", "missing"]), + ], +) +def test_filter_status_user_type_and_nested_or(stores, status, expected): + db = stores() + seed(db) + condition = {"or": [{"user_name": "B"}, {"user_id": "u"}]} + before = deepcopy(condition) + result = db.export_graph( + filter=condition, status=status, user_id="u", memory_type=["LongTermMemory", "DreamDiary"] + ) + assert identities(result) == [("A", key) for key in expected] + assert result["total_nodes"] == len(expected) and condition == before + assert db.export_graph(user_id="absent")["total_nodes"] == 0 + assert db.export_graph(memory_type=["InsightMemory"])["nodes"] == [] + + +def test_full_single_cube_edges_and_null_vector(stores, database): + db = stores() + seed(db) + db.add_node("z", "no vector", {"created_at": "2026-01-03"}) + admin, _ = database + _query( + admin, + """SELECT * FROM ag_catalog.cypher('memos_graph', $$MATCH (a:Memory {id:'same',user_name:'A'}),(b:Memory {id:'same',user_name:'B'}) CREATE (a)-[:DIRTY]->(b)$$) AS (r agtype)""", + ) + result = db.export_graph(include_embedding=True, status=[]) + assert result["nodes"][0] == { + "id": "z", + "memory": "no vector", + "metadata": {"user_name": "A", "created_at": "2026-01-03", "embedding": None}, + } + assert result["edges"] == [{"source": "same", "target": "old", "type": "PARENT"}] + assert db.export_graph(all_cubes=True, status=[])["total_edges"] == 2 + + +def test_one_snapshot_survives_writes_between_count_nodes_and_edges(stores, monkeypatch): + db, writer = stores(maxconn=1), stores() + seed(db) + original = db._get_connection + seen = [] + + def hook(stage, cursor, rows): + if stage == "execute" and cursor.statement.startswith("SELECT COUNT(*)"): + seen.append("count") + writer.add_node("late", "after count", {"created_at": "2027-01-01"}) + elif stage == "fetchall" and cursor.statement.startswith("SELECT business_id,"): + seen.append("nodes") + writer.delete_edge("same", "old", "PARENT") + writer.add_node("same", "changed", {"status": "deleted"}) + + @contextmanager + def observed(**kwargs): + with original(**kwargs) as raw: + yield ObservedConnection(raw, hook) + + with monkeypatch.context() as patch: + patch.setattr(db, "_get_connection", observed) + result = db.export_graph(status=[]) + assert seen == ["count", "nodes"] + assert result["total_nodes"] == 4 and result["total_edges"] == 1 + assert result["nodes"][0]["memory"] == "A same" + assert db.export_graph(status=[])["total_nodes"] == 5 + assert db.export_graph(status=[])["total_edges"] == 0 + db.add_node("after", "write works", {}) + + +def test_bad_time_decode_and_readonly_failure_restore_session(stores, database): + db = stores(maxconn=1) + db.add_node("bad", "fixture", {"created_at": "not a time"}) + with pytest.raises(GraphDBError): + db.export_graph(page=99, page_size=1) + db.add_node("bad", "fixed", {}) + with pytest.raises(GraphDBError), db._cursor(snapshot=True) as cursor: + cursor.execute('DELETE FROM memos_graph."Memory"') + db.add_node("after", "works", {}) + assert db.export_graph()["total_nodes"] == 2 + admin, _ = database + _query( + admin, + """UPDATE memos_graph."Memory" SET properties='{}'::agtype WHERE business_id='bad' """, + ) + with pytest.raises(GraphDBError, match="Inconsistent"): + db.export_graph() + + +def test_physical_empty_scope_special_values_and_no_input_changes(stores): + db = stores(user_name=None, use_multi_db=True) + meta = {"user_id": " u' ", "memory_type": "DreamDiary", "status": "activated"} + db.add_node(" ID' ", "exact", meta) + assert db.export_graph(user_id=" u' ", memory_type=[])["nodes"] == [ + {"id": " ID' ", "memory": "exact", "metadata": {**meta, "user_name": ""}} + ] diff --git a/tests/graph_dbs/test_opengauss_files_integration.py b/tests/graph_dbs/test_opengauss_files_integration.py new file mode 100644 index 000000000..3d1eb63ef --- /dev/null +++ b/tests/graph_dbs/test_opengauss_files_integration.py @@ -0,0 +1,234 @@ +"""Actual Tree/Cube restoration and internal record HTTP contracts.""" + +import json +import os + +from contextlib import contextmanager +from copy import deepcopy +from types import SimpleNamespace + +import pytest + +from fastapi.testclient import TestClient + +from memos.api.config import APIConfig +from memos.exceptions import GraphDBError +from memos.mem_cube.general import GeneralMemCube +from memos.memories.textual.tree import TreeTextMemory +from tests.api.test_server_router import mock_init_server +from tests.graph_dbs.test_opengauss_consumers_integration import environment as _environment +from tests.graph_dbs.test_opengauss_consumers_integration import model_factories +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores, environment = _database, _stores, _environment +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +def seed(db): + for owner in ("A", "B"): + for key, state, vector in [ + ("active", "activated", [1, 0, 0]), + ("archived", "archived", None), + ("deleted", "deleted", [0, 1, 0]), + ]: + db.add_node( + key, + owner + ":" + key, + { + "status": state, + "embedding": vector, + "memory_type": "LongTermMemory", + "created_at": { + "active": "2026-01-03", + "archived": "2026-01-02", + "deleted": "2026-01-01", + }[key], + "user_id": "u", + "tags": ["中文"], + "info": {"nested": {"x": 1}}, + }, + user_name=owner, + ) + db.add_edge("active", "archived", "PARENT", user_name=owner) + db.add_edge("archived", "deleted", "FOLLOWS", user_name=owner) + + +def tree(db): + memory = TreeTextMemory.__new__(TreeTextMemory) + memory.config = SimpleNamespace(memory_filename="memory.json") + memory.graph_store = db + return memory + + +@pytest.mark.parametrize("include", [False, True]) +def test_tree_file_empty_and_merge_restore(stores, tmp_path, include): + db = stores() + seed(db) + memory = tree(db) + memory.dump(str(tmp_path), include_embedding=include, user_name="A") + saved = json.loads((tmp_path / "memory.json").read_text()) + assert [n["id"] for n in saved["nodes"]] == ["active", "archived", "deleted"] + assert [n["metadata"]["status"] for n in saved["nodes"]] == ["activated", "archived", "deleted"] + assert all(n["metadata"]["user_name"] == "A" for n in saved["nodes"]) + assert saved["edges"] == [ + {"source": "active", "target": "archived", "type": "PARENT"}, + {"source": "archived", "target": "deleted", "type": "FOLLOWS"}, + ] + if include: + assert [n["metadata"]["embedding"] for n in saved["nodes"]] == [[1, 0, 0], None, [0, 1, 0]] + else: + assert all("embedding" not in n["metadata"] for n in saved["nodes"]) + other = db.export_graph(user_name="B", status=[], include_embedding=True) + db.clear(user_name="A") + memory.load(str(tmp_path), user_name="A") + restored = db.export_graph(user_name="A", status=[], include_embedding=include) + assert restored == saved + db.add_node("extra", "outside backup", {}, user_name="A") + db.add_edge("extra", "active", "PARENT", user_name="A") + db.update_node("active", {"embedding": [0, 0, 1]}, user_name="A") + memory.load(str(tmp_path), user_name="A") + assert db.get_node("extra")["memory"] == "outside backup" + assert len(db.export_graph(status=[])["edges"]) == 3 + assert db.get_node("active", include_embedding=True)["metadata"]["embedding"] == ( + [1, 0, 0] if include else None + ) + assert db.export_graph(user_name="B", status=[], include_embedding=True) == other + + +def test_cube_dump_explicit_full_backup(environment, monkeypatch, tmp_path): + monkeypatch.setenv("MEMOS_USER_NAME", "A") + model_factories(monkeypatch) + config = APIConfig.get_default_cube_config() + config.text_mem.config.graph_db.config.user_name = "A" + cube = GeneralMemCube(config) + try: + cube.text_mem.graph_store.clear(user_name="A") + cube.text_mem.graph_store.clear(user_name="B") + seed(cube.text_mem.graph_store) + before = cube.text_mem.graph_store.export_graph(status=[], include_embedding=True) + cube.dump(str(tmp_path)) + saved = json.loads((tmp_path / cube.text_mem.config.memory_filename).read_text()) + assert saved == before and saved["total_nodes"] == 3 + cube.text_mem.graph_store.clear() + cube.load(str(tmp_path)) + assert cube.text_mem.graph_store.export_graph(status=[], include_embedding=True) == before + finally: + cube.close() + + +@pytest.mark.parametrize("failure", ["cross_cube", "dangling", "sql"]) +def test_invalid_restore_and_database_fault_are_atomic(stores, database, tmp_path, failure): + from psycopg2 import sql + + db = stores() + seed(db) + before = db.export_graph(all_cubes=True, status=[], include_embedding=True) + data = deepcopy(db.export_graph(status=[], include_embedding=True)) + data["nodes"][0]["memory"] = "must not partially apply" + if failure == "cross_cube": + data["nodes"][-1]["metadata"]["user_name"] = "B" + if failure == "dangling": + data["edges"].append({"source": "active", "target": "absent", "type": "PARENT"}) + (tmp_path / "memory.json").write_text(json.dumps(data)) + admin, config = database + if failure == "sql": + _query( + admin, + sql.SQL('REVOKE UPDATE ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(ValueError if failure == "cross_cube" else GraphDBError): + tree(db).load(str(tmp_path), user_name="A") + finally: + if failure == "sql": + _query( + admin, + sql.SQL('GRANT UPDATE ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.export_graph(all_cubes=True, status=[], include_embedding=True) == before + + +@pytest.fixture +def internal(stores, monkeypatch, tmp_path): + db = stores() + seed(db) + db.add_node("unique", "one owner", {}, user_name="B") + monkeypatch.setenv("FILE_LOCAL_PATH", str(tmp_path)) + with contextmanager(mock_init_server.__wrapped__)() as app: + import memos.api.routers.server_router as router + + monkeypatch.setattr(router, "graph_db", db) + yield TestClient(app, raise_server_exceptions=False), db + + +def test_internal_ownership_and_record_side_effects(internal): + client, db = internal + + def post(route, body, status=200): + result = client.post("/product/" + route, json=body) + assert result.status_code == status, result.text + return result.json()["data"] + + assert post("get_user_names_by_memory_ids", {"memory_ids": ["unique", "absent"]}) == { + "unique": "B", + "absent": None, + } + post("get_user_names_by_memory_ids", {"memory_ids": ["active"]}, 500) + assert post("exist_mem_cube_id", {"mem_cube_id": "A"}) == {"A": True} + assert post("exist_mem_cube_id", {"mem_cube_id": "absent"}) == {"absent": False} + before_b = db.export_graph(user_name="B", status=[], include_embedding=True) + post("delete_memory_by_record_id", {"mem_cube_id": "A", "record_id": "batch1"}) + assert db.get_node("active")["metadata"]["status"] == "deleted" + assert db.get_node("archived")["metadata"]["status"] == "deleted" + assert db.get_node("deleted")["metadata"].get("delete_record_id") is None + post("recover_memory_by_record_id", {"mem_cube_id": "A", "delete_record_id": "absent"}) + assert db.get_node("active")["metadata"]["status"] == "deleted" + post("recover_memory_by_record_id", {"mem_cube_id": "A", "delete_record_id": "batch1"}) + assert db.get_node("active")["metadata"]["status"] == "activated" + assert db.get_node("archived")["metadata"]["status"] == "activated" + assert db.get_node("deleted")["metadata"]["status"] == "deleted" + post("delete_memory_by_record_id", {"mem_cube_id": "missing", "record_id": "zero"}) + assert db.export_graph(user_name="B", status=[], include_embedding=True) == before_b + + +@pytest.mark.parametrize( + "route,payload", + [ + ("get_user_names_by_memory_ids", {"memory_ids": ["unique"]}), + ("exist_mem_cube_id", {"mem_cube_id": "A"}), + ("delete_memory_by_record_id", {"mem_cube_id": "A", "record_id": "batch"}), + ("recover_memory_by_record_id", {"mem_cube_id": "A", "delete_record_id": "batch"}), + ], +) +def test_internal_storage_failure_is_not_success(internal, database, route, payload): + from psycopg2 import sql + + client, db = internal + admin, config = database + before = db.export_graph(all_cubes=True, status=[], include_embedding=True) + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + result = client.post("/product/" + route, json=payload) + assert result.status_code == 500 and result.json()["data"] is None + finally: + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.export_graph(all_cubes=True, status=[], include_embedding=True) == before diff --git a/tests/graph_dbs/test_opengauss_filters.py b/tests/graph_dbs/test_opengauss_filters.py new file mode 100644 index 000000000..c777611df --- /dev/null +++ b/tests/graph_dbs/test_opengauss_filters.py @@ -0,0 +1,230 @@ +"""Fail-closed filtering and deletion inputs; business results use real SQL in IT.""" + +import json + +from copy import deepcopy +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def store(monkeypatch, psycopg2_stub): + pool = MagicMock() + cursor = pool.transaction.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", pool.transaction) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_ensure_open", pool.ensure_open) + monkeypatch.setattr(PartialOpenGaussGraphDB, "close", pool.close) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + db = PartialOpenGaussGraphDB( + OpenGaussGraphDBConfig( + host="localhost", + user="memos", + password="synthetic", + db_name="test", + user_name="A", + embedding_dimension=3, + ) + ) + return db, pool, cursor + + +BAD_FILTERS = [ + [], + "bad", + False, + {"and": []}, + {"or": []}, + {"and": {}}, + {"or": "bad"}, + {"and": [None]}, + {"or": [{}]}, + {"and": [{"or": []}]}, + {"status": {}}, + {"status": {"unknown": "x"}}, + {"status": {"in": "x"}}, + {"confidence": {"gt": True}}, + {"confidence": {"gt": "1"}}, + {"confidence": {"lt": float("inf")}}, + {"confidence": {"gte": 10**400}}, + {"created_at": {"gte": "not-date"}}, + {"created_at": {"lt": 123}}, + {"created_at": "2026-02-30"}, + {"memory": {"like": []}}, + {"bad; DROP TABLE x": 1}, + {"info.a.b": 1}, + {"": 1}, + {12: "x"}, + {"status": float("nan")}, + {"status": ("x",)}, + {"info": {"=": {1: "x"}}}, +] + + +@pytest.mark.parametrize("condition", BAD_FILTERS) +@pytest.mark.parametrize("method", ["read", "list", "delete"]) +def test_bad_business_filters_never_borrow(store, condition, method): + db, pool, _ = store + with pytest.raises(ValueError): + if method == "read": + db.get_by_metadata([], filter=condition) + elif method == "list": + db.get_all_memory_items("LongTermMemory", filter=condition) + else: + db.delete_node_by_prams(filter=condition) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "filters", + [ + None, + {}, + "bad", + [None], + [{}], + [{"field": "x"}], + [{"field": "x", "value": 1, "extra": 1}], + [{"field": "x.y", "value": 1}], + [{"field": "x", "op": "bad", "value": 1}], + [{"field": "x", "op": [], "value": 1}], + ], +) +def test_bad_metadata_filters_never_borrow(store, filters): + db, pool, _ = store + with pytest.raises(ValueError): + db.get_by_metadata(filters) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "kwargs", + [ + {"user_name": ""}, + {"knowledgebase_ids": "K"}, + {"knowledgebase_ids": [""]}, + {"user_name_flag": False}, + {"user_name_flag": 1}, + {"status": 1}, + {"status": ""}, + ], +) +def test_bad_read_options_never_borrow(store, kwargs): + db, pool, _ = store + with pytest.raises(ValueError): + db.get_by_metadata([], **kwargs) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("kwargs", [{"scope": ""}, {"scope": None}, {"include_embedding": 1}]) +def test_bad_listing_options_never_borrow(store, kwargs): + db, pool, _ = store + with pytest.raises(ValueError): + db.get_all_memory_items(**{"scope": "LongTermMemory", **kwargs}) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "kwargs", + [ + {"memory_ids": "a"}, + {"file_ids": [None]}, + {"writable_cube_ids": [""]}, + {"writable_cube_ids": "A"}, + {"memory_ids": ["a"], "file_ids": ["f"]}, + {"memory_ids": ["a"], "filter": {"status": "deleted"}}, + {"file_ids": ["f"], "filter": {"status": "deleted"}}, + ], +) +def test_bad_delete_options_never_borrow(store, kwargs): + db, pool, _ = store + with pytest.raises(ValueError): + db.delete_node_by_prams(**kwargs) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "kwargs", + [ + {}, + {"filter": {}}, + {"memory_ids": []}, + {"file_ids": []}, + {"memory_ids": ["a"], "writable_cube_ids": []}, + ], +) +def test_empty_delete_never_borrows(store, kwargs): + db, pool, _ = store + assert db.delete_node_by_prams(**kwargs) == 0 + pool.transaction.assert_not_called() + + +def test_bound_values_scopes_and_no_input_mutation(store): + db, _, cursor = store + value = "中文 ' $memos$ ; -- %_\\\n" + condition = {"or": [{"project": value}, {"memory": {"like": value}}]} + original = deepcopy(condition) + cursor.fetchall.return_value = [("a",), ("a",), ("k",)] + assert db.get_by_metadata( + [], user_name="B", knowledgebase_ids=["K", "K"], filter=condition + ) == ["a", "a", "k"] + statement, params = cursor.execute.call_args.args + assert "scope = ANY(%(scopes)s)" in statement and "ORDER BY scope, business_id" in statement + assert params["scopes"] == ["B", "K"] + assert value not in statement and "project" not in statement + assert condition == original + + +def test_multi_scope_projection_and_physical_empty_scope(store): + db, _, cursor = store + cursor.fetchall.return_value = [ + ( + "a", + "memo", + json.dumps( + { + "id": "a", + "memory": "memo", + "user_name": "K", + "status": "deleted", + } + ), + "[1,0,0]", + "K", + ) + ] + assert db.get_all_memory_items( + "InsightMemory", knowledgebase_ids=["K"], include_embedding=True + ) == [ + { + "id": "a", + "memory": "memo", + "metadata": {"status": "deleted", "embedding": [1, 0, 0], "user_name": "K"}, + } + ] + db.config.user_name = None + db.config.use_multi_db = True + cursor.fetchall.return_value = [] + assert db.get_by_metadata([]) == [] + assert cursor.execute.call_args.args[1]["scopes"] == [""] + + +@pytest.mark.parametrize( + "operation", + [ + lambda db: db.get_by_metadata([]), + lambda db: db.get_all_memory_items("LongTermMemory"), + lambda db: db.delete_node_by_prams(), + ], +) +def test_closed_store_rejects_even_empty_operations(store, operation): + db, pool, _ = store + pool.ensure_open.side_effect = GraphDBError("closed") + with pytest.raises(GraphDBError, match="closed"): + operation(db) + pool.transaction.assert_not_called() diff --git a/tests/graph_dbs/test_opengauss_fulltext.py b/tests/graph_dbs/test_opengauss_fulltext.py new file mode 100644 index 000000000..7d2824265 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_fulltext.py @@ -0,0 +1,347 @@ +"""Native BM25 contracts, including bounded recovery of a failed write index.""" + +import json + +from copy import deepcopy +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def fulltext_store(monkeypatch, psycopg2_stub): + transaction = MagicMock() + cursor = ( + transaction.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + ) + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", transaction) + config = OpenGaussGraphDBConfig( + host="localhost", + user="fixture", + password="synthetic-password", + db_name="fixture", + user_name="A", + embedding_dimension=3, + ) + return PartialOpenGaussGraphDB(config), transaction, cursor + + +def _hit(node_id="one", score=2.5, scope="A", vector=None, **metadata): + return ( + node_id, + "fixture", + json.dumps({"id": node_id, "memory": "fixture", "user_name": scope, **metadata}), + vector, + scope, + score, + ) + + +def test_fulltext_raw_scores_threshold_and_projection(fulltext_store): + db, _, cursor = fulltext_store + cursor.fetchall.return_value = [ + _hit("low", 1), + _hit("high", 2.5, vector="[1,0,0]", tags=["x"], score_shadow=99), + ] + hits = db.search_by_fulltext( + ["星港"], + threshold=2.5, + return_fields=["memory", "tags", "absent", "id", "score", "memory"], + include_embedding=True, + ) + assert hits == [ + { + "id": "high", + "score": 2.5, + "memory": "fixture", + "tags": ["x"], + "absent": None, + "embedding": [1, 0, 0], + } + ] + + +def test_fulltext_terms_filters_and_scope_are_bound_and_input_is_unchanged(fulltext_store): + db, _, cursor = fulltext_store + cursor.fetchall.return_value = [] + kwargs = { + "query_words": [" '星港' ", '"回归"', "星港", "'it''s'", " ", "x'; DROP TABLE y; --"], + "user_name": "B", + "cube_name": "A'", + "knowledgebase_ids": ["K", "K"], + "scope": "LongTermMemory", + "status": "archived", + "filter": {"project": "星港"}, + "search_filter": {"session_id": "bound-session'"}, + } + before = deepcopy(kwargs) + db.search_by_fulltext(**kwargs) + statement, params = cursor.execute.call_args.args + assert params["query"] == "星港 回归 it's x'; DROP TABLE y; --" + assert ["A'", "K"] in params.values() + assert ( + "scope = ANY(%(scopes)s)" in statement + and "ORDER BY memory <&> %(query)s DESC LIMIT %(top_k)s" in statement + ) + for value in ("星港", "bound-session'", "DROP TABLE", "LongTermMemory", "archived"): + assert value not in statement + assert kwargs == before + + +@pytest.mark.parametrize("top_k", [1, 5, 200, 201, 210]) +def test_fulltext_uses_taat_without_clamping_sql_limit(fulltext_store, top_k): + db, _, cursor = fulltext_store + cursor.fetchall.return_value = [_hit()] + assert db.search_by_fulltext(["星港"], top_k=top_k) == [{"id": "one", "score": 2.5}] + cursor.execute.assert_any_call("SET LOCAL enable_bm25_taat = on") + assert cursor.execute.call_args.args[1]["top_k"] == top_k + + +@pytest.mark.parametrize("words", [[], ["", " ", "''", '""']]) +def test_fulltext_empty_words_do_not_borrow(fulltext_store, words): + db, transaction, _ = fulltext_store + assert db.search_by_fulltext(words) == [] + transaction.assert_not_called() + + +@pytest.mark.parametrize("words", [None, "星港", ("星港",), [1], [None], [True]]) +def test_fulltext_bad_word_types_are_not_stringified(fulltext_store, words): + db, transaction, _ = fulltext_store + with pytest.raises(ValueError): + db.search_by_fulltext(words) + transaction.assert_not_called() + + +@pytest.mark.parametrize( + "kwargs", + [ + {"top_k": -1}, + {"top_k": True}, + {"top_k": 1.5}, + {"threshold": -1}, + {"threshold": True}, + {"threshold": "1"}, + {"threshold": float("nan")}, + {"threshold": float("inf")}, + {"threshold": 10**400}, + {"scope": ""}, + {"cube_name": " "}, + {"status": False}, + {"search_filter": []}, + {"filter": {"or": []}}, + {"return_fields": "memory"}, + {"return_fields": [None]}, + {"return_fields": ["bad; field"]}, + {"include_embedding": 1}, + {"tsquery_config": "custom"}, + {"tsvector_field": "bad;"}, + ], +) +def test_fulltext_invalid_options_fail_even_for_empty_query(fulltext_store, kwargs): + db, transaction, _ = fulltext_store + with pytest.raises(ValueError): + db.search_by_fulltext([], **kwargs) + transaction.assert_not_called() + + +@pytest.mark.parametrize("config", [None, "jiebacfg", "jiebaqry"]) +def test_fulltext_accepts_only_documented_consumer_aliases(fulltext_store, config): + db, _, cursor = fulltext_store + cursor.fetchall.return_value = [_hit()] + assert db.search_by_fulltext(["星港"], tsvector_field="memory", tsquery_config=config) + + +def test_fulltext_zero_limit_and_closed_store(fulltext_store): + db, transaction, _ = fulltext_store + assert db.search_by_fulltext(["星港"], top_k=0) == [] + transaction.assert_not_called() + db.close() + with pytest.raises(GraphDBError, match="closed"): + db.search_by_fulltext([]) + + +@pytest.mark.parametrize("score", [None, "bad", -1, float("nan"), float("inf"), 10**400]) +def test_fulltext_corrupt_scores_fail_explicitly(fulltext_store, score): + db, _, cursor = fulltext_store + cursor.fetchall.return_value = [_hit(score=score)] + with pytest.raises(GraphDBError): + db.search_by_fulltext(["星港"]) + + +def test_fulltext_same_id_scope_ties_and_embedding_field(fulltext_store): + db, _, cursor = fulltext_store + cursor.fetchall.return_value = [ + _hit("same", scope="K", vector="[1,0,0]"), + _hit("same", vector="[0,1,0]"), + ] + hits = db.search_by_fulltext( + ["星港"], knowledgebase_ids=["K"], return_fields=["user_name", "embedding"] + ) + assert [(h["id"], h["user_name"], h["embedding"]) for h in hits] == [ + ("same", "A", [0, 1, 0]), + ("same", "K", [1, 0, 0]), + ] + + +REBUILD_MESSAGE = "Last batch insert document failed, scanned score maybe affected, please reindex or recreate bm25 index [memory_fulltext_bm25]." + + +def _failure(code, message): + import psycopg2 + + class DriverFailure(psycopg2.Error): + @property + def pgcode(self): + return code + + @property + def diag(self): + return SimpleNamespace(message_primary=message) + + error = GraphDBError("query failed") + error.__cause__ = DriverFailure() + return error + + +def test_fulltext_known_rollback_error_repairs_once_then_retries_read(fulltext_store, monkeypatch): + db, transaction, cursor = fulltext_store + repair = MagicMock() + monkeypatch.setattr(db, "_rebuild_fulltext_index", repair) + cursor.fetchall.side_effect = [_failure("42809", REBUILD_MESSAGE), [_hit()]] + assert db.search_by_fulltext(["星港"]) == [{"id": "one", "score": 2.5}] + repair.assert_called_once_with() + assert transaction.call_count == 2 + + +@pytest.mark.parametrize( + "code,message", + [ + (None, None), + ("42501", REBUILD_MESSAGE), + ("42809", "No BM25 index is used to the scan, please check the plan."), + ], +) +def test_fulltext_unrelated_errors_do_not_rebuild(fulltext_store, monkeypatch, code, message): + failure = GraphDBError("no cause") if code is None else _failure(code, message) + db, transaction, cursor = fulltext_store + repair = MagicMock() + monkeypatch.setattr(db, "_rebuild_fulltext_index", repair) + cursor.fetchall.side_effect = failure + with pytest.raises(GraphDBError) as caught: + db.search_by_fulltext(["星港"]) + assert caught.value is failure + repair.assert_not_called() + assert transaction.call_count == 1 + + +@pytest.mark.parametrize("repair_fails", [False, True]) +def test_fulltext_recovery_failure_is_not_empty_or_infinite( + fulltext_store, monkeypatch, repair_fails +): + db, _, cursor = fulltext_store + failure = _failure("42809", REBUILD_MESSAGE) + cursor.fetchall.side_effect = failure + repair = MagicMock(side_effect=GraphDBError("repair failed") if repair_fails else None) + monkeypatch.setattr(db, "_rebuild_fulltext_index", repair) + with pytest.raises(GraphDBError): + db.search_by_fulltext(["星港"]) + repair.assert_called_once_with() + assert cursor.fetchall.call_count == (1 if repair_fails else 2) + + +INDEX_ROW = ( + "memos_graph", + "Memory", + "bm25", + "bm25_text_ops", + "5", + True, + True, + None, + None, + ["dict_path=DEFAULT"], +) + + +@pytest.mark.parametrize("existing", [None, INDEX_ROW]) +def test_fulltext_index_creation_and_compatible_reuse(fulltext_store, existing): + db, _, cursor = fulltext_store + cursor.fetchone.side_effect = [None, INDEX_ROW] if existing is None else [existing] + db._init_fulltext_index(cursor) + queries = [call.args[0] for call in cursor.execute.call_args_list] + assert sum("CREATE INDEX" in query for query in queries) == (existing is None) + assert not any("DROP" in query for query in queries) + + +@pytest.mark.parametrize( + "offset,value", + [ + (0, "other"), + (1, "Other"), + (2, "btree"), + (3, "text_ops"), + (4, "6"), + (5, False), + (6, False), + (7, "predicate"), + (8, "expression"), + (9, ["dict_path=/custom"]), + ], +) +def test_fulltext_incompatible_index_is_not_replaced(fulltext_store, offset, value): + db, _, cursor = fulltext_store + row = list(INDEX_ROW) + row[offset] = value + cursor.fetchone.return_value = tuple(row) + with pytest.raises(GraphDBError): + db._init_fulltext_index(cursor) + assert not any( + "CREATE" in call.args[0] or "DROP" in call.args[0] for call in cursor.execute.call_args_list + ) + + +@pytest.mark.parametrize("present", [False, True]) +def test_fulltext_rebuild_locks_and_only_repairs_the_owned_index(fulltext_store, present): + db, _, cursor = fulltext_store + cursor.fetchone.return_value = INDEX_ROW if present else None + if present: + db._rebuild_fulltext_index() + cursor.execute.assert_any_call("REINDEX INDEX memos_graph.memory_fulltext_bm25") + else: + with pytest.raises(GraphDBError): + db._rebuild_fulltext_index() + assert not any("REINDEX" in call.args[0] for call in cursor.execute.call_args_list) + cursor.execute.assert_any_call("SELECT pg_advisory_xact_lock(621503, %(lock)s)", {"lock": 1}) + + +@pytest.mark.parametrize("failure", ["none", "one_deadlock", "two_deadlocks", "permission"]) +def test_owned_test_graph_cleanup_only_retries_one_deadlock(monkeypatch, psycopg2_stub, failure): + import psycopg2 + + from tests.graph_dbs import test_opengauss_integration as integration + + query = MagicMock() + monkeypatch.setattr(integration, "_query", query) + if failure == "one_deadlock": + query.side_effect = [psycopg2.errors.DeadlockDetected(), []] + elif failure == "two_deadlocks": + query.side_effect = psycopg2.errors.DeadlockDetected() + elif failure == "permission": + query.side_effect = psycopg2.errors.InsufficientPrivilege() + if failure in ("two_deadlocks", "permission"): + with pytest.raises(psycopg2.Error): + integration._drop_test_graph(None) + else: + integration._drop_test_graph(None) + assert query.call_count == (2 if "deadlock" in failure else 1) + assert all( + call.args[1] == "SELECT ag_catalog.drop_graph('memos_graph',true)" + for call in query.call_args_list + ) diff --git a/tests/graph_dbs/test_opengauss_import_clear.py b/tests/graph_dbs/test_opengauss_import_clear.py new file mode 100644 index 000000000..c8db39468 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_import_clear.py @@ -0,0 +1,101 @@ +"""Atomic import and Cube clearing validation and failure contracts.""" + +from copy import deepcopy +from unittest.mock import MagicMock + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity import capacity as _capacity +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +capacity, pool_case = _capacity, _pool_case +CALLS = [ + ("import_graph", ({"nodes": [{"id": "a", "memory": "text"}]},)), + ("clear", ()), + ("drop_database", ()), +] + + +@pytest.mark.parametrize( + "data", + [ + None, + [], + {"nodes": {}}, + {"edges": {}}, + {"nodes": [None]}, + {"nodes": [{"id": "a"}]}, + {"nodes": [{"id": "a", "memory": "x", "metadata": None}]}, + {"nodes": [{"id": "a", "memory": "x"}, {"id": "a", "memory": "y"}]}, + {"nodes": [{"id": "a", "memory": "x", "metadata": {"embedding": [1, 2]}}]}, + {"nodes": [{"id": "a", "memory": "x", "metadata": {"user_name": "B"}}]}, + {"nodes": [{"id": "a", "memory": "x", "metadata": {"info": {"x": ["\0"]}}}]}, + {"nodes": [{"id": "a", "memory": "x", "metadata": {"\0": 1}}]}, + {"edges": [None]}, + {"edges": [{}]}, + {"edges": [{"source": "a", "target": "b", "type": "ANY"}]}, + {"edges": [{"source": "a", "target": "b", "type": "PARENT", "user_name": "B"}]}, + {"edges": [{"source": "", "target": "b", "type": "PARENT"}]}, + ], +) +def test_import_validation_is_complete_before_borrow(capacity, data): + db, pool, _, _ = capacity + before = deepcopy(data) + with pytest.raises(ValueError): + db.import_graph(data) + assert data == before + pool.getconn.assert_not_called() + + +def test_empty_import_does_not_borrow(capacity): + db, pool, _, _ = capacity + assert db.import_graph({}) is None + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("method,args", CALLS) +def test_closed_maintenance_rejects_calls(capacity, method, args): + db, pool, _, _ = capacity + db.close() + with pytest.raises(GraphDBError, match="closed"): + getattr(db, method)(*args) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize("method,args", CALLS) +@pytest.mark.parametrize("stage", ["execute", "fetch", "commit", "putconn"]) +def test_import_clear_transaction_failure(capacity, method, args, stage): + import psycopg2 + + db, pool, connection, cursor = capacity + error = psycopg2.ProgrammingError("maintenance injection") + if stage == "execute": + cursor.execute.side_effect = error + elif stage == "fetch": + cursor.fetchall.side_effect = error + elif stage == "commit": + connection.commit.side_effect = error + else: + pool.putconn.side_effect = error + with pytest.raises(GraphDBError) as caught: + getattr(db, method)(*args) + assert caught.value.__cause__ is error + assert db._active == 0 and pool.getconn.call_count == 1 + + +def test_clear_chunks_share_one_transaction(capacity, monkeypatch): + db, _, connection, cursor = capacity + keys = [f"x{i:04}" for i in range(1200)] + cursor.fetchall.return_value = [(key,) for key in keys] + monkeypatch.setattr(db, "_cypher", MagicMock()) + db.clear(user_name="B") + deletes = [ + c.args[1] for c in cursor.execute.call_args_list if c.args[0].startswith("DELETE FROM ONLY") + ] + assert deletes == [ + {"scope": "B", "ids": batch} for batch in [keys[:500], keys[500:1000], keys[1000:]] + ] + assert db._cypher.call_count == 3 + connection.commit.assert_called_once() diff --git a/tests/graph_dbs/test_opengauss_import_clear_integration.py b/tests/graph_dbs/test_opengauss_import_clear_integration.py new file mode 100644 index 000000000..581ffc853 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_import_clear_integration.py @@ -0,0 +1,367 @@ +"""Real import round trips, rollback and fixed-set clearing.""" + +import os + +from concurrent.futures import ThreadPoolExecutor +from copy import deepcopy +from threading import Event + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs import test_opengauss_capacity_integration as support +from tests.graph_dbs.test_opengauss_integration import _query + + +database, stores = support.database, support.stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +def fixture(): + return { + "nodes": [ + { + "id": key, + "memory": "星港 " + key, + "metadata": { + "memory_type": "WorkingMemory", + "status": state, + "embedding": [1, 0, 0], + "created_at": "2026-01-01T00:00:00Z", + "info": {"keep": [True, None, 3]}, + }, + } + for key, state in [("a", "activated"), ("b", "archived"), ("c", "deleted")] + ], + "edges": [ + {"source": a, "target": b, "type": t} + for a, b, t in [ + ("a", "b", "PARENT"), + ("b", "a", "PARENT"), + ("a", "a", "FOLLOWS"), + ("a", "b", "RELATE"), + ] + ], + } + + +def test_import_export_roundtrip_all_status_vectors_and_edges(stores, database): + db = stores() + data = fixture() + data["edges"].append(deepcopy(data["edges"][0])) + before = deepcopy(data) + db.import_graph(data) + assert data == before + backup = db.export_graph(status=[], include_embedding=True) + assert backup["total_nodes"] == 3 and backup["total_edges"] == 4 + assert sorted((e["source"], e["target"], e["type"]) for e in backup["edges"]) == [ + ("a", "a", "FOLLOWS"), + ("a", "b", "PARENT"), + ("a", "b", "RELATE"), + ("b", "a", "PARENT"), + ] + db.add_node("a", "foreign", {}, user_name="B") + foreign = db.get_node("a", user_name="B") + db.clear() + assert db.export_graph(status=[])["nodes"] == [] + db.import_graph(backup) + assert db.export_graph(status=[], include_embedding=True) == backup + assert db.get_node("a", user_name="B") == foreign + support.visible(db, ["a"]) + db.drop_database() + assert db.get_node("a", user_name="B") == foreign + assert db.export_graph(status=[])["nodes"] == [] + db.import_graph({}) + + +def test_upsert_preserves_graph_identity_and_file_external_data(stores, database): + db = stores() + admin, _ = database + db.import_graph(fixture()) + old = support.snapshot(admin) + db.import_graph( + { + "nodes": [{"id": "a", "memory": "replacement", "metadata": {"status": "activated"}}], + "edges": [{"source": "a", "target": "c", "type": "NEW", "user_name": "A"}], + } + ) + new = support.snapshot(admin) + assert new["nodes"][0][1] == old["nodes"][0][1] + assert new["nodes"][1:] == old["nodes"][1:] + assert db.get_node("a", True) == { + "id": "a", + "memory": "replacement", + "metadata": {"status": "activated", "embedding": None}, + } + assert db.export_graph(status=[])["total_edges"] == 5 + db.import_graph({"edges": [{"source": "b", "target": "c", "type": "EXISTING"}]}) + assert db.edge_exists("b", "c", "EXISTING") + + +def test_missing_endpoint_and_late_node_or_edge_sql_failure_roll_back(stores, database): + db = stores() + admin, _ = database + db.import_graph(fixture()) + before = support.snapshot(admin) + with pytest.raises(GraphDBError, match="endpoint"): + db.import_graph( + { + "nodes": [{"id": "new", "memory": "x"}], + "edges": [{"source": "new", "target": "missing", "type": "PARENT"}], + } + ) + assert support.snapshot(admin) == before + _query( + admin, + """CREATE FUNCTION memos_graph.fail_import() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.business_id='z' THEN RAISE EXCEPTION 'import injection'; END IF; RETURN NEW; END $$""", + ) + _query( + admin, + """CREATE TRIGGER fail_import BEFORE INSERT ON memos_graph."Memory" FOR EACH ROW EXECUTE PROCEDURE memos_graph.fail_import()""", + ) + try: + with pytest.raises(GraphDBError): + db.import_graph( + {"nodes": [{"id": "a", "memory": "changed"}, {"id": "z", "memory": "failure"}]} + ) + finally: + _query(admin, 'DROP TRIGGER fail_import ON memos_graph."Memory"') + _query(admin, "DROP FUNCTION memos_graph.fail_import()") + assert support.snapshot(admin) == before + # A real AGE label permission failure happens after input nodes have been upserted. + _, config = database + from psycopg2 import sql + + _query( + admin, + sql.SQL('REVOKE INSERT ON memos_graph."PARENT" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError): + db.import_graph( + { + "nodes": [{"id": "z", "memory": "new"}], + "edges": [{"source": "z", "target": "b", "type": "PARENT"}], + } + ) + finally: + _query( + admin, + sql.SQL('GRANT INSERT ON memos_graph."PARENT" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert support.snapshot(admin) == before + support.visible(db, ["a"]) + + +@pytest.mark.parametrize("failure", [None, "x0500", "x1000", "timeout", "permission"]) +def test_clear_three_blocks_rollback_and_search_visibility(stores, database, monkeypatch, failure): + from psycopg2 import sql + + db = stores(maxconn=1) + admin, config = database + support.seed_large(db, admin) + before = support.snapshot(admin) + if failure is None: + db.clear() + assert support.snapshot(admin) == {"nodes": [], "edges": []} + assert db.search_by_embedding([1, 0, 0]) == [] + assert db.search_by_fulltext(["批量"]) == [] + return + if failure == "permission": + _query( + admin, + sql.SQL('REVOKE DELETE ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as caught: + db.clear() + assert caught.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('GRANT DELETE ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + else: + + def hook(stage, cursor, rows): + if ( + stage == "execute" + and "pg_advisory_xact_lock" in cursor.statement + and failure == "timeout" + ): + cursor.cursor.execute("SET LOCAL statement_timeout='500ms'") + + support.observe(db, monkeypatch, hook) + with ( + support.failing_delete( + admin, + node_id="x1000" if failure == "timeout" else failure, + action="PERFORM pg_sleep(2)" + if failure == "timeout" + else "RAISE EXCEPTION 'clear failure'", + ), + pytest.raises(GraphDBError) as caught, + ): + db.clear() + assert caught.value.__cause__.pgcode == ("57014" if failure == "timeout" else "P0001") + assert support.snapshot(admin) == before + assert len(db.search_by_embedding([1, 0, 0], top_k=1300)) == 1200 + assert len(db.search_by_fulltext(["批量"], top_k=1300)) == 1200 + + +def test_clear_fixed_set_excludes_later_insert_and_serializes_update(stores, database, monkeypatch): + db, writer = stores(), stores() + db.import_graph(fixture()) + selected, release = Event(), Event() + + def hook(stage, cursor, rows): + if stage == "fetchall" and "FOR UPDATE" in cursor.statement: + selected.set() + assert release.wait(8) + + with monkeypatch.context() as patch: + support.observe(db, patch, hook) + with ThreadPoolExecutor(max_workers=2) as executor: + clear = executor.submit(db.clear) + try: + assert selected.wait(8) + writer.add_node("later", "new insertion", {}) + update = executor.submit(writer.add_node, "a", "rebuilt", {}) + assert not update.done() + finally: + release.set() + clear.result(timeout=8) + update.result(timeout=8) + assert support.ids(db) == ["a", "later"] + assert db.get_node("a")["memory"] == "rebuilt" + assert db.get_edges("a") == [] + + +@pytest.mark.parametrize("operation", ["import", "clear"]) +@pytest.mark.parametrize("stage", ["before_commit", "after_commit", "putconn"]) +def test_commit_uncertainty_is_observable_without_replay( + stores, database, monkeypatch, operation, stage +): + import psycopg2 + + db = stores(maxconn=1) + admin, _ = database + db.import_graph(fixture()) + before = support.snapshot(admin) + pool = db.connection_pool + getconn, putconn = pool.getconn, pool.putconn + calls = [] + error = psycopg2.OperationalError("synthetic commit confirmation loss") + + class Connection: + def __init__(self, raw): + self.raw = raw + + def __getattr__(self, name): + return getattr(self.raw, name) + + def commit(self): + calls.append("commit") + if stage != "before_commit": + self.raw.commit() + if stage != "putconn": + raise error + + def borrow(): + calls.append("borrow") + return Connection(getconn()) + + def release(connection, close=False): + if stage == "putconn": + raise error + putconn(connection.raw, close=close) + + with monkeypatch.context() as patch: + patch.setattr(pool, "getconn", borrow) + patch.setattr(pool, "putconn", release) + with pytest.raises(GraphDBError) as caught: + if operation == "import": + db.import_graph({"nodes": [{"id": "z", "memory": "committed"}]}) + else: + db.clear() + assert caught.value.__cause__ is error + assert calls == ["borrow", "commit"] + if stage == "before_commit": + assert support.snapshot(admin) == before + else: + check = stores() if stage == "putconn" else db + assert support.ids(check) == (["a", "b", "c", "z"] if operation == "import" else []) + + +@pytest.mark.parametrize("operation", ["clear", "hard_delete"]) +def test_scoped_delete_actual_cancel_restores_data_and_replaces_connection( + stores, database, operation +): + db = stores(maxconn=1) + admin, _ = database + db.import_graph(fixture()) + if operation == "hard_delete": + db.delete_node_by_mem_cube_id("A", "R") + before = support.snapshot(admin) + raw = db.connection_pool.getconn() + pid = _query(raw, "SELECT pg_backend_pid()")[0][0] + raw.rollback() + db.connection_pool.putconn(raw) + _query(admin, "SELECT pg_advisory_lock(621503, 9000)") + try: + with ( + support.failing_delete( + admin, node_id="a", action="PERFORM pg_advisory_xact_lock(621503, 9000)" + ), + ThreadPoolExecutor(max_workers=1) as executor, + ): + future = ( + executor.submit(db.clear) + if operation == "clear" + else executor.submit(db.delete_node_by_mem_cube_id, "A", "R", True) + ) + try: + support.wait_until( + lambda: _query( + admin, + "SELECT 1 FROM pg_locks WHERE pid=%s AND locktype='advisory' AND NOT granted", + (pid,), + ) + ) + raw.cancel() + with pytest.raises(GraphDBError) as caught: + future.result(timeout=8) + assert caught.value.__cause__.pgcode == "57014" + finally: + _query(admin, "SELECT pg_advisory_unlock(621503, 9000)") + finally: + _query(admin, "SELECT pg_advisory_unlock_all()") + assert raw.closed and support.snapshot(admin) == before + if operation == "hard_delete": + assert db.search_by_embedding([1, 0, 0]) == [] + assert db.search_by_fulltext(["星港"]) == [] + db.recover_memory_by_mem_cube_id("A", "R") + assert db.get_node("a")["metadata"]["status"] == "activated" + assert db.search_by_embedding([1, 0, 0]) + assert db.search_by_fulltext(["星港"]) + return + support.visible(db, ["a"]) + + +def test_physical_scope_clear_does_not_remove_named_cube(stores): + db = stores(use_multi_db=True, user_name=None) + db.import_graph(fixture()) + db.import_graph({"nodes": [{"id": "a", "memory": "named"}]}, user_name="A") + db.drop_database() + assert db.export_graph(status=[])["nodes"] == [] + assert db.get_node("a", user_name="A")["memory"] == "named" diff --git a/tests/graph_dbs/test_opengauss_integration.py b/tests/graph_dbs/test_opengauss_integration.py new file mode 100644 index 000000000..c9c2ff008 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_integration.py @@ -0,0 +1,2202 @@ +"""Opt-in, real-driver AGE storage tests in a disposable database. + +Set MEMOS_OPENGAUSS_TEST_CONFIG to an OpenGaussGraphDBConfig JSON object for a +test administrator. These tests create/drop only their own UUID database and +role; the application under test uses an ordinary role, without SYSADMIN. +""" + +import json +import math +import os +import secrets +import threading +import time +import uuid + +from concurrent.futures import ThreadPoolExecutor + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.dependency import require_python_package +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +@require_python_package( + import_name="psycopg2", + install_command="pip install psycopg2-binary", + install_link="https://pypi.org/project/psycopg2-binary/", +) +def _connect(config): + import psycopg2.sql + + return psycopg2.connect( + host=config.host, + port=config.port, + dbname=config.db_name, + user=config.user, + password=config.password, + connect_timeout=5, + application_name="memos-opengauss-it-admin", + ) + + +def _query(admin, statement, params=()): + with admin.cursor() as cursor: + cursor.execute("SELECT 'null'::ag_catalog.agtype") + cursor.execute("SET search_path = ag_catalog, public") + cursor.execute(statement, params) + return cursor.fetchall() if cursor.description else [] + + +@pytest.fixture(scope="module") +def database(): + if not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"): + pytest.skip("openGauss integration tests not enabled") + config = OpenGaussGraphDBConfig.model_validate_json(os.environ["MEMOS_OPENGAUSS_TEST_CONFIG"]) + manager = _connect(config) + from psycopg2 import sql + + manager.autocommit = True + suffix = uuid.uuid4().hex[:12] + name, role = "memos_it_" + suffix, "memos_it_app_" + suffix + password = "It_A7_" + secrets.token_hex(8) + created_database = created_role = False + admin = None + try: + with manager.cursor() as cursor: + cursor.execute( + sql.SQL("CREATE USER {} PASSWORD %s").format(sql.Identifier(role)), (password,) + ) + created_role = True + cursor.execute( + sql.SQL("CREATE DATABASE {} DBCOMPATIBILITY 'PG'").format(sql.Identifier(name)) + ) + created_database = True + target = config.model_copy(update={"db_name": name}) + admin = _connect(target) + admin.autocommit = True + with admin.cursor() as cursor: + cursor.execute("CREATE EXTENSION age") + cursor.execute( + sql.SQL("GRANT CREATE ON DATABASE {} TO {}").format( + sql.Identifier(name), sql.Identifier(role) + ) + ) + cursor.execute( + sql.SQL("GRANT USAGE ON SCHEMA ag_catalog TO {}").format(sql.Identifier(role)) + ) + cursor.execute( + sql.SQL("GRANT SELECT ON ag_catalog.ag_graph,ag_catalog.ag_label TO {}").format( + sql.Identifier(role) + ) + ) + application = target.model_copy( + update={ + "user": role, + "password": password, + "user_name": "A", + "embedding_dimension": 3, + "minconn": 1, + "maxconn": 2, + } + ) + yield admin, application + finally: + if admin is not None: + admin.close() + with manager.cursor() as cursor: + if created_database: + cursor.execute(sql.SQL("DROP DATABASE {}").format(sql.Identifier(name))) + if created_role: + cursor.execute(sql.SQL("DROP USER {}").format(sql.Identifier(role))) + manager.close() + + +@pytest.fixture +def stores(database): + admin, config = database + opened = [] + if _query(admin, "SELECT 1 FROM ag_catalog.ag_graph WHERE name='memos_graph'"): + _drop_test_graph(admin) + + def make(**overrides): + store = PartialOpenGaussGraphDB(config.model_copy(update=overrides)) + opened.append(store) + return store + + yield make + for store in opened: + store.close() + # Client close sends Terminate without waiting for backend termination. + # Verify server cleanup within a fixed deadline; a persistent leak fails. + deadline = time.monotonic() + 5 + while True: + remaining = _query( + admin, + "SELECT count(*) FROM pg_stat_activity WHERE datname=%s AND application_name='memos-opengauss'", + (config.db_name,), + ) + if remaining == [(0,)] or time.monotonic() >= deadline: + break + time.sleep(0.01) + assert remaining == [(0,)] + + +def _drop_test_graph(admin): + # The WLM space-accounting worker can race AGE's drop lock order. Only + # this owned test object is retried, once, after the autocommit rollback. + import psycopg2 + + statement = "SELECT ag_catalog.drop_graph('memos_graph',true)" + try: + _query(admin, statement) + except psycopg2.errors.DeadlockDetected: + _query(admin, statement) + + +def test_registered_age_label_and_ordinary_application_role(stores, database): + db = stores() + admin, config = database + assert _query( + admin, "SELECT rolsystemadmin FROM pg_roles WHERE rolname=%s", (config.user,) + ) == [(False,)] + assert _query( + admin, + "SELECT l.kind FROM ag_catalog.ag_label l JOIN ag_catalog.ag_graph g ON g.oid=l.graph WHERE g.name='memos_graph' AND l.name='Memory'", + ) == [("v",)] + db.add_node( + "one", + "中文 emoji 🚀 ' ; DROP TABLE Memory; --\n", + {"embedding": [1, 0, 0], "info": {"nested": [True, None, 3]}}, + ) + row = db.get_node("one", include_embedding=True) + assert row == { + "id": "one", + "memory": "中文 emoji 🚀 ' ; DROP TABLE Memory; --\n", + "metadata": {"embedding": [1, 0, 0], "info": {"nested": [True, None, 3]}}, + } + + +@pytest.mark.parametrize( + "value", [None, [], {}, "raw or malformed JSON {", [{"type": "chat", "content": "fixture"}]] +) +def test_structured_and_serialized_metadata_round_trip(stores, value): + db = stores() + metadata = { + "sources": value, + "usage": value, + "info": value, + "created_at": "2026-01-01T00:00:00Z", + } + db.add_node("one", "", metadata) + assert db.get_node("one") == {"id": "one", "memory": "", "metadata": metadata} + assert db.get_node("one", include_embedding=True)["metadata"]["embedding"] is None + + +def test_scopes_batch_order_last_upsert_and_missing_records(stores): + db = stores() + db.add_nodes_batch( + [ + {"id": "same", "memory": "before"}, + {"id": "two", "memory": "second"}, + {"id": "same", "memory": "last", "metadata": {"status": "archived"}}, + ] + ) + db.add_node("same", "scope B", {}, user_name="B") + assert db.get_nodes(["two", "absent", "same", "two"]) == [ + {"id": "two", "memory": "second", "metadata": {}}, + {"id": "same", "memory": "last", "metadata": {"status": "archived"}}, + ] + assert db.get_node("same", user_name="B")["memory"] == "scope B" + assert db.get_node("absent") is None + db.update_node("absent", {"memory": "no insert"}) + assert db.get_node("absent") is None + with pytest.raises(ValueError): + db.update_node("same", {"user_name": "B"}) + assert db.get_node("same", user_name="B")["memory"] == "scope B" + + +def test_upsert_and_update_preserve_graphid_edges_and_projection(stores, database): + db = stores() + admin, _ = database + db.add_nodes_batch( + [ + { + "id": "one", + "memory": "before", + "metadata": {"embedding": [1, 0, 0], "tags": ["old"]}, + }, + {"id": "two", "memory": "target"}, + ] + ) + graphid = _query( + admin, + 'SELECT id::text FROM memos_graph."Memory" WHERE scope=%s AND business_id=%s', + ("A", "one"), + )[0][0] + _query( + admin, + "SELECT * FROM ag_catalog.cypher('memos_graph',$$MATCH (a:Memory {id:'one',user_name:'A'}),(b:Memory {id:'two',user_name:'A'}) CREATE (a)-[:RELATE_TO]->(b) RETURN id(a)$$) AS (id agtype)", + ) + db.add_node("one", "replacement", {"embedding": [0, 1, 0], "tags": ["retained"]}) + db.update_node("one", {"info": {"updated": True}, "memory": "updated"}) + assert db.get_node("one", include_embedding=True) == { + "id": "one", + "memory": "updated", + "metadata": {"embedding": [0, 1, 0], "tags": ["retained"], "info": {"updated": True}}, + } + rows = _query( + admin, + "SELECT * FROM ag_catalog.cypher('memos_graph',$$MATCH (a:Memory)-[r:RELATE_TO]->(b:Memory) RETURN id(a),a.memory,b.id$$) AS (id agtype,memory agtype,target agtype)", + ) + assert len(rows) == 1 + assert [json.loads(value) for value in rows[0]] == [int(graphid), "updated", "two"] + db.update_node("one", {"embedding": []}) + assert db.get_node("one", include_embedding=True)["metadata"]["embedding"] is None + assert _query( + admin, + 'SELECT id::text,embedding::text FROM memos_graph."Memory" WHERE business_id=%s', + ("one",), + ) == [(graphid, None)] + + +@pytest.mark.parametrize("position", [0, 1, 2]) +def test_execution_failure_rolls_back_entire_batch(stores, database, position): + db = stores() + admin, _ = database + db.add_node("keep", "existing", {}) + _query( + admin, + "CREATE OR REPLACE FUNCTION public.memos_it_reject() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.business_id='bad' THEN RAISE EXCEPTION 'synthetic rejected row'; END IF; RETURN NEW; END $$", + ) + _query( + admin, + 'CREATE TRIGGER memos_it_reject BEFORE INSERT ON memos_graph."Memory" FOR EACH ROW EXECUTE PROCEDURE public.memos_it_reject()', + ) + nodes = [{"id": str(i), "memory": "temporary"} for i in range(3)] + nodes[position]["id"] = "bad" + with pytest.raises(GraphDBError) as error: + db.add_nodes_batch(nodes) + assert error.value.__cause__.pgcode == "P0001" + assert _query(admin, 'SELECT scope,business_id,memory FROM memos_graph."Memory"') == [ + ("A", "keep", "existing") + ] + assert db.get_node("keep")["memory"] == "existing" + + +def test_other_unique_constraints_cannot_upsert_another_scope(stores, database): + db = stores() + admin, _ = database + db.add_node("foreign", "same text", {}, user_name="B") + _query(admin, 'CREATE UNIQUE INDEX memos_it_text_unique ON memos_graph."Memory" (memory)') + with pytest.raises(GraphDBError) as error: + db.add_node("new", "same text", {}, user_name="A") + assert error.value.__cause__.pgcode == "23505" + assert db.get_node("new") is None + assert db.get_node("foreign", user_name="B") == { + "id": "foreign", + "memory": "same text", + "metadata": {}, + } + + +def test_reinitialization_and_concurrent_initialization_preserve_data(stores, database): + barrier = threading.Barrier(2, timeout=10) + + def initialize(): + barrier.wait() + return stores() + + with ThreadPoolExecutor(2) as executor: + one, two = [ + future.result(timeout=20) + for future in [executor.submit(initialize), executor.submit(initialize)] + ] + one.add_node("persisted", "existing", {}) + one.close() + three = stores() + assert ( + two.get_node("persisted") + == three.get_node("persisted") + == {"id": "persisted", "memory": "existing", "metadata": {}} + ) + + +def test_concurrent_upserts_are_unique_and_complete(stores, database): + db = stores() + admin, _ = database + barrier = threading.Barrier(2, timeout=10) + + def write(writer): + barrier.wait() + db.add_node("one", writer, {"info": {"writer": writer}}) + + with ThreadPoolExecutor(2) as executor: + futures = [executor.submit(write, writer) for writer in ["first", "second"]] + for future in futures: + future.result(timeout=15) + assert _query(admin, 'SELECT count(*) FROM memos_graph."Memory"') == [(1,)] + node = db.get_node("one") + assert node in [ + {"id": "one", "memory": writer, "metadata": {"info": {"writer": writer}}} + for writer in ["first", "second"] + ] + + +def test_partial_update_reads_after_row_lock_without_losing_fields(stores): + db = stores() + db.add_node("one", "before", {}) + barrier = threading.Barrier(2, timeout=10) + + def update(): + barrier.wait() + db.update_node("one", {"tags": ["second"]}) + + with ThreadPoolExecutor(1) as executor: + with db._get_connection() as connection, connection.cursor() as cursor: + cursor.execute( + 'UPDATE memos_graph."Memory" SET properties=%s::agtype WHERE scope=%s AND business_id=%s', + ( + json.dumps( + {"id": "one", "memory": "before", "user_name": "A", "info": {"first": 1}} + ), + "A", + "one", + ), + ) + future = executor.submit(update) + barrier.wait() + future.result(timeout=15) + assert db.get_node("one") == { + "id": "one", + "memory": "before", + "metadata": {"info": {"first": 1}, "tags": ["second"]}, + } + + +def test_failed_transactions_dirty_sessions_and_replacement_connections(stores): + db = stores(maxconn=1) + db.add_node("one", "keep", {}) + with ( + pytest.raises(GraphDBError) as error, + db._get_connection() as connection, + connection.cursor() as cursor, + ): + cursor.execute("SELECT 1/0") + assert error.value.__cause__.pgcode == "22012" + with db._get_connection() as connection, connection.cursor() as cursor: + cursor.execute("SET search_path = pg_catalog") + with db._get_connection() as connection, connection.cursor() as cursor: + cursor.execute("SHOW search_path") + assert cursor.fetchone()[0] == "ag_catalog, public" + cursor.execute( + "SELECT * FROM ag_catalog.cypher('memos_graph',$$MATCH (n:Memory) RETURN n.id$$) AS (id agtype)" + ) + assert json.loads(cursor.fetchone()[0]) == "one" + connection.set_session(isolation_level="SERIALIZABLE", readonly=True, autocommit=True) + with db._get_connection() as connection, connection.cursor() as cursor: + assert connection.autocommit is False + cursor.execute("SHOW transaction_isolation") + assert cursor.fetchone()[0] == "read committed" + cursor.execute("SHOW transaction_read_only") + assert cursor.fetchone()[0] == "off" + connection.close() + with pytest.raises(GraphDBError, match="closed connection"): + db.get_node("one") + assert db.get_node("one")["memory"] == "keep" + + +def test_bounded_wait_recovery_close_and_post_close_empty_calls(stores): + db = stores(maxconn=1, connection_wait_timeout=1) + with db._get_connection(), pytest.raises(GraphDBError, match="wait timed out"): + db.get_node("missing") + assert db.get_node("missing") is None + db.close() + db.close() + for action in [ + lambda: db.add_nodes_batch([]), + lambda: db.get_nodes([]), + lambda: db.update_node("one", {}), + ]: + with pytest.raises(GraphDBError, match="closed"): + action() + + +def test_incompatible_dimensions_do_not_change_existing_data(stores): + db = stores() + db.add_node("one", "keep", {"embedding": [1, 0, 0]}) + with pytest.raises(GraphDBError, match="incompatible"): + stores(embedding_dimension=4) + assert db.get_node("one", include_embedding=True)["metadata"]["embedding"] == [1, 0, 0] + + +def test_wrong_credentials_fail_without_password_in_public_error(stores): + import psycopg2 + + with pytest.raises(GraphDBError) as error: + stores(password="Synthetic_Wrong_Password_A7") + assert isinstance(error.value.__cause__, psycopg2.OperationalError) + assert "Invalid username/password,login denied" in str(error.value.__cause__) + assert "Synthetic_Wrong_Password_A7" not in str(error.value) + + +def test_unregistered_schema_is_not_adopted_or_destroyed(stores, database): + admin, _ = database + _query(admin, "CREATE SCHEMA memos_graph") + with pytest.raises(GraphDBError): + stores() + assert _query(admin, "SELECT 1 FROM pg_namespace WHERE nspname='memos_graph'") == [(1,)] + _query(admin, "DROP SCHEMA memos_graph") + + +def test_incomplete_registered_label_is_rejected_with_data_preserved(stores, database): + from psycopg2 import sql + + admin, config = database + _query(admin, "SELECT ag_catalog.create_graph('memos_graph')") + _query( + admin, + "SELECT * FROM ag_catalog.cypher('memos_graph',$$CREATE (n:Memory {id:'legacy'}) RETURN n.id$$) AS (id agtype)", + ) + _query( + admin, + sql.SQL("GRANT USAGE ON SCHEMA memos_graph TO {}").format(sql.Identifier(config.user)), + ) + with pytest.raises(GraphDBError, match="incompatible"): + stores() + assert _query(admin, 'SELECT count(*) FROM memos_graph."Memory"') == [(1,)] + + +def test_missing_extension_fails_and_closes_its_connections(stores, database): + from psycopg2 import sql + + admin, _ = database + name = "memos_it_no_age_" + uuid.uuid4().hex[:12] + _query(admin, sql.SQL("CREATE DATABASE {} DBCOMPATIBILITY 'PG'").format(sql.Identifier(name))) + try: + with pytest.raises(GraphDBError) as error: + stores(db_name=name) + # This openGauss build reports 3F001 for an absent ag_catalog schema. + assert error.value.__cause__.pgcode == "3F001" + assert _query(admin, "SELECT count(*) FROM pg_stat_activity WHERE datname=%s", (name,)) == [ + (0,) + ] + finally: + _query(admin, sql.SQL("DROP DATABASE {}").format(sql.Identifier(name))) + + +def test_missing_create_privilege_fails_without_partial_graph(stores, database): + from psycopg2 import sql + + admin, config = database + _query( + admin, + sql.SQL("REVOKE CREATE ON DATABASE {} FROM {}").format( + sql.Identifier(config.db_name), sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as error: + stores() + assert error.value.__cause__.pgcode == "42501" + assert _query( + admin, "SELECT count(*) FROM ag_catalog.ag_graph WHERE name='memos_graph'" + ) == [(0,)] + finally: + _query( + admin, + sql.SQL("GRANT CREATE ON DATABASE {} TO {}").format( + sql.Identifier(config.db_name), sql.Identifier(config.user) + ), + ) + + +def test_missing_unique_constraint_is_not_silently_repaired(stores, database): + from psycopg2 import sql + + db = stores() + admin, _ = database + db.add_node("one", "keep", {}) + name = _query( + admin, + "SELECT conname FROM pg_constraint WHERE conrelid=%s::regclass AND contype='u'", + ('memos_graph."Memory"',), + )[0][0] + _query( + admin, + sql.SQL('ALTER TABLE memos_graph."Memory" DROP CONSTRAINT {}').format(sql.Identifier(name)), + ) + with pytest.raises(GraphDBError, match="incompatible"): + stores() + assert db.get_node("one")["memory"] == "keep" + + +def test_registered_graph_without_memory_label_can_be_completed(stores, database): + _, config = database + connection = _connect(config) + try: + with connection.cursor() as cursor: + cursor.execute("SELECT 'null'::ag_catalog.agtype") + cursor.execute("SET search_path = ag_catalog, public") + cursor.execute("SELECT ag_catalog.create_graph('memos_graph')") + connection.commit() + finally: + connection.close() + db = stores() + db.add_node("one", "completed", {}) + assert db.get_node("one")["memory"] == "completed" + + +def test_optional_physical_scope_stays_separate_from_explicit_scope(stores): + db = stores(use_multi_db=True, user_name=None) + db.add_node("same", "physical default", {}) + db.add_node("same", "logical B", {}, user_name="B") + assert db.get_node("same")["memory"] == "physical default" + assert db.get_node("same", user_name="B")["memory"] == "logical B" + + +@pytest.mark.parametrize("position", [0, 1, 2]) +def test_preparation_failure_leaves_existing_data_untouched(stores, database, position): + db = stores() + admin, _ = database + db.add_node("keep", "existing", {}) + nodes = [{"id": str(i), "memory": "temporary"} for i in range(3)] + nodes[position]["metadata"] = {"embedding": [1, 0]} + with pytest.raises(ValueError, match="embedding"): + db.add_nodes_batch(nodes) + assert _query(admin, 'SELECT scope,business_id,memory FROM memos_graph."Memory"') == [ + ("A", "keep", "existing") + ] + + +def _relation_fixture(db): + for scope in ("A", "B"): + db.add_nodes_batch( + [ + {"id": "one", "memory": "source", "metadata": {"embedding": [1, 0, 0]}}, + {"id": "two", "memory": "target"}, + {"id": "three", "memory": "unconnected"}, + ], + user_name=scope, + ) + db.add_edge("one", "two", "RELATE_TO", user_name=scope) + db.add_edge("two", "one", "FOLLOWS") + db.add_edge("one", "one", "SELF") + db.add_edge("one", "two", "PARENT") + + +@pytest.mark.parametrize( + "direction,expected", + [ + ( + "OUTGOING", + [ + {"from": "one", "to": "one", "type": "SELF"}, + {"from": "one", "to": "two", "type": "PARENT"}, + {"from": "one", "to": "two", "type": "RELATE_TO"}, + ], + ), + ( + "INCOMING", + [ + {"from": "one", "to": "one", "type": "SELF"}, + {"from": "two", "to": "one", "type": "FOLLOWS"}, + ], + ), + ( + "ANY", + [ + {"from": "one", "to": "one", "type": "SELF"}, + {"from": "one", "to": "two", "type": "PARENT"}, + {"from": "one", "to": "two", "type": "RELATE_TO"}, + {"from": "two", "to": "one", "type": "FOLLOWS"}, + ], + ), + ], +) +def test_age_edge_directions_return_true_endpoints(stores, direction, expected): + db = stores() + _relation_fixture(db) + assert db.get_edges("one", direction=direction) == expected + assert db.get_edges("one", user_name="B") == [{"from": "one", "to": "two", "type": "RELATE_TO"}] + + +def test_age_edge_existence_neighbors_and_exact_deletion(stores): + db = stores() + _relation_fixture(db) + assert db.edge_exists("one", "two", "RELATE_TO") + assert not db.edge_exists("one", "two", "RELATE_TO", "INCOMING") + assert db.edge_exists("one", "two", "FOLLOWS", "INCOMING") + assert db.edge_exists("one", "two", direction="ANY") + assert db.edge_exists("one", "one", "SELF") + assert db.get_neighbors("one", "ANY", "out") == ["one", "two"] + assert db.get_neighbors("one", "FOLLOWS", "in") == ["two"] + assert db.get_neighbors("one", "ANY", "both") == ["one", "two"] + assert db.get_neighbors("three", "ANY") == [] + assert db.get_edges("absent") == [] + assert not db.edge_exists("one", "absent") + db.delete_edge("one", "two", "RELATE_TO") + db.delete_edge("one", "two", "RELATE_TO") + assert db.get_edges("one") == [ + {"from": "one", "to": "one", "type": "SELF"}, + {"from": "one", "to": "two", "type": "PARENT"}, + {"from": "two", "to": "one", "type": "FOLLOWS"}, + ] + assert db.edge_exists("one", "two", "RELATE_TO", user_name="B") + assert db.get_edges("one", "NOT_CREATED") == [] + + +@pytest.mark.parametrize( + "source,target", [("absent", "two"), ("one", "absent"), ("absent", "absent")] +) +def test_age_missing_endpoints_never_create_nodes_or_edges(stores, source, target): + db = stores() + db.add_node("one", "source", {}) + db.add_node("two", "other scope", {}, user_name="B") + db.add_edge(source, target, "RELATE_TO") + db.delete_edge(source, target, "RELATE_TO") + assert db.get_edges("one") == [] + assert db.get_node("absent") is None + + +def test_age_cross_scope_write_and_external_cross_scope_read_are_blocked(stores, database): + db = stores() + admin, _ = database + db.add_node("one", "A source", {}) + db.add_node("two", "B target", {}, user_name="B") + db.add_edge("one", "two", "RELATE_TO") + assert _query(admin, "SELECT count(*) FROM memos_graph._ag_label_edge") == [(0,)] + _query( + admin, + "SELECT * FROM ag_catalog.cypher('memos_graph',$$MATCH (a:Memory {id:'one',user_name:'A'}),(b:Memory {id:'two',user_name:'B'}) CREATE (a)-[:RELATE_TO]->(b)$$) AS (result agtype)", + ) + assert db.get_edges("one") == [] + assert db.get_edges("two", user_name="B") == [] + assert not db.edge_exists("one", "two") + + +def test_age_duplicate_parallel_self_edges_and_concurrency(stores, database): + db = stores() + admin, _ = database + db.add_nodes_batch([{"id": "one", "memory": "source"}, {"id": "two", "memory": "target"}]) + barrier = threading.Barrier(2) + + def write(_): + barrier.wait(timeout=5) + db.add_edge("one", "two", "RELATE_TO") + + with ThreadPoolExecutor(2) as executor: + list(executor.map(write, range(2))) + db.add_edge("one", "two", "RELATE_TO") + db.add_edge("one", "two", "PARENT") + db.add_edge("one", "one", "SELF") + assert _query(admin, "SELECT count(*) FROM memos_graph._ag_label_edge") == [(3,)] + assert db.get_edges("one") == [ + {"from": "one", "to": "one", "type": "SELF"}, + {"from": "one", "to": "two", "type": "PARENT"}, + {"from": "one", "to": "two", "type": "RELATE_TO"}, + ] + + +def test_age_hard_delete_detaches_all_edges_and_preserves_other_scope(stores, database): + db = stores() + admin, _ = database + _relation_fixture(db) + db.delete_node("one") + db.delete_node("one") + db.delete_node("absent") + assert db.get_nodes(["one", "two", "three"]) == [ + {"id": "two", "memory": "target", "metadata": {}}, + {"id": "three", "memory": "unconnected", "metadata": {}}, + ] + assert db.get_node("one", include_embedding=True, user_name="B") == { + "id": "one", + "memory": "source", + "metadata": {"embedding": [1, 0, 0]}, + } + assert db.get_edges("two") == [] + assert db.get_edges("one", user_name="B") == [{"from": "one", "to": "two", "type": "RELATE_TO"}] + assert _query(admin, "SELECT count(*) FROM memos_graph._ag_label_edge") == [(1,)] + assert _query( + admin, + 'SELECT count(*) FROM ONLY memos_graph."Memory" WHERE scope=%s AND business_id=%s', + ("A", "one"), + ) == [(0,)] + + +@pytest.mark.parametrize("failure", ["trigger", "permission"]) +def test_age_detach_failure_rolls_back_edges_node_and_retrieval_columns(stores, database, failure): + from psycopg2 import sql + + db = stores() + admin, config = database + _relation_fixture(db) + if failure == "trigger": + _query( + admin, + "CREATE OR REPLACE FUNCTION public.memos_it_reject_delete() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'synthetic rejected delete'; END $$", + ) + _query( + admin, + 'CREATE TRIGGER memos_it_reject_delete BEFORE DELETE ON memos_graph."Memory" FOR EACH ROW EXECUTE PROCEDURE public.memos_it_reject_delete()', + ) + else: + owner = _query(admin, "SELECT current_user")[0][0] + _query( + admin, + sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format(sql.Identifier(owner)), + ) + _query( + admin, + sql.SQL('GRANT SELECT, UPDATE ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + with pytest.raises(GraphDBError) as error: + db.delete_node("one") + assert error.value.__cause__.pgcode == ("P0001" if failure == "trigger" else "42501") + assert db.get_node("one", include_embedding=True) == { + "id": "one", + "memory": "source", + "metadata": {"embedding": [1, 0, 0]}, + } + assert db.get_edges("one") == [ + {"from": "one", "to": "one", "type": "SELF"}, + {"from": "one", "to": "two", "type": "PARENT"}, + {"from": "one", "to": "two", "type": "RELATE_TO"}, + {"from": "two", "to": "one", "type": "FOLLOWS"}, + ] + assert db.edge_exists("one", "two", "RELATE_TO", user_name="B") + with db._get_connection() as connection, connection.cursor() as cursor: + cursor.execute("SELECT count(*) FROM pg_prepared_statements") + assert cursor.fetchone() == (0,) + if failure == "trigger": + _query(admin, 'DROP TRIGGER memos_it_reject_delete ON memos_graph."Memory"') + else: + _query( + admin, + sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format( + sql.Identifier(config.user) + ), + ) + db.delete_node("one") + assert db.get_node("one") is None + + +@pytest.mark.parametrize("physical", [False, True]) +def test_age_bound_special_ids_and_scopes_round_trip(stores, physical): + scope = None if physical else "中文 $$ ' ; --" + db = stores(use_multi_db=physical, user_name=scope) + source, target = "a\" $$ ' \\ ; --", "中文\n$target" + db.add_nodes_batch([{"id": source, "memory": "source"}, {"id": target, "memory": "target"}]) + db.add_edge(source, target, "Mixed_case_1") + assert db.get_edges(source) == [{"from": source, "to": target, "type": "Mixed_case_1"}] + db.delete_edge(source, target, "Mixed_case_1") + assert not db.edge_exists(source, target) + db.add_edge(source, target, "Mixed_case_1") + db.delete_node(source) + assert db.get_node(source) is None + assert db.get_node(target)["memory"] == "target" + + +@pytest.mark.parametrize("existing_edge", [False, True]) +def test_age_concurrent_delete_and_edge_creation_leave_no_dangling_edges( + stores, database, existing_edge +): + db = stores() + admin, _ = database + db.add_nodes_batch([{"id": "one", "memory": "source"}, {"id": "two", "memory": "target"}]) + if existing_edge: + db.add_edge("one", "two", "RELATE_TO") + barrier = threading.Barrier(2) + + def change(delete): + barrier.wait(timeout=5) + if delete: + db.delete_node("one") + else: + db.add_edge("one", "two", "RELATE_TO") + + with ThreadPoolExecutor(2) as executor: + list(executor.map(change, [True, False])) + assert db.get_node("one") is None + assert db.get_node("two") == {"id": "two", "memory": "target", "metadata": {}} + assert db.get_edges("two") == [] + assert _query(admin, "SELECT count(*) FROM memos_graph._ag_label_edge") == [(0,)] + + +def test_age_maximum_type_length_and_native_graph_after_sql_delete(stores, database): + db = stores() + admin, _ = database + db.add_nodes_batch([{"id": "one", "memory": "source"}, {"id": "two", "memory": "target"}]) + edge_type = "R" * 63 + db.add_edge("one", "two", edge_type) + assert db.get_edges("one", edge_type) == [{"from": "one", "to": "two", "type": edge_type}] + db.delete_node("one") + assert _query( + admin, + "SELECT * FROM ag_catalog.cypher('memos_graph',$$MATCH (n:Memory) RETURN n.id$$) AS (id agtype)", + ) == [('"two"',)] + db.add_node("one", "recreated", {}) + assert db.get_edges("one") == [] + db.add_edge("one", "two", edge_type) + assert db.edge_exists("one", "two", edge_type) + + +def _traversal_fixture(db): + db.add_nodes_batch([{"id": name, "memory": "memory " + name} for name in [*"abcdefg", "z"]]) + db.add_nodes_batch([{"id": name, "memory": "B " + name} for name in "abcdefg"], user_name="B") + for source, target, kind in [ + ("a", "b", "R"), + ("a", "c", "R"), + ("b", "d", "R"), + ("c", "d", "R"), + ("d", "a", "R"), + ("e", "a", "R"), + ("b", "c", "SIDE"), + ("a", "b", "OTHER"), + ("b", "b", "SELF"), + ("d", "f", "R"), + ("f", "g", "R"), + ]: + db.add_edge(source, target, kind) + db.add_edge("a", "g", "R", user_name="B") + + +@pytest.mark.parametrize( + "source,target,depth,expected", + [ + ("a", "d", 1, []), + ("a", "d", 2, ["a", "b", "d"]), + ("a", "g", 3, []), + ("a", "g", 4, ["a", "b", "d", "f", "g"]), + ("a", "a", 1, ["a"]), + ("a", "e", 5, []), + ("a", "z", 5, []), + ("missing", "a", 3, []), + ("a", "missing", 3, []), + ("missing", "missing", 3, []), + ], +) +def test_age_bounded_shortest_path(stores, source, target, depth, expected): + db = stores() + _traversal_fixture(db) + assert db.get_path(source, target, depth) == expected + assert db.get_path("a", "g", 1, user_name="B") == ["a", "g"] + + +def test_age_traversal_reaches_fifth_hop_and_stops_at_configured_bound(stores): + db = stores() + db.add_nodes_batch([{"id": str(i), "memory": str(i)} for i in range(7)]) + for i in range(6): + db.add_edge(str(i), str(i + 1), "R") + db.add_edge("6", "0", "R") + assert db.get_path("0", "5", 5) == ["0", "1", "2", "3", "4", "5"] + assert db.get_path("0", "6", 5) == [] + assert [n["id"] for n in db.get_subgraph("0", 1)["neighbors"]] == ["1", "6"] + with pytest.raises(ValueError, match="depth"): + db.get_path("0", "6", 6) + with pytest.raises(ValueError, match="depth"): + db.get_subgraph("0", 0) + + +def test_age_subgraph_preserves_full_nodes_and_induced_edge_shape(stores, database): + db = stores() + _traversal_fixture(db) + result = db.get_subgraph("a", 1) + assert result == { + "core_node": {"id": "a", "memory": "memory a", "metadata": {}}, + "neighbors": [ + {"id": name, "memory": "memory " + name, "metadata": {}} + for name in ["b", "c", "d", "e"] + ], + "edges": [ + {"source": "a", "target": "b", "type": "OTHER"}, + {"source": "a", "target": "b", "type": "R"}, + {"source": "a", "target": "c", "type": "R"}, + {"source": "b", "target": "b", "type": "SELF"}, + {"source": "b", "target": "c", "type": "SIDE"}, + {"source": "b", "target": "d", "type": "R"}, + {"source": "c", "target": "d", "type": "R"}, + {"source": "d", "target": "a", "type": "R"}, + {"source": "e", "target": "a", "type": "R"}, + ], + } + assert [n["id"] for n in db.get_subgraph("a", 2)["neighbors"]] == ["b", "c", "d", "e", "f"] + assert db.get_subgraph("z") == { + "core_node": {"id": "z", "memory": "memory z", "metadata": {}}, + "neighbors": [], + "edges": [], + } + admin, _ = database + assert _query(admin, 'SELECT count(*) FROM memos_graph."Memory"') == [(15,)] + assert _query(admin, "SELECT count(*) FROM memos_graph._ag_label_edge") == [(12,)] + + +def test_age_subgraph_center_status_and_missing_center(stores): + db = stores() + db.add_node("a", "center", {"status": "archived", "embedding": [1, 0, 0]}) + db.add_node("b", "neighbor", {"status": "deleted", "info": {"nested": [1]}}) + db.add_edge("a", "b", "R") + empty = {"core_node": None, "neighbors": [], "edges": []} + assert db.get_subgraph("missing") == db.get_subgraph("a") == empty + for status in ("archived", "", None): + assert db.get_subgraph("a", center_status=status) == { + "core_node": {"id": "a", "memory": "center", "metadata": {"status": "archived"}}, + "neighbors": [ + { + "id": "b", + "memory": "neighbor", + "metadata": {"status": "deleted", "info": {"nested": [1]}}, + } + ], + "edges": [{"source": "a", "target": "b", "type": "R"}], + } + + +def test_age_traversal_cannot_bridge_scopes_through_external_edges(stores, database): + db = stores() + _traversal_fixture(db) + admin, _ = database + _query( + admin, + "SELECT * FROM ag_catalog.cypher('memos_graph',$$" + "MATCH (a:Memory {id:'a',user_name:'A'}),(b:Memory {id:'b',user_name:'B'})," + "(z:Memory {id:'z',user_name:'A'}) CREATE (a)-[:BRIDGE]->(b)-[:BRIDGE]->(z)" + "$$) AS (result agtype)", + ) + assert db.get_path("a", "z", 5) == [] + assert [n["id"] for n in db.get_subgraph("a", 5)["neighbors"]] == list("bcdefg") + assert all(e["type"] != "BRIDGE" for e in db.get_subgraph("a", 5)["edges"]) + + +@pytest.mark.parametrize("physical", [False, True]) +def test_age_traversal_preserves_special_ids_and_scope(stores, physical): + scope = None if physical else " 范围 $memos$ ' \n" + db = stores(use_multi_db=physical, user_name=scope) + source, target = '"中文 $memos$ ; --\n"', "目标 $$ ' \\\n" + db.add_node(source, "source", {"tags": ["标签 ' $$"]}) + db.add_node(target, "target", {"tags": ["标签 ' $$"]}) + db.add_edge(source, target, "R") + assert db.get_path(source, target, 1) == [source, target] + assert db.get_subgraph(source, 1)["edges"] == [ + {"source": source, "target": target, "type": "R"} + ] + assert [n["id"] for n in db.get_neighbors_by_tag(["标签 ' $$"], [target])] == [source] + + +def test_age_tag_neighbors_rank_visibility_and_optional_embeddings(stores): + db = stores() + for node_id, metadata in [ + ("b", {"tags": ["red", "blue", "blue"]}), + ("a", {"tags": '["red", "blue"]', "embedding": [1, 0, 0]}), + ("c", {"tags": ["red"]}), + ("archive", {"status": "archived", "tags": ["red"]}), + ("reason", {"type": "reasoning", "tags": ["red"]}), + ("legacy", {"node_type": "reasoning", "tags": ["red"]}), + ("working", {"memory_type": "WorkingMemory", "tags": ["red"]}), + ("empty", {"tags": None}), + ]: + db.add_node(node_id, "memory " + node_id, metadata) + db.add_node("foreign", "B", {"tags": ["red", "blue"]}, user_name="B") + assert db.get_neighbors_by_tag(["red", "blue", "red"], [], top_k=2) == [ + {"id": "a", "memory": "memory a", "metadata": {"tags": '["red", "blue"]'}}, + {"id": "b", "memory": "memory b", "metadata": {"tags": ["red", "blue", "blue"]}}, + ] + assert [n["id"] for n in db.get_neighbors_by_tag(["red"], ["a"])] == ["b", "c"] + assert db.get_neighbors_by_tag(["red", "red"], [], min_overlap=2) == [] + assert db.get_neighbors_by_tag(["blue"], [], top_k=1, include_embedding=True)[0]["metadata"][ + "embedding" + ] == [1, 0, 0] + assert db.get_neighbors_by_tag(["red"], [], user_name="B") == [ + {"id": "foreign", "memory": "B", "metadata": {"tags": ["red", "blue"]}} + ] + assert ( + db.get_neighbors_by_tag(["red"], ["a"], include_embedding=True)[0]["metadata"]["embedding"] + is None + ) + + +@pytest.mark.parametrize( + "method,args", + [ + ("get_path", ("a", "b")), + ("get_subgraph", ("a",)), + ("get_neighbors_by_tag", (["red"], [])), + ], +) +def test_age_traversal_select_failure_rolls_back_and_recovers(stores, database, method, args): + from psycopg2 import sql + + db = stores(maxconn=1) + db.add_node("a", "center", {"tags": ["red"]}) + db.add_node("b", "neighbor", {}) + db.add_edge("a", "b", "R") + admin, config = database + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as error: + getattr(db, method)(*args) + assert error.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.get_path("a", "b") == ["a", "b"] + assert db.get_subgraph("a")["neighbors"] == [{"id": "b", "memory": "neighbor", "metadata": {}}] + assert db.get_neighbors_by_tag(["red"], []) == [ + {"id": "a", "memory": "center", "metadata": {"tags": ["red"]}} + ] + + +def test_age_bad_tag_data_fails_without_poisoning_the_connection(stores): + db = stores(maxconn=1) + db.add_node("a", "invalid tags", {"tags": "not-json"}) + with pytest.raises(GraphDBError, match="tags"): + db.get_neighbors_by_tag(["red"], []) + db.update_node("a", {"tags": ["red"]}) + assert db.get_neighbors_by_tag(["red"], []) == [ + {"id": "a", "memory": "invalid tags", "metadata": {"tags": ["red"]}} + ] + + +@pytest.fixture +def filtered_store(stores): + db = stores() + for node_id, metadata in [ + ( + "a", + { + "status": "activated", + "tags": ["x", "y"], + "confidence": 0.8, + "created_at": "2026-01-01T00:00:00Z", + "info": {"project": "星港", "done": True}, + "file_ids": ["f1", "f2"], + }, + ), + ( + "b", + { + "status": "archived", + "tags": ["y"], + "confidence": 0.2, + "created_at": "2026-01-01T08:00:00+08:00", + "file_ids": ["f2"], + }, + ), + ( + "c", + { + "status": "deleted", + "tags": [], + "confidence": 0.5, + "created_at": "2026-01-02T00:00:00", + "info": {"project": None}, + }, + ), + ("d", {}), + ]: + db.add_node( + node_id, + "memo " + node_id, + {"memory_type": "LongTermMemory", "embedding": [1, 0, 0], **metadata}, + ) + db.add_node("a", "B memo", {"memory_type": "LongTermMemory"}, user_name="B") + db.add_node("k", "K memo", {"memory_type": "LongTermMemory"}, user_name="K") + return db + + +@pytest.mark.parametrize( + "condition, expected", + [ + ({}, ["a", "b", "c", "d"]), + ({"status": "activated"}, ["a"]), + ({"confidence": {"gt": 0.5}}, ["a"]), + ({"confidence": {"gte": 0.5}}, ["a", "c"]), + ({"confidence": {"lt": 0.5}}, ["b"]), + ({"confidence": {"lte": 0.5}}, ["b", "c"]), + ({"status": {"in": ["archived", "deleted"]}}, ["b", "c"]), + ({"status": {"in": []}}, []), + ({"tags": "x"}, ["a"]), + ({"tags": ["x", "y"]}, ["a"]), + ({"tags": []}, ["a", "b", "c"]), + ({"tags": {"in": ["z", "y"]}}, ["a", "b"]), + ({"tags": {"contains": "x"}}, ["a"]), + ({"tags": {"contains": []}}, []), + ({"file_ids": {"contains": ["f1", "f2"]}}, ["a", "b"]), + ({"project": "星港"}, ["a"]), + ({"info.project": None}, ["c"]), + ({"info.done": True}, ["a"]), + ({"info.done": 1}, []), + ({"info": {"=": {"project": "星港", "done": True}}}, ["a"]), + ({"memory": {"like": "emo a"}}, ["a"]), + ({"memory": {"nolike": "a"}}, ["b", "c", "d"]), + ({"memory": {"starts_with": "memo"}}, ["a", "b", "c", "d"]), + ({"memory": {"ends_with": "b"}}, ["b"]), + ({"created_at": "2026-01-01T00:00:00Z"}, ["a", "b"]), + ({"created_at": {"gte": "2026-01-01", "lte": "2026-01-01T00:00:00Z"}}, ["a", "b"]), + ({"created_at": {"gt": "2026-01-01T08:00:00+08:00"}}, ["c"]), + ({"created_at": {"lt": "2026-01-02T00:00:00Z"}}, ["a", "b"]), + ( + {"or": [{"status": "deleted"}, {"and": [{"tags": "x"}, {"confidence": {"gte": 0.8}}]}]}, + ["a", "c"], + ), + ({"or": [{"id": "a"}, {"id": "b"}], "status": "archived"}, ["b"]), + ({"user_name": "B"}, []), + ], +) +def test_filter_business_results_real_sql(filtered_store, condition, expected): + assert filtered_store.get_by_metadata([], filter=condition) == expected + + +@pytest.mark.parametrize( + "field, op, value, expected", + [ + ("status", "=", "deleted", ["c"]), + ("status", "in", ["archived", "deleted"], ["b", "c"]), + ("tags", "contains", ["x", "z"], ["a"]), + ("confidence", ">", 0.5, ["a"]), + ("confidence", ">=", 0.5, ["a", "c"]), + ("confidence", "<", 0.5, ["b"]), + ("confidence", "<=", 0.5, ["b", "c"]), + ("memory", "like", "emo a", ["a"]), + ("memory", "starts_with", "memo", ["a", "b", "c", "d"]), + ("memory", "ends_with", "b", ["b"]), + ("created_at", "=", "2026-01-01T08:00:00+08:00", ["a", "b"]), + ], +) +def test_filter_metadata_results_real_sql(filtered_store, field, op, value, expected): + assert filtered_store.get_by_metadata([{"field": field, "op": op, "value": value}]) == expected + + +def test_filter_scopes_status_projection_and_literal_strings(filtered_store): + db = filtered_store + assert db.get_by_metadata([], status="activated") == ["a", "d"] + assert db.get_by_metadata([], user_name="B") == ["a"] + assert db.get_by_metadata([], knowledgebase_ids=["B", "K", "K"]) == [ + "a", + "b", + "c", + "d", + "a", + "k", + ] + assert db.get_by_metadata([], knowledgebase_ids=[]) == ["a", "b", "c", "d"] + assert db.get_by_metadata( + [{"field": "tags", "op": "contains", "value": "y"}], filter={"status": "archived"} + ) == ["b"] + rows = db.get_all_memory_items("LongTermMemory", status="activated", include_embedding=True) + assert [row["id"] for row in rows] == ["a", "d"] + assert all(row["metadata"]["embedding"] == [1, 0, 0] for row in rows) + rows = db.get_all_memory_items("LongTermMemory", knowledgebase_ids=["B"]) + assert [(row["metadata"]["user_name"], row["id"]) for row in rows] == [ + ("A", "a"), + ("A", "b"), + ("A", "c"), + ("A", "d"), + ("B", "a"), + ] + literal = "中文🚀 %_\\ ' $memos$ ; --\n" + db.add_node("special", literal, {"tags": '["x"]', "status": None}) + assert db.get_by_metadata([], filter={"memory": {"like": literal}}) == ["special"] + assert db.get_by_metadata([], filter={"memory": {"like": "%"}}) == ["special"] + assert db.get_by_metadata([], filter={"tags": {"contains": "x"}}) == ["a"] + assert db.get_by_metadata([], status="activated") == ["a", "d"] + + +def test_filter_all_types_and_no_hidden_limit(stores): + db = stores() + types = [ + "WorkingMemory", + "LongTermMemory", + "UserMemory", + "OuterMemory", + "ToolSchemaMemory", + "ToolTrajectoryMemory", + "RawFileMemory", + "SkillMemory", + "PreferenceMemory", + "Context", + "InsightMemory", + "DreamDiary", + "ProfileMemory", + ] + for kind in types: + db.add_node(kind, kind, {"memory_type": kind}) + assert db.get_all_memory_items(kind) == [ + {"id": kind, "memory": kind, "metadata": {"memory_type": kind}} + ] + db.add_nodes_batch( + [ + {"id": f"n{i:03}", "memory": "", "metadata": {"memory_type": "LongTermMemory"}} + for i in range(105) + ] + ) + assert len(db.get_all_memory_items("LongTermMemory")) == 106 + + +def test_filter_soft_delete_and_restore_preserve_identity_graph_and_vector( + filtered_store, database +): + db = filtered_store + db.add_edge("a", "d", "R") + admin, _ = database + before = _query( + admin, "SELECT id FROM memos_graph.\"Memory\" WHERE scope='A' AND business_id='a'" + ) + db.update_node("a", {"status": "deleted", "evolve_to": ["d"]}) + assert db.get_by_metadata([], status="activated") == ["d"] + assert db.get_by_metadata([], status="deleted") == ["a", "c"] + node = db.get_node("a", include_embedding=True) + assert node["memory"] == "memo a" and node["metadata"]["embedding"] == [1, 0, 0] + assert node["metadata"]["evolve_to"] == ["d"] + assert db.get_neighbors_by_tag(["x"], []) == [] + assert db.edge_exists("a", "d", "R") + db.update_node("a", {"status": "activated"}) + assert db.get_by_metadata([], status="activated") == ["a", "d"] + assert [node["id"] for node in db.get_neighbors_by_tag(["x"], [])] == ["a"] + assert ( + _query(admin, "SELECT id FROM memos_graph.\"Memory\" WHERE scope='A' AND business_id='a'") + == before + ) + assert db.get_node("a", user_name="B")["memory"] == "B memo" + + +@pytest.mark.parametrize( + "selector, expected", + [ + ({"memory_ids": ["a", "a", "missing"]}, ["b", "c", "d"]), + ({"file_ids": ["f2"]}, ["c", "d"]), + ({"filter": {"status": {"in": ["archived", "deleted"]}}}, ["a", "d"]), + ], +) +def test_filter_hard_delete_counts_edges_and_scope(filtered_store, database, selector, expected): + db = filtered_store + db.add_edge("a", "b", "R") + db.add_edge("b", "c", "R") + db.add_edge("a", "a", "SELF") + assert db.delete_node_by_prams(**selector) == 4 - len(expected) + assert db.get_by_metadata([]) == expected + assert db.get_node("a", user_name="B")["memory"] == "B memo" + assert all( + edge["from"] in expected and edge["to"] in expected + for node_id in "abcd" + for edge in db.get_edges(node_id) + ) + admin, _ = database + expected_edges = 1 if expected == ["b", "c", "d"] else 0 + assert _query(admin, 'SELECT count(*) FROM memos_graph."R"') == [(expected_edges,)] + assert _query(admin, 'SELECT count(*) FROM memos_graph."SELF"') == [(int("a" in expected),)] + assert db.delete_node_by_prams(**selector) == 0 + + +def test_filter_delete_explicit_scopes_and_empty_selection(filtered_store): + db = filtered_store + assert db.delete_node_by_prams(memory_ids=["a"], writable_cube_ids=[]) == 0 + assert db.delete_node_by_prams(filter={}) == 0 + assert db.delete_node_by_prams(memory_ids=["a"], writable_cube_ids=["B", "B", "K"]) == 1 + assert db.get_node("a") is not None + assert db.get_node("k", user_name="K") is not None + assert ( + db.delete_node_by_prams( + filter={"memory_type": "LongTermMemory"}, writable_cube_ids=["A", "K"] + ) + == 5 + ) + assert db.get_by_metadata([], knowledgebase_ids=["B", "K"]) == [] + + +def test_filter_delete_batch_rollback_and_connection_recovery(filtered_store, database): + db = filtered_store + admin, _ = database + db.add_edge("a", "b", "R") + _query( + admin, + "CREATE FUNCTION public.memos_filter_reject() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.business_id='b' THEN RAISE EXCEPTION 'synthetic batch reject'; END IF; RETURN OLD; END $$", + ) + _query( + admin, + 'CREATE TRIGGER memos_filter_reject BEFORE DELETE ON memos_graph."Memory" FOR EACH ROW EXECUTE PROCEDURE public.memos_filter_reject()', + ) + try: + with pytest.raises(GraphDBError) as error: + db.delete_node_by_prams(file_ids=["f2"]) + assert error.value.__cause__.pgcode == "P0001" + assert db.get_by_metadata([]) == ["a", "b", "c", "d"] + assert db.edge_exists("a", "b", "R") + finally: + _query(admin, 'DROP TRIGGER memos_filter_reject ON memos_graph."Memory"') + _query(admin, "DROP FUNCTION public.memos_filter_reject()") + assert db.delete_node_by_prams(file_ids=["f2"]) == 2 + + +def test_filter_invalid_stored_time_is_not_empty_success(filtered_store): + db = filtered_store + db.add_node("bad", "", {"created_at": "broken"}) + with pytest.raises(GraphDBError): + db.get_by_metadata([], filter={"created_at": {"gte": "2026-01-01"}}) + assert db.get_by_metadata([], filter={"id": "bad"}) == ["bad"] + + +def test_filter_age_fraction_round_trip_preserves_strings(stores): + db = stores() + metadata = { + "confidence": 0.8, + "negative": -0.2, + "info": {"nested": [0.125, -0.125]}, + "background": 'literal .8, -.2 and "quoted: .8"; slash \\ .7', + } + db.add_node("fraction", "text: .8 and -.2", metadata) + assert db.get_node("fraction") == { + "id": "fraction", + "memory": "text: .8 and -.2", + "metadata": metadata, + } + db.update_node("fraction", {"status": "deleted"}) + assert db.get_node("fraction")["metadata"] == {**metadata, "status": "deleted"} + + +def test_filter_nonmatching_types_null_and_structured_values(stores): + db = stores() + db.add_node("a", "", {"info": {"value": None}, "confidence": "0.8", "tags": "x"}) + db.add_node("b", "", {"info": "serialized or invalid", "confidence": True, "tags": {"x": 1}}) + db.add_node("c", "", {"info": {"value": 1.0}, "confidence": -0.2, "tags": ["x"]}) + db.add_node("d", "", {"info": {"value": "1"}, "confidence": None, "tags": None}) + assert db.get_by_metadata([], filter={"value": {"in": [None, 1]}}) == ["a", "c"] + assert db.get_by_metadata([], filter={"value": {"in": []}}) == [] + assert db.get_by_metadata([], filter={"confidence": {"lt": 0}}) == ["c"] + assert db.get_by_metadata([], filter={"tags": {"contains": "x"}}) == ["c"] + assert db.get_by_metadata([], filter={"confidence": {"like": "0.8"}}) == ["a"] + assert db.get_by_metadata([], filter={"confidence": {"nolike": "no"}}) == ["a"] + assert db.get_all_memory_items("missing", filter={"id": "a"}) == [] + + +def test_filter_physical_scope_and_quoted_explicit_scope(stores): + db = stores(use_multi_db=True, user_name=None) + db.add_node("id", "physical", {"memory_type": "Context"}) + quoted = " scope ' $memos$ -- 中文 " + db.add_node("id", "quoted", {"memory_type": "Context"}, user_name=quoted) + assert db.get_by_metadata([]) == ["id"] + assert db.get_all_memory_items("Context") == [ + {"id": "id", "memory": "physical", "metadata": {"memory_type": "Context"}} + ] + assert db.get_by_metadata([], user_name=quoted) == ["id"] + assert db.delete_node_by_prams(memory_ids=["id"]) == 1 + assert db.get_by_metadata([]) == [] + assert db.get_node("id", user_name=quoted)["memory"] == "quoted" + assert db.delete_node_by_prams(memory_ids=["id"], writable_cube_ids=[quoted]) == 1 + + +@pytest.mark.parametrize("operation", ["metadata", "list", "delete"]) +def test_filter_permission_failure_restores_connection(filtered_store, database, operation): + from psycopg2 import sql + + db = filtered_store + admin, config = database + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as error: + if operation == "metadata": + db.get_by_metadata([]) + elif operation == "list": + db.get_all_memory_items("LongTermMemory") + else: + db.delete_node_by_prams(memory_ids=["a"]) + assert error.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.get_by_metadata([]) == ["a", "b", "c", "d"] + + +def test_filter_delete_and_edge_creation_are_serialized(filtered_store, database): + db = filtered_store + db.add_edge("a", "b", "R") + barrier = threading.Barrier(2) + + def remove(): + barrier.wait(timeout=5) + return db.delete_node_by_prams(file_ids=["f1"]) + + def connect(): + barrier.wait(timeout=5) + db.add_edge("a", "c", "R") + + with ThreadPoolExecutor(max_workers=2) as executor: + removed, connected = executor.submit(remove), executor.submit(connect) + assert removed.result(timeout=10) == 1 + connected.result(timeout=10) + assert db.get_by_metadata([]) == ["b", "c", "d"] + assert _query(database[0], 'SELECT count(*) FROM memos_graph."R"') == [(0,)] + + +def _vector_fixture(db): + """Independent vectors: cosine scores for e1 are 1, .9, .5 and 0.""" + for node_id, vector in [ + ("same", [1, 0, 0]), + ("near", [0.8, 0.6, 0]), + ("orthogonal", [0, 1, 0]), + ("opposite", [-1, 0, 0]), + ]: + db.add_node( + node_id, + node_id, + { + "memory_type": "LongTermMemory", + "status": "activated", + "embedding": vector, + "info": {"project": "星港"}, + "session_id": "s1", + }, + ) + for node_id, status in [ + ("archived", "archived"), + ("deleted", "deleted"), + ("null-status", None), + ]: + db.add_node( + node_id, + node_id, + {"memory_type": "LongTermMemory", "status": status, "embedding": [1, 0, 0]}, + ) + db.add_node("no-vector", "no-vector", {"memory_type": "LongTermMemory"}) + db.add_node("zero", "zero", {"memory_type": "LongTermMemory", "embedding": [0, 0, 0]}) + db.add_node( + "same", "scope B", {"memory_type": "LongTermMemory", "embedding": [1, 0, 0]}, user_name="B" + ) + + +def test_datavec_scores_scopes_thresholds_and_projection(stores): + db = stores() + _vector_fixture(db) + hits = db.search_by_embedding( + [1, 0, 0], + top_k=10, + scope="LongTermMemory", + return_fields=["memory", "user_name", "info"], + include_embedding=True, + ) + assert [hit["id"] for hit in hits] == ["same", "near", "orthogonal", "opposite"] + assert [hit["score"] for hit in hits] == pytest.approx([1, 0.9, 0.5, 0], abs=1e-6) + assert all(hit["user_name"] == "A" and hit["info"] == {"project": "星港"} for hit in hits) + assert hits[0]["embedding"] == [1, 0, 0] + assert [hit["id"] for hit in db.search_by_embedding([1, 0, 0], threshold=0.5, top_k=10)] == [ + "same", + "near", + "orthogonal", + ] + assert [ + hit["id"] for hit in db.search_by_embedding([1, 0, 0], threshold=0.50001, top_k=10) + ] == ["same", "near"] + assert db.search_by_embedding([1, 0, 0], threshold=1, top_k=1) == [{"id": "same", "score": 1}] + assert [hit["id"] for hit in db.search_by_embedding([1, 0, 0], status="archived")] == [ + "archived" + ] + assert db.search_by_embedding([1, 0, 0], scope="UserMemory") == [] + assert db.search_by_embedding([1, 0, 0], top_k=0) == [] + + +def test_datavec_filters_and_same_id_multi_scope_projection(stores): + db = stores() + _vector_fixture(db) + assert db.search_by_embedding([1, 0, 0], filter={"project": "missing"}) == [] + assert db.search_by_embedding([1, 0, 0], search_filter={"session_id": "s2"}) == [] + assert [ + hit["id"] + for hit in db.search_by_embedding( + [1, 0, 0], filter={"project": "星港"}, search_filter={"session_id": "s1"}, top_k=2 + ) + ] == ["same", "near"] + hits = db.search_by_embedding( + [1, 0, 0], + knowledgebase_ids=["B", "B"], + threshold=1, + top_k=10, + return_fields=["user_name", "memory", "unknown"], + ) + assert hits == [ + {"id": "same", "score": 1, "user_name": "A", "memory": "same", "unknown": None}, + {"id": "same", "score": 1, "user_name": "B", "memory": "scope B", "unknown": None}, + ] + assert db.search_by_embedding([1, 0, 0], user_name="A", cube_name="B", top_k=1) == [ + {"id": "same", "score": 1} + ] + assert db.get_node("same", user_name="A", cube_name="B")["memory"] == "scope B" + + +def test_datavec_updates_soft_delete_restore_and_completion_deletion(stores): + db = stores() + _vector_fixture(db) + db.update_node("same", {"status": "deleted"}) + assert db.search_by_embedding([1, 0, 0], top_k=1)[0]["id"] == "near" + db.update_node("same", {"status": "activated", "embedding": [0, 1, 0]}) + assert db.search_by_embedding([0, 1, 0], top_k=10)[0]["score"] == 1 + db.update_node("same", {"embedding": None}) + assert "same" not in [hit["id"] for hit in db.search_by_embedding([1, 0, 0], top_k=10)] + hits = db.search_by_embedding([1, 0, 0], top_k=1) + db.add_node("near", "other scope", {"embedding": [1, 0, 0]}, user_name="B") + db.delete_node("near") + assert db.get_nodes([hit["id"] for hit in hits], cube_name="A") == [] + assert db.get_node("near", user_name="B")["memory"] == "other scope" + + +@pytest.mark.parametrize("dimension", [1, 1024, 2000, 2001, 3072, 16000]) +def test_datavec_real_supported_dimensions_and_index_policy(stores, database, dimension): + db = stores(embedding_dimension=dimension) + vector = [1] + [0] * (dimension - 1) + db.add_node("one", "fixture", {"embedding": vector}) + assert db.search_by_embedding(vector, top_k=1) == [{"id": "one", "score": 1}] + admin, _ = database + count = _query( + admin, + "SELECT count(*) FROM pg_indexes WHERE schemaname='memos_graph' AND indexname='memory_embedding_hnsw'", + )[0][0] + assert count == (1 if dimension <= 2000 else 0) + + +def test_datavec_index_repair_preserves_nodes_and_rejects_incompatible_index(stores, database): + db = stores() + db.add_node("one", "fixture", {"embedding": [1, 0, 0]}) + admin, _ = database + graphid = _query(admin, 'SELECT id::text FROM memos_graph."Memory"')[0][0] + _query(admin, "DROP INDEX memos_graph.memory_embedding_hnsw") + repaired = stores() + assert repaired.search_by_embedding([1, 0, 0], top_k=1) == [{"id": "one", "score": 1}] + assert _query(admin, 'SELECT id::text FROM memos_graph."Memory"') == [(graphid,)] + _query(admin, "DROP INDEX memos_graph.memory_embedding_hnsw") + _query(admin, 'CREATE INDEX memory_embedding_hnsw ON memos_graph."Memory" (memory)') + with pytest.raises(GraphDBError, match="index"): + stores() + assert db.get_node("one")["memory"] == "fixture" + + +def test_datavec_hnsw_plan_and_strong_scope_filter(stores, database): + db = stores() + db.add_nodes_batch( + [ + {"id": f"b{i}", "memory": "other", "metadata": {"embedding": [1, 0, 0]}} + for i in range(150) + ], + user_name="B", + ) + db.add_nodes_batch( + [ + {"id": f"a{i}", "memory": "ours", "metadata": {"embedding": [0.8, 0.6, 0]}} + for i in range(5) + ] + ) + with db._get_connection() as connection, connection.cursor() as cursor: + cursor.execute( + 'EXPLAIN SELECT /*+ indexscan(m memory_embedding_hnsw) */ business_id FROM ONLY memos_graph."Memory" m WHERE scope=%s AND vector_norm(embedding)>0 ORDER BY embedding <=> %s::vector LIMIT 3', + ("A", "[1,0,0]"), + ) + plan = "\n".join(row[0] for row in cursor.fetchall()) + assert "Ann Index Scan" in plan and "memory_embedding_hnsw" in plan + hits = db.search_by_embedding([1, 0, 0], top_k=5, return_fields=["user_name"]) + assert [hit["id"] for hit in hits] == ["a0", "a1", "a2", "a3", "a4"] + assert all(hit["user_name"] == "A" for hit in hits) + + +def test_datavec_real_retriever_completion_respects_cube_alias(stores, monkeypatch): + from memos.memories.textual.tree_text_memory.retrieve.recall import GraphMemoryRetriever + + monkeypatch.delenv("MEMOS_POLARDB_LIGHTWEIGHT_SEARCH_ENABLED", raising=False) + db = stores() + first = "00000000-0000-4000-8000-000000000001" + second = "00000000-0000-4000-8000-000000000002" + db.add_node(first, "same", {"memory_type": "LongTermMemory", "embedding": [1, 0, 0]}) + db.add_node(second, "near", {"memory_type": "LongTermMemory", "embedding": [0.8, 0.6, 0]}) + db.add_node( + first, "scope B", {"memory_type": "LongTermMemory", "embedding": [1, 0, 0]}, user_name="B" + ) + retriever = GraphMemoryRetriever(db, None, include_embedding=True) + results = retriever._vector_recall([[1, 0, 0]], "LongTermMemory", top_k=2, user_name="A") + assert [item.id for item in results] == [first, second] + assert [item.metadata.relativity for item in results] == pytest.approx([1, 0.9], abs=1e-6) + assert results[0].metadata.embedding == [1, 0, 0] + external = retriever._vector_recall( + [[1, 0, 0]], "LongTermMemory", top_k=1, user_name="A", cube_name="B" + ) + assert [item.memory for item in external] == ["scope B"] + + +def test_datavec_query_permission_failure_and_pool_recovery(stores, database): + from psycopg2 import sql + + db = stores() + db.add_node("one", "fixture", {"embedding": [1, 0, 0]}) + admin, config = database + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as caught: + db.search_by_embedding([1, 0, 0]) + assert caught.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.search_by_embedding([1, 0, 0], top_k=1) == [{"id": "one", "score": 1}] + + +@pytest.mark.parametrize("failure", ["cancel", "timeout"]) +def test_datavec_cancel_timeout_and_subsequent_query_recovery(stores, database, failure): + from psycopg2 import sql + + admin, config = database + if failure == "timeout": + _query( + admin, + sql.SQL("ALTER ROLE {} SET statement_timeout='1s'").format(sql.Identifier(config.user)), + ) + locker = None + try: + db = stores() + db.add_node("one", "fixture", {"embedding": [1, 0, 0]}) + locker = _connect(config) + with locker.cursor() as cursor: + cursor.execute('LOCK TABLE memos_graph."Memory" IN ACCESS EXCLUSIVE MODE') + with ThreadPoolExecutor(1) as executor: + future = executor.submit(db.search_by_embedding, [1, 0, 0]) + try: + if failure == "cancel": + deadline = time.monotonic() + 5 + while True: + active = _query( + admin, + "SELECT pid FROM pg_stat_activity WHERE datname=%s AND application_name='memos-opengauss' AND state='active' AND query LIKE 'SELECT business_id%%'", + (config.db_name,), + ) + if active or time.monotonic() >= deadline: + break + time.sleep(0.01) + assert len(active) == 1 + assert _query(admin, "SELECT pg_cancel_backend(%s)", (active[0][0],)) == [ + (True,) + ] + with pytest.raises(GraphDBError) as caught: + future.result(timeout=5) + assert caught.value.__cause__.pgcode == "57014" + finally: + locker.rollback() + assert db.search_by_embedding([1, 0, 0], top_k=1) == [{"id": "one", "score": 1}] + finally: + if locker is not None: + locker.close() + if failure == "timeout": + _query( + admin, + sql.SQL("ALTER ROLE {} RESET statement_timeout").format( + sql.Identifier(config.user) + ), + ) + + +def test_datavec_index_creation_permission_failure_preserves_data(stores, database): + from psycopg2 import sql + + db = stores() + db.add_node("one", "fixture", {"embedding": [1, 0, 0]}) + admin, config = database + _query(admin, "DROP INDEX memos_graph.memory_embedding_hnsw") + owner = _query(admin, "SELECT current_user")[0][0] + _query( + admin, sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format(sql.Identifier(owner)) + ) + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format(sql.Identifier(config.user)), + ) + try: + with pytest.raises(GraphDBError) as caught: + stores() + assert caught.value.__cause__.pgcode == "42501" + assert db.get_node("one")["memory"] == "fixture" + finally: + _query( + admin, + sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format( + sql.Identifier(config.user) + ), + ) + repaired = stores() + assert repaired.search_by_embedding([1, 0, 0], top_k=1) == [{"id": "one", "score": 1}] + + +def test_datavec_failed_write_keeps_index_results_and_extreme_queries(stores): + db = stores() + db.add_node("one", "fixture", {"embedding": [1, 0, 0]}) + with ( + pytest.raises(GraphDBError), + db._get_connection() as connection, + connection.cursor() as cursor, + ): + cursor.execute( + "UPDATE memos_graph.\"Memory\" SET embedding='[0,1,0]'::vector WHERE scope=%s", ("A",) + ) + cursor.execute("SELECT 1/0") + for scale in (1e-300, 1e300): + assert db.search_by_embedding([scale, 0, 0], top_k=1) == [{"id": "one", "score": 1}] + + +def test_datavec_invalid_inputs_preserve_real_rows_and_physical_scope(stores, database): + db = stores(use_multi_db=True, user_name=None) + db.add_node("one", "fixture", {"embedding": [1, 0, 0]}) + db.add_node("one", "other scope", {"embedding": [0, 1, 0]}, user_name="B") + admin, _ = database + statement = 'SELECT id::text, scope, business_id, properties, embedding::text FROM ONLY memos_graph."Memory" ORDER BY scope' + before = _query(admin, statement) + for vector in ( + [1, 0], + [1, 0, 0, 0], + [0, 0, 0], + [True, 0, 0], + [float("nan"), 0, 0], + [float("inf"), 0, 0], + ["1", 0, 0], + ): + with pytest.raises(ValueError): + db.search_by_embedding(vector) + for kwargs in ({"top_k": -1}, {"top_k": True}, {"return_fields": ["id; DROP TABLE x"]}): + with pytest.raises(ValueError): + db.search_by_embedding([1, 0, 0], **kwargs) + assert _query(admin, statement) == before + assert db.search_by_embedding([1, 0, 0], top_k=10, return_fields=["user_name"]) == [ + {"id": "one", "score": 1, "user_name": ""} + ] + + +def _bm25_fixture(db): + for node_id, scope, memory in [ + ("both", "A", "星港 回归"), + ("star", "A", "星港"), + ("regression", "A", "回归"), + ("none", "A", "月球"), + ("both", "B", "星港 星港 回归"), + ]: + db.add_node( + node_id, + memory, + { + "memory_type": "LongTermMemory", + "embedding": [1, 0, 0], + "info": {"project": "星港"}, + "session_id": "s1", + }, + user_name=scope, + ) + + +def test_bm25_real_scores_match_independent_formula_and_projection(stores, database): + db = stores() + _bm25_fixture(db) + idf = math.log(1 + (5 - 3 + 0.5) / (3 + 0.5)) + one_term = idf * 2.2 / (1 + 1.2 * (0.25 + 0.75 / 1.6)) + both_terms = 2 * idf * 2.2 / (1 + 1.2 * (0.25 + 0.75 * 2 / 1.6)) + hits = db.search_by_fulltext( + ["星港", "回归"], + return_fields=["memory", "info", "unknown", "user_name"], + include_embedding=True, + ) + assert [hit["id"] for hit in hits] == ["both", "regression", "star"] + assert [hit["score"] for hit in hits] == pytest.approx( + [both_terms, one_term, one_term], abs=1e-6 + ) + assert hits[0]["memory"] == "星港 回归" and hits[0]["info"] == {"project": "星港"} + assert all( + hit["embedding"] == [1, 0, 0] and hit["unknown"] is None and hit["user_name"] == "A" + for hit in hits + ) + assert [h["id"] for h in db.search_by_fulltext(["星港", "回归"], threshold=0.8)] == ["both"] + rare = db.search_by_fulltext(["月球"], threshold=1) + assert len(rare) == 1 and rare[0]["score"] > 1 + admin, _ = database + plan = _query( + admin, + 'EXPLAIN SELECT business_id FROM ONLY memos_graph."Memory" WHERE scope=%s ORDER BY memory <&> %s DESC LIMIT 5', + ("A", "星港"), + ) + assert "memory_fulltext_bm25" in "\n".join(row[0] for row in plan) + + +@pytest.mark.parametrize( + "words,expected", + [ + (["星港"], ["star", "both"]), + (["'星港'", '"回归"', "星港"], ["both", "regression", "star"]), + (["星港 & 回归"], ["both", "regression", "star"]), + (["星港 | 回归"], ["both", "regression", "star"]), + (["unmatch"], []), + (["!!!"], []), + ([], []), + ([" ", "''"], []), + (["星港'; DROP TABLE absent; --"], ["star", "both"]), + ], +) +def test_bm25_words_are_native_or_text_not_sql_or_tsquery(stores, words, expected): + db = stores() + _bm25_fixture(db) + assert [h["id"] for h in db.search_by_fulltext(words)] == expected + assert db.get_node("both")["memory"] == "星港 回归" + + +@pytest.mark.parametrize("top_k", [1, 5, 200, 201, 210]) +def test_bm25_more_than_200_candidates_survive_scope_filter(stores, top_k): + db = stores() + db.add_nodes_batch( + [{"id": f"b{i}", "memory": "星港 星港 星港"} for i in range(220)], user_name="B" + ) + db.add_nodes_batch( + [ + {"id": f"a{i:03}", "memory": "星港", "metadata": {"info": {"project": "ours"}}} + for i in range(215) + ] + ) + hits = db.search_by_fulltext( + ["星港"], top_k=top_k, filter={"project": "ours"}, return_fields=["user_name"] + ) + assert len(hits) == top_k and len({h["id"] for h in hits}) == top_k + assert all(h["user_name"] == "A" and h["id"].startswith("a") for h in hits) + assert [h["id"] for h in hits] == sorted(h["id"] for h in hits) + + +def test_bm25_filters_status_cube_alias_and_same_id_scopes(stores): + db = stores() + _bm25_fixture(db) + db.update_node("star", {"status": "archived"}) + assert [h["id"] for h in db.search_by_fulltext(["星港"])] == ["both"] + assert [h["id"] for h in db.search_by_fulltext(["星港"], status="archived")] == ["star"] + assert db.search_by_fulltext(["星港"], scope="UserMemory") == [] + assert db.search_by_fulltext(["星港"], filter={"project": "missing"}) == [] + assert db.search_by_fulltext(["星港"], search_filter={"session_id": "missing"}) == [] + hits = db.search_by_fulltext( + ["星港"], + scope="LongTermMemory", + filter={"project": "星港"}, + search_filter={"session_id": "s1"}, + knowledgebase_ids=["B", "B"], + return_fields=["user_name"], + ) + assert {(h["id"], h["user_name"]) for h in hits} == {("both", "A"), ("both", "B")} + assert ( + db.search_by_fulltext(["星港"], user_name="A", cube_name="B", return_fields=["user_name"])[ + 0 + ]["user_name"] + == "B" + ) + db.update_node("star", {"status": "activated"}) + assert {h["id"] for h in db.search_by_fulltext(["星港"])} == {"star", "both"} + + +def test_bm25_update_delete_and_rollback_recovery_preserve_graph(stores, database): + db = stores() + db.add_node("one", "星港", {"embedding": [1, 0, 0]}) + db.add_node("two", "月球", {}) + db.add_edge("one", "two", "RELATE_TO") + admin, _ = database + graphid = _query( + admin, 'SELECT id::text FROM memos_graph."Memory" WHERE business_id=%s', ("one",) + ) + db.update_node("one", {"memory": "回归"}) + assert db.search_by_fulltext(["星港"]) == [] + assert [h["id"] for h in db.search_by_fulltext(["回归"])] == ["one"] + with pytest.raises(GraphDBError), db._get_connection() as conn, conn.cursor() as cursor: + cursor.execute( + 'UPDATE memos_graph."Memory" SET memory=%s WHERE business_id=%s', + ("rollbackmarker", "one"), + ) + cursor.execute("SELECT 1/0") + assert [h["id"] for h in db.search_by_fulltext(["回归"])] == ["one"] + assert db.search_by_fulltext(["rollbackmarker"]) == [] + assert db.edge_exists("one", "two", "RELATE_TO") + assert ( + _query(admin, 'SELECT id::text FROM memos_graph."Memory" WHERE business_id=%s', ("one",)) + == graphid + ) + db.delete_node("one") + assert db.search_by_fulltext(["回归"]) == [] + + +def test_bm25_missing_index_reinitialization_and_incompatible_index(stores, database): + db = stores() + db.add_node("one", "星港", {}) + admin, _ = database + _query(admin, "DROP INDEX memos_graph.memory_fulltext_bm25") + with pytest.raises(GraphDBError): + db.search_by_fulltext(["星港"]) + repaired = stores() + assert [h["id"] for h in repaired.search_by_fulltext(["星港"])] == ["one"] + _query(admin, "DROP INDEX memos_graph.memory_fulltext_bm25") + _query(admin, 'CREATE INDEX memory_fulltext_bm25 ON memos_graph."Memory" (memory)') + with pytest.raises(GraphDBError): + stores() + assert db.get_node("one")["memory"] == "星港" + + +def test_bm25_repair_permission_failure_is_explicit_and_recoverable(stores, database): + from psycopg2 import sql + + db = stores() + db.add_node("one", "星港", {}) + with pytest.raises(GraphDBError), db._get_connection() as conn, conn.cursor() as cursor: + cursor.execute('UPDATE memos_graph."Memory" SET memory=%s', ("rollbackmarker",)) + cursor.execute("SELECT 1/0") + admin, config = database + owner = _query(admin, "SELECT current_user")[0][0] + _query( + admin, sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format(sql.Identifier(owner)) + ) + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format(sql.Identifier(config.user)), + ) + try: + with pytest.raises(GraphDBError) as caught: + db.search_by_fulltext(["星港"]) + assert caught.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format( + sql.Identifier(config.user) + ), + ) + assert [h["id"] for h in db.search_by_fulltext(["星港"])] == ["one"] + + +def test_bm25_parallel_recovery_after_rolled_back_write(stores): + db = stores() + db.add_node("one", "星港", {}) + with pytest.raises(GraphDBError), db._get_connection() as conn, conn.cursor() as cursor: + cursor.execute('UPDATE memos_graph."Memory" SET memory=%s', ("rollbackmarker",)) + cursor.execute("SELECT 1/0") + with ThreadPoolExecutor(2) as executor: + futures = [executor.submit(db.search_by_fulltext, ["星港"]) for _ in range(2)] + assert [[h["id"] for h in future.result(timeout=10)] for future in futures] == [ + ["one"], + ["one"], + ] + assert db.search_by_fulltext(["rollbackmarker"]) == [] + + +def test_bm25_retriever_fast_graph_switch_and_cube_completion(stores): + from memos.memories.textual.tree_text_memory.retrieve.recall import GraphMemoryRetriever + from memos.memories.textual.tree_text_memory.retrieve.retrieval_mid_structs import ( + ParsedTaskGoal, + ) + + db = stores() + node_id = "00000000-0000-4000-8000-000000000001" + db.add_node(node_id, "星港", {"memory_type": "LongTermMemory", "embedding": [1, 0, 0]}) + db.add_node(node_id, "星港 回归", {"memory_type": "LongTermMemory"}, user_name="B") + retriever = GraphMemoryRetriever(db, None, include_embedding=True) + parsed = ParsedTaskGoal(keys=["星港"], tags=[]) + assert ( + retriever.retrieve( + query="星港", + top_k=5, + parsed_goal=parsed, + memory_scope="LongTermMemory", + query_embedding=[], + user_name="A", + use_fast_graph=False, + ) + == [] + ) + results = retriever.retrieve( + query="星港", + top_k=5, + parsed_goal=parsed, + memory_scope="LongTermMemory", + query_embedding=[], + user_name="A", + use_fast_graph=True, + ) + assert [r.id for r in results] == [node_id] and results[0].metadata.relativity > 0 + assert results[0].metadata.embedding == [1, 0, 0] + other = retriever._fulltext_recall(["星港"], "LongTermMemory", user_name="A", cube_name="B") + assert [r.memory for r in other] == ["星港 回归"] + + +@pytest.mark.parametrize("failure", ["cancel", "timeout"]) +def test_bm25_cancel_timeout_do_not_hide_failure_or_break_pool(stores, database, failure): + from psycopg2 import sql + + admin, config = database + if failure == "timeout": + _query( + admin, + sql.SQL("ALTER ROLE {} SET statement_timeout='1s'").format(sql.Identifier(config.user)), + ) + locker = None + try: + db = stores() + db.add_node("one", "星港", {}) + locker = _connect(config) + with locker.cursor() as cursor: + cursor.execute('LOCK TABLE memos_graph."Memory" IN ACCESS EXCLUSIVE MODE') + with ThreadPoolExecutor(1) as executor: + future = executor.submit(db.search_by_fulltext, ["星港"]) + try: + if failure == "cancel": + deadline = time.monotonic() + 5 + while True: + active = _query( + admin, + "SELECT pid FROM pg_stat_activity WHERE datname=%s AND application_name='memos-opengauss' AND state='active' AND query LIKE 'SELECT business_id%%'", + (config.db_name,), + ) + if active or time.monotonic() >= deadline: + break + time.sleep(0.01) + assert len(active) == 1 + assert _query(admin, "SELECT pg_cancel_backend(%s)", (active[0][0],)) == [ + (True,) + ] + with pytest.raises(GraphDBError) as caught: + future.result(timeout=5) + assert caught.value.__cause__.pgcode == "57014" + finally: + locker.rollback() + assert [h["id"] for h in db.search_by_fulltext(["星港"])] == ["one"] + finally: + if locker is not None: + locker.close() + if failure == "timeout": + _query( + admin, + sql.SQL("ALTER ROLE {} RESET statement_timeout").format( + sql.Identifier(config.user) + ), + ) + + +def test_bm25_query_permission_failure_then_restore(stores, database): + from psycopg2 import sql + + db = stores() + db.add_node("one", "星港", {}) + admin, config = database + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as caught: + db.search_by_fulltext(["星港"]) + assert caught.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert [h["id"] for h in db.search_by_fulltext(["星港"])] == ["one"] + + +def test_bm25_index_creation_permission_failure_preserves_graph(stores, database): + from psycopg2 import sql + + db = stores() + db.add_node("one", "星港", {}) + admin, config = database + owner = _query(admin, "SELECT current_user")[0][0] + _query(admin, "DROP INDEX memos_graph.memory_fulltext_bm25") + _query( + admin, sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format(sql.Identifier(owner)) + ) + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format(sql.Identifier(config.user)), + ) + try: + with pytest.raises(GraphDBError) as caught: + stores() + assert caught.value.__cause__.pgcode == "42501" + assert db.get_node("one")["memory"] == "星港" + finally: + _query( + admin, + sql.SQL('ALTER TABLE memos_graph."Memory" OWNER TO {}').format( + sql.Identifier(config.user) + ), + ) + repaired = stores() + assert [h["id"] for h in repaired.search_by_fulltext(["星港"])] == ["one"] + + +def test_bm25_failed_batch_rolls_back_all_nodes_and_search_recovers(stores): + db = stores() + db.add_node("one", "星港", {}) + with pytest.raises(GraphDBError): + db.add_nodes_batch( + [ + {"id": "transient", "memory": "rollbackmarker"}, + { + "id": "overflow", + "memory": "rollbackmarker", + "metadata": {"embedding": [1e100, 0, 0]}, + }, + ] + ) + assert db.get_nodes(["transient", "overflow"]) == [] + assert [h["id"] for h in db.search_by_fulltext(["星港"])] == ["one"] + assert db.search_by_fulltext(["rollbackmarker"]) == [] + + +def test_bm25_invalid_inputs_and_physical_scope_preserve_storage(stores, database): + db = stores(use_multi_db=True, user_name=None) + db.add_node("one", "星港", {}) + db.add_node("one", "星港", {}, user_name="B") + admin, _ = database + statement = 'SELECT id::text,scope,business_id,memory,properties FROM ONLY memos_graph."Memory" ORDER BY scope' + before = _query(admin, statement) + for words in (None, "星港", [True], [None]): + with pytest.raises(ValueError): + db.search_by_fulltext(words) + for kwargs in ( + {"top_k": -1}, + {"threshold": float("nan")}, + {"tsquery_config": "custom"}, + {"return_fields": ["bad;"]}, + ): + with pytest.raises(ValueError): + db.search_by_fulltext(["星港"], **kwargs) + assert _query(admin, statement) == before + hits = db.search_by_fulltext(["星港"], return_fields=["user_name"]) + assert len(hits) == 1 and hits[0]["user_name"] == "" and hits[0]["id"] == "one" diff --git a/tests/graph_dbs/test_opengauss_like.py b/tests/graph_dbs/test_opengauss_like.py new file mode 100644 index 000000000..fa654b910 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_like.py @@ -0,0 +1,93 @@ +"""Bound LIKE, strict inputs and projections for default Feedback.""" + +import json + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity import capacity as _capacity +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +capacity, pool_case = _capacity, _pool_case + + +@pytest.mark.parametrize( + "kwargs", + [ + {"query_word": None}, + {"query_word": 1}, + {"query_word": "a\0"}, + {"query_word": "bad\\"}, + {"scope": ""}, + {"scope": False}, + {"status": False}, + {"status": ""}, + {"user_name": ""}, + {"search_filter": []}, + {"filter": {"and": []}}, + {"knowledgebase_ids": "B"}, + {"knowledgebase_ids": [""]}, + {"return_fields": "id"}, + {"return_fields": [1]}, + {"return_fields": ["a.b"]}, + {"unknown": 1}, + ], +) +def test_like_rejects_before_borrow(capacity, kwargs): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + db.search_by_keywords_like(**({"query_word": "%word%"} | kwargs)) + pool.getconn.assert_not_called() + + +def test_closed_like_precedes_validation(capacity): + db, pool, _, _ = capacity + db.close() + with pytest.raises(GraphDBError): + db.search_by_keywords_like(None) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize( + "fields", [None, [], ["id", "memory", "user_name", "embedding", "tags", "absent", "id"]] +) +@pytest.mark.parametrize("cross", [False, True]) +def test_like_bound_pattern_projection_and_stable_order(capacity, fields, cross): + db, _, _, cursor = capacity + cursor.fetchall.return_value = [ + ( + "id", + "中文", + json.dumps({"id": "id", "memory": "中文", "user_name": "A", "tags": ["x"]}), + "[1,0,0]", + "A", + ) + ] + pattern = "%' OR true --_\\%" + result = db.search_by_keywords_like( + pattern, + scope="LongTermMemory", + search_filter={"user_id": "u"}, + filter={"tag": "x"}, + knowledgebase_ids=["B"] if cross else None, + return_fields=fields, + ) + expected = {"id": "id"} + if fields: + expected.update(memory="中文", user_name="A", embedding=[1, 0, 0], tags=["x"], absent=None) + if cross: + expected["user_name"] = "A" + assert result == [expected] + query, params = cursor.execute.call_args.args + assert pattern not in query and params["memos_like"] == pattern + assert "LIKE %(memos_like)s" in query and "LIMIT" not in query + assert 'ORDER BY scope COLLATE "C", business_id COLLATE "C"' in query + + +def test_like_empty_success_and_invalid_decode(capacity): + db, _, _, cursor = capacity + assert db.search_by_keywords_like("") == [] + cursor.fetchall.return_value = [("id", "text", "invalid", None, "A")] + with pytest.raises(GraphDBError): + db.search_by_keywords_like("%") diff --git a/tests/graph_dbs/test_opengauss_like_integration.py b/tests/graph_dbs/test_opengauss_like_integration.py new file mode 100644 index 000000000..599ced617 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_like_integration.py @@ -0,0 +1,344 @@ +"""Literal LIKE and default Feedback writes against ordinary-role openGauss.""" + +import os +import uuid + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError +from memos.mem_feedback.simple_feedback import SimpleMemFeedback +from memos.memories.textual.tree_text_memory.organize.manager import MemoryManager +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import ( + database as _database, +) +from tests.graph_dbs.test_opengauss_integration import ( + stores as _stores, +) + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +def test_like_pattern_scopes_status_projection_and_order(stores): + db = stores() + for owner in ["B", "A"]: + for key, text in [ + ("quote", "星港 O'Reilly"), + ("percent", "a%b"), + ("underscore", "a_b"), + ("wild", "acb"), + ("slash", "a\\b"), + ("empty", ""), + ]: + db.add_node( + key, + text, + { + "memory_type": "LongTermMemory", + "status": "activated", + "user_id": "u", + "embedding": [1, 0, 0], + "info": {"tag": "x"}, + }, + user_name=owner, + ) + db.add_node("archived", "星港", {"status": "archived", "memory_type": "UserMemory"}) + for pattern, expected in [ + ("%O'Reilly%", ["quote"]), + ("%星港%", ["quote"]), + ("a_b", ["percent", "slash", "underscore", "wild"]), + (r"a\%b", ["percent"]), + (r"a\_b", ["underscore"]), + (r"a\\b", ["slash"]), + ("", ["empty"]), + ("acb", ["wild"]), + ("%missing%", []), + ]: + assert db.search_by_keywords_like(pattern) == [{"id": key} for key in expected] + assert db.search_by_keywords_like("%星港%", status="archived", scope="UserMemory") == [ + {"id": "archived"} + ] + assert db.search_by_keywords_like("%", search_filter={"user_id": "absent"}) == [] + assert db.search_by_keywords_like("%", filter={"info.tag": "missing"}) == [] + result = db.search_by_keywords_like( + "%星港%", + knowledgebase_ids=["B", "A"], + return_fields=["memory", "embedding", "user_name", "absent"], + ) + assert result == [ + { + "id": "quote", + "memory": "星港 O'Reilly", + "embedding": [1, 0, 0], + "user_name": owner, + "absent": None, + } + for owner in ["A", "B"] + ] + assert db.search_by_keywords_like("%星港%", knowledgebase_ids=["B"]) == [ + {"id": "quote", "user_name": owner} for owner in ["A", "B"] + ] + with pytest.raises(ValueError): + db.search_by_keywords_like("bad\\") + assert db.search_by_keywords_like("acb") == [{"id": "wild"}] + + +def test_like_has_no_hidden_limit(stores): + db = stores() + db.add_nodes_batch( + [ + {"id": str(i).zfill(4), "memory": "many targets", "metadata": {"status": "activated"}} + for i in range(1001) + ] + ) + assert db.search_by_keywords_like("%targets%") == [{"id": str(i).zfill(4)} for i in range(1001)] + + +@pytest.fixture +def feedback(stores): + db = stores() + embedder = Mock(config=SimpleNamespace(embedding_dims=3)) + embedder.embed.side_effect = lambda texts, **kwargs: [[0, 1, 0] for _ in texts] + manager = MemoryManager( + db, + embedder, + Mock(), + memory_size={"WorkingMemory": 100, "LongTermMemory": 100, "UserMemory": 100}, + is_reorganize=False, + ) + fb = SimpleMemFeedback( + Mock(), embedder, db, manager, SimpleNamespace(memory_version_switch="off"), Mock(), Mock() + ) + fb._keyword_replace_judgement = Mock( + return_value={ + "if_keyword_replace": "true", + "original": "旧词", + "target": "新词", + "doc_scope": "NONE", + } + ) + fb._generate_answer = Mock(return_value="updated") + return db, fb + + +@pytest.mark.parametrize("kind", ["WorkingMemory", "LongTermMemory"]) +def test_default_feedback_keyword_replacement_real_mutation(feedback, kind): + db, fb = feedback + ids = {key: str(uuid.uuid5(uuid.NAMESPACE_DNS, key)) for key in ["target", "other", "deleted"]} + for owner, key, agent, state in [ + ("A", "target", "app", "activated"), + ("B", "target", "app", "activated"), + ("A", "other", "else", "activated"), + ("A", "deleted", "app", "deleted"), + ]: + db.add_node( + ids[key], + "旧词 内容", + { + "memory_type": kind, + "status": state, + "tags": ["旧词"], + "embedding": [1, 0, 0], + "info": {"agent_id": agent}, + }, + user_name=owner, + ) + before_b = db.export_graph(user_name="B", status=[], include_embedding=True) + assert fb.DB_IDX_READY is False + result = fb.process_feedback( + "u", "A", [], "请将旧词改为新词", info={"agent_id": "app"}, corrected_answer=True + ) + updates = result["record"]["update"] + assert result["answer"] == "updated" and len(updates) == 1 and result["record"]["add"] == [] + new_id = updates[0]["id"] + uuid.UUID(new_id) + assert updates[0]["origin_memory"] == "旧词 内容" and updates[0]["archived_id"] == ids["target"] + node = db.get_node(new_id, include_embedding=True) + assert node["memory"] == "新词 内容" and node["metadata"]["tags"] == ["新词"] + assert node["metadata"]["embedding"] == [0, 1, 0] and node["metadata"]["memory_type"] == kind + if kind == "WorkingMemory": + assert new_id == ids["target"] + else: + assert ( + new_id != ids["target"] + and db.get_node(ids["target"])["metadata"]["status"] == "archived" + ) + assert node["metadata"]["covered_history"] == ids["target"] + assert db.get_node(ids["other"])["memory"] == "旧词 内容" + assert db.get_node(ids["deleted"])["metadata"]["status"] == "deleted" + assert db.export_graph(user_name="B", status=[], include_embedding=True) == before_b + expected_ids = {new_id} + if kind == "LongTermMemory": + working_id = node["metadata"]["working_binding"] + assert working_id != new_id + working = db.get_node(working_id, include_embedding=True) + assert working["metadata"]["memory_type"] == "WorkingMemory" + assert working["memory"] == "新词 内容" and working["metadata"]["embedding"] == [0, 1, 0] + expected_ids.add(working_id) + assert db.search_by_keywords_like("%新词%") == [{"id": value} for value in sorted(expected_ids)] + assert new_id in [n["id"] for n in db.search_by_embedding([0, 1, 0], scope=kind)] + assert new_id in [n["id"] for n in db.search_by_fulltext(["新词"], scope=kind)] + + +@pytest.mark.parametrize( + "permission,kind", + [("SELECT", "WorkingMemory"), ("UPDATE", "WorkingMemory"), ("CONSTRAINT", "LongTermMemory")], +) +def test_default_feedback_sql_error_visible_and_recovers(feedback, database, permission, kind): + from psycopg2 import sql + + db, fb = feedback + key = str(uuid.uuid4()) + db.add_node(key, "旧词 内容", {"memory_type": kind, "status": "activated", "tags": []}) + admin, config = database + if permission == "CONSTRAINT": + _query( + admin, + "ALTER TABLE memos_graph.\"Memory\" ADD CONSTRAINT reject_feedback CHECK (memory <> '新词 内容')", + ) + else: + _query( + admin, + sql.SQL("REVOKE " + permission + ' ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError): + fb.process_feedback("u", "A", [], "替换") + finally: + if permission == "CONSTRAINT": + _query(admin, 'ALTER TABLE memos_graph."Memory" DROP CONSTRAINT reject_feedback') + else: + _query( + admin, + sql.SQL("GRANT " + permission + ' ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.get_node(key)["memory"] == "旧词 内容" + assert db.get_node(key)["metadata"]["status"] == "activated" + + +def test_answer_failure_preserves_real_feedback_update(feedback): + db, fb = feedback + key = str(uuid.uuid4()) + db.add_node( + key, "旧词 内容", {"memory_type": "WorkingMemory", "status": "activated", "tags": []} + ) + fb._generate_answer = Mock(side_effect=ValueError("optional model failure")) + result = fb.process_feedback("u", "A", [], "替换", corrected_answer=True) + assert result["answer"] == "" and result["answer_status"] == "unavailable" + assert result["record"]["update"][0]["id"] == key + assert db.get_node(key)["memory"] == "新词 内容" + + +def test_feedback_http_and_actual_scheduler_handler_fail_on_storage( + feedback, database, monkeypatch, tmp_path +): + import json + + from contextlib import contextmanager + + from fastapi.testclient import TestClient + from psycopg2 import sql + + from memos.api.handlers.base_handler import HandlerDependencies + from memos.api.handlers.feedback_handler import FeedbackHandler + from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem + from memos.mem_scheduler.schemas.task_schemas import RunningTaskItem + from memos.mem_scheduler.task_schedule_modules.dispatcher import SchedulerDispatcher + from memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler import ( + FeedbackMessageHandler, + ) + from tests.api.test_server_router import mock_init_server + + db, fb = feedback + db.add_node( + str(uuid.uuid4()), "旧词 内容", {"memory_type": "WorkingMemory", "status": "activated"} + ) + api = FeedbackHandler( + HandlerDependencies( + mem_reader=Mock(), + mem_scheduler=Mock(), + searcher=Mock(), + reranker=Mock(), + feedback_server=fb, + ) + ) + context = Mock() + context.get_feedback_server.return_value = fb + scheduler_handler = FeedbackMessageHandler(context) + tracker = Mock() + dispatcher = SchedulerDispatcher(max_workers=1, metrics=Mock(), status_tracker=tracker) + payload = { + "user_id": "u", + "writable_cube_ids": ["A"], + "history": [], + "feedback_content": "替换", + "async_mode": "sync", + } + msg = ScheduleMessageItem( + item_id="feedback-fault", + user_id="u", + mem_cube_id="A", + label="mem_feedback", + content=json.dumps(payload), + ) + task = RunningTaskItem( + user_id="u", mem_cube_id="A", task_info="feedback", task_name="feedback", messages=[msg] + ) + dispatcher._running_tasks[task.item_id] = task + admin, config = database + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + monkeypatch.setenv("FILE_LOCAL_PATH", str(tmp_path)) + try: + with contextmanager(mock_init_server.__wrapped__)() as app: + import memos.api.routers.server_router as router + + monkeypatch.setattr(router, "feedback_handler", api) + response = TestClient(app, raise_server_exceptions=False).post( + "/product/feedback", json=payload + ) + assert response.status_code == 500 and response.json()["data"] is None + future = dispatcher.dispatcher_executor.submit( + dispatcher._create_task_wrapper(scheduler_handler, task), [msg] + ) + with pytest.raises(GraphDBError): + future.result(timeout=10) + assert task.status == "failed" + tracker.task_failed.assert_called_once() + tracker.task_completed.assert_not_called() + finally: + dispatcher.close() + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + + +def test_like_discards_terminated_connection_and_recovers(stores, database): + db = stores(maxconn=1) + db.add_node("one", "recover", {}) + with db._get_connection() as connection, connection.cursor() as cursor: + cursor.execute("SELECT pg_backend_pid()") + pid = cursor.fetchone()[0] + admin, _ = database + _query(admin, "SELECT pg_terminate_backend(%s)", (pid,)) + with pytest.raises(GraphDBError): + db.search_by_keywords_like("%recover%") + assert db.search_by_keywords_like("%recover%") == [{"id": "one"}] diff --git a/tests/graph_dbs/test_opengauss_memory_pairs_integration.py b/tests/graph_dbs/test_opengauss_memory_pairs_integration.py new file mode 100644 index 000000000..fc9e75f72 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_memory_pairs_integration.py @@ -0,0 +1,117 @@ +"""Working and graph memory creation uses one real openGauss transaction.""" + +import os +import uuid + +from concurrent.futures import ThreadPoolExecutor +from threading import Barrier +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError +from memos.memories.textual.item import TextualMemoryItem +from memos.memories.textual.tree_text_memory.organize.manager import MemoryManager +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +@pytest.mark.parametrize("batch", [False, True]) +@pytest.mark.parametrize("kind", ["LongTermMemory", "UserMemory"]) +def test_real_pairs_retry_scope_and_indexes(stores, batch, kind): + db = stores() + manager = MemoryManager(db, Mock(), Mock()) + item = TextualMemoryItem( + memory="pairmarker", metadata={"memory_type": kind, "embedding": [1, 0, 0]} + ) + working = str(uuid.uuid5(uuid.NAMESPACE_URL, "memos:working:" + item.id)) + db.add_node(item.id, "other cube", {"memory_type": kind}, user_name="B") + for _ in range(2): + assert manager.add([item], "A", mode="async", use_batch=batch) == [item.id] + nodes = db.export_graph(user_name="A", status=[], include_embedding=True)["nodes"] + assert {n["id"] for n in nodes} == {item.id, working} + final = db.get_node(item.id, user_name="A", include_embedding=True) + temporary = db.get_node(working, user_name="A", include_embedding=True) + assert final["metadata"]["memory_type"] == kind + assert final["metadata"]["working_binding"] == working + assert temporary["metadata"]["memory_type"] == "WorkingMemory" + assert not temporary["metadata"].get("working_binding") + for node in [final, temporary]: + assert node["memory"] == "pairmarker" and node["metadata"]["embedding"] == [1, 0, 0] + assert {n["id"] for n in db.search_by_embedding([1, 0, 0], scope=None)} == { + item.id, + working, + } + assert {n["id"] for n in db.search_by_fulltext(["pairmarker"], scope=None)} == { + item.id, + working, + } + assert db.get_node(item.id, user_name="B")["memory"] == "other cube" + + +@pytest.mark.parametrize("batch", [False, True]) +@pytest.mark.parametrize("existing", [False, True]) +def test_second_node_failure_rolls_back_pair_and_indexes(stores, database, batch, existing): + db = stores() + manager = MemoryManager(db, Mock(), Mock()) + manager.is_reorganize = True + manager.reorganizer = Mock() + item = TextualMemoryItem( + memory="beforemarker", metadata={"memory_type": "LongTermMemory", "embedding": [1, 0, 0]} + ) + if existing: + manager.add([item], "A", mode="async", use_batch=batch) + before = db.export_graph(status=[], include_embedding=True) + manager.reorganizer.reset_mock() + item.memory = "rejectpairmarker" + item.metadata.embedding = [0, 1, 0] + admin, _ = database + _query( + admin, + """ALTER TABLE memos_graph."Memory" ADD CONSTRAINT reject_pair + CHECK (memory <> 'rejectpairmarker' OR business_id <> %s)""", + (item.id,), + ) + try: + with pytest.raises(GraphDBError): + manager.add([item], "A", mode="async", use_batch=batch) + finally: + _query(admin, 'ALTER TABLE memos_graph."Memory" DROP CONSTRAINT reject_pair') + assert db.export_graph(status=[], include_embedding=True) == before + assert db.search_by_fulltext(["rejectpairmarker"], scope=None) == [] + manager.reorganizer.add_message.assert_not_called() + assert manager.add([item], "A", mode="async", use_batch=batch) == [item.id] + nodes = db.export_graph(status=[], include_embedding=True)["nodes"] + assert len(nodes) == 2 and all(n["memory"] == "rejectpairmarker" for n in nodes) + + +@pytest.mark.parametrize("batch", [False, True]) +def test_concurrent_rewrite_keeps_pair_consistent(stores, batch): + db = stores() + manager = MemoryManager(db, Mock(), Mock()) + primary = str(uuid.uuid4()) + barrier = Barrier(2) + + def write(text): + item = TextualMemoryItem( + id=primary, memory=text, metadata={"memory_type": "LongTermMemory"} + ) + barrier.wait(timeout=5) + assert manager.add([item], "A", mode="async", use_batch=batch) == [primary] + + with ThreadPoolExecutor(2) as pool: + list(pool.map(write, ["first", "second"])) + nodes = db.export_graph(status=[])["nodes"] + assert len(nodes) == 2 + assert nodes[0]["memory"] == nodes[1]["memory"] in {"first", "second"} + primary_node = next(node for node in nodes if node["id"] == primary) + assert primary_node["metadata"]["working_binding"] == next( + node["id"] for node in nodes if node["id"] != primary + ) diff --git a/tests/graph_dbs/test_opengauss_records.py b/tests/graph_dbs/test_opengauss_records.py new file mode 100644 index 000000000..afba38393 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_records.py @@ -0,0 +1,89 @@ +"""Ownership and deletion record inputs, failures and exact results.""" + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_capacity import capacity as _capacity +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +capacity, pool_case = _capacity, _pool_case +CALLS = [ + ("get_user_names_by_memory_ids", (["a"],)), + ("exist_user_name", ("A",)), + ("delete_node_by_mem_cube_id", ("A", "r")), + ("recover_memory_by_mem_cube_id", ("A", "r")), +] + + +@pytest.mark.parametrize("method,args", CALLS) +def test_closed_record_methods_do_not_borrow(capacity, method, args): + db, pool, _, _ = capacity + db.close() + with pytest.raises(GraphDBError, match="closed"): + getattr(db, method)(*args) + pool.getconn.assert_not_called() + + +@pytest.mark.parametrize( + "method,args", + [ + ("get_user_names_by_memory_ids", (None,)), + ("get_user_names_by_memory_ids", (["a\0"],)), + ("exist_user_name", (None,)), + ("exist_user_name", ("",)), + ("delete_node_by_mem_cube_id", (None, "r")), + ("delete_node_by_mem_cube_id", ("A", "")), + ("delete_node_by_mem_cube_id", ("A", "r", 1)), + ("delete_node_by_mem_cube_id", ("A", "r", None)), + ("recover_memory_by_mem_cube_id", ("A", "r\0")), + ], +) +def test_record_validation_precedes_borrow(capacity, method, args): + db, pool, _, _ = capacity + with pytest.raises(ValueError): + getattr(db, method)(*args) + pool.getconn.assert_not_called() + + +def test_ownership_empty_unique_missing_and_ambiguous(capacity): + db, pool, _, cursor = capacity + assert db.get_user_names_by_memory_ids([]) == {} + pool.getconn.assert_not_called() + cursor.fetchall.return_value = [("a", ""), ("b", "B")] + inputs = ["b", "a", "missing", "a"] + assert db.get_user_names_by_memory_ids(inputs) == {"b": "B", "a": "", "missing": None} + assert inputs == ["b", "a", "missing", "a"] + cursor.fetchall.return_value = [("a", "A"), ("a", "B")] + with pytest.raises(GraphDBError, match="multiple"): + db.get_user_names_by_memory_ids(["a"]) + + +@pytest.mark.parametrize("method,args", CALLS) +@pytest.mark.parametrize("stage", ["execute", "fetch", "commit", "putconn"]) +def test_record_driver_failures_are_not_empty_success(capacity, method, args, stage): + import psycopg2 + + db, pool, connection, cursor = capacity + error = psycopg2.ProgrammingError("record injection") + if stage == "execute": + cursor.execute.side_effect = error + elif stage == "fetch": + cursor.fetchall.side_effect = error + cursor.fetchone.side_effect = error + elif stage == "commit": + connection.commit.side_effect = error + else: + pool.putconn.side_effect = error + with pytest.raises(GraphDBError) as caught: + getattr(db, method)(*args) + assert caught.value.__cause__ is error and db._active == 0 + assert pool.getconn.call_count == 1 + + +def test_bad_record_properties_roll_back(capacity): + db, _, connection, cursor = capacity + cursor.fetchall.return_value = [("a", "memory", "{", None)] + with pytest.raises(GraphDBError): + db.delete_node_by_mem_cube_id("A", "r") + connection.commit.assert_not_called() diff --git a/tests/graph_dbs/test_opengauss_records_integration.py b/tests/graph_dbs/test_opengauss_records_integration.py new file mode 100644 index 000000000..15a07c5b3 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_records_integration.py @@ -0,0 +1,293 @@ +"""Real scoped soft deletion, recovery and ownership ambiguity.""" + +import json +import os + +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timezone +from threading import Event + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs import test_opengauss_capacity_integration as support +from tests.graph_dbs.test_opengauss_integration import _query + + +database, stores = support.database, support.stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +def seed(db): + for key, state in [ + ("a", "activated"), + ("b", "archived"), + ("c", "resolving"), + ("d", None), + ("e", "missing"), + ("old", "deleted"), + ]: + meta = { + "status": state, + "memory_type": "LongTermMemory", + "created_at": "2026-01-01", + "updated_at": "2026-01-01", + "embedding": [1, 0, 0], + "delete_record_id": "previous", + "info": {"keep": True}, + } + if state == "missing": + del meta["status"] + db.add_node(key, "星港 " + key, meta) + db.add_edge("a", "b", "PARENT") + db.add_node("foreign", "B", {"status": "deleted"}, user_name="B") + + +def test_ownership_all_states_empty_and_ambiguous_identity(stores): + db = stores() + seed(db) + assert db.exist_user_name("empty") == {"empty": False} + assert db.exist_user_name("B") == {"B": True} + assert db.get_user_names_by_memory_ids(["missing", "old", "foreign", "old"]) == { + "missing": None, + "old": "A", + "foreign": "B", + } + db.add_node("old", "same ID", {}, user_name="B") + with pytest.raises(GraphDBError, match="multiple"): + db.get_user_names_by_memory_ids(["old", "foreign"]) + physical = stores(use_multi_db=True, user_name=None) + physical.add_node("physical", "empty scope", {}) + assert physical.get_user_names_by_memory_ids(["physical"]) == {"physical": ""} + + +def test_soft_delete_recovery_and_batches_preserve_unrelated_data(stores, database): + db = stores() + seed(db) + before = {key: db.get_node(key, True) for key in ["a", "b", "c", "d", "e", "old"]} + admin, _ = database + original = support.snapshot(admin) + start = datetime.now(timezone.utc) + assert db.delete_node_by_mem_cube_id("A", "R1") == 5 + end = datetime.now(timezone.utc) + times = set() + for key in ["a", "b", "c", "d", "e"]: + row = db.get_node(key, True) + stamp = row["metadata"]["delete_time"] + assert start <= datetime.fromisoformat(stamp) <= end + times.add(stamp) + expected = { + **before[key], + "metadata": { + **before[key]["metadata"], + "status": "deleted", + "delete_record_id": "R1", + "delete_time": stamp, + }, + } + assert row == expected + assert len(times) == 1 and db.get_node("old", True) == before["old"] + assert support.snapshot(admin)["edges"] == original["edges"] + assert db.get_memory_count("LongTermMemory") == 6 + assert db.get_structure_optimization_candidates("LongTermMemory") == [] + assert db.export_graph()["nodes"] == [] + assert db.search_by_embedding([1, 0, 0], top_k=20) == [] + assert db.search_by_fulltext(["星港"], top_k=20) == [] + assert db.get_by_metadata([], filter={"delete_record_id": "R1"}) == ["a", "b", "c", "d", "e"] + assert db.delete_node_by_mem_cube_id("A", "R1") == 0 + assert db.recover_memory_by_mem_cube_id("A", "wrong") == 0 + db.add_node("new", "new", {"status": "activated"}) + assert db.delete_node_by_mem_cube_id("A", "R2") == 1 + assert db.recover_memory_by_mem_cube_id("A", "R1") == 5 + assert db.recover_memory_by_mem_cube_id("A", "R1") == 0 + for key in ["a", "b", "c", "d", "e"]: + assert db.get_node(key, True) == { + **before[key], + "metadata": { + **before[key]["metadata"], + "status": "activated", + "delete_record_id": "", + "delete_time": "", + }, + } + assert [n["id"] for n in db.get_structure_optimization_candidates("LongTermMemory")] == [ + "c", + "d", + "e", + ] + assert len(db.search_by_embedding([1, 0, 0], top_k=20)) == 5 + assert len(db.search_by_fulltext(["星港"], top_k=20)) == 5 + assert db.delete_node_by_mem_cube_id("A", "R1", True) == 0 + assert db.delete_node_by_mem_cube_id("A", "R2", True) == 1 + assert db.delete_node_by_mem_cube_id("A", "R2", True) == 0 + assert db.get_node("new") is None + assert db.get_node("foreign", user_name="B")["memory"] == "B" + assert db.delete_node_by_mem_cube_id("A", "R1") == 5 + assert db.delete_node_by_mem_cube_id("A", "R1", True) == 5 + assert support.ids(db) == ["old"] + assert db.get_edges("a") == [] + + +@pytest.mark.parametrize("operation", ["delete", "recover"]) +def test_late_property_update_failure_rolls_back_entire_record(stores, database, operation): + db = stores() + seed(db) + admin, _ = database + if operation == "recover": + db.delete_node_by_mem_cube_id("A", "R") + before = support.snapshot(admin) + _query( + admin, + """CREATE FUNCTION memos_graph.record_fail() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.business_id='e' THEN RAISE EXCEPTION 'record failure'; END IF; RETURN NEW; END $$""", + ) + _query( + admin, + """CREATE TRIGGER record_fail BEFORE UPDATE ON memos_graph."Memory" FOR EACH ROW EXECUTE PROCEDURE memos_graph.record_fail()""", + ) + try: + with pytest.raises(GraphDBError): + if operation == "delete": + db.delete_node_by_mem_cube_id("A", "R") + else: + db.recover_memory_by_mem_cube_id("A", "R") + finally: + _query(admin, 'DROP TRIGGER record_fail ON memos_graph."Memory"') + _query(admin, "DROP FUNCTION memos_graph.record_fail()") + assert support.snapshot(admin) == before + + +@pytest.mark.parametrize("failure", [None, "x0500", "x1000", "permission", "timeout"]) +def test_hard_delete_record_three_blocks_is_atomic(stores, database, monkeypatch, failure): + from psycopg2 import sql + + db = stores(maxconn=1) + admin, config = database + support.seed_large(db, admin) + assert db.delete_node_by_mem_cube_id("A", "R") == 1200 + before = support.snapshot(admin) + if failure is None: + assert db.delete_node_by_mem_cube_id("A", "R", True) == 1200 + assert support.snapshot(admin) == {"nodes": [], "edges": []} + return + if failure == "permission": + _query( + admin, + sql.SQL('REVOKE DELETE ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError) as caught: + db.delete_node_by_mem_cube_id("A", "R", True) + assert caught.value.__cause__.pgcode == "42501" + finally: + _query( + admin, + sql.SQL('GRANT DELETE ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + else: + + def hook(stage, cursor, rows): + if stage == "execute" and "pg_advisory_xact_lock" in cursor.statement: + cursor.cursor.execute("SET LOCAL statement_timeout='500ms'") + + if failure == "timeout": + support.observe(db, monkeypatch, hook) + with ( + support.failing_delete( + admin, + node_id="x1000" if failure == "timeout" else failure, + action="PERFORM pg_sleep(2)" + if failure == "timeout" + else "RAISE EXCEPTION 'hard delete failure'", + ), + pytest.raises(GraphDBError) as caught, + ): + db.delete_node_by_mem_cube_id("A", "R", True) + assert caught.value.__cause__.pgcode == ("57014" if failure == "timeout" else "P0001") + assert support.snapshot(admin) == before + assert db.search_by_embedding([1, 0, 0], top_k=1300) == [] + assert db.search_by_fulltext(["批量"], top_k=1300) == [] + assert db.recover_memory_by_mem_cube_id("A", "R") == 1200 + assert len(db.search_by_embedding([1, 0, 0], top_k=1300)) == 1200 + assert len(db.search_by_fulltext(["批量"], top_k=1300)) == 1200 + + +def test_soft_delete_fixed_set_leaves_concurrent_insert_for_next_call(stores, monkeypatch): + db, writer = stores(), stores() + seed(db) + selected, resume = Event(), Event() + + def hook(stage, cursor, rows): + if stage == "fetchall" and "FOR UPDATE" in cursor.statement: + selected.set() + assert resume.wait(8) + + with monkeypatch.context() as patch: + support.observe(db, patch, hook) + with ThreadPoolExecutor(max_workers=1) as executor: + pending = executor.submit(db.delete_node_by_mem_cube_id, "A", "R") + try: + assert selected.wait(8) + writer.add_node("late", "new", {"status": "activated"}) + finally: + resume.set() + assert pending.result(timeout=8) == 5 + assert db.get_node("late")["metadata"]["status"] == "activated" + assert db.delete_node_by_mem_cube_id("A", "R") == 1 + assert db.recover_memory_by_mem_cube_id("A", "R") == 6 + + +@pytest.mark.parametrize("operation", ["delete", "recover"]) +def test_record_change_preserves_update_committed_while_waiting_for_row( + stores, database, operation +): + db, writer = stores(maxconn=1), stores(maxconn=1) + admin, _ = database + seed(db) + if operation == "recover": + db.delete_node_by_mem_cube_id("A", "R") + node = writer.get_node("a") + props = { + **node["metadata"], + "id": "a", + "memory": node["memory"], + "user_name": "A", + "info": {"concurrent": "kept"}, + } + raw = db.connection_pool.getconn() + pid = _query(raw, "SELECT pg_backend_pid()")[0][0] + raw.rollback() + db.connection_pool.putconn(raw) + update = writer.connection_pool.getconn() + try: + with update.cursor() as cursor: + cursor.execute( + 'UPDATE memos_graph."Memory" SET properties=%s::ag_catalog.agtype WHERE scope=%s AND business_id=%s', + (json.dumps(props), "A", "a"), + ) + with ThreadPoolExecutor(max_workers=1) as executor: + method = ( + db.delete_node_by_mem_cube_id + if operation == "delete" + else db.recover_memory_by_mem_cube_id + ) + future = executor.submit(method, "A", "R") + try: + support.wait_until( + lambda: _query( + admin, "SELECT 1 FROM pg_locks WHERE pid=%s AND NOT granted", (pid,) + ) + ) + finally: + update.commit() + assert future.result(timeout=8) == 5 + finally: + update.rollback() + writer.connection_pool.putconn(update) + assert db.get_node("a")["metadata"]["info"] == {"concurrent": "kept"} + assert db.get_node("a", True)["metadata"]["embedding"] == [1, 0, 0] diff --git a/tests/graph_dbs/test_opengauss_relations.py b/tests/graph_dbs/test_opengauss_relations.py new file mode 100644 index 000000000..ef2d643c2 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_relations.py @@ -0,0 +1,209 @@ +"""Scoped AGE adjacency and hard-delete contracts, including query failures.""" + +import json + +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def store(monkeypatch, psycopg2_stub): + pool = MagicMock() + cursor = pool.transaction.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", pool.transaction) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_ensure_open", pool.ensure_open) + monkeypatch.setattr(PartialOpenGaussGraphDB, "close", pool.close) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + config = OpenGaussGraphDBConfig( + host="localhost", + user="memos", + password="synthetic", + db_name="test", + user_name="A", + embedding_dimension=3, + ) + return PartialOpenGaussGraphDB(config), pool, cursor + + +@pytest.mark.parametrize("method", ["add_edge", "delete_edge"]) +def test_mutations_bind_scoped_endpoints_and_serialize(store, method): + db, pool, cursor = store + source, target, scope = "中文 ' $$ ; --", 'two\\n"', " B $$ ' " + cursor.fetchall.return_value = [(source,), (target,)] + getattr(db, method)(source, target, "RELATE_TO", user_name=scope) + calls = cursor.execute.call_args_list + assert "pg_advisory_xact_lock" in calls[0].args[0] + assert "ORDER BY business_id FOR UPDATE" in calls[1].args[0] + assert calls[1].args[1] == {"scope": scope, "ids": sorted([source, target])} + prepare = next(call.args[0] for call in calls if call.args[0].startswith("PREPARE")) + assert ("MERGE" if method == "add_edge" else "DELETE r") in prepare + assert "a.user_name = $scope" in prepare and "b.user_name = $scope" in prepare + execute = next(call for call in calls if call.args[0].startswith("EXECUTE")) + expected = {"scope": scope, "source": source, "target": target} + if method == "delete_edge": + expected["type"] = "RELATE_TO" + assert json.loads(execute.args[1]["parameters"]) == expected + assert source not in prepare and scope not in prepare + assert calls[-1].args[0] == "DEALLOCATE memos_age" + pool.transaction.assert_called_once() + + +@pytest.mark.parametrize("method", ["add_edge", "delete_edge", "delete_node"]) +def test_missing_endpoints_are_noops(store, method): + db, _, cursor = store + cursor.fetchall.return_value = [] + args = ("missing",) if method == "delete_node" else ("missing", "two", "RELATE_TO") + getattr(db, method)(*args) + assert not any(call.args[0].startswith("PREPARE") for call in cursor.execute.call_args_list) + + +def test_hard_delete_removes_native_edges_then_sql_label_row(store): + db, _, cursor = store + cursor.fetchall.return_value = [("one",)] + db.delete_node("one", user_name="B") + calls = cursor.execute.call_args_list + assert "DELETE r" in next(call.args[0] for call in calls if call.args[0].startswith("PREPARE")) + bound = next(call.args[1]["parameters"] for call in calls if call.args[0].startswith("EXECUTE")) + assert json.loads(bound) == {"id": "one", "scope": "B"} + assert calls[-1].args == ( + 'DELETE FROM ONLY "memos_graph"."Memory" WHERE scope = %(scope)s AND business_id = %(id)s', + {"scope": "B", "id": "one"}, + ) + + +@pytest.mark.parametrize( + "method", ["add_edge", "delete_edge", "edge_exists", "get_edges", "get_neighbors"] +) +@pytest.mark.parametrize("bad_type", [None, 7, "", " ", "x`]->(n) DELETE n //", "A" * 64, "中文"]) +def test_invalid_types_reject_before_borrowing(store, method, bad_type): + db, pool, _ = store + args = ( + ("one", "two", bad_type) + if method in ("add_edge", "delete_edge", "edge_exists") + else ("one", bad_type) + ) + with pytest.raises(ValueError, match="type"): + getattr(db, method)(*args) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("method", ["add_edge", "delete_edge"]) +def test_any_is_read_only(store, method): + db, pool, _ = store + with pytest.raises(ValueError, match="type"): + getattr(db, method)("one", "two", "ANY") + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "method,args", + [ + ("delete_node", ("",)), + ("add_edge", (" ", "two", "R")), + ("delete_edge", ("one", None, "R")), + ("edge_exists", ("one", 3)), + ("get_edges", (None,)), + ("get_neighbors", ("", "R")), + ], +) +def test_invalid_ids_reject_before_borrowing(store, method, args): + db, pool, _ = store + with pytest.raises(ValueError, match="id"): + getattr(db, method)(*args) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "method,args", + [ + ("get_edges", ("one", "ANY", "out")), + ("edge_exists", ("one", "two", "ANY", "both")), + ("get_neighbors", ("one", "ANY", "ANY")), + ("get_edges", ("one", "ANY", None)), + ("get_neighbors", ("one", "ANY", [])), + ], +) +def test_invalid_directions_reject_before_borrowing(store, method, args): + db, pool, _ = store + with pytest.raises(ValueError, match="direction"): + getattr(db, method)(*args) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("direction", ["OUTGOING", "INCOMING", "ANY"]) +@pytest.mark.parametrize("edge_type", ["ANY", "RELATE_TO"]) +def test_projection_is_sorted_unique_and_scoped(store, direction, edge_type): + db, _, cursor = store + cursor.fetchall.return_value = [ + ('"two"', '"one"', '"PARENT"'), + ('"one"', '"two"', '"RELATE_TO"'), + ('"one"', '"two"', '"RELATE_TO"'), + ] + assert db.get_edges("one", edge_type, direction, user_name="B") == [ + {"from": "one", "to": "two", "type": "RELATE_TO"}, + {"from": "two", "to": "one", "type": "PARENT"}, + ] + calls = cursor.execute.call_args_list + prepare = next(call.args[0] for call in calls if call.args[0].startswith("PREPARE")) + assert "a.user_name = $scope" in prepare and "b.user_name = $scope" in prepare + assert ("type(r) = $type" in prepare) == (edge_type != "ANY") + bound = next(call.args[1]["parameters"] for call in calls if call.args[0].startswith("EXECUTE")) + assert json.loads(bound)["scope"] == "B" + + +@pytest.mark.parametrize( + "rows", [[("bad-json", '"two"', '"R"')], [("null", '"two"', '"R"')], [('"one"',)]] +) +def test_bad_projection_is_an_explicit_error(store, rows): + db, _, cursor = store + cursor.fetchall.return_value = rows + with pytest.raises(GraphDBError, match="edge"): + db.get_edges("one") + + +@pytest.mark.parametrize( + "method,args", + [ + ("delete_node", ("one",)), + ("add_edge", ("one", "two", "R")), + ("delete_edge", ("one", "two", "R")), + ("edge_exists", ("one", "two")), + ("get_edges", ("one",)), + ("get_neighbors", ("one", "ANY")), + ], +) +def test_closed_store_rejects_graph_operations(store, method, args): + db, pool, _ = store + pool.ensure_open.side_effect = GraphDBError("closed") + with pytest.raises(GraphDBError, match="closed"): + getattr(db, method)(*args) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("stage", ["prepare", "execute", "fetch", "deallocate"]) +def test_cypher_failures_preserve_error_and_exit_transaction(store, stage): + db, pool, cursor = store + error = ValueError("synthetic stage failure") + if stage == "fetch": + cursor.fetchall.side_effect = error + else: + + def execute(statement, *args): + if statement.startswith( + {"prepare": "PREPARE", "execute": "EXECUTE", "deallocate": "DEALLOCATE memos_age"}[ + stage + ] + ): + raise error + + cursor.execute.side_effect = execute + with pytest.raises(ValueError) as caught: + db.get_edges("one") + assert caught.value is error + assert pool.transaction.return_value.__exit__.call_args.args[1] is error diff --git a/tests/graph_dbs/test_opengauss_search_consumers_integration.py b/tests/graph_dbs/test_opengauss_search_consumers_integration.py new file mode 100644 index 000000000..5b164294a --- /dev/null +++ b/tests/graph_dbs/test_opengauss_search_consumers_integration.py @@ -0,0 +1,297 @@ +"""Real graph candidates through existing ranking/API/dispatcher consumers. + +Models are controlled; this is not a live-model or deployed-service E2E test. +""" + +import os + +from contextlib import contextmanager +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from fastapi.testclient import TestClient + +from memos.api.handlers.base_handler import HandlerDependencies +from memos.api.handlers.search_handler import SearchHandler +from memos.api.product_models import APISearchRequest +from memos.exceptions import GraphDBError +from memos.mem_scheduler.optimized_scheduler import OptimizedScheduler +from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem +from memos.mem_scheduler.schemas.task_schemas import RunningTaskItem +from memos.mem_scheduler.task_schedule_modules.dispatcher import SchedulerDispatcher +from memos.memories.textual.tree_text_memory.retrieve.recall import GraphMemoryRetriever +from memos.memories.textual.tree_text_memory.retrieve.searcher import Searcher +from tests.api.test_server_router import mock_init_server +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import ( + database as _database, +) +from tests.graph_dbs.test_opengauss_integration import ( + stores as _stores, +) + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +@pytest.fixture +def pipeline(stores): + db = stores() + for owner, key, memory, vector in [ + ("A", "00000000-0000-0000-0000-000000000001", "apple orchard", [1, 0, 0]), + ("A", "00000000-0000-0000-0000-000000000002", "apple harvest", [1, 0, 0]), + ("B", "00000000-0000-0000-0000-000000000001", "blue ocean", [0, 1, 0]), + ("B", "00000000-0000-0000-0000-000000000002", "sailing yachts", [0, 1, 0]), + ("B", "00000000-0000-0000-0000-000000000003", "星港 travel", None), + ]: + db.add_node( + key, + memory, + { + "embedding": vector, + "memory_type": "LongTermMemory", + "status": "activated", + "created_at": "2026-01-01T00:00:00Z", + }, + user_name=owner, + ) + embedder = Mock() + embedder.embed.return_value = [[0, 0, 1]] + retriever = GraphMemoryRetriever(db, embedder, include_embedding=True) + + def vector_search(**kwargs): + return retriever._vector_recall( + [[1, 1, 0]], + "LongTermMemory", + top_k=kwargs["top_k"], + user_name=kwargs["user_name"], + search_filter=kwargs.get("search_filter"), + ) + + # Control only model/query work; run the real DataVec candidate consumer and SingleCube paths. + searcher = Searcher.__new__(Searcher) + searcher.graph_store = db + searcher.embedder = embedder + searcher.reranker = None + searcher._extract_weighted_keyword_terms = Mock(return_value=["星港"]) + searcher.retrieve = lambda **kw: searcher._retrieve_from_keyword( + kw["query"], + None, + [[1, 1, 0]], + kw["top_k"], + "LongTermMemory", + user_name=kw["user_name"], + search_filter=kw.get("search_filter"), + rerank=False, + ) + scheduler = Mock() + scheduler.retriever.enhance_memories_with_query.side_effect = lambda **kw: ( + kw["memories"], + None, + ) + mixture = SimpleNamespace( + config=SimpleNamespace(use_redis_queue=True), + searcher=searcher, + api_module=Mock(get_history_memories=Mock(return_value=[])), + history_memory_turns=1, + reranker=Mock(rerank=Mock(return_value=[])), + submit_memory_history_async_task=Mock(), + ) + scheduler.mix_search_memories = lambda **kw: OptimizedScheduler.mix_search_memories( + mixture, **kw + ) + handler = SearchHandler( + HandlerDependencies( + naive_mem_cube=SimpleNamespace(text_mem=SimpleNamespace(search=vector_search)), + mem_reader=Mock(), + mem_scheduler=scheduler, + searcher=searcher, + deepsearch_agent=Mock(), + reranker=None, + ) + ) + yield db, handler, searcher + + +@pytest.mark.parametrize( + "dedup,top_k,expected", + [ + ( + "no", + 4, + [ + ( + "A", + [ + "00000000-0000-0000-0000-000000000001", + "00000000-0000-0000-0000-000000000002", + ], + ), + ( + "B", + [ + "00000000-0000-0000-0000-000000000001", + "00000000-0000-0000-0000-000000000002", + ], + ), + ], + ), + ( + "sim", + 1, + [ + ("A", ["00000000-0000-0000-0000-000000000001"]), + ("B", ["00000000-0000-0000-0000-000000000001"]), + ], + ), + ( + "mmr", + 1, + [ + ("A", ["00000000-0000-0000-0000-000000000001"]), + ("B", ["00000000-0000-0000-0000-000000000001"]), + ], + ), + ], +) +def test_real_candidates_final_identity_order_and_no_mutation(pipeline, dedup, top_k, expected): + db, handler, _ = pipeline + before = db.export_graph(all_cubes=True, status=[], include_embedding=True) + req = APISearchRequest( + query="query", + user_id="u", + readable_cube_ids=["A", "B", "A"], + dedup=dedup, + top_k=top_k, + mode="fast", + relativity=0, + ) + result = handler.handle_search_memories(req).data["text_mem"] + assert [(b["cube_id"], [m["id"] for m in b["memories"]]) for b in result] == expected + for bucket in result: + for mem in bucket["memories"]: + assert mem["memory"] in ( + {"apple orchard", "apple harvest"} + if bucket["cube_id"] == "A" + else {"blue ocean", "sailing yachts"} + ) + assert mem["metadata"]["relativity"] == pytest.approx((1 + 2**-0.5) / 2) + assert db.export_graph(all_cubes=True, status=[], include_embedding=True) == before + + +def test_keyword_missing_embedding_and_valid_empty(pipeline): + db, handler, searcher = pipeline + pairs = searcher.retrieve(query="星港", top_k=10, user_name="B") + assert [m.id for m, _ in pairs] == ["00000000-0000-0000-0000-000000000003"] + node = pairs[0][0].model_dump() + assert node["metadata"]["embedding"] is None + assert handler._extract_embeddings([node]) == [[0, 0, 1]] + searcher.embedder.embed.assert_called_once_with(["星港 travel"]) + assert ( + db.get_node("00000000-0000-0000-0000-000000000003", include_embedding=True, user_name="B")[ + "metadata" + ]["embedding"] + is None + ) + assert searcher.retrieve(query="星港", top_k=10, user_name="A") == [] + + +@pytest.fixture +def client(pipeline, monkeypatch, tmp_path): + monkeypatch.setenv("FILE_LOCAL_PATH", str(tmp_path)) + with contextmanager(mock_init_server.__wrapped__)() as app: + import memos.api.routers.server_router as router + + monkeypatch.setattr(router, "search_handler", pipeline[1]) + yield TestClient(app, raise_server_exceptions=False) + + +@pytest.mark.parametrize("mode", ["fast", "fine", "mixture"]) +@pytest.mark.parametrize("fault", ["sql", "filter"]) +def test_required_reads_fail_through_actual_modes_and_http(pipeline, database, client, mode, fault): + from psycopg2 import sql + + db, handler, _ = pipeline + admin, config = database + payload = { + "query": "query", + "user_id": "u", + "readable_cube_ids": ["A", "B"], + "mode": mode, + "dedup": "no", + } + if fault == "filter": + payload["filter"] = {"and": [{"tag": {"unknown": "x"}}]} + else: + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + with pytest.raises(GraphDBError if fault == "sql" else ValueError): + handler.handle_search_memories(APISearchRequest(**payload)) + response = client.post("/product/search", json=payload) + assert response.status_code == (500 if fault == "sql" else 400), response.text + assert response.json()["data"] is None + finally: + if fault == "sql": + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) + assert db.get_node("00000000-0000-0000-0000-000000000001")["memory"] == "apple orchard" + + +def test_scheduler_marks_required_storage_failure_failed(pipeline, database): + from psycopg2 import sql + + db, handler, _ = pipeline + admin, config = database + tracker = Mock() + dispatcher = SchedulerDispatcher(max_workers=1, metrics=Mock(), status_tracker=tracker) + message = ScheduleMessageItem( + item_id="required-search", user_id="u", mem_cube_id="A", label="query", content="query" + ) + task = RunningTaskItem( + user_id="u", + mem_cube_id="A", + task_info="required read", + task_name="query", + messages=[message], + ) + dispatcher._running_tasks[task.item_id] = task + req = APISearchRequest(query="query", user_id="u", readable_cube_ids=["A", "B"], mode="fast") + work = dispatcher._create_task_wrapper( + lambda messages: handler.handle_search_memories(req), task + ) + _query( + admin, + sql.SQL('REVOKE SELECT ON memos_graph."Memory" FROM {}').format( + sql.Identifier(config.user) + ), + ) + try: + future = dispatcher.dispatcher_executor.submit(work, [message]) + with pytest.raises(GraphDBError): + future.result(timeout=10) + assert task.status == "failed" + tracker.task_failed.assert_called_once() + assert tracker.task_failed.call_args.kwargs["error_message"] + tracker.task_completed.assert_not_called() + finally: + dispatcher.close() + _query( + admin, + sql.SQL('GRANT SELECT ON memos_graph."Memory" TO {}').format( + sql.Identifier(config.user) + ), + ) diff --git a/tests/graph_dbs/test_opengauss_sql.py b/tests/graph_dbs/test_opengauss_sql.py new file mode 100644 index 000000000..47737a417 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_sql.py @@ -0,0 +1,138 @@ +"""Shared cursor lifetime and independently composable named SQL fragments.""" + +import re + +from copy import deepcopy +from unittest.mock import MagicMock + +import pytest + +from memos.exceptions import GraphDBError +from tests.graph_dbs.test_opengauss_connection import pool_case as _pool_case + + +pool_case = _pool_case + + +@pytest.mark.parametrize( + "utc,lock", [(False, None), (True, None), (False, 1), (True, 2), (False, 0)] +) +def test_cursor_owns_one_transaction_and_orders_optional_setup(pool_case, utc, lock): + _, db, pool, connection, _ = pool_case + cursor = connection.cursor.return_value.__enter__.return_value + with db._cursor(utc=utc, lock=lock) as yielded: + assert yielded is cursor + yielded.execute("SELECT 1") + connection.commit.assert_not_called() + statements = [call.args for call in cursor.execute.call_args_list][3:] + expected = [] + if utc: + expected.append(("SET LOCAL TIME ZONE 'UTC'",)) + if lock is not None: + expected.append(("SELECT pg_advisory_xact_lock(621503, %(lock)s)", {"lock": lock})) + assert statements == [*expected, ("SELECT 1",)] + pool.getconn.assert_called_once() + connection.commit.assert_called_once() + pool.putconn.assert_called_once_with(connection, close=False) + + +@pytest.mark.parametrize("stage", ["cursor", "utc", "lock", "body", "close"]) +def test_cursor_setup_body_and_close_errors_use_existing_rollback(pool_case, stage): + driver, db, pool, connection, _ = pool_case + manager = MagicMock() + cursor = manager.__enter__.return_value + error = driver.ProgrammingError("shared cursor failure") + connection.cursor.side_effect = [ + connection.cursor.return_value, + error if stage == "cursor" else manager, + ] + + def execute(statement, params=None): + if (stage == "utc" and "TIME ZONE" in statement) or ( + stage == "lock" and "advisory" in statement + ): + raise error + + cursor.execute.side_effect = execute + if stage == "close": + manager.__exit__.side_effect = error + with pytest.raises(GraphDBError) as caught, db._cursor(utc=True, lock=2): + if stage == "body": + raise error + assert caught.value.__cause__ is error + connection.commit.assert_not_called() + assert connection.rollback.call_count == 2 + pool.putconn.assert_called_once_with(connection, close=False) + + +def test_cursor_closed_store_does_not_borrow(pool_case): + _, db, pool, _, _ = pool_case + db.close() + with pytest.raises(GraphDBError, match="closed"), db._cursor(): + pytest.fail("Closed cursor yielded") + pool.getconn.assert_not_called() + + +def test_fragment_field_reuses_bindings_without_mutating_inputs(pool_case): + _, db, _, _, _ = pool_case + path = ["info", "department"] + sql, params = db._text_field_expr(path, prefix="memos_field") + assert path == ["info", "department"] + assert len(params) == 2 + assert set(params.values()) == {'"info"', '"department"'} + assert set(re.findall(r"%\((\w+)\)s", sql)) == set(params) + assert len(re.findall(r"%\((\w+)\)s", sql)) > len(params) + params["memos_field_root"] = "modified" + assert db._text_field_expr(path, prefix="memos_field")[1]["memos_field_root"] == '"info"' + + +def test_binding_merge_rejects_collisions_without_touching_inputs(pool_case): + _, db, _, _, _ = pool_case + left, right = {"limit": 18}, {"limit": 65} + with pytest.raises(ValueError, match="Duplicate"): + db._merge_bindings(left, right) + assert left == {"limit": 18} and right == {"limit": 65} + assert db._merge_bindings(left, {"other": 65}) == {"limit": 18, "other": 65} + assert db._merge_bindings() == {} + + +def test_nested_repeated_field_conditions_have_distinct_parameters(pool_case): + _, db, _, _, _ = pool_case + condition = { + "and": [ + {"confidence": {"gt": 18, "lt": 65}}, + {"or": [{"confidence": 21}, {"confidence": 42}]}, + ] + } + before = deepcopy(condition) + sql, params = db._build_filter_where_clause(condition, prefix="memos_filter") + assert condition == before + values = list(params.values()) + for value in [18, 65, "21", "42"]: + assert value in values + assert "%s" not in sql + assert set(re.findall(r"%\((\w+)\)s", sql)) == set(params) + other_sql, other_params = db._build_filter_where_clause(condition, prefix="memos_other") + assert not set(params) & set(other_params) + assert sql != other_sql + + +def test_grouping_helpers_validate_before_borrow_and_decode_before_commit(pool_case, monkeypatch): + _, db, pool, connection, _ = pool_case + predicate, params = db._validate_grouping(["info.v"], " WHERE business_id=%(id)s ", {"id": "x"}) + assert predicate == "business_id=%(id)s" and params == {"id": "x"} + query, bound = db._build_grouping_query(["info.v"], predicate, params, "A") + assert "GROUP BY" in query and bound["memos_scope"] == "A" + assert params == {"id": "x"} + pool.getconn.assert_not_called() + original = db._decode_grouped_rows + + def decode(fields, rows): + connection.commit.assert_not_called() + return original(fields, rows) + + monkeypatch.setattr(db, "_decode_grouped_rows", decode) + cursor = connection.cursor.return_value.__enter__.return_value + cursor.fetchall.return_value = [('"value"', 2)] + assert db.get_grouped_counts(["info.v"]) == [{"info.v": "value", "count": 2}] + connection.commit.assert_called_once() diff --git a/tests/graph_dbs/test_opengauss_sql_integration.py b/tests/graph_dbs/test_opengauss_sql_integration.py new file mode 100644 index 000000000..776f80231 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_sql_integration.py @@ -0,0 +1,120 @@ +"""Named SQL composition through public methods and real AGE/DataVec/BM25.""" + +import os + +from copy import deepcopy + +import pytest + +from tests.graph_dbs.test_opengauss_integration import database as _database +from tests.graph_dbs.test_opengauss_integration import stores as _stores + + +database, stores = _database, _stores +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG"), reason="openGauss integration tests not enabled" +) + + +@pytest.mark.parametrize("method", ["metadata", "vector", "fulltext"]) +def test_repeated_fields_combine_across_independent_predicates(stores, method): + db = stores() + for node_id, confidence, status in [ + ("a", 21, "activated"), + ("b", 42, "activated"), + ("c", 65, "activated"), + ("d", 21, "archived"), + ("e", 10, "activated"), + ]: + db.add_node( + node_id, + "星港 回归", + { + "confidence": confidence, + "status": status, + "embedding": [1, 0, 0], + "memory_type": "WorkingMemory", + "session_id": "shared", + }, + ) + db.add_node( + "foreign", + "星港 回归", + { + "confidence": 21, + "status": "activated", + "embedding": [1, 0, 0], + "memory_type": "WorkingMemory", + "session_id": "shared", + }, + user_name="B", + ) + condition = { + "and": [ + {"confidence": {"gt": 18, "lt": 65}}, + {"or": [{"confidence": 21}, {"confidence": 42}]}, + ] + } + original = deepcopy(condition) + if method == "metadata": + result = db.get_by_metadata( + [{"field": "confidence", "op": "gte", "value": 20}], + filter=condition, + status="activated", + ) + else: + search = db.search_by_embedding if method == "vector" else db.search_by_fulltext + result = [ + hit["id"] + for hit in search( + [1, 0, 0] if method == "vector" else ["星港"], + search_filter={"session_id": "shared"}, + filter=condition, + scope="WorkingMemory", + status="activated", + top_k=10, + ) + ] + assert result == ["a", "b"] + assert condition == original + + +def test_array_like_and_repeated_nested_field_bindings(stores): + db = stores() + literal = "50%_\\'中文" + for node_id, text, tags in [ + ("a", literal, ["red", "blue"]), + ("b", "50xx中文", ["red", "blue"]), + ("c", literal, ["green"]), + ]: + db.add_node(node_id, "fixture", {"info": {"label": text}, "tags": tags}) + assert db.get_by_metadata( + [], + filter={ + "and": [ + {"info.label": {"like": literal, "starts_with": "50%", "ends_with": "中文"}}, + {"tags": {"contains": ["red", "green"], "all": ["blue"]}}, + ] + }, + ) == ["a"] + + +def test_grouping_user_bindings_are_separate_and_reusable(stores): + db = stores() + for scope in ("A", "B"): + db.add_node( + "one", "星港", {"info": {"department": "研发", "team": "存储"}}, user_name=scope + ) + fields = ["info.department", "info.team"] + params = {"scope": "one"} + assert db.get_grouped_counts( + fields, + "(business_id=%(scope)s OR business_id=%(scope)s) AND memory LIKE '%%星港%%'", + params, + ) == [{"info.department": "研发", "info.team": "存储", "count": 1}] + assert params == {"scope": "one"} + with pytest.raises(ValueError, match="reserved"): + db.get_grouped_counts( + fields, "business_id=%(memos_group_0_root)s", {"memos_group_0_root": "one"} + ) + assert db.get_memory_count("WorkingMemory") == 0 diff --git a/tests/graph_dbs/test_opengauss_store.py b/tests/graph_dbs/test_opengauss_store.py new file mode 100644 index 000000000..35d8a59ff --- /dev/null +++ b/tests/graph_dbs/test_opengauss_store.py @@ -0,0 +1,447 @@ +"""AGE node storage contract; database tests live beside these unit tests.""" + +import importlib +import json + +from copy import deepcopy +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def config(): + return OpenGaussGraphDBConfig( + host="localhost", + user="memos", + password="synthetic-password", + db_name="test", + user_name="A", + embedding_dimension=3, + maxconn=2, + ) + + +@pytest.fixture +def store(config, monkeypatch, psycopg2_stub): + pool = MagicMock() + connection = pool.transaction.return_value.__enter__.return_value + cursor = connection.cursor.return_value.__enter__.return_value + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", pool.transaction) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_ensure_open", pool.ensure_open) + monkeypatch.setattr(PartialOpenGaussGraphDB, "close", pool.close) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + return PartialOpenGaussGraphDB(config), pool, cursor + + +def test_upsert_preserves_input_and_binds_scope_and_payload(store): + db, pool, cursor = store + metadata = { + "memory_type": "LongTermMemory", + "embedding": [1, 0, 0], + "sources": [{"type": "chat", "content": "中文 ' ; DROP TABLE Memory; --\n"}], + "info": {"nested": [True, None, 3]}, + "created_at": "2026-01-01T00:00:00Z", + } + original = deepcopy(metadata) + db.add_node("same", "fixture memory", metadata) + assert metadata == original + query, params = cursor.execute.call_args.args + assert "ON CONFLICT (scope, business_id)" in query + assert "id =" not in query.split("DO UPDATE", 1)[1] + assert (params["scope"], params["id"], params["memory"]) == ("A", "same", "fixture memory") + assert json.loads(params["embedding"]) == [1, 0, 0] + properties = json.loads(params["properties"]) + assert properties == { + **{k: v for k, v in metadata.items() if k != "embedding"}, + "id": "same", + "memory": "fixture memory", + "user_name": "A", + } + pool.transaction.assert_called_once() + + +def test_reads_reorder_and_deduplicate_without_leaking_scope(store): + db, _, cursor = store + cursor.fetchall.return_value = [ + ( + "two", + "second", + json.dumps({"id": "two", "memory": "second", "user_name": "A", "status": "archived"}), + None, + ), + ( + "one", + "first", + json.dumps({"id": "one", "memory": "first", "user_name": "A", "tags": ["x"]}), + "[1,0,0]", + ), + ] + assert db.get_nodes(["one", "missing", "two", "one"], include_embedding=True) == [ + {"id": "one", "memory": "first", "metadata": {"tags": ["x"], "embedding": [1, 0, 0]}}, + {"id": "two", "memory": "second", "metadata": {"status": "archived", "embedding": None}}, + ] + assert cursor.execute.call_args.args[1] == {"scope": "A", "ids": ["one", "missing", "two"]} + + +@pytest.mark.parametrize("position", [0, 1, 2]) +def test_bad_batch_is_rejected_before_acquiring_a_connection(store, position): + db, pool, _ = store + nodes = [{"id": str(i), "memory": "ok"} for i in range(3)] + nodes[position]["metadata"] = {"embedding": [1, 2]} + with pytest.raises(ValueError, match="embedding"): + db.add_nodes_batch(nodes) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("scope", ["B", " 中文 Cube ' ; -- "]) +def test_explicit_scope_is_preserved_and_overrides_config(store, scope): + db, _, cursor = store + db.add_node("id", "", {"user_name": scope}, user_name=scope) + assert cursor.execute.call_args.args[1]["scope"] == scope + + +@pytest.mark.parametrize("scope", ["", " \t", False, 12, [], {}]) +def test_bad_scope_never_borrows(store, scope): + db, pool, _ = store + with pytest.raises(ValueError, match="user_name"): + db.get_nodes(["one"], user_name=scope) + pool.transaction.assert_not_called() + + +def test_physical_database_scope_is_explicitly_bounded(store): + db, _, cursor = store + db.config.use_multi_db = True + db.config.user_name = None + db.add_node("id", "physical", {}) + assert cursor.execute.call_args.args[1]["scope"] == "" + db.config.use_multi_db = False + with pytest.raises(ValueError, match="user_name"): + db.get_node("id") + + +@pytest.mark.parametrize( + "node", + [ + {}, + {"id": "x"}, + {"memory": "ok"}, + None, + [], + {"id": None, "memory": "ok"}, + {"id": " ", "memory": "ok"}, + {"id": False, "memory": "ok"}, + {"id": "x", "memory": None}, + {"id": "x", "memory": False}, + {"id": "x", "metadata": [], "memory": "ok"}, + {"id": "x", "metadata": None, "memory": "ok"}, + ], +) +def test_invalid_node_has_no_database_side_effect(store, node): + db, pool, _ = store + with pytest.raises(ValueError): + db.add_nodes_batch([node]) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("nodes", [None, {}, "wrong", ("one",)]) +def test_batches_require_a_list(store, nodes): + db, pool, _ = store + with pytest.raises(ValueError, match="nodes"): + db.add_nodes_batch(nodes) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("embedding", [None, []]) +def test_vectorless_nodes_are_allowed_without_random_embeddings(store, embedding): + db, _, cursor = store + db.add_node( + "dream", "", {"embedding": embedding, "memory_type": "CustomMemory", "status": "pending"} + ) + assert cursor.execute.call_args.args[1]["embedding"] is None + + +@pytest.mark.parametrize( + "metadata", + [ + {"id": "overridden"}, + {"memory": "overridden"}, + {"user_name": "B"}, + {"embedding": [1, 2]}, + {"embedding": [1, 2, 3, 4]}, + {"embedding": [True, 0, 0]}, + {"embedding": [1, "0", 0]}, + {"embedding": [1, float("nan"), 0]}, + {"embedding": [1, float("inf"), 0]}, + {"embedding": False}, + {"embedding": "[1,0,0]"}, + {"embedding": (1, 0, 0)}, + {"info": {1: "integer key"}}, + {"info": ("tuple",)}, + {"embedding": [10**400, 0, 0]}, + {"info": {1, 2}}, + {"info": float("nan")}, + ], +) +def test_invalid_metadata_and_vectors_do_not_borrow(store, metadata): + db, pool, _ = store + with pytest.raises(ValueError): + db.add_node("x", "ok", metadata) + pool.transaction.assert_not_called() + + +def test_empty_batches_and_reads_do_not_borrow_but_check_lifecycle(store): + db, pool, _ = store + db.add_nodes_batch([]) + assert db.get_nodes([]) == [] + pool.transaction.assert_not_called() + assert pool.ensure_open.call_count == 2 + + +@pytest.mark.parametrize("ids", [None, "one", {}, [None], [False], [""]]) +def test_invalid_read_ids_do_not_borrow(store, ids): + db, pool, _ = store + with pytest.raises(ValueError): + db.get_nodes(ids) + pool.transaction.assert_not_called() + + +def test_single_read_missing_and_vector_projection(store): + db, _, cursor = store + cursor.fetchall.return_value = [] + assert db.get_node("missing") is None + cursor.fetchall.return_value = [ + ( + "one", + "first", + json.dumps( + { + "id": "one", + "memory": "first", + "user_name": "A", + "sources": "raw string", + "info": None, + } + ), + None, + ) + ] + assert db.get_node("one") == { + "id": "one", + "memory": "first", + "metadata": {"sources": "raw string", "info": None}, + } + assert "embedding::text" not in cursor.execute.call_args.args[0] + with pytest.raises(ValueError, match="include_embedding"): + db.get_node("one", include_embedding="false") + + +@pytest.mark.parametrize( + "properties", ["broken", "null", "[]", "{}", '{"id":"other","memory":"first","user_name":"A"}'] +) +def test_damaged_or_inconsistent_properties_are_errors(store, properties): + from memos.exceptions import GraphDBError + + db, _, cursor = store + cursor.fetchall.return_value = [("one", "first", properties, None)] + with pytest.raises(GraphDBError): + db.get_node("one") + + +def test_partial_update_keeps_fields_and_does_not_mutate_input(store): + db, _, cursor = store + cursor.fetchone.return_value = ( + "one", + "before", + json.dumps( + { + "id": "one", + "memory": "before", + "user_name": "A", + "tags": ["keep"], + "info": {"old": 1}, + } + ), + "[1,0,0]", + ) + fields = {"info": {"new": 2}, "memory": "after"} + original = deepcopy(fields) + db.update_node("one", fields) + query, params = cursor.execute.call_args.args + assert query.startswith("UPDATE") + assert (params["memory"], params["embedding"]) == ("after", "[1, 0, 0]") + assert json.loads(params["properties"]) == { + "id": "one", + "memory": "after", + "user_name": "A", + "tags": ["keep"], + "info": {"new": 2}, + } + assert (params["scope"], params["id"]) == ("A", "one") + assert fields == original + assert any("FOR UPDATE" in call.args[0] for call in cursor.execute.call_args_list) + + +@pytest.mark.parametrize( + "fields", [{"tags": ["new"]}, {"embedding": None}, {"embedding": []}, {"embedding": [0, 1, 0]}] +) +def test_partial_update_embedding_presence_controls_preservation(store, fields): + db, _, cursor = store + cursor.fetchone.return_value = ( + "one", + "before", + json.dumps({"id": "one", "memory": "before", "user_name": "A"}), + "[1,0,0]", + ) + db.update_node("one", fields) + params = cursor.execute.call_args.args[1] + assert params["memory"] == "before" + expected = fields.get("embedding", [1, 0, 0]) + assert (None if params["embedding"] is None else json.loads(params["embedding"])) == ( + expected or None + ) + + +def test_missing_update_is_noop_and_empty_update_does_not_borrow(store): + db, pool, cursor = store + db.update_node("missing", {}) + pool.transaction.assert_not_called() + cursor.fetchone.return_value = None + db.update_node("missing", {"tags": []}) + assert "FOR UPDATE" in cursor.execute.call_args.args[0] + db.close() + pool.close.assert_called_once() + + +@pytest.mark.parametrize( + "fields", + [None, [], {"id": "other"}, {"user_name": "B"}, {"memory": None}, {"embedding": [False, 0, 0]}], +) +def test_invalid_updates_do_not_borrow(store, fields): + db, pool, _ = store + with pytest.raises(ValueError): + db.update_node("one", fields) + pool.transaction.assert_not_called() + + +def _columns(): + return [ + ("id", "graphid", True, True, 1), + ("properties", "agtype", True, True, 2), + ("scope", "text", True, False, 3), + ("business_id", "text", True, False, 4), + ("memory", "text", True, False, 5), + ("embedding", "vector(3)", False, False, 6), + ] + + +@pytest.fixture +def schema_case(config, monkeypatch, psycopg2_stub): + module = importlib.import_module("memos.graph_dbs.opengauss") + pool = MagicMock() + cursor = pool.transaction.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + cursor.fetchone.side_effect = [ + (1,), + ("v", "memos_graph", "Memory"), + (1,), + ("memos_graph", "Memory", "hnsw", "vector_cosine_ops", "6", True, True, None, None), + ( + "memos_graph", + "Memory", + "bm25", + "bm25_text_ops", + "5", + True, + True, + None, + None, + ["dict_path=DEFAULT"], + ), + ] + cursor.fetchall.return_value = _columns() + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", pool.transaction) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_ensure_open", pool.ensure_open) + monkeypatch.setattr(PartialOpenGaussGraphDB, "close", pool.close) + return module, config, pool, cursor + + +@pytest.mark.parametrize("graph", [None, (1,)]) +def test_initialization_creates_missing_graph_or_label_without_private_id_helpers( + schema_case, graph +): + module, config, pool, cursor = schema_case + cursor.fetchone.side_effect = [ + graph, + None, + (1,), + None, + ("memos_graph", "Memory", "hnsw", "vector_cosine_ops", "6", True, True, None, None), + ( + "memos_graph", + "Memory", + "bm25", + "bm25_text_ops", + "5", + True, + True, + None, + None, + ["dict_path=DEFAULT"], + ), + ] + db = PartialOpenGaussGraphDB(config) + assert db.config is not config + queries = [call.args[0] for call in cursor.execute.call_args_list] + assert any("pg_advisory_xact_lock" in query for query in queries) + assert sum("create_graph" in query for query in queries) == (1 if graph is None else 0) + assert any("create_vlabel" in query for query in queries) + assert not any("_make_graph_id" in query for query in queries) + pool.close.assert_not_called() + + +def test_compatible_initialization_does_not_run_destructive_ddl(schema_case): + module, config, _, cursor = schema_case + PartialOpenGaussGraphDB(config) + assert not any( + "CREATE " in call.args[0] or "ALTER " in call.args[0] or "DROP " in call.args[0] + for call in cursor.execute.call_args_list + ) + + +@pytest.mark.parametrize("failure", ["label", "columns", "unique"]) +def test_incompatible_schema_is_rejected_and_pool_closed(schema_case, failure): + from memos.exceptions import GraphDBError + + module, config, pool, cursor = schema_case + if failure == "label": + cursor.fetchone.side_effect = [(1,), ("e", "memos_graph", "Memory")] + elif failure == "columns": + cursor.fetchall.return_value = _columns()[:-1] + [ + ("embedding", "vector(4)", False, False, 6) + ] + else: + cursor.fetchone.side_effect = [(1,), ("v", "memos_graph", "Memory"), None] + with pytest.raises(GraphDBError, match="incompatible"): + PartialOpenGaussGraphDB(config) + pool.close.assert_called_once() + + +def test_initialization_failure_preserves_original_even_if_close_fails(schema_case, monkeypatch): + from memos.exceptions import GraphDBError + + module, config, pool, _ = schema_case + error = ValueError("original") + + def fail(self): + raise error + + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", fail) + pool.close.side_effect = GraphDBError("cleanup") + with pytest.raises(ValueError) as caught: + PartialOpenGaussGraphDB(config) + assert caught.value is error diff --git a/tests/graph_dbs/test_opengauss_structure.py b/tests/graph_dbs/test_opengauss_structure.py new file mode 100644 index 000000000..4290b3946 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_structure.py @@ -0,0 +1,75 @@ +"""The openGauss provider follows the existing single-class backend layout.""" + +import ast +import importlib +import inspect + +from pathlib import Path + +from memos.configs.graph_db import GraphDBConfigFactory +from memos.graph_dbs.base import BaseGraphDB + + +def test_one_backend_class_owns_pool_storage_and_filters(): + module = importlib.import_module("memos.graph_dbs.opengauss") + cls = module.OpenGaussGraphDB + assert issubclass(cls, BaseGraphDB) + assert cls.__abstractmethods__ == frozenset() + assert not inspect.isabstract(cls) + support = importlib.import_module("tests.graph_dbs.opengauss_test_support") + assert support.PartialOpenGaussGraphDB is cls + assert "opengauss" in GraphDBConfigFactory.backend_to_class + tree = ast.parse(Path(module.__file__).read_text()) + assert [node.name for node in ast.walk(tree) if isinstance(node, ast.ClassDef)] == [ + "OpenGaussGraphDB" + ] + for name in [ + "__init__", + "_get_connection", + "close", + "parse_filter", + "_build_filter_where_clause", + "_build_single_filter_condition", + "get_by_metadata", + "get_all_memory_items", + "delete_node_by_prams", + "search_by_embedding", + "search_by_fulltext", + "get_grouped_counts", + "get_memory_count", + "node_not_exist", + "remove_oldest_memory", + "get_structure_optimization_candidates", + "export_graph", + "import_graph", + "clear", + "drop_database", + "get_user_names_by_memory_ids", + "exist_user_name", + "delete_node_by_mem_cube_id", + "recover_memory_by_mem_cube_id", + "get_context_chain", + "deduplicate_nodes", + "detect_conflicts", + "merge_nodes", + ]: + assert name in cls.__dict__ + assert "ThreadedConnectionPool" in inspect.getsource(cls.__init__) + for name in ["_opengauss_connection.py", "_opengauss_store.py", "_opengauss_filter.py"]: + assert not Path(module.__file__).with_name(name).exists() + + +def test_parse_filter_preserves_input_and_routes_business_fields(): + from copy import deepcopy + + module = importlib.import_module("memos.graph_dbs.opengauss") + condition = { + "and": [{"project": "星港"}, {"or": [{"status": "activated"}, {"info.done": True}]}] + } + before = deepcopy(condition) + assert module.OpenGaussGraphDB.parse_filter(condition) == { + "and": [{"info.project": "星港"}, {"or": [{"status": "activated"}, {"info.done": True}]}] + } + assert condition == before + assert module.OpenGaussGraphDB.parse_filter(None) is None + assert module.OpenGaussGraphDB.parse_filter({}) == {} diff --git a/tests/graph_dbs/test_opengauss_traversal.py b/tests/graph_dbs/test_opengauss_traversal.py new file mode 100644 index 000000000..6c1154bf6 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_traversal.py @@ -0,0 +1,419 @@ +"""Bounded graph traversal and ranked tag candidates, with fixed business results.""" + +import json + +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def store(monkeypatch, psycopg2_stub): + pool = MagicMock() + cursor = pool.transaction.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", pool.transaction) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_ensure_open", pool.ensure_open) + monkeypatch.setattr(PartialOpenGaussGraphDB, "close", pool.close) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + db = PartialOpenGaussGraphDB( + OpenGaussGraphDBConfig( + host="localhost", + user="memos", + password="synthetic", + db_name="test", + user_name="A", + embedding_dimension=3, + ) + ) + return db, pool, cursor + + +@pytest.fixture +def graph(store, monkeypatch): + """Model driver rows, keeping expected paths/subgraphs independent of traversal.""" + db, pool, cursor = store + nodes = {("A", name): {} for name in [*"abcdefg", "z"]} + nodes.update({("B", name): {} for name in "abcdefg"}) + edges = [ + ("A", "a", "b", "R"), + ("A", "a", "c", "R"), + ("A", "b", "d", "R"), + ("A", "c", "d", "R"), + ("A", "d", "a", "R"), + ("A", "e", "a", "R"), + ("A", "b", "c", "SIDE"), + ("A", "a", "b", "OTHER"), + ("A", "b", "b", "SELF"), + ("A", "d", "f", "R"), + ("A", "f", "g", "R"), + ("B", "a", "g", "R"), + ] + + def execute(statement, parameters): + scope, ids = parameters["scope"], parameters["ids"] + selected = [ + (name, metadata) + for (owner, name), metadata in nodes.items() + if owner == scope and (name not in ids if "NOT" in statement else name in ids) + ] + if statement.startswith("SELECT business_id FROM"): + cursor.fetchall.return_value = [(name,) for name, _ in selected] + else: + cursor.fetchall.return_value = [ + ( + name, + "memory " + name, + json.dumps( + {**metadata, "id": name, "memory": "memory " + name, "user_name": scope} + ), + "[1, 0, 0]" if "embedding::text" in statement else None, + ) + for name, metadata in selected + ] + + def cypher(cursor, body, parameters, columns): + assert "a.user_name = $scope" in body and "b.user_name = $scope" in body + scope, ids = parameters["scope"], set(parameters["ids"]) + result = [] + for owner, source, target, kind in reversed(edges): + if owner != scope: + continue + if "(a.id IN $ids OR b.id IN $ids)" in body: + matches = source in ids or target in ids + elif "a.id IN $ids AND b.id IN $ids" in body: + matches = source in ids and target in ids + else: + matches = source in ids + if matches: + result.append(tuple(json.dumps(value) for value in (source, target, kind))) + return result + + cursor.execute.side_effect = execute + monkeypatch.setattr(db, "_cypher", MagicMock(side_effect=cypher)) + return db, pool, cursor, nodes, edges + + +@pytest.mark.parametrize( + "source,target,depth,expected", + [ + ("a", "d", 1, []), + ("a", "d", 2, ["a", "b", "d"]), + ("a", "g", 3, []), + ("a", "g", 4, ["a", "b", "d", "f", "g"]), + ("a", "g", 5, ["a", "b", "d", "f", "g"]), + ("a", "a", 1, ["a"]), + ("a", "e", 5, []), + ("a", "z", 5, []), + ("z", "a", 5, []), + ("missing", "a", 3, []), + ("a", "missing", 3, []), + ("missing", "missing", 3, []), + ], +) +def test_directed_shortest_path_is_bounded_and_deterministic( + graph, source, target, depth, expected +): + db, pool, _, _, _ = graph + assert db.get_path(source, target, depth) == expected + pool.transaction.assert_called_once() + + +def test_equal_length_paths_compare_full_sequences_not_only_last_hop(graph): + db, _, _, _, edges = graph + edges[:] = [ + ("A", *edge) + for edge in [ + ("a", "b", "R"), + ("a", "c", "R"), + ("b", "f", "R"), + ("c", "d", "R"), + ("f", "g", "R"), + ("d", "g", "R"), + ] + ] + assert db.get_path("a", "g", 3) == ["a", "b", "f", "g"] + + +def test_path_honors_explicit_scope(graph): + db, _, _, _, _ = graph + assert db.get_path("a", "g", 1, user_name="B") == ["a", "g"] + + +def test_fifth_hop_is_included_but_sixth_is_not(graph): + db, _, _, _, edges = graph + edges[:] = [ + ("A", source, target, "R") for source, target in zip("abcdef", "bcdefg", strict=True) + ] + assert db.get_path("a", "f", 5) == ["a", "b", "c", "d", "e", "f"] + assert db.get_path("a", "g", 5) == [] + + +def test_subgraph_contains_incoming_boundary_edges_parallel_edges_and_self_loop(graph): + db, pool, _, _, _ = graph + result = db.get_subgraph("a", depth=1) + assert result == { + "core_node": {"id": "a", "memory": "memory a", "metadata": {}}, + "neighbors": [ + {"id": name, "memory": "memory " + name, "metadata": {}} + for name in ["b", "c", "d", "e"] + ], + "edges": [ + {"source": "a", "target": "b", "type": "OTHER"}, + {"source": "a", "target": "b", "type": "R"}, + {"source": "a", "target": "c", "type": "R"}, + {"source": "b", "target": "b", "type": "SELF"}, + {"source": "b", "target": "c", "type": "SIDE"}, + {"source": "b", "target": "d", "type": "R"}, + {"source": "c", "target": "d", "type": "R"}, + {"source": "d", "target": "a", "type": "R"}, + {"source": "e", "target": "a", "type": "R"}, + ], + } + pool.transaction.assert_called_once() + + +@pytest.mark.parametrize( + "depth,names", [(2, ["b", "c", "d", "e", "f"]), (5, ["b", "c", "d", "e", "f", "g"])] +) +def test_subgraph_expands_actual_depth_without_repeating_cycle(graph, depth, names): + db, _, _, _, _ = graph + result = db.get_subgraph("a", depth) + assert [node["id"] for node in result["neighbors"]] == names + assert len(result["edges"]) == (10 if depth == 2 else 11) + assert db._cypher.call_count <= depth + 1 + + +def test_isolated_subgraph_and_explicit_scope(graph): + db, _, _, _, _ = graph + assert db.get_subgraph("z") == { + "core_node": {"id": "z", "memory": "memory z", "metadata": {}}, + "neighbors": [], + "edges": [], + } + assert db.get_subgraph("a", user_name="B") == { + "core_node": {"id": "a", "memory": "memory a", "metadata": {}}, + "neighbors": [{"id": "g", "memory": "memory g", "metadata": {}}], + "edges": [{"source": "a", "target": "g", "type": "R"}], + } + + +@pytest.mark.parametrize("status", ["archived", None]) +def test_center_status_filters_center_only_and_can_be_disabled(graph, status): + db, _, _, nodes, _ = graph + nodes[("A", "a")] = {"status": status} + nodes[("A", "b")] = {"status": "deleted"} + assert db.get_subgraph("a") == {"core_node": None, "neighbors": [], "edges": []} + for required in (None, "", status): + result = db.get_subgraph("a", center_status=required) + assert result["core_node"]["metadata"] == {"status": status} + assert result["neighbors"][0]["metadata"] == {"status": "deleted"} + + +def test_missing_center_does_not_query_edges(graph): + db, _, _, _, _ = graph + assert db.get_subgraph("missing") == {"core_node": None, "neighbors": [], "edges": []} + db._cypher.assert_not_called() + + +@pytest.mark.parametrize("method,args", [("get_path", ("a", "b")), ("get_subgraph", ("a",))]) +@pytest.mark.parametrize("depth", [0, -1, 6, True, False, 1.0, "2", None, []]) +def test_invalid_depth_rejected_before_borrowing(store, method, args, depth): + db, pool, _ = store + with pytest.raises(ValueError, match="depth"): + getattr(db, method)(*args, depth) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "method,args,kwargs", + [ + ("get_path", (None, "b"), {}), + ("get_path", ("a", " "), {}), + ("get_subgraph", ("",), {}), + ("get_subgraph", ("a",), {"center_status": []}), + ("get_path", ("a", "b"), {"user_name": " "}), + ("get_subgraph", ("a",), {"user_name": 3}), + ], +) +def test_invalid_traversal_inputs_do_not_borrow(store, method, args, kwargs): + db, pool, _ = store + with pytest.raises(ValueError): + getattr(db, method)(*args, **kwargs) + pool.transaction.assert_not_called() + + +def test_frontier_values_are_bound_and_both_endpoints_scoped(store): + db, _, cursor = store + cursor.fetchall.return_value = [('"one"', '"two"', '"R"')] + scope, node_id = " B $$ ' ", "中文 $memos$ ' ; --\n" + result = db._read_edges(cursor, "a.id IN $ids", {"scope": scope, "ids": [node_id]}) + assert result == [{"from": "one", "to": "two", "type": "R"}] + calls = cursor.execute.call_args_list + prepare = next(call.args[0] for call in calls if call.args[0].startswith("PREPARE")) + assert "a.user_name = $scope" in prepare and "b.user_name = $scope" in prepare + assert node_id not in prepare and scope not in prepare + bound = next(call.args[1]["parameters"] for call in calls if call.args[0].startswith("EXECUTE")) + assert json.loads(bound) == {"scope": scope, "ids": [node_id]} + + +@pytest.mark.parametrize( + "direction,rows,expected", + [ + ("out", [('"one"', '"two"', '"R"'), ('"one"', '"one"', '"SELF"')], ["one", "two"]), + ("in", [('"two"', '"one"', '"R"')], ["two"]), + ("both", [('"two"', '"one"', '"R"'), ('"one"', '"two"', '"OTHER"')], ["two"]), + ], +) +def test_existing_neighbor_projection_survives_shared_query_refactor( + store, direction, rows, expected +): + db, _, cursor = store + cursor.fetchall.return_value = rows + assert db.get_neighbors("one", "ANY", direction) == expected + + +@pytest.mark.parametrize("method,args", [("get_path", ("a", "g")), ("get_subgraph", ("a",))]) +def test_expansion_failure_does_not_return_partial_results(graph, method, args): + db, pool, _, _, _ = graph + error = ValueError("synthetic second-hop failure") + db._cypher.side_effect = [[('"a"', '"b"', '"R"')], error] + with pytest.raises(ValueError) as caught: + getattr(db, method)(*args) + assert caught.value is error + assert pool.transaction.return_value.__exit__.call_args.args[1] is error + + +@pytest.fixture +def candidates(graph): + db, pool, cursor, nodes, _ = graph + nodes.clear() + nodes.update( + { + ("A", "b"): {"tags": ["red", "blue", "blue"]}, + ("A", "a"): {"tags": '["red", "blue"]'}, + ("A", "c"): {"tags": ["red"]}, + ("A", "archived"): {"status": "archived", "tags": ["red", "blue"]}, + ("A", "null-status"): {"status": None, "tags": ["red"]}, + ("A", "reasoning"): {"type": "reasoning", "tags": ["red"]}, + ("A", "legacy"): {"node_type": "reasoning", "tags": ["red"]}, + ("A", "working"): {"memory_type": "WorkingMemory", "tags": ["red"]}, + ("A", "no-tags"): {}, + ("A", "null-tags"): {"tags": None}, + ("B", "foreign"): {"tags": ["red", "blue"]}, + } + ) + return db, pool, cursor, nodes + + +def test_tag_overlap_rank_ties_visibility_exclusions_and_projection(candidates): + db, pool, _, _ = candidates + assert db.get_neighbors_by_tag(["red", "blue", "red"], [], top_k=2) == [ + {"id": "a", "memory": "memory a", "metadata": {"tags": '["red", "blue"]'}}, + {"id": "b", "memory": "memory b", "metadata": {"tags": ["red", "blue", "blue"]}}, + ] + assert [node["id"] for node in db.get_neighbors_by_tag(["red", "red"], ["a"])] == ["b", "c"] + assert db.get_neighbors_by_tag(["red", "red"], [], min_overlap=2) == [] + assert db.get_neighbors_by_tag(["none"], []) == [] + assert pool.transaction.call_count == 4 + + +def test_tag_explicit_scope_and_embedding_return(candidates): + db, _, _, _ = candidates + assert db.get_neighbors_by_tag(["blue"], [], include_embedding=True, user_name="B") == [ + { + "id": "foreign", + "memory": "memory foreign", + "metadata": {"tags": ["red", "blue"], "embedding": [1, 0, 0]}, + }, + ] + + +def test_tag_sql_binds_special_scope_and_excluded_ids(store): + db, _, cursor = store + cursor.fetchall.return_value = [] + scope, excluded = " 范围 ' $$ ", "节点 $memos$ ' ; --\n" + assert db.get_neighbors_by_tag(["标签 ' $$"], [excluded], user_name=scope) == [] + statement, parameters = cursor.execute.call_args.args + assert parameters == {"scope": scope, "ids": [excluded]} + assert scope not in statement and excluded not in statement + + +@pytest.mark.parametrize("tags,top_k", [([], 5), (["red"], 0)]) +def test_empty_tag_queries_do_not_borrow(store, tags, top_k): + db, pool, _ = store + assert db.get_neighbors_by_tag(tags, [], top_k=top_k) == [] + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "kwargs", + [ + {"tags": "red"}, + {"tags": None}, + {"tags": [3]}, + {"tags": [" "]}, + {"exclude_ids": "a"}, + {"exclude_ids": [None]}, + {"exclude_ids": [""]}, + {"top_k": -1}, + {"top_k": True}, + {"top_k": 1.5}, + {"min_overlap": 0}, + {"min_overlap": True}, + {"min_overlap": "1"}, + {"include_embedding": 1}, + {"user_name": ""}, + ], +) +def test_invalid_tag_arguments_reject_before_borrowing(store, kwargs): + db, pool, _ = store + with pytest.raises(ValueError): + db.get_neighbors_by_tag(**{"tags": ["red"], "exclude_ids": [], **kwargs}) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize("tags", ["not-json", '"red"', "{}", "[1]", [1], {}, 1, "[null]"]) +def test_corrupted_candidate_tags_fail_explicitly(candidates, tags): + db, _, _, nodes = candidates + nodes[("A", "a")]["tags"] = tags + with pytest.raises(GraphDBError, match="tags"): + db.get_neighbors_by_tag(["red"], []) + + +@pytest.mark.parametrize( + "method,args", + [ + ("get_path", ("a", "b")), + ("get_subgraph", ("a",)), + ("get_neighbors_by_tag", ([], [])), + ], +) +def test_closed_store_rejects_all_traversal_calls(store, method, args): + db, pool, _ = store + pool.ensure_open.side_effect = GraphDBError("closed") + with pytest.raises(GraphDBError, match="closed"): + getattr(db, method)(*args) + pool.transaction.assert_not_called() + + +@pytest.mark.parametrize( + "method,args", + [ + ("get_path", ("a", "b")), + ("get_subgraph", ("a",)), + ("get_neighbors_by_tag", (["red"], [])), + ], +) +def test_read_failures_propagate_and_leave_transaction(store, method, args): + db, pool, cursor = store + error = ValueError("synthetic SELECT failure") + cursor.execute.side_effect = error + with pytest.raises(ValueError) as caught: + getattr(db, method)(*args) + assert caught.value is error + assert pool.transaction.return_value.__exit__.call_args.args[1] is error diff --git a/tests/graph_dbs/test_opengauss_vector.py b/tests/graph_dbs/test_opengauss_vector.py new file mode 100644 index 000000000..ad70c7852 --- /dev/null +++ b/tests/graph_dbs/test_opengauss_vector.py @@ -0,0 +1,342 @@ +"""DataVec score, scope, projection and bounded candidate recovery contracts.""" + +import json + +from copy import deepcopy +from unittest.mock import MagicMock + +import pytest + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.exceptions import GraphDBError +from tests.graph_dbs.opengauss_test_support import PartialOpenGaussGraphDB + + +@pytest.fixture +def vector_store(monkeypatch, psycopg2_stub): + transaction = MagicMock() + cursor = ( + transaction.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + ) + monkeypatch.setattr(psycopg2_stub.pool, "ThreadedConnectionPool", MagicMock()) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_init_schema", lambda self: None) + monkeypatch.setattr(PartialOpenGaussGraphDB, "_get_connection", transaction) + config = OpenGaussGraphDBConfig( + host="localhost", + user="fixture", + password="synthetic-password", + db_name="fixture", + user_name="A", + embedding_dimension=3, + ) + return PartialOpenGaussGraphDB(config), transaction, cursor + + +def _hit(node_id="one", distance=0.0, scope="A", vector=None, **metadata): + return ( + node_id, + "fixture", + json.dumps({"id": node_id, "memory": "fixture", "user_name": scope, **metadata}), + vector, + scope, + distance, + ) + + +def test_vector_math_and_score_order_are_independent_of_driver_order(vector_store): + db, _, cursor = vector_store + cursor.fetchall.return_value = [ + _hit("opposite", 2), + _hit("near", 0.2), + _hit("same", 0), + _hit("orthogonal", 1), + ] + hits = db.search_by_embedding([1, 0, 0], top_k=4) + assert [hit["id"] for hit in hits] == ["same", "near", "orthogonal", "opposite"] + assert [hit["score"] for hit in hits] == pytest.approx([1, 0.9, 0.5, 0]) + assert all(set(hit) == {"id", "score"} for hit in hits) + assert cursor.fetchall.call_count == 1 + + +@pytest.mark.parametrize( + "threshold,expected", + [(None, ["a", "b"]), (0, ["a", "b"]), (0.5, ["a", "b"]), (0.500001, ["a"]), (1, ["a"])], +) +def test_threshold_is_inclusive_and_has_no_hidden_distance_cutoff( + vector_store, threshold, expected +): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit("a", 0), _hit("b", 1)] + assert [ + hit["id"] for hit in db.search_by_embedding([1, 0, 0], top_k=2, threshold=threshold) + ] == expected + + +def test_underfilled_ann_is_replaced_by_one_same_scope_exact_query(vector_store): + db, transaction, cursor = vector_store + cursor.fetchall.side_effect = [ + [_hit("old", scope="B")], + [_hit("new", 0.2, scope="B"), _hit("other", 1, scope="B")], + ] + hits = db.search_by_embedding([1, 0, 0], top_k=2, user_name="B", filter={"project": "星港"}) + assert [hit["id"] for hit in hits] == ["new", "other"] + queries = [ + call for call in cursor.execute.call_args_list if "SELECT business_id" in call.args[0] + ] + assert len(queries) == 2 + assert " + 0" not in queries[0].args[0] + assert " + 0" in queries[1].args[0] + assert queries[0].args[1] == queries[1].args[1] + assert ["B"] in queries[0].args[1].values() + transaction.assert_called_once() + + +def test_zero_hits_are_bounded_and_high_dimensions_use_exact_only(vector_store): + db, _, cursor = vector_store + cursor.fetchall.return_value = [] + assert db.search_by_embedding([1, 0, 0]) == [] + assert cursor.fetchall.call_count == 2 + cursor.reset_mock() + db.config.embedding_dimension = 2001 + assert db.search_by_embedding([1] + [0] * 2000) == [] + assert cursor.fetchall.call_count == 1 + assert " + 0" in cursor.execute.call_args.args[0] + + +def test_scopes_filters_and_values_are_bound_without_mutating_inputs(vector_store): + db, _, cursor = vector_store + cursor.fetchall.return_value = [] + arguments = { + "user_name": "B", + "cube_name": "A'中文", + "knowledgebase_ids": ["K", "K"], + "scope": "LongTermMemory", + "status": "archived", + "search_filter": {"session_id": "bound-session'"}, + "filter": {"project": "中文'; DROP TABLE Memory; --"}, + } + before = deepcopy(arguments) + db.search_by_embedding([1, 0, 0], **arguments) + statement, params = cursor.execute.call_args.args + assert "scope = ANY(%(scopes)s)" in statement + assert "vector_norm(embedding) > 0" in statement + assert ["A'中文", "K"] in params.values() + for literal in ("LongTermMemory", "archived", "bound-session'", "中文'; DROP TABLE Memory; --"): + assert literal not in statement + assert ( + json.dumps(literal, ensure_ascii=False) in params.values() + or json.dumps(literal) in params.values() + ) + assert arguments == before + + +@pytest.mark.parametrize( + "fields,include,keys", + [ + (None, False, []), + ([], False, []), + (["id", "score", "tags", "tags", "absent"], False, ["tags", "absent"]), + (["embedding"], False, ["embedding"]), + (None, True, ["embedding"]), + ], +) +def test_projection_deduplicates_and_protects_identity_score(vector_store, fields, include, keys): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit(vector="[1,0,0]", tags=["x"], score=99)] + hit = db.search_by_embedding( + [1, 0, 0], top_k=1, return_fields=fields, include_embedding=include + )[0] + assert hit["id"] == "one" and hit["score"] == 1 + assert set(hit) == {"id", "score", *keys} + if "embedding" in keys: + assert hit["embedding"] == [1, 0, 0] + if "absent" in keys: + assert hit["absent"] is None + + +def test_projected_payload_keeps_json_types_and_scope(vector_store): + db, _, cursor = vector_store + metadata = { + "info": {"fraction": 0.8}, + "sources": [{"content": "fixture"}], + "memory_type": "Context", + } + cursor.fetchall.return_value = [_hit(scope="K", **metadata)] + hit = db.search_by_embedding( + [1, 0, 0], + top_k=1, + knowledgebase_ids=["K"], + return_fields=["memory", "user_name", *metadata], + )[0] + assert hit == {"id": "one", "score": 1, "memory": "fixture", "user_name": "K", **metadata} + + +@pytest.mark.parametrize( + "vector", + [ + None, + [], + (1, 0, 0), + "[1,0,0]", + [1, 0], + [1, 0, 0, 0], + [0, 0, 0], + [True, 0, 0], + ["1", 0, 0], + [float("nan"), 0, 0], + [float("inf"), 0, 0], + [10**400, 0, 0], + ], +) +def test_invalid_query_vectors_fail_before_borrowing(vector_store, vector): + db, transaction, _ = vector_store + with pytest.raises(ValueError): + db.search_by_embedding(vector) + transaction.assert_not_called() + + +@pytest.mark.parametrize( + "kwargs", + [ + {"top_k": -1}, + {"top_k": True}, + {"top_k": 1.5}, + {"threshold": -0.1}, + {"threshold": 1.1}, + {"threshold": float("nan")}, + {"threshold": float("inf")}, + {"threshold": "0.5"}, + {"threshold": True}, + {"scope": ""}, + {"scope": False}, + {"status": ""}, + {"user_name": ""}, + {"cube_name": " "}, + {"knowledgebase_ids": "K"}, + {"search_filter": []}, + {"search_filter": {"bad;": 1}}, + {"filter": {"or": []}}, + {"return_fields": "memory"}, + {"return_fields": [None]}, + {"return_fields": ["memory; DROP TABLE x"]}, + {"include_embedding": "false"}, + ], +) +def test_invalid_arguments_even_with_zero_limit_fail_before_borrowing(vector_store, kwargs): + db, transaction, _ = vector_store + with pytest.raises(ValueError): + db.search_by_embedding([1, 0, 0], **{"top_k": 0, **kwargs}) + transaction.assert_not_called() + + +@pytest.mark.parametrize("scale", [1e-300, 1e300]) +def test_finite_extreme_query_is_scaled_without_mutation(vector_store, scale): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit()] + vector = [scale, 0, 0] + assert db.search_by_embedding(vector, top_k=1) == [{"id": "one", "score": 1}] + assert vector == [scale, 0, 0] + assert json.loads(cursor.execute.call_args.args[1]["vector"]) == [1, 0, 0] + + +def test_zero_limit_and_closed_store(vector_store): + db, transaction, _ = vector_store + assert db.search_by_embedding([1, 0, 0], top_k=0) == [] + transaction.assert_not_called() + db.close() + with pytest.raises(GraphDBError, match="closed"): + db.search_by_embedding([1, 0, 0], top_k=0) + + +@pytest.mark.parametrize("distance", [None, "bad", float("nan"), float("inf"), -1, 3]) +def test_corrupt_distance_is_not_a_successful_empty_result(vector_store, distance): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit(distance=distance)] + with pytest.raises(GraphDBError): + db.search_by_embedding([1, 0, 0], top_k=1) + + +@pytest.mark.parametrize( + "vector", ["broken", "{}", "null", [True, 0, 0], [float("nan"), 0, 0], [10**400, 0, 0], 3] +) +def test_corrupt_projected_vector_is_explicit_error(vector_store, vector): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit(vector=vector)] + with pytest.raises(GraphDBError): + db.search_by_embedding([1, 0, 0], top_k=1, include_embedding=True) + + +@pytest.mark.parametrize( + "vector,expected", + [("[1,0,0]", [1, 0, 0]), ([1, 0, 0], [1, 0, 0]), ((1, 0, 0), [1, 0, 0]), (None, None)], +) +def test_supported_vector_shapes_are_decoded(vector_store, vector, expected): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit(vector=vector)] + assert ( + db.search_by_embedding([1, 0, 0], top_k=1, include_embedding=True)[0]["embedding"] + == expected + ) + + +def test_candidate_ties_keep_distinct_cube_identities(vector_store): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit("same", scope="K"), _hit("same", scope="A")] + hits = db.search_by_embedding( + [1, 0, 0], top_k=2, knowledgebase_ids=["K"], return_fields=["user_name"] + ) + assert [hit["user_name"] for hit in hits] == ["A", "K"] + + +@pytest.mark.parametrize("cube,expected", [(None, "B"), ("K", "K")]) +def test_node_completion_uses_the_same_cube_alias(vector_store, cube, expected): + db, _, cursor = vector_store + cursor.fetchall.return_value = [_hit(scope=expected)[:4]] + assert db.get_node("one", user_name="B", cube_name=cube)["id"] == "one" + assert cursor.execute.call_args.args[1]["scope"] == expected + + +INDEX_ROW = ("memos_graph", "Memory", "hnsw", "vector_cosine_ops", "6", True, True, None, None) + + +@pytest.mark.parametrize("existing", [None, INDEX_ROW]) +def test_vector_index_is_created_or_verified(vector_store, existing): + db, _, cursor = vector_store + cursor.fetchone.side_effect = [existing, INDEX_ROW] if existing is None else [existing] + db._init_vector_index(cursor) + queries = [call.args[0] for call in cursor.execute.call_args_list] + assert sum("CREATE INDEX" in query for query in queries) == (existing is None) + assert not any("DROP" in query for query in queries) + + +@pytest.mark.parametrize( + "offset,value", + [ + (0, "other"), + (1, "Other"), + (2, "btree"), + (3, "vector_l2_ops"), + (4, "5"), + (5, False), + (6, False), + (7, "predicate"), + (8, "expression"), + ], +) +def test_incompatible_vector_index_is_rejected_without_replacement(vector_store, offset, value): + db, _, cursor = vector_store + row = list(INDEX_ROW) + row[offset] = value + cursor.fetchone.return_value = tuple(row) + with pytest.raises(GraphDBError, match="index"): + db._init_vector_index(cursor) + assert not any( + "CREATE" in call.args[0] or "DROP" in call.args[0] for call in cursor.execute.call_args_list + ) + + +def test_high_dimension_storage_does_not_attempt_unsupported_hnsw(vector_store): + db, _, cursor = vector_store + db.config.embedding_dimension = 3072 + db._init_vector_index(cursor) + cursor.execute.assert_not_called() diff --git a/tests/mem_cube/test_close_diagnostics.py b/tests/mem_cube/test_close_diagnostics.py new file mode 100644 index 000000000..3143e5553 --- /dev/null +++ b/tests/mem_cube/test_close_diagnostics.py @@ -0,0 +1,30 @@ +"""Every failed owned resource remains observable and retryable.""" + +from unittest.mock import Mock + +import pytest + +from memos.lifecycle import CloseState +from memos.mem_cube.general import GeneralMemCube + + +def test_multiple_cleanup_failures_are_logged_and_only_failures_retry(monkeypatch): + log = Mock() + monkeypatch.setattr("memos.mem_cube.general.logger", log) + first, second = ValueError("first failure"), OSError("second failure") + memories = [ + Mock(close=Mock(side_effect=[first, None])), + Mock(), + Mock(close=Mock(side_effect=[second, None])), + ] + cube = GeneralMemCube.__new__(GeneralMemCube) + cube._close_state = CloseState() + cube._owned_memories = list(memories) + with pytest.raises(ValueError) as caught: + cube.close() + assert caught.value is first + assert log.exception.call_count == 2 + assert cube._owned_memories == [memories[0], memories[2]] + cube.close() + assert [m.close.call_count for m in memories] == [2, 1, 2] + assert cube._owned_memories == [] diff --git a/tests/mem_feedback/test_feedback.py b/tests/mem_feedback/test_feedback.py index 3f86ac482..a06576e7f 100644 --- a/tests/mem_feedback/test_feedback.py +++ b/tests/mem_feedback/test_feedback.py @@ -33,6 +33,7 @@ def test_process_feedback_runs_answer_and_core_workflows(): assert result == { "answer": "corrected answer", + "answer_status": "generated", "record": {"add": [{"id": "memory-1"}], "update": []}, } feedback._generate_answer.assert_called_once_with( diff --git a/tests/mem_feedback/test_feedback_outcomes.py b/tests/mem_feedback/test_feedback_outcomes.py new file mode 100644 index 000000000..3f3076c72 --- /dev/null +++ b/tests/mem_feedback/test_feedback_outcomes.py @@ -0,0 +1,94 @@ +"""Feedback is a memory operation with an optional, independently generated answer.""" + +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError, LLMError, MemOSError +from memos.mem_feedback.simple_feedback import SimpleMemFeedback + + +@pytest.fixture +def feedback(): + instance = SimpleMemFeedback(*(Mock() for _ in range(7))) + instance._generate_answer = Mock(return_value="Corrected answer") + instance.process_feedback_core = Mock( + return_value={"record": {"add": [{"id": "created"}], "update": [{"id": "updated"}]}} + ) + return instance + + +@pytest.mark.parametrize("failure", [ValueError("model"), TimeoutError("model"), OSError("model")]) +def test_optional_answer_failure_preserves_committed_records(feedback, failure): + feedback._generate_answer.side_effect = failure + result = feedback.process_feedback("u", "cube", [], "feedback", corrected_answer=True) + assert result == { + "answer": "", + "answer_status": "unavailable", + "record": {"add": [{"id": "created"}], "update": [{"id": "updated"}]}, + } + feedback.process_feedback_core.assert_called_once() + + +@pytest.mark.parametrize("answer", [None, "", " ", {}, "Corrected answer"]) +def test_answer_status_is_machine_readable(feedback, answer): + feedback._generate_answer.return_value = answer + result = feedback.process_feedback("u", "cube", [], "feedback", corrected_answer=True) + valid = answer == "Corrected answer" + assert result["answer_status"] == ("generated" if valid else "unavailable") + assert result["answer"] == (answer if valid else "") + + +def test_no_answer_requested_does_not_call_model(feedback): + result = feedback.process_feedback("u", "cube", [], "feedback") + assert result["answer_status"] == "not_requested" and result["answer"] == "" + feedback._generate_answer.assert_not_called() + + +@pytest.mark.parametrize( + "failure", [GraphDBError("storage"), ValueError("storage input"), OSError("core")] +) +def test_core_failure_wins_even_if_answer_also_fails(feedback, failure, caplog): + feedback._generate_answer.side_effect = ValueError("optional answer") + feedback.process_feedback_core.side_effect = failure + with pytest.raises(type(failure)) as caught: + feedback.process_feedback("u", "cube", [], "feedback", corrected_answer=True) + assert caught.value is failure + assert "answer" in caplog.text.lower() + feedback._generate_answer.assert_called_once() + feedback.process_feedback_core.assert_called_once() + + +def test_invalid_model_memory_is_not_reported_as_caller_value_error(): + feedback = SimpleMemFeedback(*(Mock() for _ in range(7))) + feedback._keyword_replace_judgement = Mock(return_value=None) + feedback._feedback_judgement = Mock( + return_value=[{"validity": "true", "corrected_info": "fact", "key": "k", "tags": 17}] + ) + feedback._batch_embed = Mock(return_value=[[1.0]]) + feedback._feedback_memory = Mock() + with pytest.raises(MemOSError, match="model"): + feedback.process_feedback_core("u", "cube", [{"role": "user", "content": "old"}], "new") + feedback._feedback_memory.assert_not_called() + + +def test_core_unexpected_failure_is_not_empty_success(): + feedback = SimpleMemFeedback(*(Mock() for _ in range(7))) + feedback._keyword_replace_judgement = Mock(side_effect=OSError("required operation")) + with pytest.raises(MemOSError): + feedback.process_feedback_core("u", "cube", [], "feedback") + + +@pytest.mark.parametrize("outcome", [None, {}, ValueError("invalid model JSON")]) +def test_required_judgement_failure_is_not_a_noop(outcome): + feedback = SimpleMemFeedback(*(Mock() for _ in range(7))) + feedback._keyword_replace_judgement = Mock(return_value=None) + feedback._feedback_judgement = Mock(return_value=outcome) + if isinstance(outcome, Exception): + feedback._feedback_judgement.side_effect = outcome + feedback._feedback_memory = Mock() + with pytest.raises(LLMError, match="model"): + feedback.process_feedback_core( + "u", "cube", [{"role": "user", "content": "history"}], "feedback" + ) + feedback._feedback_memory.assert_not_called() diff --git a/tests/mem_feedback/test_feedback_storage_contract.py b/tests/mem_feedback/test_feedback_storage_contract.py new file mode 100644 index 000000000..d13a0dc03 --- /dev/null +++ b/tests/mem_feedback/test_feedback_storage_contract.py @@ -0,0 +1,238 @@ +"""Regression branches around the Feedback boundaries changed for storage errors.""" + +from threading import Event +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError, MemOSError +from memos.mem_feedback.simple_feedback import SimpleMemFeedback +from memos.memories.textual.item import TextualMemoryItem +from memos.memories.textual.tree_text_memory.organize.manager import MemoryManager + + +@pytest.fixture +def feedback(): + f = SimpleMemFeedback(Mock(), Mock(), Mock(), Mock(), Mock(), Mock(), Mock()) + f._generate_answer = Mock(return_value="answer") + f._keyword_replace_judgement = Mock(return_value=None) + f._pure_add = Mock(return_value={"record": {"add": ["new"], "update": []}}) + f._feedback_judgement = Mock(return_value=None) + f._batch_embed = Mock(side_effect=lambda texts: [[1, 0, 0] for _ in texts]) + f._feedback_memory = Mock(return_value={"record": {"add": [], "update": ["changed"]}}) + return f + + +@pytest.mark.parametrize( + "judge", + [ + None, + {"if_keyword_replace": "false"}, + {"if_keyword_replace": "true", "original": "NONE"}, + {"if_keyword_replace": "true", "original": "word", "target": "NONE"}, + ], +) +def test_nonkeyword_pure_add_contract(feedback, judge): + feedback._keyword_replace_judgement.return_value = judge + assert feedback.process_feedback_core( + "u", "A", [], "fact", {"agent_id": "a"}, feedback_time="2026-01-01" + )["record"]["add"] == ["new"] + + +@pytest.mark.parametrize( + "judgements,expected", + [ + ([], []), + ([{"validity": "false", "user_attitude": "irrelevant"}], ["new"]), + ([{"validity": "false", "user_attitude": "negative"}], []), + ([{"validity": "true"}], []), + ([{"validity": "true", "corrected_info": ""}], []), + ([{"validity": "true", "corrected_info": "fact"}], []), + ([{"validity": "true", "corrected_info": "fact", "key": "k"}], []), + ], +) +def test_core_no_valid_feedback_or_irrelevant(feedback, judgements, expected): + feedback._feedback_judgement.return_value = judgements + result = feedback.process_feedback_core( + "u", "A", [{"role": "user", "content": "history"}], "feedback" + ) + assert result["record"]["add"] == expected and result["record"]["update"] == [] + + +def test_core_valid_memory_and_empty_input(feedback): + assert feedback.process_feedback_core("u", "A", [], " ") == { + "record": {"add": [], "update": []} + } + feedback._feedback_judgement.return_value = [ + {"validity": "true", "corrected_info": "fact", "key": "k", "tags": ["x"]}, + {}, + ] + result = feedback.process_feedback_core( + "u", + "A", + [{"role": "user", "content": "history"}], + "feedback", + feedback_time="2026-01-01T00:00:00", + session_id="s", + ) + assert result == {"record": {"add": [], "update": ["changed"]}} + mem = feedback._feedback_memory.call_args.args[2][0] + assert ( + mem.memory == "fact" and mem.metadata.tags == ["x"] and mem.metadata.embedding == [1, 0, 0] + ) + feedback._keyword_replace_judgement.side_effect = RuntimeError("model unavailable") + with pytest.raises(MemOSError, match="core"): + feedback.process_feedback_core("u", "A", [], "feedback") + + +@pytest.mark.parametrize("error", [TimeoutError("timeout"), RuntimeError("model unavailable")]) +def test_feedback_core_errors_do_not_become_empty_records(feedback, error): + feedback.process_feedback_core = Mock(side_effect=error) + with pytest.raises(type(error)) as caught: + feedback.process_feedback("u", "A", [], "feedback") + assert caught.value is error + + +def test_feedback_answer_and_core_execute_concurrently(feedback): + answer_started, core_started = Event(), Event() + + def answer(*args, **kwargs): + answer_started.set() + assert core_started.wait(3) + return "answer" + + def core(*args, **kwargs): + core_started.set() + assert answer_started.wait(3) + return {"record": {"add": ["stored"], "update": []}} + + feedback._generate_answer = answer + feedback.process_feedback_core = core + result = feedback.process_feedback("u", "A", [], "feedback", corrected_answer=True) + assert result == { + "answer": "answer", + "answer_status": "generated", + "record": {"add": ["stored"], "update": []}, + } + + +@pytest.mark.parametrize( + "original,tokenized,has_hits", [("old word", ["old", "word"], False), ("旧词", ["旧词"], True)] +) +def test_existing_tfidf_branch_is_only_stub_regression( + feedback, original, tokenized, has_hits, monkeypatch +): + feedback.DB_IDX_READY = True + monkeypatch.setattr( + "memos.mem_feedback.feedback.detect_lang", lambda text: "zh" if original == "旧词" else "en" + ) + feedback._tokenize_chinese = Mock(return_value=tokenized) + feedback.graph_store.search_by_keywords_tfidf.return_value = [{"id": "one"}] if has_hits else [] + feedback.graph_store.search_by_fulltext.return_value = [] + memory = TextualMemoryItem(memory=original, metadata={"tags": [original, "new"]}) + feedback.graph_store.get_node.return_value = memory.model_dump() + feedback._single_update_operation = Mock(return_value={"id": memory.id}) + feedback._doc_filter = Mock(side_effect=lambda scope, memories: memories) + result = feedback.process_keyword_replace( + "u", "A", {"original": original, "target": "new", "doc_scope": "doc"} + ) + assert len(result["record"]["update"]) == int(has_hits) + feedback.graph_store.search_by_keywords_like.assert_not_called() + + +@pytest.mark.parametrize( + "text,tags,fail", + [("no match", [], False), ("old text", [], False), ("old text", ["old", "new"], True)], +) +def test_keyword_replace_filter_tags_and_required_update_failure(feedback, text, tags, fail): + memory = TextualMemoryItem(memory=text, metadata={"tags": tags}) + feedback.graph_store.search_by_keywords_like.return_value = [{"id": memory.id}] + feedback.graph_store.get_node.return_value = memory.model_dump() + feedback._single_update_operation = ( + Mock(side_effect=RuntimeError("optional failure")) + if fail + else Mock(return_value={"id": memory.id}) + ) + if fail: + with pytest.raises(RuntimeError): + feedback.process_keyword_replace( + "u", "A", {"original": "old", "target": "new"}, info={"ignored": 1} + ) + else: + result = feedback.process_keyword_replace( + "u", "A", {"original": "old", "target": "new"}, info={"ignored": 1} + ) + assert len(result["record"]["update"]) == int("old" in text) + if "old" in text: + new = feedback._single_update_operation.call_args.args[1] + assert new.memory == "new text" and new.metadata.tags == ["new"] + + +@pytest.fixture +def manager(): + m = MemoryManager.__new__(MemoryManager) + m.reorganizer = Mock() + m.memory_size = {"WorkingMemory": 20} + m.graph_store = Mock() + m.is_reorganize = False + return m + + +@pytest.mark.parametrize( + "kind", ["WorkingMemory", "OuterMemory", "Context", "RawFileMemory", "LongTermMemory"] +) +def test_manager_selected_kinds(manager, kind): + memory = TextualMemoryItem(memory="one", metadata={"memory_type": kind}) + expected = [memory.id] if kind in {"RawFileMemory", "LongTermMemory"} else [] + assert manager._process_memory(memory, "A") == expected + if kind == "Context": + manager.graph_store.add_nodes_batch.assert_not_called() + else: + assert manager.graph_store.add_nodes_batch.call_args.kwargs == {"user_name": "A"} + + +@pytest.mark.parametrize( + "failure", [GraphDBError("storage"), ValueError("input"), RuntimeError("driver")] +) +def test_manager_update_failure_kind(manager, failure): + manager.graph_store.add_nodes_batch.side_effect = failure + memory = TextualMemoryItem(memory="one", metadata={"memory_type": "LongTermMemory"}) + with pytest.raises(type(failure)) as caught: + manager._process_memory(memory, "A") + assert caught.value is failure + + +def test_manager_parallel_failure_and_empty(manager): + manager._process_memory = Mock(return_value=["id"]) + assert manager._add_memories_parallel([Mock()], "A") == ["id"] + assert manager._add_memories_parallel([], "A") == [] + manager._process_memory.side_effect = RuntimeError("write") + with pytest.raises(RuntimeError): + manager._add_memories_parallel([Mock()], "A") + manager._cleanup_working_memory("A") + manager.graph_store.remove_oldest_memory.side_effect = RuntimeError("optional") + manager._cleanup_working_memory("A") + + +def test_scheduler_feedback_optional_errors_and_empty_batch(monkeypatch): + from memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler import ( + FeedbackMessageHandler, + ) + + handler = FeedbackMessageHandler(Mock()) + handler.process_single_feedback = Mock() + handler.batch_handler("u", "A", [Mock()]) + handler.batch_handler("u", "A", []) + handler.process_single_feedback.side_effect = RuntimeError("optional") + with pytest.raises(RuntimeError): + handler.batch_handler("u", "A", [Mock()]) + monkeypatch.setattr( + "memos.mem_scheduler.task_schedule_modules.base_handler.group_messages_by_user_and_mem_cube", + lambda **kw: {"u": {"empty": [], "A": [Mock()]}}, + ) + operation = Mock(side_effect=RuntimeError("optional")) + handler.handle_exception = Mock() + handler.process_grouped_messages([], operation) + handler.handle_exception.assert_called_once() + operation.side_effect = None + handler.process_grouped_messages([], operation) diff --git a/tests/mem_feedback/test_feedback_task_records.py b/tests/mem_feedback/test_feedback_task_records.py new file mode 100644 index 000000000..891d5fd2e --- /dev/null +++ b/tests/mem_feedback/test_feedback_task_records.py @@ -0,0 +1,84 @@ +"""Successful Feedback tasks preserve existing knowledge-base event formats.""" + +import json + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem +from memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler import ( + FeedbackMessageHandler, +) + + +@pytest.mark.parametrize("playground", [False, True]) +@pytest.mark.parametrize("decoded", [False, True]) +def test_feedback_records_emit_only_complete_changes(monkeypatch, playground, decoded): + monkeypatch.setattr( + "memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler.is_playground_api", + lambda: playground, + ) + context = Mock() + context.get_feedback_server.return_value.process_feedback.return_value = { + "record": { + "add": [ + {"id": "a", "memory": "one", "source_doc_id": "doc"}, + {"id": "b", "text": "two"}, + SimpleNamespace(id="c", memory="three", origin_memory="old"), + {"id": "missing-memory"}, + {"memory": "missing-id"}, + ], + "update": [ + {"id": "d", "memory": "after", "origin_memory": "before"}, + {"id": "e", "text": "after", "old_memory": "before"}, + SimpleNamespace(id="f", memory="after", origin_memory="before"), + {"id": "g", "text": "after", "original_content": "before"}, + {"id": "missing-memory"}, + {}, + ], + } + } + handler = FeedbackMessageHandler(context) + payload = {"feedback_content": "change", "task_id": "business"} + message = ScheduleMessageItem( + user_id="u", mem_cube_id="A", label="mem_feedback", content=json.dumps(payload) + ) + if decoded: + message = message.model_copy(update={"content": payload}) + handler.process_single_feedback(message) + if playground: + context.services.create_event_log.assert_not_called() + else: + event = context.services.create_event_log.call_args.kwargs + changes = event["memcube_log_content"] + assert [item["memory_id"] for item in changes] == list("abcdefg") + assert [item["operation"] for item in changes] == ["ADD"] * 3 + ["UPDATE"] * 4 + assert all(item["original_content"] == "before" for item in changes[3:]) + assert changes[0]["source_doc_id"] == "doc" + assert event["memory_len"] == 7 + assert context.services.create_event_log.return_value.task_id == "business" + context.services.submit_web_logs.assert_called_once() + + +@pytest.mark.parametrize( + "result", [None, {}, {"record": None}, {"record": {"add": None, "update": []}}] +) +def test_empty_feedback_records_emit_no_content_event(monkeypatch, result): + monkeypatch.setattr( + "memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler.is_playground_api", + lambda: False, + ) + context = Mock() + context.get_feedback_server.return_value.process_feedback.return_value = result + handler = FeedbackMessageHandler(context) + message = ScheduleMessageItem( + user_id="u", mem_cube_id="A", label="mem_feedback", content="{}", task_id="fallback" + ) + handler.process_single_feedback(message) + assert ( + context.get_feedback_server.return_value.process_feedback.call_args.kwargs["task_id"] + == "fallback" + ) + context.services.create_event_log.assert_not_called() diff --git a/tests/mem_feedback/test_storage_errors.py b/tests/mem_feedback/test_storage_errors.py new file mode 100644 index 000000000..7290a018d --- /dev/null +++ b/tests/mem_feedback/test_storage_errors.py @@ -0,0 +1,153 @@ +"""Required Feedback storage failures remain visible across all boundaries.""" + +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError +from memos.mem_feedback.simple_feedback import SimpleMemFeedback +from memos.memories.textual.item import TextualMemoryItem +from memos.memories.textual.tree_text_memory.organize.manager import MemoryManager + + +@pytest.fixture +def feedback(): + fb = SimpleMemFeedback(Mock(), Mock(), Mock(), Mock(), Mock(), Mock(), Mock()) + fb._generate_answer = Mock(return_value="answer") + fb._keyword_replace_judgement = Mock( + return_value={"if_keyword_replace": "true", "original": "old", "target": "new"} + ) + return fb + + +@pytest.mark.parametrize("failure", [GraphDBError("storage"), ValueError("input")]) +@pytest.mark.parametrize("stage", ["read", "update"]) +def test_feedback_default_errors_reach_caller(feedback, failure, stage): + if stage == "read": + feedback.graph_store.search_by_keywords_like.side_effect = failure + else: + memory = TextualMemoryItem(memory="old word", metadata={"tags": ["old"]}) + feedback.graph_store.search_by_keywords_like.return_value = [{"id": memory.id}] + feedback.graph_store.get_node.return_value = memory.model_dump() + feedback._batch_embed = Mock(return_value=[[1, 0, 0]]) + feedback._single_update_operation = Mock(side_effect=failure) + with pytest.raises(type(failure)) as caught: + feedback.process_feedback("u", "A", [], "replace old by new") + assert caught.value is failure + + +@pytest.mark.parametrize( + "method", ["_add_memories_parallel", "_process_memory", "_cleanup_working_memory"] +) +@pytest.mark.parametrize("failure", [GraphDBError("storage"), ValueError("input")]) +def test_manager_required_failure_reaches_feedback(method, failure): + manager = MemoryManager.__new__(MemoryManager) + manager.reorganizer = Mock() + manager.memory_size = {"WorkingMemory": 20} + manager.graph_store = Mock(remove_oldest_memory=Mock(side_effect=failure)) + manager.graph_store.add_nodes_batch.side_effect = failure + manager.is_reorganize = False + memory = TextualMemoryItem(memory="old", metadata={"memory_type": "LongTermMemory"}) + if method == "_add_memories_parallel": + manager._process_memory = Mock(side_effect=failure) + args = ( + [] + if method == "_cleanup_working_memory" + else [memory if method == "_process_memory" else [memory]] + ) + with pytest.raises(type(failure)) as caught: + getattr(manager, method)(*args, user_name="A") + assert caught.value is failure + + +def test_working_and_graph_are_submitted_in_one_write(): + manager = MemoryManager(Mock(), Mock(), Mock()) + memory = TextualMemoryItem(memory="one", metadata={"memory_type": "LongTermMemory"}) + assert manager._process_memory(memory, "A") == [memory.id] + manager.graph_store.add_node.assert_not_called() + manager.graph_store.add_nodes_batch.assert_called_once() + working, graph = manager.graph_store.add_nodes_batch.call_args.args[0] + assert working["id"] != graph["id"] == memory.id + assert graph["metadata"]["working_binding"] == working["id"] + + +@pytest.mark.parametrize("error", [GraphDBError("uncertain write"), ValueError("invalid")]) +def test_storage_write_is_not_retried_or_wrapped(feedback, error): + operation = Mock(side_effect=error) + with pytest.raises(type(error)) as caught: + feedback._retry_db_operation(operation) + assert caught.value is error + operation.assert_called_once() + + +def test_keyword_update_unexpected_failure_cannot_become_empty_success(feedback): + memory = TextualMemoryItem(memory="old word", metadata={"tags": ["old"]}) + feedback.graph_store.search_by_keywords_like.return_value = [{"id": memory.id}] + feedback.graph_store.get_node.return_value = memory.model_dump() + feedback._batch_embed = Mock(return_value=[[1, 0, 0]]) + feedback._single_update_operation = Mock(side_effect=OSError("failed update")) + with pytest.raises(OSError): + feedback.process_keyword_replace("u", "A", {"original": "old", "target": "new"}) + + +@pytest.mark.parametrize("error", [GraphDBError("storage"), ValueError("input"), OSError("core")]) +def test_scheduler_feedback_batch_does_not_report_storage_success(error): + from memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler import ( + FeedbackMessageHandler, + ) + + handler = FeedbackMessageHandler.__new__(FeedbackMessageHandler) + handler.process_single_feedback = Mock(side_effect=error) + with pytest.raises(type(error)) as caught: + handler.batch_handler("u", "A", [Mock()]) + assert caught.value is error + + +@pytest.mark.parametrize("error", [GraphDBError("storage"), ValueError("input"), OSError("core")]) +def test_scheduler_group_boundary_does_not_swallow_required_error(error): + from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem + from memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler import ( + FeedbackMessageHandler, + ) + + handler = FeedbackMessageHandler(Mock()) + handler.batch_handler = Mock(side_effect=error) + msg = ScheduleMessageItem(user_id="u", mem_cube_id="A", label="mem_feedback", content="{}") + with pytest.raises(type(error)) as caught: + handler([msg]) + assert caught.value is error + + +@pytest.mark.parametrize("content", ["{", "[]", '{"feedback_content": "change"}']) +def test_feedback_failure_marks_dispatcher_task_failed(content): + from memos.mem_scheduler.base_scheduler import BaseScheduler + from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem + from memos.mem_scheduler.schemas.task_schemas import RunningTaskItem + from memos.mem_scheduler.task_schedule_modules.handlers.feedback_handler import ( + FeedbackMessageHandler, + ) + + scheduler = BaseScheduler({"use_redis_queue": False}) + tracker = Mock() + scheduler.status_tracker = tracker + scheduler.dispatcher._maybe_emit_task_completion = Mock() + context = Mock() + context.get_feedback_server.return_value.process_feedback.side_effect = GraphDBError( + "write failed" + ) + handler = FeedbackMessageHandler(context) + message = ScheduleMessageItem( + user_id="u", mem_cube_id="A", label="mem_feedback", content=content + ) + task = RunningTaskItem( + user_id="u", mem_cube_id="A", task_info="feedback", task_name="feedback", messages=[message] + ) + scheduler.dispatcher._running_tasks[task.item_id] = task + try: + with pytest.raises((GraphDBError, ValueError)): + scheduler.dispatcher._create_task_wrapper(handler, task)([message]) + assert task.status == "failed" + tracker.task_failed.assert_called_once() + tracker.task_completed.assert_not_called() + finally: + scheduler.close() diff --git a/tests/mem_scheduler/process_support.py b/tests/mem_scheduler/process_support.py new file mode 100644 index 000000000..a53d4cb97 --- /dev/null +++ b/tests/mem_scheduler/process_support.py @@ -0,0 +1,63 @@ +"""Bounded process probes with separate startup and behavior deadlines.""" + +import multiprocessing +import traceback + +from contextlib import contextmanager +from multiprocessing.connection import wait + +import pytest + + +# Cold imports load the framework and optional libraries on CI runners. This +# budget does not extend the shorter task/drain/deadlock checks after readiness. +STARTUP_TIMEOUT = 60 + + +def _run_probe(connection, target, args): + try: + target(connection, *args) + except BaseException: + connection.send(("error", traceback.format_exc())) + raise + finally: + connection.close() + + +@contextmanager +def spawned_probe(target, *args): + context = multiprocessing.get_context("spawn") + connection, child_connection = context.Pipe() + process = context.Process(target=_run_probe, args=(child_connection, target, args)) + try: + process.start() + child_connection.close() + yield process, connection + finally: + child_connection.close() + connection.close() + if process.pid is not None: + if process.is_alive(): + process.terminate() + process.join(5) + if process.is_alive(): + process.kill() + process.join(5) + assert not process.is_alive(), f"Probe process {process.pid} could not be reaped" + process.close() + + +def receive_probe_message(connection, process, *, phase, timeout): + ready = wait([connection, process.sentinel], timeout=timeout) + if connection in ready: + try: + message = connection.recv() + except EOFError: + pytest.fail(f"{phase}: child {process.pid} closed its pipe without a result") + if message[0] == "error": + pytest.fail(f"{phase}: child {process.pid} failed:\n{message[1]}", pytrace=False) + return message + if process.sentinel in ready: + process.join() + pytest.fail(f"{phase}: child {process.pid} exited with code {process.exitcode}") + pytest.fail(f"{phase}: child {process.pid} did not respond within {timeout}s") diff --git a/tests/mem_scheduler/test_borrowed_feedback.py b/tests/mem_scheduler/test_borrowed_feedback.py new file mode 100644 index 000000000..6a0c6a866 --- /dev/null +++ b/tests/mem_scheduler/test_borrowed_feedback.py @@ -0,0 +1,125 @@ +"""Borrowing one component must not suppress initialization of another.""" + +from concurrent.futures import ThreadPoolExecutor +from threading import Event +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.exceptions import ConfigurationError, MemOSError +from memos.mem_scheduler.base_scheduler import BaseScheduler + + +@pytest.fixture +def scheduler(): + scheduler = BaseScheduler({"use_redis_queue": False, "thread_pool_max_workers": 2}) + yield scheduler + scheduler.close() + + +def test_borrowed_cube_builds_feedback_from_same_dependencies(scheduler, monkeypatch): + default_factory = Mock(side_effect=AssertionError("must not create a default cube")) + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", default_factory) + text = SimpleNamespace( + extractor_llm=Mock(), + embedder=Mock(), + graph_store=Mock(), + memory_manager=Mock(), + reranker=Mock(), + ) + cube = SimpleNamespace(text_mem=text, close=Mock()) + scheduler.mem_reader = Mock() + scheduler.searcher = Mock() + scheduler.mem_cube = cube + feedback = scheduler.feedback_server + assert feedback is not None + assert feedback is scheduler.feedback_server + assert feedback.graph_store is text.graph_store + assert feedback.memory_manager is text.memory_manager + assert feedback.llm is text.extractor_llm + assert feedback.embedder is text.embedder + assert feedback.mem_reader is scheduler.mem_reader + assert feedback.searcher is scheduler.searcher + assert scheduler.mem_cube is cube and scheduler.components is None + scheduler.close() + cube.close.assert_not_called() + text.graph_store.close.assert_not_called() + default_factory.assert_not_called() + + +def test_missing_dependencies_fail_explicitly_and_can_be_supplied(scheduler): + cube = SimpleNamespace(text_mem=SimpleNamespace()) + scheduler.mem_cubes = {"borrowed": cube} + with pytest.raises(ConfigurationError, match="Feedback"): + _ = scheduler.feedback_server + assert scheduler.mem_cube is cube and scheduler.components is None + external = Mock() + scheduler.feedback_server = external + assert scheduler.feedback_server is external + scheduler.close() + external.close.assert_not_called() + + +def test_borrowed_feedback_survives_owned_cube_initialization(scheduler, monkeypatch): + external = Mock() + owned = {"naive_mem_cube": Mock(), "feedback_server": Mock(), "graph_db": Mock()} + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", lambda: owned) + scheduler.feedback_server = external + assert scheduler.mem_cube is owned["naive_mem_cube"] + assert scheduler.feedback_server is external + scheduler.close() + owned["graph_db"].close.assert_called_once() + external.close.assert_not_called() + + +def test_changing_cube_discards_only_automatically_assembled_feedback(scheduler, monkeypatch): + first, second = Mock(), Mock() + first_feedback, second_feedback = Mock(), Mock() + scheduler.mem_cube = first + factory = Mock(side_effect=[first_feedback, second_feedback]) + monkeypatch.setattr(scheduler, "_feedback_for_borrowed_cube", factory) + assert scheduler.feedback_server is first_feedback + scheduler.mem_cube = first + assert scheduler.feedback_server is first_feedback + scheduler.mem_cube = second + assert scheduler.feedback_server is second_feedback + explicit = Mock() + scheduler.feedback_server = explicit + scheduler.mem_cube = first + assert scheduler.feedback_server is explicit + + +def test_failed_borrowed_assembly_can_retry(scheduler, monkeypatch): + scheduler.mem_cube = Mock() + feedback = Mock() + factory = Mock(side_effect=[ConfigurationError("missing dependency"), feedback]) + monkeypatch.setattr(scheduler, "_feedback_for_borrowed_cube", factory) + with pytest.raises(ConfigurationError): + _ = scheduler.feedback_server + assert scheduler._feedback_server is None + assert scheduler.feedback_server is feedback + + +def test_close_prevents_late_borrowed_feedback_publication(scheduler, monkeypatch): + scheduler.mem_cube = Mock() + entered, release = Event(), Event() + + def assemble(): + entered.set() + assert release.wait(3) + return Mock() + + monkeypatch.setattr(scheduler, "_feedback_for_borrowed_cube", assemble) + with ThreadPoolExecutor(2) as pool: + creation = pool.submit(lambda: scheduler.feedback_server) + assert entered.wait(3) + closing = pool.submit(scheduler.close) + try: + assert scheduler._close_state.closing.wait(3) + finally: + release.set() + with pytest.raises(MemOSError): + creation.result(timeout=3) + closing.result(timeout=3) + assert scheduler._feedback_server is None diff --git a/tests/mem_scheduler/test_close_reentrancy.py b/tests/mem_scheduler/test_close_reentrancy.py new file mode 100644 index 000000000..c30bfbdba --- /dev/null +++ b/tests/mem_scheduler/test_close_reentrancy.py @@ -0,0 +1,104 @@ +"""Run self-close probes in owned processes so a regression cannot hang pytest.""" + +import pytest + +from tests.mem_scheduler.process_support import ( + STARTUP_TIMEOUT, + receive_probe_message, + spawned_probe, +) + + +def _close_probe(connection, mode): + from threading import Event, Thread + from unittest.mock import Mock + + import memos.mem_scheduler.base_scheduler as module + + from memos.exceptions import MemOSError + from memos.mem_scheduler.base_scheduler import BaseScheduler + + s = BaseScheduler({"use_redis_queue": False, "thread_pool_max_workers": 2}) + pool = s.dispatcher.dispatcher_executor + graph = Mock() + s.components = {"graph_db": graph, "naive_mem_cube": Mock(), "feedback_server": Mock()} + connection.send(("ready",)) + assert connection.recv() == ("run",) + + def rejected_close(): + try: + (s.dispatcher.close if mode == "dispatcher" else s.close)() + except MemOSError: + return + raise AssertionError("close from an owned worker must fail") + + if mode == "concurrent": + entered, release, draining, done = (Event() for _ in range(4)) + original = pool.shutdown + + def shutdown(*args, **kwargs): + draining.set() + return original(*args, **kwargs) + + pool.shutdown = shutdown + + def worker(): + entered.set() + assert release.wait(5) + rejected_close() + + future = pool.submit(worker) + assert entered.wait(5) + errors = [] + + def owner(): + try: + s.close() + except BaseException as exc: + errors.append(exc) + finally: + done.set() + + thread = Thread(target=owner, daemon=True) + thread.start() + assert draining.wait(5) + release.set() + future.result(timeout=3) + assert done.wait(3) + assert not errors, errors + thread.join() + elif mode == "initializing": + expected = s.components + s.components = None + + def initialize(): + rejected_close() + return expected + + module.init_components = initialize + assert s.mem_cube is expected["naive_mem_cube"] + s.close() + else: + pool.submit(rejected_close).result(timeout=3) + # Rejected self-close must not terminally close the scheduler or dispatcher. + assert pool.submit(lambda: 42).result(timeout=3) == 42 + assert s.mem_cube is s.components["naive_mem_cube"] + s.close() + graph.close.assert_called_once() + connection.send(("owner closed safely",)) + + +@pytest.mark.parametrize("mode", ["worker", "dispatcher", "concurrent", "initializing"]) +def test_owned_worker_and_initialization_callbacks_cannot_close_owner(mode): + with spawned_probe(_close_probe, mode) as (child, connection): + assert receive_probe_message( + connection, child, phase="scheduler initialization", timeout=STARTUP_TIMEOUT + ) == ("ready",) + # Only initialized processes enter the bounded deadlock probe. Import + # time cannot consume its budget or hide an actual self-close deadlock. + connection.send(("run",)) + assert receive_probe_message( + connection, child, phase=f"self-close ({mode})", timeout=20 + ) == ("owner closed safely",) + child.join(5) + assert child.exitcode == 0 diff --git a/tests/mem_scheduler/test_consumer_shutdown.py b/tests/mem_scheduler/test_consumer_shutdown.py new file mode 100644 index 000000000..e1651c587 --- /dev/null +++ b/tests/mem_scheduler/test_consumer_shutdown.py @@ -0,0 +1,115 @@ +"""Consumer loop shutdown keeps context, dispatch and metrics behavior intact.""" + +from datetime import datetime, timezone +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.context.context import get_current_context +from memos.mem_scheduler.base_mixins.queue_ops import BaseSchedulerQueueMixin + + +def consumer(messages): + owner = SimpleNamespace( + _running=True, + _consume_interval=0.1, + consume_batch=1, + use_redis_queue=False, + enable_parallel_dispatch=True, + dispatcher=Mock(max_workers=2), + metrics=Mock(), + memos_message_queue=Mock(get_messages=Mock(return_value=messages)), + ) + owner.dispatcher.get_running_task_count.return_value = 0 + owner._stop_event = Mock( + is_set=Mock(return_value=False), + wait=Mock(side_effect=lambda _: setattr(owner, "_running", False)), + ) + return owner + + +@pytest.mark.parametrize( + "timestamp", [None, 0, datetime(2026, 1, 1), datetime(2026, 1, 1, tzinfo=timezone.utc)] +) +@pytest.mark.parametrize("redis_queue", [False, True]) +def test_consumer_dispatch_preserves_context_and_observes_stop(monkeypatch, timestamp, redis_queue): + message = SimpleNamespace( + timestamp=timestamp, + user_id="u", + user_name="u", + label="test", + trace_id="trace", + api_path="/test", + ) + owner = consumer([message]) + owner.use_redis_queue = redis_queue + previous = get_current_context() + emit = Mock() + monkeypatch.setattr("memos.mem_scheduler.base_mixins.queue_ops.emit_monitor_event", emit) + BaseSchedulerQueueMixin._message_consumer(owner) + owner.dispatcher.dispatch.assert_called_once_with([message]) + owner._stop_event.wait.assert_called_once_with(0.1) + current = get_current_context() + assert (current.to_dict() if current else None) == (previous.to_dict() if previous else None) + assert ( + emit.call_args.args[2]["queue_wait_ms"] is None + if timestamp is None + else emit.call_args.args[2]["queue_wait_ms"] >= 0 + ) + + +@pytest.mark.parametrize( + "case", + [ + "empty", + "serial", + "full", + "short", + "dispatch_error", + "metrics_error", + "queue_error", + "empty_redis", + ], +) +def test_consumer_failure_and_idle_paths_are_stoppable(monkeypatch, case): + message = SimpleNamespace( + timestamp=None, user_id="u", user_name="u", label="test", trace_id="trace", api_path="/test" + ) + owner = consumer([message]) + monkeypatch.setattr("memos.mem_scheduler.base_mixins.queue_ops.emit_monitor_event", Mock()) + if case == "empty": + owner.memos_message_queue.get_messages.return_value = [] + elif case == "serial": + owner.enable_parallel_dispatch = False + elif case == "full": + owner.dispatcher.get_running_task_count.return_value = 2 + elif case == "short": + owner.consume_batch = 2 + elif case == "dispatch_error": + owner.dispatcher.dispatch.side_effect = ValueError("dispatch") + elif case == "metrics_error": + owner.metrics.task_dequeued.side_effect = ValueError("metrics") + else: + owner.memos_message_queue.get_messages.side_effect = ValueError( + "No messages available in Redis queue" if case == "empty_redis" else "queue failure" + ) + BaseSchedulerQueueMixin._message_consumer(owner) + assert not owner._running + owner._stop_event.wait.assert_called_once_with(0.1) + + +@pytest.mark.parametrize("error", [False, True]) +def test_metrics_monitor_handles_queue_shapes_and_stops(error): + owner = consumer([]) + owner.memos_message_queue.qsize.side_effect = [ + 5, + ValueError("queue") + if error + else {"total_size": 7, "p:u:c:t": 3, "local": 4, "short:key": 0}, + ] + BaseSchedulerQueueMixin._monitor_loop(owner) + owner._stop_event.wait.assert_called_once_with(15) + if not error: + assert owner.metrics.update_queue_length.call_args_list[0].args == (3, "u") + assert owner.metrics.update_queue_length.call_args_list[1].args == (4, "local") diff --git a/tests/mem_scheduler/test_final_close.py b/tests/mem_scheduler/test_final_close.py new file mode 100644 index 000000000..df8e53789 --- /dev/null +++ b/tests/mem_scheduler/test_final_close.py @@ -0,0 +1,631 @@ +"""R09-04/05: event barriers make close/initialization/recovery races deterministic.""" + +from concurrent.futures import ThreadPoolExecutor +from threading import Event, Thread +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError, MemOSError +from memos.mem_scheduler.base_scheduler import BaseScheduler +from memos.mem_scheduler.monitors.dispatcher_monitor import SchedulerDispatcherMonitor + + +@pytest.fixture +def scheduler(): + instance = BaseScheduler({"use_redis_queue": False, "thread_pool_max_workers": 2}) + yield instance + # Also usable during Red, without assuming the new close API exists. + instance._running = False + instance.dispatcher.dispatcher_executor.shutdown(wait=True, cancel_futures=True) + + +def owned_components(): + return {"graph_db": Mock(), "naive_mem_cube": Mock(), "feedback_server": Mock()} + + +def wait_for_close(component): + assert component._close_state.closing.wait(3) + + +def test_lazy_properties_share_one_creation(monkeypatch, scheduler): + components = owned_components() + factory = Mock(return_value=components) + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", factory) + with ThreadPoolExecutor(4) as pool: + cubes = list(pool.map(lambda _: scheduler.mem_cube, range(8))) + feedback = list(pool.map(lambda _: scheduler.feedback_server, range(8))) + assert all(cube is components["naive_mem_cube"] for cube in cubes) + assert all(server is components["feedback_server"] for server in feedback) + factory.assert_called_once_with() + scheduler.close() + scheduler.close() + components["graph_db"].close.assert_called_once_with() + + +def test_nonfinal_stop_quiesces_monitor_before_dispatcher(scheduler): + order = [] + scheduler._running = True + scheduler.stop_consumer = Mock(side_effect=lambda: order.append("consumer")) + scheduler.dispatcher_monitor = Mock(stop=Mock(side_effect=lambda: order.append("monitor"))) + scheduler.dispatcher.shutdown = Mock(side_effect=lambda: order.append("dispatcher")) + scheduler.stop() + assert order == ["consumer", "monitor", "dispatcher"] + + +@pytest.mark.parametrize("property_name", ["mem_cube", "feedback_server"]) +def test_lazy_failure_is_visible_and_can_retry_before_close(monkeypatch, scheduler, property_name): + error = GraphDBError("construction") + components = owned_components() + factory = Mock(side_effect=[error, components]) + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", factory) + with pytest.raises(GraphDBError) as raised: + getattr(scheduler, property_name) + assert raised.value is error + getattr(scheduler, property_name) + scheduler.close() + components["graph_db"].close.assert_called_once_with() + + +def test_borrowed_components_never_lazy_initialize_or_close(monkeypatch, scheduler): + components = owned_components() + factory = Mock(side_effect=AssertionError("must not initialize")) + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", factory) + scheduler.mem_cube = components["naive_mem_cube"] + scheduler.feedback_server = components["feedback_server"] + assert scheduler.mem_cube is components["naive_mem_cube"] + assert scheduler.feedback_server is components["feedback_server"] + scheduler.close() + components["graph_db"].close.assert_not_called() + components["naive_mem_cube"].close.assert_not_called() + + +@pytest.mark.parametrize( + "entry", + [ + "mem_cube", + "feedback_server", + "start", + "start_consumer", + "start_background_monitor", + "submit_messages", + ], +) +def test_closed_scheduler_rejects_work_without_creating_resources(monkeypatch, scheduler, entry): + factory = Mock(side_effect=AssertionError("must not initialize")) + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", factory) + scheduler.close() + with pytest.raises(MemOSError, match="clos"): + if entry == "submit_messages": + scheduler.submit_messages([]) + elif entry in ("mem_cube", "feedback_server"): + getattr(scheduler, entry) + else: + getattr(scheduler, entry)() + factory.assert_not_called() + + +def test_close_waits_for_unpublished_initialization(monkeypatch, scheduler): + entered, release, closing = Event(), Event(), Event() + components = owned_components() + + def initialize(): + entered.set() + assert release.wait(3) + return components + + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", initialize) + scheduler.rabbitmq_close = Mock(side_effect=closing.set) + with ThreadPoolExecutor(2) as pool: + initialization = pool.submit(lambda: scheduler.mem_cube) + assert entered.wait(3) + close = pool.submit(scheduler.close) + wait_for_close(scheduler) + assert not close.done() + release.set() + with pytest.raises(MemOSError): + initialization.result(3) + close.result(3) + assert scheduler.current_mem_cube is None + components["graph_db"].close.assert_called_once_with() + + +def test_final_close_waits_for_task_even_when_stop_wait_false(monkeypatch, scheduler): + entered, release, closing = Event(), Event(), Event() + components = owned_components() + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", lambda: components) + assert scheduler.mem_cube is components["naive_mem_cube"] + scheduler.rabbitmq_close = Mock(side_effect=closing.set) + + def task(): + entered.set() + assert release.wait(3) + components["graph_db"].close.assert_not_called() + + scheduler.dispatcher.stop_wait = False + task_future = scheduler.dispatcher.dispatcher_executor.submit(task) + assert entered.wait(3) + with ThreadPoolExecutor(1) as pool: + close = pool.submit(scheduler.close) + wait_for_close(scheduler) + assert not close.done() + release.set() + task_future.result(3) + close.result(3) + components["graph_db"].close.assert_called_once_with() + + +def test_close_failure_retries_only_unreleased_resources(monkeypatch, scheduler): + components = owned_components() + components["graph_db"].close.side_effect = [GraphDBError("pool"), None] + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", lambda: components) + assert scheduler.mem_cube is components["naive_mem_cube"] + scheduler.rabbitmq_close = Mock() + with pytest.raises(MemOSError): + scheduler.close() + with pytest.raises(MemOSError): + scheduler.start() + scheduler.close() + scheduler.rabbitmq_close.assert_called_once_with() + assert components["graph_db"].close.call_count == 2 + + +def test_running_recovery_preserves_tasks_and_final_close_drains_retired_pool(scheduler): + monitor = SchedulerDispatcherMonitor({}) + monitor.initialize(scheduler.dispatcher) + scheduler.dispatcher_monitor = monitor + old = scheduler.dispatcher.dispatcher_executor + entered, release = Event(), Event() + task = old.submit(lambda: (entered.set(), release.wait(3))) + assert entered.wait(3) + monitor._restart_pool("dispatcher", monitor._pools["dispatcher"]) + current = scheduler.dispatcher.dispatcher_executor + assert current is not old + assert current.submit(lambda: 7).result(3) == 7 + with ThreadPoolExecutor(1) as pool: + close = pool.submit(scheduler.close) + release.set() + close.result(3) + assert task.done() + assert old._shutdown and current._shutdown + monitor._restart_pool("dispatcher", monitor._pools["dispatcher"]) + assert scheduler.dispatcher.dispatcher_executor is current + + +def test_recovery_cannot_publish_new_executor_after_close(monkeypatch, scheduler): + monitor = SchedulerDispatcherMonitor({}) + monitor.initialize(scheduler.dispatcher) + scheduler.dispatcher_monitor = monitor + old = scheduler.dispatcher.dispatcher_executor + entered, release, closing = Event(), Event(), Event() + candidate = ThreadPoolExecutor(1) + + def new_executor(**kwargs): + entered.set() + assert release.wait(3) + return candidate + + monkeypatch.setattr( + "memos.mem_scheduler.monitors.dispatcher_monitor.ContextThreadPoolExecutor", new_executor + ) + scheduler.rabbitmq_close = Mock(side_effect=closing.set) + # Recovery belongs to the monitor thread; shutdown joins that actual owner. + monitor._monitor_thread = Thread( + target=monitor._restart_pool, args=("dispatcher", monitor._pools["dispatcher"]) + ) + with ThreadPoolExecutor(1) as pool: + monitor._monitor_thread.start() + assert entered.wait(3) + close = pool.submit(scheduler.close) + wait_for_close(monitor) + assert not close.done() + release.set() + monitor._monitor_thread.join(3) + assert not monitor._monitor_thread.is_alive() + close.result(3) + assert scheduler.dispatcher.dispatcher_executor is old + assert candidate._shutdown + + +def test_stop_does_not_close_graph_but_final_close_does(monkeypatch, scheduler): + components = owned_components() + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", lambda: components) + assert scheduler.mem_cube is components["naive_mem_cube"] + scheduler.stop() + components["graph_db"].close.assert_not_called() + scheduler.close() + components["graph_db"].close.assert_called_once_with() + + +@pytest.mark.parametrize("part", ["channel", "connection", "loop"]) +def test_rabbitmq_release_failures_are_visible_and_continue(scheduler, part): + scheduler.rabbitmq_channel = Mock(is_open=True) + scheduler.rabbitmq_connection = Mock(is_open=True) + targets = { + "channel": scheduler.rabbitmq_channel.close, + "connection": scheduler.rabbitmq_connection.close, + "loop": scheduler.rabbitmq_connection.ioloop.stop, + } + targets[part].side_effect = ValueError("release failed") + with pytest.raises(MemOSError): + scheduler.rabbitmq_close() + for target in targets.values(): + target.assert_called_once_with() + + +def test_close_checks_rabbitmq_thread_exit_without_holding_callback_lock(scheduler): + thread = Mock() + thread.is_alive.side_effect = [True, True] + scheduler._io_loop_thread = thread + with pytest.raises(MemOSError): + scheduler.rabbitmq_close() + thread.join.assert_called_once_with(timeout=5) + + +def test_initialize_modules_cannot_publish_after_final_close(monkeypatch, scheduler): + import memos.mem_scheduler.base_scheduler as module + + entered, release = Event(), Event() + for name in ("MemoryPostProcessor", "SchedulerRetriever", "ActivationMemoryManager"): + monkeypatch.setattr(module, name, Mock()) + monkeypatch.setattr(module.AuthConfig, "default_config_exists", lambda: False) + monkeypatch.setattr(module.AuthConfig, "from_local_env", lambda: None) + + def general_monitor(**kwargs): + entered.set() + assert release.wait(3) + return Mock() + + monkeypatch.setattr(module, "SchedulerGeneralMonitor", general_monitor) + with ThreadPoolExecutor(2) as pool: + initialization = pool.submit(scheduler.initialize_modules, Mock()) + assert entered.wait(3) + close = pool.submit(scheduler.close) + wait_for_close(scheduler) + assert not close.done() + release.set() + with pytest.raises(MemOSError): + initialization.result(3) + close.result(3) + assert not scheduler.dispatcher_monitor._running + assert not scheduler.dispatcher_monitor._monitor_thread.is_alive() + + +def test_close_waits_for_start_already_in_progress(monkeypatch, scheduler): + from memos.mem_scheduler.base_mixins.queue_ops import BaseSchedulerQueueMixin + + entered, release = Event(), Event() + + def start(self): + entered.set() + assert release.wait(3) + self._running = True + + monkeypatch.setattr(BaseSchedulerQueueMixin, "start", start) + with ThreadPoolExecutor(2) as pool: + starting = pool.submit(scheduler.start) + assert entered.wait(3) + close = pool.submit(scheduler.close) + wait_for_close(scheduler) + release.set() + starting.result(3) + close.result(3) + assert scheduler._running is False + + +@pytest.mark.parametrize("kind", ["thread", "process"]) +def test_stop_retains_live_consumer_for_final_close(scheduler, kind): + from memos.mem_scheduler.schemas.general_schemas import STARTUP_BY_PROCESS + + worker = Mock(is_alive=Mock(return_value=True)) + scheduler._running = True + if kind == "process": + scheduler.scheduler_startup_mode = STARTUP_BY_PROCESS + scheduler._consumer_process = worker + else: + scheduler._consumer_thread = worker + scheduler.stop_consumer() + assert getattr(scheduler, "_consumer_" + kind) is worker + + +def test_async_memory_history_submission_rejects_closed_scheduler(scheduler): + from memos.mem_scheduler.optimized_scheduler import OptimizedScheduler + + request = SimpleNamespace( + query="query", user_id="u", top_k=3, internet_search=False, chat_history=[] + ) + context = SimpleNamespace(mem_cube_id="A") + scheduler.memos_message_queue.submit_messages = Mock() + submit = OptimizedScheduler.submit_memory_history_async_task + task = submit(scheduler, request, context) + assert task.startswith("mix_search_u_") + scheduler.close() + with pytest.raises(MemOSError): + submit(scheduler, request, context) + scheduler.memos_message_queue.submit_messages.assert_called_once() + + +@pytest.mark.parametrize("resource", ["monitor", "consumer", "dispatcher"]) +def test_failed_drain_retains_owned_graph_until_retry(monkeypatch, scheduler, resource): + components = owned_components() + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", lambda: components) + assert scheduler.mem_cube is components["naive_mem_cube"] + if resource == "monitor": + scheduler.dispatcher_monitor = Mock() + scheduler.dispatcher_monitor.close.side_effect = [ValueError("monitor"), None] + elif resource == "consumer": + scheduler._consumer_thread = Mock(join=Mock(side_effect=[ValueError("join"), None])) + else: + original = scheduler.dispatcher.close + scheduler.dispatcher.close = Mock(side_effect=[ValueError("task"), None]) + with pytest.raises(MemOSError): + scheduler.close() + components["graph_db"].close.assert_not_called() + scheduler.close() + components["graph_db"].close.assert_called_once_with() + if resource == "dispatcher": + original() + + +def test_join_self_and_live_process_are_visible_errors(scheduler): + import threading + + scheduler._consumer_thread = threading.current_thread() + with pytest.raises(MemOSError): + scheduler._join_consumers() + scheduler._consumer_thread = None + process = Mock(is_alive=Mock(side_effect=[True, True, False])) + scheduler._consumer_process = process + with pytest.raises(MemOSError): + scheduler._join_consumers() + scheduler._join_consumers() + assert process.join.call_count == 3 + process.terminate.assert_called_once() + + +def test_existing_components_are_reused_when_feedback_reference_is_reset(monkeypatch, scheduler): + components = owned_components() + factory = Mock(return_value=components) + monkeypatch.setattr("memos.mem_scheduler.base_scheduler.init_components", factory) + assert scheduler.mem_cube is components["naive_mem_cube"] + scheduler._feedback_server = None + assert scheduler.feedback_server is components["feedback_server"] + factory.assert_called_once_with() + scheduler.close() + + +def test_real_consumer_and_metrics_threads_exit_on_close(scheduler): + scheduler.memos_message_queue.get_messages = Mock(return_value=[]) + scheduler.submit_messages([]) + scheduler.start() + threads = scheduler._consumer_thread, scheduler._monitor_thread + scheduler.close() + assert all(not thread.is_alive() for thread in threads) + + +def test_dispatcher_cleanup_retries_only_failed_executor(scheduler): + original = scheduler.dispatcher.dispatcher_executor + failed = Mock(shutdown=Mock(side_effect=[ValueError("executor"), None])) + retired = Mock() + scheduler.dispatcher.dispatcher_executor = failed + scheduler.dispatcher._retired_executors.extend([original, retired]) + with pytest.raises(MemOSError): + scheduler.dispatcher.close() + scheduler.dispatcher.close() + retired.shutdown.assert_called_once_with(wait=True, cancel_futures=True) + assert failed.shutdown.call_count == 2 + scheduler.dispatcher.dispatcher_executor = original + + +def test_monitor_recovery_failure_keeps_old_executor_for_final_cleanup(scheduler): + monitor = SchedulerDispatcherMonitor({}) + monitor.initialize(scheduler.dispatcher) + original = scheduler.dispatcher.dispatcher_executor + failed = Mock(shutdown=Mock(side_effect=ValueError("shutdown"))) + scheduler.dispatcher._retired_executors.append(failed) + info = {"executor": failed, "max_workers": 1} + monitor._restart_pool("dispatcher", info) + assert not monitor._restart_in_progress + assert scheduler.dispatcher.dispatcher_executor is original + failed.shutdown.side_effect = None + monitor.close() + scheduler.close() + + +def test_monitor_health_failure_and_stop_signal_are_observed(): + monitor = SchedulerDispatcherMonitor({}) + monitor._stop_event.wait = Mock(side_effect=[False, True]) + monitor._check_pools_health = Mock(side_effect=ValueError("health")) + monitor._monitor_loop() + monitor._check_pools_health.assert_called_once_with() + + +@pytest.mark.parametrize("shutdown", [False, True]) +@pytest.mark.parametrize("failure", [False, True]) +def test_nonfinal_monitor_stop_handles_executor_state(shutdown, failure): + monitor = SchedulerDispatcherMonitor({}) + monitor._running = True + executor = Mock(_shutdown=shutdown) + if failure: + executor.shutdown.side_effect = ValueError("shutdown") + monitor.register_pool("one", executor, 1) + monitor.stop() + assert not monitor._running + assert executor.shutdown.call_count == (0 if shutdown else 1) + monitor.stop() + + +def test_monitor_start_duplicate_stop_and_self_join(): + import threading + + monitor = SchedulerDispatcherMonitor({}) + assert monitor.start() + assert not monitor.start() + monitor.stop() + assert not monitor._monitor_thread.is_alive() + monitor._monitor_thread = threading.current_thread() + with pytest.raises(MemOSError): + monitor.stop() + with pytest.raises(MemOSError): + monitor.close() + monitor._monitor_thread = None + monitor.close() + + +def test_retired_worker_cannot_close_owner_and_recovery_keeps_wait_log( + scheduler, monkeypatch, caplog +): + import memos.mem_scheduler.monitors.dispatcher_monitor as module + + monitor = SchedulerDispatcherMonitor({}) + monitor.initialize(scheduler.dispatcher) + scheduler.dispatcher_monitor = monitor + entered, released = Event(), Event() + + def work(): + entered.set() + assert released.wait(3) + with pytest.raises(MemOSError): + scheduler.close() + with pytest.raises(MemOSError): + scheduler.dispatcher.close() + + task = scheduler.dispatcher.dispatcher_executor.submit(work) + assert entered.wait(3) + monkeypatch.setattr(module, "perf_counter", Mock(side_effect=[0, 2])) + try: + monitor._restart_pool("dispatcher", monitor._pools["dispatcher"]) + finally: + released.set() + task.result(3) + assert "Long lock wait: 2.000s" in caplog.text + assert not scheduler._close_state.closing.is_set() + scheduler.close() + + +def test_nonfinal_stop_without_optional_dispatcher_or_monitor(scheduler): + dispatcher = scheduler.dispatcher + scheduler._running = True + scheduler.stop_consumer = Mock() + scheduler._monitor_thread = Mock() + scheduler.dispatcher = None + try: + scheduler.stop() + scheduler._monitor_thread.join.assert_called_once_with(timeout=2.0) + finally: + scheduler.dispatcher = dispatcher + scheduler._monitor_thread = None + scheduler.close() + + +def test_resource_close_callback_cannot_recursively_close_owner(): + from memos.lifecycle import CloseState + + state = CloseState() + with pytest.raises(MemOSError, match="callback"): + state.close(lambda: state.close(lambda: None)) + with pytest.raises(MemOSError, match="clos"): + state.ensure_open() + cleanup = Mock() + state.close(cleanup) + state.close(cleanup) + cleanup.assert_called_once() + + +def test_rabbitmq_closed_connection_and_exited_thread(scheduler): + scheduler.rabbitmq_connection = Mock(is_open=False) + thread = Mock(is_alive=Mock(side_effect=[True, False])) + scheduler._io_loop_thread = thread + scheduler.rabbitmq_close() + scheduler.rabbitmq_connection.close.assert_not_called() + thread.join.assert_called_once_with(timeout=5) + + +def test_rabbitmq_self_join_is_reported(scheduler): + import threading + + scheduler._io_loop_thread = threading.current_thread() + with pytest.raises(MemOSError): + scheduler.rabbitmq_close() + + +@pytest.mark.parametrize("auth_source", ["path", "default", "env", "invalid", "missing"]) +@pytest.mark.parametrize("has_dispatcher", [False, True]) +def test_module_initialization_sources_and_partial_queue_configuration( + monkeypatch, tmp_path, scheduler, auth_source, has_dispatcher +): + import memos.mem_scheduler.base_scheduler as module + + original_dispatcher, original_queue = scheduler.dispatcher, scheduler.memos_message_queue + scheduler.enable_parallel_dispatch = False + scheduler.use_redis_queue = True + if not has_dispatcher: + scheduler.dispatcher = None + scheduler.memos_message_queue = None + for name in ( + "SchedulerGeneralMonitor", + "SchedulerRetriever", + "MemoryPostProcessor", + "ActivationMemoryManager", + "TaskStatusTracker", + ): + monkeypatch.setattr(module, name, Mock()) + monkeypatch.setattr( + module.AuthConfig, "default_config_exists", lambda: auth_source == "default" + ) + auth = None if auth_source == "missing" else SimpleNamespace(rabbitmq=None) + monkeypatch.setattr(module.AuthConfig, "from_local_config", Mock(return_value=auth)) + getter = ( + Mock(side_effect=ValueError("auth")) + if auth_source == "invalid" + else Mock(return_value=auth) + ) + monkeypatch.setattr(module.AuthConfig, "from_local_env", getter) + if auth_source == "path": + path = tmp_path / "auth.json" + path.write_text("{}") + scheduler.auth_config_path = path + reader = Mock() + try: + scheduler.initialize_modules(Mock(), mem_reader=reader, redis_client=Mock()) + assert scheduler.mem_reader is reader + finally: + scheduler.dispatcher, scheduler.memos_message_queue = original_dispatcher, original_queue + scheduler.close() + + +def test_failed_module_construction_preserves_cause(monkeypatch, scheduler): + import memos.mem_scheduler.base_scheduler as module + + error = ValueError("monitor unavailable") + monkeypatch.setattr(module, "SchedulerGeneralMonitor", Mock(side_effect=error)) + with pytest.raises(ValueError) as raised: + scheduler.initialize_modules(Mock()) + assert raised.value is error + scheduler.close() + + +@pytest.mark.parametrize( + "kind,alive", + [ + ("process", [False, False]), + ("process", [True, False, False]), + ("process", [True, True, False, False]), + ("thread", [True, False]), + ("thread", [False]), + ], +) +def test_stop_consumer_exit_states(scheduler, kind, alive): + from memos.mem_scheduler.schemas.general_schemas import STARTUP_BY_PROCESS + + worker = Mock(is_alive=Mock(side_effect=alive)) + scheduler._running = True + if kind == "process": + scheduler.scheduler_startup_mode = STARTUP_BY_PROCESS + scheduler._consumer_process = worker + else: + scheduler._consumer_thread = worker + scheduler.stop_consumer() + assert not scheduler._running + scheduler.stop_consumer() diff --git a/tests/mem_scheduler/test_process_close.py b/tests/mem_scheduler/test_process_close.py new file mode 100644 index 000000000..245a6477b --- /dev/null +++ b/tests/mem_scheduler/test_process_close.py @@ -0,0 +1,40 @@ +"""Legacy process handles are retained until cleanup succeeds. + +Independent process execution and draining are covered in test_worker_isolation. +""" + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.exceptions import MemOSError +from memos.mem_scheduler.base_mixins.queue_ops import BaseSchedulerQueueMixin +from memos.mem_scheduler.base_scheduler import BaseScheduler + + +@pytest.mark.parametrize("failure", ["survives", "terminate_error"]) +def test_unterminated_process_keeps_resources_and_allows_retry(failure): + scheduler = BaseScheduler({"use_redis_queue": False, "thread_pool_max_workers": 1}) + graph = Mock() + scheduler.components = {"graph_db": graph} + process = Mock(is_alive=Mock(return_value=True)) + if failure == "terminate_error": + process.terminate.side_effect = OSError("cannot terminate") + scheduler._consumer_process = process + try: + with pytest.raises(MemOSError): + scheduler.close() + process.terminate.assert_called_once() + graph.close.assert_not_called() + assert not scheduler.dispatcher.dispatcher_executor._shutdown + finally: + process.is_alive.return_value = False + scheduler.close() + graph.close.assert_called_once() + + +def test_starting_running_consumer_does_not_replace_stop_signal(): + owner = SimpleNamespace(_running=True) + BaseSchedulerQueueMixin.start_consumer(owner) + assert vars(owner) == {"_running": True} diff --git a/tests/mem_scheduler/test_rabbitmq_shutdown.py b/tests/mem_scheduler/test_rabbitmq_shutdown.py index 5cb594c43..3096e6746 100644 --- a/tests/mem_scheduler/test_rabbitmq_shutdown.py +++ b/tests/mem_scheduler/test_rabbitmq_shutdown.py @@ -73,8 +73,8 @@ def native_running_loop(): monitor.start() task = scheduler.dispatcher.dispatcher_executor.submit(lambda: 42) assert task.result(timeout=2) == 42 - shutdown = Mock(wraps=scheduler.dispatcher.shutdown) - monkeypatch.setattr(scheduler.dispatcher, "shutdown", shutdown) + close = Mock(wraps=scheduler.dispatcher.close) + monkeypatch.setattr(scheduler.dispatcher, "close", close) error = Mock() monkeypatch.setattr(lifecycle.logger, "exception", error) try: @@ -89,14 +89,14 @@ def native_running_loop(): ): time.sleep(0.005) assert not any(t.is_alive() for t in scheduler.dispatcher.dispatcher_executor._threads) - shutdown.assert_called_once() + close.assert_called_once() assert scheduler._rabbitmq_stop_flag is True error.assert_not_called() finally: scheduler._running = False consumer.join(timeout=1) monitor.join(timeout=1) - scheduler.dispatcher.shutdown() + scheduler.close() def test_unexpected_join_error_is_not_hidden(): diff --git a/tests/mem_scheduler/test_recall_fallback.py b/tests/mem_scheduler/test_recall_fallback.py new file mode 100644 index 000000000..6c693b3fc --- /dev/null +++ b/tests/mem_scheduler/test_recall_fallback.py @@ -0,0 +1,79 @@ +"""Optional model parsing cannot discard successfully retrieved memories.""" + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +import memos.api.handlers # noqa: F401 - initialize the existing handler import cycle + +from memos.api.product_models import APISearchRequest +from memos.exceptions import GraphDBError +from memos.mem_scheduler.memory_manage_modules.enhancement_pipeline import EnhancementPipeline +from memos.memories.textual.item import TextualMemoryItem +from memos.multi_mem_cube.single_cube import SingleCubeView +from memos.types.general_types import FineStrategy, SearchMode, UserContext + + +@pytest.mark.parametrize( + "response", + [ + "", + "not JSON", + '{"hint": null}', + '{"hint": 42}', + '{"hint":"x","trigger_recall":"false"}', + ValueError("model"), + OSError("offline"), + ], +) +def test_optional_recall_failure_preserves_raw_search_result(monkeypatch, response): + monkeypatch.setattr("memos.multi_mem_cube.single_cube.FINE_STRATEGY", FineStrategy.RECREATE) + llm = Mock() + if isinstance(response, Exception): + llm.generate.side_effect = response + else: + llm.generate.return_value = response + pipeline = EnhancementPipeline(llm, SimpleNamespace(), lambda **_: "fixture") + raw = TextualMemoryItem(memory="already retrieved") + searcher = Mock(post_retrieve=Mock(return_value=[raw])) + retriever = SimpleNamespace( + enhance_memories_with_query=lambda **_: ([], True), + recall_for_missing_memories=pipeline.recall_for_missing_memories, + ) + view = SingleCubeView( + "A", Mock(), Mock(), SimpleNamespace(retriever=retriever), Mock(), searcher + ) + view._postformat_memories = Mock(side_effect=lambda memories, *args, **kwargs: memories) + assert view._search_text( + APISearchRequest(query="q", user_id="u"), + UserContext(user_id="u", mem_cube_id="A"), + SearchMode.FINE, + ) == [raw] + searcher.search.assert_not_called() + + +@pytest.mark.parametrize( + "response,expected", + [ + ("{}", ("", False)), + ('{"hint":""}', ("", False)), + ('{"hint":"topic"}', ("topic", False)), + ('{"hint":"topic","trigger_recall":true}', ("topic", True)), + ], +) +def test_valid_recall_contract(response, expected): + pipeline = EnhancementPipeline( + Mock(generate=Mock(return_value=response)), SimpleNamespace(), lambda **_: "fixture" + ) + assert pipeline.recall_for_missing_memories("q", ["memory"]) == expected + + +def test_recall_does_not_reclassify_storage_error(): + failure = GraphDBError("storage") + pipeline = EnhancementPipeline( + Mock(generate=Mock(side_effect=failure)), SimpleNamespace(), lambda **_: "fixture" + ) + with pytest.raises(GraphDBError) as caught: + pipeline.recall_for_missing_memories("q", []) + assert caught.value is failure diff --git a/tests/mem_scheduler/test_worker_integration.py b/tests/mem_scheduler/test_worker_integration.py new file mode 100644 index 000000000..b90c4c5bd --- /dev/null +++ b/tests/mem_scheduler/test_worker_integration.py @@ -0,0 +1,138 @@ +"""Independent worker, real Redis transport and real openGauss resource ownership. + +MEMOS_WORKER_REDIS_TEST_PORT must point at an isolated, disposable Redis instance. +""" + +import multiprocessing +import os +import uuid + +from functools import partial + +import pytest +import redis + +from memos.configs.graph_db import OpenGaussGraphDBConfig +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.mem_scheduler.base_scheduler import BaseScheduler +from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem +from memos.mem_scheduler.task_schedule_modules.task_queue import ScheduleTaskQueue +from memos.mem_scheduler.worker import run_scheduler_worker +from tests.graph_dbs.test_opengauss_integration import _query +from tests.graph_dbs.test_opengauss_integration import database as _database + + +database = _database +pytestmark = pytest.mark.skipif( + not os.getenv("MEMOS_OPENGAUSS_TEST_CONFIG") or not os.getenv("MEMOS_WORKER_REDIS_TEST_PORT"), + reason="requires openGauss and an isolated worker-test Redis", +) + + +class _ReportingGraph: + def __init__(self, graph, results): + self.graph, self.results = graph, results + + def close(self): + self.graph.close() + self.results.put(("closed", os.getpid())) + + +def _build_storage_worker(config, results): + scheduler = BaseScheduler( + {"use_redis_queue": True, "thread_pool_max_workers": 2, "consume_interval_seconds": 0.01} + ) + try: + graph = OpenGaussGraphDB(OpenGaussGraphDBConfig(**config)) + scheduler.components = {"graph_db": _ReportingGraph(graph, results)} + + def persist(messages): + for message in messages: + graph.add_node( + message.item_id, + message.content, + {"memory_type": "WorkingMemory"}, + user_name=message.mem_cube_id, + ) + results.put((message.content, os.getpid())) + + scheduler.register_handlers({"worker-test": persist}) + results.put(("ready", os.getpid())) + return scheduler + except BaseException: + scheduler.close() + raise + + +def test_real_cross_process_queue_and_database_sessions(database, monkeypatch): + admin, config = database + prefix = "worker-test-" + uuid.uuid4().hex + port = int(os.environ["MEMOS_WORKER_REDIS_TEST_PORT"]) + for key, value in { + "MEMOS_ENABLE_REDIS": "true", + "MEMSCHEDULER_REDIS_HOST": "127.0.0.1", + "MEMSCHEDULER_REDIS_PORT": str(port), + "MEMSCHEDULER_REDIS_DB": "0", + "MEMSCHEDULER_REDIS_STREAM_KEY_PREFIX": prefix, + "MEMSCHEDULER_REDIS_TIMEOUT": "2", + "MEMSCHEDULER_REDIS_CONNECT_TIMEOUT": "2", + }.items(): + monkeypatch.setenv(key, value) + monkeypatch.delenv("MEMSCHEDULER_REDIS_PASSWORD", raising=False) + ctx = multiprocessing.get_context("spawn") + stop, results = ctx.Event(), ctx.Queue() + transport = ScheduleTaskQueue(use_redis_queue=True, maxsize=100) + inspect_redis = redis.Redis(host="127.0.0.1", port=port, decode_responses=True) + before_clients = len(inspect_redis.client_list()) + child = ctx.Process( + target=run_scheduler_worker, + args=(partial(_build_storage_worker, config.model_dump(), results), stop), + ) + try: + child.start() + assert results.get(timeout=15) == ("ready", child.pid) + for index in range(2): + content = f"late task {index}" + transport.submit_messages( + ScheduleMessageItem( + item_id=f"{prefix}-{index}", + user_id=prefix, + task_id=f"{prefix}-task", + mem_cube_id="A", + label="worker-test", + content=content, + ) + ) + # A different process discovers new streams on the existing 30 s + # refresh cycle; the test budget must include that documented delay. + assert results.get(timeout=45) == (content, child.pid) + stop.set() + assert results.get(timeout=10) == ("closed", child.pid) + child.join(10) + assert child.exitcode == 0 + assert _query( + admin, 'SELECT memory FROM ONLY memos_graph."Memory" ORDER BY business_id' + ) == [("late task 0",), ("late task 1",)] + assert _query( + admin, + "SELECT count(*) FROM pg_stat_activity WHERE datname=%s AND application_name='memos-opengauss'", + (config.db_name,), + ) == [(0,)] + assert len(inspect_redis.client_list()) == before_clients + finally: + stop.set() + if child.pid is not None: + child.join(10) + if child.is_alive(): + child.terminate() + child.join(5) + child.close() + results.close() + results.join_thread() + transport.memos_message_queue._stop_stream_keys_refresh_thread() + transport.memos_message_queue.redis_close() + keys = list(inspect_redis.scan_iter(match=f"*{prefix}*")) + if keys: + inspect_redis.delete(*keys) + assert list(inspect_redis.scan_iter(match=f"*{prefix}*")) == [] + inspect_redis.close() diff --git a/tests/mem_scheduler/test_worker_isolation.py b/tests/mem_scheduler/test_worker_isolation.py new file mode 100644 index 000000000..4d9706390 --- /dev/null +++ b/tests/mem_scheduler/test_worker_isolation.py @@ -0,0 +1,252 @@ +"""Schedulers belong to one process; independent workers construct their own resources.""" + +import multiprocessing +import os +import queue + +from dataclasses import dataclass +from functools import partial +from threading import Event +from unittest.mock import Mock + +import pytest + +from memos.exceptions import ConfigurationError +from memos.mem_scheduler.base_scheduler import BaseScheduler +from memos.mem_scheduler.schemas.message_schemas import ScheduleMessageItem +from tests.mem_scheduler.process_support import ( + STARTUP_TIMEOUT, + receive_probe_message, + spawned_probe, +) + + +def test_legacy_process_start_fails_before_mutating_running_state(monkeypatch): + process = Mock() + monkeypatch.setattr(multiprocessing, "Process", process) + scheduler = BaseScheduler({"use_redis_queue": False, "scheduler_startup_mode": "process"}) + try: + with pytest.raises(ConfigurationError, match="worker"): + scheduler.start_consumer() + assert not scheduler._running + assert scheduler._consumer_process is None + process.assert_not_called() + finally: + scheduler.close() + + +def test_typed_config_does_not_silently_ignore_process_mode(): + from pydantic import ValidationError + + from memos.configs.mem_scheduler import GeneralSchedulerConfig + + with pytest.raises(ValidationError, match="scheduler_startup_mode"): + GeneralSchedulerConfig(scheduler_startup_mode="process") + + +def test_failed_thread_start_remains_closable_and_retryable(monkeypatch): + scheduler = BaseScheduler({"use_redis_queue": False}) + thread = Mock() + thread.start.side_effect = OSError("cannot start thread") + with monkeypatch.context() as patch: + patch.setattr( + "memos.mem_scheduler.base_mixins.queue_ops.ContextThread", Mock(return_value=thread) + ) + with pytest.raises(OSError): + scheduler.start_consumer() + assert not scheduler._running and scheduler._consumer_thread is None + try: + scheduler.start_consumer() + assert scheduler._consumer_thread.is_alive() + finally: + scheduler.close() + + +def test_owner_pid_validation_reports_configuration_error(): + scheduler = BaseScheduler({"use_redis_queue": False}) + original = scheduler._owner_pid + try: + scheduler._owner_pid = -1 + with pytest.raises(ConfigurationError, match="fork"): + scheduler.close() + finally: + scheduler._owner_pid = original + scheduler.close() + + +def _try_inherited_scheduler(scheduler, results): + for operation in ( + scheduler.start_consumer, + lambda: scheduler.submit_messages([]), + lambda: scheduler.mem_cube, + lambda: scheduler.feedback_server, + scheduler.stop, + scheduler.stop_consumer, + scheduler.close, + ): + try: + operation() + except ConfigurationError: + results.put("rejected") + else: + results.put("unsafe") + + +@pytest.mark.skipif("fork" not in multiprocessing.get_all_start_methods(), reason="requires fork") +def test_inherited_scheduler_is_rejected_before_acquiring_parent_locks(): + ctx = multiprocessing.get_context("fork") + results = ctx.Queue() + scheduler = BaseScheduler({"use_redis_queue": False, "thread_pool_max_workers": 1}) + graph = Mock() + scheduler.components = {"graph_db": graph, "naive_mem_cube": Mock(), "feedback_server": Mock()} + # The fork happens while a different parent thread holds the lifecycle lock. + locked, release = Event(), Event() + + def hold_lock(): + with scheduler._initialization_lock: + locked.set() + release.wait(15) + + task = scheduler.dispatcher.dispatcher_executor.submit(hold_lock) + child = ctx.Process(target=_try_inherited_scheduler, args=(scheduler, results)) + try: + assert locked.wait(3) + child.start() + assert [results.get(timeout=5) for _ in range(7)] == ["rejected"] * 7 + child.join(5) + assert child.exitcode == 0 + graph.close.assert_not_called() + finally: + release.set() + task.result(timeout=3) + if child.pid is not None: + if child.is_alive(): + child.terminate() + child.join(5) + child.close() + scheduler.close() + results.close() + results.join_thread() + graph.close.assert_called_once() + + +@dataclass +class _ChildGraph: + results: object + + def close(self): + self.results.send(("closed", os.getpid())) + + +def _build_worker(incoming, results, release): + scheduler = BaseScheduler( + {"use_redis_queue": False, "thread_pool_max_workers": 1, "consume_interval_seconds": 0.01} + ) + scheduler.components = {"graph_db": _ChildGraph(results)} + + def receive(batch_size): + try: + content = incoming.get_nowait() + except queue.Empty: + return [] + return [ScheduleMessageItem(user_id="u", mem_cube_id="cube", label="test", content=content)] + + def handle(messages): + results.send(("started", os.getpid())) + assert release.wait(15) + results.send((messages[0].content, os.getpid())) + + # The adapter represents an external data transport; no parent runtime + # objects (scheduler, executor, connection or ordinary Queue) cross the boundary. + scheduler.memos_message_queue.get_messages = receive + scheduler.register_handlers({"test": handle}) + results.send(("created", os.getpid())) + return scheduler + + +def _run_worker(results, incoming, stop, release): + from memos.mem_scheduler.worker import run_scheduler_worker + + run_scheduler_worker(partial(_build_worker, incoming, results, release), stop) + + +@pytest.mark.parametrize("parent_busy", [False, True]) +def test_spawn_worker_handles_late_task_drains_and_preserves_parent(parent_busy): + ctx = multiprocessing.get_context("spawn") + incoming = ctx.Queue() + stop, release = ctx.Event(), ctx.Event() + parent_release, parent_started = Event(), Event() + parent = BaseScheduler({"use_redis_queue": False, "thread_pool_max_workers": 1}) + graph = Mock() + parent.components = {"graph_db": graph} + + def parent_work(): + parent_started.set() + if parent_busy: + # The parent stays busy throughout cold child startup. The outer + # finally always releases it, including failed startup/behavior checks. + parent_release.wait() + return os.getpid() + + task = parent.dispatcher.dispatcher_executor.submit(parent_work) + try: + assert parent_started.wait(3) + if not parent_busy: + assert task.result(timeout=3) == os.getpid() + with spawned_probe(_run_worker, incoming, stop, release) as (child, results): + try: + assert receive_probe_message( + results, child, phase="worker startup", timeout=STARTUP_TIMEOUT + ) == ("created", child.pid) + incoming.put("late-task-done") + assert receive_probe_message( + results, child, phase="task acceptance", timeout=5 + ) == ("started", child.pid) + stop.set() + assert child.is_alive() # accepted task still owns the graph + release.set() + assert receive_probe_message(results, child, phase="task drain", timeout=5) == ( + "late-task-done", + child.pid, + ) + assert receive_probe_message(results, child, phase="graph release", timeout=5) == ( + "closed", + child.pid, + ) + child.join(10) + assert child.exitcode == 0 + finally: + stop.set() + release.set() + child.join(10) + graph.close.assert_not_called() + parent_release.set() + assert task.result(timeout=3) == os.getpid() + assert parent.dispatcher.dispatcher_executor.submit(lambda: 42).result(timeout=3) == 42 + finally: + stop.set() + release.set() + parent_release.set() + parent.close() + incoming.close() + incoming.join_thread() + graph.close.assert_called_once() + + +@pytest.mark.parametrize("stage", ["start", "wait", "success"]) +def test_worker_always_closes_returned_scheduler(stage): + from memos.mem_scheduler.worker import run_scheduler_worker + + scheduler, stop = Mock(), Mock() + failure = OSError("worker failure") + if stage == "start": + scheduler.start.side_effect = failure + elif stage == "wait": + stop.wait.side_effect = failure + if stage == "success": + run_scheduler_worker(lambda: scheduler, stop) + else: + with pytest.raises(OSError) as caught: + run_scheduler_worker(lambda: scheduler, stop) + assert caught.value is failure + scheduler.close.assert_called_once() diff --git a/tests/memories/textual/test_binding_uuid_forms.py b/tests/memories/textual/test_binding_uuid_forms.py new file mode 100644 index 000000000..c56e108e8 --- /dev/null +++ b/tests/memories/textual/test_binding_uuid_forms.py @@ -0,0 +1,28 @@ +"""Binding identity is UUID identity, including mixed accepted spellings.""" + +from uuid import UUID, uuid4 + +import pytest + +from memos.memories.textual.tree_text_memory.organize.manager import extract_working_binding_ids + + +VALUE = UUID("a1234567-1234-5678-9abc-123456789abc") +FORMS = [str(VALUE), VALUE.hex, VALUE.urn, "{" + str(VALUE) + "}", str(VALUE).upper()] + + +@pytest.mark.parametrize("primary", FORMS) +@pytest.mark.parametrize("binding", FORMS) +def test_same_uuid_never_becomes_cleanup_candidate(primary, binding): + assert ( + extract_working_binding_ids([{"id": primary, "metadata": {"working_binding": binding}}]) + == set() + ) + + +@pytest.mark.parametrize("primary", [None, 42, "legacy-id", str(uuid4())]) +def test_independent_binding_preserves_its_storage_key(primary): + binding = VALUE.urn + assert extract_working_binding_ids( + [{"id": primary, "metadata": {"working_binding": binding}}] + ) == {binding} diff --git a/tests/memories/textual/test_memory_pair_identity.py b/tests/memories/textual/test_memory_pair_identity.py new file mode 100644 index 000000000..4eca43ddc --- /dev/null +++ b/tests/memories/textual/test_memory_pair_identity.py @@ -0,0 +1,181 @@ +"""Independent identities and one storage operation for each logical memory.""" + +import uuid + +from threading import Event +from unittest.mock import Mock + +import pytest + +from memos.exceptions import GraphDBError +from memos.memories.textual.item import TextualMemoryItem +from memos.memories.textual.tree_text_memory.organize.manager import ( + MemoryManager, + extract_working_binding_ids, +) + + +@pytest.fixture +def manager(): + result = MemoryManager(Mock(), Mock(), Mock()) + result.is_reorganize = True + result.reorganizer = Mock() + return result + + +@pytest.mark.parametrize("batch", [False, True]) +@pytest.mark.parametrize("kind", ["LongTermMemory", "UserMemory"]) +@pytest.mark.parametrize("fast", [False, True]) +def test_independent_pair_preserves_input_and_retry_identity(manager, batch, kind, fast): + old_binding = str(uuid.uuid4()) + item = TextualMemoryItem( + memory="production pair", + metadata={ + "memory_type": kind, + "embedding": [1, 0, 0], + "tags": ["mode:fast"] if fast else [], + "background": f"source [working_binding:{old_binding}]", + "working_binding": old_binding, + "info": {"agent_id": "app"}, + }, + ) + before = item.model_dump() + expected_working = str(uuid.uuid5(uuid.NAMESPACE_URL, "memos:working:" + item.id)) + for _ in range(2): + assert manager.add([item], "A", mode="async", use_batch=batch) == [item.id] + nodes = manager.graph_store.add_nodes_batch.call_args.args[0] + assert len(nodes) == 2 + working, final = nodes + assert working["id"] == expected_working != final["id"] == item.id + assert working["metadata"]["memory_type"] == "WorkingMemory" + assert "working_binding" not in working["metadata"] + assert final["metadata"]["memory_type"] == kind + assert final["metadata"]["working_binding"] == expected_working + for node in nodes: + assert node["memory"] == item.memory + assert node["metadata"]["embedding"] == [1, 0, 0] + assert node["metadata"]["info"] == {"agent_id": "app"} + assert old_binding not in node["metadata"]["background"] + if fast: + assert f"[working_binding:{expected_working}]" in final["metadata"]["background"] + assert manager.graph_store.add_nodes_batch.call_args.kwargs == {"user_name": "A"} + assert item.model_dump() == before + manager.graph_store.add_node.assert_not_called() + + +@pytest.mark.parametrize("batch", [False, True]) +@pytest.mark.parametrize( + "kind", + [ + "WorkingMemory", + "OuterMemory", + "RawFileMemory", + "ToolSchemaMemory", + "ToolTrajectoryMemory", + "SkillMemory", + "PreferenceMemory", + "Context", + ], +) +def test_single_types_do_not_create_phantom_bindings(manager, batch, kind): + item = TextualMemoryItem(memory="one", metadata={"memory_type": kind, "tags": ["mode:fast"]}) + graph = kind not in {"WorkingMemory", "OuterMemory", "Context"} + assert manager.add([item], "A", mode="async", use_batch=batch) == ([item.id] if graph else []) + if kind == "Context": + manager.graph_store.add_nodes_batch.assert_not_called() + else: + nodes = manager.graph_store.add_nodes_batch.call_args.args[0] + assert len(nodes) == 1 and nodes[0]["id"] == item.id + assert nodes[0]["metadata"]["memory_type"] == (kind if graph else "WorkingMemory") + assert "working_binding" not in nodes[0]["metadata"] + + +@pytest.mark.parametrize("batch", [False, True]) +@pytest.mark.parametrize( + "failure", [GraphDBError("storage"), ValueError("invalid"), RuntimeError("driver")] +) +def test_failed_write_is_visible_and_does_not_notify(manager, batch, failure): + manager.graph_store.add_node.side_effect = failure + manager.graph_store.add_nodes_batch.side_effect = failure + with pytest.raises(type(failure)) as caught: + manager.add( + [TextualMemoryItem(memory="one", metadata={"memory_type": "LongTermMemory"})], + "A", + use_batch=batch, + ) + assert caught.value is failure + manager.reorganizer.add_message.assert_not_called() + manager.graph_store.remove_oldest_memory.assert_not_called() + + +def test_batch_boundary_never_splits_pair(manager): + items = [ + TextualMemoryItem(memory=str(i), metadata={"memory_type": "LongTermMemory"}) + for i in range(6) + ] + assert manager._add_memories_batch(items, "A", batch_size=5) == [item.id for item in items] + calls = manager.graph_store.add_nodes_batch.call_args_list + assert len(calls) == 2 + for call in calls: + nodes = call.args[0] + ids = {node["id"] for node in nodes} + for node in nodes: + if node["metadata"]["memory_type"] == "LongTermMemory": + assert node["metadata"]["working_binding"] in ids + + +def test_parallel_results_follow_input_order(manager): + first, second = TextualMemoryItem(memory="first"), TextualMemoryItem(memory="second") + second_done = Event() + + def process(item, owner): + if item is first: + assert second_done.wait(3) + else: + second_done.set() + return [item.id] + + manager._process_memory = process + assert manager._add_memories_parallel([first, second], "A") == [first.id, second.id] + + +def test_empty_batch_and_invalid_batch_size(manager): + assert manager.add([], "A", mode="async") == [] + with pytest.raises(ValueError, match="batch_size"): + manager._add_memories_batch([], "A", batch_size=0) + manager.graph_store.add_nodes_batch.assert_not_called() + + +def test_missing_background_and_nontext_legacy_metadata(manager): + item = TextualMemoryItem( + memory="one", metadata={"memory_type": "LongTermMemory", "background": None} + ) + assert manager.add([item], "A", mode="async") == [item.id] + assert extract_working_binding_ids([{"id": item.id, "metadata": {"background": 3}}]) == set() + + +@pytest.mark.parametrize("form", ["braced", "hex", "urn"]) +def test_legacy_self_binding_preserves_accepted_uuid_spellings(form): + value = uuid.uuid4() + primary = {"braced": "{" + str(value) + "}", "hex": value.hex, "urn": value.urn}[form] + item = TextualMemoryItem(id=primary, memory="legacy", metadata={"working_binding": primary}) + assert extract_working_binding_ids([item]) == set() + + +@pytest.mark.parametrize("as_dict", [False, True]) +@pytest.mark.parametrize("source", ["structured", "legacy", "self", "invalid", "none"]) +def test_binding_cleanup_uses_existing_field_and_rejects_self(as_dict, source): + primary, working, legacy = [str(uuid.uuid4()) for _ in range(3)] + metadata = {} + if source == "structured": + metadata = {"working_binding": working, "background": f"[working_binding:{legacy}]"} + elif source == "legacy": + metadata = {"background": f"[working_binding:{working}]"} + elif source == "self": + metadata = {"working_binding": primary, "background": f"[working_binding:{primary}]"} + elif source == "invalid": + metadata = {"working_binding": "invalid"} + item = TextualMemoryItem(id=primary, memory="one", metadata=metadata) + assert extract_working_binding_ids([item.model_dump() if as_dict else item]) == ( + {working} if source in {"structured", "legacy"} else set() + ) diff --git a/tests/memories/textual/test_tree.py b/tests/memories/textual/test_tree.py index a72709ec5..6969f10b1 100644 --- a/tests/memories/textual/test_tree.py +++ b/tests/memories/textual/test_tree.py @@ -5,6 +5,7 @@ import pytest from memos.configs.memory import TreeTextMemoryConfig +from memos.exceptions import MemoryError from memos.memories.textual.item import TextualMemoryItem, TreeNodeTextualMemoryMetadata from memos.memories.textual.tree import TreeTextMemory @@ -116,8 +117,9 @@ def test_load_file_not_exists(mock_tree_text_memory, tmp_path): mock_tree_text_memory.graph_store.import_graph = MagicMock() result = tmp_path / "does_not_exist" - mock_tree_text_memory.load(str(result)) - # Should log a warning but not raise + with pytest.raises(MemoryError): + mock_tree_text_memory.load(str(result)) + mock_tree_text_memory.graph_store.import_graph.assert_not_called() def test_dump_and_load_success(tmp_path, mock_tree_text_memory): diff --git a/tests/memories/textual/test_tree_files.py b/tests/memories/textual/test_tree_files.py new file mode 100644 index 000000000..6f101e602 --- /dev/null +++ b/tests/memories/textual/test_tree_files.py @@ -0,0 +1,218 @@ +"""Strict restore and atomic publication of Tree backup files.""" + +import json + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +import memos.memories.textual.tree as module + +from memos.exceptions import GraphDBError, MemCubeError, MemoryError +from memos.graph_dbs.opengauss import OpenGaussGraphDB +from memos.mem_cube.general import GeneralMemCube +from memos.memories.textual.tree import TreeTextMemory + + +@pytest.fixture +def memory(): + memory = TreeTextMemory.__new__(TreeTextMemory) + memory.config = SimpleNamespace(memory_filename="memory.json") + memory.graph_store = Mock(spec=OpenGaussGraphDB) + memory.graph_store.export_graph.return_value = {"nodes": [{"id": "中文"}], "edges": []} + return memory + + +@pytest.mark.parametrize("legacy", [False, True]) +@pytest.mark.parametrize("include", [False, True]) +def test_dump_projection_and_load(memory, tmp_path, legacy, include): + if legacy: + memory.graph_store = Mock() + memory.graph_store.export_graph.return_value = {"nodes": [], "edges": []} + memory.dump(str(tmp_path), include_embedding=include, user_name="A") + kwargs = {"include_embedding": include, "user_name": "A"} + if not legacy: + kwargs["status"] = [] + memory.graph_store.export_graph.assert_called_once_with(**kwargs) + data = json.loads((tmp_path / "memory.json").read_text(encoding="utf-8")) + assert data == memory.graph_store.export_graph.return_value + memory.load(str(tmp_path), user_name="A") + memory.graph_store.import_graph.assert_called_once_with(data, user_name="A") + assert [p.name for p in tmp_path.iterdir()] == ["memory.json"] + + +@pytest.mark.parametrize( + "content,cause", + [(None, FileNotFoundError), (b"{", json.JSONDecodeError), (b"\xff", UnicodeDecodeError)], +) +@pytest.mark.parametrize("legacy", [False, True]) +def test_load_file_failures_visible(memory, tmp_path, content, cause, legacy): + if legacy: + memory.graph_store = Mock() + if content is not None: + (tmp_path / "memory.json").write_bytes(content) + with pytest.raises(MemoryError) as error: + memory.load(str(tmp_path)) + assert isinstance(error.value.__cause__, cause) + memory.graph_store.import_graph.assert_not_called() + + +def test_load_io_failure(memory, tmp_path, monkeypatch): + failure = PermissionError("read denied") + monkeypatch.setattr("builtins.open", Mock(side_effect=failure)) + with pytest.raises(MemoryError) as error: + memory.load(str(tmp_path)) + assert error.value.__cause__ is failure + + +@pytest.mark.parametrize("operation", ["dump", "load"]) +@pytest.mark.parametrize("failure", [GraphDBError("storage"), ValueError("invalid graph")]) +def test_storage_errors_preserve_identity_and_cause(memory, tmp_path, operation, failure): + failure.__cause__ = OSError("original storage error") + (tmp_path / "memory.json").write_text('{"nodes": [], "edges": []}', encoding="utf-8") + getattr( + memory.graph_store, "export_graph" if operation == "dump" else "import_graph" + ).side_effect = failure + before = (tmp_path / "memory.json").read_bytes() + with pytest.raises(type(failure)) as error: + getattr(memory, operation)(str(tmp_path)) + assert error.value is failure and error.value.__cause__.args == ("original storage error",) + assert (tmp_path / "memory.json").read_bytes() == before + assert len(list(tmp_path.iterdir())) == 1 + + +@pytest.mark.parametrize( + "stage", ["mkdir", "create", "serialize", "write", "flush", "fsync", "close", "replace"] +) +def test_atomic_failure_preserves_old_file_and_cleans_temp(memory, tmp_path, monkeypatch, stage): + target = tmp_path / "memory.json" + target.write_bytes(b"previous backup") + failure = TypeError("serialize") if stage == "serialize" else OSError(stage) + if stage == "mkdir": + monkeypatch.setattr(module.os, "makedirs", Mock(side_effect=failure)) + elif stage == "create": + monkeypatch.setattr(module.tempfile, "NamedTemporaryFile", Mock(side_effect=failure)) + elif stage == "serialize": + monkeypatch.setattr(module.json, "dump", Mock(side_effect=failure)) + elif stage in ("fsync", "replace"): + monkeypatch.setattr(module.os, stage, Mock(side_effect=failure)) + else: + create = module.tempfile.NamedTemporaryFile + + def failing_file(*args, **kwargs): + raw = create(*args, **kwargs) + + class File: + name = raw.name + + def __enter__(self): + return self + + def __exit__(self, *exc): + raw.close() + if stage == "close": + raise failure + + def write(self, data): + if stage == "write": + raise failure + return raw.write(data) + + def flush(self): + if stage == "flush": + raise failure + return raw.flush() + + def fileno(self): + return raw.fileno() + + return File() + + monkeypatch.setattr(module.tempfile, "NamedTemporaryFile", failing_file) + with pytest.raises(MemoryError) as error: + memory.dump(str(tmp_path)) + assert error.value.__cause__ is failure + assert target.read_bytes() == b"previous backup" + assert [p.name for p in tmp_path.iterdir()] == ["memory.json"] + + +def test_atomic_publication_occurs_after_closed_complete_file(memory, tmp_path, monkeypatch): + replace = module.os.replace + calls = [] + sync = module.os.fsync + + def observed_sync(fd): + calls.append("fsync") + return sync(fd) + + def observed_replace(source, destination): + assert calls == ["fsync"] + assert ( + json.loads(module.Path(source).read_text(encoding="utf-8")) + == memory.graph_store.export_graph.return_value + ) + calls.append("replace") + return replace(source, destination) + + monkeypatch.setattr(module.os, "fsync", observed_sync) + monkeypatch.setattr(module.os, "replace", observed_replace) + memory.dump(str(tmp_path)) + assert calls == ["fsync", "replace"] + + +@pytest.mark.parametrize( + "cleanup_error", [FileNotFoundError("gone"), PermissionError("cleanup denied")] +) +def test_cleanup_failure_does_not_mask_primary(memory, tmp_path, monkeypatch, cleanup_error): + primary = OSError("replace") + monkeypatch.setattr(module.os, "replace", Mock(side_effect=primary)) + unlink = module.os.unlink + + def failed_unlink(path): + unlink(path) + raise cleanup_error + + monkeypatch.setattr(module.os, "unlink", failed_unlink) + with pytest.raises(MemoryError) as error: + memory.dump(str(tmp_path)) + assert error.value.__cause__ is primary + + +@pytest.mark.parametrize("kind", ["opengauss", "other_tree", "other_memory", "none"]) +@pytest.mark.parametrize("selection", [None, [], ["text_mem", "act_mem", "para_mem", "pref_mem"]]) +def test_cube_dump_selection_and_embedding(memory, tmp_path, kind, selection): + cube = GeneralMemCube.__new__(GeneralMemCube) + cube.config = Mock(config_filename="config.json") + if kind == "none": + cube._text_mem = None + elif kind == "other_memory": + cube._text_mem = Mock() + else: + memory.dump = Mock() + if kind == "other_tree": + memory.graph_store = Mock() + cube._text_mem = memory + cube._act_mem, cube._para_mem, cube._pref_mem = (Mock() for _ in range(3)) + cube.dump(str(tmp_path), memory_types=selection) + for name in ("text_mem", "act_mem", "para_mem", "pref_mem"): + component = getattr(cube, name) + if component is None: + continue + if selection == []: + component.dump.assert_not_called() + else: + kwargs = ( + {"include_embedding": True} if name == "text_mem" and kind == "opengauss" else {} + ) + component.dump.assert_called_once_with(str(tmp_path), **kwargs) + + +def test_cube_empty_memories_and_directory_guard(tmp_path): + cube = GeneralMemCube.__new__(GeneralMemCube) + cube.config = Mock(config_filename="config.json") + cube._text_mem = cube._act_mem = cube._para_mem = cube._pref_mem = None + cube.dump(str(tmp_path / "new")) + (tmp_path / "occupied").touch() + with pytest.raises(MemCubeError): + cube.dump(str(tmp_path)) diff --git a/tests/memories/textual/test_tree_manager.py b/tests/memories/textual/test_tree_manager.py index 1ad730ee5..24ae80c3a 100644 --- a/tests/memories/textual/test_tree_manager.py +++ b/tests/memories/textual/test_tree_manager.py @@ -72,22 +72,23 @@ def test_process_memory_adds_nodes(memory_manager): confidence=80.0, ), ) - memory_manager._process_memory(memory) # Only pass the single memory item - assert memory_manager.graph_store.add_node.called + memory_manager._process_memory(memory) + nodes = memory_manager.graph_store.add_nodes_batch.call_args.args[0] + assert len(nodes) == 2 and nodes[0]["id"] != nodes[1]["id"] -def test_add_to_graph_memory_merges(memory_manager, mock_graph_store): +def test_process_user_memory_writes_bound_pair(memory_manager, mock_graph_store): memory = TextualMemoryItem( memory="to merge", metadata=TreeNodeTextualMemoryMetadata( embedding=[0.1] * 5, memory_type="UserMemory", confidence=80.0 ), ) - memory_manager._add_to_graph_memory(memory, "UserMemory") - assert mock_graph_store.add_node.called + memory_manager._process_memory(memory) + assert mock_graph_store.add_nodes_batch.called -def test_add_to_graph_memory_creates_new_node(memory_manager, mock_graph_store): +def test_process_long_term_memory_writes_bound_pair(memory_manager, mock_graph_store): mock_graph_store.search_by_embedding.return_value = [{"id": "id1", "score": 0.5}] memory = TextualMemoryItem( memory="new memory", @@ -98,8 +99,8 @@ def test_add_to_graph_memory_creates_new_node(memory_manager, mock_graph_store): key="topic", ), ) - memory_manager._add_to_graph_memory(memory, "LongTermMemory") - assert mock_graph_store.add_node.called + memory_manager._process_memory(memory) + assert mock_graph_store.add_nodes_batch.called def test_inherit_edges(memory_manager, mock_graph_store): diff --git a/tests/memories/textual/test_tree_ownership.py b/tests/memories/textual/test_tree_ownership.py new file mode 100644 index 000000000..fb6b5a3c7 --- /dev/null +++ b/tests/memories/textual/test_tree_ownership.py @@ -0,0 +1,211 @@ +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from memos.api.config import APIConfig +from memos.exceptions import GraphDBError +from memos.mem_cube.general import GeneralMemCube +from memos.memories.textual.base import BaseTextMemory +from memos.memories.textual.simple_tree import SimpleTreeTextMemory +from memos.memories.textual.tree import TreeTextMemory + + +@pytest.fixture +def tree_config(monkeypatch): + monkeypatch.setenv("GRAPH_DB_BACKEND", "neo4j-community") + monkeypatch.setenv("MOS_ENABLE_DEFAULT_CUBE_CONFIG", "true") + monkeypatch.setenv("ENABLE_INTERNET", "false") + monkeypatch.setenv("ENABLE_ACTIVATION_MEMORY", "false") + monkeypatch.setenv("ENABLE_PREFERENCE_MEMORY", "false") + return APIConfig.get_default_cube_config().text_mem.config + + +@pytest.fixture +def tree_factories(monkeypatch): + import memos.memories.textual.tree as module + + for name in ( + "LLMFactory", + "EmbedderFactory", + "GraphStoreFactory", + "RerankerFactory", + "InternetRetrieverFactory", + ): + monkeypatch.setattr(getattr(module, name), "from_config", Mock(return_value=Mock())) + monkeypatch.setattr(module, "MemoryManager", Mock()) + return module + + +def test_tree_owns_graph_and_retries_close(tree_config, tree_factories): + tree = TreeTextMemory(tree_config) + tree.graph_store.close.side_effect = [GraphDBError("close"), None] + with pytest.raises(GraphDBError): + tree.close() + tree.close() + tree.close() + assert tree.graph_store.close.call_count == 2 + + +@pytest.mark.parametrize("stage", ["RerankerFactory", "MemoryManager"]) +def test_tree_construction_failure_releases_graph(tree_config, tree_factories, stage): + target = getattr(tree_factories, stage) + if stage.endswith("Factory"): + target = target.from_config + error = ValueError("consumer failed") + target.side_effect = error + with pytest.raises(ValueError) as raised: + TreeTextMemory(tree_config) + assert raised.value is error + tree_factories.GraphStoreFactory.from_config.return_value.close.assert_called_once_with() + + +def test_simple_tree_borrows_graph(tree_config): + graph = Mock() + tree = SimpleTreeTextMemory(Mock(), Mock(), Mock(), graph, Mock(), Mock(), tree_config) + tree.close() + tree.close() + graph.close.assert_not_called() + + +def cube_config(*backends): + return SimpleNamespace( + **{ + name: SimpleNamespace(backend=backend) + for name, backend in zip( + ("text_mem", "act_mem", "para_mem", "pref_mem"), backends, strict=True + ) + } + ) + + +def test_cube_keeps_replaced_owned_memory_and_deduplicates(monkeypatch): + owned = Mock(spec=BaseTextMemory) + owned.close = Mock() + borrowed = Mock(spec=BaseTextMemory) + borrowed.close = Mock() + monkeypatch.setattr( + "memos.mem_cube.general.MemoryFactory.from_config", Mock(return_value=owned) + ) + cube = GeneralMemCube(cube_config("tree_text", "uninitialized", "uninitialized", "tree_text")) + cube.text_mem = borrowed + cube.close() + cube.close() + owned.close.assert_called_once_with() + borrowed.close.assert_not_called() + + +def test_cube_partial_construction_closes_created_memory(monkeypatch): + first = Mock() + error = ValueError("second failed") + monkeypatch.setattr( + "memos.mem_cube.general.MemoryFactory.from_config", Mock(side_effect=[first, error]) + ) + with pytest.raises(ValueError) as raised: + GeneralMemCube(cube_config("tree_text", "kv_cache", "uninitialized", "uninitialized")) + assert raised.value is error + first.close.assert_called_once_with() + + +def test_cube_preserves_text_initialization_timing_log(monkeypatch, caplog): + memory = Mock() + monkeypatch.setattr( + "memos.mem_cube.general.MemoryFactory.from_config", Mock(return_value=memory) + ) + with caplog.at_level("INFO"): + cube = GeneralMemCube( + cube_config("tree_text", "uninitialized", "uninitialized", "uninitialized") + ) + assert "init_text_mem in " in caplog.text + cube.close() + + +def test_cube_load_failure_closes_new_cube(monkeypatch): + config = cube_config("tree_text", "uninitialized", "uninitialized", "uninitialized") + monkeypatch.setattr( + "memos.mem_cube.general.GeneralMemCubeConfig.from_json_file", lambda _: config + ) + memory = Mock() + monkeypatch.setattr( + "memos.mem_cube.general.MemoryFactory.from_config", Mock(return_value=memory) + ) + error = GraphDBError("restore failed") + monkeypatch.setattr(GeneralMemCube, "load", Mock(side_effect=error)) + with pytest.raises(GraphDBError) as raised: + GeneralMemCube.init_from_dir("unused") + assert raised.value is error + memory.close.assert_called_once_with() + + +def test_cube_close_continues_after_failure_and_retries_only_failed_memory(monkeypatch): + first, second, legacy = Mock(), Mock(), SimpleNamespace() + first.close.side_effect = [GraphDBError("close failed"), None] + monkeypatch.setattr( + "memos.mem_cube.general.MemoryFactory.from_config", + Mock(side_effect=[first, second, legacy]), + ) + cube = GeneralMemCube(cube_config("a", "b", "c", "uninitialized")) + with pytest.raises(GraphDBError): + cube.close() + second.close.assert_called_once_with() + cube.close() + second.close.assert_called_once_with() + assert first.close.call_count == 2 + + +def test_cube_construct_and_restore_keep_primary_error_on_cleanup_failure(monkeypatch): + memory = Mock(close=Mock(side_effect=GraphDBError("cleanup"))) + failure = ValueError("construct") + factory = Mock(side_effect=[memory, failure]) + monkeypatch.setattr("memos.mem_cube.general.MemoryFactory.from_config", factory) + with pytest.raises(ValueError) as raised: + GeneralMemCube(cube_config("a", "b", "uninitialized", "uninitialized")) + assert raised.value is failure + config = cube_config("a", "uninitialized", "uninitialized", "uninitialized") + monkeypatch.setattr( + "memos.mem_cube.general.GeneralMemCubeConfig.from_json_file", lambda _: config + ) + factory.side_effect = None + factory.return_value = memory + monkeypatch.setattr(GeneralMemCube, "load", Mock(side_effect=failure)) + with pytest.raises(ValueError) as raised: + GeneralMemCube.init_from_dir("unused") + assert raised.value is failure + + +def test_cube_load_success_and_default_config_merge(monkeypatch): + config = cube_config("uninitialized", "uninitialized", "uninitialized", "uninitialized") + config.cube_id = "C" + monkeypatch.setattr( + "memos.mem_cube.general.GeneralMemCubeConfig.from_json_file", lambda _: config + ) + merge = Mock(return_value=config) + monkeypatch.setattr("memos.mem_cube.general.merge_config_with_default", merge) + load = Mock() + monkeypatch.setattr(GeneralMemCube, "load", load) + cube = GeneralMemCube.init_from_dir("unused", ["text_mem"], default_config=config) + assert cube.config is config + merge.assert_called_once_with(config, config) + load.assert_called_once_with("unused", ["text_mem"]) + cube.close() + + +def test_tree_optional_internet_and_cleanup_failure(tree_config, tree_factories): + tree_config.internet_retriever = SimpleNamespace(backend="test") + tree = TreeTextMemory(tree_config) + assert ( + tree.internet_retriever is tree_factories.InternetRetrieverFactory.from_config.return_value + ) + tree.close() + graph = tree_factories.GraphStoreFactory.from_config.return_value + graph.close.side_effect = GraphDBError("cleanup") + primary = ValueError("internet") + tree_factories.InternetRetrieverFactory.from_config.side_effect = primary + with pytest.raises(ValueError) as raised: + TreeTextMemory(tree_config) + assert raised.value is primary + borrowed = SimpleTreeTextMemory( + Mock(), Mock(), Mock(), graph, Mock(), Mock(), tree_config, internet_retriever="existing" + ) + assert borrowed.internet_retriever == "existing" + borrowed.close() diff --git a/tests/memories/textual/test_tree_reorganizer_candidates.py b/tests/memories/textual/test_tree_reorganizer_candidates.py new file mode 100644 index 000000000..8c9701181 --- /dev/null +++ b/tests/memories/textual/test_tree_reorganizer_candidates.py @@ -0,0 +1,138 @@ +"""Reorganizer candidate projection and optimize_structure control flow.""" + +import json + +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from memos.memories.textual.tree_text_memory.organize import reorganizer as module +from tests.graph_dbs.test_postgres_400_compat import postgres_db as _postgres_db + + +postgres_db = _postgres_db + + +@pytest.fixture +def case(): + graph = MagicMock() + graph.node_not_exist.return_value = 0 + graph.get_memory_count.return_value = 20 + graph.get_structure_optimization_candidates.return_value = [ + { + "id": f"00000000-0000-4000-8000-{i:012}", + "memory": "fixture", + "metadata": {"memory_type": "LongTermMemory", "embedding": [1, 0, 0]}, + } + for i in range(20) + ] + instance = module.GraphStructureReorganizer(graph, MagicMock(), MagicMock(), False) + instance._is_optimizing = {"LongTermMemory": False, "UserMemory": False} + instance._partition = MagicMock(side_effect=lambda nodes: [nodes]) + instance._process_cluster_and_write = MagicMock() + return instance, graph + + +def test_candidate_embedding_and_cube_reach_nonempty_partition(case): + instance, graph = case + instance.optimize_structure(user_name="B") + graph.get_structure_optimization_candidates.assert_called_once_with( + "LongTermMemory", include_embedding=True, user_name="B" + ) + nodes = instance._partition.call_args.args[0] + assert len(nodes) == 20 and all(n.metadata.embedding == [1, 0, 0] for n in nodes) + instance._process_cluster_and_write.assert_called_once_with(nodes, "LongTermMemory", 10, 4, "B") + assert not instance._is_optimizing["LongTermMemory"] + + +@pytest.mark.parametrize("reason", ["busy", "missing", "empty", "too-small", "no-groups"]) +def test_early_returns_do_not_submit_clusters(case, reason): + instance, graph = case + if reason == "busy": + instance._is_optimizing["LongTermMemory"] = True + elif reason == "missing": + graph.node_not_exist.return_value = 1 + elif reason in {"empty", "too-small"}: + graph.get_structure_optimization_candidates.return_value = ( + [] + if reason == "empty" + else graph.get_structure_optimization_candidates.return_value[:19] + ) + else: + instance._partition.return_value = [] + instance._partition.side_effect = None + instance.optimize_structure(user_name="A") + instance._process_cluster_and_write.assert_not_called() + assert instance._is_optimizing["LongTermMemory"] == (reason == "busy") + if reason in {"busy", "missing"}: + graph.get_structure_optimization_candidates.assert_not_called() + + +@pytest.mark.parametrize("stage", [1, 2, 3, 4]) +def test_each_deadline_resets_busy_flag(case, monkeypatch, stage): + instance, _ = case + logger = MagicMock() + monkeypatch.setattr(module, "logger", logger) + monkeypatch.setattr( + module, "time", SimpleNamespace(time=MagicMock(side_effect=[0] * stage + [601])) + ) + instance.optimize_structure() + logger.error.assert_called_once() + assert not instance._is_optimizing["LongTermMemory"] + assert instance._process_cluster_and_write.call_count == int(stage == 4) + + +def test_candidate_failure_propagates_and_cluster_failure_is_logged(case, monkeypatch): + instance, graph = case + graph.get_structure_optimization_candidates.side_effect = ValueError("candidate failure") + with pytest.raises(ValueError, match="candidate failure"): + instance.optimize_structure() + assert not instance._is_optimizing["LongTermMemory"] + graph.get_structure_optimization_candidates.side_effect = None + instance._process_cluster_and_write.side_effect = ValueError("cluster failure") + logger = MagicMock() + monkeypatch.setattr(module, "logger", logger) + assert instance.optimize_structure() is None + assert "Cluster processing failed" in logger.warning.call_args.args[0] + assert not instance._is_optimizing["LongTermMemory"] + + +@pytest.mark.parametrize("backend", ["neo4j", "polardb", "postgres"]) +def test_existing_candidate_methods_return_requested_embedding(backend, postgres_db): + props = { + "id": "memory-1", + "memory": "fixture", + "status": "activated", + "memory_type": "LongTermMemory", + "user_name": "tenant-a", + } + if backend == "neo4j": + from memos.graph_dbs.neo4j import Neo4jGraphDB + + db = Neo4jGraphDB.__new__(Neo4jGraphDB) + db.config = SimpleNamespace(user_name="tenant-a", use_multi_db=False) + db.db_name = "test" + db.driver = MagicMock() + db.driver.session.return_value.__enter__.return_value.run.return_value = [ + {"id": "memory-1", "node": {**props, "embedding": [1, 0, 0]}} + ] + elif backend == "polardb": + from memos.graph_dbs.polardb import PolarDBGraphDB + + db = PolarDBGraphDB.__new__(PolarDBGraphDB) + db.config = {"user_name": "tenant-a"} + db.db_name = "test" + db._get_connection = MagicMock() + cursor = db._get_connection.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value + cursor.fetchall.return_value = [("[1,0,0]", json.dumps({"properties": props}) + "::vertex")] + else: + db, cursor = postgres_db + cursor.fetchall.return_value = [ + ("memory-1", "fixture", props, None, None, "[1,0,0]", "tenant-a") + ] + result = db.get_structure_optimization_candidates( + "LongTermMemory", include_embedding=True, user_name="tenant-a" + ) + assert len(result) == 1 and result[0]["id"] == "memory-1" + assert result[0]["metadata"]["embedding"] == [1, 0, 0] diff --git a/tests/test_close_state.py b/tests/test_close_state.py new file mode 100644 index 000000000..dc6405a07 --- /dev/null +++ b/tests/test_close_state.py @@ -0,0 +1,47 @@ +"""Concurrent cleanup reports one outcome without sharing raised tracebacks.""" + +from concurrent.futures import ThreadPoolExecutor +from threading import Event +from unittest.mock import Mock + +import pytest + +from memos.exceptions import MemOSError +from memos.lifecycle import CloseState + + +def test_waiter_has_own_exception_and_original_cause(monkeypatch): + state = CloseState() + entered, release, waiting = Event(), Event(), Event() + original = ValueError("resource failed") + + def cleanup(): + entered.set() + assert release.wait(3) + raise original + + with ThreadPoolExecutor(2) as pool: + owner = pool.submit(state.close, cleanup) + assert entered.wait(3) + wait = state._attempt.done.wait + + def observed_wait(): + waiting.set() + return wait(3) + + monkeypatch.setattr(state._attempt.done, "wait", observed_wait) + waiter = pool.submit(state.close, Mock()) + try: + assert waiting.wait(3) + finally: + release.set() + with pytest.raises(ValueError) as first: + owner.result(3) + with pytest.raises(MemOSError) as second: + waiter.result(3) + assert first.value is original + assert second.value is not original and second.value.__cause__ is original + retry = Mock() + state.close(retry) + state.close(retry) + retry.assert_called_once()