Skip to content

fix: memory error #2028

Closed
Closed

Description

@CarltonXiang

Pre-submission checklist | 提交前检查

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
  • This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题

Bug Description | 问题描述

i use the plugin in hermes, the error.log of hermes as following:

2026-05-13 16:37:59,610 WARNING [248119d5525e] run_agent: Tool memory_search returned error (30.00s): {"error": "[timeout] memory.search did not respond within 30.0s"}
2026-05-13 16:39:05,618 WARNING [248119d5525e] plugins.memory.memtensor: MemOS: sync_turn turn.end failed — [timeout] turn.end did not respond within 30.0s

How to Reproduce | 如何重现

start
after one or tow hour, ask the agent " What is there in your memory?"

Environment | 环境信息

  • pyhton 3.11
  • ubuntu 26
  • memos v2.0.1

Additional Context | 其他信息

No response

Willingness to Implement | 实现意愿

  • I'm willing to implement this myself | 我愿意自己解决
  • I would like someone else to implement this | 我希望其他人来解决

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jul 2, 2026
  2. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 222a060feaf4fae7
    Working branch: bugfix/autodev-2028
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  3. self-assigned this
    on Jul 2, 2026
  4. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 222a060feaf4fae7 completed.

    Summary: Fixes Hermes plugin memory_search / turn.end 30 s timeout regression reported in issue #2028. The Python-side JSON-RPC bridge client silently allowed callers to park for the full per-request timeout whenever the Node bridge subprocess died or hung, and the keepalive path only recovered from transport_closed — a BridgeError("timeout") was logged at DEBUG and dropped, so the dead bridge was reused forever.

    Three cooperating fixes were applied to apps/memos-local-plugin/adapters/hermes/memos_provider/: (1) MemosBridgeClient._read_loop now uses try/finally + a new _abort_pending helper so reader-thread exit immediately wakes every parked waiter with transport_closed; (2) MemosBridgeClient.request checks Popen.poll() before writing to stdin, so a dead subprocess fast-fails; (3) MemTensorProvider.handle_tool_call now routes memos_search / memos_get / memos_timeline / memos_skill_list / memos_environment / memos_skill_get through a new _bridge_request_with_retry helper that reconnects and retries once on transport_closed. Keepalive reconnects on timeout / dead subprocess in addition to transport_closed via _should_reconnect_after_keepalive_failure. Write-path (sync_turn) retry semantics are unchanged.

    Tests: added 7 BridgeRecoveryTests covering all four layers (reader-exit wake-up, dead-subprocess fast-fail, keepalive-timeout reconnect, keepalive-dead-subprocess reconnect, keepalive-generic-error no reconnect, tool retry success, tool retry failure surfaced). Full runs: python3 -m unittest tests.python.test_bridge_client → 33 tests OK (26 → 33); tests.python.test_hermes_provider_pipeline → 16 tests OK; ruff check + ruff format --check clean.

    Note: issue #2028 is a re-report of the closed #1722 regression because the fix landed on bugfix/autodev-1722 against the same base branch (dev-20260624-v2.0.22) and was not merged forward. The same design applies verbatim.

    Base branch: main
    Branch: bugfix/autodev-2028
    Commit: 3359962
    PR: #2030
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  5. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:codingAI agent is coding | AI 正在编码
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Jul 2, 2026
  6. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 0ceb1ab3d6a84736
    Working branch: bugfix/autodev-2028
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  7. added a commit that references this issue on Jul 2, 2026
    3e808ad
  8. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 0ceb1ab3d6a84736 completed.

    Summary: Root cause of the reported "[timeout] memory.search did not respond within 30.0s" and "[timeout] turn.end did not respond within 30.0s" errors (issue #2028) is that the Hermes memtensor provider (apps/memos-local-plugin/adapters/hermes/memos_provider/init.py) was calling _bridge.request(...) for four heavy pipeline methods — memory.search (invoked from both memos_search and the keyword branch of memos_environment), turn.start, and turn.end — without a timeout= kwarg, so all four fell back to the 30s JSON-RPC default. After 1-2 hours of continuous Hermes use, the accumulated MemOS memory makes the V7 retrieval / capture / reflection pipeline legitimately run past 30s and the calls time out even though the bridge subprocess is alive.

    The fix introduces a module-level _LONG_RPC_TIMEOUT (default 75.0s, override via MEMOS_HERMES_LONG_RPC_TIMEOUT; unparseable or non-positive values fall back to the default) and threads it through those four call sites. 75s matches the ceiling that feedback.submit and sync_turn's _ensure_bridge already opt into, so we're closing an internal inconsistency rather than inventing a new magic number. Short-latency RPCs (session.open, core.health, memory.get_trace/policy/world, memory.timeline, skill.list/get, subagent.record, session.close, memory.list_world_models) intentionally stay on the existing 30s fast-path so genuine regressions there stay visible.

    TDD evidence: four new tests were added to tests/python/test_bridge_client.py (test_memos_search_uses_long_rpc_timeout, test_memos_environment_search_uses_long_rpc_timeout, test_sync_turn_uses_long_rpc_timeout_for_turn_end, test_prefetch_uses_long_rpc_timeout_for_turn_start). All four failed against the pre-fix code (recorded RPC kwargs had no timeout key, so kwargs.get("timeout", 0.0) came back as 0.0 < 75.0) and pass after the fix. Combined pytest run for the two Hermes test modules: python3 -m unittest tests.python.test_bridge_client tests.python.test_hermes_provider_pipeline reports Ran 46 tests in 30.153s / OK (30 in test_bridge_client, 16 in test_hermes_provider_pipeline). ruff check and ruff format --check are clean on both modified files.

    opsp artefacts (task.md, proposal.md, design.md, spec.md, verification-report.md) have been archived to the sibling memos-autodev-specs repo under 2026-07-02-2028-fix-memory-error/ and pushed to main. Working branch bugfix/autodev-2028 has been pushed to origin at commit 3e808ad. The change is scoped to the plugin and does not touch src/memos/api/ so no OpenAPI regeneration is needed.

    Base branch: main
    Branch: bugfix/autodev-2028
    Commit: 3e808ad
    PR: #2031
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  9. 2 remaining items

  10. added
    ai:codingAI agent is coding | AI 正在编码
    and removed
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jul 2, 2026
  11. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: f7f2ab26022eef23
    Working branch: bugfix/autodev-2028
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  12. added a commit that references this issue on Jul 2, 2026
    02af2c7
  13. Memtensor-AI commented on Jul 2, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task f7f2ab26022eef23 completed.

    Summary: Fixes #2028 by recovering the Hermes MemOS plugin from a hung Node bridge subprocess. Issue #2028 has a byte-for-byte identical error signature to already-solved #1722 ([timeout] memory.search did not respond within 30.0s + [timeout] turn.end did not respond within 30.0s), but the #1722 fix was never merged onto base branch dev-20260624-v2.0.22; the current base still carries all three cooperating defects. This change ports that fix forward.

    The fix addresses three Python-adapter defects that turn "Node bridge subprocess dies quietly" into "every memory tool times out for 30 s and then fails forever". First, MemosBridgeClient._read_loop was wrapped in try/finally that calls the new _abort_pending() helper on any exit, waking every parked JSON-RPC waiter with transport_closed in under a second (previously they parked on the full per-request timeout). Second, MemosBridgeClient.request now checks Popen.poll() before writing, so a request against an already-exited subprocess fast-fails instead of buffering into a dead pipe. Third, the keepalive loop was routed through a new _should_reconnect_after_keepalive_failure() predicate that reconnects on BridgeError("timeout", …) and dead-subprocess in addition to the pre-existing transport_closed trigger. Finally, all six read-path memory tools (memos_search, memos_get, memos_timeline, memos_environment, memos_skill_list, memos_skill_get) were routed through the new _bridge_request_with_retry() helper that reconnects + retries once on transport_closed, mirroring the existing sync_turn pattern.

    Verified with 7 new unit tests (2 at bridge level: reader-exit abort + poll-based fast-fail; 3 at keepalive predicate level: timeout / dead-subprocess / no-storm-on-transient; 2 at read-path retry level: reconnect-and-recover + retry-still-fails-surfaces-error). Combined test_bridge_client + test_hermes_provider_pipeline run: 49 tests pass in 0.28 s — down from 32.19 s pre-fix, direct evidence that the reader-thread abort actually wakes pending waiters. ruff check and ruff format --check clean across the plugin and its tests.

    Scope is Python-only: no wire-protocol change, no dependency changes, no Node bridge changes, no config changes. The per-request 30 s timeout budget is preserved for genuinely slow calls. Shutdown races are unchanged (the existing _bridge_keepalive_stop gate still guards reconnects). Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller.

    Base branch: main
    Branch: bugfix/autodev-2028
    Commit: 02af2c7
    PR: #2036
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  14. added
    ai:testingAI agent is running tests | AI 正在运行测试
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Jul 2, 2026
  15. added a commit that references this issue on Jul 2, 2026
    1771568
  16. added and removed
    ai:testingAI agent is running tests | AI 正在运行测试
    on Jul 2, 2026
  17. added 2 commits that reference this issue on Sep 18, 2026
    45657ed
    f261d03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:pluginOpenClaw & Hermestypes:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions