You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题
Bug Description | 问题描述
i use the plugin in hermes, the error.log of hermes as following:
2026-05-13 16:37:59,610 WARNING [248119d5525e] run_agent: Tool memory_search returned error (30.00s): {"error": "[timeout] memory.search did not respond within 30.0s"}
2026-05-13 16:39:05,618 WARNING [248119d5525e] plugins.memory.memtensor: MemOS: sync_turn turn.end failed — [timeout] turn.end did not respond within 30.0s
How to Reproduce | 如何重现
start
after one or tow hour, ask the agent " What is there in your memory?"
Environment | 环境信息
pyhton 3.11
ubuntu 26
memos v2.0.1
Additional Context | 其他信息
No response
Willingness to Implement | 实现意愿
I'm willing to implement this myself | 我愿意自己解决
I would like someone else to implement this | 我希望其他人来解决
Summary: Fixes Hermes plugin memory_search / turn.end 30 s timeout regression reported in issue #2028. The Python-side JSON-RPC bridge client silently allowed callers to park for the full per-request timeout whenever the Node bridge subprocess died or hung, and the keepalive path only recovered from transport_closed — a BridgeError("timeout") was logged at DEBUG and dropped, so the dead bridge was reused forever.
Three cooperating fixes were applied to apps/memos-local-plugin/adapters/hermes/memos_provider/: (1) MemosBridgeClient._read_loop now uses try/finally + a new _abort_pending helper so reader-thread exit immediately wakes every parked waiter with transport_closed; (2) MemosBridgeClient.request checks Popen.poll() before writing to stdin, so a dead subprocess fast-fails; (3) MemTensorProvider.handle_tool_call now routes memos_search / memos_get / memos_timeline / memos_skill_list / memos_environment / memos_skill_get through a new _bridge_request_with_retry helper that reconnects and retries once on transport_closed. Keepalive reconnects on timeout / dead subprocess in addition to transport_closed via _should_reconnect_after_keepalive_failure. Write-path (sync_turn) retry semantics are unchanged.
Tests: added 7 BridgeRecoveryTests covering all four layers (reader-exit wake-up, dead-subprocess fast-fail, keepalive-timeout reconnect, keepalive-dead-subprocess reconnect, keepalive-generic-error no reconnect, tool retry success, tool retry failure surfaced). Full runs: python3 -m unittest tests.python.test_bridge_client → 33 tests OK (26 → 33); tests.python.test_hermes_provider_pipeline → 16 tests OK; ruff check + ruff format --check clean.
Note: issue #2028 is a re-report of the closed #1722 regression because the fix landed on bugfix/autodev-1722 against the same base branch (dev-20260624-v2.0.22) and was not merged forward. The same design applies verbatim.
Summary: Root cause of the reported "[timeout] memory.search did not respond within 30.0s" and "[timeout] turn.end did not respond within 30.0s" errors (issue #2028) is that the Hermes memtensor provider (apps/memos-local-plugin/adapters/hermes/memos_provider/init.py) was calling _bridge.request(...) for four heavy pipeline methods — memory.search (invoked from both memos_search and the keyword branch of memos_environment), turn.start, and turn.end — without a timeout= kwarg, so all four fell back to the 30s JSON-RPC default. After 1-2 hours of continuous Hermes use, the accumulated MemOS memory makes the V7 retrieval / capture / reflection pipeline legitimately run past 30s and the calls time out even though the bridge subprocess is alive.
The fix introduces a module-level _LONG_RPC_TIMEOUT (default 75.0s, override via MEMOS_HERMES_LONG_RPC_TIMEOUT; unparseable or non-positive values fall back to the default) and threads it through those four call sites. 75s matches the ceiling that feedback.submit and sync_turn's _ensure_bridge already opt into, so we're closing an internal inconsistency rather than inventing a new magic number. Short-latency RPCs (session.open, core.health, memory.get_trace/policy/world, memory.timeline, skill.list/get, subagent.record, session.close, memory.list_world_models) intentionally stay on the existing 30s fast-path so genuine regressions there stay visible.
TDD evidence: four new tests were added to tests/python/test_bridge_client.py (test_memos_search_uses_long_rpc_timeout, test_memos_environment_search_uses_long_rpc_timeout, test_sync_turn_uses_long_rpc_timeout_for_turn_end, test_prefetch_uses_long_rpc_timeout_for_turn_start). All four failed against the pre-fix code (recorded RPC kwargs had no timeout key, so kwargs.get("timeout", 0.0) came back as 0.0 < 75.0) and pass after the fix. Combined pytest run for the two Hermes test modules: python3 -m unittest tests.python.test_bridge_client tests.python.test_hermes_provider_pipeline reports Ran 46 tests in 30.153s / OK (30 in test_bridge_client, 16 in test_hermes_provider_pipeline). ruff check and ruff format --check are clean on both modified files.
opsp artefacts (task.md, proposal.md, design.md, spec.md, verification-report.md) have been archived to the sibling memos-autodev-specs repo under 2026-07-02-2028-fix-memory-error/ and pushed to main. Working branch bugfix/autodev-2028 has been pushed to origin at commit 3e808ad. The change is scoped to the plugin and does not touch src/memos/api/ so no OpenAPI regeneration is needed.
Summary: Fixes #2028 by recovering the Hermes MemOS plugin from a hung Node bridge subprocess. Issue #2028 has a byte-for-byte identical error signature to already-solved #1722 ([timeout] memory.search did not respond within 30.0s + [timeout] turn.end did not respond within 30.0s), but the #1722 fix was never merged onto base branch dev-20260624-v2.0.22; the current base still carries all three cooperating defects. This change ports that fix forward.
The fix addresses three Python-adapter defects that turn "Node bridge subprocess dies quietly" into "every memory tool times out for 30 s and then fails forever". First, MemosBridgeClient._read_loop was wrapped in try/finally that calls the new _abort_pending() helper on any exit, waking every parked JSON-RPC waiter with transport_closed in under a second (previously they parked on the full per-request timeout). Second, MemosBridgeClient.request now checks Popen.poll() before writing, so a request against an already-exited subprocess fast-fails instead of buffering into a dead pipe. Third, the keepalive loop was routed through a new _should_reconnect_after_keepalive_failure() predicate that reconnects on BridgeError("timeout", …) and dead-subprocess in addition to the pre-existing transport_closed trigger. Finally, all six read-path memory tools (memos_search, memos_get, memos_timeline, memos_environment, memos_skill_list, memos_skill_get) were routed through the new _bridge_request_with_retry() helper that reconnects + retries once on transport_closed, mirroring the existing sync_turn pattern.
Verified with 7 new unit tests (2 at bridge level: reader-exit abort + poll-based fast-fail; 3 at keepalive predicate level: timeout / dead-subprocess / no-storm-on-transient; 2 at read-path retry level: reconnect-and-recover + retry-still-fails-surfaces-error). Combined test_bridge_client + test_hermes_provider_pipeline run: 49 tests pass in 0.28 s — down from 32.19 s pre-fix, direct evidence that the reader-thread abort actually wakes pending waiters. ruff check and ruff format --check clean across the plugin and its tests.
Scope is Python-only: no wire-protocol change, no dependency changes, no Node bridge changes, no config changes. The per-request 30 s timeout budget is preserved for genuinely slow calls. Shutdown races are unchanged (the existing _bridge_keepalive_stop gate still guards reconnects). Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller.
Activity
Memtensor-AI commentedon Jul 2, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
222a060feaf4fae7Working branch:
bugfix/autodev-2028Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(memos-local-plugin): recover memory_search + turn.end after hung …
Memtensor-AI commentedon Jul 2, 2026
✅ AutoDev task
222a060feaf4fae7completed.Summary: Fixes Hermes plugin memory_search / turn.end 30 s timeout regression reported in issue #2028. The Python-side JSON-RPC bridge client silently allowed callers to park for the full per-request timeout whenever the Node bridge subprocess died or hung, and the keepalive path only recovered from transport_closed — a BridgeError("timeout") was logged at DEBUG and dropped, so the dead bridge was reused forever.
Three cooperating fixes were applied to apps/memos-local-plugin/adapters/hermes/memos_provider/: (1) MemosBridgeClient._read_loop now uses try/finally + a new _abort_pending helper so reader-thread exit immediately wakes every parked waiter with transport_closed; (2) MemosBridgeClient.request checks Popen.poll() before writing to stdin, so a dead subprocess fast-fails; (3) MemTensorProvider.handle_tool_call now routes memos_search / memos_get / memos_timeline / memos_skill_list / memos_environment / memos_skill_get through a new _bridge_request_with_retry helper that reconnects and retries once on transport_closed. Keepalive reconnects on timeout / dead subprocess in addition to transport_closed via _should_reconnect_after_keepalive_failure. Write-path (sync_turn) retry semantics are unchanged.
Tests: added 7 BridgeRecoveryTests covering all four layers (reader-exit wake-up, dead-subprocess fast-fail, keepalive-timeout reconnect, keepalive-dead-subprocess reconnect, keepalive-generic-error no reconnect, tool retry success, tool retry failure surfaced). Full runs: python3 -m unittest tests.python.test_bridge_client → 33 tests OK (26 → 33); tests.python.test_hermes_provider_pipeline → 16 tests OK; ruff check + ruff format --check clean.
Note: issue #2028 is a re-report of the closed #1722 regression because the fix landed on bugfix/autodev-1722 against the same base branch (dev-20260624-v2.0.22) and was not merged forward. The same design applies verbatim.
Base branch:
mainBranch:
bugfix/autodev-2028Commit:
3359962PR: #2030
Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Memtensor-AI commentedon Jul 2, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
0ceb1ab3d6a84736Working branch:
bugfix/autodev-2028Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(memos-local-plugin): extend memory.search/turn.end RPC timeout to…
Memtensor-AI commentedon Jul 2, 2026
✅ AutoDev task
0ceb1ab3d6a84736completed.Summary: Root cause of the reported "[timeout] memory.search did not respond within 30.0s" and "[timeout] turn.end did not respond within 30.0s" errors (issue #2028) is that the Hermes memtensor provider (apps/memos-local-plugin/adapters/hermes/memos_provider/init.py) was calling
_bridge.request(...)for four heavy pipeline methods — memory.search (invoked from both memos_search and the keyword branch of memos_environment), turn.start, and turn.end — without atimeout=kwarg, so all four fell back to the 30s JSON-RPC default. After 1-2 hours of continuous Hermes use, the accumulated MemOS memory makes the V7 retrieval / capture / reflection pipeline legitimately run past 30s and the calls time out even though the bridge subprocess is alive.The fix introduces a module-level
_LONG_RPC_TIMEOUT(default 75.0s, override viaMEMOS_HERMES_LONG_RPC_TIMEOUT; unparseable or non-positive values fall back to the default) and threads it through those four call sites. 75s matches the ceiling thatfeedback.submitandsync_turn's_ensure_bridgealready opt into, so we're closing an internal inconsistency rather than inventing a new magic number. Short-latency RPCs (session.open, core.health, memory.get_trace/policy/world, memory.timeline, skill.list/get, subagent.record, session.close, memory.list_world_models) intentionally stay on the existing 30s fast-path so genuine regressions there stay visible.TDD evidence: four new tests were added to tests/python/test_bridge_client.py (test_memos_search_uses_long_rpc_timeout, test_memos_environment_search_uses_long_rpc_timeout, test_sync_turn_uses_long_rpc_timeout_for_turn_end, test_prefetch_uses_long_rpc_timeout_for_turn_start). All four failed against the pre-fix code (recorded RPC kwargs had no
timeoutkey, sokwargs.get("timeout", 0.0)came back as 0.0 < 75.0) and pass after the fix. Combined pytest run for the two Hermes test modules:python3 -m unittest tests.python.test_bridge_client tests.python.test_hermes_provider_pipelinereportsRan 46 tests in 30.153s / OK(30 in test_bridge_client, 16 in test_hermes_provider_pipeline).ruff checkandruff format --checkare clean on both modified files.opsp artefacts (task.md, proposal.md, design.md, spec.md, verification-report.md) have been archived to the sibling memos-autodev-specs repo under 2026-07-02-2028-fix-memory-error/ and pushed to main. Working branch bugfix/autodev-2028 has been pushed to origin at commit 3e808ad. The change is scoped to the plugin and does not touch src/memos/api/ so no OpenAPI regeneration is needed.
Base branch:
mainBranch:
bugfix/autodev-2028Commit:
3e808adPR: #2031
Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
2 remaining items
Memtensor-AI commentedon Jul 2, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
f7f2ab26022eef23Working branch:
bugfix/autodev-2028Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(memos-local-plugin): recover memory_search + turn.end after hung …
Memtensor-AI commentedon Jul 2, 2026
✅ AutoDev task
f7f2ab26022eef23completed.Summary: Fixes #2028 by recovering the Hermes MemOS plugin from a hung Node bridge subprocess. Issue #2028 has a byte-for-byte identical error signature to already-solved #1722 (
[timeout] memory.search did not respond within 30.0s+[timeout] turn.end did not respond within 30.0s), but the #1722 fix was never merged onto base branchdev-20260624-v2.0.22; the current base still carries all three cooperating defects. This change ports that fix forward.The fix addresses three Python-adapter defects that turn "Node bridge subprocess dies quietly" into "every memory tool times out for 30 s and then fails forever". First,
MemosBridgeClient._read_loopwas wrapped in try/finally that calls the new_abort_pending()helper on any exit, waking every parked JSON-RPC waiter withtransport_closedin under a second (previously they parked on the full per-request timeout). Second,MemosBridgeClient.requestnow checksPopen.poll()before writing, so a request against an already-exited subprocess fast-fails instead of buffering into a dead pipe. Third, the keepalive loop was routed through a new_should_reconnect_after_keepalive_failure()predicate that reconnects onBridgeError("timeout", …)and dead-subprocess in addition to the pre-existingtransport_closedtrigger. Finally, all six read-path memory tools (memos_search,memos_get,memos_timeline,memos_environment,memos_skill_list,memos_skill_get) were routed through the new_bridge_request_with_retry()helper that reconnects + retries once ontransport_closed, mirroring the existingsync_turnpattern.Verified with 7 new unit tests (2 at bridge level: reader-exit abort + poll-based fast-fail; 3 at keepalive predicate level: timeout / dead-subprocess / no-storm-on-transient; 2 at read-path retry level: reconnect-and-recover + retry-still-fails-surfaces-error). Combined
test_bridge_client+test_hermes_provider_pipelinerun: 49 tests pass in 0.28 s — down from 32.19 s pre-fix, direct evidence that the reader-thread abort actually wakes pending waiters.ruff checkandruff format --checkclean across the plugin and its tests.Scope is Python-only: no wire-protocol change, no dependency changes, no Node bridge changes, no config changes. The per-request 30 s timeout budget is preserved for genuinely slow calls. Shutdown races are unchanged (the existing
_bridge_keepalive_stopgate still guards reconnects). Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller.Base branch:
mainBranch:
bugfix/autodev-2028Commit:
02af2c7PR: #2036
Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Fix #2028: fix: memory error (#2036)
Fix MemTensor#2028: fix: memory error (MemTensor#2031)
Fix MemTensor#2028: fix: memory error (MemTensor#2036)