You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix #1722: fix: memory_search returned error - #2017
Fixed issue #1722: after Hermes ran for 1-2 hours, memory_search and turn.end permanently returned [timeout] … did not respond within 30.0s and never recovered. Root cause: three cooperating defects in apps/memos-local-plugin/adapters/hermes/memos_provider/ on the Python side turned "bridge subprocess dies quietly" into "every memory tool blocks 30 s and then fails forever" — the reader thread exited silently on EOF (leaving pending waiters parked until each individual timeout expired), the keepalive only reconnected on transport_closed (ignoring BridgeError("timeout", …) — the exact symptom), and the read-path tools had no retry mirroring sync_turn.
Fix, without touching the JSON-RPC wire protocol or the Node bridge: (1) MemosBridgeClient._read_loop now runs in try/except/finally and always calls a new _abort_pending("bridge subprocess exited") that wakes every parked waiter with BridgeError("transport_closed", …) in <1 s; (2) MemosBridgeClient.request short-circuits with transport_closed when Popen.poll() already reports a non-None exit code, so writes never go to a dead pipe; (3) close() was refactored to share teardown via a new _close_called flag; (4) _start_bridge_keepalive now delegates to _should_reconnect_after_keepalive_failure(err) — reconnects on transport_closed, BridgeError("timeout", …), and any error observed while the subprocess is already dead; (5) handle_tool_call routes every read-path tool (memos_search, memos_get, memos_timeline, memos_environment, memos_skill_list, memos_skill_get) through a new _bridge_request_with_retry helper that reconnects + retries once on transport_closed, mirroring sync_turn.
Tests: 7 new cases in apps/memos-local-plugin/tests/python/test_bridge_client.py cover reader-exit abort, fast-fail on dead poll(), read-path recover branch, read-path still-broken branch, keepalive reconnect on health timeout, keepalive reconnect on dead subprocess, and the negative case (do NOT reconnect on transient generic error while the subprocess is alive — avoids reconnect storms). All 33 bridge tests + 16 hermes pipeline tests pass; ruff check and ruff format --check are clean across the modified files.
Branch bugfix/autodev-1722 pushed to origin at commit 7c9fc1f0. Archive artifacts (task.md + proposal/spec/design/verification-report) were also synced to memos-autodev-specs main. No wire-protocol / config / dependency changes; no impact outside the Hermes adapter.
Race condition (fragile test timing).ready.set() fires at the very top of _do_request, beforeclient.request(...) is called — so when ready.wait() returns, the worker thread has only just started executing request(). The 50 ms sleep is the sole guard between here and the worker entering waiter.wait(timeout=5.0). On a heavily loaded CI runner the 50 ms budget can be consumed before request() reaches waiter.wait(), meaning the test will simulate subprocess death while the worker is still inside the poll() fast-fail check — exercising the wrong code path (fast-fail rather than _abort_pending) yet still "passing".
A deterministic fix is to move ready.set() to just after the waiter is parked. The cleanest approach without changing the production code is to use a second threading.Event that is set from a callback installed via _pending introspection, or — more simply — to move the ready.set() call after a deliberate blocking stub:
def_do_request() ->None:
# park on the waiter first, then signal the main threaddef_hook(*_, **__):
ready.set() # signal *after* we're inside request()raiseBridgeError("never", "placeholder") # replaced by abort_pending# Alternatively: patch the _ServerStream to signal on first iteration
...
A simpler pattern: set ready via a patched threading.Event.wait wrapper, or use a longer-but-bounded worker.join(timeout) with an explicit assertion rather than relying on timing.
💡 Suggested Change
Before:
worker = threading.Thread(target=_do_request, daemon=True)
worker.start()
self.assertTrue(ready.wait(timeout=1.0))
# Give the worker a beat to actually enter `waiter.wait(...)`.
time.sleep(0.05)
After:
worker = threading.Thread(target=_do_request, daemon=True)
worker.start()
self.assertTrue(ready.wait(timeout=1.0))
# Give the worker a beat to actually enter `waiter.wait(...)`.
# NOTE: this sleep is inherently racy — on slow CI machines 50 ms may
# not be enough for request() to reach waiter.wait(), causing the test
# to exercise the fast-fail poll() path instead of _abort_pending.
# Consider using a deterministic synchronization primitive instead.
time.sleep(0.05)
Dead code in this diff — terminate() is never called by any of the new tests. None of the five new test methods invoke terminate(), and the production close() path calls proc.terminate() only inside bridge_client.py's close(). None of the new tests exercise the full close() path through FakePopen.terminate(). The method is harmless to keep as infrastructure for future tests, but the magic constant -15 (SIGTERM) should at minimum be documented or replaced with a named constant. More importantly, if terminate() is meant to gate on the production close() path, at least one test should verify the behaviour.
Race condition: subprocess can exit between poll() and _pending registration.
The poll() pre-flight check at line 185 and the self._pending[rpc_id] = entry insertion at line 196 are not atomic — they happen outside the same critical section. The sequence of events that reintroduces issue #1722:
poll() returns None (subprocess alive).
The subprocess exits; reader thread calls _abort_pending, which clears _pending (which is currently empty).
This thread acquires _lock, inserts the entry, writes to stdin, and releases _lock.
The entry now lives in _pending permanently and waits the full timeout for a response that will never arrive.
The window is small but non-zero under load. To close it, move the poll() check inside the _lock block, before (or immediately after) the entry is registered in _pending, and add a double-check after acquiring the lock:
Alternatively, in _abort_pending, after setting self._closed = True under _host_handlers_cv, re-check _pending under _lock to drain any entries that snuck in after the initial clear().
💡 Suggested Change
Before:
exit_code = self._proc.poll()
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
with self._lock:
After:
with self._lock:
exit_code = self._proc.poll()
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
Shared mutable error dict assigned to all pending entries.
The same error dict object reference is written to every entry's entry["error"] field. If any consumer code downstream (e.g., in future error-enrichment logic) mutates the dict — by adding context keys, popping fields, or modifying nested values — it would silently corrupt the error visible to all other concurrent callers that received the same reference.
Consider copying the dict per entry or using a frozen/immutable representation:
forentryinentries:
entry["error"] =dict(error) # shallow copy per entryentry["event"].set()
_close_called guard is set before the critical section completes, making a partial failure unrecoverable.
_close_called = True is assigned unconditionally before with self._host_handlers_cv and all subsequent cleanup steps (stdin close, process wait, SIGTERM/SIGKILL, _abort_pending). If any of those steps raises an unhandled exception — for example self._proc.stdin.close() leaking beyond contextlib.suppress, or a truly unexpected error in _abort_pending — close() can never be called again because the early-return guard has already tripped. This permanently leaks the client and its background reader thread.
The guard's intent is only to prevent concurrent/re-entrant calls, not to survive exceptions inside close(). Consider protecting only the concurrent-call race with a threading.Lock, or wrap the body in try/finally so cleanup still runs on exceptions:
defclose(self) ->None:
ifself._close_called:
returnself._close_called=Truetry:
withself._host_handlers_cv:
...
# ... rest of cleanup ...exceptException:
logger.exception("MemOS: unexpected error during bridge close")
_reconnect_bridge is protected by self._reconnect_lock, but reads and writes to self._bridge outside that lock are unprotected. Two concurrent tool invocations that both catch a transport_closed error will each call _reconnect_bridge, each potentially spawning a new subprocess and leaking the other. More critically, the keepalive thread can null-out self._bridge mid-reconnect while this code is about to call self._bridge.request(...) after the retry, producing an AttributeError instead of a BridgeError.
Suggested fix: wrap the entire try/except/reconnect/retry block under self._reconnect_lock, matching the same protection already applied inside _reconnect_bridge. This ensures only one thread reconnects at a time and all others wait for the restored bridge:
withself._reconnect_lock:
try:
returnself._bridge.request(method, params, **kwargs) or {}
exceptExceptionaserr:
ifnotself._is_transport_closed(err):
raise
...
self._reconnect_bridge(self._session_id, timeout=30.0)
assertself._bridgeisnotNonereturnself._bridge.request(method, params, **kwargs) or {}
Falsy-value collapse via or {} may swallow valid empty responses (Medium)
self._bridge.request(...) or {} collapses any falsy return value — including an empty list [] — to {}. An empty list is a perfectly valid JSON-RPC result for memory.search with zero hits; replacing it with {} breaks call sites that iterate the result (e.g., for hit in resp["hits"] will raise KeyError). The same pattern appears on the retry path (line 1823).
Suggested fix: use an explicit None-guard instead:
Return type annotation dict[str, Any] is too narrow (Low)
bridge_client.request() can return a list (e.g., search-result arrays) as well as a dict. Annotating the return as dict[str, Any] will produce type-checker errors at call sites that treat the result as a list, and misrepresents the actual contract.
Suggested fix: broaden to dict[str, Any] | list[Any] (or whatever union is already used at other request() call sites in this file).
Overly broad contextlib.suppress(Exception) around proc.poll() (Low)
Suppressing all exceptions hides legitimate programming mistakes. If proc.poll() raises (e.g., OSError on a zombie process in certain OS states), the broad suppression silently leaves the decision at False, causing the reconnect to be skipped.
Suggested fix: narrow the suppression to only the expected OS-level exception:
Race condition: subprocess can exit between poll() and _pending registration.
The poll() pre-flight check and the self._pending[rpc_id] = entry insertion are not atomic — they happen in separate critical sections. The problematic sequence:
poll() returns None (subprocess alive).
The subprocess exits; reader thread calls _abort_pending, which clears _pending (currently empty).
This thread acquires _lock, inserts the new entry, writes to stdin, and releases _lock.
The orphaned entry now lives in _pending permanently and waits the full timeout.
This is the same stall that fix #1722 aims to prevent. To close the window, move the poll() check inside the _lock block — after acquiring the lock but before registering the entry:
withself._lock:
# Re-check atomically under the lock that protects _pending.exit_code=self._proc.poll()
ifexit_codeisnotNone:
raiseBridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
rpc_id=self._next_id
...
💡 Suggested Change
Before:
exit_code = self._proc.poll()
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
with self._lock:
After:
with self._lock:
exit_code = self._proc.poll()
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
Shared mutable error dict assigned to all pending entries.
The same error dict object reference is written to every entry["error"]. If any downstream consumer ever mutates the dict (e.g., adds context keys, pops fields, or modifies nested values), it would silently corrupt the error visible to all other concurrent callers that received the same reference.
Use a per-entry copy to isolate each caller:
forentryinentries:
entry["error"] =dict(error) # shallow copy per entryentry["event"].set()
_close_called guard is set before cleanup completes, making a partial failure unrecoverable.
_close_called = True is assigned before the with self._host_handlers_cv block and all subsequent cleanup (stdin close, process wait, SIGTERM/SIGKILL, _abort_pending). If any of those steps raises an unhandled exception, close() can never be retried because the guard has already tripped, permanently leaking the client and its background reader thread.
The guard's intent is only to prevent re-entrant concurrent calls. Consider wrapping the body in try/except so cleanup errors are logged but don't leave the client in a permanently-uncloseable state:
defclose(self) ->None:
ifself._close_called:
returnself._close_called=Truetry:
withself._host_handlers_cv:
...
# ... rest of cleanup ...exceptException:
logger.exception("MemOS: unexpected error during bridge close")
Race condition: subprocess can exit between poll() and _pending registration.
The poll() pre-flight check (line 185) and the self._pending[rpc_id] = entry insertion (line 196) are not atomic — they happen outside the same critical section. The problematic sequence:
poll() returns None (subprocess alive).
The subprocess exits; the reader thread calls _abort_pending, which clears _pending (currently empty).
This thread acquires _lock, inserts the new entry, writes to stdin, and releases _lock.
The orphaned entry now lives in _pending permanently and waits the full timeout — the exact stall that this fix aims to prevent.
To close the window, move the poll() check inside the _lock block, before registering the entry:
withself._lock:
# Re-check atomically under the lock that protects _pending.exit_code=self._proc.poll()
ifexit_codeisnotNone:
raiseBridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
rpc_id=self._next_id
...
💡 Suggested Change
Before:
exit_code = self._proc.poll()
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
with self._lock:
After:
with self._lock:
exit_code = self._proc.poll()
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
Shared mutable error dict assigned to all pending entries.
The same error dict object reference is written into every entry["error"]. If any downstream consumer ever mutates the dict (e.g., adds context keys, pops fields, or modifies nested values such as data), it would silently corrupt the error visible to all concurrent callers that hold the same reference.
Use a shallow copy per entry to isolate callers:
forentryinentries:
entry["error"] =dict(error) # shallow copy per entryentry["event"].set()
_close_called guard is set before cleanup completes, making a partial failure unrecoverable.
_close_called = True is assigned unconditionally before the with self._host_handlers_cv block and all subsequent cleanup steps (stdin close, process wait, SIGTERM/SIGKILL, _abort_pending). If any of those steps raises an unhandled exception, close() can never be retried because the early-return guard has already tripped, permanently leaking the client and its background reader thread.
The guard's intent is only to prevent concurrent/re-entrant calls, not to survive exceptions inside close(). Consider wrapping the body in try/except so cleanup errors are logged but don't leave the client in a permanently-uncloseable state:
defclose(self) ->None:
ifself._close_called:
returnself._close_called=Truetry:
withself._host_handlers_cv:
...
# ... rest of cleanup ...exceptException:
logger.exception("MemOS: unexpected error during bridge close")
Generated by cloud-assistant via Open Code Review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ai:generatedGenerated or modified by AI | 由 AI 生成或修改types:bugSomething isn't working | 功能异常
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixed issue #1722: after Hermes ran for 1-2 hours,
memory_searchandturn.endpermanently returned[timeout] … did not respond within 30.0sand never recovered. Root cause: three cooperating defects inapps/memos-local-plugin/adapters/hermes/memos_provider/on the Python side turned "bridge subprocess dies quietly" into "every memory tool blocks 30 s and then fails forever" — the reader thread exited silently on EOF (leaving pending waiters parked until each individual timeout expired), the keepalive only reconnected ontransport_closed(ignoringBridgeError("timeout", …)— the exact symptom), and the read-path tools had no retry mirroringsync_turn.Fix, without touching the JSON-RPC wire protocol or the Node bridge: (1)
MemosBridgeClient._read_loopnow runs in try/except/finally and always calls a new_abort_pending("bridge subprocess exited")that wakes every parked waiter withBridgeError("transport_closed", …)in <1 s; (2)MemosBridgeClient.requestshort-circuits withtransport_closedwhenPopen.poll()already reports a non-None exit code, so writes never go to a dead pipe; (3)close()was refactored to share teardown via a new_close_calledflag; (4)_start_bridge_keepalivenow delegates to_should_reconnect_after_keepalive_failure(err)— reconnects ontransport_closed,BridgeError("timeout", …), and any error observed while the subprocess is already dead; (5)handle_tool_callroutes every read-path tool (memos_search,memos_get,memos_timeline,memos_environment,memos_skill_list,memos_skill_get) through a new_bridge_request_with_retryhelper that reconnects + retries once ontransport_closed, mirroringsync_turn.Tests: 7 new cases in
apps/memos-local-plugin/tests/python/test_bridge_client.pycover reader-exit abort, fast-fail on deadpoll(), read-path recover branch, read-path still-broken branch, keepalive reconnect on health timeout, keepalive reconnect on dead subprocess, and the negative case (do NOT reconnect on transient generic error while the subprocess is alive — avoids reconnect storms). All 33 bridge tests + 16 hermes pipeline tests pass;ruff checkandruff format --checkare clean across the modified files.Branch
bugfix/autodev-1722pushed tooriginat commit7c9fc1f0. Archive artifacts (task.md + proposal/spec/design/verification-report) were also synced to memos-autodev-specs main. No wire-protocol / config / dependency changes; no impact outside the Hermes adapter.Related Issue (Required): Fixes #1722
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Automated tests are pending.
Checklist
@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.
Reviewer Checklist