You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix #1722: fix: memory_search returned error - #2027
Fixes#1722 — memory_search returned [timeout] memory.search did not respond within 30.0s after the hermes plugin's Node bridge went silent, followed by the same failure on turn.end. Three cooperating Python-side defects: (a) MemosBridgeClient._read_loop iterated stdout with no try/finally so parked JSON-RPC waiters stayed asleep past EOF for the full 30 s per-request timeout; (b) _start_bridge_keepalive reconnected only on transport_closed and dropped BridgeError("timeout", …) — the exact code raised for the reported symptom — at DEBUG; (c) read-path memory tools (memos_search, memos_get, memos_timeline, memos_environment, memos_skill_list, memos_skill_get) called _bridge.request once with no retry, so even the first user-visible call to a stale bridge surfaced a bare failure.
Fix keeps the JSON-RPC wire protocol untouched. Reader loop now aborts every pending waiter with transport_closed on any exit via the new _abort_pending helper (also reused by close()). request() fast-fails on Popen.poll() != None before ever writing to a dead pipe. Keepalive delegates to a new _should_reconnect_after_keepalive_failure helper that reconnects on transport_closed, BridgeError("timeout", …), and any error observed on an already-dead subprocess (while staying quiet for generic transient errors on a live subprocess, to prevent reconnect storms). Every read-path tool now goes through _bridge_request_with_retry, which mirrors the existing sync_turn write-path pattern — reconnect + retry once on transport_closed, surface any second failure verbatim.
Tests: added 8 new cases under apps/memos-local-plugin/tests/python/test_bridge_client.py covering reader-thread abort, fast-fail on dead subprocess, close() wake-pending, keepalive reconnect on timeout / dead subprocess / no-op on transient generic error, and handle_tool_call retry that both succeeds and surfaces a second transport failure. python3 -m unittest tests.python.test_bridge_client = 34 tests OK; tests.python.test_hermes_provider_pipeline = 16 tests OK; ruff check + ruff format --check clean.
Branch bugfix/autodev-1722 pushed to origin at c65d96e. opsp artifacts (task.md + proposal/spec/design/verification-report) archived to memos-autodev-specs main under 2026-07-02-1722-fix-memorysearch-returned-error/.
Race condition (TOCTOU): _closed check is outside _lock, so a new pending entry can be orphaned.
The sequence is:
Thread A reads self._closed → False, passes the guard at line 173.
Thread A calls self._proc.poll() → None, passes the fail-fast check.
_abort_pending (from reader-thread finally) runs on another thread: sets _closed = True, clears _pending, wakes all waiters.
Thread A acquires _lock, adds a new entry to _pending (line 190), and sends the payload.
Nobody will ever resolve that entry — it stalls until timeout fires.
The fix is to re-check _closed inside _lockafter registering the pending entry, and remove it + raise immediately if _closed is now True. Alternatively, hold a single lock that guards both _closed and _pending throughout the registration path.
💡 Suggested Change
Before:
if self._closed:
raise BridgeError("transport_closed", "bridge client is closed")
# Layer 2 (fix for #1722): fail-fast when the subprocess has
# already exited. Otherwise the pipe write may silently succeed
# into an OS buffer and the caller stalls for the full
# per-request timeout before failing.
exit_code = self._proc.poll() if self._proc is not None else None
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
with self._lock:
rpc_id = self._next_id
self._next_id += 1
waiter = threading.Event()
entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
self._pending[rpc_id] = entry
After:
with self._lock:
# Re-check _closed inside _lock to close the TOCTOU window
# between the guard above and the entry registration below.
if self._closed:
raise BridgeError("transport_closed", "bridge client is closed")
exit_code = self._proc.poll() if self._proc is not None else None
if exit_code is not None:
raise BridgeError(
"transport_closed",
f"bridge subprocess exited (code={exit_code})",
)
rpc_id = self._next_id
self._next_id += 1
waiter = threading.Event()
entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
self._pending[rpc_id] = entry
Race condition in _abort_pending: gap between releasing _host_handlers_cv and acquiring _lock can leave new _pending entries orphaned.
_closed is set to True inside _host_handlers_cv, but _pending is cleared inside a separate_lock. Between the two lock acquisitions, a racing thread can:
Read _closed == True → False (before _host_handlers_cv is released), register an entry in _pending while _abort_pending still holds _host_handlers_cv.
_abort_pending releases _host_handlers_cv, acquires _lock, clears _pending — but the new entry was added after the clear if insertion races past the _lock acquisition order.
More importantly, there is no guarantee the _closed = True write (visible to request()) and the _pending.clear() appear atomic to a racing request() call. Consider consolidating _closed under _lock (or ensuring _abort_pending re-checks _pending after holding _lock for the full critical section).
Unexpected reader-loop exceptions are silently swallowed at DEBUG level.
except Exception is deliberately broad, but logger.debug(...) means that in production — where DEBUG is typically off — all unexpected exceptions (e.g., AttributeError, OSError during line iteration, MemoryError) will be invisible. This makes post-mortem diagnosis very difficult. Consider logging at WARNING or higher for truly unexpected exceptions, reserving DEBUG only for the expected clean-EOF path.
💡 Suggested Change
Before:
except Exception:
logger.debug("bridge reader loop terminated with exception", exc_info=True)
After:
except Exception:
logger.warning("bridge reader loop terminated with exception", exc_info=True)
Inconsistency: timeout errors are not retried here, but they trigger reconnect in _should_reconnect_after_keepalive_failure.
_is_transport_closed only matches BridgeError(code='transport_closed', ...). A BridgeError('timeout', ...) — raised by MemosBridgeClient.request when the Node bridge stops answering (see bridge_client.py line 205) — will propagate uncaught from this helper without a retry or reconnect attempt, even though _should_reconnect_after_keepalive_failure explicitly treats timeout as a reconnect trigger.
This means a hung bridge causes memos_search / skill.list / etc. to return {"error": "[timeout] memory.search did not respond within 30.0s"} to the LLM on first occurrence instead of reconnecting and retrying, and the bridge is only cleaned up later by the keepalive thread — reproducing the original #1722 symptom for a different error code.
Suggested fix: extend the retry condition to include timeout:
Or better, extract the shared predicate used by both _bridge_request_with_retry and _should_reconnect_after_keepalive_failure into a single _is_retriable_transport_error helper to avoid this divergence.
TOCTOU race on self._bridge between _reconnect_bridge and the retry call.
_reconnect_bridge acquires _reconnect_lock to write self._bridge = new_bridge, then returns and releases the lock. This helper then reads self._bridge outside that lock for the retry. The keepalive thread runs every 5 s and can concurrently call _reconnect_bridge again — which would set self._bridge = None briefly (line 1851) or replace it with another new client — between the assert and the .request(...) call.
Capture the bridge reference immediately after reconnect to avoid this window:
self._reconnect_bridge(self._session_id, timeout=30.0)
br=self._bridgeifbrisNone:
raiseBridgeError("transport_closed", "reconnect did not produce a bridge")
returnbr.request(method, params, **kwargs)
Racy synchronization via time.sleep(0.05): This sleep is used as a heuristic to wait until the background thread has written the request and parked it in _pending. On a slow or heavily loaded CI machine the 50 ms may not be enough — the EOF signal (_done = True + _event.set()) could arrive before the waiter is actually parked, meaning _abort_pending fires before the entry exists in self._pending, the worker then parks and waits the full timeout=10.0 s, and the worker.join(timeout=2.0) returns with the worker still alive, causing the test to fail with a misleading message.
A robust alternative is to use an threading.Event that FakePopen's stdin write path sets when the first request arrives, replacing the sleep:
request_received=threading.Event()
# ... monkeypatch stdin.write to set event ...worker.start()
request_received.wait(timeout=2.0) # deterministicself._fake.stdout._done=True
...
This same pattern applies to test_close_wakes_pending_waiters.
Busy-polling private client._closed without synchronization: _closed is set inside _host_handlers_cv (a threading.Condition) in the production code, but the test reads it without holding that lock. While CPython's GIL makes bare boolean reads effectively atomic in practice, this is still technically a data race that could be flagged by threading sanitizers.
More importantly, this busy-wait loop exits via timeout (after 2 s) if the reader thread hasn't set _closed yet. In that case the subsequent assertRaises(BridgeError) may still pass — but for the wrong reason: the subprocess poll check at the top of request() also raises transport_closed, so the test can pass even if the reader thread never processed the EOF, masking a potential bug in _abort_pending.
Consider exposing a test-only synchronization hook (e.g., join the reader thread client._reader.join(timeout=2.0)) to deterministically wait for the reader loop to finish before asserting.
test_close_wakes_pending_waiters leaks background threads: client.close() is called as the operation under test, which is correct. However, MemosBridgeClient.__init__ always spawns a reader thread (self._reader) and a stderr reader thread (self._stderr_reader). After close(), the stderr reader thread may still be alive (it loops over self._proc.stderr which FakePopen.stderr is an io.StringIO — the loop ends immediately, so this is likely fine in practice). Still, it's worth adding an assertion or a join to ensure both daemon threads have exited to avoid any cross-test interference.
Racy synchronization via time.sleep(0.05): This sleep heuristic is unreliable on slow or heavily loaded CI machines. If the 50 ms elapses before the worker thread has written the request and parked it in _pending, the EOF signal (_done = True + _event.set()) will fire before the waiter exists, so _abort_pending finds an empty _pending dict, clears nothing, and the worker subsequently parks and waits the full 10 s — causing worker.join(timeout=2.0) to return with the worker still alive and a spurious test failure.
The same risk exists in test_close_wakes_pending_waiters.
A robust alternative is to synchronize on the arrival of the request at the fake stdin. The FakePopen stdin write path already appends to _stdin_lines; you could patch it to also set a threading.Event, e.g.:
Busy-polling client._closed without a lock and ambiguous pass/fail semantics: _closed is set under _host_handlers_cv (a threading.Condition) in the production code, but here it is read bare without that lock. While CPython's GIL makes this atomically safe in practice, it is technically a data race.
More critically, this busy-wait loop exits after 2 s even if the reader thread hasn't set _closed yet. In that case the subsequent request() call will still raise transport_closed — but via the poll() check at the top of request(), not via _abort_pending. This means the test can pass even if the reader-thread EOF→waiter notification path (_abort_pending) is entirely broken, defeating the purpose of the test.
Consider joining the reader thread directly after triggering EOF to ensure it has completed before asserting:
client._reader.join(timeout=2.0) # wait for reader to finishself.assertFalse(client._reader.is_alive(), "reader thread did not exit")
This makes the test deterministic and specifically exercises the _abort_pending path.
Racy synchronization via time.sleep(0.05): This sleep is an unreliable heuristic — on a slow or heavily loaded CI machine the 50 ms may elapse before the worker thread has written the request and parked it in _pending. In that case _abort_pending fires against an empty _pending dict (clearing nothing), the worker then parks and waits the full 10 s timeout, and worker.join(timeout=2.0) returns with the worker still alive — causing a spurious test failure.
The same issue exists in test_close_wakes_pending_waiters (line 435).
A deterministic alternative is to synchronize on the moment the request arrives at the fake stdin. The FakePopen stdin write path already accumulates lines in _stdin_lines; a small threading.Event patched into that path would let the test wait reliably:
Busy-polling client._closed without a lock, with ambiguous pass semantics: _closed is written under _host_handlers_cv (a threading.Condition) in the production code, but is read here without holding that lock — a technical data race (safe under CPython's GIL but fragile).
More importantly, this busy-wait exits after 2 s even if the reader thread hasn't set _closed yet. In that timeout case, the subsequent request() call will still raise transport_closed — but via the poll() check at the top of request(), not via _abort_pending. This means the test can pass even if the reader-thread EOF→waiter notification path (_abort_pending) is entirely broken, defeating the test's stated purpose.
A robust fix is to join the reader thread directly after triggering EOF:
self._fake.stdout._done=Trueself._fake.stdout._event.set()
client._reader.join(timeout=2.0) # wait for reader to finishself.assertFalse(client._reader.is_alive(), "reader thread did not exit")
# Now _closed is guaranteed to be set via _abort_pending
Generated by cloud-assistant via Open Code Review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ai:generatedGenerated or modified by AI | 由 AI 生成或修改types:bugSomething isn't working | 功能异常
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #1722 —
memory_searchreturned[timeout] memory.search did not respond within 30.0safter the hermes plugin's Node bridge went silent, followed by the same failure onturn.end. Three cooperating Python-side defects: (a)MemosBridgeClient._read_loopiterated stdout with no try/finally so parked JSON-RPC waiters stayed asleep past EOF for the full 30 s per-request timeout; (b)_start_bridge_keepalivereconnected only ontransport_closedand droppedBridgeError("timeout", …)— the exact code raised for the reported symptom — at DEBUG; (c) read-path memory tools (memos_search,memos_get,memos_timeline,memos_environment,memos_skill_list,memos_skill_get) called_bridge.requestonce with no retry, so even the first user-visible call to a stale bridge surfaced a bare failure.Fix keeps the JSON-RPC wire protocol untouched. Reader loop now aborts every pending waiter with
transport_closedon any exit via the new_abort_pendinghelper (also reused byclose()).request()fast-fails onPopen.poll() != Nonebefore ever writing to a dead pipe. Keepalive delegates to a new_should_reconnect_after_keepalive_failurehelper that reconnects ontransport_closed,BridgeError("timeout", …), and any error observed on an already-dead subprocess (while staying quiet for generic transient errors on a live subprocess, to prevent reconnect storms). Every read-path tool now goes through_bridge_request_with_retry, which mirrors the existingsync_turnwrite-path pattern — reconnect + retry once on transport_closed, surface any second failure verbatim.Tests: added 8 new cases under
apps/memos-local-plugin/tests/python/test_bridge_client.pycovering reader-thread abort, fast-fail on dead subprocess,close()wake-pending, keepalive reconnect ontimeout/ dead subprocess / no-op on transient generic error, andhandle_tool_callretry that both succeeds and surfaces a second transport failure.python3 -m unittest tests.python.test_bridge_client= 34 tests OK;tests.python.test_hermes_provider_pipeline= 16 tests OK;ruff check+ruff format --checkclean.Branch
bugfix/autodev-1722pushed to origin at c65d96e. opsp artifacts (task.md + proposal/spec/design/verification-report) archived tomemos-autodev-specsmain under2026-07-02-1722-fix-memorysearch-returned-error/.Related Issue (Required): Fixes #1722
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Automated tests are pending.
Checklist
@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.
Reviewer Checklist