Skip to content

Fix #1722: fix: memory_search returned error - #2027

Closed
Memtensor-AI wants to merge 0 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722
Closed

Memtensor-AI wants to merge 0 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fixes #1722 — memory_search returned [timeout] memory.search did not respond within 30.0s after the hermes plugin's Node bridge went silent, followed by the same failure on turn.end. Three cooperating Python-side defects: (a) MemosBridgeClient._read_loop iterated stdout with no try/finally so parked JSON-RPC waiters stayed asleep past EOF for the full 30 s per-request timeout; (b) _start_bridge_keepalive reconnected only on transport_closed and dropped BridgeError("timeout", …) — the exact code raised for the reported symptom — at DEBUG; (c) read-path memory tools (memos_search, memos_get, memos_timeline, memos_environment, memos_skill_list, memos_skill_get) called _bridge.request once with no retry, so even the first user-visible call to a stale bridge surfaced a bare failure.

Fix keeps the JSON-RPC wire protocol untouched. Reader loop now aborts every pending waiter with transport_closed on any exit via the new _abort_pending helper (also reused by close()). request() fast-fails on Popen.poll() != None before ever writing to a dead pipe. Keepalive delegates to a new _should_reconnect_after_keepalive_failure helper that reconnects on transport_closed, BridgeError("timeout", …), and any error observed on an already-dead subprocess (while staying quiet for generic transient errors on a live subprocess, to prevent reconnect storms). Every read-path tool now goes through _bridge_request_with_retry, which mirrors the existing sync_turn write-path pattern — reconnect + retry once on transport_closed, surface any second failure verbatim.

Tests: added 8 new cases under apps/memos-local-plugin/tests/python/test_bridge_client.py covering reader-thread abort, fast-fail on dead subprocess, close() wake-pending, keepalive reconnect on timeout / dead subprocess / no-op on transient generic error, and handle_tool_call retry that both succeeds and surfaces a second transport failure. python3 -m unittest tests.python.test_bridge_client = 34 tests OK; tests.python.test_hermes_provider_pipeline = 16 tests OK; ruff check + ruff format --check clean.

Branch bugfix/autodev-1722 pushed to origin at c65d96e. opsp artifacts (task.md + proposal/spec/design/verification-report) archived to memos-autodev-specs main under 2026-07-02-1722-fix-memorysearch-returned-error/.

Related Issue (Required): Fixes #1722

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Automated tests are pending.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.

Reviewer Checklist

@Memtensor-AI Memtensor-AI added ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常 labels Jul 2, 2026
@Memtensor-AI Memtensor-AI mentioned this pull request Jul 2, 2026
3 of 5 tasks
@Memtensor-AI
Memtensor-AI force-pushed the bugfix/autodev-1722 branch from c65d96e to a4f1b5b Compare July 2, 2026 03:40
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2027
Task: d24d1e93a3e30f26
Base: dev-20260624-v2.0.22
Head: bugfix/autodev-1722

🔍 OpenCodeReview found 12 issue(s) in this PR.


1. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L173-L190)

Race condition (TOCTOU): _closed check is outside _lock, so a new pending entry can be orphaned.

The sequence is:

  1. Thread A reads self._closed → False, passes the guard at line 173.
  2. Thread A calls self._proc.poll() → None, passes the fail-fast check.
  3. _abort_pending (from reader-thread finally) runs on another thread: sets _closed = True, clears _pending, wakes all waiters.
  4. Thread A acquires _lock, adds a new entry to _pending (line 190), and sends the payload.
  5. Nobody will ever resolve that entry — it stalls until timeout fires.

The fix is to re-check _closed inside _lock after registering the pending entry, and remove it + raise immediately if _closed is now True. Alternatively, hold a single lock that guards both _closed and _pending throughout the registration path.

💡 Suggested Change

Before:

        if self._closed:
            raise BridgeError("transport_closed", "bridge client is closed")
        # Layer 2 (fix for #1722): fail-fast when the subprocess has
        # already exited. Otherwise the pipe write may silently succeed
        # into an OS buffer and the caller stalls for the full
        # per-request timeout before failing.
        exit_code = self._proc.poll() if self._proc is not None else None
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

After:

        with self._lock:
            # Re-check _closed inside _lock to close the TOCTOU window
            # between the guard above and the entry registration below.
            if self._closed:
                raise BridgeError("transport_closed", "bridge client is closed")
            exit_code = self._proc.poll() if self._proc is not None else None
            if exit_code is not None:
                raise BridgeError(
                    "transport_closed",
                    f"bridge subprocess exited (code={exit_code})",
                )
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

2. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L294-L313)

Race condition in _abort_pending: gap between releasing _host_handlers_cv and acquiring _lock can leave new _pending entries orphaned.

_closed is set to True inside _host_handlers_cv, but _pending is cleared inside a separate _lock. Between the two lock acquisitions, a racing thread can:

  1. Read _closed == True → False (before _host_handlers_cv is released), register an entry in _pending while _abort_pending still holds _host_handlers_cv.
  2. _abort_pending releases _host_handlers_cv, acquires _lock, clears _pending — but the new entry was added after the clear if insertion races past the _lock acquisition order.

More importantly, there is no guarantee the _closed = True write (visible to request()) and the _pending.clear() appear atomic to a racing request() call. Consider consolidating _closed under _lock (or ensuring _abort_pending re-checks _pending after holding _lock for the full critical section).


3. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L384-L385)

Unexpected reader-loop exceptions are silently swallowed at DEBUG level.

except Exception is deliberately broad, but logger.debug(...) means that in production — where DEBUG is typically off — all unexpected exceptions (e.g., AttributeError, OSError during line iteration, MemoryError) will be invisible. This makes post-mortem diagnosis very difficult. Consider logging at WARNING or higher for truly unexpected exceptions, reserving DEBUG only for the expected clean-EOF path.

💡 Suggested Change

Before:

        except Exception:
            logger.debug("bridge reader loop terminated with exception", exc_info=True)

After:

        except Exception:
            logger.warning("bridge reader loop terminated with exception", exc_info=True)

4. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1292-L1296)

Inconsistency: timeout errors are not retried here, but they trigger reconnect in _should_reconnect_after_keepalive_failure.

_is_transport_closed only matches BridgeError(code='transport_closed', ...). A BridgeError('timeout', ...) — raised by MemosBridgeClient.request when the Node bridge stops answering (see bridge_client.py line 205) — will propagate uncaught from this helper without a retry or reconnect attempt, even though _should_reconnect_after_keepalive_failure explicitly treats timeout as a reconnect trigger.

This means a hung bridge causes memos_search / skill.list / etc. to return {"error": "[timeout] memory.search did not respond within 30.0s"} to the LLM on first occurrence instead of reconnecting and retrying, and the bridge is only cleaned up later by the keepalive thread — reproducing the original #1722 symptom for a different error code.

Suggested fix: extend the retry condition to include timeout:

except BridgeError as err:
    if not self._is_transport_closed(err) and not (isinstance(err, BridgeError) and err.code == "timeout"):
        raise

Or better, extract the shared predicate used by both _bridge_request_with_retry and _should_reconnect_after_keepalive_failure into a single _is_retriable_transport_error helper to avoid this divergence.


5. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1302-L1304)

TOCTOU race on self._bridge between _reconnect_bridge and the retry call.

_reconnect_bridge acquires _reconnect_lock to write self._bridge = new_bridge, then returns and releases the lock. This helper then reads self._bridge outside that lock for the retry. The keepalive thread runs every 5 s and can concurrently call _reconnect_bridge again — which would set self._bridge = None briefly (line 1851) or replace it with another new client — between the assert and the .request(...) call.

Capture the bridge reference immediately after reconnect to avoid this window:

self._reconnect_bridge(self._session_id, timeout=30.0)
br = self._bridge
if br is None:
    raise BridgeError("transport_closed", "reconnect did not produce a bridge")
return br.request(method, params, **kwargs)

6. apps/memos-local-plugin/tests/python/test_bridge_client.py (L379-L384)

Racy synchronization via time.sleep(0.05): This sleep is used as a heuristic to wait until the background thread has written the request and parked it in _pending. On a slow or heavily loaded CI machine the 50 ms may not be enough — the EOF signal (_done = True + _event.set()) could arrive before the waiter is actually parked, meaning _abort_pending fires before the entry exists in self._pending, the worker then parks and waits the full timeout=10.0 s, and the worker.join(timeout=2.0) returns with the worker still alive, causing the test to fail with a misleading message.

A robust alternative is to use an threading.Event that FakePopen's stdin write path sets when the first request arrives, replacing the sleep:

request_received = threading.Event()
# ... monkeypatch stdin.write to set event ...
worker.start()
request_received.wait(timeout=2.0)  # deterministic
self._fake.stdout._done = True
...

This same pattern applies to test_close_wakes_pending_waiters.


7. apps/memos-local-plugin/tests/python/test_bridge_client.py (L409-L411)

Busy-polling private client._closed without synchronization: _closed is set inside _host_handlers_cv (a threading.Condition) in the production code, but the test reads it without holding that lock. While CPython's GIL makes bare boolean reads effectively atomic in practice, this is still technically a data race that could be flagged by threading sanitizers.

More importantly, this busy-wait loop exits via timeout (after 2 s) if the reader thread hasn't set _closed yet. In that case the subsequent assertRaises(BridgeError) may still pass — but for the wrong reason: the subprocess poll check at the top of request() also raises transport_closed, so the test can pass even if the reader thread never processed the EOF, masking a potential bug in _abort_pending.

Consider exposing a test-only synchronization hook (e.g., join the reader thread client._reader.join(timeout=2.0)) to deterministically wait for the reader loop to finish before asserting.


8. apps/memos-local-plugin/tests/python/test_bridge_client.py (L437-L441)

test_close_wakes_pending_waiters leaks background threads: client.close() is called as the operation under test, which is correct. However, MemosBridgeClient.__init__ always spawns a reader thread (self._reader) and a stderr reader thread (self._stderr_reader). After close(), the stderr reader thread may still be alive (it loops over self._proc.stderr which FakePopen.stderr is an io.StringIO — the loop ends immediately, so this is likely fine in practice). Still, it's worth adding an assertion or a join to ensure both daemon threads have exited to avoid any cross-test interference.


9. apps/memos-local-plugin/tests/python/test_bridge_client.py (L379-L384)

Racy synchronization via time.sleep(0.05): This sleep heuristic is unreliable on slow or heavily loaded CI machines. If the 50 ms elapses before the worker thread has written the request and parked it in _pending, the EOF signal (_done = True + _event.set()) will fire before the waiter exists, so _abort_pending finds an empty _pending dict, clears nothing, and the worker subsequently parks and waits the full 10 s — causing worker.join(timeout=2.0) to return with the worker still alive and a spurious test failure.

The same risk exists in test_close_wakes_pending_waiters.

A robust alternative is to synchronize on the arrival of the request at the fake stdin. The FakePopen stdin write path already appends to _stdin_lines; you could patch it to also set a threading.Event, e.g.:

request_received = threading.Event()
original_write = self._fake.stdin.write
self._fake.stdin.write = lambda line: (original_write(line), request_received.set())
worker.start()
request_received.wait(timeout=2.0)  # deterministic
self._fake.stdout._done = True
...

10. apps/memos-local-plugin/tests/python/test_bridge_client.py (L409-L411)

Busy-polling client._closed without a lock and ambiguous pass/fail semantics: _closed is set under _host_handlers_cv (a threading.Condition) in the production code, but here it is read bare without that lock. While CPython's GIL makes this atomically safe in practice, it is technically a data race.

More critically, this busy-wait loop exits after 2 s even if the reader thread hasn't set _closed yet. In that case the subsequent request() call will still raise transport_closed — but via the poll() check at the top of request(), not via _abort_pending. This means the test can pass even if the reader-thread EOF→waiter notification path (_abort_pending) is entirely broken, defeating the purpose of the test.

Consider joining the reader thread directly after triggering EOF to ensure it has completed before asserting:

client._reader.join(timeout=2.0)  # wait for reader to finish
self.assertFalse(client._reader.is_alive(), "reader thread did not exit")

This makes the test deterministic and specifically exercises the _abort_pending path.


11. apps/memos-local-plugin/tests/python/test_bridge_client.py (L379-L384)

Racy synchronization via time.sleep(0.05): This sleep is an unreliable heuristic — on a slow or heavily loaded CI machine the 50 ms may elapse before the worker thread has written the request and parked it in _pending. In that case _abort_pending fires against an empty _pending dict (clearing nothing), the worker then parks and waits the full 10 s timeout, and worker.join(timeout=2.0) returns with the worker still alive — causing a spurious test failure.

The same issue exists in test_close_wakes_pending_waiters (line 435).

A deterministic alternative is to synchronize on the moment the request arrives at the fake stdin. The FakePopen stdin write path already accumulates lines in _stdin_lines; a small threading.Event patched into that path would let the test wait reliably:

request_received = threading.Event()
original_write = self._fake.stdin.write
self._fake.stdin.write = lambda line: (original_write(line), request_received.set())
worker.start()
request_received.wait(timeout=2.0)  # deterministic
self._fake.stdout._done = True
...

12. apps/memos-local-plugin/tests/python/test_bridge_client.py (L409-L411)

Busy-polling client._closed without a lock, with ambiguous pass semantics: _closed is written under _host_handlers_cv (a threading.Condition) in the production code, but is read here without holding that lock — a technical data race (safe under CPython's GIL but fragile).

More importantly, this busy-wait exits after 2 s even if the reader thread hasn't set _closed yet. In that timeout case, the subsequent request() call will still raise transport_closed — but via the poll() check at the top of request(), not via _abort_pending. This means the test can pass even if the reader-thread EOF→waiter notification path (_abort_pending) is entirely broken, defeating the test's stated purpose.

A robust fix is to join the reader thread directly after triggering EOF:

self._fake.stdout._done = True
self._fake.stdout._event.set()
client._reader.join(timeout=2.0)  # wait for reader to finish
self.assertFalse(client._reader.is_alive(), "reader thread did not exit")
# Now _closed is guaranteed to be set via _abort_pending

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants