Skip to content

Fix #1722: fix: memory_search returned error - #2029

Closed
Memtensor-AI wants to merge 1 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722
Closed

Memtensor-AI wants to merge 1 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fixes #1722 — the Hermes plugin's memory_search / turn.end calls stopped returning "[timeout] … did not respond within 30.0s" after the Node bridge subprocess dies quietly. Three cooperating Python-side defects in apps/memos-local-plugin were repaired in the JSON-RPC adapter without touching the wire protocol or the Node bridge.

Layered changes: (1) MemosBridgeClient._read_loop is now wrapped in try/finally and, on any exit (EOF or exception), calls the new _abort_pending helper that wakes every parked JSON-RPC waiter with BridgeError("transport_closed", …) in under one second, instead of letting each caller time out individually after 30 s. (2) MemosBridgeClient.request short-circuits with transport_closed when Popen.poll() reports the subprocess has already exited, preventing writes to a dead pipe. (3) MemTensorProvider._start_bridge_keepalive now reconnects on BridgeError("timeout", …) or any error observed while the subprocess is dead, via the new _should_reconnect_after_keepalive_failure gate — a live subprocess with a generic parse error still does NOT reconnect (avoids reconnect storms). (4) MemTensorProvider.handle_tool_call routes every read-path memory tool (memos_search / memos_get / memos_timeline / memos_environment / memos_skill_list / memos_skill_get) through the new _bridge_request_with_retry helper, mirroring the existing sync_turn write-path reconnect+retry.

Verification: 7 new unit tests added, covering reader-thread abort, fast-fail poll(), the three keepalive branches, and the read-path retry (both success and second-failure). Full test run: python3 -m unittest tests.python.test_bridge_client tests.python.test_hermes_provider_pipeline → 49 tests OK; ruff check and ruff format --check clean.

Related Issue (Required): Fixes #1722

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Automated tests are pending.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.

Reviewer Checklist

…bridge

Fixes MemTensor#1722 — after the Hermes agent runs for 1-2 hours every
memory_search call surfaces `[timeout] memory.search did not respond
within 30.0s` and the follow-up sync_turn fails with the same code.

Three cooperating Python-side defects were repaired in the JSON-RPC
adapter (no wire-protocol change; Node bridge untouched):

1. MemosBridgeClient._read_loop now wraps its dispatch in try/finally
   and, on any exit (EOF or exception), calls the new _abort_pending
   helper that wakes every parked waiter with a transport_closed error
   in <1s — instead of letting each caller time out individually after
   30s.
2. MemosBridgeClient.request short-circuits with transport_closed when
   Popen.poll() reports the subprocess has already exited, preventing
   writes into an OS-buffered dead pipe.
3. MemTensorProvider._start_bridge_keepalive now reconnects on
   BridgeError('timeout', …) or any error observed while the subprocess
   is dead, in addition to the existing transport_closed trigger, via
   the new _should_reconnect_after_keepalive_failure gate.
4. MemTensorProvider.handle_tool_call routes every read-path memory
   tool (memos_search / memos_get / memos_timeline / memos_environment
   / memos_skill_list / memos_skill_get) through
   _bridge_request_with_retry, mirroring the write-path pattern used
   by sync_turn.

Tests: added 7 new unit cases covering all four layers; 33 bridge
tests + 16 pipeline tests pass; ruff check + ruff format clean.
@Memtensor-AI Memtensor-AI added ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常 labels Jul 2, 2026
@Memtensor-AI Memtensor-AI mentioned this pull request Jul 2, 2026
3 of 5 tasks
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2029
Task: be988fdfd3182ed6
Base: dev-20260624-v2.0.22
Head: bugfix/autodev-1722

🔍 OpenCodeReview found 22 issue(s) in this PR.


1. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L180-L191)

TOCTOU race: entry can be orphaned in _pending after _abort_pending runs.

The subprocess exit is detected via poll() outside _lock, then the entry is inserted into _pending inside _lock. Between these two points, the _read_loop's finally block can call _abort_pending(), which:

  1. Sets self._closed = True (under _host_handlers_cv)
  2. Iterates and clears self._pending (under _lock)

If the _abort_pending call completes before line 191, the new entry is inserted into an already-cleared _pending with _closed = True. No one will ever wake it — the caller blocks for the full timeout and then raises a misleading [timeout] error, which is the exact bug (#1722) this PR aims to prevent.

The pre-existing _closed check at line 173 doesn't guard this window because it is read outside _lock, and _closed is set under a different lock (_host_handlers_cv), not _lock.

Suggested fix: After inserting the entry inside _lock, re-check self._closed while still holding _lock. If it is set, remove the entry and raise immediately:

with self._lock:
    if self._closed:  # re-check under the lock (abort_pending may have just run)
        raise BridgeError("transport_closed", "bridge client is closed")
    rpc_id = self._next_id
    ...
    self._pending[rpc_id] = entry
    try:
        self._proc.stdin.write(payload + "\n")
        self._proc.stdin.flush()
    except (BrokenPipeError, OSError) as err:
        self._pending.pop(rpc_id, None)
        raise BridgeError("transport_closed", str(err)) from err

This eliminates the window because _abort_pending acquires _lock before clearing _pending, so a _closed state set by _abort_pending will be visible immediately after the re-check under _lock.

💡 Suggested Change

Before:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

After:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            if self._closed:  # re-check under lock: _abort_pending may have run
                raise BridgeError("transport_closed", "bridge client is closed")
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

2. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L362-L367)

Reader-thread exceptions are silently swallowed at DEBUG level.

An unexpected exception on the reader thread is terminal — it ends the ability to receive any further responses from the bridge subprocess. Logging this at DEBUG level means operators will see no signal in INFO/WARNING logs when the bridge dies unexpectedly (e.g. OOM kill surfaced as OSError during stdout iteration, or a bug in the dispatch code).

Consider logging at WARNING or ERROR so it is visible in production log aggregation without needing debug verbosity.

💡 Suggested Change

Before:

        except Exception:
            # Any read-side exception (subprocess crash surfaced as OSError,
            # stdout iteration errors) is terminal for the reader thread.
            # We fall through to the finally block so pending waiters are
            # released instead of parking on their per-request timeouts.
            logger.debug("bridge: reader loop terminated by exception", exc_info=True)

After:

        except Exception:
            # Any read-side exception (subprocess crash surfaced as OSError,
            # stdout iteration errors) is terminal for the reader thread.
            # We fall through to the finally block so pending waiters are
            # released instead of parking on their per-request timeouts.
            logger.warning("bridge: reader loop terminated by exception", exc_info=True)

3. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L376-L388)

Docstring claims idempotency but the method always takes the slow path; also misses the TOCTOU window noted above.

The docstring says 'subsequent calls are no-ops', but there is no early-return guard when _closed is already True. On every call — even after _pending is already empty — both locks are acquired. While harmless in practice (iterating an empty dict is cheap), the idempotency claim is misleading.

More importantly, by setting _closed = True under _host_handlers_cv first, then acquiring _lock second, there is a brief window where _closed = True but _pending has not yet been cleared. A concurrent request() call that reads _closed = False (under no lock) and then inserts into _pending just after _pending.clear() will be left orphaned.

Consider adding an early return to make the idempotency guarantee explicit and correct:

def _abort_pending(self, reason: str) -> None:
    with self._host_handlers_cv:
        if self._closed:  # already aborted
            return
        self._closed = True
        self._host_handlers_cv.notify_all()
    with self._lock:
        for entry in list(self._pending.values()):
            ...
        self._pending.clear()
💡 Suggested Change

Before:

    def _abort_pending(self, reason: str) -> None:
        """Mark the client closed and wake every parked JSON-RPC waiter.

        Called from both ``_read_loop``'s finally block (subprocess
        died) and ``close`` (client shutting down). Idempotent —
        subsequent calls with the same client are no-ops because
        ``_pending`` is cleared and ``_closed`` is latched.
        """
        with self._host_handlers_cv:
            self._closed = True
            self._host_handlers_cv.notify_all()
        with self._lock:
            for entry in list(self._pending.values()):

After:

    def _abort_pending(self, reason: str) -> None:
        """Mark the client closed and wake every parked JSON-RPC waiter.

        Called from both ``_read_loop``'s finally block (subprocess
        died) and ``close`` (client shutting down). Idempotent —
        subsequent calls with the same client are no-ops because
        ``_pending`` is cleared and ``_closed`` is latched.
        """
        with self._host_handlers_cv:
            if self._closed:  # already aborted — nothing left to do
                return
            self._closed = True
            self._host_handlers_cv.notify_all()
        with self._lock:
            for entry in list(self._pending.values()):

4. apps/memos-local-plugin/tests/python/test_bridge_client.py (L319-L321)

Redundant local import: threading is already imported at module level (line 18). This local import threading as _th shadows the top-level import unnecessarily and violates the project's existing import style. Replace all _th.Thread references below with threading.Thread.

💡 Suggested Change

Before:

        import threading as _th

        t = _th.Thread(target=_issue)

After:

        t = threading.Thread(target=_issue)

5. apps/memos-local-plugin/tests/python/test_bridge_client.py (L341-L352)

Missing test case for the TOCTOU race between poll() and _pending registration in request(). The production bridge_client.py checks poll() outside self._lock (line 180), then registers the waiter entry inside self._lock (line 191). Between those two points, _abort_pending can run (triggered by the reader-thread EOF), set _closed = True, clear _pending, and exit — after which request() still inserts a new entry that will never be woken, causing the caller to block until timeout expires.

The existing test_reader_exit_marks_pending_as_transport_closed only tests the case where the entry is already in _pending before EOF arrives. A complementary test should simulate the race where EOF arrives while request() is between poll() and the with self._lock block (i.e., the entry is registered after _abort_pending has already cleared pending). Without this, the race goes undetected and can silently produce a full-timeout hang in production.


6. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1802-L1804)

[High] AssertionError during shutdown if _reconnect_bridge returns early.

_reconnect_bridge is a no-op when _bridge_keepalive_stop is set (the shutdown guard on lines 1808–1810). In that case it returns without touching self._bridge, so self._bridge is still None at this point and the assert raises an AssertionError — an opaque crash instead of a clean failure.

This is the same guard pattern used in the original sync_turn reconnect path, but there it is guarded by _ensure_bridge which handles None gracefully. Here there is no such protection.

Suggested fix: replace the assertion with a descriptive raise:

self._reconnect_bridge(self._session_id, timeout=30.0)
if self._bridge is None:
    raise BridgeError("transport_closed", "bridge unavailable after reconnect attempt")
return self._bridge.request(method, params, **kwargs) or {}

7. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1792-L1796)

[High] Retry gate uses _is_transport_closed instead of _should_reconnect_after_keepalive_failure, missing the core scenario of issue #1722.

The reported symptom (issue #1722) is a wedged Node event loop where every bridge call hits the 10 s BridgeError('timeout', ...). _should_reconnect_after_keepalive_failure was explicitly extended to handle this case (line 1756), but _bridge_request_with_retry still guards only on _is_transport_closed, which does not match BridgeError.code == "timeout".

As a result:

  • The keepalive thread now correctly reconnects on timeout.
  • But the read-path tool calls (memos_search, memos_get, etc.) that triggered the user-visible error still re-raise BridgeError('timeout') without reconnecting.

Note: with the new _abort_pending mechanism in bridge_client.py, a dead subprocess now surfaces as transport_closed (not timeout), so the subprocess-death case is covered. However, the wedged-event-loop scenario (subprocess alive, but event loop frozen) will still raise timeout on the read path without any retry.

Suggested fix: replace _is_transport_closed with _should_reconnect_after_keepalive_failure in this guard, and update the log message accordingly:

except Exception as err:
    if not self._should_reconnect_after_keepalive_failure(err):
        raise
    logger.warning(
        "MemOS: bridge %s during %s; reconnecting and retrying once — %s",
        self._describe_keepalive_error(err),
        method,
        err,
    )

8. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1361)

[Medium] or {} in _bridge_request_with_retry silently coerces a list/null response into {}, breaking skill.list callers.

_bridge_request_with_retry returns self._bridge.request(...) or {}. If the bridge returns a list (e.g., skill.list returns [{...}, ...]) or None, the or {} replaces it with an empty dict. Previously json.dumps(self._bridge.request(...)) would correctly encode the list or null.

This is a silent behavioral regression: the caller receives {} where it expects a list, and the LLM model sees an empty object instead of the skill list.

Suggested fix: let _bridge_request_with_retry return Any and use None-guarding only:

def _bridge_request_with_retry(
    self,
    method: str,
    params: dict[str, Any] | None = None,
    *,
    timeout: float | None = None,
) -> Any:
    ...
    try:
        return self._bridge.request(method, params, **kwargs)
    ...
    self._reconnect_bridge(self._session_id, timeout=30.0)
    ...
    return self._bridge.request(method, params, **kwargs)

Alternatively, change the return expression to ... or None and let callers handle None explicitly.


9. apps/memos-local-plugin/tests/python/test_bridge_client.py (L319-L321)

Redundant local import: threading is already imported at module level (line 18). Using import threading as _th here and then _th.Thread(...) below is inconsistent with the rest of the test file and violates PEP 8. Replace with the module-level threading.Thread directly.

💡 Suggested Change

Before:

        import threading as _th

        t = _th.Thread(target=_issue)

After:

        t = threading.Thread(target=_issue)

10. apps/memos-local-plugin/tests/python/test_bridge_client.py (L341-L352)

Missing coverage for the TOCTOU race between poll() and _pending registration.

In production bridge_client.py, request() calls self._proc.poll() outside self._lock (line 180), then registers the waiter entry inside self._lock (line 191). Between those two steps, _abort_pending() can run (triggered by the reader thread reaching EOF), set _closed = True, clear _pending, and exit — after which request() inserts a new entry that will never be woken, causing the caller to block until timeout expires with a misleading timeout error.

The existing test_reader_exit_marks_pending_as_transport_closed only covers the case where the entry is already in _pending before EOF. A test that simulates EOF arriving after poll() but before the lock acquisition would catch this race, which is directly relevant to issue #1722's root cause (unexpected long timeouts).


11. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L180-L191)

TOCTOU race: entry can be orphaned in _pending after _abort_pending runs.

The subprocess exit is detected via poll() outside _lock, then the waiter entry is inserted into _pending inside _lock. Between these two points, the reader thread's finally block can call _abort_pending(), which:

  1. Sets self._closed = True (under _host_handlers_cv)
  2. Clears self._pending (under _lock)

If _abort_pending completes before line 191, the new entry is inserted into an already-cleared _pending with no one to ever wake it — the caller parks until the full timeout fires, raising a misleading [timeout] error. This is the exact bug (#1722) this PR aims to prevent.

The _closed check at line 173 doesn't guard this window because _closed is read outside any lock, while _abort_pending sets it under a different lock (_host_handlers_cv) before acquiring _lock to clear _pending.

Suggested fix: Re-check self._closed inside _lock after inserting the entry, and raise immediately if it is set:

with self._lock:
    if self._closed:  # re-check under lock: _abort_pending may have just run
        raise BridgeError("transport_closed", "bridge client is closed")
    rpc_id = self._next_id
    ...
    self._pending[rpc_id] = entry
    try:
        self._proc.stdin.write(payload + "\n")
        ...

This is safe because _abort_pending always acquires _lock before touching _pending, so if _closed is True inside _lock, the pending dict is already cleared and no waker is coming.


12. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L362-L367)

Terminal reader-thread exceptions are silently swallowed at DEBUG level.

This except Exception catches failures that are fatal to the entire bridge communication channel (e.g., OSError from a crashed subprocess, unexpected bugs in the dispatch code). Logging at DEBUG means operators running with INFO-level logging will see zero signal when the reader thread unexpectedly dies. Other critical bridge failures in this file use logger.error or logger.warning.

Consider changing to logger.warning or logger.error to make unexpected reader-thread crashes visible in production logs.


13. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L384-L388)

_abort_pending docstring claims idempotency but no early-return guard exists.

The docstring says 'subsequent calls are no-ops because _pending is cleared and _closed is latched', but the implementation always acquires both locks and iterates _pending unconditionally on every call. This also widens the TOCTOU window described above: between setting _closed = True (under _host_handlers_cv) and clearing _pending (under _lock), a concurrent request() caller that already passed the outer _closed check can insert a new entry that gets orphaned.

Adding an early return inside the _host_handlers_cv block satisfies the docstring's guarantee atomically — the first call sets the latch and clears pending; all subsequent calls return immediately:

with self._host_handlers_cv:
    if self._closed:  # already aborted — nothing left to do
        return
    self._closed = True
    self._host_handlers_cv.notify_all()
with self._lock:
    ...
    self._pending.clear()

14. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L180-L191)

TOCTOU race: the waiter entry can be orphaned in _pending after _abort_pending runs.

The poll() check is done outside _lock, but the entry is inserted into _pending inside _lock. Between these two points, the reader thread's finally block can call _abort_pending(), which (1) sets _closed = True under _host_handlers_cv and (2) calls _pending.clear() under _lock. If _abort_pending completes before line 191, the new entry is inserted into an already-cleared _pending with no one to ever wake it — the caller then blocks until timeout fires and gets a misleading [timeout] error, the exact bug (#1722) this PR aims to fix.

The _closed check at line 173 does not guard this because it reads _closed outside any lock, while _abort_pending sets it under a different lock (_host_handlers_cv).

Suggested fix: re-check self._closed inside _lock at the top of the with self._lock: block, and raise immediately if set:

with self._lock:
    if self._closed:  # re-check under lock: _abort_pending may have just run
        raise BridgeError("transport_closed", "bridge client is closed")
    rpc_id = self._next_id
    ...
    self._pending[rpc_id] = entry

This is safe because _abort_pending also acquires _lock before touching _pending, so a _closed state observed inside _lock guarantees the pending dict is already cleared.


15. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L362-L367)

Terminal reader-thread exceptions are logged only at DEBUG level.

This except Exception block catches fatal read-side errors (e.g., OSError from a subprocess crash) that permanently end bridge communication. Logging at DEBUG means that with standard INFO-level production logging, operators see no signal at all when the reader thread unexpectedly dies. Other critical bridge failures in this file use logger.error or logger.warning.

Suggested fix: change to logger.warning(...) or logger.error(...).


16. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L384-L388)

_abort_pending is documented as idempotent but has no early-return guard.

The docstring states 'subsequent calls are no-ops because _pending is cleared and _closed is latched', but the implementation always acquires both locks unconditionally on every call. More importantly, this two-step lock acquisition widens the TOCTOU window noted in request(): between setting _closed = True under _host_handlers_cv and clearing _pending under _lock, a concurrent request() caller that already passed the outer _closed check can insert an entry into _pending that will never be woken.

Adding an early-return inside the _host_handlers_cv block satisfies the docstring's guarantee and reduces the race window:

with self._host_handlers_cv:
    if self._closed:  # already aborted — nothing left to do
        return
    self._closed = True
    self._host_handlers_cv.notify_all()
with self._lock:
    ...
    self._pending.clear()

17. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L180-L191)

TOCTOU race: the waiter entry can be orphaned in _pending after _abort_pending runs.

The poll() fast-fail check is done outside _lock, but the entry is inserted into _pending inside _lock. Between these two points, the reader thread's finally block can complete _abort_pending(), which (1) sets _closed = True under _host_handlers_cv and (2) calls _pending.clear() under _lock. If _abort_pending completes before line 191, the new entry is inserted into an already-cleared _pending with no one to ever wake it. The caller then blocks until timeout fires and gets a misleading [timeout] error — the exact bug (#1722) this PR aims to fix.

The _closed check at line 173 does not guard this because it reads _closed outside any lock, while _abort_pending sets it under a different lock (_host_handlers_cv).

Suggested fix: re-check self._closed inside _lock at the top of the with self._lock: block and raise immediately if set.

💡 Suggested Change

Before:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

After:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            if self._closed:  # re-check under lock: _abort_pending may have just run
                raise BridgeError("transport_closed", "bridge client is closed")
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

18. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L362-L367)

Terminal reader-thread exceptions are logged only at DEBUG level.

This except Exception block catches failures that are fatal to the entire bridge communication channel (e.g., OSError from a crashed subprocess). Logging at DEBUG means operators running with standard INFO-level production logging will see no signal when the reader thread unexpectedly dies. Other critical bridge failures in this file use logger.error or logger.warning.

Suggested fix: change to logger.warning(...) or logger.error(...).

💡 Suggested Change

Before:

        except Exception:
            # Any read-side exception (subprocess crash surfaced as OSError,
            # stdout iteration errors) is terminal for the reader thread.
            # We fall through to the finally block so pending waiters are
            # released instead of parking on their per-request timeouts.
            logger.debug("bridge: reader loop terminated by exception", exc_info=True)

After:

        except Exception:
            # Any read-side exception (subprocess crash surfaced as OSError,
            # stdout iteration errors) is terminal for the reader thread.
            # We fall through to the finally block so pending waiters are
            # released instead of parking on their per-request timeouts.
            logger.warning("bridge: reader loop terminated by exception", exc_info=True)

19. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L384-L388)

_abort_pending is documented as idempotent but has no early-return guard, and the two-step lock acquisition widens the TOCTOU window.

The docstring states 'subsequent calls are no-ops because _pending is cleared and _closed is latched', but the implementation always acquires both locks unconditionally. More importantly, between setting _closed = True under _host_handlers_cv and clearing _pending under _lock, a concurrent request() caller that already passed the outer _closed check (line 173) can insert an entry into _pending that will never be woken.

Suggested fix: add an early return inside the _host_handlers_cv block before setting _closed:

with self._host_handlers_cv:
    if self._closed:  # already aborted -- nothing left to do
        return
    self._closed = True
    self._host_handlers_cv.notify_all()
with self._lock:
    ...
    self._pending.clear()

20. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L180-L191)

TOCTOU race: the poll() check outside _lock and the _pending insertion inside _lock are not atomic. Between these two points the reader thread's finally block can complete _abort_pending(), which sets _closed=True (under _host_handlers_cv) and calls _pending.clear() (under _lock). The new entry is then inserted into an already-cleared _pending with no one to ever wake it, so the caller blocks until the full timeout fires and receives a misleading [timeout] error — the exact bug (#1722) this PR aims to fix.

The _closed check at line 173 does not guard this window because it reads _closed outside any lock, while _abort_pending sets it under a different lock (_host_handlers_cv).

Suggested fix: re-check self._closed inside _lock at the top of the with self._lock: block and raise immediately if set.

💡 Suggested Change

Before:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

After:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:
            if self._closed:  # re-check under lock: _abort_pending may have just run
                raise BridgeError("transport_closed", "bridge client is closed")
            rpc_id = self._next_id
            self._next_id += 1
            waiter = threading.Event()
            entry: dict[str, Any] = {"event": waiter, "result": None, "error": None}
            self._pending[rpc_id] = entry

21. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L367)

Fatal reader-thread exceptions are logged only at DEBUG level. This catches failures that permanently end bridge communication (e.g. OSError from a crashed subprocess). With standard INFO-level production logging, operators see no signal when the reader thread unexpectedly dies. Other critical bridge failures in this file use logger.error or logger.warning. Consider changing to logger.warning(...) or logger.error(...).


22. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L384-L388)

_abort_pending is documented as idempotent but has no early-return guard. The docstring states 'subsequent calls are no-ops' but the method always acquires both locks unconditionally. More importantly, the two-step lock acquisition widens the TOCTOU window in request(): between setting _closed=True (under _host_handlers_cv) and clearing _pending (under _lock), a concurrent request() caller that already passed the outer _closed check can insert an entry into _pending that will never be woken.

Suggested fix: add an early return inside the _host_handlers_cv block before setting _closed:
with self._host_handlers_cv:
if self._closed: # already aborted -- nothing left to do
return
self._closed = True
self._host_handlers_cv.notify_all()

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants