Skip to content

Fix #1722: fix: memory_search returned error - #2017

Closed
Memtensor-AI wants to merge 0 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722
Closed

Memtensor-AI wants to merge 0 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fixed issue #1722: after Hermes ran for 1-2 hours, memory_search and turn.end permanently returned [timeout] … did not respond within 30.0s and never recovered. Root cause: three cooperating defects in apps/memos-local-plugin/adapters/hermes/memos_provider/ on the Python side turned "bridge subprocess dies quietly" into "every memory tool blocks 30 s and then fails forever" — the reader thread exited silently on EOF (leaving pending waiters parked until each individual timeout expired), the keepalive only reconnected on transport_closed (ignoring BridgeError("timeout", …) — the exact symptom), and the read-path tools had no retry mirroring sync_turn.

Fix, without touching the JSON-RPC wire protocol or the Node bridge: (1) MemosBridgeClient._read_loop now runs in try/except/finally and always calls a new _abort_pending("bridge subprocess exited") that wakes every parked waiter with BridgeError("transport_closed", …) in <1 s; (2) MemosBridgeClient.request short-circuits with transport_closed when Popen.poll() already reports a non-None exit code, so writes never go to a dead pipe; (3) close() was refactored to share teardown via a new _close_called flag; (4) _start_bridge_keepalive now delegates to _should_reconnect_after_keepalive_failure(err) — reconnects on transport_closed, BridgeError("timeout", …), and any error observed while the subprocess is already dead; (5) handle_tool_call routes every read-path tool (memos_search, memos_get, memos_timeline, memos_environment, memos_skill_list, memos_skill_get) through a new _bridge_request_with_retry helper that reconnects + retries once on transport_closed, mirroring sync_turn.

Tests: 7 new cases in apps/memos-local-plugin/tests/python/test_bridge_client.py cover reader-exit abort, fast-fail on dead poll(), read-path recover branch, read-path still-broken branch, keepalive reconnect on health timeout, keepalive reconnect on dead subprocess, and the negative case (do NOT reconnect on transient generic error while the subprocess is alive — avoids reconnect storms). All 33 bridge tests + 16 hermes pipeline tests pass; ruff check and ruff format --check are clean across the modified files.

Branch bugfix/autodev-1722 pushed to origin at commit 7c9fc1f0. Archive artifacts (task.md + proposal/spec/design/verification-report) were also synced to memos-autodev-specs main. No wire-protocol / config / dependency changes; no impact outside the Hermes adapter.

Related Issue (Required): Fixes #1722

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Automated tests are pending.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.

Reviewer Checklist

@Memtensor-AI Memtensor-AI added ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常 labels Jul 1, 2026
@Memtensor-AI Memtensor-AI mentioned this pull request Jul 1, 2026
3 of 5 tasks
@Memtensor-AI
Memtensor-AI force-pushed the bugfix/autodev-1722 branch from 7c9fc1f to a4f1b5b Compare July 1, 2026 10:10
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2017
Task: ddc1cdc6358ca509
Base: dev-20260624-v2.0.22
Head: bugfix/autodev-1722

🔍 OpenCodeReview found 15 issue(s) in this PR.


1. apps/memos-local-plugin/tests/python/test_bridge_client.py (L383-L387)

Race condition (fragile test timing). ready.set() fires at the very top of _do_request, before client.request(...) is called — so when ready.wait() returns, the worker thread has only just started executing request(). The 50 ms sleep is the sole guard between here and the worker entering waiter.wait(timeout=5.0). On a heavily loaded CI runner the 50 ms budget can be consumed before request() reaches waiter.wait(), meaning the test will simulate subprocess death while the worker is still inside the poll() fast-fail check — exercising the wrong code path (fast-fail rather than _abort_pending) yet still "passing".

A deterministic fix is to move ready.set() to just after the waiter is parked. The cleanest approach without changing the production code is to use a second threading.Event that is set from a callback installed via _pending introspection, or — more simply — to move the ready.set() call after a deliberate blocking stub:

def _do_request() -> None:
    # park on the waiter first, then signal the main thread
    def _hook(*_, **__):
        ready.set()  # signal *after* we're inside request()
        raise BridgeError("never", "placeholder")  # replaced by abort_pending
    # Alternatively: patch the _ServerStream to signal on first iteration
    ...

A simpler pattern: set ready via a patched threading.Event.wait wrapper, or use a longer-but-bounded worker.join(timeout) with an explicit assertion rather than relying on timing.

💡 Suggested Change

Before:

        worker = threading.Thread(target=_do_request, daemon=True)
        worker.start()
        self.assertTrue(ready.wait(timeout=1.0))
        # Give the worker a beat to actually enter `waiter.wait(...)`.
        time.sleep(0.05)

After:

        worker = threading.Thread(target=_do_request, daemon=True)
        worker.start()
        self.assertTrue(ready.wait(timeout=1.0))
        # Give the worker a beat to actually enter `waiter.wait(...)`.
        # NOTE: this sleep is inherently racy — on slow CI machines 50 ms may
        # not be enough for request() to reach waiter.wait(), causing the test
        # to exercise the fast-fail poll() path instead of _abort_pending.
        # Consider using a deterministic synchronization primitive instead.
        time.sleep(0.05)

2. apps/memos-local-plugin/tests/python/test_bridge_client.py (L79-L81)

Dead code in this diff — terminate() is never called by any of the new tests. None of the five new test methods invoke terminate(), and the production close() path calls proc.terminate() only inside bridge_client.py's close(). None of the new tests exercise the full close() path through FakePopen.terminate(). The method is harmless to keep as infrastructure for future tests, but the magic constant -15 (SIGTERM) should at minimum be documented or replaced with a named constant. More importantly, if terminate() is meant to gate on the production close() path, at least one test should verify the behaviour.


3. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L185-L191)

Race condition: subprocess can exit between poll() and _pending registration.

The poll() pre-flight check at line 185 and the self._pending[rpc_id] = entry insertion at line 196 are not atomic — they happen outside the same critical section. The sequence of events that reintroduces issue #1722:

  1. poll() returns None (subprocess alive).
  2. The subprocess exits; reader thread calls _abort_pending, which clears _pending (which is currently empty).
  3. This thread acquires _lock, inserts the entry, writes to stdin, and releases _lock.
  4. The entry now lives in _pending permanently and waits the full timeout for a response that will never arrive.

The window is small but non-zero under load. To close it, move the poll() check inside the _lock block, before (or immediately after) the entry is registered in _pending, and add a double-check after acquiring the lock:

with self._lock:
    if self._closed:  # re-check under lock
        raise BridgeError("transport_closed", "bridge client is closed")
    exit_code = self._proc.poll()
    if exit_code is not None:
        raise BridgeError(
            "transport_closed",
            f"bridge subprocess exited (code={exit_code})",
        )
    rpc_id = self._next_id
    ...

Alternatively, in _abort_pending, after setting self._closed = True under _host_handlers_cv, re-check _pending under _lock to drain any entries that snuck in after the initial clear().

💡 Suggested Change

Before:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:

After:

        with self._lock:
            exit_code = self._proc.poll()
            if exit_code is not None:
                raise BridgeError(
                    "transport_closed",
                    f"bridge subprocess exited (code={exit_code})",
                )

4. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L396-L398)

Shared mutable error dict assigned to all pending entries.

The same error dict object reference is written to every entry's entry["error"] field. If any consumer code downstream (e.g., in future error-enrichment logic) mutates the dict — by adding context keys, popping fields, or modifying nested values — it would silently corrupt the error visible to all other concurrent callers that received the same reference.

Consider copying the dict per entry or using a frozen/immutable representation:

for entry in entries:
    entry["error"] = dict(error)  # shallow copy per entry
    entry["event"].set()

5. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L257-L260)

_close_called guard is set before the critical section completes, making a partial failure unrecoverable.

_close_called = True is assigned unconditionally before with self._host_handlers_cv and all subsequent cleanup steps (stdin close, process wait, SIGTERM/SIGKILL, _abort_pending). If any of those steps raises an unhandled exception — for example self._proc.stdin.close() leaking beyond contextlib.suppress, or a truly unexpected error in _abort_pending — close() can never be called again because the early-return guard has already tripped. This permanently leaks the client and its background reader thread.

The guard's intent is only to prevent concurrent/re-entrant calls, not to survive exceptions inside close(). Consider protecting only the concurrent-call race with a threading.Lock, or wrap the body in try/finally so cleanup still runs on exceptions:

def close(self) -> None:
    if self._close_called:
        return
    self._close_called = True
    try:
        with self._host_handlers_cv:
            ...
        # ... rest of cleanup ...
    except Exception:
        logger.exception("MemOS: unexpected error during bridge close")

6. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1811-L1823)

Thread-safety race on self._bridge (High)

_reconnect_bridge is protected by self._reconnect_lock, but reads and writes to self._bridge outside that lock are unprotected. Two concurrent tool invocations that both catch a transport_closed error will each call _reconnect_bridge, each potentially spawning a new subprocess and leaking the other. More critically, the keepalive thread can null-out self._bridge mid-reconnect while this code is about to call self._bridge.request(...) after the retry, producing an AttributeError instead of a BridgeError.

Suggested fix: wrap the entire try/except/reconnect/retry block under self._reconnect_lock, matching the same protection already applied inside _reconnect_bridge. This ensures only one thread reconnects at a time and all others wait for the restored bridge:

with self._reconnect_lock:
    try:
        return self._bridge.request(method, params, **kwargs) or {}
    except Exception as err:
        if not self._is_transport_closed(err):
            raise
        ...
        self._reconnect_bridge(self._session_id, timeout=30.0)
        assert self._bridge is not None
        return self._bridge.request(method, params, **kwargs) or {}

7. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1812)

Falsy-value collapse via or {} may swallow valid empty responses (Medium)

self._bridge.request(...) or {} collapses any falsy return value — including an empty list [] — to {}. An empty list is a perfectly valid JSON-RPC result for memory.search with zero hits; replacing it with {} breaks call sites that iterate the result (e.g., for hit in resp["hits"] will raise KeyError). The same pattern appears on the retry path (line 1823).

Suggested fix: use an explicit None-guard instead:

result = self._bridge.request(method, params, **kwargs)
return result if result is not None else {}

8. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1794)

Return type annotation dict[str, Any] is too narrow (Low)

bridge_client.request() can return a list (e.g., search-result arrays) as well as a dict. Annotating the return as dict[str, Any] will produce type-checker errors at call sites that treat the result as a list, and misrepresents the actual contract.

Suggested fix: broaden to dict[str, Any] | list[Any] (or whatever union is already used at other request() call sites in this file).


9. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1872-L1877)

Overly broad contextlib.suppress(Exception) around proc.poll() (Low)

Suppressing all exceptions hides legitimate programming mistakes. If proc.poll() raises (e.g., OSError on a zombie process in certain OS states), the broad suppression silently leaves the decision at False, causing the reconnect to be skipped.

Suggested fix: narrow the suppression to only the expected OS-level exception:

with contextlib.suppress(OSError):
    if proc.poll() is not None:
        return True

10. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L185-L191)

Race condition: subprocess can exit between poll() and _pending registration.

The poll() pre-flight check and the self._pending[rpc_id] = entry insertion are not atomic — they happen in separate critical sections. The problematic sequence:

  1. poll() returns None (subprocess alive).
  2. The subprocess exits; reader thread calls _abort_pending, which clears _pending (currently empty).
  3. This thread acquires _lock, inserts the new entry, writes to stdin, and releases _lock.
  4. The orphaned entry now lives in _pending permanently and waits the full timeout.

This is the same stall that fix #1722 aims to prevent. To close the window, move the poll() check inside the _lock block — after acquiring the lock but before registering the entry:

with self._lock:
    # Re-check atomically under the lock that protects _pending.
    exit_code = self._proc.poll()
    if exit_code is not None:
        raise BridgeError(
            "transport_closed",
            f"bridge subprocess exited (code={exit_code})",
        )
    rpc_id = self._next_id
    ...
💡 Suggested Change

Before:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:

After:

        with self._lock:
            exit_code = self._proc.poll()
            if exit_code is not None:
                raise BridgeError(
                    "transport_closed",
                    f"bridge subprocess exited (code={exit_code})",
                )

11. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L396-L398)

Shared mutable error dict assigned to all pending entries.

The same error dict object reference is written to every entry["error"]. If any downstream consumer ever mutates the dict (e.g., adds context keys, pops fields, or modifies nested values), it would silently corrupt the error visible to all other concurrent callers that received the same reference.

Use a per-entry copy to isolate each caller:

for entry in entries:
    entry["error"] = dict(error)  # shallow copy per entry
    entry["event"].set()

12. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L257-L260)

_close_called guard is set before cleanup completes, making a partial failure unrecoverable.

_close_called = True is assigned before the with self._host_handlers_cv block and all subsequent cleanup (stdin close, process wait, SIGTERM/SIGKILL, _abort_pending). If any of those steps raises an unhandled exception, close() can never be retried because the guard has already tripped, permanently leaking the client and its background reader thread.

The guard's intent is only to prevent re-entrant concurrent calls. Consider wrapping the body in try/except so cleanup errors are logged but don't leave the client in a permanently-uncloseable state:

def close(self) -> None:
    if self._close_called:
        return
    self._close_called = True
    try:
        with self._host_handlers_cv:
            ...
        # ... rest of cleanup ...
    except Exception:
        logger.exception("MemOS: unexpected error during bridge close")

13. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L185-L191)

Race condition: subprocess can exit between poll() and _pending registration.

The poll() pre-flight check (line 185) and the self._pending[rpc_id] = entry insertion (line 196) are not atomic — they happen outside the same critical section. The problematic sequence:

  1. poll() returns None (subprocess alive).
  2. The subprocess exits; the reader thread calls _abort_pending, which clears _pending (currently empty).
  3. This thread acquires _lock, inserts the new entry, writes to stdin, and releases _lock.
  4. The orphaned entry now lives in _pending permanently and waits the full timeout — the exact stall that this fix aims to prevent.

To close the window, move the poll() check inside the _lock block, before registering the entry:

with self._lock:
    # Re-check atomically under the lock that protects _pending.
    exit_code = self._proc.poll()
    if exit_code is not None:
        raise BridgeError(
            "transport_closed",
            f"bridge subprocess exited (code={exit_code})",
        )
    rpc_id = self._next_id
    ...
💡 Suggested Change

Before:

        exit_code = self._proc.poll()
        if exit_code is not None:
            raise BridgeError(
                "transport_closed",
                f"bridge subprocess exited (code={exit_code})",
            )
        with self._lock:

After:

        with self._lock:
            exit_code = self._proc.poll()
            if exit_code is not None:
                raise BridgeError(
                    "transport_closed",
                    f"bridge subprocess exited (code={exit_code})",
                )

14. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L396-L398)

Shared mutable error dict assigned to all pending entries.

The same error dict object reference is written into every entry["error"]. If any downstream consumer ever mutates the dict (e.g., adds context keys, pops fields, or modifies nested values such as data), it would silently corrupt the error visible to all concurrent callers that hold the same reference.

Use a shallow copy per entry to isolate callers:

for entry in entries:
    entry["error"] = dict(error)  # shallow copy per entry
    entry["event"].set()

15. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L256-L262)

_close_called guard is set before cleanup completes, making a partial failure unrecoverable.

_close_called = True is assigned unconditionally before the with self._host_handlers_cv block and all subsequent cleanup steps (stdin close, process wait, SIGTERM/SIGKILL, _abort_pending). If any of those steps raises an unhandled exception, close() can never be retried because the early-return guard has already tripped, permanently leaking the client and its background reader thread.

The guard's intent is only to prevent concurrent/re-entrant calls, not to survive exceptions inside close(). Consider wrapping the body in try/except so cleanup errors are logged but don't leave the client in a permanently-uncloseable state:

def close(self) -> None:
    if self._close_called:
        return
    self._close_called = True
    try:
        with self._host_handlers_cv:
            ...
        # ... rest of cleanup ...
    except Exception:
        logger.exception("MemOS: unexpected error during bridge close")

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants