Skip to content

Fix #1722: fix: memory_search returned error - #2015

Closed
Memtensor-AI wants to merge 1 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722
Closed

Memtensor-AI wants to merge 1 commit into
MemTensor:dev-20260624-v2.0.22from
Memtensor-AI:bugfix/autodev-1722

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fixes GitHub issue #1722 (memory_search timing out after long Hermes sessions) in the Hermes MemOS Python adapter. Three cooperating defects allowed a dead or hung Node bridge subprocess to poison every subsequent request without triggering a reconnect: the reader thread returned silently on subprocess exit leaving JSON-RPC waiters parked for their full 30 s timeout, the keepalive only reconnected on transport_closed (ignoring the timeout error the previous defect produced), and the read-path memory tools had no reconnect+retry unlike sync_turn.

Fix layered across MemosBridgeClient and MemTensorProvider: (1) the reader loop's new finally block calls a shared _abort_pending helper that wakes every pending waiter with transport_closed in under a second; (2) request() fast-fails via Popen.poll() when the subprocess is already dead so we never write to a buffered dead pipe; (3) a new _should_reconnect_after_keepalive_failure helper reconnects on transport_closed / timeout / dead subprocess; (4) a new _bridge_request_with_retry helper runs one reconnect+retry on transport_closed for every read-path memory tool. The JSON-RPC wire protocol and the Node bridge are untouched.

Test evidence: added 7 unit tests covering reader-exit cleanup, poll()-based fast-fail, read-path recovery, keepalive reconnect triggers, and the "still broken after reconnect" surface-verbatim behaviour. Full bridge-client suite (33 tests) and adjacent Hermes provider pipeline suite (16 tests) pass. ruff check and ruff format --check both clean.

Ready to review: 3 files changed, 427 insertions, 88 deletions, commit d63b3f5 on bugfix/autodev-1722. Specs archive pushed to memos-autodev-specs main (commit da2b3b7).

Related Issue (Required): Fixes #1722

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Automated tests are pending.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.

Reviewer Checklist

…ensor#1722)

Long-running Hermes sessions (1-2 h) were hitting
`[timeout] memory.search did not respond within 30.0s` and staying broken
for the rest of the session. Three cooperating defects in the Python
bridge adapter allowed a dead / hung Node bridge subprocess to poison
every subsequent request without ever triggering a reconnect:

1. `MemosBridgeClient._read_loop` returned silently on stdout EOF, so
   pending JSON-RPC waiters parked for their full 30 s timeout.
2. `MemTensorProvider._start_bridge_keepalive` only reconnected on
   `transport_closed`; a `BridgeError("timeout", ...)` from the health
   probe was dropped at DEBUG.
3. `handle_tool_call` for `memos_search` (and the other read-path memory
   tools) had no reconnect+retry, unlike `sync_turn`.

Fix:

* Reader thread's `finally` wakes every pending waiter with
  `transport_closed` (new `_abort_pending` helper).
* `request()` fast-fails via `Popen.poll()` when the subprocess is
  already dead so we never write to a buffered dead pipe.
* Keepalive delegates to `_should_reconnect_after_keepalive_failure`,
  which reconnects on transport_closed / timeout / dead subprocess.
* Read-path memory tools now share `_bridge_request_with_retry`, which
  reconnects and retries once on `transport_closed` (matches sync_turn).

Adds 7 unit tests covering reader-exit cleanup, poll()-based fast-fail,
read-path recovery, keepalive reconnect triggers, and the "still broken
after reconnect" surface-verbatim behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Memtensor-AI Memtensor-AI added ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常 labels Jul 1, 2026
@Memtensor-AI Memtensor-AI mentioned this pull request Jul 1, 2026
3 of 5 tasks
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2015
Task: 304d489a58d3731c
Base: dev-20260624-v2.0.22
Head: bugfix/autodev-1722

🔍 OpenCodeReview found 7 issue(s) in this PR.


1. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L392-L397)

Race condition (TOCTOU) in _abort_pending(): self._closed = True (under _host_handlers_cv) and self._pending.clear() (under _lock) are acquired in two separate critical sections. Between releasing _host_handlers_cv and acquiring _lock, another thread in request() can pass the if self._closed: guard (observed False before the first lock was acquired), insert a new entry into self._pending, and then have that entry cleared by the subsequent self._pending.clear() — all before setting entry["event"]. That caller would then block for its full timeout, reproducing the exact issue this PR is fixing.

To fix this atomically, both self._closed = True and the extraction/clear of self._pending should happen while the same lock is held. Since request() checks self._closed and inserts into self._pending both under self._lock, the fix is to set self._closed = True inside the with self._lock: block:

def _abort_pending(self, reason: str) -> None:
    with self._host_handlers_cv:
        self._host_handlers_cv.notify_all()
    with self._lock:
        self._closed = True          # atomic with the pending-map clear
        pending = list(self._pending.values())
        self._pending.clear()
    for entry in pending:
        entry["error"] = {...}
        entry["event"].set()

2. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L254-L259)

close() early-return path skips _abort_pending(): When self._closed is already True (set by the reader thread via _abort_pending()), the early-return branch calls _teardown_process() and returns — without calling _abort_pending(). Any request that was inserted into self._pending between the time _abort_pending() ran (in the reader thread) and the time the caller invokes close() will never be woken and will block until its full timeout expires.

Suggested fix: call _abort_pending() unconditionally on both paths, since it is designed to be idempotent (the second call finds _pending empty):

def close(self) -> None:
    if self._closed:
        self._teardown_process()
        self._abort_pending("bridge closed")  # idempotent; wakes any late arrivals
        return
    ...

3. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L270-L275)

_teardown_process() is not protected against concurrent invocation: There is no guard preventing two threads from calling _teardown_process() simultaneously (e.g., one hits the early-return path while another takes the normal close() path). Both will attempt to close self._proc.stdin, call self._proc.wait(), and potentially terminate()/kill() the same process. This can produce ValueError: I/O operation on closed file on the second stdin close, and calling kill() on an already-dead process raises ProcessLookupError on some platforms.

Consider guarding with a dedicated flag (protected by self._lock) that is set atomically on first entry:

def _teardown_process(self) -> None:
    with self._lock:
        if getattr(self, "_teardown_started", False):
            return
        self._teardown_started = True
    ...

4. apps/memos-local-plugin/adapters/hermes/memos_provider/bridge_client.py (L373-L375)

Read-loop crash is logged at DEBUG level: When the for line in self._proc.stdout loop raises an unexpected exception (e.g., an OSError from the OS, a bug in event dispatch), the error is only logged at logger.debug(...). This makes it very difficult to diagnose production problems — the bridge silently vanishes from the perspective of the operator. Since this is an unexpected crash (not a normal subprocess EOF), consider logging at WARNING or ERROR level:

except Exception as err:
    exit_reason = f"bridge read loop crashed: {err}"
    logger.warning("bridge: read loop raised %s", err, exc_info=True)

5. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1817-L1818)

Race condition: self._bridge can become None between the assertion and the .request() call.

The keepalive thread (running _reconnect_bridge periodically) can set self._bridge = None between line 1817 (assert self._bridge is not None) and line 1818 (return self._bridge.request(...)). This would raise an AttributeError: 'NoneType' object has no attribute 'request' in production — far more confusing than the original transport error.

Fix: capture self._bridge into a local variable before calling .request() so the reference is stable for the duration of the call:

# One reconnect, one retry, then surface whatever comes back.
self._reconnect_bridge(self._session_id, timeout=30.0)
bridge = self._bridge
assert bridge is not None
return bridge.request(method, params, **call_kwargs)

The same pattern should also be applied to the first call on line 1806 (return self._bridge.request(...)) — capture self._bridge into a local before the try block.

💡 Suggested Change

Before:

        assert self._bridge is not None
        return self._bridge.request(method, params, **call_kwargs)

After:

        bridge = self._bridge
        assert bridge is not None
        return bridge.request(method, params, **call_kwargs)

6. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1816)

Hardcoded 30-second reconnect timeout ignores the caller's timeout parameter.

If a caller ever passes a short timeout (e.g. 5 s for a latency-sensitive query), the reconnect phase alone can block for 30 s, violating the caller's expectation. While current call sites all omit timeout, the mismatch is a latent footgun once timeout is used.

Consider using the caller-supplied timeout for reconnect as well, falling back to 30 s only when timeout is None:

reconnect_timeout = timeout if timeout is not None else 30.0
self._reconnect_bridge(self._session_id, timeout=reconnect_timeout)

7. apps/memos-local-plugin/adapters/hermes/memos_provider/__init__.py (L1868-L1877)

The _should_reconnect_after_keepalive_failure poll branch silently degrades if _proc is renamed or removed from BridgeClient.

The getattr(bridge, '_proc', None) approach couples this method to a private implementation detail of MemosBridgeClient. If the attribute is ever renamed, this branch silently returns False instead of True, causing the keepalive to miss a dead subprocess — with no warning or test coverage (the # pragma: no cover comment confirms this).

Since bridge_client.py now raises BridgeError("transport_closed", ...) in request() whenever poll() returns non-None (the fast-fail path added in this PR), a dead subprocess will already surface as transport_closed on the health probe, making the poll() check here redundant. If the poll check is still desired as an extra layer, consider exposing a public is_alive() -> bool method on MemosBridgeClient instead of using reflection.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Command failed: git clone --depth 1 --branch bugfix/autodev-1722 git@github.com:MemTensor/MemOS.git /data/test-workspaces/304d489a58d3731c/repo
Cloning into '/data/test-workspaces/304d489a58d3731c/repo'...
warning: Could not find remote branch bugfix/autodev-1722 to clone.
fatal: Remote branch bugfix/autodev-1722 not found in upstream origin
Branch: bugfix/autodev-1722

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 types:bug Something isn't working | 功能异常

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants