Skip to content

bug(hermes): _bridge_ok cache permanently disables provider after transient startup failure #1797

Closed

Description

@MuBeiGe

Summary

daemon_manager.py's ensure_bridge_running(probe_only=True) caches _bridge_ok as a module-level global with no expiry mechanism. Once set to False by a transient failure (subprocess creation race, env-var load ordering), it stays False for the lifetime of the gateway process. This causes MemTensorProvider.is_available() to return False permanently, and agent_init.py skips adding the provider — MemOS is silently disabled despite the bridge daemon being perfectly healthy.

Root Cause

In adapters/hermes/memos_provider/daemon_manager.py, the caching logic at lines 130–133:

if _bridge_ok is not None and probe_only:
    return _bridge_ok  # ← returns stale False forever

The sequence that triggers the bug:

  1. Hermes gateway starts, loads .env (which sets MEMOS_NODE_BINARY)
  2. agent_init.py calls is_available() → ensure_bridge_running(probe_only=True)
  3. If _node_available() fails at this moment (e.g. transient subprocess error on Windows, or .env not yet loaded in os.environ), _bridge_ok is set to False
  4. Meanwhile ensure_viewer_daemon() (called during initialize() without probe_only) finds the bridge already running on port 18800 via _probe_viewer() and returns True — without ever calling ensure_bridge_running() again, so _bridge_ok stays False forever
  5. is_available() returns False → agent_init.py:994 skips add_provider() → MemOS offline for the gateway's lifetime

Symptoms

  • Repeated "MemOS: Node.js not found on PATH" warnings in logs (from non-session contexts, no session ID prefix)
  • Bridge health endpoint (/api/v1/health) shows llm.available: true and embedder.available: true
  • But bridge_client never logs any activity — the provider was never registered
  • hermes doctor shows memtensor as unavailable

Proposed Fix

Three changes to daemon_manager.py, ~20 lines net new code:

1. TTL-based cache expiry

Add _bridge_ok_at: float = 0.0 timestamp and BRIDGE_OK_TTL_SEC = 60.0. Cached results expire after 60 seconds, forcing revalidation.

2. Running-bridge fallback

When _node_available() fails, check whether a bridge is already alive via _probe_viewer() == "running_memos". A live bridge process is definitive proof the environment is viable (the bridge itself was launched with Node.js).

3. shutdown_bridge() resets both variables

Fixed ensure_bridge_running:

def ensure_bridge_running(*, probe_only: bool = False) -> bool:
    global _bridge_ok, _bridge_ok_at
    with _lock:
        now = time.time()
        if _bridge_ok is not None and probe_only:
            if (now - _bridge_ok_at) < BRIDGE_OK_TTL_SEC:
                return _bridge_ok
            # Cache expired — fall through to revalidate.

        script = _bridge_script()
        if not script.exists():
            logger.warning("MemOS: bridge script missing at %s", script)
            _bridge_ok = False
            _bridge_ok_at = now
            return False

        if _node_available():
            _bridge_ok = True
            _bridge_ok_at = now
            return True

        # Node binary check failed. Check if bridge is already running.
        if _probe_viewer() == "running_memos":
            _bridge_ok = True
            _bridge_ok_at = now
            return True

        logger.warning("MemOS: Node.js not found on PATH")
        _bridge_ok = False
        _bridge_ok_at = now
        return False

Environment

  • MemOS version: 2.0.5 (@memtensor/memos-local-plugin)
  • Hermes Agent on Windows 10
  • Node.js v24.14.1 (path set via MEMOS_NODE_BINARY in .env)

Notes

Happy to submit a PR if this direction looks right. The fix has been tested on Windows — syntax verified, logic tested with fresh import.

Activity

  1. deleted a comment from MaiHHConnect on May 27, 2026
  2. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 29, 2026
  3. self-assigned this
    on Jun 30, 2026
  4. Memtensor-AI commented on Jun 30, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 0c949574dc352cff
    Working branch: bugfix/autodev-1797
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  5. added a commit that references this issue on Jun 30, 2026
    72f1756
  6. Memtensor-AI commented on Jun 30, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 0c949574dc352cff completed.

    Summary: Fixes #1797: _bridge_ok cache in apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py had no expiry, so a single transient _node_available() failure during Hermes gateway startup permanently disabled MemTensorProvider. The cache now expires after 60 seconds, and _node_available() failures fall through to a _probe_viewer() check that accepts an already-running MemOS bridge on :18800 as proof the environment is viable. shutdown_bridge() also resets the new _bridge_ok_at timestamp.

    Added regression tests (BridgeOkCacheTests, 6 cases) covering cold-start revalidation, cache-hit short-circuit within TTL, TTL-expiry recovery, running-bridge fallback, shutdown reset, and full-call (non-probe) revalidation. Full plugin python suite green: 48 passed (42 existing + 6 new). Ruff lint and format both clean. Branch bugfix/autodev-1797 pushed to origin; openspec artifacts archived to memos-autodev-specs.

    Base branch: main
    Branch: bugfix/autodev-1797
    Commit: 72f1756
    PR: #2001
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  7. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 30, 2026
  8. Memtensor-AI commented on Jul 1, 2026

    @Memtensor-AI
    Collaborator

    ❌ AutoDev task 0c949574dc352cff failed.

    Summary: Claude Code 未通过 submit_result 工具提交结果,且无文本输出
    Error: Claude SDK stream ended without invoking submit_result and without producing any text

  9. added a commit that references this issue on Jul 2, 2026
    c507b04
  10. added and removed
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    ai:failedAI task failed | AI 任务失败
    on Jul 2, 2026
  11. added a commit that references this issue on Sep 18, 2026
    f54400c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:pluginOpenClaw & Hermestypes:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions