Skip to content

fix(plugin): defer rescue-spawn while the managed bridge unit is active - #2589

Open
syzsunshine219 wants to merge 2 commits into
MemTensor:mainfrom
akushonkamen:fix/2467-defer-rescue-spawn-to-managed-unit
Open

syzsunshine219 wants to merge 2 commits into
MemTensor:mainfrom
akushonkamen:fix/2467-defer-rescue-spawn-to-managed-unit

Conversation

@syzsunshine219

@syzsunshine219 syzsunshine219 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Restores the change proposed in #2483 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: 6f042035938e667c4529416eb8e07b84d6033ed8

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2483.


Original change description

During a systemd restart window (~8-10s) nothing is bound on :18800 yet, so _probe_viewer() can only classify the port as free/running_memos/occupied/unknown and the free verdict is read as spawn clearance. The session host then rescue-spawns its own --daemon (_bridge_command(daemon=True) + Popen(start_new_session=True)), wins first bind, and memos-bridge.service crash-loops forever on "port busy" while the rogue answers /api/v1/health green. Killing the rogue does not help: both host keepalive paths (legacy _start_bridge_keepalive and shared_bridge_runtime.reconnect:219, whose before_spawn runs on every respawn) re-enter ensure_viewer_daemon within ~3s, and killExistingBridge in bridge.cts (via daemon/bridge.pid) makes the war bidirectional.

Description

Add a managed-unit gate to the rescue-spawn path of ensure_viewer_daemon() (apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py:538, acquired after the viewer-start.lock and before the Popen):

  • _managed_daemon_gate() (daemon_manager.py:403) runs systemctl is-active on MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service, env-overridable, empty disables the gate). While the unit reports active/activating, it waits up to 20s (1s poll, daemon_manager.py:68-69) for the managed daemon to take :18800.
  • If the managed daemon takes the port, the gate returns running and ensure_viewer_daemon() reports the viewer as up and returns True without spawning.
  • Only when the unit leaves active/activating (failed/inactive) does the rescue-spawn proceed as before.
  • Any failure of the check (no systemctl, no bus, timeout) degrades to the current behavior — macOS, Windows, containers and hand-managed daemons are unaffected.

Root cause and full trace are in #2467.

Related Issue (Required): Fixes #2467

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Red (pre-fix baseline, acceptance run): from apps/memos-local-plugin:

timeout 120 python3 -m unittest tests.python.test_daemon_manager -v

→ Ran 12 tests, FAILED (failures=3, errors=6) — the 3 failures are exactly the broken popen.assert_not_called() assertions: Popen was invoked with the rogue command line from the issue (node --import tsx .../bridge.mts --agent=hermes --daemon); the 6 errors reference the new helpers that did not exist yet (expected red).

Green (post-fix, re-run on this branch at commit 6f042035):

cd apps/memos-local-plugin
timeout 120 python3 -m unittest tests.python.test_daemon_manager -v
# Ran 12 tests in 0.128s — OK
timeout 300 python3 -m unittest tests.python.test_daemon_manager tests.python.test_bridge_client tests.python.test_bridge_script_resolution tests.python.test_hermes_install_environment tests.python.test_hermes_provider_pipeline tests.python.test_runtime_home tests.python.test_shared_bridge_runtime
# Ran 144 tests in 7.472s — OK

Lint (pyproject.toml:186 pins ruff ^0.11.8; run via uvx at the locked version because no local ruff binary):

uvx ruff@0.11.8 check --no-fix adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# All checks passed!
uvx ruff@0.11.8 format --check adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# 2 files already formatted

(CI enforces both: .github/workflows/python-tests.yml:95-96.)

  • Unit Test
  • Test Script Or Test Steps (please provide)

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

Reviewer Checklist

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Made sure Checks passed
  • Tests have been provided

Transparency note: this fix was produced by an AI agent (Claude Code, GLM) working via the akushonkamen account, as already disclosed in the issue claim comment. All commands and numbers above were executed against this branch.

…ve (MemTensor#2467)

During a systemd restart window (~8-10s) :18800 probes "free" while the
supervised daemon is mid-start. ensure_viewer_daemon treated that as
spawn clearance, so a session host rescue-spawned its own --daemon,
won first bind, and the memos-bridge.service unit crash-looped forever
on "port busy" while the rogue answered /api/v1/health green (413
restarts/24h observed in MemTensor#2467). Killing the rogue did not help: the
host keepalive/reconnect path re-entered ensure_viewer_daemon within
~3s (shared_bridge_runtime.reconnect runs before_spawn on every
respawn).

Gate the rescue-spawn path on `systemctl is-active` for
MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service): while the
unit is active/activating, wait up to 20s for the managed daemon to
take the port (report the viewer up when it does) instead of racing
it; spawn again only once the unit leaves active/activating. Any
failure of the check (no systemctl, no systemd bus, timeout) degrades
to today's unconditional rescue behavior, so macOS/Windows/containers
and manually managed daemons are unaffected.

Unit-side yield (bind-retry backoff for a daemon losing first-bind to
a mid-start unit) is left as follow-up work.
Acceptance requires the repo-enforced `ruff format --check`
(.github/workflows/python-tests.yml:95-96) to pass. No semantic
change: one gate log string joined onto a single line, patch.object
call wrapping normalized in the new test file. Verified: uvx
ruff@0.11.8 format --check . -> 636 files already formatted; ruff
check --no-fix . -> All checks passed; python test suite still
144/144 OK.
@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Nov 9, 2026
@syzsunshine219
syzsunshine219 marked this pull request as ready for review November 9, 2026 15:48
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2589
Task: ed8bac24fe2cca64
Base: main
Head: fix/2467-defer-rescue-spawn-to-managed-unit
Head SHA: 6f042035938e667c4529416eb8e07b84d6033ed8

🔍 OpenCodeReview found 3 issue(s) in this PR.


1. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L444-L446)

The log message interpolates the current state into a phrase that expects the previous one, so it reads as a contradiction: with state == "failed" it prints "managed bridge unit memos-bridge.service is no longer failed — proceeding with rescue-spawn", even though the unit being failed is precisely why the rescue spawn proceeds. Suggested rewrite to name the current state directly (same pattern as the other two messages in this function).

💡 Suggested Change

Before:

                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",
                unit,
                state or "active/activating",

After:

                "MemOS: managed bridge unit %s stopped being active/activating "
                "(now %s) — proceeding with rescue-spawn",
                unit,
                state or "inactive",

2. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L398-L400)

except Exception returns None with no logging at all. If systemctl exists but fails for an unexpected reason (dbus permission error, bus not reachable, TimeoutExpired), the #2467 protection is silently disabled in production and there is no trace in the logs to explain why a rescue-spawn still raced the managed unit. A debug-level breadcrumb keeps the documented graceful degradation while making the disabled-guard case diagnosable.

💡 Suggested Change

Before:

    except Exception:
        return None
    return (completed.stdout or "").strip().lower() or None

After:

    except Exception as err:
        logger.debug("MemOS: systemctl is-active %s failed — %s", unit, err)
        return None
    return (completed.stdout or "").strip().lower() or None

3. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L441-L444)

Inside the wait loop, _systemd_unit_state() returning None (systemctl timeout / transient bus failure) is indistinguishable from a real state change, so the gate falls through to "spawn" and rescue-spawns a competing --daemon — exactly the #2467 port war this function exists to prevent. The fail-open degrade to None makes sense for the initial probe (non-systemd hosts must keep today's behavior), but once we have already observed active/activating the host demonstrably runs systemd, so a None here should be treated as "still waiting" (or fail closed at the deadline) rather than "unit stopped".

💡 Suggested Change

Before:

        state = _systemd_unit_state(unit)
        if state not in ("active", "activating"):
            logger.info(
                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",

After:

        state = _systemd_unit_state(unit)
        if state is not None and state not in ("active", "activating"):
            logger.info(
                "MemOS: managed bridge unit %s is no longer active/activating — proceeding with rescue-spawn",

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: GIT PREPARATION FAILED

Git preparation did not complete. Tests were not executed; inspect the recorded command evidence.

Details: Executor error: Git fetch failed (unknown, attempt 1/2, timeout 300000ms).
Connection to github.com closed by remote host.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Branch: fix/2467-defer-rescue-spawn-to-managed-unit

Tested source: 6f042035938e667c4529416eb8e07b84d6033ed8
Tested base: 3a16e5fe3048a209f722806f4123fa2e533588fd
Tested commit: ef7a9b06bbc8df2d9bafe3377de36b68f2831b4f
These results apply only to this immutable source/base snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Host rescue-spawns its own bridge daemon during a systemd-daemon restart window — permanent port war, health stays green (2.0.20)

4 participants