Skip to content

fix(plugin): defer rescue-spawn while the managed bridge unit is active - #2638

Open
syzsunshine219 wants to merge 2 commits into
mainfrom
codex/maintainer-edit-pr-2589-20261119
Open

syzsunshine219 wants to merge 2 commits into
mainfrom
codex/maintainer-edit-pr-2589-20261119

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

The source branch for the change proposed in #2589 now lives in MemTensor/MemOS, so repository writers can contribute fixes through the normal upstream branch permissions.

The code and commit authors are unchanged. This PR retains target branch main and exact source head 6f042035938e667c4529416eb8e07b84d6033ed8 from #2589.

Previous discussion and review remain in #2589. Review and checks must be re-evaluated for this replacement PR; no previous approval or new test result is asserted by this migration.

Previous PR description and implementation details

Restores the change proposed in #2483 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: 6f042035938e667c4529416eb8e07b84d6033ed8

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2483.


Original change description

During a systemd restart window (~8-10s) nothing is bound on :18800 yet, so _probe_viewer() can only classify the port as free/running_memos/occupied/unknown and the free verdict is read as spawn clearance. The session host then rescue-spawns its own --daemon (_bridge_command(daemon=True) + Popen(start_new_session=True)), wins first bind, and memos-bridge.service crash-loops forever on "port busy" while the rogue answers /api/v1/health green. Killing the rogue does not help: both host keepalive paths (legacy _start_bridge_keepalive and shared_bridge_runtime.reconnect:219, whose before_spawn runs on every respawn) re-enter ensure_viewer_daemon within ~3s, and killExistingBridge in bridge.cts (via daemon/bridge.pid) makes the war bidirectional.

Description

Add a managed-unit gate to the rescue-spawn path of ensure_viewer_daemon() (apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py:538, acquired after the viewer-start.lock and before the Popen):

  • _managed_daemon_gate() (daemon_manager.py:403) runs systemctl is-active on MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service, env-overridable, empty disables the gate). While the unit reports active/activating, it waits up to 20s (1s poll, daemon_manager.py:68-69) for the managed daemon to take :18800.
  • If the managed daemon takes the port, the gate returns running and ensure_viewer_daemon() reports the viewer as up and returns True without spawning.
  • Only when the unit leaves active/activating (failed/inactive) does the rescue-spawn proceed as before.
  • Any failure of the check (no systemctl, no bus, timeout) degrades to the current behavior — macOS, Windows, containers and hand-managed daemons are unaffected.

Root cause and full trace are in #2467.

Related Issue (Required): Fixes #2467

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Red (pre-fix baseline, acceptance run): from apps/memos-local-plugin:

timeout 120 python3 -m unittest tests.python.test_daemon_manager -v

→ Ran 12 tests, FAILED (failures=3, errors=6) — the 3 failures are exactly the broken popen.assert_not_called() assertions: Popen was invoked with the rogue command line from the issue (node --import tsx .../bridge.mts --agent=hermes --daemon); the 6 errors reference the new helpers that did not exist yet (expected red).

Green (post-fix, re-run on this branch at commit 6f042035):

cd apps/memos-local-plugin
timeout 120 python3 -m unittest tests.python.test_daemon_manager -v
# Ran 12 tests in 0.128s — OK
timeout 300 python3 -m unittest tests.python.test_daemon_manager tests.python.test_bridge_client tests.python.test_bridge_script_resolution tests.python.test_hermes_install_environment tests.python.test_hermes_provider_pipeline tests.python.test_runtime_home tests.python.test_shared_bridge_runtime
# Ran 144 tests in 7.472s — OK

Lint (pyproject.toml:186 pins ruff ^0.11.8; run via uvx at the locked version because no local ruff binary):

uvx ruff@0.11.8 check --no-fix adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# All checks passed!
uvx ruff@0.11.8 format --check adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# 2 files already formatted

(CI enforces both: .github/workflows/python-tests.yml:95-96.)

  • Unit Test
  • Test Script Or Test Steps (please provide)

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

Reviewer Checklist

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Made sure Checks passed
  • Tests have been provided

Transparency note: this fix was produced by an AI agent (Claude Code, GLM) working via the akushonkamen account, as already disclosed in the issue claim comment. All commands and numbers above were executed against this branch.

…ve (#2467)

During a systemd restart window (~8-10s) :18800 probes "free" while the
supervised daemon is mid-start. ensure_viewer_daemon treated that as
spawn clearance, so a session host rescue-spawned its own --daemon,
won first bind, and the memos-bridge.service unit crash-looped forever
on "port busy" while the rogue answered /api/v1/health green (413
restarts/24h observed in #2467). Killing the rogue did not help: the
host keepalive/reconnect path re-entered ensure_viewer_daemon within
~3s (shared_bridge_runtime.reconnect runs before_spawn on every
respawn).

Gate the rescue-spawn path on `systemctl is-active` for
MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service): while the
unit is active/activating, wait up to 20s for the managed daemon to
take the port (report the viewer up when it does) instead of racing
it; spawn again only once the unit leaves active/activating. Any
failure of the check (no systemctl, no systemd bus, timeout) degrades
to today's unconditional rescue behavior, so macOS/Windows/containers
and manually managed daemons are unaffected.

Unit-side yield (bind-retry backoff for a daemon losing first-bind to
a mid-start unit) is left as follow-up work.
Acceptance requires the repo-enforced `ruff format --check`
(.github/workflows/python-tests.yml:95-96) to pass. No semantic
change: one gate log string joined onto a single line, patch.object
call wrapping normalized in the new test file. Verified: uvx
ruff@0.11.8 format --check . -> 636 files already formatted; ruff
check --no-fix . -> All checks passed; python test suite still
144/144 OK.
@syzsunshine219 syzsunshine219 added the area:plugin OpenClaw & Hermes label Nov 19, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Nov 19, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2638
Task: 7d278856737e0da6
Base: main
Head: codex/maintainer-edit-pr-2589-20261119
Head SHA: 6f042035938e667c4529416eb8e07b84d6033ed8

🔍 OpenCodeReview found 2 issue(s) in this PR.


1. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L441-L444)

Inside the wait loop, _systemd_unit_state() returning None means "undetectable" (systemctl timeout, DBus hiccup, transient OSError) per its own docstring, but the loop treats it as "unit stopped" and immediately resumes the rescue-spawn. Since we only enter this loop after a first call reported active/activating, a single transient systemctl failure during exactly the restart window this fix targets (systemd is busiest then, and the timeout=2.0 is easy to hit) reopens the #2467 port war. Only terminal states (inactive/failed/deactivating-style "stopped") should release the deferral; None should keep polling until the deadline (then fail closed with refuse, consistent with the unknown probe path).

💡 Suggested Change

Before:

        state = _systemd_unit_state(unit)
        if state not in ("active", "activating"):
            logger.info(
                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",

After:

        state = _systemd_unit_state(unit)
        if state is None:
            # Transient systemctl failure is not evidence the unit stopped;
            # keep deferring until the deadline instead of racing the unit.
            if time.time() >= deadline:
                logger.warning(
                    "MemOS: managed bridge unit %s state undetectable after %.0fs and "
                    "port %d still free; refusing to spawn a competing daemon",
                    unit,
                    MANAGED_UNIT_GRACE_SEC,
                    HERMES_VIEWER_PORT,
                )
                return "refuse"
            time.sleep(MANAGED_UNIT_POLL_SEC)
            continue
        if state not in ("active", "activating"):
            logger.info(
                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",

2. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L444-L446)

The message formats the new (non-active) state into "is no longer %s", so state == "failed" reads "unit X is no longer failed — proceeding with rescue-spawn" and state == "inactive" reads "is no longer inactive", which is factually wrong and misleading during triage. The or "active/activating" fallback also prints a state that was never observed (the None case). Suggest logging the observed state as the current state.

💡 Suggested Change

Before:

                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",
                unit,
                state or "active/activating",

After:

                "MemOS: managed bridge unit %s left active/activating (now %s) — "
                "proceeding with rescue-spawn",
                unit,
                state or "undetectable",

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Host rescue-spawns its own bridge daemon during a systemd-daemon restart window — permanent port war, health stays green (2.0.20)

5 participants