Skip to content

fix(plugin): defer rescue-spawn while the managed bridge unit is active - #2483

Closed
akushonkamen wants to merge 2 commits into
MemTensor:dev-v2.0.36from
akushonkamen:fix/2467-defer-rescue-spawn-to-managed-unit
Closed

akushonkamen wants to merge 2 commits into
MemTensor:dev-v2.0.36from
akushonkamen:fix/2467-defer-rescue-spawn-to-managed-unit

Conversation

@akushonkamen

Copy link
Copy Markdown

During a systemd restart window (~8-10s) nothing is bound on :18800 yet, so _probe_viewer() can only classify the port as free/running_memos/occupied/unknown and the free verdict is read as spawn clearance. The session host then rescue-spawns its own --daemon (_bridge_command(daemon=True) + Popen(start_new_session=True)), wins first bind, and memos-bridge.service crash-loops forever on "port busy" while the rogue answers /api/v1/health green. Killing the rogue does not help: both host keepalive paths (legacy _start_bridge_keepalive and shared_bridge_runtime.reconnect:219, whose before_spawn runs on every respawn) re-enter ensure_viewer_daemon within ~3s, and killExistingBridge in bridge.cts (via daemon/bridge.pid) makes the war bidirectional.

Description

Add a managed-unit gate to the rescue-spawn path of ensure_viewer_daemon() (apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py:538, acquired after the viewer-start.lock and before the Popen):

  • _managed_daemon_gate() (daemon_manager.py:403) runs systemctl is-active on MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service, env-overridable, empty disables the gate). While the unit reports active/activating, it waits up to 20s (1s poll, daemon_manager.py:68-69) for the managed daemon to take :18800.
  • If the managed daemon takes the port, the gate returns running and ensure_viewer_daemon() reports the viewer as up and returns True without spawning.
  • Only when the unit leaves active/activating (failed/inactive) does the rescue-spawn proceed as before.
  • Any failure of the check (no systemctl, no bus, timeout) degrades to the current behavior — macOS, Windows, containers and hand-managed daemons are unaffected.

Root cause and full trace are in #2467.

Related Issue (Required): Fixes #2467

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Red (pre-fix baseline, acceptance run): from apps/memos-local-plugin:

timeout 120 python3 -m unittest tests.python.test_daemon_manager -v

→ Ran 12 tests, FAILED (failures=3, errors=6) — the 3 failures are exactly the broken popen.assert_not_called() assertions: Popen was invoked with the rogue command line from the issue (node --import tsx .../bridge.mts --agent=hermes --daemon); the 6 errors reference the new helpers that did not exist yet (expected red).

Green (post-fix, re-run on this branch at commit 6f042035):

cd apps/memos-local-plugin
timeout 120 python3 -m unittest tests.python.test_daemon_manager -v
# Ran 12 tests in 0.128s — OK
timeout 300 python3 -m unittest tests.python.test_daemon_manager tests.python.test_bridge_client tests.python.test_bridge_script_resolution tests.python.test_hermes_install_environment tests.python.test_hermes_provider_pipeline tests.python.test_runtime_home tests.python.test_shared_bridge_runtime
# Ran 144 tests in 7.472s — OK

Lint (pyproject.toml:186 pins ruff ^0.11.8; run via uvx at the locked version because no local ruff binary):

uvx ruff@0.11.8 check --no-fix adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# All checks passed!
uvx ruff@0.11.8 format --check adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# 2 files already formatted

(CI enforces both: .github/workflows/python-tests.yml:95-96.)

  • Unit Test
  • Test Script Or Test Steps (please provide)

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

Reviewer Checklist

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Made sure Checks passed
  • Tests have been provided

Transparency note: this fix was produced by an AI agent (Claude Code, GLM) working via the akushonkamen account, as already disclosed in the issue claim comment. All commands and numbers above were executed against this branch.

…ve (MemTensor#2467)

During a systemd restart window (~8-10s) :18800 probes "free" while the
supervised daemon is mid-start. ensure_viewer_daemon treated that as
spawn clearance, so a session host rescue-spawned its own --daemon,
won first bind, and the memos-bridge.service unit crash-looped forever
on "port busy" while the rogue answered /api/v1/health green (413
restarts/24h observed in MemTensor#2467). Killing the rogue did not help: the
host keepalive/reconnect path re-entered ensure_viewer_daemon within
~3s (shared_bridge_runtime.reconnect runs before_spawn on every
respawn).

Gate the rescue-spawn path on `systemctl is-active` for
MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service): while the
unit is active/activating, wait up to 20s for the managed daemon to
take the port (report the viewer up when it does) instead of racing
it; spawn again only once the unit leaves active/activating. Any
failure of the check (no systemctl, no systemd bus, timeout) degrades
to today's unconditional rescue behavior, so macOS/Windows/containers
and manually managed daemons are unaffected.

Unit-side yield (bind-retry backoff for a daemon losing first-bind to
a mid-start unit) is left as follow-up work.
Acceptance requires the repo-enforced `ruff format --check`
(.github/workflows/python-tests.yml:95-96) to pass. No semantic
change: one gate log string joined onto a single line, patch.object
call wrapping normalized in the new test file. Verified: uvx
ruff@0.11.8 format --check . -> 636 files already formatted; ruff
check --no-fix . -> All checks passed; python test suite still
144/144 OK.
@akushonkamen

Copy link
Copy Markdown
Author

Local green evidence for the autotest, since external-PR runs on this scope are known to fail at environment preparation (#2476):

cd apps/memos-local-plugin
timeout 120 python3 -m unittest tests.python.test_daemon_manager -v
# Ran 12 tests in 0.128s — OK
timeout 300 python3 -m unittest tests.python.test_daemon_manager tests.python.test_bridge_client tests.python.test_bridge_script_resolution tests.python.test_hermes_install_environment tests.python.test_hermes_provider_pipeline tests.python.test_runtime_home tests.python.test_shared_bridge_runtime
# Ran 144 tests in 7.472s — OK
uvx ruff@0.11.8 check --no-fix adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# All checks passed!
uvx ruff@0.11.8 format --check adapters/hermes/memos_provider/daemon_manager.py tests/python/test_daemon_manager.py
# 2 files already formatted

ruff is pinned to ^0.11.8 in pyproject.toml:186 and CI enforces both check and format (.github/workflows/python-tests.yml:95-96).

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2483
Task: 64e0c144b3e044f5
Base: dev-v2.0.36
Head: fix/2467-defer-rescue-spawn-to-managed-unit

🔍 OpenCodeReview found 5 issue(s) in this PR.


1. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L451-L454)

Dead code: state or 'unknown' can never evaluate to 'unknown' at this point. The re-check _systemd_unit_state(unit) at line 441 is immediately followed by if state not in ('active', 'activating'): return 'spawn'. Any falsy state (i.e. None) would have returned 'spawn' above, so when execution reaches the deadline branch state is always 'active' or 'activating'. Drop the fallback, or replace it with assert state is not None to make the invariant explicit.

💡 Suggested Change

Before:

                "MemOS: managed bridge unit %s still %s after %.0fs and port "
                "%d still free; refusing to spawn a competing daemon",
                unit,
                state or "unknown",

After:

                "MemOS: managed bridge unit %s still %s after %.0fs and port "
                "%d still free; refusing to spawn a competing daemon",
                unit,
                state,

2. apps/memos-local-plugin/tests/python/test_daemon_manager.py (L146-L151)

The three rescue-spawn tests (test_managed_unit_failure_releases_rescue_spawn, test_no_managed_unit_keeps_unconditional_rescue_spawn, test_inactive_managed_unit_still_allows_rescue_spawn) mock subprocess.Popen but do NOT mock log_file.open("a") or _bridge_command.

In ensure_viewer_daemon, the real call sequence before Popen is:

log_handle = log_file.open("a", encoding="utf-8")  # real FS write
_viewer_process = subprocess.Popen(...)              # mocked
log_handle.close()

The log_file is _plugin_root() / "logs" / "daemon-start.log". Because only Popen is patched, the production code will attempt to create the logs/ directory and open the log file for real, producing an artefact in the plugin source tree. Additionally, _bridge_command() is called just before Popen to build the command list; it calls _node_binary(), which raises RuntimeError("Node.js not found on PATH") when Node is absent — silently failing the test for the wrong reason on any CI agent without Node.

Fix: also patch _bridge_command (to avoid the Node dependency) and patch log_file.open or the Path.open call (to avoid real FS I/O). The cleanest approach is to mock _bridge_command at the module level and pass a runtime_home=tmp_path so the log directory stays in a temp location:

patch.object(daemon_manager_mod, "_bridge_command", return_value=["/usr/bin/node", "bridge.cjs", "--daemon"]),
💡 Suggested Change

Before:

patch.object(
                daemon_manager_mod.subprocess, "Popen", return_value=FakeDaemon()
            ) as popen,
        ):
            self.assertTrue(daemon_manager_mod.ensure_viewer_daemon())
            popen.assert_called_once()

After:

            patch.object(daemon_manager_mod, "_bridge_command",
                         return_value=["/usr/bin/node", "bridge.cjs", "--daemon"]),
            patch.object(daemon_manager_mod, "_plugin_root",
                         return_value=Path(tempfile.mkdtemp())),
            patch.object(
                daemon_manager_mod.subprocess, "Popen", return_value=FakeDaemon()
            ) as popen,

3. apps/memos-local-plugin/tests/python/test_daemon_manager.py (L55-L58)

tearDown resets three module-level globals but omits the _lock state reset. More critically, ensure_viewer_daemon is called inside with _lock: — if a test causes an unexpected exception inside the lock, the RLock is still acquired by that thread and will be released when the thread unwinds, so that's fine for RLock. However, the tearDown does not reset _viewer_last_probe_at to a value that will actually expire the TTL cache check: the cache-bypass condition is probe_only=True and _viewer_status == 'running_memos' and now - _viewer_last_probe_at < VIEWER_PROBE_TTL_SEC. Since tearDown sets _viewer_status = None and _viewer_last_probe_at = 0.0, the TTL shortcut is not taken on the next test. But if tests are added in the future that call ensure_viewer_daemon(probe_only=True) and leave _viewer_status = "running_memos" and _viewer_last_probe_at = time.time(), the tearDown's fixed 0.0 won't be safe unless all three fields are always reset together.

A more robust tearDown saves and restores originals rather than hard-coding sentinel values:

def setUp(self):
    self._orig_status = daemon_manager_mod._viewer_status
    self._orig_probe_at = daemon_manager_mod._viewer_last_probe_at
    self._orig_process = daemon_manager_mod._viewer_process

def tearDown(self):
    daemon_manager_mod._viewer_status = self._orig_status
    daemon_manager_mod._viewer_last_probe_at = self._orig_probe_at
    daemon_manager_mod._viewer_process = self._orig_process

4. apps/memos-local-plugin/tests/python/test_daemon_manager.py (L60-L71)

The grace window (50 ms) and poll interval (10 ms) are real wall-clock values; time.time() and time.sleep() are not mocked. On a heavily loaded CI runner, the first loop iteration — which includes at least one _probe_viewer call, one _systemd_unit_state call, a deadline comparison, and a time.sleep(0.01) — could theoretically consume more than 50 ms before the deadline is evaluated for the first time, causing _managed_daemon_gate() to return "refuse" instead of looping as expected. The same applies to test_active_managed_unit_blocks_rescue_spawn.

Mocking time.time with a controlled sequence eliminates the timing dependency:

fake_times = iter([0.0, 0.0, 0.04])  # before deadline on first two checks
with patch("daemon_manager.time.time", side_effect=fake_times), ...

5. apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py (L444-L446)

Misleading fallback: at this point state can legitimately be None. _systemd_unit_state returns None when systemctl is missing, the systemd bus is unreachable, or the 2s call times out — and that None reaches here directly. Substituting "active/activating" therefore logs that the unit left a known active state, when the actual cause is that systemd became unqueryable (and the docstring documents that None means "undetectable"). Since this log line is the operator's only signal for why a rescue-spawn resumed mid-restart, use a distinct string such as "undetectable" so the two causes stay distinguishable.

💡 Suggested Change

Before:

                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",
                unit,
                state or "active/activating",

After:

                "MemOS: managed bridge unit %s is no longer %s — proceeding with rescue-spawn",
                unit,
                state or "undetectable",

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Environment preparation failed before any gating tests executed. Failed scopes: memos_local_plugin
Branch: fix/2467-defer-rescue-spawn-to-managed-unit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants