Skip to content

[Bug] Host rescue-spawns its own bridge daemon during a systemd-daemon restart window — permanent port war, health stays green (2.0.20) #2467

Open

Description

@kiwipaulrob

Environment

  • memos-local-plugin 2.0.20 (shared-bridge runtime; systemd unit memos-bridge.service runs dist/bridge.cjs --daemon)
  • Single Hermes instance; hosts = hermes-gateway (many sessions), hermes-dashboard, TUI — which normally spawn bridge.mjs --no-viewer clients against :18800

Summary
When the systemd-managed daemon is briefly unavailable (e.g. during a restart), a session host can rescue-spawn its own full daemon (bridge.mjs --agent=hermes --home=<plugin> --daemon). If that process binds :18800 before the systemd daemon finishes its ~8–10 s boot, the unit can never bind again: it crash-loops on port :18800 busy while the rogue process keeps serving. Critically, /api/v1/health stays green (answered by the rogue), so health-based monitoring reports the system as healthy. Unattended, this persists indefinitely — only stopping the spawning host clears it (killing the rogue alone respawns in ~3 s via the host's keepalive).

Observed (2026-10-07, this deployment)

  • 11:22:57 — watchdog restarted the unit after a legitimate down verdict (LLM-lane outage); recovery failed at 11:23:16.
  • 11:23:31 — the gateway process (the rogue's parent PID, cgroup hermes-gateway.service) spawned bridge.mjs ... --daemon; it bound :18800 in ~1 s while the systemd replacement was still initializing (it reached pipeline.ready at +8 s and only then saw the port busy).
  • Result: 413 unit start attempts in ~24 h, a restart attempt every ~26 s, port :18800 busy ×3,363 — while health answered ok:true throughout, and the watchdog's post-restart rogue sweep lost the race to the ~3 s respawn keepalive (it logged "recovery OK" every cycle).

Impact

  • 400+ restarts/day of churn (CPU, logs, journal); the process actually serving sessions runs outside systemd supervision; every loop tick interrupts in-flight recovery/backfill work; health + monitoring are actively misleading; clearing it requires stopping the spawning host first (operator footgun).

Related

Suggested directions (not prescriptive)

  • A host should not rescue-spawn a --daemon when a managed daemon exists or is mid-start: check systemctl is-active memos-bridge / the singleton pidfile / probe :18800 with a short retry window before spawning.
  • If rescue-spawning is intentional, make the rescued daemon yield to an in-progress systemd start (bind-retry with backoff) instead of racing for first-bind.
  • Optional: when the port is already owned by another bridge.* process, exit with a distinctive log line the supervisors can watch for (the unit currently logs port busy ... retrying (n/10) and then exits silently-ish at default log levels).

Verified local workaround (now standard ops here): stop unit → stop the spawning host (gateway) → sweep rogues → systemctl reset-failed memos-bridge; systemctl start memos-bridge (binds in ~2 s) → start host → verify the listener's cgroup is memos-bridge.service and no --daemon child reappears. Detection added locally: a 30-min guard alerts when the :18800 listener's cgroup ≠ memos-bridge.service or NRestarts climbs between runs.

Activity

  1. akushonkamen commented on Oct 12, 2026

    @akushonkamen

    Confirmed the root cause and picking this up (fix in progress).

    Root cause (verified on dev-v2.0.36, apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py):

    • _probe_viewer() (:276) can only classify the port as free/running_memos/occupied/unknown. During a systemd restart window (:18800 not yet bound) the connection-refused → bind-probe path reports free, and ensure_viewer_daemon() (:359) treats that as spawn clearance — there is no intermediate "managed daemon is starting" state.
    • It then spawns the exact rogue command line from this report: _bridge_command(daemon=True) (:167-184, --daemon at :178-179) via subprocess.Popen(..., start_new_session=True) (:441-448). The detached daemon binds :18800 in ~1s, the systemd replacement loses first-bind, and bridge.cts (:461-498) exits after its EADDRINUSE retries → permanent crash-loop, with the rogue answering /api/v1/health (green). Killing the rogue doesn't help because the host keepalive/reconnect path re-enters ensure_viewer_daemon (~3s), and killExistingBridge (bridge.cts:137-162) makes the war bidirectional via daemon/bridge.pid.

    Fix direction: gate the rescue-spawn path on a supervisor signal — when systemctl is-active memos-bridge.service (env-overridable, degrades to a no-op where systemd is absent) reports active/activating while the port probes free, wait a bounded window for the managed daemon to take the port instead of racing it; only spawn if the unit leaves active/activating. Non-systemd hosts keep the current rescue behavior.

    Note: I'm an AI agent (Claude Code, GLM) working via the akushonkamen account; the change will come as a PR against dev-v2.0.36. Since I can't run systemd locally, verification will be unit-level (unittest, mirroring the existing ViewerDaemonTests stubs) plus a disclosure in the PR.

  2. akushonkamen commented on Oct 12, 2026

    @akushonkamen

    PR with the fix: #2483 (against dev-v2.0.36).

    The rescue-spawn path in ensure_viewer_daemon() is now gated on systemctl is-active MEMOS_BRIDGE_SYSTEMD_UNIT (default memos-bridge.service): while the unit is active/activating and :18800 probes free, it waits up to 20s for the managed daemon to take the port instead of racing it; any check failure degrades to the current behavior. Verification: pre-fix tests.python.test_daemon_manager = 3 failures + 6 errors (the failures are the broken popen.assert_not_called()), post-fix 12/12 OK, regression suite 144/144 OK, ruff 0.11.8 check + format clean.

  3. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  4. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Nov 13, 2026
  5. kiwipaulrob commented on Nov 13, 2026

    @kiwipaulrob
    ContributorAuthor

    Still relevant — please don't let this auto-close. The rescue-spawned bridge daemon during the systemd-daemon restart window leaves a permanent port conflict while health checks stay green, which makes it particularly nasty to notice. Happy to retest against a newer build or provide more detail if it helps triage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:pluginOpenClaw & Hermesstatus:needs-triageNeeds initial triage | 需要初步判断 & 问题复现status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions