Repository navigation
Description
Environment
- memos-local-plugin 2.0.20 (shared-bridge runtime; systemd unit
memos-bridge.servicerunsdist/bridge.cjs --daemon) - Single Hermes instance; hosts = hermes-gateway (many sessions), hermes-dashboard, TUI — which normally spawn
bridge.mjs --no-viewerclients against :18800
Summary
When the systemd-managed daemon is briefly unavailable (e.g. during a restart), a session host can rescue-spawn its own full daemon (bridge.mjs --agent=hermes --home=<plugin> --daemon). If that process binds :18800 before the systemd daemon finishes its ~8–10 s boot, the unit can never bind again: it crash-loops on port :18800 busy while the rogue process keeps serving. Critically, /api/v1/health stays green (answered by the rogue), so health-based monitoring reports the system as healthy. Unattended, this persists indefinitely — only stopping the spawning host clears it (killing the rogue alone respawns in ~3 s via the host's keepalive).
Observed (2026-10-07, this deployment)
- 11:22:57 — watchdog restarted the unit after a legitimate
downverdict (LLM-lane outage); recovery failed at 11:23:16. - 11:23:31 — the gateway process (the rogue's parent PID, cgroup
hermes-gateway.service) spawnedbridge.mjs ... --daemon; it bound :18800 in ~1 s while the systemd replacement was still initializing (it reachedpipeline.readyat +8 s and only then saw the port busy). - Result: 413 unit start attempts in ~24 h, a restart attempt every ~26 s,
port :18800 busy×3,363 — while health answeredok:truethroughout, and the watchdog's post-restart rogue sweep lost the race to the ~3 s respawn keepalive (it logged "recovery OK" every cycle).
Impact
- 400+ restarts/day of churn (CPU, logs, journal); the process actually serving sessions runs outside systemd supervision; every loop tick interrupts in-flight recovery/backfill work; health + monitoring are actively misleading; clearing it requires stopping the spawning host first (operator footgun).
Related
- fix: initialize() spawns new headless bridge without closing old one — process leak #1927 (closed — old
initialize()spawn leak), fix(hermes-adapter): widen _ACTIVE_CLIENTS key with owner_id to prevent intra-process bridge fight #2292 (closed — superseded by shared-bridge mode; though 2.0.20 still exhibits host daemon-spawn behavior under this trigger), [Bug] dirty-closed rescan misses freshly-marked episodes for a full cursor lap; interrupted drains lose phase-2 accounting #2459 (dirty-rescan cursor accounting).
Suggested directions (not prescriptive)
- A host should not rescue-spawn a
--daemonwhen a managed daemon exists or is mid-start: checksystemctl is-active memos-bridge/ the singleton pidfile / probe :18800 with a short retry window before spawning. - If rescue-spawning is intentional, make the rescued daemon yield to an in-progress systemd start (bind-retry with backoff) instead of racing for first-bind.
- Optional: when the port is already owned by another
bridge.*process, exit with a distinctive log line the supervisors can watch for (the unit currently logsport busy ... retrying (n/10)and then exits silently-ish at default log levels).
Verified local workaround (now standard ops here): stop unit → stop the spawning host (gateway) → sweep rogues → systemctl reset-failed memos-bridge; systemctl start memos-bridge (binds in ~2 s) → start host → verify the listener's cgroup is memos-bridge.service and no --daemon child reappears. Detection added locally: a 30-min guard alerts when the :18800 listener's cgroup ≠ memos-bridge.service or NRestarts climbs between runs.
Activity
akushonkamen commentedon Oct 12, 2026
Confirmed the root cause and picking this up (fix in progress).
Root cause (verified on
dev-v2.0.36,apps/memos-local-plugin/adapters/hermes/memos_provider/daemon_manager.py):_probe_viewer()(:276) can only classify the port asfree/running_memos/occupied/unknown. During a systemd restart window (:18800 not yet bound) the connection-refused → bind-probe path reportsfree, andensure_viewer_daemon()(:359) treats that as spawn clearance — there is no intermediate "managed daemon is starting" state._bridge_command(daemon=True)(:167-184,--daemonat :178-179) viasubprocess.Popen(..., start_new_session=True)(:441-448). The detached daemon binds :18800 in ~1s, the systemd replacement loses first-bind, andbridge.cts(:461-498) exits after its EADDRINUSE retries → permanent crash-loop, with the rogue answering/api/v1/health(green). Killing the rogue doesn't help because the host keepalive/reconnect path re-entersensure_viewer_daemon(~3s), andkillExistingBridge(bridge.cts:137-162) makes the war bidirectional viadaemon/bridge.pid.Fix direction: gate the rescue-spawn path on a supervisor signal — when
systemctl is-active memos-bridge.service(env-overridable, degrades to a no-op where systemd is absent) reportsactive/activatingwhile the port probes free, wait a bounded window for the managed daemon to take the port instead of racing it; only spawn if the unit leaves active/activating. Non-systemd hosts keep the current rescue behavior.Note: I'm an AI agent (Claude Code, GLM) working via the akushonkamen account; the change will come as a PR against
dev-v2.0.36. Since I can't run systemd locally, verification will be unit-level (unittest, mirroring the existingViewerDaemonTestsstubs) plus a disclosure in the PR.akushonkamen commentedon Oct 12, 2026
PR with the fix: #2483 (against
dev-v2.0.36).The rescue-spawn path in
ensure_viewer_daemon()is now gated onsystemctl is-active MEMOS_BRIDGE_SYSTEMD_UNIT(defaultmemos-bridge.service): while the unit is active/activating and :18800 probes free, it waits up to 20s for the managed daemon to take the port instead of racing it; any check failure degrades to the current behavior. Verification: pre-fixtests.python.test_daemon_manager= 3 failures + 6 errors (the failures are the brokenpopen.assert_not_called()), post-fix 12/12 OK, regression suite 144/144 OK, ruff 0.11.8 check + format clean.github-actions commentedon Nov 13, 2026
This issue has been automatically marked as stale due to inactivity.
kiwipaulrob commentedon Nov 13, 2026
Still relevant — please don't let this auto-close. The rescue-spawned bridge daemon during the systemd-daemon restart window leaves a permanent port conflict while health checks stay green, which makes it particularly nasty to notice. Happy to retest against a newer build or provide more detail if it helps triage.