Skip to content

fix: kill a live-but-unreachable broker when replacing it - #50

Merged
axisrow merged 1 commit into
mainfrom
fix/broker-orphan-on-unreachable-endpoint
Aug 3, 2026
Merged

fix: kill a live-but-unreachable broker when replacing it#50
axisrow merged 1 commit into
mainfrom
fix/broker-orphan-on-unreachable-endpoint

Conversation

@axisrow

@axisrow axisrow commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Summary

Investigated upstream openai/codex-plugin-cc#580 ("give the app-server broker an idle timeout, and kill the one it replaces") for portability here. Verified each claimed defect empirically on macOS against this fork's own code before porting anything — did not trust the upstream diff on its own.

Defect 1 (no idle timeout) — not ported, already covered. This fork's in-process BrokerController (lib/broker-controller.mjs) already ships a causal, test-driven idle-timeout mechanism (CODEX_COMPANION_BROKER_IDLE_TIMEOUT_MS, default 15 min), architected around an injected clock rather than polling socket count on a wall-clock interval. Predates this investigation. No gap to close.

Defect 2 (stale-replacement path never kills the process it replaces) — partially already fixed, one gap closed here. 478be42 already tree-kills a just-spawned broker that never becomes ready. The remaining gap: loadReusableBrokerSessionUnlocked only tree-kills the recorded pid when the readiness probe confirms the endpoint is live, out of deliberate caution about killing a recycled pid (see the comment it replaces). When the endpoint is unreachable but the process is still alive — e.g. the broker's unix socket file gets swept by external tmp cleanup while the process itself keeps running — that pid was never trusted, so the process is orphaned for its full idle-timeout window and invisible to session-lifecycle-hook.mjs once the state file is overwritten by the replacement broker.

Reproduced locally: spawn a real broker via ensureBrokerSession, delete its socket file out from under it, call ensureBrokerSession again — the original process stays alive (confirmed via isProcessAlive) until it eventually self-exits on its own idle timer, not because anything killed it.

Fix: trust the pid for tree-kill when either the readiness probe succeeded OR the pid is alive right now (isProcessAlive). A live pid recorded in the state file has been the same process continuously since it was written — there's no recycling window to guard against — so killing it is exactly as safe as the existing endpoint-confirmed path. A dead/absent pid still skips the kill, preserving the original PID-reuse guard for the case it actually protects (confirmed the existing "stale reachable brokers preserved" test, which uses a fabricated dead pid 999999, still passes unchanged).

Not ported: upstream's widened stale-probe retry (150ms → 150ms + 1s). This fork already serializes ensureBrokerSession via a file lock (broker-lock.mjs), so the concurrent-load probe race that retry targets doesn't reproduce here — confirmed empirically with concurrent ensureBrokerSession calls under contention; the lock keeps them from stepping on each other regardless of probe timing.

Not touched: upstream's stacked-on #577 (Windows Git Bash/taskkill fix) — irrelevant on macOS, out of scope here.

Test plan

  • Added replacing a live broker whose endpoint became unreachable still kills its process in tests/broker-lifecycle.test.mjs — spawns a real broker, deletes its socket file, asserts the replacement path still terminates the orphaned process. Uses a 10-minute idle timeout so the assertion can't pass "by accident" via the broker's own self-shutdown (the default test env's 2s idle timeout would otherwise mask a still-broken kill path).
  • Verified the test fails without the fix (times out waiting for the orphan to die) and passes with it.
  • npm test — 190/190 passing, both in this worktree (branched fresh off main) and the working branch it was authored on.
  • npm run build passes for the changed files. Note: tsc -p tsconfig.app-server.json currently fails on unrelated pre-existing type errors in plugins/codex/scripts/lib/codex.mjs (lines 1209) that exist on main independent of this PR — reproduced by stashing this change entirely and re-running build on a clean main checkout. Flagging separately, not fixed here (out of scope).

No version bump (targeted fix, no upstream-influenced behavior on the always-covered path).

Investigated upstream openai#580 ("give the app-server
broker an idle timeout, and kill the one it replaces") for portability
to this fork. Two independent defects were claimed; verified each on
macOS against this fork's own broker-lifecycle.mjs/broker-controller.mjs
rather than trusting the diff.

Defect 1 (no idle timeout): NOT reproduced as a gap here. This fork's
in-process BrokerController (lib/broker-controller.mjs) already ships
a causal, test-driven idle-timeout mechanism (CODEX_COMPANION_BROKER_IDLE_TIMEOUT_MS,
default 15 min) predating this investigation, architected around an
injected clock rather than the broker polling its own socket count on
a wall-clock interval. No port needed.

Defect 2 (stale-replacement path never kills the process it replaces):
partially already fixed here (478be42 already tree-kills a
just-spawned broker that never becomes ready). The remaining gap is
narrower than upstream's diff implies: loadReusableBrokerSessionUnlocked
only tree-kills the recorded pid when the readiness probe confirms the
endpoint is live, out of deliberate caution about killing a recycled
pid. When the endpoint is unreachable (e.g. the broker's unix socket
file is swept by external tmp cleanup while the process itself is
still running -- reproduced locally by deleting the socket file out
from under a spawned broker) that pid was never trusted, so the
process is orphaned for its full idle-timeout window and invisible to
session-lifecycle-hook.mjs once the state file is overwritten by the
replacement broker.

Fix: trust the pid for tree-kill when either the probe succeeded OR
the pid is alive right now (isProcessAlive). A live pid recorded in
the state file has been the same process continuously since it was
written -- there is no recycling window to guard against -- so killing
it is as safe as the existing endpoint-confirmed path. A dead/absent
pid still skips the kill, preserving the original PID-reuse guard for
the case it actually protects.

Did not port upstream's widened stale-probe retry (150ms -> 150ms + 1s):
this fork already serializes ensureBrokerSession via a file lock
(broker-lock.mjs), so concurrent-load probe races that upstream's
retry targets don't reproduce here (confirmed empirically). Did not
touch upstream#577 (Windows Git Bash/taskkill fix) -- irrelevant on
macOS.

Added a regression test that spawns a real broker, deletes its socket
file, and asserts the replacement path still terminates the orphaned
process (using a long idle timeout so the assertion can't pass "by
accident" via the broker's own self-shutdown).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HbkiKZR4w8hZUmNNTdb6kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant