fix: kill a live-but-unreachable broker when replacing it - #50
Merged
Conversation
Investigated upstream openai#580 ("give the app-server broker an idle timeout, and kill the one it replaces") for portability to this fork. Two independent defects were claimed; verified each on macOS against this fork's own broker-lifecycle.mjs/broker-controller.mjs rather than trusting the diff. Defect 1 (no idle timeout): NOT reproduced as a gap here. This fork's in-process BrokerController (lib/broker-controller.mjs) already ships a causal, test-driven idle-timeout mechanism (CODEX_COMPANION_BROKER_IDLE_TIMEOUT_MS, default 15 min) predating this investigation, architected around an injected clock rather than the broker polling its own socket count on a wall-clock interval. No port needed. Defect 2 (stale-replacement path never kills the process it replaces): partially already fixed here (478be42 already tree-kills a just-spawned broker that never becomes ready). The remaining gap is narrower than upstream's diff implies: loadReusableBrokerSessionUnlocked only tree-kills the recorded pid when the readiness probe confirms the endpoint is live, out of deliberate caution about killing a recycled pid. When the endpoint is unreachable (e.g. the broker's unix socket file is swept by external tmp cleanup while the process itself is still running -- reproduced locally by deleting the socket file out from under a spawned broker) that pid was never trusted, so the process is orphaned for its full idle-timeout window and invisible to session-lifecycle-hook.mjs once the state file is overwritten by the replacement broker. Fix: trust the pid for tree-kill when either the probe succeeded OR the pid is alive right now (isProcessAlive). A live pid recorded in the state file has been the same process continuously since it was written -- there is no recycling window to guard against -- so killing it is as safe as the existing endpoint-confirmed path. A dead/absent pid still skips the kill, preserving the original PID-reuse guard for the case it actually protects. Did not port upstream's widened stale-probe retry (150ms -> 150ms + 1s): this fork already serializes ensureBrokerSession via a file lock (broker-lock.mjs), so concurrent-load probe races that upstream's retry targets don't reproduce here (confirmed empirically). Did not touch upstream#577 (Windows Git Bash/taskkill fix) -- irrelevant on macOS. Added a regression test that spawns a real broker, deletes its socket file, and asserts the replacement path still terminates the orphaned process (using a long idle timeout so the assertion can't pass "by accident" via the broker's own self-shutdown). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HbkiKZR4w8hZUmNNTdb6kB
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigated upstream openai/codex-plugin-cc#580 ("give the app-server broker an idle timeout, and kill the one it replaces") for portability here. Verified each claimed defect empirically on macOS against this fork's own code before porting anything — did not trust the upstream diff on its own.
Defect 1 (no idle timeout) — not ported, already covered. This fork's in-process
BrokerController(lib/broker-controller.mjs) already ships a causal, test-driven idle-timeout mechanism (CODEX_COMPANION_BROKER_IDLE_TIMEOUT_MS, default 15 min), architected around an injected clock rather than polling socket count on a wall-clock interval. Predates this investigation. No gap to close.Defect 2 (stale-replacement path never kills the process it replaces) — partially already fixed, one gap closed here.
478be42already tree-kills a just-spawned broker that never becomes ready. The remaining gap:loadReusableBrokerSessionUnlockedonly tree-kills the recorded pid when the readiness probe confirms the endpoint is live, out of deliberate caution about killing a recycled pid (see the comment it replaces). When the endpoint is unreachable but the process is still alive — e.g. the broker's unix socket file gets swept by external tmp cleanup while the process itself keeps running — that pid was never trusted, so the process is orphaned for its full idle-timeout window and invisible tosession-lifecycle-hook.mjsonce the state file is overwritten by the replacement broker.Reproduced locally: spawn a real broker via
ensureBrokerSession, delete its socket file out from under it, callensureBrokerSessionagain — the original process stays alive (confirmed viaisProcessAlive) until it eventually self-exits on its own idle timer, not because anything killed it.Fix: trust the pid for tree-kill when either the readiness probe succeeded OR the pid is alive right now (
isProcessAlive). A live pid recorded in the state file has been the same process continuously since it was written — there's no recycling window to guard against — so killing it is exactly as safe as the existing endpoint-confirmed path. A dead/absent pid still skips the kill, preserving the original PID-reuse guard for the case it actually protects (confirmed the existing "stale reachable brokers preserved" test, which uses a fabricated dead pid999999, still passes unchanged).Not ported: upstream's widened stale-probe retry (150ms → 150ms + 1s). This fork already serializes
ensureBrokerSessionvia a file lock (broker-lock.mjs), so the concurrent-load probe race that retry targets doesn't reproduce here — confirmed empirically with concurrentensureBrokerSessioncalls under contention; the lock keeps them from stepping on each other regardless of probe timing.Not touched: upstream's stacked-on #577 (Windows Git Bash/taskkill fix) — irrelevant on macOS, out of scope here.
Test plan
replacing a live broker whose endpoint became unreachable still kills its processintests/broker-lifecycle.test.mjs— spawns a real broker, deletes its socket file, asserts the replacement path still terminates the orphaned process. Uses a 10-minute idle timeout so the assertion can't pass "by accident" via the broker's own self-shutdown (the default test env's 2s idle timeout would otherwise mask a still-broken kill path).npm test— 190/190 passing, both in this worktree (branched fresh offmain) and the working branch it was authored on.npm run buildpasses for the changed files. Note:tsc -p tsconfig.app-server.jsoncurrently fails on unrelated pre-existing type errors inplugins/codex/scripts/lib/codex.mjs(lines 1209) that exist onmainindependent of this PR — reproduced by stashing this change entirely and re-running build on a cleanmaincheckout. Flagging separately, not fixed here (out of scope).No version bump (targeted fix, no upstream-influenced behavior on the always-covered path).