Skip to content

fix: stop losing turn-lifecycle events (dangling sub-agents, stuck 'thinking') - #273

Merged
saucam merged 2 commits into
mainfrom
fix/subagent-lifecycle-leak
Aug 2, 2026
Merged

fix: stop losing turn-lifecycle events (dangling sub-agents, stuck 'thinking')#273
saucam merged 2 commits into
mainfrom
fix/subagent-lifecycle-leak

Conversation

@saucam

@saucam saucam commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Sub-agent counts climbed turn after turn and never came down, and sessions sat spinning at "thinking" after the model had visibly answered. Both trace to the same place: the provider's event channel loses turn-lifecycle events, silently.

This PR fixes the causes and keeps reconciliation as a backstop.


Cause 1 — mid-turn accounting counted pushes that produce no turn_done

#pendingMidTurnCount exists so #consumeEvents can absorb the intermediate turn_done the SDK emits when a mid-turn push starts a new query. It was incremented for every mid-turn push — including "later", which merges into the running turn and starts no query at all (ClaudeProvider: shouldQuery = priority !== "later").

session.send exposes priority on the wire, so any client sending "later" while the session was working left the counter one too high. The consumer then hit its terminal turn_done, treated it as an intermediate boundary, decremented, re-asserted "thinking" (session.ts:3379) and continued — waiting for a turn_done that was never going to be emitted.

That is the stuck spinner. It also stranded that turn's sub-agents and tools, because the loop never exited, so neither the boundary flush nor the consumer's finally ever ran. Recovery came only from the 5-minute stall watchdog or the next send — which is why it looked intermittent.

Now only querying pushes are counted.

Cause 2 — #emit dropped events silently

// before
#emit(event) { try { this.#currentTurnQueue?.push(event); } catch {} }

Three distinct losses in one expression — null queue, closed queue, full queue — indistinguishable and none logged. A lost subagent_stop left its sub-agent dangling with a live delegated ZeroID token; a lost tool_complete stranded status at tool_running. Nothing recorded either, which is why this was invisible for so long.

Now #emit reports the disposition, and id-keyed lifecycle events (subagent_stop, tool_complete) are buffered and replayed into the next turn rather than vanishing — their handlers are idempotent on unknown ids, so a late duplicate costs nothing. turn_done and streamed text are deliberately not carried: a stale turn_done would end the next turn the instant it began. Buffer is bounded (100); the log is deduped per loop generation so a persistent fault reports once, not once per event.

The fourth, invisible loss

After the consumer broke on turn_done, the turn queue stayed open and unread until the next turn replaced it. Late events were accepted into a queue nobody would ever drain — push succeeded, so there was no error and no log to catch. TurnRun.endTurn() (optional, called once from the consumer's finally) closes it, converting that silent case into the observable carryover path.

Cause 3 — the sweep was in the wrong place

#completeActiveTools() has always run in the consumer's finally precisely because provider events can be lost. Sub-agents were simply never added to that same backstop. The sweep now sits beside it, covering every exit path — clean turn_done, error, stall recovery, ownership loss — plus the mid-turn continuation branch, which continues without dispatching to #handleProviderEvent and so was missed entirely by the earlier turn_done-case sweep.

Abort-path sweeps (interrupt(), #teardownProvider()) stay as defence in depth, since those abort the SDK query and the hooks provably never fire.

What was leaking

All bounded by session destroy (deactivateSessionAgent cascades), but real while a session lives:

Session.#subagents one entry per orphan — this drove the wrong count
Session.#subagentRegistrations a retained Promise<void> per orphan
AgentIdentityManager.#agents identityId, wimseUri, token, apiKey — a live credential for a dead sub-agent

The third is the one that matters: revocation is meant to ride the sub-agent's own stop, and instead waited for session teardown.

Verification

src/tests/session-subagent-lifecycle.test.ts — 13 cases. Verified by reverting the source and re-running:

Fail against original main (the sweep work): turn-end sweep · non-accumulation across turns · ZeroID revocation of an orphan · sweep on interrupt · sweep on provider teardown · info_update broadcast.

Fail against the first commit (the root-cause work): the "later" mid-turn push no longer swallowing its terminal turn_donewhich fails by timing out, the bug exactly · sub-agent reconciliation at an absorbed mid-turn boundary · endTurn() closing the turn stream.

Also pinned: "now" pushes still correctly absorb their intermediate boundary (so the fix didn't just disable the mechanism), the normal subagent_stop path still works, orphans are revoked exactly once, and the carryover policy excludes turn_done/text.

  • bun run typecheck clean (root + protocol + core)
  • bun run lint clean, 347 files
  • bun test2203 pass, 19 skip, 0 fail across 152 files

Known gaps

  • The provider's carryover/replay path has no automated integration test — exercising it needs a live SDK query loop. Only the policy (which event types carry) is unit-tested. The Session-side half (endTurn) is covered.
  • A dropped turn_done still hangs the consumer until the stall watchdog. Making the queue close a non-lossy terminal signal would need the provider to distinguish terminal from intermediate turn_done, duplicating a fragile state machine that currently lives only in Session — deliberately not attempted here. With Cause 1 fixed, the reachable path to that hang is closed; what remains is now logged rather than silent.
  • active: boolean on each sub-agent entry is still vestigial (set true, never false). Untouched because subagents is client-visible in the protocol (types.ts:220).

🤖 Generated with Claude Code

The provider's `subagent_stop` event was the ONLY path that removed an
entry from Session's #subagents map. That event originates in the Claude
SDK's SubagentStop hook, which cannot fire once the query is aborted --
and interrupt, setModel, rotate and switchProvider all abort it via
#abortController.abort(). Every such abort therefore orphaned each
in-flight sub-agent permanently, with nothing anywhere reconciling the
map afterwards.

Three consequences:

- subagentSnapshot (feeding /who and toInfo().subagents) only ever grew.
  The displayed sub-agent count climbed turn after turn and never came
  back down, which is how this was noticed.
- #subagents and #subagentRegistrations grew unbounded for the lifetime
  of a long-lived session.
- Worst: each orphan kept a LIVE delegated ZeroID token. Revocation is
  meant to ride the sub-agent's own stop; instead it fell through to
  deactivateSessionAgent's cascade at session destroy, so a dead
  sub-agent's credential stayed valid as long as the session lived. That
  quietly weakens the per-agent revocation guarantee.

Adds #sweepStaleSubagents, which revokes and drops whatever remains. A
Task sub-agent cannot outlive the turn that spawned it, so it runs at
turn_done as the principled backstop, plus on the two abort paths that
leave the session alive: interrupt() and #teardownProvider(). It is
idempotent and returns immediately on an empty map (the common case), and
double-revocation is free because deactivateSubagent no-ops on an id it
already dropped -- so a trailing subagent_stop racing a sweep costs
nothing.

Not swept at destroy(): deactivateSessionAgent already cascades over the
session's sub-agent keys there, and the Session object is discarded.

Tests: 6 of the 8 new cases fail against the pre-fix code, covering the
turn-end sweep, non-accumulation across turns, both abort paths, the
identity revocation, and the info_update broadcast so clients see the
count drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

The sweep added earlier reconciled dangling sub-agents but never asked why
they dangled. Root-causing that turned up two defects, one of which is the
"model answered but the spinner keeps spinning" report.

1. Mid-turn accounting counted pushes that produce no turn_done.

#pendingMidTurnCount exists so #consumeEvents can absorb the intermediate
turn_done the SDK emits when a mid-turn push starts a NEW query. It was
incremented for every mid-turn push, including "later" -- which merges
into the running turn and starts no query at all (ClaudeProvider:
shouldQuery = priority !== "later"). Any client sending with an explicit
"later" priority while the session was working therefore left the counter
one too high, and the consumer swallowed the turn's REAL terminal
turn_done as if it were an intermediate boundary: it re-asserted
"thinking" and continue'd, waiting for a turn_done that would never be
emitted.

That is the stuck spinner. It also stranded the turn's sub-agents and
tools, because the loop never exited and so neither the boundary flush nor
the consumer's finally ever ran. Recovery came only from the 5-minute
stall watchdog or the next send. Now only querying pushes are counted.

2. The provider's event channel dropped events silently.

ClaudeProvider.#emit was `try { queue?.push(e) } catch {}` -- three silent
losses in one expression: a null queue, a closed queue, and a full queue,
all indistinguishable and none logged. A lost subagent_stop left its
sub-agent dangling with a live delegated ZeroID token; a lost
tool_complete stranded the status at tool_running. Nothing said so.

#emit now reports the disposition, and id-keyed lifecycle events
(subagent_stop, tool_complete) are buffered and replayed into the next
turn instead of vanishing -- their handlers are idempotent on unknown ids,
so a late duplicate costs nothing. turn_done and streamed text are
deliberately not carried: a stale turn_done would end the next turn as it
began. The buffer is bounded and the log is deduped per loop generation.

There was also a fourth, invisible loss: after the consumer broke on
turn_done, the turn queue stayed OPEN and unread until the next turn
replaced it, so late events were accepted into a queue nobody would drain
-- push succeeded, so no error, no log. TurnRun.endTurn() (optional, called
once from the consumer's finally) closes it, converting that into the
observable carryover path.

3. Sweep moved to where reconciliation already lives.

#completeActiveTools has always run in the consumer's finally precisely
because provider events can be lost; sub-agents were simply never added to
the same backstop. The sweep now sits beside it, covering every exit path,
plus the mid-turn continuation branch -- which continue's without
dispatching to #handleProviderEvent and so was missed entirely by the
previous turn_done-case sweep. The abort-path sweeps stay as defence in
depth for interrupt/teardown, where hooks provably never fire.

Tests: 5 new cases on top of the existing 8. The three covering these
changes fail against the previous commit -- the stuck-spinner case by
timing out, which is the bug exactly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@saucam saucam changed the title fix: sweep orphaned sub-agents at turn boundaries and on abort paths fix: stop losing turn-lifecycle events (dangling sub-agents, stuck 'thinking') Aug 1, 2026
@saucam
saucam merged commit 925d427 into main Aug 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants