Skip to content

Answer new messages on sessions whose sidecar toolbox has crashed - #54

Open
josephschorr wants to merge 3 commits into
mainfrom
wake-past-failed-sidecar
Open

josephschorr wants to merge 3 commits into
mainfrom
wake-past-failed-sidecar

Conversation

@josephschorr

@josephschorr josephschorr commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

A parked session whose separately-running sidecar toolbox had crashed could not be woken: a new user message was never answered, and the session's leftover pods were never cleaned up.

Now a crashed sidecar toolbox no longer holds the session back. The new message wakes the session, the agent starts without that toolbox, and it tells the user the toolbox is unavailable. If the toolbox recovers, the agent picks it up. A toolbox that is still starting up no longer delays the wake either. Once the conversation goes quiet, the session sleeps and archives as usual.

A toolbox that never becomes ready for any other reason — it cannot be scheduled, it was evicted, it keeps failing its readiness check — is now given five minutes and then treated the same way as a crashed one, instead of holding the session indefinitely.

An agent restarted after a toolbox failure was already reported is told again that the toolbox is unavailable; the user is not notified a second time.

@vercel

vercel Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
openagentprimitives Ignored Ignored Preview Nov 16, 2026 5:38pm UTC

Request Review

@samkim samkim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR lets a woken (parked, Idle) session proceed past a separate-pod sidecar that has crashed or cannot pull its image: the wake is consumed, a runner is started, the failure stays on status, and the runner now boots without the sidecar instead of polling 120s and failing the session. The controller change does what it says for the terminal-failure case.

Verdict: ready after the inline fixes. No blocking defect was confirmed. What is left is one plausible gap in how a woken runner tells the model about an already-reported crash, missing test coverage, and two comment/readability nits.

Issues, highest first:

  • Test coverage gap (minor): the new runner boot branches have no test, the controller tests do not assert the 30s requeue or the 2s hold on the pass after the wake, and no bronzethread bundle covers a post-wake sidecar crash. See inline.
  • A woken runner does not tell the model the toolbox is gone if the crash was already reported before the session slept (minor, plausible). The new boot path relies on the refresher, but the refresher suppresses already-reported failures across runner restarts and advisories are not kept in history. See inline at internal/cmd/runner/main.go:1414.
  • Redundant condition at pkg/controllers/agentsession/controller.go:2817 and a now-inaccurate comment at controller.go:3306 (nits). See inline.

Questions for the author:

  1. A separate-pod sidecar that is neither Ready nor in a recognized terminal Waiting state (pod evicted so phase is Failed, exited 0, Unschedulable, or running but failing readiness) still sets sidecarPodNotReady and holds the runner every 2s with no deadline, and the pod is never recreated because Create returns AlreadyExists. This is the same on main and on first boot, so the PR does not cause it, but after this PR the stuck session sits in Pending with the wake already consumed. Should such pods be classified as terminal failures (so they fall through like a crash loop) or bounded by a deadline like detectorReadyDeadline? Both are valid and it is your call.
  2. Has mage test:integration and mage test:e2e been run? The new tests need envtest, and the repo's ship gate requires all three suites; please quote the results in the PR.

Not anchored to a diff line: none beyond the question above (controller.go:2710 is the sidecarPodNotReady assignment it refers to).

This review was drafted by an automated pipeline and checked against the code before submission.

Comment thread internal/cmd/runner/main.go Outdated
"sidecar", rt.Name)
continue
}
if rt.RunMode == "separate-pod" && rt.PodFailure != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] New runner boot branches for a terminally failed separate-pod sidecar have no test

Problem: run() now skips a separate-pod sidecar with PodFailure != nil (here) and breaks out of the IP poll when a failure is recorded mid-poll (about line 1443, with the continue at about line 1449). Nothing in internal/cmd/runner/*_test.go exercises either branch, and the existing never-ready-with-no-failure path that still ends in failSession(SidecarBootFailed) is also unpinned. A regression would silently turn a recoverable sidecar crash back into a failed session.
Fix: extract the per-sidecar decision into a small pure function in internal/cmd/runner (inputs: the ResolvedSidecarToolbox and whether a probe URL resolved; output: probe, skip, poll, or boot-fail) and call it from the loop at main.go:1396-1460, keeping behavior identical. Do not otherwise restructure run().
Test: add a table-driven test, for example TestSidecarBootDecision in internal/cmd/runner/sidecar_boot_test.go, with cases: separate-pod with PodFailure and no IP gives skip; separate-pod with IP and a stale PodFailure gives probe; separate-pod with no IP and no failure gives poll; in-pod with no port gives boot-fail; AwaitingSecret gives skip. Use testify require/assert and subtest names of the form "condition: expected outcome". Run go test -run TestSidecarBootDecision ./internal/cmd/runner/.
Constraints: CLAUDE.md "Test conventions" (table-driven for 4+ shared-shape cases, subtest names state condition plus outcome).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extracted the per-sidecar branch into sidecarBootDecision (internal/cmd/runner/sidecar_boot.go) with a table test; the poll loop re-evaluates it on each refreshed snapshot. One small difference: a sidecar that flips back to awaiting its secret mid-poll is now skipped rather than polled to the timeout.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: sidecarBootDecision (internal/cmd/runner/sidecar_boot.go:35-48) is used by the boot loop and re-evaluated each poll; TestSidecarBootDecision covers the requested cases and passes. The mid-poll AwaitingSecret skip is fine.

Comment thread internal/cmd/runner/main.go Outdated
// (a crash loop, an unpullable image). It will not become Ready,
// so polling for its IP would only end in failing the whole
// session. Boot without it: the tool refresher reports the
// failure to the agent and user at the top of the first turn, and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] After a wake, the refresher does not report a sidecar failure that an earlier runner already reported

Problem: this comment (and the PR description) relies on the tool refresher to tell the agent the toolbox is unavailable on the first turn. But newSidecarToolRefresher seeds its reported set from the sidecar-failure-reported annotation (internal/cmd/runner/sidecar_refresh.go:239-246), so a crash reported during an earlier turn is skipped by terminallyFailedSidecars in the woken runner. The advisory is transient (pkg/agent/runner/loop_replay.go:176-184) and is not in the replayed history. Result: a session that crashed its sidecar, slept, and was woken boots without the toolbox and the model gets no [system] notice explaining why those tools are missing. Before this PR this path was unreachable, because the runner failed at the 120s IP poll.
Fix: keep the annotation suppressing the user-facing notify (so the channel is not spammed on every restart), but always emit sidecarFailureAdvisory(rt) on the first refresher pass of a process for every sidecar that currently has PodFailure != nil. One way: in the !seeded block, collect the currently failed sidecars and append their advisories to res without calling notify. Alternatively, correct this comment and the PR description if the omission is intended.
Test: in internal/cmd/runner/sidecar_refresh_test.go, add a case where the AgentSession already carries the annotation with a fingerprint matching a resolved sidecar's PodFailure: assert the first call returns one advisory and notify is not called, and a second call returns no advisory. Run go test -run TestSidecarToolRefresher ./internal/cmd/runner/ (check the actual test name first).
Constraints: CLAUDE.md "Never silently drop errors" (the agent should know why a toolbox is unavailable).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. The refresher's first pass in a process now re-advises the agent for any sidecar whose current failure was already reported, without calling notify again. Covered by TestSidecarToolRefresher_RestartedRunnerReAdvisesAgentWithoutReNotifying.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: sidecar_refresh.go:247-281 appends advisories for already-notified failed sidecars on the first pass without calling notify; TestSidecarToolRefresher_RestartedRunnerReAdvisesAgentWithoutReNotifying passes.

// crash is set, otherwise left still starting), parks it Idle with the runner
// pod gone, stamps a wake as a new user message would, and reconciles. It
// returns the session and whether a runner pod exists afterwards.
func wakeWithSeparatePodSidecar(t *testing.T, name string, crash bool) (spiceboxv1alpha1.AgentSession, bool) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Wake tests do not assert the requeue cadence or the post-wake hold

Problem: wakeWithSeparatePodSidecar drives the reconciler with reconcileToWork (line 63), which discards every ctrl.Result. The PR's claims that a terminally failed sidecar requeues at sidecarTerminalRequeue (30s), that a merely-starting sidecar holds the pass after the wake at 2s, and that the wake is consumed exactly once are therefore unasserted. TestReconcileSecretGatedSidecar_StartingSidecarDoesNotBlockWake also only checks the runner is absent after four blind passes.
Fix: in the helper, do the wake reconcile with r.Reconcile directly and return its ctrl.Result. Then assert RequeueAfter == 1*time.Second for the wake pass. Follow with one more reconcile and assert RequeueAfter == 2*time.Second with no runner pod for the starting case, and RequeueAfter == 30*time.Second with a runner pod for the crash case.
Test: extend TestReconcileSecretGatedSidecar_CrashLoopDoesNotBlockWake and TestReconcileSecretGatedSidecar_StartingSidecarDoesNotBlockWake in this file, and add a case where status.resolvedSidecarToolboxes[0].podFailure is cleared after the pod goes Ready and the requeue drops to zero. Also assert LastWakeAt is unchanged after the second pass (wake consumed once). Run go test -run 'TestReconcileSecretGatedSidecar' ./pkg/controllers/agentsession/ with the suite's envtest setup (check how testenv.Shared is gated before choosing build tags).
Constraints: CLAUDE.md ship gate: also run mage test:integration and mage test:e2e. A bronzethread bundle for the post-wake crash is worth considering if a crash-looping pod can be simulated in the harness; if not, say why in the PR.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wake tests now reconcile pass by pass and assert the 1s wake requeue, the 2s hold for a starting sidecar, the 30s re-check for a crashed one, and that LastWakeAt is unchanged afterwards; a new test covers recovery clearing the failure and the requeue. No bronzethread bundle: the e2e harness has no kubelet and its in-process runner factory bypasses the runner's boot loop, so a bundle could not exercise a crashed sidecar pod.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: the helper returns each pass's ctrl.Result; tests assert 1s, 2s, 30s, unchanged LastWakeAt, and recovery clearing the failure. The no-bundle reason (no kubelet in the e2e harness) is accepted.

// suppresses the idle sweep, the session would never sleep or archive
// either. FALL THROUGH, so the wake is consumed and the runner is (re)created;
// just re-check the pod at a relaxed cadence.
if sidecarTerminalFailure && (sidecarRequeue == 0 || sidecarTerminalRequeue < sidecarRequeue) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Redundant comparison in the requeue assignment

Problem: at this point sidecarRequeue is either 0 or 5 * time.Second (only the AwaitingSecret branch sets it), so sidecarTerminalRequeue < sidecarRequeue (30s < 5s) is never true and the condition reduces to sidecarRequeue == 0. The result is correct (the 5s AwaitingSecret value wins), but the second clause suggests a general min that this code never needs.
Fix: use sidecarRequeue = min(sidecarRequeue, sidecarTerminalRequeue) guarded for zero, or simply if sidecarTerminalFailure && sidecarRequeue == 0 { sidecarRequeue = sidecarTerminalRequeue } with a comment that the 5s AwaitingSecret value wins when both apply.
Test: covered by the extended wake tests above; no new test required. Run go build ./pkg/controllers/agentsession/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified to sidecarRequeue == 0, with a comment that the AwaitingSecret interval wins when both apply.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: controller.go:2819 is now sidecarTerminalFailure && sidecarRequeue == 0, with the comment.

// is already Running by now, so this blocks nothing.
return ctrl.Result{RequeueAfter: sidecarSecretRequeue}, r.applyStatus(ctx, &sess)
// so the sidecar pod is created as soon as the runner produces it, or while a
// sidecar sits terminally failed. The runner is already Running by now, so

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Comment claims the runner is already Running at the final return

Problem: "The runner is already Running by now, so this blocks nothing" is no longer accurate. A post-wake pass with a terminally failed sidecar now falls through to runner creation and can reach this return with the runner still Pending or Creating.
Fix: reword to say the requeue does not hold the runner, because the sidecar step no longer returns early for AwaitingSecret or terminally failed sidecars, rather than asserting the runner phase.
Test: none (comment only).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded: the requeue does not hold the runner because the sidecar step no longer returns early for those cases.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: the comment near controller.go:3307 no longer claims the runner phase.

@josephschorr

Copy link
Copy Markdown
Member Author

On the two questions in the review:

  1. A separate-pod sidecar that is not Ready and not in a recognized terminal state now has a five-minute deadline, measured from when it last stopped being ready (or from pod creation if it never was). Past it the pod is recorded as failed with reason NotReadyDeadlineExceeded and follows the same path as a crash loop. Covered by TestSidecarPodReadyDeadlineFailure and TestReconcileSecretGatedSidecar_NotReadyPastDeadlineStopsHoldingWokenRunner.
  2. Local gate: mage test:integration passed. mage test:unit and mage test:e2e did not come back clean on a heavily loaded machine (load average 16–36): two pkg/web/webui/chat tests, TestPassthroughSelfService, TestChatResume_NoDuplicateDrainedMessage, and the bronzethread package hitting its 25-minute timeout with no bundle failing. Each passes when re-run in isolation, including the full bronzethread and steelthread packages. I could not get a single clean full run on this box, so CI is the reference.

@samkim samkim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 (commit a60f71b, one commit since round 1, 581b2a4). The delta extracts the runner's per-sidecar boot decision into sidecarBootDecision, makes a woken runner re-advise the agent about an already-reported sidecar failure, tightens the wake tests to assert requeue cadence, and adds a 5-minute deadline for a separate-pod sidecar that stays not-Ready.

Verdict: ready after the inline fixes. No blocker; both remaining items are minor.

Earlier issues: 5 of 5 verified. The runner boot branches are now unit-tested, the woken runner re-advises without re-notifying, the wake tests assert 1s/2s/30s and that LastWakeAt is not rewritten, and the requeue condition and comment are fixed. CI on a60f71b passed unit, integration and e2e.

Open, highest impact first:

  1. Minor: the new 5-minute deadline fails a first-boot session whose sidecar pod is Pending on a capacity shortfall. #44, already on main but not yet merged into this branch, holds bundle and detector pods for up to an hour in that same case, so a burst would fail at the sidecar stage only. See inline on sidecarpod.go:393.
  2. Minor, plausible: a crash-looping sidecar pod older than 5m can flap between CrashLoopBackOff and NotReadyDeadlineExceeded, and each flap can send the user another unavailable-toolbox notice. See inline on sidecarpod.go:393.

Questions for the author:

  1. Should a separate-pod sidecar pod that is Pending on a capacity shortfall get the same hold as bundles and detectors in #44 (needs a rebase on main to reuse capacityShortfall), or is failing it at 5m the intended behavior? The inline comment lists both options and does not pick one.

Nothing confirmed was left without a diff anchor.

This review was drafted by an automated pipeline and checked against the code before submission.

// Not Ready and not recognizably terminal — evicted, unschedulable, running
// but failing readiness. Waiting is right only for so long: past the
// deadline it is reported exactly like a terminal failure.
return sidecarPodState{Failure: sidecarPodReadyDeadlineFailure(&live, r.now())}, nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] The 5-minute not-Ready deadline fails a first-boot session on a capacity shortfall

Problem: this return classifies every not-Ready pod without a terminal Waiting reason by sidecarPodReadyDeadlineFailure. A pod that is Pending with PodScheduled=False and a scheduler message such as "0/3 nodes: 3 Insufficient cpu" has no PodReady condition, so the clock runs from CreationTimestamp (about line 420). On first boot the caller (controller.go:2683, !podAlreadyStarted) turns any Failure into markBootFailed(SidecarBootFailed). In a burst of sessions whose class has a separate-pod sidecar, the sidecar pods that cannot be scheduled for 5m fail their sessions. origin/main (#44, pkg/controllers/agentsession/capacity_hold.go, not yet on this branch) deliberately holds booting bundle and detector pods up to 1h in exactly that case, because failing a burst at the readiness deadline was the bug it fixed. After merge the sidecar pod is the one stage that still fails at 5m. Before this PR the pod held with no deadline, so the hard failure is new.
Verify: this is a design choice, so do not implement until the author answers question 1 in the review body. The options are (a) when the session is booting and the pod's PodScheduled=False message passes capacityShortfall (from #44, git show origin/main:pkg/controllers/agentsession/capacity_hold.go), extend the deadline using effectiveCapacityHoldTimeout() instead of sidecarReadyDeadline, which needs this branch rebased on main; (b) keep 5m for sidecars and document that in the sidecarReadyDeadline comment and the PR description.
Test: if (a), add a case to TestSidecarPodReadyDeadlineFailure in pkg/controllers/agentsession/sidecarpod_test.go for an "Insufficient cpu" Unschedulable pod 6m old that returns nil for a booting session. Run go test -run TestSidecarPodReadyDeadlineFailure ./pkg/controllers/agentsession/. If (b), the existing case at sidecarpod_test.go:580 already pins it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went with (a): rebased on main, and a booting session's sidecar pod that is Pending on a capacity shortfall now gets effectiveCapacityHoldTimeout() instead of 5m (sidecarReadyDeadlineFor). Once the pod is scheduled, the kubelet's PodReady transition restarts the clock, so no release-adjusted start is needed. Covered by TestSidecarReadyDeadlineFor.

// Not Ready and not recognizably terminal — evicted, unschedulable, running
// but failing readiness. Waiting is right only for so long: past the
// deadline it is reported exactly like a terminal failure.
return sidecarPodState{Failure: sidecarPodReadyDeadlineFailure(&live, r.now())}, nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor, plausible] A crash-looping pod older than 5m can flap between CrashLoopBackOff and NotReadyDeadlineExceeded

Problem: sidecarPodFailure (above) only matches containers whose State is Waiting with a terminal reason (pkg/platform/podstatus/podstatus.go:36-45). While a crash-looping container is briefly Running or Terminated between restarts there is no Waiting reason, so the pod falls through to the deadline classifier. A never-ready pod's PodReady LastTransitionTime does not move, so after 5m the pass records NotReadyDeadlineExceeded ... (phase Running). The next back-off flips the record back to CrashLoopBackOff. The controller owns Pods (controller.go:1139), so each container-state change triggers a reconcile that can land in the window. The runner fingerprint is Reason|Message (internal/cmd/runner/sidecar_refresh.go:152), so each flap can send the user a second ToolboxUnavailable notice with a different reason.
Verify: reproduce with a table case for a pod with Phase Running, a ContainerStatus whose State.Running is set and whose LastTerminationState.Terminated is set (exit 1), a PodReady=False condition with LastTransitionTime 10m ago. Confirm reconcileSidecarPod yields NotReadyDeadlineExceeded for it. If so, fix: in sidecarPodReadyDeadlineFailure, return nil (still waiting) when any container status has RestartCount > 0 or LastTerminationState.Terminated != nil, or classify that case with sidecarPodFailure's CrashLoopBackOff reason, so a restarting container keeps one stable reason. Pick whichever keeps the failure Message stable across the two states.
Test: add a case to TestSidecarPodReadyDeadlineFailure in pkg/controllers/agentsession/sidecarpod_test.go named like "crash-looping container between restarts, past the deadline: no deadline failure (or stable reason)". Run go test -run TestSidecarPodReadyDeadlineFailure ./pkg/controllers/agentsession/.
Constraints: the Message must stay free of elapsed time and per-pass values (see the function comment): it is the runner's dedup fingerprint.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. A container whose last failed termination finished during the current not-Ready stretch is now classified as CrashLoopBackOff with the same record the back-off state produces (sidecarPodRestartingFailure), so the fingerprint holds across restarts. The empty-log fallback also now names the exit code instead of the kubelet back-off text, whose delay changed on every restart. Covered by TestSidecarPodRestartingFailure.

Comment thread internal/cmd/runner/main.go Outdated
"sidecar", rt.Name)
continue
}
if rt.RunMode == "separate-pod" && rt.PodFailure != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: sidecarBootDecision (internal/cmd/runner/sidecar_boot.go:35-48) is used by the boot loop and re-evaluated each poll; TestSidecarBootDecision covers the requested cases and passes. The mid-poll AwaitingSecret skip is fine.

Comment thread internal/cmd/runner/main.go Outdated
// (a crash loop, an unpullable image). It will not become Ready,
// so polling for its IP would only end in failing the whole
// session. Boot without it: the tool refresher reports the
// failure to the agent and user at the top of the first turn, and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: sidecar_refresh.go:247-281 appends advisories for already-notified failed sidecars on the first pass without calling notify; TestSidecarToolRefresher_RestartedRunnerReAdvisesAgentWithoutReNotifying passes.

// crash is set, otherwise left still starting), parks it Idle with the runner
// pod gone, stamps a wake as a new user message would, and reconciles. It
// returns the session and whether a runner pod exists afterwards.
func wakeWithSeparatePodSidecar(t *testing.T, name string, crash bool) (spiceboxv1alpha1.AgentSession, bool) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: the helper returns each pass's ctrl.Result; tests assert 1s, 2s, 30s, unchanged LastWakeAt, and recovery clearing the failure. The no-bundle reason (no kubelet in the e2e harness) is accepted.

// suppresses the idle sweep, the session would never sleep or archive
// either. FALL THROUGH, so the wake is consumed and the runner is (re)created;
// just re-check the pod at a relaxed cadence.
if sidecarTerminalFailure && (sidecarRequeue == 0 || sidecarTerminalRequeue < sidecarRequeue) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: controller.go:2819 is now sidecarTerminalFailure && sidecarRequeue == 0, with the comment.

// is already Running by now, so this blocks nothing.
return ctrl.Result{RequeueAfter: sidecarSecretRequeue}, r.applyStatus(ctx, &sess)
// so the sidecar pod is created as soon as the runner produces it, or while a
// sidecar sits terminally failed. The runner is already Running by now, so

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at a60f71b: the comment near controller.go:3307 no longer claims the runner phase.

A parked session whose separate-pod sidecar was crash-looping never
consumed a wake: the sidecar step returned early on every pass, ahead of
the wake handling. Because a pending wake also suppresses the idle sweep,
the session never slept or archived either, so the user's message went
unanswered and its pods were never reaped.

A sidecar recorded as terminally failed no longer gates the reconcile;
the failure is already on status for the runner to surface, so the pass
falls through, the wake is consumed and the runner is created. A sidecar
that is merely starting still holds the runner, but no longer holds a
pending wake.

The runner now boots without a separate-pod sidecar recorded as
terminally failed instead of polling for its IP and then failing the
session; the tool refresher reports the failure and picks the sidecar up
if it recovers.
… woken agent about a toolbox that already failed

A separate-pod sidecar that is neither Ready nor in a recognized terminal
container state (evicted, unschedulable, failing readiness) is now treated as
failed once it has been not-Ready for five minutes, measured from when it last
stopped being ready. It then follows the same path as a crash loop.

A runner started after the failure was already reported to the user now tells
the agent once more, since advisories are not replayed from history; the user
is not notified again.

The runner's per-sidecar boot decision is extracted into a pure function with
a table test, and the wake tests assert each pass's requeue interval.
…one failure record across crash-loop restarts
@josephschorr
josephschorr force-pushed the wake-past-failed-sidecar branch from a60f71b to f6d09d6 Compare November 16, 2026 17:38

This branch was successfully deployed

1 active (outdated) deployment
Preview — a60f71b8 Deployed Nov 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants