Repository navigation
Answer new messages on sessions whose sidecar toolbox has crashed - #54
josephschorr wants to merge 3 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
samkim
left a comment
There was a problem hiding this comment.
This PR lets a woken (parked, Idle) session proceed past a separate-pod sidecar that has crashed or cannot pull its image: the wake is consumed, a runner is started, the failure stays on status, and the runner now boots without the sidecar instead of polling 120s and failing the session. The controller change does what it says for the terminal-failure case.
Verdict: ready after the inline fixes. No blocking defect was confirmed. What is left is one plausible gap in how a woken runner tells the model about an already-reported crash, missing test coverage, and two comment/readability nits.
Issues, highest first:
- Test coverage gap (minor): the new runner boot branches have no test, the controller tests do not assert the 30s requeue or the 2s hold on the pass after the wake, and no bronzethread bundle covers a post-wake sidecar crash. See inline.
- A woken runner does not tell the model the toolbox is gone if the crash was already reported before the session slept (minor, plausible). The new boot path relies on the refresher, but the refresher suppresses already-reported failures across runner restarts and advisories are not kept in history. See inline at
internal/cmd/runner/main.go:1414. - Redundant condition at
pkg/controllers/agentsession/controller.go:2817and a now-inaccurate comment atcontroller.go:3306(nits). See inline.
Questions for the author:
- A separate-pod sidecar that is neither Ready nor in a recognized terminal Waiting state (pod evicted so phase is Failed, exited 0, Unschedulable, or running but failing readiness) still sets
sidecarPodNotReadyand holds the runner every 2s with no deadline, and the pod is never recreated because Create returns AlreadyExists. This is the same onmainand on first boot, so the PR does not cause it, but after this PR the stuck session sits in Pending with the wake already consumed. Should such pods be classified as terminal failures (so they fall through like a crash loop) or bounded by a deadline likedetectorReadyDeadline? Both are valid and it is your call. - Has
mage test:integrationandmage test:e2ebeen run? The new tests need envtest, and the repo's ship gate requires all three suites; please quote the results in the PR.
Not anchored to a diff line: none beyond the question above (controller.go:2710 is the sidecarPodNotReady assignment it refers to).
This review was drafted by an automated pipeline and checked against the code before submission.
| "sidecar", rt.Name) | ||
| continue | ||
| } | ||
| if rt.RunMode == "separate-pod" && rt.PodFailure != nil { |
There was a problem hiding this comment.
[minor] New runner boot branches for a terminally failed separate-pod sidecar have no test
Problem: run() now skips a separate-pod sidecar with PodFailure != nil (here) and breaks out of the IP poll when a failure is recorded mid-poll (about line 1443, with the continue at about line 1449). Nothing in internal/cmd/runner/*_test.go exercises either branch, and the existing never-ready-with-no-failure path that still ends in failSession(SidecarBootFailed) is also unpinned. A regression would silently turn a recoverable sidecar crash back into a failed session.
Fix: extract the per-sidecar decision into a small pure function in internal/cmd/runner (inputs: the ResolvedSidecarToolbox and whether a probe URL resolved; output: probe, skip, poll, or boot-fail) and call it from the loop at main.go:1396-1460, keeping behavior identical. Do not otherwise restructure run().
Test: add a table-driven test, for example TestSidecarBootDecision in internal/cmd/runner/sidecar_boot_test.go, with cases: separate-pod with PodFailure and no IP gives skip; separate-pod with IP and a stale PodFailure gives probe; separate-pod with no IP and no failure gives poll; in-pod with no port gives boot-fail; AwaitingSecret gives skip. Use testify require/assert and subtest names of the form "condition: expected outcome". Run go test -run TestSidecarBootDecision ./internal/cmd/runner/.
Constraints: CLAUDE.md "Test conventions" (table-driven for 4+ shared-shape cases, subtest names state condition plus outcome).
There was a problem hiding this comment.
Extracted the per-sidecar branch into sidecarBootDecision (internal/cmd/runner/sidecar_boot.go) with a table test; the poll loop re-evaluates it on each refreshed snapshot. One small difference: a sidecar that flips back to awaiting its secret mid-poll is now skipped rather than polled to the timeout.
There was a problem hiding this comment.
Verified at a60f71b: sidecarBootDecision (internal/cmd/runner/sidecar_boot.go:35-48) is used by the boot loop and re-evaluated each poll; TestSidecarBootDecision covers the requested cases and passes. The mid-poll AwaitingSecret skip is fine.
| // (a crash loop, an unpullable image). It will not become Ready, | ||
| // so polling for its IP would only end in failing the whole | ||
| // session. Boot without it: the tool refresher reports the | ||
| // failure to the agent and user at the top of the first turn, and |
There was a problem hiding this comment.
[minor] After a wake, the refresher does not report a sidecar failure that an earlier runner already reported
Problem: this comment (and the PR description) relies on the tool refresher to tell the agent the toolbox is unavailable on the first turn. But newSidecarToolRefresher seeds its reported set from the sidecar-failure-reported annotation (internal/cmd/runner/sidecar_refresh.go:239-246), so a crash reported during an earlier turn is skipped by terminallyFailedSidecars in the woken runner. The advisory is transient (pkg/agent/runner/loop_replay.go:176-184) and is not in the replayed history. Result: a session that crashed its sidecar, slept, and was woken boots without the toolbox and the model gets no [system] notice explaining why those tools are missing. Before this PR this path was unreachable, because the runner failed at the 120s IP poll.
Fix: keep the annotation suppressing the user-facing notify (so the channel is not spammed on every restart), but always emit sidecarFailureAdvisory(rt) on the first refresher pass of a process for every sidecar that currently has PodFailure != nil. One way: in the !seeded block, collect the currently failed sidecars and append their advisories to res without calling notify. Alternatively, correct this comment and the PR description if the omission is intended.
Test: in internal/cmd/runner/sidecar_refresh_test.go, add a case where the AgentSession already carries the annotation with a fingerprint matching a resolved sidecar's PodFailure: assert the first call returns one advisory and notify is not called, and a second call returns no advisory. Run go test -run TestSidecarToolRefresher ./internal/cmd/runner/ (check the actual test name first).
Constraints: CLAUDE.md "Never silently drop errors" (the agent should know why a toolbox is unavailable).
There was a problem hiding this comment.
Confirmed. The refresher's first pass in a process now re-advises the agent for any sidecar whose current failure was already reported, without calling notify again. Covered by TestSidecarToolRefresher_RestartedRunnerReAdvisesAgentWithoutReNotifying.
There was a problem hiding this comment.
Verified at a60f71b: sidecar_refresh.go:247-281 appends advisories for already-notified failed sidecars on the first pass without calling notify; TestSidecarToolRefresher_RestartedRunnerReAdvisesAgentWithoutReNotifying passes.
| // crash is set, otherwise left still starting), parks it Idle with the runner | ||
| // pod gone, stamps a wake as a new user message would, and reconciles. It | ||
| // returns the session and whether a runner pod exists afterwards. | ||
| func wakeWithSeparatePodSidecar(t *testing.T, name string, crash bool) (spiceboxv1alpha1.AgentSession, bool) { |
There was a problem hiding this comment.
[minor] Wake tests do not assert the requeue cadence or the post-wake hold
Problem: wakeWithSeparatePodSidecar drives the reconciler with reconcileToWork (line 63), which discards every ctrl.Result. The PR's claims that a terminally failed sidecar requeues at sidecarTerminalRequeue (30s), that a merely-starting sidecar holds the pass after the wake at 2s, and that the wake is consumed exactly once are therefore unasserted. TestReconcileSecretGatedSidecar_StartingSidecarDoesNotBlockWake also only checks the runner is absent after four blind passes.
Fix: in the helper, do the wake reconcile with r.Reconcile directly and return its ctrl.Result. Then assert RequeueAfter == 1*time.Second for the wake pass. Follow with one more reconcile and assert RequeueAfter == 2*time.Second with no runner pod for the starting case, and RequeueAfter == 30*time.Second with a runner pod for the crash case.
Test: extend TestReconcileSecretGatedSidecar_CrashLoopDoesNotBlockWake and TestReconcileSecretGatedSidecar_StartingSidecarDoesNotBlockWake in this file, and add a case where status.resolvedSidecarToolboxes[0].podFailure is cleared after the pod goes Ready and the requeue drops to zero. Also assert LastWakeAt is unchanged after the second pass (wake consumed once). Run go test -run 'TestReconcileSecretGatedSidecar' ./pkg/controllers/agentsession/ with the suite's envtest setup (check how testenv.Shared is gated before choosing build tags).
Constraints: CLAUDE.md ship gate: also run mage test:integration and mage test:e2e. A bronzethread bundle for the post-wake crash is worth considering if a crash-looping pod can be simulated in the harness; if not, say why in the PR.
There was a problem hiding this comment.
The wake tests now reconcile pass by pass and assert the 1s wake requeue, the 2s hold for a starting sidecar, the 30s re-check for a crashed one, and that LastWakeAt is unchanged afterwards; a new test covers recovery clearing the failure and the requeue. No bronzethread bundle: the e2e harness has no kubelet and its in-process runner factory bypasses the runner's boot loop, so a bundle could not exercise a crashed sidecar pod.
There was a problem hiding this comment.
Verified at a60f71b: the helper returns each pass's ctrl.Result; tests assert 1s, 2s, 30s, unchanged LastWakeAt, and recovery clearing the failure. The no-bundle reason (no kubelet in the e2e harness) is accepted.
| // suppresses the idle sweep, the session would never sleep or archive | ||
| // either. FALL THROUGH, so the wake is consumed and the runner is (re)created; | ||
| // just re-check the pod at a relaxed cadence. | ||
| if sidecarTerminalFailure && (sidecarRequeue == 0 || sidecarTerminalRequeue < sidecarRequeue) { |
There was a problem hiding this comment.
[nit] Redundant comparison in the requeue assignment
Problem: at this point sidecarRequeue is either 0 or 5 * time.Second (only the AwaitingSecret branch sets it), so sidecarTerminalRequeue < sidecarRequeue (30s < 5s) is never true and the condition reduces to sidecarRequeue == 0. The result is correct (the 5s AwaitingSecret value wins), but the second clause suggests a general min that this code never needs.
Fix: use sidecarRequeue = min(sidecarRequeue, sidecarTerminalRequeue) guarded for zero, or simply if sidecarTerminalFailure && sidecarRequeue == 0 { sidecarRequeue = sidecarTerminalRequeue } with a comment that the 5s AwaitingSecret value wins when both apply.
Test: covered by the extended wake tests above; no new test required. Run go build ./pkg/controllers/agentsession/.
There was a problem hiding this comment.
Simplified to sidecarRequeue == 0, with a comment that the AwaitingSecret interval wins when both apply.
There was a problem hiding this comment.
Verified at a60f71b: controller.go:2819 is now sidecarTerminalFailure && sidecarRequeue == 0, with the comment.
| // is already Running by now, so this blocks nothing. | ||
| return ctrl.Result{RequeueAfter: sidecarSecretRequeue}, r.applyStatus(ctx, &sess) | ||
| // so the sidecar pod is created as soon as the runner produces it, or while a | ||
| // sidecar sits terminally failed. The runner is already Running by now, so |
There was a problem hiding this comment.
[nit] Comment claims the runner is already Running at the final return
Problem: "The runner is already Running by now, so this blocks nothing" is no longer accurate. A post-wake pass with a terminally failed sidecar now falls through to runner creation and can reach this return with the runner still Pending or Creating.
Fix: reword to say the requeue does not hold the runner, because the sidecar step no longer returns early for AwaitingSecret or terminally failed sidecars, rather than asserting the runner phase.
Test: none (comment only).
There was a problem hiding this comment.
Reworded: the requeue does not hold the runner because the sidecar step no longer returns early for those cases.
There was a problem hiding this comment.
Verified at a60f71b: the comment near controller.go:3307 no longer claims the runner phase.
|
On the two questions in the review:
|
samkim
left a comment
There was a problem hiding this comment.
Round 2 (commit a60f71b, one commit since round 1, 581b2a4). The delta extracts the runner's per-sidecar boot decision into sidecarBootDecision, makes a woken runner re-advise the agent about an already-reported sidecar failure, tightens the wake tests to assert requeue cadence, and adds a 5-minute deadline for a separate-pod sidecar that stays not-Ready.
Verdict: ready after the inline fixes. No blocker; both remaining items are minor.
Earlier issues: 5 of 5 verified. The runner boot branches are now unit-tested, the woken runner re-advises without re-notifying, the wake tests assert 1s/2s/30s and that LastWakeAt is not rewritten, and the requeue condition and comment are fixed. CI on a60f71b passed unit, integration and e2e.
Open, highest impact first:
- Minor: the new 5-minute deadline fails a first-boot session whose sidecar pod is Pending on a capacity shortfall. #44, already on main but not yet merged into this branch, holds bundle and detector pods for up to an hour in that same case, so a burst would fail at the sidecar stage only. See inline on
sidecarpod.go:393. - Minor, plausible: a crash-looping sidecar pod older than 5m can flap between
CrashLoopBackOffandNotReadyDeadlineExceeded, and each flap can send the user another unavailable-toolbox notice. See inline onsidecarpod.go:393.
Questions for the author:
- Should a separate-pod sidecar pod that is Pending on a capacity shortfall get the same hold as bundles and detectors in #44 (needs a rebase on main to reuse
capacityShortfall), or is failing it at 5m the intended behavior? The inline comment lists both options and does not pick one.
Nothing confirmed was left without a diff anchor.
This review was drafted by an automated pipeline and checked against the code before submission.
| // Not Ready and not recognizably terminal — evicted, unschedulable, running | ||
| // but failing readiness. Waiting is right only for so long: past the | ||
| // deadline it is reported exactly like a terminal failure. | ||
| return sidecarPodState{Failure: sidecarPodReadyDeadlineFailure(&live, r.now())}, nil |
There was a problem hiding this comment.
[minor] The 5-minute not-Ready deadline fails a first-boot session on a capacity shortfall
Problem: this return classifies every not-Ready pod without a terminal Waiting reason by sidecarPodReadyDeadlineFailure. A pod that is Pending with PodScheduled=False and a scheduler message such as "0/3 nodes: 3 Insufficient cpu" has no PodReady condition, so the clock runs from CreationTimestamp (about line 420). On first boot the caller (controller.go:2683, !podAlreadyStarted) turns any Failure into markBootFailed(SidecarBootFailed). In a burst of sessions whose class has a separate-pod sidecar, the sidecar pods that cannot be scheduled for 5m fail their sessions. origin/main (#44, pkg/controllers/agentsession/capacity_hold.go, not yet on this branch) deliberately holds booting bundle and detector pods up to 1h in exactly that case, because failing a burst at the readiness deadline was the bug it fixed. After merge the sidecar pod is the one stage that still fails at 5m. Before this PR the pod held with no deadline, so the hard failure is new.
Verify: this is a design choice, so do not implement until the author answers question 1 in the review body. The options are (a) when the session is booting and the pod's PodScheduled=False message passes capacityShortfall (from #44, git show origin/main:pkg/controllers/agentsession/capacity_hold.go), extend the deadline using effectiveCapacityHoldTimeout() instead of sidecarReadyDeadline, which needs this branch rebased on main; (b) keep 5m for sidecars and document that in the sidecarReadyDeadline comment and the PR description.
Test: if (a), add a case to TestSidecarPodReadyDeadlineFailure in pkg/controllers/agentsession/sidecarpod_test.go for an "Insufficient cpu" Unschedulable pod 6m old that returns nil for a booting session. Run go test -run TestSidecarPodReadyDeadlineFailure ./pkg/controllers/agentsession/. If (b), the existing case at sidecarpod_test.go:580 already pins it.
There was a problem hiding this comment.
Went with (a): rebased on main, and a booting session's sidecar pod that is Pending on a capacity shortfall now gets effectiveCapacityHoldTimeout() instead of 5m (sidecarReadyDeadlineFor). Once the pod is scheduled, the kubelet's PodReady transition restarts the clock, so no release-adjusted start is needed. Covered by TestSidecarReadyDeadlineFor.
| // Not Ready and not recognizably terminal — evicted, unschedulable, running | ||
| // but failing readiness. Waiting is right only for so long: past the | ||
| // deadline it is reported exactly like a terminal failure. | ||
| return sidecarPodState{Failure: sidecarPodReadyDeadlineFailure(&live, r.now())}, nil |
There was a problem hiding this comment.
[minor, plausible] A crash-looping pod older than 5m can flap between CrashLoopBackOff and NotReadyDeadlineExceeded
Problem: sidecarPodFailure (above) only matches containers whose State is Waiting with a terminal reason (pkg/platform/podstatus/podstatus.go:36-45). While a crash-looping container is briefly Running or Terminated between restarts there is no Waiting reason, so the pod falls through to the deadline classifier. A never-ready pod's PodReady LastTransitionTime does not move, so after 5m the pass records NotReadyDeadlineExceeded ... (phase Running). The next back-off flips the record back to CrashLoopBackOff. The controller owns Pods (controller.go:1139), so each container-state change triggers a reconcile that can land in the window. The runner fingerprint is Reason|Message (internal/cmd/runner/sidecar_refresh.go:152), so each flap can send the user a second ToolboxUnavailable notice with a different reason.
Verify: reproduce with a table case for a pod with Phase Running, a ContainerStatus whose State.Running is set and whose LastTerminationState.Terminated is set (exit 1), a PodReady=False condition with LastTransitionTime 10m ago. Confirm reconcileSidecarPod yields NotReadyDeadlineExceeded for it. If so, fix: in sidecarPodReadyDeadlineFailure, return nil (still waiting) when any container status has RestartCount > 0 or LastTerminationState.Terminated != nil, or classify that case with sidecarPodFailure's CrashLoopBackOff reason, so a restarting container keeps one stable reason. Pick whichever keeps the failure Message stable across the two states.
Test: add a case to TestSidecarPodReadyDeadlineFailure in pkg/controllers/agentsession/sidecarpod_test.go named like "crash-looping container between restarts, past the deadline: no deadline failure (or stable reason)". Run go test -run TestSidecarPodReadyDeadlineFailure ./pkg/controllers/agentsession/.
Constraints: the Message must stay free of elapsed time and per-pass values (see the function comment): it is the runner's dedup fingerprint.
There was a problem hiding this comment.
Confirmed. A container whose last failed termination finished during the current not-Ready stretch is now classified as CrashLoopBackOff with the same record the back-off state produces (sidecarPodRestartingFailure), so the fingerprint holds across restarts. The empty-log fallback also now names the exit code instead of the kubelet back-off text, whose delay changed on every restart. Covered by TestSidecarPodRestartingFailure.
| "sidecar", rt.Name) | ||
| continue | ||
| } | ||
| if rt.RunMode == "separate-pod" && rt.PodFailure != nil { |
There was a problem hiding this comment.
Verified at a60f71b: sidecarBootDecision (internal/cmd/runner/sidecar_boot.go:35-48) is used by the boot loop and re-evaluated each poll; TestSidecarBootDecision covers the requested cases and passes. The mid-poll AwaitingSecret skip is fine.
| // (a crash loop, an unpullable image). It will not become Ready, | ||
| // so polling for its IP would only end in failing the whole | ||
| // session. Boot without it: the tool refresher reports the | ||
| // failure to the agent and user at the top of the first turn, and |
There was a problem hiding this comment.
Verified at a60f71b: sidecar_refresh.go:247-281 appends advisories for already-notified failed sidecars on the first pass without calling notify; TestSidecarToolRefresher_RestartedRunnerReAdvisesAgentWithoutReNotifying passes.
| // crash is set, otherwise left still starting), parks it Idle with the runner | ||
| // pod gone, stamps a wake as a new user message would, and reconciles. It | ||
| // returns the session and whether a runner pod exists afterwards. | ||
| func wakeWithSeparatePodSidecar(t *testing.T, name string, crash bool) (spiceboxv1alpha1.AgentSession, bool) { |
There was a problem hiding this comment.
Verified at a60f71b: the helper returns each pass's ctrl.Result; tests assert 1s, 2s, 30s, unchanged LastWakeAt, and recovery clearing the failure. The no-bundle reason (no kubelet in the e2e harness) is accepted.
| // suppresses the idle sweep, the session would never sleep or archive | ||
| // either. FALL THROUGH, so the wake is consumed and the runner is (re)created; | ||
| // just re-check the pod at a relaxed cadence. | ||
| if sidecarTerminalFailure && (sidecarRequeue == 0 || sidecarTerminalRequeue < sidecarRequeue) { |
There was a problem hiding this comment.
Verified at a60f71b: controller.go:2819 is now sidecarTerminalFailure && sidecarRequeue == 0, with the comment.
| // is already Running by now, so this blocks nothing. | ||
| return ctrl.Result{RequeueAfter: sidecarSecretRequeue}, r.applyStatus(ctx, &sess) | ||
| // so the sidecar pod is created as soon as the runner produces it, or while a | ||
| // sidecar sits terminally failed. The runner is already Running by now, so |
There was a problem hiding this comment.
Verified at a60f71b: the comment near controller.go:3307 no longer claims the runner phase.
A parked session whose separate-pod sidecar was crash-looping never consumed a wake: the sidecar step returned early on every pass, ahead of the wake handling. Because a pending wake also suppresses the idle sweep, the session never slept or archived either, so the user's message went unanswered and its pods were never reaped. A sidecar recorded as terminally failed no longer gates the reconcile; the failure is already on status for the runner to surface, so the pass falls through, the wake is consumed and the runner is created. A sidecar that is merely starting still holds the runner, but no longer holds a pending wake. The runner now boots without a separate-pod sidecar recorded as terminally failed instead of polling for its IP and then failing the session; the tool refresher reports the failure and picks the sidecar up if it recovers.
… woken agent about a toolbox that already failed A separate-pod sidecar that is neither Ready nor in a recognized terminal container state (evicted, unschedulable, failing readiness) is now treated as failed once it has been not-Ready for five minutes, measured from when it last stopped being ready. It then follows the same path as a crash loop. A runner started after the failure was already reported to the user now tells the agent once more, since advisories are not replayed from history; the user is not notified again. The runner's per-sidecar boot decision is extracted into a pure function with a table test, and the wake tests assert each pass's requeue interval.
…one failure record across crash-loop restarts
a60f71b to
f6d09d6
Compare
A parked session whose separately-running sidecar toolbox had crashed could not be woken: a new user message was never answered, and the session's leftover pods were never cleaned up.
Now a crashed sidecar toolbox no longer holds the session back. The new message wakes the session, the agent starts without that toolbox, and it tells the user the toolbox is unavailable. If the toolbox recovers, the agent picks it up. A toolbox that is still starting up no longer delays the wake either. Once the conversation goes quiet, the session sleeps and archives as usual.
A toolbox that never becomes ready for any other reason — it cannot be scheduled, it was evicted, it keeps failing its readiness check — is now given five minutes and then treated the same way as a crashed one, instead of holding the session indefinitely.
An agent restarted after a toolbox failure was already reported is told again that the toolbox is unavailable; the user is not notified a second time.