Repository navigation
Admit sessions to the cluster as whole sets, in a fair queue - #64
Open
josephschorr wants to merge 33 commits into
Open
josephschorr wants to merge 33 commits into
josephschorr wants to merge 33 commits into
Conversation
Pure package describing a session's pod demand, when it joined its class's line, and the order sessions are admitted in. Adds the Admitted condition and its reasons.
Add a node capacity model with all-or-nothing whole-set reservation, and export schedfit.EffectiveRequests plus a pod-slot count on Headroom.
Admit whole pod sets from per-class lines, picking the line with the lowest dominant share first; a head that does not fit blocks only its own line.
DemandFor renders every pod a session will create (runner, one sandbox per tool bundle, content-guard detectors, separate-pod sidecars) through the reconciler's own builders and prices each with schedfit.EffectiveRequests, without writing anything. EffectiveRequests now defaults an unset request to the container's limit, as the API server does, so a built-but-not-created sandbox (limits only) is no longer priced at zero. Detector resolution moves into pure helpers shared with the reconciler, and toolchain resolution accepts a client.Reader.
Identity precedence and the tier-resolved sandbox backend now live in desiredBundleSession, called by both the reconciler's bundle loop and DemandFor, so the two cannot drift.
With Reconciler.AdmissionEnabled, a booting session that is not yet admitted creates none of its pods: it waits in line under CapacityHold=True/WaitingForAdmission and fails with HoldExpired once it has waited past the capacity-hold timeout, measured from WaitingSince. Settings resolution and the pre-start gates still run so the admitter can size the session's demand. Reclaiming a session's pods records Admitted=False/Reclaimed: in the SleptAt write on idle sleep, and after the terminal reap. The zero value keeps today's ungated behaviour.
The admitter rebuilds every admission line from AgentSession status on each pass, prices each waiting session's whole pod set, and admits in fair-share order against node headroom. It writes Admitted=True and the waiting sessions' line positions (only when they change) to status, so a restarted operator resumes the same line. The operator gains --session-admission (default true) and --admission-min-warm, and passes the admission switch to the AgentSession reconciler. CapacityModel now reports cluster totals for the dominant-share calculation.
The uncached session list was cluster-wide, which 403s under the per-namespace RBAC --watch-namespaces exists for and reaches sessions in namespaces the operator does not own. With a watch set, the pass now lists one namespace at a time, using the same set the manager cache is scoped to.
An admitted booting session whose bundle pod stays unschedulable on a capacity shortfall for two minutes reaps its pods and goes back to its line as Admitted=False/PlacementMismatch, written after the reap succeeds. JoinedAt is unchanged, so it keeps its place; the waiting timeout restarts at the release. Only with admission on; the pod-level capacity hold is unchanged otherwise.
Bundles can now open several named sessions across more than one fake Channel, scope transcript steps to one session, hold a turn until other sessions reach a state, and run with whole-set admission against a fixed capacity. The driver watches every session for the whole run and checks that none holds part of a pod set, that admission never exceeds the capacity, and the bundle's ordering claims. admission-burst runs three reviewer sessions and one session of a second agent on room for two: the reviewer line is FIFO and the second agent is admitted ahead of the burst's tail.
OpeningText for the unnamed thread returned the first user turn's text, which scoped every unscoped step to the parent session and stranded a delegated child's requests.
A placement-mismatch release now fails the session, as the waiting timeout does, once it has been in line longer than the capacity-hold timeout since it joined: a stall the admitter's model cannot see would otherwise re-admit and release forever, since each release restarts both the waiting clock and the bundle deadline. The release also clears the bundle stage's CapacityHold, whose give-up time described the pods it just reaped. status.admission is admitter-owned; the runner-forgery webhook now refuses a runner write to it.
Pure planning: oldest-idle warm sessions of the head's line, freed on a cloned model until the head fits.
After deciding, the admitter plans an early reclaim for each line whose head does not fit: warm Idle sessions of that line idle at least --admission-min-warm, oldest first, until the head fits. It asks for each through the existing sleep path with a reclaim-requested annotation. The AgentSession reconciler treats that annotation as "sleep now" (still behind the SessionHold guard), removes it once the session is slept, and clears it as stale when the session left Idle.
A reclaimed session's pods stay in the capacity snapshot until they are gone, and each status write and pod event in between wakes the admitter, which then saw the head still not fit and reclaimed the next-oldest warm session too. Before a line plans, free the still-bound pods of its in-flight reclaims (requested but not yet slept, or slept and Reclaimed with pods still terminating) and keep them out of the candidates. Log, rather than discard, a head that does not fit after its planned reclaims are freed.
AgentClass gains spec.maxConcurrentSessions. The admitter reads it for every line with a waiting session and hands it to Decide as that line's cap; delegated children are neither counted nor blocked. A class whose cap cannot be read holds its non-child sessions back for the pass. The admission-child-at-cap bundle delegates into a capped line that is already full and asserts the delegate tool result is the child's answer.
The capacity-hold notice now appends the session's place in its class's admission line when status.admission records one.
A session's wait was measured from creation, so one that cleared a start approval, identity choice or credential gate more than the capacity-hold timeout after creation failed on its first gated pass. The admitter also admitted sessions that had a Pending phase and resolved settings but had not reached the gate (parked, fatal settings, pin-blocked), reserving capacity for them. The gate now stamps Admitted=False/Queued at arrival; the waiting timeout and give-up notice run from it, and the admitter considers only queued sessions. WriteOwned persists a deliberately re-stamped transition time on a same-status reason change, so a released session arriving again is queued from now.
JoinedAt fell back to creation once a cold-woken session had been admitted (or when its wake and the admission transition shared a second), so a misfit release measured its wait from creation and failed any channel session older than the capacity-hold timeout. JoinedAt is now the last wake if set, else creation.
Admission defaults on, so at upgrade every session already booting, running or warm idle had no Admitted condition. A booting one was gated with its pods still up, and the admitter priced its whole set on top of headroom that already subtracted those pods; a running one was never counted toward its class's cap or share. A non-terminal session with no Admitted condition whose status records pods, and that is not slept or held, is pre-admission: the gate lets it through without queueing it, and the admitter adopts it (Admitted=True/Adopted) once it sees one of its pods, so it counts as a holder. A pre-admission session that sleeps is released like an admitted one, so its next wake queues.
Three cases let the model admit sets the scheduler could not place, so the misfit release reaped, re-admitted and re-noticed the same session every few minutes: tainted pools counted as room, a namespace-scoped operator not seeing other namespaces' pods, and unknown capacity admitting everything. The node model now records each node's NoSchedule/NoExecute taints and places a pod only where its tolerations (carried on PodDemand from the built spec) tolerate them. A namespace-scoped admitter treats capacity as unknown. Sessions admitted under unknown capacity get the reason AdmittedUnknownCapacity, which the misfit release skips, leaving them to the pod-level hold. Unknown capacity is logged on transition only.
Warm idle holders count toward an AgentClass's session cap, but early reclaim only planned for heads that did not fit, so a line held at its cap by idle sessions waited out their full sleep grace. Decide now reports which lines' heads wait on the cap. For such a head the admitter asks the line's oldest non-child warm idle session past the minimum warm time to sleep early, one per head, and not while an earlier request for the line is still in flight.
A held session's pods were reaped but it stayed Admitted=True, so it kept a full reservation (all of it unplaced) and a cap slot for as long as the hold stood. The Held projection's status write now records Admitted=False/Reclaimed; on release the session rejoins the line through the gate.
A sandbox pod adopted from a SandboxWarmPool keeps its pool-minted name, so matching demand entries to pods by name never found it: the holder's whole bundle demand stayed unplaced and was reserved on top of a snapshot that already counted the pod, for the session's lifetime, and an early reclaim could not credit it. A bundle's demand entry now carries its SpiceboxSession, and the admitter finds its pod by that SpiceboxSession's label (which adoption injects); runner, detector and sidecar entries still match by their deterministic names.
Reserve a holder's unplaced pods one at a time, so one pod the model cannot place no longer drops the reservation for the rest. Clear a stale line position from sessions that are no longer waiting. Wrap a content-guard detector's config error and give a nil detector spec its own message. gofmt the AgentSession webhook.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Admission reserves an admitted session's unplaced pods one at a time, and the fixed-set model charged a whole set for each, so a single admitted reviewer filled the burst scenario's two-set cluster. Whenever the first session was admitted alone, the second waited for it to leave. Charge each set to its session's runner pod instead.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sessions that need several pods could deadlock a busy cluster: each session got part of its pods scheduled and then held them while it waited for the rest, so no session could ever finish starting. This change adds an admission queue in front of session startup. A session now gets its whole set of pods at once, or waits in line until it can.
AgentClass.spec.maxConcurrentSessionscan optionally cap how many sessions a class runs at once. Delegated child sessions skip the line, so a parent never waits on its own children.--admission-min-warm(default 5m). After that it can be put to sleep early to make room for someone waiting.--session-admission=false.New status: an
Admittedcondition andstatus.admission(position, line length, join time). Only the operator can write these.Testing
mage test:unit,mage test:integrationandmage test:e2eall pass.Follow-ups