Skip to content

Admit sessions to the cluster as whole sets, in a fair queue - #64

Open
josephschorr wants to merge 33 commits into
mainfrom
feat/session-admission-queue
Open

josephschorr wants to merge 33 commits into
mainfrom
feat/session-admission-queue

Conversation

@josephschorr

Copy link
Copy Markdown
Member

Summary

Sessions that need several pods could deadlock a busy cluster: each session got part of its pods scheduled and then held them while it waited for the rest, so no session could ever finish starting. This change adds an admission queue in front of session startup. A session now gets its whole set of pods at once, or waits in line until it can.

  • One line, whole sets. A single leader-only admitter compares each waiting session's full pod demand against node capacity, including taints, and admits it only when everything fits.
  • Fair share between agents. Admission favours the agent class using the least of the cluster. AgentClass.spec.maxConcurrentSessions can optionally cap how many sessions a class runs at once. Delegated child sessions skip the line, so a parent never waits on its own children.
  • Warm sessions yield under contention. An idle session stays warm for at least --admission-min-warm (default 5m). After that it can be put to sleep early to make room for someone waiting.
  • Position in line. A waiting session is told where it stands ("You're 3rd in line for …"). Its wait is still bounded by the existing capacity-hold timeout.
  • Self-correcting. If the scheduler can't place a set the admitter expected to fit, the session gives its pods back and returns to its place in line. When capacity can't be measured reliably (for example, a namespace-scoped operator), sessions are admitted and the existing per-pod capacity hold takes over.
  • Safe rollout. Sessions already running when the queue is enabled are adopted, not gated. Sessions that were asleep or on hold at that point rejoin the line when they resume. Admission is on by default and can be turned off with --session-admission=false.

New status: an Admitted condition and status.admission (position, line length, join time). Only the operator can write these.

Testing

  • Unit tests for the line model, capacity model, fair-share decisions and reclaim planning.
  • envtest coverage for the gate, the admitter, adoption on upgrade and operator restart.
  • New bronzethread scenarios: a burst of multi-pod sessions on a small cluster, and a child session admitted while its class is at its cap.
  • mage test:unit, mage test:integration and mage test:e2e all pass.

Follow-ups

  • Update the position-in-line message in place as the line moves.
  • A session that hits a settings error after joining the line should leave it.
  • Cap-based reclaim should also consider warm sessions that have no pods placed yet.

Pure package describing a session's pod demand, when it joined its
class's line, and the order sessions are admitted in. Adds the Admitted
condition and its reasons.
Add a node capacity model with all-or-nothing whole-set reservation, and export schedfit.EffectiveRequests plus a pod-slot count on Headroom.
Admit whole pod sets from per-class lines, picking the line with the lowest
dominant share first; a head that does not fit blocks only its own line.
DemandFor renders every pod a session will create (runner, one sandbox
per tool bundle, content-guard detectors, separate-pod sidecars) through
the reconciler's own builders and prices each with
schedfit.EffectiveRequests, without writing anything.

EffectiveRequests now defaults an unset request to the container's
limit, as the API server does, so a built-but-not-created sandbox (limits
only) is no longer priced at zero. Detector resolution moves into pure
helpers shared with the reconciler, and toolchain resolution accepts a
client.Reader.
Identity precedence and the tier-resolved sandbox backend now live in
desiredBundleSession, called by both the reconciler's bundle loop and
DemandFor, so the two cannot drift.
With Reconciler.AdmissionEnabled, a booting session that is not yet
admitted creates none of its pods: it waits in line under
CapacityHold=True/WaitingForAdmission and fails with HoldExpired once it
has waited past the capacity-hold timeout, measured from WaitingSince.
Settings resolution and the pre-start gates still run so the admitter can
size the session's demand. Reclaiming a session's pods records
Admitted=False/Reclaimed: in the SleptAt write on idle sleep, and after
the terminal reap. The zero value keeps today's ungated behaviour.
The admitter rebuilds every admission line from AgentSession status on
each pass, prices each waiting session's whole pod set, and admits in
fair-share order against node headroom. It writes Admitted=True and the
waiting sessions' line positions (only when they change) to status, so
a restarted operator resumes the same line.

The operator gains --session-admission (default true) and
--admission-min-warm, and passes the admission switch to the
AgentSession reconciler. CapacityModel now reports cluster totals for
the dominant-share calculation.
The uncached session list was cluster-wide, which 403s under the
per-namespace RBAC --watch-namespaces exists for and reaches sessions
in namespaces the operator does not own. With a watch set, the pass now
lists one namespace at a time, using the same set the manager cache is
scoped to.
An admitted booting session whose bundle pod stays unschedulable on a
capacity shortfall for two minutes reaps its pods and goes back to its
line as Admitted=False/PlacementMismatch, written after the reap
succeeds. JoinedAt is unchanged, so it keeps its place; the waiting
timeout restarts at the release. Only with admission on; the pod-level
capacity hold is unchanged otherwise.
Bundles can now open several named sessions across more than one fake
Channel, scope transcript steps to one session, hold a turn until other
sessions reach a state, and run with whole-set admission against a fixed
capacity. The driver watches every session for the whole run and checks
that none holds part of a pod set, that admission never exceeds the
capacity, and the bundle's ordering claims.

admission-burst runs three reviewer sessions and one session of a second
agent on room for two: the reviewer line is FIFO and the second agent is
admitted ahead of the burst's tail.
OpeningText for the unnamed thread returned the first user turn's text,
which scoped every unscoped step to the parent session and stranded a
delegated child's requests.
A placement-mismatch release now fails the session, as the waiting
timeout does, once it has been in line longer than the capacity-hold
timeout since it joined: a stall the admitter's model cannot see would
otherwise re-admit and release forever, since each release restarts
both the waiting clock and the bundle deadline. The release also clears
the bundle stage's CapacityHold, whose give-up time described the pods
it just reaped.

status.admission is admitter-owned; the runner-forgery webhook now
refuses a runner write to it.
Pure planning: oldest-idle warm sessions of the head's line, freed on a cloned model until the head fits.
After deciding, the admitter plans an early reclaim for each line whose
head does not fit: warm Idle sessions of that line idle at least
--admission-min-warm, oldest first, until the head fits. It asks for
each through the existing sleep path with a reclaim-requested
annotation. The AgentSession reconciler treats that annotation as
"sleep now" (still behind the SessionHold guard), removes it once the
session is slept, and clears it as stale when the session left Idle.
A reclaimed session's pods stay in the capacity snapshot until they are
gone, and each status write and pod event in between wakes the
admitter, which then saw the head still not fit and reclaimed the
next-oldest warm session too. Before a line plans, free the still-bound
pods of its in-flight reclaims (requested but not yet slept, or slept
and Reclaimed with pods still terminating) and keep them out of the
candidates. Log, rather than discard, a head that does not fit after
its planned reclaims are freed.
AgentClass gains spec.maxConcurrentSessions. The admitter reads it for
every line with a waiting session and hands it to Decide as that line's
cap; delegated children are neither counted nor blocked. A class whose
cap cannot be read holds its non-child sessions back for the pass.

The admission-child-at-cap bundle delegates into a capped line that is
already full and asserts the delegate tool result is the child's answer.
The capacity-hold notice now appends the session's place in its class's
admission line when status.admission records one.
A session's wait was measured from creation, so one that cleared a
start approval, identity choice or credential gate more than the
capacity-hold timeout after creation failed on its first gated pass.
The admitter also admitted sessions that had a Pending phase and
resolved settings but had not reached the gate (parked, fatal settings,
pin-blocked), reserving capacity for them.

The gate now stamps Admitted=False/Queued at arrival; the waiting
timeout and give-up notice run from it, and the admitter considers only
queued sessions. WriteOwned persists a deliberately re-stamped
transition time on a same-status reason change, so a released session
arriving again is queued from now.
JoinedAt fell back to creation once a cold-woken session had been
admitted (or when its wake and the admission transition shared a
second), so a misfit release measured its wait from creation and failed
any channel session older than the capacity-hold timeout. JoinedAt is
now the last wake if set, else creation.
Admission defaults on, so at upgrade every session already booting,
running or warm idle had no Admitted condition. A booting one was gated
with its pods still up, and the admitter priced its whole set on top of
headroom that already subtracted those pods; a running one was never
counted toward its class's cap or share.

A non-terminal session with no Admitted condition whose status records
pods, and that is not slept or held, is pre-admission: the gate lets it
through without queueing it, and the admitter adopts it
(Admitted=True/Adopted) once it sees one of its pods, so it counts as a
holder. A pre-admission session that sleeps is released like an
admitted one, so its next wake queues.
Three cases let the model admit sets the scheduler could not place, so
the misfit release reaped, re-admitted and re-noticed the same session
every few minutes: tainted pools counted as room, a namespace-scoped
operator not seeing other namespaces' pods, and unknown capacity
admitting everything.

The node model now records each node's NoSchedule/NoExecute taints and
places a pod only where its tolerations (carried on PodDemand from the
built spec) tolerate them. A namespace-scoped admitter treats capacity
as unknown. Sessions admitted under unknown capacity get the reason
AdmittedUnknownCapacity, which the misfit release skips, leaving them
to the pod-level hold. Unknown capacity is logged on transition only.
Warm idle holders count toward an AgentClass's session cap, but early
reclaim only planned for heads that did not fit, so a line held at its
cap by idle sessions waited out their full sleep grace.

Decide now reports which lines' heads wait on the cap. For such a head
the admitter asks the line's oldest non-child warm idle session past
the minimum warm time to sleep early, one per head, and not while an
earlier request for the line is still in flight.
A held session's pods were reaped but it stayed Admitted=True, so it
kept a full reservation (all of it unplaced) and a cap slot for as long
as the hold stood. The Held projection's status write now records
Admitted=False/Reclaimed; on release the session rejoins the line
through the gate.
A sandbox pod adopted from a SandboxWarmPool keeps its pool-minted
name, so matching demand entries to pods by name never found it: the
holder's whole bundle demand stayed unplaced and was reserved on top of
a snapshot that already counted the pod, for the session's lifetime,
and an early reclaim could not credit it.

A bundle's demand entry now carries its SpiceboxSession, and the
admitter finds its pod by that SpiceboxSession's label (which adoption
injects); runner, detector and sidecar entries still match by their
deterministic names.
Reserve a holder's unplaced pods one at a time, so one pod the model
cannot place no longer drops the reservation for the rest. Clear a
stale line position from sessions that are no longer waiting. Wrap a
content-guard detector's config error and give a nil detector spec its
own message. gofmt the AgentSession webhook.
@vercel

vercel Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
openagentprimitives Ignored Ignored Preview Nov 11, 2026 6:56am UTC

Request Review

Admission reserves an admitted session's unplaced pods one at a time, and
the fixed-set model charged a whole set for each, so a single admitted
reviewer filled the burst scenario's two-set cluster. Whenever the first
session was admitted alone, the second waited for it to leave. Charge each
set to its session's runner pod instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant