feat: show storage quota and maintenance-shortened sessions in the notice line - #11
Merged
Merged
Conversation
…tice line
Two things a session should tell you about itself and until now did not: that
your writes are about to start failing, and that the allocation you got is not
the one you asked for.
Quota. Bodhi's quota_check lives only on the head node, so the obvious
implementation is `ssh head quota_check` on a timer — every session depending
on passwordless ssh back to a host it has no other reason to talk to. It turns
out not to be necessary. The hard limit is a shared file, and the daemons that
know usage listen to the compute nodes directly, answering "QUOTA <uid>" with
"OK <kilobytes>" per target. So a session can just ask: the whole probe is
bash and /dev/tcp, takes about a second, and reproduces quota_check -b's
numbers exactly (30.2T of 30T, 100.7%).
The result is cached per user rather than per session. Quota is a property of
the account and a user often has six sessions open; there is no sense in each
of them interrogating nine daemons on its own schedule. One probe per interval
serves all of them, and --check-quota rewrites the cache and pokes every
session so the warning clears within a tick rather than at the end of the
ten-minute poll. That is the command to hand an agent that has just deleted
something: leaving a stale warning on the user's screen makes it look like the
deletion did not work. The bodhi-storage skill and --agent-context now both
say so.
Maintenance. Slurm will not start a job that runs into a reservation; it
defers it until the window closes, which can be a day or more. At the default
day-long request a session therefore just stops starting as maintenance
approaches, with no obvious reason. The previous code caught this and printed
the shorter command to run instead, which is correct but hands the arithmetic
back to the user at the moment they wanted a shell. It now trims the request
to fit and says so, and the session carries the fact for its whole life rather
than only in launch output that scrolls away.
The notice line is now ranked rather than single-purpose: over-quota (red)
outranks a short session (yellow), which outranks the scrolling Claude hint.
The first two are static — a marquee is right for an invitation and wrong for
a warning — and share the line when both apply, since both stay true for the
whole session. A session showing a static warning also stops paying for the
marquee's 0.3s redraw.
Two bugs found by testing against the live cluster rather than by reading:
- next_maintenance_window printed without a trailing newline, so `read`
reported failure after setting its variables and the `|| return 0` skipped
trimming entirely. A one-day request was not being shortened at all.
- quota_probe queried the daemons before reading the quota file. Bash's
/dev/tcp has no connect timeout, so on a cluster where those addresses are
unroutable this could stall the status loop for minutes. The local file
read now gates the network half, and somewhere without the quota file
never opens a socket.
Every input is overridable (SINTERACTIVE_QUOTA_FILE, _HOSTS, _PORT, _TIMEOUT,
_POLL) and every failure is silent, so a cluster that does its quotas
differently simply never shows the notice.
Closes #8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things a session should tell you about itself and until now did not: that
your writes are about to start failing, and that the allocation you got is not
the one you asked for.
Closes #8.
The notice line
Rendered from the real functions, with live numbers:
It is now ranked rather than single-purpose: over-quota (red) outranks a
short session (yellow), which outranks the scrolling Claude Code hint. The
first two are static — a marquee is right for an invitation and wrong for a
warning — and share the line when both apply, since both stay true for the
whole session. A session showing a static warning also stops paying for the
marquee's 0.3s redraw.
Quota, without the head node
quota_checklives only onamc-bodhi, so the obvious implementation isssh head quota_checkon a timer — every session depending on passwordlessSSH back to a host it has no other reason to talk to.
It turns out not to be necessary. The hard limit is a shared file, and the
daemons that know usage listen to the compute nodes directly, answering
QUOTA <uid>withOK <kilobytes>per target. So a session just asks. Thewhole probe is bash and
/dev/tcp, takes about a second, and reproducesquota_check -bexactly:Cached per user, not per session — quota is a property of the account and
you often have six sessions open; one probe per interval serves all of them.
--check-quotarewrites the cache and pokes every session, so the warningclears within a tick rather than at the end of the ten-minute poll. That is
the command to hand an agent that has just deleted something: leaving a stale
warning on your screen makes it look like the deletion did not work. The
bodhi-storageskill and--agent-contextnow both say so.Verified end to end on a live session — simulating freeing space cleared the
red notice within 8 seconds, and restoring the real limit brought it back.
Maintenance: trim, don't refuse
Slurm defers a job that would run into a reservation until the window closes,
so at the default day-long request a session simply stops starting as
maintenance approaches. The previous code caught this and printed the shorter
command to run instead — correct, but it hands the arithmetic back to you at
the moment you wanted a shell:
Confirmed the granted allocation matches:
TIME_LIMIT 17:11:00. A requestthat already fits is left alone (a
--time=1hlaunch got its full hour and nonotice), a launch is refused when under 10 minutes remain, and an explicit
--reservationis untouched.Two bugs found by testing against the cluster
next_maintenance_windowprinted without a trailing newline, soreadreported failure after setting its variables and the
|| return 0skippedtrimming entirely. A one-day request was not being shortened at all.
quota_probequeried the daemons before reading the quota file. Bash's/dev/tcphas no connect timeout, so where those addresses are unroutablethis could stall the status loop for minutes. The local file read now gates
the network half — somewhere without the quota file never opens a socket.
Portability
Every input is overridable (
SINTERACTIVE_QUOTA_FILE,_HOSTS,_PORT,_TIMEOUT,_POLL) and every failure is silent, so a cluster that does itsquotas differently never shows the notice.
All four
validate.ymlgates pass locally.🤖 Generated with Claude Code