Conversation
…ody bytes
The hub is the fleet's whole control plane, every agent shares one
TURMA_AGENT_TOKEN, and `restart: unless-stopped` turns a crash into a
repeating outage. At the deployed `mem_limit: 256m` it could be OOM-killed
by anyone holding that token, two different ways, and neither fix bounds
the other's cost.
Measured on `main`, node:24-alpine at -m 256m:
- 1024 overlapping 0.9 MiB heartbeats -> OOMKilled, zero requests answered.
- 2 concurrent 30 MiB heartbeats -> OOMKilled (XERK-258).
XERK-273 attributed the first to the socket count. Re-measuring says
otherwise: a socket costs ~28 KiB, so 1024 idle-bodied connections peak at
49 MiB and 4096 at 135 MiB. The bill in that row was the bodies. Both
bounds are therefore needed — a connection cap safe against a worst-case
32 MiB body would have to be ~4, and no byte budget can see a socket that
has not sent a body yet.
Connection cap (XERK-273)
`server.maxConnections`, default 256, `MAX_CONNECTIONS` to override. Sized
against real use (~72 sockets: agents' control channels plus viewers' SSE,
HTTP and terminal sockets), not against what survives. It counts upgraded
WebSockets too, which is why it must clear steady-state use with room to
spare. Node destroys an over-cap socket before parsing, so a `drop` handler
logs the refusal — rate-limited, since a flood is thousands per second.
In-flight body budget (XERK-258)
Every memory ceiling is now a fraction of the container's cgroup limit
rather than a fixed number, logged at boot. The flat 128 MiB upload ceiling
was double the whole container and so could never fire before the OOM
killer did.
A body is charged BODY_PARSE_COST (3x) its wire size, not its wire size:
the bill is the JS string plus the object graph `JSON.parse` builds beside
it. Charging wire bytes was tried first and admitted both 30 MiB beats
against a 64 MiB budget, OOM-ing anyway. Raw Buffer bodies keep 1x —
nothing parses them.
A body arriving into an idle hub is always admitted, which is what keeps a
single 65 MiB migration bundle working; everything after it is admitted
only while the sum stays under budget. The per-request ceiling only ever
tightens, so a 64 GB host does not inherit an 8 GB single-body ceiling.
Refusals are 503 + {error}, never 413: the body was fine, the hub was
momentarily full. Both readRawBody callers answered a flat 413 and now draw
that distinction. On a 503 the migration relay holds the migration in
`exporting` and `_migration_upload` retries 5xx (never 4xx) — a bundle is
the largest body an agent sends, nothing else would retry it, and a lost
one strands the move.
Verified at -m 256m, all shapes surviving with real 200s served:
1 x 30 MiB 52 MiB peak 1x200
2 x 30 MiB 59 MiB peak 1x200, 1x503
6 x 30 MiB 84 MiB peak 1x200, 5x503
512 x 0.9MiB 114 MiB peak 23x200, 233x503, 256 refused
1024 x 0.9MiB 71 MiB peak 23x200, 233x503, 768 refused
Ordinary beats succeed immediately afterwards, so nothing leaks the budget.
…re promised
QA found the first cut had traded one availability hole for a cheaper one,
and re-measuring the worst case found a second. Both are fixed here; the
OOM matrix still holds.
1. Reservation wedge (QA, HIGH)
Charging the DECLARED Content-Length reserved budget before any byte
arrived, so one socket that declared a 32 MiB body and then sent nothing
held more than the whole budget until the 300s request timeout — and every
other body on the hub was refused 503 meanwhile. It needed no bandwidth and
no credentials (/api/login reads its body before any auth gate) and was
renewable: a cheaper, longer fleet-wide outage than the OOM this exists to
prevent, and worst at the deployed 256m where one socket sufficed.
A declared length is a CLAIM. It is still CHECKED — against the per-request
cap, and against the budget — but never CHARGED, because a claim that is
never held is never denied to anyone else. Only buffered bytes are charged.
The idle-hub exemption had to become sticky per read (`sole`) to survive
that: judged per top-up, a lone 65 MiB migration bundle would sail past the
budget on its first chunk and then be refused mid-stream by its own charge.
Verified: 200 sockets each declaring 32 MiB and sending nothing, against
the real container — ordinary beats and an 8 MiB beat both still 200.
2. Refusal churn (found while re-measuring)
256 sockets (the connection cap) each sending a 30 MiB body still OOM-killed
the hub, with not one of those bodies buffered. Two causes, both about what
a REFUSED request costs:
- Node hands us whatever accumulated in one `data` event — megabytes, not
one buffer — so refusing on the first chunk still costs a chunk per
socket. Hence the declared-length check above, which refuses before any
of it arrives.
- Node then DUMPS an unread body when the response finishes, to keep the
connection alive: it resumes the stream we paused and reads all 30 MiB,
discarding it. Discarded bytes are still read into memory. So a budget
refusal now closes the connection (`endRefusedConnection`) after `finish`
— the 503 is on the wire first, and the connection was not reusable
anyway since we never read its body.
Draining is also rationed for budget refusals (BUDGET_DRAIN_SLACK, 64 KiB vs
1 MiB): it is a courtesy paid out of the resource that has just run out, and
unlike an oversize refusal it hits many requests at once.
Matrix at -m 256m, all surviving, hub serving afterwards:
1 x 30 MiB 56 MiB peak 1x200
2 x 30 MiB 59 MiB peak 1x200, 1x503
6 x 30 MiB 58 MiB peak 1x200, 5x503
256 x 30 MiB 56 MiB peak 1x200, 233x503 <- was OOMKilled
512 x 0.9 MiB 119 MiB peak 194x200, 317x503
1024 x 0.9 MiB 115 MiB peak 256x200, 237x503
Throughput went UP as a side effect: 256 served on the 1024 row against 23
before, because refused sockets are now released instead of drained.
…ize path A second QA pass returned FAIL with three HIGH defects. All three are fixed; one of them was a regression I had introduced against origin/main. 1. The ceilings were unreachable under any concurrency (QA) The idle-hub exemption required the hub to hold EXACTLY zero bytes, so one trickling request defeated it: a real 65 MiB migration bundle was refused with 3 KB in flight, stranding the move, and HEARTBEAT_MAX went on advertising a 32 MiB beat that — at 3x parse cost against a 64 MiB budget — no concurrent moment would accept. The effective ceiling was TOTAL/3. Replaced with two lanes (`bodyLaneFor`): the shared budget, and ONE big body at a time that need not fit it. Worst case is one max-size body plus the whole shared budget, which is what the container is sized against. The lane is re-judged on every top-up, not latched. Latching it — which is what I wrote first — removed incremental enforcement altogether: several bodies entered the shared lane while small, never consulted it again, and grew past the ceiling together. Two 30 MiB beats OOM-killed the hub with the budget nominally in force. A body that outgrows the shared lane is now promoted to the big one if free, else refused. Verified: 8, 20 and 30 MiB beats all 200 with a trickle in flight. 2. The oversize/413 drain path OOM-killed the hub (QA, pre-existing on main) Refusals on SIZE took none of the new protections, so the hub drained cap + 1 MiB per socket and Node dumped the rest. 256 sockets streaming past the cap killed it; so did the unauthenticated variant through /api/login. QA confirmed origin/main dies the same way, so it is pre-existing — but it is the same class this ticket exists to close and needs no credentials, so it is fixed here rather than filed. Draining is now capped by CONCURRENCY (DRAIN_CONCURRENCY_MAX), and past the cap the refusal stops reading and closes. The count is the right unit: the cost is concurrent read churn, not bytes, since draining keeps nothing. 3. Refusing oversize on the DECLARATION broke XERK-235 (my regression) Refusing at header time is correct for the budget and wrong for size. It makes Node close the connection under a request still being written, and a client that writes its whole body before reading loses the response. That client is python urllib — exactly what hub-agent.py posts with. Measured: origin/main returns a readable 413 to urllib; my previous commit returned `[Errno 32] Broken pipe` at every size tried. That is XERK-235's loop, where an agent never learns its limit and re-sends the same body every beat. An oversize body is therefore read to `cap` again, as on main. The budget is what bounds it now: those bytes are charged like any other, so a flood is refused 503 before buffering and only what the hub can afford ever buffers its way to a 413. Restored and re-measured with real urllib: HTTP 413 with its limit, and 256-socket oversize floods survive at 98-123 MiB. Matrix at -m 256m, all surviving, hub serving afterwards: 1 x 30 MiB 52 MiB 2 x 30 MiB 63 MiB 6 x 30 MiB 78 MiB 256 x 30 MiB 118 MiB 512 x 0.9 MiB 107 MiB 1024 x 0.9MiB 88 MiB 256 x 33 MiB (oversize, authed) 98 MiB 256 x 2 MiB (oversize, unauth'd) 123 MiB 200 sockets declaring 32 MiB, sending nothing: ordinary and 8 MiB beats 200
Third QA pass: all three of the previous HIGH defects verified fixed, and
the lane and drain bookkeeping came back clean under targeted attack (no
`bigLaneTaken` leak across 5 abort waves, no two bodies in the big lane, no
drain-slot leak across 5 waves x 24 oversize bodies, budget back to 0 on all
12 framing paths). One new HIGH, caused by this change, is fixed here.
The budget bounded how MUCH may be held. Nothing bounded how LONG.
One socket that streamed 22 MiB and then simply stopped charged past the
shared budget, took the big lane, and from then on every body on the hub was
refused 503 — tiny heartbeats and the operator's own login included — until
`requestTimeout` expired 300s later. Renewable at ~22 MiB per 5 minutes, or
about 0.6 kbit/s, which is a cheaper outage than the OOM this prevents.
A hold with no progress is not slow, it is abandoned, and the two are told
apart by whether bytes are still arriving. `BODY_IDLE_TIMEOUT_MS` (20s) is
armed only while a read actually holds a charge and is reset by every chunk,
so a genuinely slow client on a bad link never meets it — it keeps sending.
On expiry the charge is released, the lane freed and the socket destroyed;
the route writes nothing, since a caller that stopped mid-body is not
waiting to read a status.
Measured at 256m, one socket streaming 22 MiB then stalling:
before beat 200 login 401 (i.e. served)
+1s .. +15s beat 503 login 503 <- was the whole 300s, renewably
+25s, +35s beat 200 login 401 <- reclaimed, staller still attached
after drop beat 200 login 401
QA also measured the matrix under a more overlapping load pattern than mine
and got higher peaks (256 x 30 MiB: 200 MiB against my 118). The PR reports
QA's figures, not mine — the headroom is roughly half what I first claimed.
Two residual holes are FILED, not fixed, as XERK-287:
- Chunked bodies have no declared length, so the cheap pre-check cannot
fire and 256 of them plus held uploads still OOM at 256m. The fix is a
concurrency cap on undeclared bodies, but the number is only safe if
agent traffic really is declared-length at the origin — and every agent
arrives through the Cloudflare tunnel, which this session cannot
measure. Capping blind would refuse legitimate heartbeats fleet-wide.
- The readable-413 window is 1 MiB wide, so urllib gets a status at 32 and
33 MiB but not at 40. Identical on origin/main, so unchanged here.
Both are pre-existing shapes that origin/main handles worse (it OOMs at two
30 MiB beats), so neither blocks this change.
The idle reclaim did what it was built to do — QA confirmed a fully silent hold is taken back at exactly the window, no progressing body is ever touched, and a real throttled 65 MiB migration is unaffected — but it did not close the wedge. Resetting the window on ANY byte is not a liveness check, it is one an attacker can forge. Warm up 22 MiB to take the big lane, then send one byte every 15s: each byte tops up the charge and re-arms the timer, so the lane is held indefinitely (QA proved it past 300s on a rig and to 75s on the deployed container) while every POST on the hub — heartbeat ingest, spawn/kill/rename, the migration relay, uploads and the operator's own login — answers 503. Steady-state cost after the warmup is ~0.5 bit/s, three orders of magnitude below what I claimed the previous commit had bought, because renewing never has to re-stream. The distinction the fix rested on was wrong, not its threshold: a dribble is neither silence nor slowness. So the window now reopens on PROGRESS — BODY_MIN_PROGRESS_BYTES (64 KiB) must arrive per window, making the rule a minimum RATE of about 3 KiB/s. That is far below anything the fleet does (agents reach the hub over a LAN or the tunnel) and far above what a dribble can fake. QA's own finding stands behind the number: they could not manufacture a >20s zero-byte gap from throttling, and a real slow migration sits in the shared lane at megabytes per step. The floor gives way to whatever the body has LEFT to send, so a nearly-complete upload is never reclaimed over its last few bytes. An attacker cannot use that: holding a big charge requires a large body, which means a large remainder. Verified at 256m against the dribble, with a legitimate slow sender running alongside it the whole time: baseline beat 200 login 401 +5s beat 503 login 503 <- one window, not 300s +25s beat 200 login 401 <- reclaimed, attacker still dribbling +45s, +65s beat 200 login 401 after drop beat 200 login 401 Re-confirmed unchanged: urllib gets its 413 at 32/33 MiB (XERK-235), and the matrix still holds (2x30 52 MiB, 256x30 136 MiB, 1024x0.9 110 MiB, none OOM-killed, hub serving after each). Also labels XERK-287 with its repo, per the ticket-filing rule.
…tended
QA's fifth pass was cut short by a session limit, but it did settle one
thing: the remainder carve-out could NOT be stretched — a body engineered to
keep a small remainder was still reclaimed at 20.0s. Its rate-floor timing
was self-contaminated, so I re-ran those cases and found a false positive of
my own by reasoning about the arithmetic.
The progress floor applied to every charged read, however small. A 60 KB
request arriving at 1 KB/s has ~55 KB left, so the floor asks for 55 KB in a
window it cannot meet, and it is dropped — while holding 180 KB of a 64 MiB
budget. That protects nothing and breaks a legitimate call. Reclaiming is
for relieving contention, not for punishing slowness.
So it now fires only under pressure (`budgetUnderPressure`): the one big lane
occupied, or the shared budget more than half spent. Those are the states in
which one body holding on actually costs another its turn. With room to
spare nothing is reclaimed at all.
That gate does not soften the attacks, because both create the pressure they
need. Measured at 256m:
Single-socket dribble (22 MiB warmup, 1 byte/15s), with a legitimate slow
sender running alongside throughout:
baseline 200/401; +5s 503/503; +25s, +45s, +65s all 200/401 with the
attacker still dribbling; 200/401 after it drops.
Multi-socket shared-lane dribble — the variant QA flagged but never built:
64 sockets each warming 384 KB (too small for the big lane, together over
the shared budget) then dribbling under the floor:
baseline 200; 503 while warming; +25s, +45s, +65s all 200. The shared
budget half-spent is itself the pressure, so the gate catches this shape
without the big lane ever being involved.
Dribbling AT the legal floor (64 KiB/window) is deliberately NOT reclaimed
— it is paying the rate the rule asks. It costs ~35 kbit/s per socket,
~2 Mbit/s across 64, against ~0.5 bit/s before this rule existed. The hub
stays available throughout; that is the budget degrading to 503s as
designed rather than an outage.
Tests restructured around this: reclaim cases stage contention by occupying
the LANE (not the budget, which would refuse the body under test), and a new
case pins the false positive — an uncontended hub must not drop a slow
caller.
…y be held QA's sixth pass corrected a factual error in my last report and found the wedge still open. Both are addressed here. The correction first: I reported floor-rate dribbling as "hub available throughout". That measurement was the 64-socket SHARED case. The single BIG-LANE socket is a total outage — QA measured the operator's own login at 503 across five windows — and my dribble probe missed it only because it dribbled UNDER the floor, where reclaim does fire. 1. The lanes are now accounted separately The big body's charge was going into the shared counter. Since a big-lane body by definition exceeds the budget, merely OCCUPYING the lane refused every other body on the hub: a 200-byte heartbeat, the operator's login, everything. One authenticated socket could hold the fleet's control plane offline for ~29 kbit/s, which is the whole outage. Kept apart, a big body costs the shared lane nothing. Occupying the lane delays other LARGE bodies — which is what an exclusive lane means — and ordinary traffic never notices. The hub's ceiling is unchanged: one max-size body plus the shared budget, which is what it is sized for. `pressure` is judged per lane accordingly: shared reads by the shared budget, a big-lane read always, since anything sitting in an exclusive lane blocks the next body whatever room exists elsewhere. Measured at 256m, one socket holding the lane and dribbling AT the floor: baseline beat 200 login 401 25 MiB 200 +20s..+80s beat 200 login 401 25 MiB 503 <- was 503/503/503 after it drops beat 200 login 401 25 MiB 200 2. A wall-clock ceiling on lane occupancy QA's read is right that no rate threshold closes the rest: a body dribbling at the floor is byte-for-byte indistinguishable from a legitimate slow migration at the same rate. So the bound is orthogonal — not "are you making progress" but "you have had the lane long enough". BIG_LANE_MAX_HOLD_MS defaults to 10 minutes, which a 65 MiB bundle clears at ~110 KiB/s, far below what a LAN or the tunnel does, and the agent retries a reset anyway. Verified with the ceiling wound to 30s against the same attack: 25 MiB bodies go 503 at +20s and 200 from +40s on, while the attacker keeps paying the floor rate. Residual, stated plainly: an attacker paying ~29 kbit/s can still delay LARGE bodies (migrations, history-heavy beats) for up to the ceiling, then must re-establish. Ordinary traffic is unaffected throughout. That is a bounded degradation of one feature rather than a control-plane outage. QA's defect 2 (a legitimate migration stalling >20s past its halfway point is reclaimed and reset, agent retries and re-ships) is not fixed here: with the lanes separated it now costs other traffic nothing, and judging it needs the same tunnel measurement XERK-287 already blocks on. Noted there. Matrix unchanged: 2x30 67 MiB, 6x30 85 MiB, 256x30 110 MiB, 1024x0.9 91 MiB, none OOM-killed, hub serving after each. Suites 1105 node / 1289 python.
QA's seventh pass found a BLOCKER in the previous commit, and it fires on
ordinary traffic with no attacker involved.
Separating the lanes was right, but a read can CHANGE lane: a large body is
admitted to the shared lane (its first chunks are small) and promoted when it
outgrows the budget. Its charge then sat in two counters while the read owned
one lane, and `release()` can only name the lane it ENDED in — so the whole
amount came off the big lane and the pre-promotion part was never given back.
One legitimate 22 MiB heartbeat therefore leaked the entire 64 MiB shared
budget, permanently. 21 MiB leaked nothing; 22 MiB leaked all of it, the
threshold being BODY_INFLIGHT_TOTAL_MAX / BODY_PARSE_COST = 21.33 MiB. Well
inside HEARTBEAT_MAX, and precisely the size XERK-235 exists because staged
history reaches. After it, every non-trivial body was refused for the life of
the process and `budgetUnderPressure("shared")` was stuck true, reinstating
the very false positive the pressure gate was added to prevent. This is the
"charge that outlives its buffer ratchets the budget shut" failure that
releaseBody's own comment warns about.
A charge must live entirely in the lane its read currently occupies, so
promotion now migrates it (`migrateToBigLane`) and release is correct by
construction.
QA's second finding has the same root cause and is closed by the same fix:
while a holder's pre-promotion charge stayed billed to shared, a holder
occupied nearly the whole shared budget, so my "ordinary traffic unaffected"
was wrong — a 0.5 MiB beat and a 4 MiB upload were refused too.
Measured at 256m, before and after one large beat:
4 x 5 MiB concurrent, then a 21/22/30 MiB beat, then 4 x 5 MiB again:
200,200,200,200 at every step (was 200,503,503,503 permanently)
With a big-lane holder active, by body size:
0.001 / 0.5 / 2 / 4 / 15 MiB -> 200 25 MiB -> 503
all 200 again the moment it drops
Residual, corrected: an attacker paying ~29 kbit/s delays only bodies that
need the exclusive lane — migration bundles and heartbeats over ~21 MiB —
for up to BIG_LANE_MAX_HOLD_MS, then must re-establish, which QA showed takes
seconds. So the ceiling bounds any single hold, not aggregate denial of those
two things. Everything else is unaffected.
The two new tests were checked against a build with the fix disabled and both
fail there, so they pin the leak rather than merely passing beside it.
Matrix: 2x30 81 MiB, 6x30 97 MiB, 256x30 170 MiB, 1024x0.9 116 MiB, none
OOM-killed, hub serving after each. QA independently measured 256x30 at
203 MiB under a tighter pattern; that is the figure to plan against. Suites
1107 node / 1289 python.
QA's eighth pass confirmed the promotion leak is gone — no byte leaked through any promotion path, across every abnormal exit, chunked bodies, uploads, migration bundles and mixed concurrent traffic — but found that the fix moved the memory ceiling, and 256 x 30 MiB now OOM-killed the container in 2 of 4 runs. That row survived on the three previous commits. The cause is structural, not a bad number. Separating the lanes was right, but it left two INDEPENDENT ceilings whose sum is the real worst case, and the shared half had been sized back when a big body still consumed it. So the true bound quietly became `shared budget + a whole big body` with nobody doing that arithmetic. `BODY_INFLIGHT_TOTAL_MAX` is now the ceiling on everything in flight across BOTH lanes (MEMORY_LIMIT/2), and shared admission counts the big lane. A big body really is occupying memory; pretending otherwise is what let the worst case drift. As it grows, ordinary traffic's room shrinks — but never to nothing, because the ceiling sits above what any one body can charge. That inequality is load-bearing and now asserted: BODY_INFLIGHT_MAX * BODY_PARSE_COST < BODY_INFLIGHT_TOTAL_MAX It is what keeps a lane-holder from starving everyone (the total outage that made the lanes separate accounts) AND what keeps the worst case inside the container. At 256m: 96 MiB of units for one max body, 128 total, so 32 remain for ordinary traffic while the lane is held. The hub logs it at boot as "128 MiB across both lanes". Effective admission in the killing flood drops from 160 units to 128. Honest limit on my own verification: my flood harness does not reproduce QA's OOM. On the old code QA measured 195-233 MiB with 2/4 OOM where mine peaked at 170; on this commit mine peaks at 139-220 across 4 runs with no OOM. That is consistent with an improvement and does NOT establish the row is fixed — QA's harness is the instrument that detects this, and it has to be the one to clear it. If it does not clear, the next lever is lowering BODY_INFLIGHT_MAX, since the inequality above leaves little room between 96 and 128. That would lower HEARTBEAT_MAX from 32 MiB to 16 MiB, which is a user-visible change to the largest beat a host may send, so it is a decision to raise rather than make quietly. Suites 1107 node / 1289 python. Lane tests restructured: staging now uses the largest body that can actually exist rather than an impossible one, since under a single ceiling a max-size body FITS the shared lane on a quiet hub — the exclusive lane exists for when it does not.
…ERK-287) QA's ninth pass confirmed the 256 x 30 MiB row is fixed on its harness (8/8 survive, where it OOM-killed 2 of 4 before), and found that held uploads are still a budget of their own sitting OUTSIDE the single ceiling this work just built. 128 + 64 = 192 MiB of a 256 MiB container. That is this change's own argument turned back on it — two independent ceilings have to be added to know the worst case, and nobody does that arithmetic. It reproduces identically on the previous commit, so it is not a regression, and closing it is a sizing decision with three user-visible levers rather than a code fix. Filed on XERK-287 with the levers and the measurements; recorded here so the next reader of these rules sees the whole worst case rather than the half this file described.
main landed three changes that touch the same memory work, so each conflict was resolved on what both sides were for rather than by picking a side. agent/hub-agent.py — `_migration_upload` main gave it a True/False return so the caller can tell the hub instead of letting the move time out; this branch gave it a 5xx retry. Both kept: it retries a transient refusal and reports the outcome either way. turma/server.js — the migration blob route (XERK-263) main now SPOOLS the bundle to disk instead of buffering it, and answers a uniform 404 so a refusal cannot name the source. Took main's side whole. The branch's 503/413 handling here is genuinely superseded: a spooled body never touches the in-flight heap budget, which is a better answer for this route than the budget was. Its comments citing the 65 MiB bundle as the reason the big lane exists are now stale for the relay and are corrected in the follow-up commit; the lane is still needed for large heartbeats. turma/server.js — `containerMemoryLimit` Both sides added one, and function declarations hoist, so main's silently won and MEMORY_LIMIT would have been computed by the wrong implementation. Now ONE function, on main's contract (null when there is no limit to read), with the fallback to host RAM applied where this branch derives from it. turma/server.js — the boot log Both print derived budgets for the same reason (they are derived, so being refused by them is otherwise the only way to learn them). Both kept. CLAUDE.md — over the 40k CI gate after the merge Both sides added to Cross-cutting contracts and both belong there, which put the file at 42,235 characters. Split by path per the file's own rule rather than trimming rationale: the body-budget mechanics move to a new `.claude/rules/turma-limits.md` scoped to `turma/server.js`, and what genuinely spans components stays — the container-limit principle, the 413/503 wire contract with the agent, the urllib caveat, and the XERK-287 gap. 36,175 characters. turma/tests/server.test.js — independent additions on both sides, both kept. The trailing `});` was shared context closing whichever last test survived, so this branch's final test needed its own. Verified after merging: 1203 node tests, 1305 python, all green. At -m 256m the boot log shows both features' derived ceilings, and the flood matrix still survives with real 200s served — 2x30 19 MiB, 256x30 145 MiB, 1024x0.9 146 MiB, none OOM-killed, hub serving after each.
…ls (XERK-263) The merge brought in XERK-263, which spools a migration bundle to disk instead of buffering it. So the 65 MiB bundle no longer reaches the in-flight heap budget at all, and citing it as the reason the exclusive lane exists is now wrong — what the lane actually carries is large HEARTBEATS. The bundle stays in the history of WHY the idle-hub rule was replaced, since that is what the rule broke at the time, but it is marked as no longer the rationale so a future reader does not restore it as one. Same correction to BIG_LANE_MAX_HOLD_MS's sizing note, which was sized against the bundle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes XERK-273 and XERK-258. At the deployed
mem_limit: 256mthe hub could beOOM-killed by anyone holding the shared
TURMA_AGENT_TOKEN, two different ways,and neither bound covers the other's cost. The hub is the fleet's whole control
plane and
restart: unless-stoppedturns a crash into a repeating outage.Measured on
main,node:24-alpineat-m 256m:mainThe ticket's diagnosis was wrong, which changed the work
XERK-273 attributed the first row to the socket count. Re-measuring says
otherwise: a socket costs ~28 KiB, so 1024 idle-bodied connections peak at
49 MiB and 4096 at 135 MiB. The bill in that row was the bodies. Both bounds
are needed and neither substitutes for the other — a connection cap safe against
a worst-case 32 MiB body would have to be ~4, and no byte budget can see a socket
that has not sent a body yet. That is why XERK-258 is folded in here rather than
left for later.
What this does
Connection cap (XERK-273).
server.maxConnections, default 256, envMAX_CONNECTIONS. Sized against real use (~72 sockets: agents' control channelsplus viewers' SSE, HTTP and terminal sockets), not against what survives. It
counts upgraded WebSockets, which is why it must clear steady-state use with room
to spare. Node destroys an over-cap socket before parsing, so a
drophandlerlogs the refusal, rate-limited.
In-flight body budget (XERK-258). Every memory ceiling is now a fraction of
the container's cgroup limit rather than a fixed number, logged at boot. The flat
128 MiB upload ceiling was double the whole container and could never fire before
the OOM killer did.
BODY_PARSE_COST(3x) its wire size, not its wire size: thebill is the JS string plus the object graph
JSON.parsebuilds beside it.Charging wire bytes admitted both 30 MiB beats and OOM'd anyway.
it. Without the big lane the hub's own advertised ceilings are unreachable
under any concurrency: a real 65 MiB migration bundle was refused with 3 KB in
flight, and
HEARTBEAT_MAXpromised a 32 MiB beat no concurrent momentaccepted. The lane is re-judged per top-up, and a promoted body's charge moves
lanes with it.
BODY_INFLIGHT_TOTAL_MAX, MEMORY_LIMIT/2).Two independent ceilings have to be added to know the worst case and nobody
does that arithmetic.
BODY_INFLIGHT_MAX * BODY_PARSE_COST < BODY_INFLIGHT_TOTAL_MAXis load-bearing in both directions — the headroomabove one max body is the room ordinary traffic runs in while the lane is held,
and it is what keeps the worst case inside the container. Asserted in the suite.
Content-Lengthis checked,never charged. An oversize body is deliberately NOT refused on its declaration,
because refusing early makes Node close under a request still being written and
python
urllib— whathub-agent.pyposts with — then loses its 413. That isXERK-235's offline loop.
BODY_MIN_PROGRESS_BYTESper
BODY_IDLE_TIMEOUT_MS, ~3 KiB/s), enforced only under contention, plusBIG_LANE_MAX_HOLD_MSas an orthogonal bound — no rate threshold separates afloor-rate attacker from a genuinely slow migration.
{error}, never 413: the body was fine, the hub wasmomentarily full. Both
readRawBodycallers answered a flat 413 and now drawthe distinction. On a 503 the migration relay holds the migration in
exportingand_migration_uploadretries 5xx (never 4xx) — nothing elsewould, and a lost bundle strands the move.
Verification
Handed to the
qaagent, which returned FAIL seven times before this. Everydefect after the first came from one of my own fixes, and the unit suite never
once objected — all of them were caught by containerised adversarial
measurement. Notable ones, all now fixed and pinned by tests:
Content-Lengthlet one silent socket — unauthenticated,via
/api/login— wedge every POST route into 503 for 300s.because a body promoted between lanes released into only one of them.
total outage — a 200-byte heartbeat and the operator's own login refused,
for ~29 kbit/s.
relied on.
Final QA verdict is PARTIAL: the row it was asked to clear survives 8/8 runs
where it OOM-killed 2 of 4 before, and the one remaining OOM is pre-existing
(reproduces on the prior commit) and filed as XERK-287.
Matrix at
-m 256m, QA's harness (deliberately more overlapping than mine — itsfigures are the ones to plan against), all
OOMKilled=falsewith/healthz200afterwards:
Also verified: python
urllibgets a readable413 {"limit":33554432}at 32 and33 MiB (XERK-235); the budget returns to 0 across all 12 framing paths, chunked
bodies, uploads, migration bundles and mixed traffic; keep-alive 5/5 with no
Connection: close; a real throttled 65 MiB migration completes.Suites: 1107 node, 1289 python. Both new leak tests were run against a build with
the fix disabled and fail there, so they pin the defect rather than passing
beside it.
Residual, in QA's words
Not fixed here — XERK-287
Held uploads are still a budget of their own outside the single ceiling, so
the true worst case is
in-flight + uploads= 192 MiB of 256, and the flood rowOOMs once attachments are staged beside it. This change's own argument applies to
it, but closing it is a sizing decision with three user-visible levers (halve the
upload relay, halve
HEARTBEAT_MAX, or raisemem_limit) rather than a codefix, so it is filed with the measurements rather than decided here. XERK-287 also
carries the chunked-body variant, which additionally needs a Cloudflare tunnel
framing measurement this session could not take.
mem_limitstays at 256m per the ticket's direction; every number above isheadroom-bound, and the measurements are there so raising it stays an evidenced
decision.
Notes
No client-facing change: web and Android already read
{error}off any non-2xxgenerically (XERK-264), so the new 503 surfaces on both with no parity work.