XERK-246: local-model failover controls in the Android client - #430
Draft
xerhab wants to merge 23 commits into
Draft
XERK-246: local-model failover controls in the Android client#430xerhab wants to merge 23 commits into
xerhab wants to merge 23 commits into
Conversation
Running out of Claude usage stops every session on a host at once. The web UI has had the failover since XERK-246; Android had neither control, which is a gap precisely when it matters most — usage runs out while you are away from a desk, and a phone was then the one surface that could do nothing about it. Two controls, matching the web: - A "run: <source>" chip in the chat compose bar, beside model and mode, that POSTs .../sessions/<id>/model-source. The agent relaunches with --resume, so the conversation, worktree and branch carry over. - A "Run against" row in the spawn composer, so new work can START on the local model. Without it you could fail existing sessions over from the phone but not begin anything once usage was gone. Both follow the HOST's localModel.available exactly as the composer's clip button follows uploadMaxBytes: an agent reporting nothing cannot do it and the hub 409s the command, so the control is hidden rather than offered and refused. The chip is shown anyway when the session is already local, so one whose host later lost its configuration keeps a visible way back. The chip paints from a memo until the heartbeat agrees — the relaunch takes several beats, and without it the value springs back and reads as a control that did nothing. The memo retires on the heartbeat reporting the switch, and ages out at 60s so one that never lands can't pin the chip on a lie; a refusal drops it at once and says why. The Claude model picker is hidden on a local session (a static chip states the model instead), as on the web: every alias it could offer — "default" included, since that resolves to the shared login's default — is one the self-hosted endpoint refuses. Pure half in core/ModelSource.kt so the Compose screens stay thin renderers. android/PARITY.md narrows the gap to the remaining 🏠 mark on session cards.
…composer QA drove the branch and found the memo did not survive leaving the chat screen, which is the one moment it exists for. - The memo lived in ChatViewModel, which is scoped to the chat's nav back-stack entry, so walking back to the session list mid-switch destroyed it and the chip sprang back to the old value — the exact "control that did nothing" reading the memo is there to prevent. It moves to AppContainer.modelSwitches, the same place and for the same reason as the compose drafts (XERK-122). The web keeps its memo at module scope across the same navigation. - The spawn composer hid the Model row and dropped the alias for a local spawn. The web does neither: sessions.html renders and sends it whatever the source. The agent drops --model for a local session itself, and the alias is what that session returns to if it is later switched back — so discarding it gave an Android-spawned session a different model from a web-spawned one. Only the CHAT bar fixes the model, because there the picker breaks a live session. - The chip now carries the web's ☁/🏠 glyph and its "Subscription" label. The amber tint alone can't answer "which model wrote this turn" for a colour-blind reader. The tooltip is not ported (a phone has no hover, and the model name is already the chip's text) and is recorded in PARITY.md. - A memo with a blank session id is no longer honoured, so it can never paint a record-less session; the web refuses it the same way. Five mutations escaped QA's mutation run because the UI/VM half had no coverage. The decisions move into pure, tested functions — ModelSource.settle / spawnValue / composerOffers / glyph and FleetViewModel.spawnRequest — and the tests now pin the settle rule, the composer gate, the chip glyph, and the exact spawn JSON (bare, subscription and local). AgentDecodeTest gains the null modelSourceAt that hub-agent sends on every session that never moved. 312 tests, 0 failures.
…found QA re-drove the branch: the memo fix, the composer parity fix and the chip glyph all held up on device, but it found the branch no longer merged and two mutations still escaping. - **Rebased onto origin/main** (XERK-251/252 rewrote the same files). The one real conflict was `observeFleet`'s `_state.update`, where main added `tunnelOnline` and this branch added `localModel` to the same `copy(...)`. Taking either side alone silently disables a whole feature — QA ran that exact resolution as a mutation and all tests stayed green. That mapping is now `ChatUiState.fromFleet(agent, session, host)`, one place, pinned by two tests, so a future merge cannot truncate it unnoticed. - **The drop-on-refusal was the last inline decision left in the ViewModel**, so deleting it kept the suite green. It is now `ModelSource.afterAttempt`, and both outcomes are asserted: holding a memo for a full minute when the hub has already refused is the same lie the TTL exists to bound. - **The TTL assertions derived their boundary from the constant under test**, so raising it from 60s to 16.7 hours passed. Both now use literals, plus a direct assertion on the constant. - `ModelSwitchStore.kt` used a literal NUL as its key separator, which made git treat the file as binary — the file that IS the memo fix would have rendered as "Binary file not shown" in the PR. It uses the escape now; same key. - The chip's missing tooltip moved to PARITY.md's "Deliberate differences", where the file's own preamble says justified differences belong. qa.md is resolved toward main — the branch had been editing a pre-XERK-251 copy and would have dropped ~180 lines of it, including the container Android recipe that is the only way to build Android on the TrueNAS host. Both QA passes' findings are re-applied on top of main's version: this workstation as a third box in "know which box you are on", the native build + emulator-driving traps, the logging-proxy recipe, and the two pre-existing behaviours QA confirmed against main (a chat restored after process death never starts the fleet poll; one malformed host silently freezes the whole fleet view while still claiming "N / N online"). 308 tests, 0 failures — 281 on main plus 27 added here, no deletions.
…calModel
Third QA pass. The five earlier fixes all held on device, but the guard I added
for the merge-conflict finding did not do what it claimed.
- **`fromFleet`'s test asserted four of its five fields and skipped
`tunnelOnline`** — the one the conflict was actually over. QA deleted it,
rebuilt, and watched the chat header say "· live" while the hub reported the
tunnel down, with the whole suite green. It is worse than a lost warning
because the field defaults TRUE: the header then asserts the tunnel is up.
Both tests now name every field, and say to add an assert whenever a field is
added there.
- **`ModelSwitchStore` had no test**, though it is a stated copy of `DraftStore`
and `.claude/rules/android.md` names `DraftStoreTest` as the convention for
exactly this class. Returning a fresh flow per call — i.e. restoring the whole
D1 bug — kept the suite green. `ModelSwitchStoreTest` mirrors DraftStoreTest's
three cases; both mutations are now caught.
- **The fixed-model chip dropped a tooltip that carried real information.** The
web's says why the chip is inert and how to get the picker back; the run
chip's only repeats the model name. That distinction was missing from
PARITY.md, which justified dropping both with the run chip's reason. The
static chip now carries the web's wording as a contentDescription — on a
MERGING wrapper, since Text writes its own semantics last and a description in
the same modifier chain never reaches the tree — and PARITY.md separates the
two cases.
- **`normalizeLocalModel` coerces the block at hub ingest**, the remedy this
repo already documents for `limits` and per-model usage and which `localModel`
shipped without. Android decodes /api/agents ATOMICALLY into typed fields, so
one host sending `available:"yes"` or a contextTokens past Int threw for the
whole array: every other host silently vanished from the phone while the tile
still read "N / N online". Verified live — a rogue host is now coerced to
`{available:false, model:null, contextTokens:null}` and the app shows 4/4.
The generalised rule is in CLAUDE.md's heartbeat contract, since it binds the
agent, the hub and three clients.
Left alone, with reasons: the three memo call sites in ChatViewModel are still
only covered by driving the app — no test in the project constructs a
ViewModel, so closing that means new test infrastructure, not a test. The pure
rules behind them are covered. Defects 7 and 8 from the second pass are
pre-existing and recorded in qa.md §6.2 rather than fixed here.
Android 311 tests, hub 928, agent 1258 — all green.
…n rule
Fourth QA pass came back PARTIAL. Three LOW findings, all fixed here.
- **`normalizeLocalModel` cut the model name with `slice(0, 60)`**, which counts
UTF-16 units: a name whose 60th unit is the high half of an astral pair
shipped a LONE SURROGATE to every client. Cosmetic in the app (Android draws
a replacement char) but genuinely unencodable — it kills `uiautomator dump`
outright, which is the tool a QA pass drives the app with. It cuts on code
points now, and the test pins the bound both ways; nothing pinned the length
at all before, so removing the cap entirely had kept the hub suite green.
- **The spawn composer's per-host lookup was inline in both screens**, so
`firstOrNull { it.key == host }` → `firstOrNull()` passed — the "fix placed in
the wrong loop" shape this repo has shipped before, and it would offer one
host's self-hosted model when spawning on another. It is
`ModelSource.hostLocalModel(agents, host)` now, tested, and the mutation fails.
- **The heartbeat rule I added to CLAUDE.md last commit was stated absolutely
and was already false.** "Any field Android decodes into a typed one needs a
normalize*" is not satisfied by the per-SESSION fields (`modelSource`,
`modelSourceAt`, `model`, `permissionMode`), which have no coercion and freeze
the fleet view the same way if an object lands in one — reproducible on main.
It is scoped to HOST-LEVEL blocks now, with the session-level gap named as
known, and it says that a `normalize*` is a whitelist that silently drops a
sub-key a newer agent adds.
Deliberately not fixed, and stated plainly rather than implied away: five
mutations still survive in the render/wiring layer — the two memo call sites in
`ChatViewModel`, the `StaticChip` description, and the two `if (…)` guards that
render the chip and the composer row. This repo has no instrumented or Compose
test source set (`android-ci.yml` runs `testDebugUnitTest` + `assembleDebug`
only), so closing them means new test infrastructure, not a test. Every one of
those behaviours was driven on the emulator by QA and by me; none of them is
protected by a test, and the PR says so.
Android 312 tests, hub 348 in server.test.js (928 across the suite) — all green.
… say why a spawn was refused Fifth QA pass. The feature held everywhere it was driven; these are the three things it found wrong, plus a rebase. - **Rebased onto b902de8.** main restructured `qa.md` and split `qa-findings.md` out of it, so the branch's copy conflicted. Resolved toward main and the five passes' notes re-applied onto the new structure — the WSL workstation as a third box in §0, a native-build + dedicated-AVD section under §2.5 (main's container recipe untouched), the rig/logging-proxy recipe, the merge-conflict-as-mutation and derived-boundary lessons in §5.6, and the two pre-existing Android behaviours in §6.2. - **The heartbeat rule I wrote two commits ago was still false**, and QA reproduced it: `capacity: {maxSessions: "eight"}` on one host drops that host from the phone while the tile still reads "N / N online". Only THREE blocks are coerced; every other host-level block Android types is raw. The rule now says that, as a live hazard rather than a solved one, instead of implying host-level is covered. - **`normalizeLocalModel` only stopped the hub MANUFACTURING a lone surrogate**; one arriving in a rogue agent's name was passed straight through, and the comment and test both read as though it were handled. Either direction kills `uiautomator dump` — the tool a QA pass drives the app with. Input surrogates are replaced now, and the test asserts both directions. - **A spawn the hub refuses said "hub unreachable".** `FleetViewModel.run` swallowed every exception into that one string, so choosing the local model on a host that lost its configuration reported a network fault for an 8ms 409. It prefers the hub's own message now, exactly as `ChatViewModel.setModelSource` already did; "hub unreachable" is left for a genuinely unanswered request. The Sessions screen still doesn't collect `messages` at all — pre-existing, and out of this branch's scope. Residual risk, stated rather than implied: 13 mutations survive, in two clusters — nine in Composable bodies and four at ViewModel call sites. Neither is an oversight; this project has no instrumented source set and no coroutine-test harness, so closing them is new infrastructure. Every one of those behaviours was driven on the emulator, including all five failure paths of the memo drop (409/500/401/HTML/socket-drop each revert the chip within ~1s and surface the hub's own text). Android 312, hub 348 in server.test.js — green. Merge with main clean.
…s, pin the wire Sixth QA pass. It found a real hole in the coercion, a fourth strike on the CLAUDE.md bullet, and two ungated wire-contract lines. - **`normalizeLocalModel` never ran on the `state.json` restore**, which is the hole that matters most: a hub restart is exactly when a new coercion ships, and the restore is the first thing it serves. A record holding a rogue block — or belonging to an OFFLINE host, where no beat will ever rewrite it — reached the phone raw and threw for the whole fleet, for up to the record's 7-day life. The loader now applies all three (`normalizeLimits` had the same hole and this branch had copied its precedent rather than `normalizeUsage`'s), with a test that fails if a coercion is added at ingest and not there. Proven by booting a hub on a state file holding `available:"yes"`: served coerced. - **The surrogate strip closed one case, not the class.** A C0 control in the model name kills `uiautomator dump` exactly as a lone surrogate does — the very failure the strip cites as its reason — and the app renders it fine, so it is invisible until the tooling dies. Both classes are stripped now. - **`CLAUDE.md`'s coercion bullet was wrong for the fourth time**: there is a FOURTH coercion, `sanitizeHeartbeat`'s `sanitizeLiveAgents`, and it is load-bearing — Android types `session.agents` too. It no longer states a count; it names the four, says to grep rather than trust the doc, and adds the restore requirement. The "every OTHER host vanishes" claim was also imprecise: that is the poll path, while per-agent SSE events decode individually, so with SSE healthy only the bad host is missing. - **The switch's own wire contract was ungated.** Renaming the `@POST` path or `ModelSourceRequest`'s field shipped green with the feature dead on the wire (404, or a 400 from the hub's enum check) — and unlike the render-layer survivors this needed no new infrastructure, since the sibling `SpawnRequest.modelSource` was already pinned. Both mutations now fail. - `qa.md`: the sixth pass's corrections, plus the two bullets this commit made stale (the restore hole and the surrogate-only claim) rewritten to match. Residual risk, corrected again: 14 mutations survive, in three locations — Composable bodies (8), ViewModel call sites (4), and `AppContainer`'s `modelSwitches` accessor. The two `HubApi` escapes are closed above. Every one of those behaviours was driven on the emulator; none is protected by a test, because the project has no instrumented source set and no coroutine-test harness. Android 313, hub 349 in server.test.js — green. No literal control bytes in either file (a NUL or a C0 makes git treat the source as binary).
…ranch typed
Seventh QA pass. Two of these are holes in my own previous fix.
- **The restore was throwing into its own `catch {}`.** `sanitizeLiveAgents`
reads two module `const`s declared ~1700 lines BELOW the restore, so at module
init they are in their temporal dead zone: the loop coerced `localModel`, hit
a ReferenceError on the session half, and the whole thing was swallowed —
half-coerced record, no log line, every suite green. The constants moved above
the restore with a comment saying why they can't live beside their function,
and a test asserts that ordering. This is the second time this branch has
shipped a coercion that didn't reach the restore; both were invisible.
- **The restore missed the fourth coercion**, `sanitizeLiveAgents`, while the
comment beside it claimed "every coercion the ingest path applies". Both paths
now go through one `normalizeRecord`, so adding a coercion covers both, and
the test asserts the two call sites rather than enumerating names — the old
test listed three and therefore could not notice the fourth missing, which is
exactly the shape that let the hole exist.
- **This branch made two session fields decode-fatal and coerced neither.**
Typing `modelSource`/`modelSourceAt` on `SessionInfo` is what did it: before
that, `ignoreUnknownKeys` skipped them and any value was harmless; after, an
object in either throws for the whole `/api/agents` array and every other host
vanishes from the phone. Measured at 7 of 12 hosts shown, no error. They are
coerced in `normalizeSessions` now, and CLAUDE.md states the general rule:
typing a field on a client and adding its hub-side coercion are one change.
- **The XML-illegal strip missed U+FFFE/U+FFFF**, the last two code points
excluded by XML 1.0's Char production, while the code and test both claimed
the whole class. They kill `uiautomator dump` exactly as a C0 does. Added,
with U+FDD0 and U+1FFFE asserted to SURVIVE so the strip doesn't overreach.
- CLAUDE.md's cold-start parenthetical was backwards (SSE carries the good hosts;
it is the poll-only case that loses the screen), and `qa-findings.md` still
claimed the split kept `qa.md` under a ceiling that does not apply to it.
Residual risk, with the seventh pass's measured numbers rather than my
arithmetic: **28 of 58 Android mutations survive**, in three locations —
Composable bodies (17), ViewModel call sites (10), and one `@Serializable` field
default. The hub half is 0 of 12. Nothing in `core/` survives. Those are all
render/wiring sites with no gate in this project (no instrumented source set, no
coroutine-test harness); each behaviour was driven on the emulator instead.
Android 313, hub 351, node 1031, agent python 1258 — green.
…-killing the hub
Eighth QA pass. The headline finding is in my own coercion, and it is the
worst thing this branch has produced.
- **`normalizeLocalModel` spread an agent-controlled string per code point
BEFORE bounding it**, and it runs BEFORE the `AGENT_RECORD_MAX` check that
exists to refuse an oversized beat. One agent-authed heartbeat with a 24 MiB
`localModel.model` OOM-killed the hub at its deployed `mem_limit: 256m` on
`node:24-alpine`, and `restart: unless-stopped` makes that a repeatable
outage loop of the fleet's whole control plane — every host's sessions,
terminals and in-flight migrations. Reproduced and A/B'd in a container at the
deployed shape: without the bound, `exit=137 oom=true` and the hub gone; with
it, 200 and still serving. The name is `slice(0, 512)`d first (far more than
the 60 code points that survive; a split astral pair is handled by the
surrogate replace immediately after), and a test holds it as a time+memory
budget rather than by reading the source.
- **`sessions: [null]` hid a host** — `normalizeSessions` skipped a non-object
element instead of dropping it, though `sessions` is typed `List<SessionInfo>`
on Android. Measured as a host silently absent while the tile still counted
it, and the whole screen replaced by the raw decoder exception with SSE also
down. Dropped now.
- A **non-array** `sessions` is deliberately still left alone: my first attempt
rewrote it to `[]`, which erased the amplifier `AGENT_RECORD_MAX` exists to
refuse — an 8 MiB string `sessions` went from 413 to 200 and the existing
XERK-235 test caught me. The rule, now written down: a coercion running
before that check may only ever SHRINK a record.
- **The TDZ test named two constants**, so a third would walk straight past it —
QA proved that with a new late-declared const that kept the suite green while
the restore silently threw again. It is behavioural now: load the real module
in a child process against a fixture holding every wrong shape at once, and
assert both the `loaded N agents` line and the coerced result.
- **A refused spawn from the Sessions screen was silent** — that pane collected
`vm.messages` from nothing, which was survivable while its routes only failed
on the network, but this branch gives it a first-class 409. It has a
SnackbarHost now; verified on device (`✗ hub unreachable` where there was
previously no feedback at all, and the hub's own text on a real refusal).
- Two `qa.md` statements my own last commit had falsified, corrected again.
Residual risk is unchanged and disclosed in the PR: no real agent/claude/gateway
in the loop, and 28 of 58 Android mutations survive in the render/wiring layer
(17 Composable bodies, 10 ViewModel call sites, 1 `@Serializable` default) —
nothing in `core/`, `data/` or `net/`, and the hub half is fully gated.
Android 313, hub 352, node 1032, agent python 1258 — green. Merge clean.
…stic
Ninth QA pass. The ordering insight in its second finding resolves three
things at once.
- **`normalizeRecord` now runs PAST the `AGENT_RECORD_MAX` gate**, not before
it. Before was wrong in both directions: a coercion there shrinks away the
amplifier the gate exists to refuse (my own attempt to rewrite a non-array
`sessions` turned an 8 MiB string from 413 into 200, caught by XERK-235's own
test), and it walks an oversized record field by field before throwing it out
— which is how a 24 MiB name reached a spread in the first place. Past the
gate the record is bounded, so a coercion is free to REWRITE. `sanitizeHeartbeat`
stays pre-gate and keeps only the shrink-only half.
That unlocks two fixes QA had filed as forced trade-offs:
- A non-array `sessions` is rewritten to `[]`. It doesn't merely hide that host
— measured, the app **cannot sign in at all**, because the login probe
decodes `/api/agents` and the throw reads as "Could not reach the hub".
- `sessions:{a:1}` in a `state.json` no longer aborts the whole restore.
`normalizeUsage` iterates `sessions || []`, so a non-iterable threw into the
restore's silent `catch {}`; `normalizeSessions` runs first now.
- **The OOM regression guard was nondeterministic** — 8 MiB against 50ms/64MB
caught the reintroduced bug only 5 runs in 8, decided by whether a GC landed
between two samples. A resource budget can't be made reliable at a 1.02x
margin, so it is now a structural assertion (the spread must be preceded by a
slice) PLUS a budget at 32 MiB with ~10x headroom. Measured: 6/6 catches with
the bug reintroduced, 4/4 clean.
- **The spawn composer kept a hidden `local` choice** after the row that shows
it disappeared, so a host losing its configuration mid-composer produced a
guaranteed 409 with nothing on screen to explain or change. It resets.
- Three more `qa.md` statements my own commits had falsified.
Residual risk, unchanged and disclosed in the PR: no real agent/claude/gateway
in the loop, and 30 render/wiring mutations with no gate (19 Composable bodies,
10 ViewModel call sites, 1 @serializable default) — nothing in core/, data/ or
net/, and the hub battery is now 12/12 caught.
Android 313, node 1068, agent python 1258 — green. Merge clean. Memo re-driven
on device after the rebuild: ☁ → 🏠 → survives HOME → survives leaving the chat
→ ☁ at t+65s.
Tenth QA pass. My previous commit's "coerce past the gate" was wrong in the
other direction, and it produced two HIGH regressions against main. Both
demonstrated end to end, both now closed and mutation-tested.
- **An expanding coercion escaped the ceiling.** `normalizeModelUsage` rewrites
`"m"` to `{model:"m"}` — ~3.5x — so measuring only the pre-coercion size let
an 8 MiB beat of bare model names park 28 MiB per host, for the record's whole
7-day life, in `state.json`, in every `/api/agents` response and every SSE
frame. Exactly the amplification XERK-235 added the ceiling to stop. Measured:
branch 200 and a 29 MB payload, main 413 and 139 bytes.
- **A throw inside the coercion left the RAW record installed**, because it now
ran after `agents[key] = next`. `normalizeUsage`'s bare `for (… of repoUsage
|| [])` throws on an object; the hub answered 400 and served the host anyway.
Worse than refusing: uncoerced `localModel.available:"yes"` is truthy, so
`localModelAvailable` handed out the switch a host cannot honour, and the
poison reached `state.json`, where the restore then aborted into its silent
`catch {}` and left every host after it uncoerced on every boot.
The fix is neither before nor after but BETWEEN: measure the raw size (the
amplifier check a shrinking coercion must not defeat), coerce, then measure the
stored size (which an expanding coercion must not escape). Coercion sits in the
middle, so it never walks an unbounded record — the property that kept the OOM
fix honest — and a throw is caught and rolls the record back, so "accepted" can
never mean "accepted and raw".
Also: `normalizeUsage` guards its iterations with `Array.isArray` instead of
`|| []`, and REWRITES a non-array `repoUsage` to `[]` rather than stepping
around it — it is `List<RepoUsage>` on Android, so serving `{a:1}` is
fleet-fatal for the phone. Guarding the loop alone would have turned main's
accidental 400 into a 200 serving that shape.
Five mutations now caught that were not: coerce-before-the-raw-check, drop the
coerced check, unguarded `repoUsage`, drop the non-array `sessions` rewrite, and
the ingest ordering itself. The order INSIDE `normalizeRecord` is deliberately
no longer load-bearing — each coercion guards its own input shape — and the
comment says so rather than justifying an order that no longer matters.
Two more `qa.md` statements corrected (the dropdown opens anywhere in the field,
not only its caret; a dump costs ~2s, not 30-60s — both measured this pass).
Residual risk unchanged: no real agent/claude/gateway in the loop, and the
render/wiring mutations with no gate in this project.
Android 313, node 1071, agent python 1258 — green. Merge clean.
xerhab
marked this pull request as draft
August 12, 2026 15:50
Contributor
Author
QA verdict: FAIL — converting to draftAn adversarial 16 of the 17 claims in the description above were confirmed, several with measurements. The failure is one incomplete fix inside this branch's own new coercion, plus a reachable dead-end in the new chat control. Blocking — caused by this changeD1 (HIGH) — an array element in const bad = payload.sessions.some((s) => !s || typeof s !== "object");
One agent-authed heartbeat carrying It carries through the D2 (MEDIUM) — the "run against" chip vanishes on an unconfirmed switch and never returns. D5 (MEDIUM) — two of the new behaviours have no gate that can turn red. 11 of 25 Android mutations escaped D7 (LOW) — the rollback backstop at Confirmed, with measurementsThe ingest ordering holds under attack: a 9.4 MiB string Filed separately (pre-existing, not caused by this branch)
Not verifiedNo real agent/ |
Eleventh QA pass — the first run by a session other than the one that wrote this branch, against the pushed tip. Sixteen of its seventeen claims held, several with measurements. These are the three that did not, two of them inside this branch's own fixes. - **An ARRAY element in `sessions` survived the new filter and blocked sign-in entirely.** `normalizeSessions`' predicate was `!s || typeof s !== "object"`, and `typeof [] === "object"` — so the element it was written to drop was the one shape it passed through. The comment above it named `null` and a bare string; the fixture used `null` and a bare string; the third non-object shape was in neither, so the fix read as complete from every angle except running it. Not merely a hidden host: the login probe decodes `/api/agents`, so the throw reads as "Could not reach the hub" and the app cannot sign in at all. Predicate is one `objectish` helper now, the fixture carries every non-object shape (`null`, string, `[1,2]`, `[]`) and asserts the surviving ids rather than a count of drops. - **The switch memo's TTL was a timer that never fired.** It aged out inside `canSwitchModelSource()`, read from `System.currentTimeMillis()` at composition time, and Compose skips recomposition while the state compares equal — so on a quiet fleet nothing re-read the clock. Reproduced twice: switching away from `local` on a host that had lost its `localModel` removed the whole "run against" chip, still gone at t+120s, with the bar reading `model: default` while the session was in fact still on the self-hosted model and no way on screen to retry or see the truth. Expiry now RETIRES the memo from the store — a state change, which is what repaints — via `settle(…, now)` plus a bounded per-memo alarm armed off the store, so a memo carried in from another nav entry is covered too. This was also a parity break: the web recomputes `localModelOffered()` unconditionally in `onPoll` every beat. - **`setModel`/`setMode` reported every refusal as "✓ model queued"**, having discarded their `Result`. Survivable while those routes only failed on the network — and then this branch's own hub commit gave `/model` a first-class 409 for a session on the self-hosted model, added precisely so an out-of-parity client could not silently drop the command. Reachable through the memo state above. Both now report the hub's own words, exactly as `setModelSource` and `FleetViewModel.run` already do. All three are mutation-checked, and the two Android ones went to `core/` rather than staying at their call site for the reason this branch already extracted `afterAttempt`: a ViewModel call site has no gate in this project, so a decision left inline is one a mutation deletes unnoticed. Reverting each fix fails its test — `Array.isArray` dropped: 1 failure; the expiry drop deleted: 1 failure; `outcomeMessage` forced to its success arm: 1 failure. `qa-findings.md` §5.8 records the three shapes, since two are §5.3 and §5.4 recurring one round later inside the code that fixed them. Android 315, node 1071 — green. Instruction files under the 40k cap. Merges clean. Not addressed here, ticketed instead: the ungated render/wiring mutations and the unreachable rollback backstop (XERK-262), the hub OOM at two concurrent large heartbeats (XERK-258, pre-existing), uncoerced `repoUsage` elements (XERK-259, pre-existing), and the unbounded command queue (XERK-261).
`/api/agents` decodes atomically on Android, so one host's wrong-typed value
throws for the entire array. The hub coerced three blocks by hand, which was
never the boundary that matters: a field is decode-fatal the moment a client
TYPES it, and Android types nearly all of AgentInfo. `repoUsage: [null]` — four
bytes from one host — made the app refuse to sign in at all ("Could not reach
the hub"), because the login probe decodes that payload.
turma/wire-shape.js is a table mirroring Models.kt plus the walker that applies
it, run from normalizeRecord (so both the heartbeat ingest and the state.json
restore get it). Two rules it exists to enforce: a list's ELEMENTS are as fatal
as its type, and `typeof [] === "object"`, so element tests go through
isPlainObject.
It coerces IN PLACE, touching only the keys it names, so it is not a whitelist:
a sub-key a newer agent adds rides through untouched rather than being dropped
fleet-wide until the table catches up. Values become the "can't tell you" one
every client already handles — "" / false / 0 / null / [] — and a nullable field
becomes null, never a zero that reads as a real measurement.
Its own module because server.js restores state.json at module init: a const
declared below that point is in its temporal dead zone, and the ReferenceError
dies in the restore's own `catch {}`, leaving records half-coerced with nothing
logged. Ordering inside normalizeRecord IS load-bearing and is documented: the
sweep runs last, after normalizeModelUsage rewrites an old agent's bare
model-name strings.
Tests: the hub suite pins the four reported shapes, element drops in every typed
list, the can't-tell values, map values (no coerceInputValues there), blocks with
a non-string discriminator, and that untyped keys survive. One test parses
Models.kt and walks it, so typing a field there without adding it here fails CI
rather than waiting for a phone to stop signing in. AgentDecodeTest pins the
client half — each shape is asserted decode-fatal, and the record the hub now
serves decodes beside a healthy host.
Resolves one conflict, in `qa-findings.md`: main added its own §5.8 and §5.9 (XERK-254's thirteen rounds) while this branch appended a §5.8 of its own. Both are kept — main's keep their numbers, this branch's becomes §5.10 — since they record different rounds and neither supersedes the other. Everything else merged clean. Main's new `QA.md` is a separate file from this branch's `qa.md`, not a rename of it. Merging rather than rebasing: the eleven commits below carry ten QA passes' evidence in their messages, and rewriting them would detach that from the code it describes. Merged because CI could not run at all while the branch was conflicted — a `pull_request` workflow builds the merge commit, and GitHub cannot create one for a DIRTY branch, so zero checks were queued on 822ecf9. Suites on the merged tree: node 1073, agent python 1277, Android 315 — green. Instruction files all under the 40k cap.
An adversarial QA pass failed the first commit on its own exemption. The sweep skipped `limits` on the grounds that `normalizeLimits` rebuilds it — but a rebuild is only as good as its own gates, and that one gated both epoch fields on `Number.isFinite`. Kotlin's `Long` takes no fractional literal, so `resetsAt: 1.5` from one host was served raw and threw for the whole `/api/agents` array: the same whole-fleet symptom this ticket exists to close, sitting inside the function meant to prevent it. Reproduced with one curl against the built image and confirmed with the real Kotlin decoder. - `normalizeLimits` now bounds `resetsAt`/`capturedAt` with `Number.isSafeInteger` (`usedPct` is a Double and keeps the looser check). A bad stamp drops that field, and an unusable `capturedAt` nulls the block — its existing degradation. - `limits` and `localModel` are in the shape table too, so they get both the type backstop and the Models.kt drift test. Nothing agent-authored is exempt now. The drift test also had two vacuous-pass modes, which is worse than the hole: it is the thing that stops this class of gap recurring. Its per-line field regex skipped every field with NO default — the dangerous kind, since those are required on the wire — and saw only the first field of a one-line `data class` (the transcript blocks). It now splits the constructor on top-level commas after stripping comments, and asserts field counts that a mis-parse drops first, so a parser that silently sees nothing can no longer read as "all covered". Re-ran the sabotage matrix: both escapes are now caught, as is dropping `limits` from the table. Also fixed inline, both found by the same pass: nothing gated `turma/Dockerfile`'s hand-maintained COPY list, so a future module omitted from it would be a container that boots to MODULE_NOT_FOUND — now a test over server.js's own local requires. And AgentDecodeTest pins the client half of the limits shape.
Twelfth QA pass, on the merged tip. It confirmed all three of the eleventh
round's fixes with fresh evidence — including a decisive isolation for the memo
alarm, driven with the fleet's heartbeats fully PAUSED, so the control returning
at t+58-63s is the alarm doing the work and not fleet churn. Every finding below
is a defect in one of those fixes.
- **The `typeof [] === "object"` fix was one level too shallow.** The identical
predicate sat FIVE LINES below the one I fixed, on `s.session` — and
`"agents" in []` is false too, so an array fell through both halves of that
guard. One heartbeat of `sessions:[{id:"s1",session:[]}]` reproduced the
original symptom exactly: served raw, and the app could not SIGN IN, because
the login probe decodes /api/agents and reads the throw as "Could not reach
the hub". The predicate is now one hoisted `objectish()` helper used at all
three sites, so there is nothing left to grep — a `function` declaration
rather than a `const`, because `sanitizeHeartbeat` calls it ~70 lines above
its definition and this file has already shipped a coercion that died of
exactly that TDZ.
I audited the remaining `typeof x === "object"` predicates rather than
assuming: `normalizeLimits` and `sanitizeLiveAgents` both have it, and both
are SAFE, because each rebuilds a whitelist from scratch — an array falls out
at the next field read instead of reaching a client. Recorded, so the next
pass does not re-derive it.
- **My first version of that fix did not actually fix it**, and the test is what
caught me: guarding the SANITIZE with the right predicate leaves the raw array
in the record, which is the thing that gets served. A non-object `session` is
REWRITTEN to null now — the "can't tell you" value every client already reads
— which is legal here only because normalizeSessions runs past the record gate.
- **The memo alarm could sleep through a backward wall-clock jump.** It derived
its delay from `currentTimeMillis`, slept on `delay` (uptime), then re-read the
wall clock; a backward jump made the re-check false, `settle` returned the SAME
instance, `MutableStateFlow` did not emit an equal value, the collector never
ran, and no new alarm was armed. Measured at t+190s after a 10-minute jump,
chip claiming the subscription while the record said local. It retires by
IDENTITY now — `compareAndSet(pending, null)` — which cannot be undone by a
clock, and no-ops if a newer switch or a settle already replaced the memo.
- **`reportedOutcome` branched on the HTTP status alone**, ignoring
`OkResponse.error`, so a `200 {ok:false,error:…}` would read "✓ model queued".
Latent, not live — no route answers that shape for /model or /mode today — but
that is precisely the position /model was in before it grew its 409, which is
how the bug this replaces was born. `FleetViewModel.run` already checked it;
the two sides agree now.
- The TTL test asserted 60_999 and 61_001 and never 61_000, so `>=` → `>`
survived a mutation battery. The exact edge is asserted.
Mutation-checked, five new: deleting the session rewrite, dropping
`Array.isArray` from `objectish`, weakening the rewrite to a falsy test,
`expired`'s `>=` → `>`, and ignoring `bodyError` — each fails its test.
**Honestly ungated:** the `compareAndSet` change itself is a ViewModel call
site, so nothing in the suite can catch its deletion. That is XERK-262 (11 of 25
mutations escape in vm/ and ui/, no coroutine-test harness exists), not
something this commit can close without building that harness.
qa-findings.md §5.11 records the four shapes.
Suites: node 1073, agent python 1277, Android 316 — green.
Second QA pass: the product fixes hold (limits dead across ingest, restore, a real container and 2000 re-fuzzed mutants — zero decode failures), but two of the guards I added to stop this class of bug recurring could still be walked through. A regression net with a hole in it is the thing that let XERK-259 exist. - The Models.kt splitter counts bracket depth and does not know string literals, so a default containing a bare `>`/`(`/`[` desynchronised it and folded the rest of the constructor into one param. The field-count FLOOR masked that only at today's count: QA proved a class grown by four fields — an ordinary paired change — then passes while silently not checking a field. Replaced the floor with a per-class count equality against the `val`s in the same comment-stripped body, which cannot drift as the class grows. - `data class X` with its `(` on the next line was invisible to the class regex, and a field of that type then skipped sub-shape validation. `\s*` around it. - The Dockerfile test matched only double-quoted, top-level, same-file requires, so a nested path, a single-quoted require, and one two hops out all escaped — each a container that boots to MODULE_NOT_FOUND with a green suite. It now walks the whole require graph from server.js and matches either quote style, a path segment, and ignores names appearing only in a Dockerfile comment. Verified by re-running every case that escaped: all six caught, the original control still caught, and a legitimate paired addition (field added to Models.kt AND the table) still green — a guard that fails on correct changes gets deleted.
Third QA pass: N1/N2/N3 are closed and the count-equality assert survived 18 engineered attempts to defeat it, but the fixes had introduced the mirror image of the bug — neither guard knew a string literal from code, so both now rejected LEGAL input. A loud failure on a correct change is what gets a guard deleted, which is the same way this class of bug gets back in. - The Models.kt parser read a default's CONTENTS as syntax: `"https://x"` hid a comment, `"/*"` opened one, `">"` broke the bracket depth, and `"use val x: Int"` invented a field. Each failed a paired, correct change with a message accusing the reader of breaking the parser. One length-preserving pass now blanks string/char/raw-string bodies and comment bodies before anything reads it as syntax, so offsets stay valid and defaults are text. - Two of the parse canaries were exact field counts, and both rot: a third field on TextBlock is an ordinary change, and CreateTicketRequest is not even on the agent record. The per-class count equality already proves every field of every class is visible, generically, so these are back to "was this class seen at all". - The require walk read prose as code: this repo's comments name modules constantly, so a stale note about a deleted one failed CI claiming it was required but missing. Sources are comment-stripped now, the existence assert is gone (a genuinely missing module takes the suite down at its own require, so that assert could only fire on a false positive), and a statically-written template-literal require counts. Verified both directions: 11 legitimate-change cases that used to fail are green (five string-noise defaults, the two rotted counts, prose in a comment and in a string), and all 9 catches still bite (M2, M14, T2, N2, dropping limits from the table, and all four Dockerfile shapes). Unrelated flake seen once and worth naming: "normalizeLocalModel bounds the name BEFORE spreading it" asserts wall-clock < 60ms and failed once under load, then passed 3/3. It belongs to #430, and this branch is not the cause — the sweep costs 0.01ms on that record, measured.
Fourth QA pass: P1/P2/P3 closed and no silent loss survives anywhere — every
Kotlin edge that desynchronises the parser now fails loudly through the
count-equality assert. Two shapes still mis-read a literal, one of them realistic
enough to redden CI on an ordinary edit.
- `blankNonCode` had no regex-literal branch, so a quote inside a character class
opened a phantom string that ran to the next matching quote — usually EOF —
leaving every comment in between unblanked and re-arming the exact P3 false
positive this was meant to kill. Not hypothetical: `/[.,;:!?'"]+$/` exists
verbatim at turma/public/chat.js:182, and adding `'` to server.js:194's
character class would have been a one-character edit that disarmed
comment-blanking for the rest of the file. Fixed without a regex-vs-division
heuristic: neither language lets a `'`/`"` literal span a line, so one that
reaches a newline unclosed was never a string and is emitted as code.
- A Kotlin raw string may END with a quote (`"""ab""""`), and closing at the
FIRST `"""` of the run left a stray quote that opened another phantom string.
The closer is now the last three of the run.
The third finding — a require written inside a string still reads as a real one —
is inherent to needing string contents to read the path out, so it is documented
as a bound beside the pattern rather than papered over.
Verified: QA's three regex shapes plus the server.js:194-with-apostrophe variant
are green as ordinary edits, while a real COPY omission is still caught WITH a
regex present; the raw-string field is green paired and caught unpaired; 21
unit shapes over blankNonCode hold (escapes, nested comments, `${}` templates,
char literals, multi-line templates); and all four real hub sources plus
Models.kt still blank to zero unblanked comment lines with length, newlines and
all 380 `val`s preserved.
Three conflicts, and one CLEAN auto-merge that was worse than any of them. - **`SessionsScreen.kt` merged with NO conflict and did not compile.** Both sides added the same snackbar independently — main for XERK-264's refusal wording, this branch because the "Run against" row gives that pane a first-class 409 — so git kept both: two `SnackbarHostState` imports, two `val snackbar`, two `SnackbarHost`s, and this branch's copy left OUTSIDE main's new `Box` (no `BoxScope`, so `align` unresolved) with a stray brace that broke the parse. Main's copy is kept, correctly placed; the one thing only this branch knew is folded into its comment. This is `qa.md` §5.6's merge-conflict-as-mutation, in its nastier form: git reported success. - **`ChatViewModel.kt`** — main refactored `setModel`/`setMode` onto a shared `report()` helper (XERK-264) while this branch put them on `reportedOutcome`, the same idea arrived at independently. Merged INTO main's helper rather than beside it, because two report paths is how the halves drift: `report` now takes the optional route-specific `failed` and reads `OkResponse.error`, so `kill()` and every other caller gets the 200-with-an-error fix too, and `reportedOutcome` is gone. Taking either side mechanically would have deleted `setModelSource` entirely — the compiler catches that one, which is the only reason it is the less dangerous of the two. - **`FleetViewModel.kt`** — main landed the identical `hubErrorMessage(e) ?: "hub unreachable"` line under XERK-264, PLUS `pendKeys` clearing that only it has. Taking "ours" here compiles clean and passes every test while silently dropping main's feature, leaving a stale spinner on every failed fleet action. Main's side kept whole; this branch's now-redundant comment dropped, since main's docstring says it better. - **`CLAUDE.md`** — not a real conflict, two independent additions to one list. Both kept, the decode-atomicity sub-bullets nested under the heartbeat bullet where they belong. Also folded in while here: `setModelSource` had the same latent `OkResponse.error` gap. It matters more there than anywhere else, because the verdict drives the MEMO as well as the wording — a refusal the hub answered 200 with would have left the chip claiming a switch that was refused. Both now read one `ModelSource.accepted(ok, bodyError)`, so the memo's fate can never be derived from the rendered string. Suites on the merged tree: node 1093, agent python 1288, Android 316, APK assembles. Instruction files all under the 36k warn line.
QA's final pass returned PASS after driving the real Android client on an emulator — the app signs in and lists every host against this branch, and shows "Could not reach the hub" against the same tree with the coercion stubbed. Its one remaining LOW was a backtick inside a JS regex re-opening the P3 false positive: backticks are exempt from the newline rule (template literals may legitimately span lines), so `` /`([^`]+)`/g `` — which glasses and veiller both contain verbatim — opened a phantom literal that swallowed the comments after it. The suggested one-liner (treat an unclosed backtick at EOF as code) does not actually fix it: the phantom literal closes on the next REAL backtick in the file, not at EOF. Verified before discarding it. So `blankNonCode` now lexes regex literals properly, JS-only (Kotlin has none, and `a / b / c` would read as one), deciding regex-vs-division on the previous significant token and scanning to the closing `/` on the same line, stepping over `[...]` classes. That turned up a second live mis-blank on the way: `/\//g` in push.js:48 — a regex whose escaped slash is followed by its closing one — read as a `//` comment and blanked the rest of that line. Harmless there, but the same shape one line further up would have hidden a require. Verified the differential rather than assuming it: every character that now blanks differently sits inside a regex literal, nothing spills across a newline, and the requires recovered from all four hub sources still equal ground truth exactly. Eight end-to-end shapes green (three backtick regexes, the Q1 quote regex, a regex containing `//`, plain division, a multi-line template) with a real COPY omission still caught alongside a backtick regex.
# Conflicts: # CLAUDE.md
XERK-259: hold the whole agent record to the shape clients type
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes XERK-246 for the Android client — the local-model failover controls the web UI already has.
What it does
Running out of Claude usage stops every session on a host at once, and that happens while you're away from a desk — the phone was the one surface that could do nothing about it. Two controls, matching the web:
.../sessions/<id>/model-source. The agent relaunches with--resume, so conversation, worktree and branch carry over.Both follow the host's
localModel.availableexactly as the 📎 followsuploadMaxBytes: an agent reporting nothing cannot do it and the hub 409s the command, so the control is hidden rather than offered and refused. The chip is shown anyway when the session is already local, so a host that later lost its configuration keeps a visible way back.The chip paints from a memo (
AppContainer.modelSwitches, not the chat VM — the VM dies when you walk back to the session list, mid-switch, which is when the memo is doing its job) until the heartbeat agrees; it ages out atSWITCH_SETTLE_MSso a switch that never lands can't pin it on a lie, and a refusal drops it at once and says why.The Claude model picker is hidden on a live local session (a static chip states the model instead), matching the web's
cc-model-fixed: every alias it could offer — "default" included — is one the self-hosted endpoint refuses. The spawn composer keeps its Model row, also matching the web.Pure half in
core/ModelSource.ktso the Compose screens stay thin renderers.android/PARITY.mdnarrows the gap to the remaining 🏠 mark on session cards.Hub-side hardening (
turma/server.js)Typing
modelSource/modelSourceAton Android'sSessionInfois what forced this: before,ignoreUnknownKeysskipped them and any value was harmless; after, an object in either throws for the whole/api/agentsarray and every other host vanishes from the phone (measured at 7 of 12 hosts shown, no error). So the branch also closes the coercion holes that turned up around it.The load-bearing outcome is the ingest ordering, arrived at over three passes:
AGENT_RECORD_MAXexists to make — a shrinking coercion must not defeat it),normalizeModelUsagerewrites"m"to{model:"m"}, ~3.5×).A throw in the middle rolls the record back, so "accepted" can never mean "accepted and raw". Ingest and the
state.jsonrestore both go through onenormalizeRecord, so adding a coercion covers both, and a test asserts the two call sites rather than enumerating names.Also closed here: an agent-controlled model name spread per code point before being bounded (see QA below), non-array
sessions/repoUsagerewritten rather than stepped around, the full XML-illegal code-point class including U+FFFE/U+FFFF, and a TDZ bug where the restore's coercion threw into its owncatch {}at module init and left every suite green.QA
Ten adversarial
qapasses on the originating session, each driving a real emulator (turma228) and a real hub. They found defects in the branch's own fixes repeatedly; the ones worth naming:normalizeLocalModelspread an agent-controlled string per code point before bounding it, and ran before theAGENT_RECORD_MAXgate. One agent-authed beat with a 24 MiBlocalModel.modelkilled the hub at its deployedmem_limit: 256monnode:24-alpine;restart: unless-stoppedmakes that a repeatable outage loop of the fleet's whole control plane. Reproduced and A/B'd in a container at the deployed shape: without the boundexit=137 oom=trueand the hub gone, with it 200 and still serving.state.json, in every/api/agentsresponse and every SSE frame. Measured: branch 200 and a 29 MB payload, main 413 and 139 bytes.localModel.available:"yes"is truthy, so it handed out a switch the host cannot honour, and the poison reachedstate.jsonwhere the restore aborted into its silentcatch {}and left every later host uncoerced on every boot.sessionsblocked login entirely — the login probe decodes/api/agents, so the throw reads as "Could not reach the hub". Not just a hidden host.vm.messagesfrom nothing so a refusal there was silent. Both fixed; verified on device.Every new test case was mutation-checked. Suites at the tip: Android 313, node 1071, agent python 1258 — green. Merges clean against current
main.Residual risk — stated plainly
@Serializabledefault. Nothing incore/,data/ornet/survives, and the hub half is fully gated. The project has no instrumented source set and no coroutine-test harness; each of those behaviours was driven on the emulator instead.Tests
core/ModelSourceTest.kt,vm/ChatUiStateTest.kt,vm/SpawnRequestTest.kt,data/ModelSwitchStoreTest.kt— the pure half, the state reads, and the spawn payload.model/AgentDecodeTest.kt— the wire block pinned, including the all-nulls shape an unconfigured host reports and the nullmodelSourceAtevery unmoved session carries.turma/tests/server.test.js— the ingest ordering, both size gates, the rollback, the restore's coercion parity and its module-init ordering, the XML-illegal class, and the OOM budget.Provenance
The authoring session (
869a5) hung before it could push. This PR was opened by a separate session that located the worktree and pushed the branch verbatim — no code was written, rebased or re-verified here, and no freshqapass was run against the tip; the QA evidence above is the authoring session's, transcribed from its commit messages. The one change made was deleting a stray untracked.probe-write-checkscratch file.