Fix runner reconciliation wake ordering and add npm CLI packaging - #64
Merged
Conversation
…ed (#67) * fix(runnerd): reset the redial backoff once a connection is established RunAgent's backoff was initialized once per process and doubled on every disconnect, and nothing anywhere put it back. A runner whose control plane routinely ends connections — an hourly credential or lease expiry, a rolling deploy — reached the 30s cap after six of them and redialed at 30-45s for the rest of that process's life, however healthy every connection in between had been. For the whole of that gap every session on the runner reads unreachable and the scheduler can place nothing on it. The sibling loop this one is modeled on, cmd/sessiond's dialLoop, has always reset on a successful dial; this one never grew the line. The delay now returns to its floor whenever a connection was ESTABLISHED, which here means exactly one thing: controld answered this connection's announce with an accept. That is the control plane's own statement that the fleet registered this runner, at a generation, on this socket — the point past which its sessions are dispatchable. Establishment rather than a completed dial, because several of controld's refusals come after the websocket handshake and before any accept (a name the credential does not cover, a generation its store cannot mint, a registration the fleet refuses, a failed reconcile), and a store outage puts the whole fleet on that path at once. Only an accept means the connection was good. Measured at the defaults against a fake control plane, seven consecutive clean closes each cost 1.04-1.45s of reachability, where before they cost 1.4s, 2.1s, 4.7s, 8s, 16s, 30s, 30s... The floor sits on AgentConfig beside the ping bounds so a test can shorten it without touching package state. Two tests: ten established-then-dropped connections must each redial at the floor — the old behavior overruns the harness's 5s bound on the ninth — and four connections dropped before any accept must keep doubling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KRpP2YdobVkHaNetLMBZaL * fix(runnerd): require an established connection to hold before resetting Review finding: keying the reset on the bare accept removed a brake that the old always-growing delay had provided by accident. The control plane evicts a runner's previous connection whenever a new one registers under the same name (runnerplane.registerRunner), so two runnerd processes configured with one runner name accept-and-evict each other indefinitely. Every dial in that loop mints a generation and writes a fleet row, so resetting on the accept alone would have held a misconfiguration that used to decay to one exchange per 30s at one per second, forever, inflating generations the whole time. The reset now needs the connection to have been established AND to have lasted RedialResetAfter (10s). Any routine lifetime is orders of magnitude past that bound — the hourly expiry this change is about is 360 times it — so the fix keeps working, while a flapping pair backs off to the cap as before. Measured again at full defaults with real 10.5s holds: five consecutive clean closes cost 1.196s, 1.288s, 1.328s, 1.466s, 1.196s. Establishment is now a local in agentSession rather than a field on agentSessionState: the accept is handled on the reader, which is the goroutine that returns, so nothing else ever reads it and it needs no atomic. Also from review: redialBackoffMin() clamps to [1ms, 30s]. Under about 2ns it reached mrand.Int63n with a zero bound, which panics and takes the process down on its first redial; above the cap it put every redial after a held connection past the ceiling RunAgent documents. The cap is a named constant now, shared with nextBackoff. And establishAndDrop's comment had its measurement bias backwards — a gap timed from the close includes the agent's notice latency, so it is an upper bound on the agent's wait, not an under-report. Two more tests: four accepted-but-immediately-dropped connections must keep backing off (74ms instead of the required 400ms without this change), and the floor's clamp at both ends. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KRpP2YdobVkHaNetLMBZaL --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…le (#66) * fix(image): install the complete Codex package, not just its executable `codex --search` failed closed in a session on a missing /usr/local/bin/codex-code-mode-host. Codex is a package, not a binary. The upstream `codex-package` archive is a manifest, `bin/codex`, the `bin/codex-code-mode-host` tool host, and bundled ripgrep, bubblewrap and zsh under `codex-path` and `codex-resources`, and Codex finds every companion by walking up from the RESOLVED path of its own executable until it finds `codex-package.json`. The toolchain script fetched `codex-${TRIPLE}.tar.gz`, which is that one executable and nothing else, and installed it into $BIN. `codex --version` was green; the first turn needing the tool host was not. Measured against the real rust-v0.153.4 archive with `codex doctor --json`, which reports Codex's own resolution rather than ours: complete package install context names package/bin/resources/path bin/codex alone bare "other"; ripgrep resolves to the OS one bin/codex + host on PATH byte-for-byte identical to the row above package minus the tool host doctor still passes; nothing notices symlink on PATH -> bin/codex identical to the complete package So the tool host goes neither on PATH nor beside sessiond in /usr/local/bin, a symlink is how the PATH entry reaches the package, and the layout has to be asserted separately because doctor does not catch a deleted host. toolchain.sh now fetches `codex-package-${CODEX_TRIPLE}.tar.gz` for both architectures, pinned by the sums published in the release's own codex-package_SHA256SUMS and checked before extraction as every other artifact here is. The tree lands at $PREFIX/lib/codex and $BIN/codex is a RELATIVE link into it, so the same link resolves in the toolchain stage and again after the final image copies both directories under /usr/local. require_codex_package rejects a manifest for another version, target or layout and any missing or non-executable member, so a build that lost a piece fails instead of shipping. The package is root-owned under /usr/local for the reason sessiond and the other agents are: a session user who could rewrite the tool host could rewrite what the agent spawns. CODEX_VERSION is unchanged at 0.153.4. Sourcing toolchain.sh now exposes the validator and installs nothing, which is how the Go tests drive it against synthetic packages. scripts/session-image-smoke.sh asks the installed codex where it resolved its own package, requires the bundled ripgrep rather than the OS one, and runs the tool host it names — offline, with no credential. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDaM4pHz5JRixZrG1yEr85 * fix(image): escape manifest values in the Codex layout check The check built its grep pattern from the version and target triple directly, so a dot in either matched any character. Only the whitespace after the colon is meant to be a pattern; the values are literals. A build-time assertion on a checksum-pinned archive, so nothing was actually loose, but a check that reads as approximate invites being trusted approximately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDaM4pHz5JRixZrG1yEr85 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: restore Claude onboarding state with credentials * harden agent home restoration * fix: fence agent credential restoration * fix: preserve agent logout fences * fix: fail closed across credential upgrades * fix: reject empty credential manifests
A session could not run an integration test. There is no way to get a database after the fact — `sudo` is not installed, the rootfs is read-only, and no package archive is on the egress allowlist — so the client AND the server halves belong in the image, which is the one thing a Dedicated runner is allowed to pull. Nothing starts. `create_main_cluster = false` keeps postgresql-common from running initdb into a directory the read-only rootfs could never use, and a policy-rc.d refuses every maintainer-script service start for the duration of the install. There is no service in the entrypoint and no port bound until a developer runs one of the documented commands. PGDG, and what the pin covers. Debian bookworm ships PostgreSQL 15 and the cell is a 17 with store tests written against it; no 17 for bookworm avoids PGDG directly or transitively. A second apt archive is a real trust decision, so the signing key's full 40-hex fingerprint is an ARG a reviewer can read — and the primary-key COUNT is asserted first and separately, because `gpg --dearmor` converts every key in the file and `signed-by=` then trusts the whole keyring. Versions resolve against a moving archive exactly as Debian's do; rainier-apt-sources.txt records both archives. libpq5 and libpq-dev are named explicitly with their major floored at the server's: apt is as free to resolve the 15-to-17 upgrade by removing the -dev package as by upgrading it, and PGDG ships one libpq for every major it carries, so an archive that has released an 18 correctly hands this image an 18.x libpq beside the 17 server. Durable state goes on the workspace volume, because that is the only writable path surviving a suspend. Sockets and pidfiles deliberately do not: protocol/workspace.TarGz refuses a socket rather than skipping it, so one under /workspace fails `rainier push`/`pull` of any tree containing it and an unclean exit leaves it there to keep failing. They live on the per-container tmpfs, which is where per-container state belongs. `rainier-pg` and `rainier-redis` are root-owned helpers beside sessiond. They install nothing and need no privilege, and the raw tools stay on PATH; they exist because three things here are not the defaults. PostgreSQL's compiled-in socket directory is on the read-only rootfs. This image generates no locales, so an initdb inheriting an unset LANG builds an SQL_ASCII database that mangles the first non-ASCII row a test inserts. And `pg_ctl -w stop` polls kill(pid, 0), which cannot tell a shut-down postmaster from an unreaped zombie — while postmaster.pid outlives the container it was written in, so it is never taken as proof a server is running and a pid that may since have been reused is never signalled. TMPDIR is still not set, and a contract test now says so. /tmp is a per-container noexec tmpfs and that is where agent scratch belongs — not in a checkpoint, an archive or `rainier pull`. Exactly one build temp moves off it, GOTMPDIR, because `go test` executes what it builds. Setting TMPDIR globally would move Claude Code's and Codex's scratch onto the volume that leaves the runner and quietly change the sandbox each believes it has. Verification. Ten new tests run both helpers against stub binaries with no docker, covering the initdb flags, the refusals, the stop paths (stale pidfile, a server still rejecting connections, a server that never stops), the deadline, the DSN, and the durable/runtime split. They are named TestSessionImage* so the qualification workflow's -run selector actually runs them. The container smoke checks that a fresh session has nothing listening and no cluster in the image, then initdbs a cluster, starts it, commits one transaction and rolls another back, reconnects over the DSN the helper prints, asserts the listener is loopback and that no socket appears under /workspace while both servers run, stops and restarts with the data intact, round-trips SQLite from shell and Python, and starts, PINGs and stops Redis — as uid 1000, read-only rootfs, no network at all. The smoke also reports now. A failing check becomes a workflow annotation and the sizes become notices, because a job log is not always reachable from where the fix has to be made, and because size is a rollout gate rather than a curiosity. The services layer measures 14 packages and 244,512 KiB installed — the build diffs its own package set to get that, so it counts the Debian-sourced dependencies too. libllvm19 and libz3-4 are 149 MiB of it, for the query JIT postgresql-17 hard-depends on; removing them means building PostgreSQL from source, which docs/session-image.md argues against.
#73) * feat(egress): a default developer egress baseline at the dispatch seam A session's network is default-deny and the only way out is egressd's per-session allowlist. Nothing in that allowlist made `npm ci` work, so the first thing every new environment did was fail: a 403 from the proxy, a human going to find out where npm keeps its tarballs, an environment edit, and a restart — once per ecosystem, per environment. The host list that fixed it for some people lived in a runbook, which means it was a default only for the people who read the runbook. This puts the default where the default actually is. controlapp's createSpec is the one seam every dispatch passes in both compositions, and it now unions a twelve-host developer baseline into the allowlist after the row's declared hosts, the launch material's clone hosts, and the agent providers' hosts — additive, deduped, and never written to the session row, so `egress_allow` still means "what a human asked for". The twelve are literal apexes reached by toolchains the session image actually ships, each established by fetching through it and following the redirects rather than by copying a vendor's firewall page. That method changed two answers: GitHub release assets now land on release-assets.githubusercontent.com, which nothing here carried, and proxy.golang.org serves its own module zips — a 4 MB one included — so storage.googleapis.com buys nothing and would have cost every GCS bucket on the internet. There are no wildcards and there is no multi-tenant namespace on the list; a test enforces both. GitHub Actions job logs, which redirect into productionresultssa*.blob.core.windows.net, are therefore NOT supported by default, and docs/default-egress.md argues that trade rather than hiding it. Reachability is not authority: none of this issues a credential, none of it lets a session publish a package, and private registries stay named per environment beside the credential that reaches them. FleetOptions.DefaultEgress is a pointer so a host can turn the baseline off (empty) or replace it, and it is host policy — no API field, no environment column, so no tenant can widen its own default. Broadening the default makes one latent gap matter: the allowlist matches the NAME a client wrote, and what that name resolves to is a nameserver's answer. egressd now resolves the host itself, refuses loopback/RFC1918/link-local (where every cloud metadata service lives)/CGNAT and the rest, and dials the address it vetted instead of re-resolving the name — which is what makes "a redirect cannot broaden private access" true rather than merely likely. AllowPrivateDestinations lifts it for a local fleet whose origins are on loopback; egressd exposes no flag for it. The self-hosted launch resolver's clone hosts are now read out of the same table rather than spelled a second time, so there is one list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(egress): reserve dial time for fallback addresses --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(cli): minimal hosted-first v0 command surface The CLI had grown twenty-odd commands, and `rainier --help` listed all of them: transfers, snapshots, secrets, environments, contexts, workspaces and the self-hosted credential vault sat beside the four things a hosted developer actually does. A new developer could not read the product off one screen, which is the only test of a command surface that matters. The public surface is now sign in, check readiness, authenticate a coding agent, and create and manage sessions — 23 lines of default help. Everything else still dispatches, under `rainier help all`, labeled Advanced. A session has three independent facts and this CLI keeps them apart: the sandbox's lifecycle (`state`), the process inside it (`child_exit_code`), and whether the control plane can reach it (`reachable`). They fail separately, so they are three columns. A session whose agent exited is still Running — it holds its filesystem and it is still attachable — and a runner that dropped its link makes a session Unavailable, not Failed. Action eligibility comes from the raw API's own transition rules and never from the display word: `SuspendSession` accepts `running` and nothing else, so Stop is not offered for a queued session merely because queued and creating share the label Starting. docs/cli-v0-contract.md is the authority for what each command promises and which Cloud APIs each promise depends on. Rainier Cloud 81d1ad3 shipped the compute enrollment and `workspace_not_ready`, so `status` reads the workspace's own compute state — both `status` and `health`, because ready capacity nobody can reach is not ready. Two contracts are still outstanding and both are isolated in cmd/rainier/readiness.go: a bearer-reachable onboarding destination, and a per-environment agent launch catalog. Neither is guessed at. New: `logout`, `status`, `info`, `stop`, `delete`, the `current` selector, and `--json` on everything a script reads — carrying the canonical API facts verbatim beside the derived ones. `diff` is removed completely: git inside the session is the source of truth. `doctor`, `suspend`, `rm` and `agent ls` remain as hidden aliases for the commands that replaced them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q3GPB683VghXL2XA4XWK7Y * fix(cli): finish v0 readiness, session safety, and output contracts * fix(credentials): fence delayed session credential deletions (#71) * feat(image): PostgreSQL 17, SQLite and Redis in the session image (#72) A session could not run an integration test. There is no way to get a database after the fact — `sudo` is not installed, the rootfs is read-only, and no package archive is on the egress allowlist — so the client AND the server halves belong in the image, which is the one thing a Dedicated runner is allowed to pull. Nothing starts. `create_main_cluster = false` keeps postgresql-common from running initdb into a directory the read-only rootfs could never use, and a policy-rc.d refuses every maintainer-script service start for the duration of the install. There is no service in the entrypoint and no port bound until a developer runs one of the documented commands. PGDG, and what the pin covers. Debian bookworm ships PostgreSQL 15 and the cell is a 17 with store tests written against it; no 17 for bookworm avoids PGDG directly or transitively. A second apt archive is a real trust decision, so the signing key's full 40-hex fingerprint is an ARG a reviewer can read — and the primary-key COUNT is asserted first and separately, because `gpg --dearmor` converts every key in the file and `signed-by=` then trusts the whole keyring. Versions resolve against a moving archive exactly as Debian's do; rainier-apt-sources.txt records both archives. libpq5 and libpq-dev are named explicitly with their major floored at the server's: apt is as free to resolve the 15-to-17 upgrade by removing the -dev package as by upgrading it, and PGDG ships one libpq for every major it carries, so an archive that has released an 18 correctly hands this image an 18.x libpq beside the 17 server. Durable state goes on the workspace volume, because that is the only writable path surviving a suspend. Sockets and pidfiles deliberately do not: protocol/workspace.TarGz refuses a socket rather than skipping it, so one under /workspace fails `rainier push`/`pull` of any tree containing it and an unclean exit leaves it there to keep failing. They live on the per-container tmpfs, which is where per-container state belongs. `rainier-pg` and `rainier-redis` are root-owned helpers beside sessiond. They install nothing and need no privilege, and the raw tools stay on PATH; they exist because three things here are not the defaults. PostgreSQL's compiled-in socket directory is on the read-only rootfs. This image generates no locales, so an initdb inheriting an unset LANG builds an SQL_ASCII database that mangles the first non-ASCII row a test inserts. And `pg_ctl -w stop` polls kill(pid, 0), which cannot tell a shut-down postmaster from an unreaped zombie — while postmaster.pid outlives the container it was written in, so it is never taken as proof a server is running and a pid that may since have been reused is never signalled. TMPDIR is still not set, and a contract test now says so. /tmp is a per-container noexec tmpfs and that is where agent scratch belongs — not in a checkpoint, an archive or `rainier pull`. Exactly one build temp moves off it, GOTMPDIR, because `go test` executes what it builds. Setting TMPDIR globally would move Claude Code's and Codex's scratch onto the volume that leaves the runner and quietly change the sandbox each believes it has. Verification. Ten new tests run both helpers against stub binaries with no docker, covering the initdb flags, the refusals, the stop paths (stale pidfile, a server still rejecting connections, a server that never stops), the deadline, the DSN, and the durable/runtime split. They are named TestSessionImage* so the qualification workflow's -run selector actually runs them. The container smoke checks that a fresh session has nothing listening and no cluster in the image, then initdbs a cluster, starts it, commits one transaction and rolls another back, reconnects over the DSN the helper prints, asserts the listener is loopback and that no socket appears under /workspace while both servers run, stops and restarts with the data intact, round-trips SQLite from shell and Python, and starts, PINGs and stops Redis — as uid 1000, read-only rootfs, no network at all. The smoke also reports now. A failing check becomes a workflow annotation and the sizes become notices, because a job log is not always reachable from where the fix has to be made, and because size is a rollout gate rather than a curiosity. The services layer measures 14 packages and 244,512 KiB installed — the build diffs its own package set to get that, so it counts the Debian-sourced dependencies too. libllvm19 and libz3-4 are 149 MiB of it, for the query JIT postgresql-17 hard-depends on; removing them means building PostgreSQL from source, which docs/session-image.md argues against. * fix(cli): close session selection and diagnostic review gaps * docs(cli): record complete race verification result * fix(cli): address independent v0 lifecycle and output review * test(relay): drain buffered output before asserting connection close --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runner registration could wake the scheduler before the runner's initial session snapshot finished reconciling. A queued session could be dispatched during that window and then requeued by the stale snapshot. This change wakes the pool only after reconciliation succeeds and adds a deterministic regression test for the ordering.
It also adds the npm wrapper package used for the
v0.0.8beta. The wrapper downloads the matching GitHub release archive, verifies its pinned SHA-256 digest, and executes the native CLI.Validation:
make verifygo test -race ./controlapp ./runnerplane ./internal/controld ./internal/e2e -run 'TestRunnerBecomesSchedulableOnlyAfterReconciliationSettles|TestRegisterRunnerIdempotentReconnectAndReplacement|TestPlacementPinQueuesWithReason|TestRegistrationAcceptsFirst' -count=3npm --prefix npm test@tokencanopy/rainier@0.0.8from npm reportsrainier v0.0.8SHA256SUMSRelease: https://github.com/tokencanopy/rainier/releases/tag/v0.0.8