feat(krater): krater staging on weft, database in patchwork_central on hem by Adambomb210 · Pull Request #32 · patchworklabsorg/infra

Adambomb210 · 2026-10-17T01:29:12Z

Krater staging on weft (staging only for now)

Krater now runs on weft, its own lab VM (pve01 VMID 361, 10.100.90.61, 4 GB, 40 GB disk), and only as staging. There is no production instance yet. One app per VM: sudo and the docker group are root, so the people who run Krater cannot read the secrets of Quilt or Weave. Adam's original description is kept below for context. Where it says alastor or production, read weft and staging.

What changed on top of Adam's commits:

Merged main twice (hosts per app, Quilt on batting, weft bring-up feat(weft): add the weft host key, give it ad-join, and size it for krater #60). Conflicts in README.md, flake.nix, secrets/secrets.nix and hosts/weft/configuration.nix are resolved.
hosts/alastor/krater.nix is now hosts/weft/krater.nix, imported by hosts/weft/configuration.nix. The krater-alastor export is gone. nixosModules.krater stays and can still run production later.
Staging only:
- hostnames krater-staging.patchworklabs.org and krater-staging-storage.patchworklabs.org
- sign-in with Weave staging (https://weave-staging.patchworklabs.org)
- database: the krater schema in patchwork_central on hem (:5432). Krater is in the new stagingOnlyApps of hosts/central-db-schema.nix, so hem creates its role and schema and skein does not. hem loads central-staging-krater.age on its own once the file exists.
- Slack: a separate Slack app in a test workspace, never the real Patchwork workspace.
- KRATER_QUILT_URL is empty, so events wait in Krater's outbox. Never set it to production Quilt.
- Krater Compute off, so no Vast account
- KRATER_ENV stays production: Krater has no staging mode, and this value only turns on its production checks. The construction border and STAGING label from the Patchwork staging standard need a change in the Krater app.
Krater stays off until its env file and its Weave staging client id exist (enable = weaveClientId != "" && pathExists krater-staging-env.age). weft builds and deploys now.
nginx on weft (tailscale0:80) serves the portal (port 3010) and the S3 gateway (port 3011). It passes alastor's X-Forwarded-Proto and appends alastor's address to X-Forwarded-For. It does no real_ip rewrite. The storage host has no nginx body limit and no request buffering. Cloudflare caps each request body at 100 MB. No Traefik on weft.
alastor routes the two staging names to weft in nixosModules.edge-alastor (krater-staging-weft, krater-staging-storage-weft).
New module option extraHosts (--add-host for the portal, the worker, the database wait and the migrations). The containers are on the krater bridge network, so Docker DNS answers them, not /etc/hosts. weft has no MagicDNS, so it pins hem.wildebeest-stargazer.ts.net to 100.67.124.123 (the pw-hem alias in jaspermayone/infra).
KRATER_TRUSTED_PROXY_COUNT is 3. Each of Cloudflare, Traefik on alastor (it trusts Cloudflare's ranges) and nginx on weft appends one address, so Krater sees client, cf_ip, alastor_ip.
Settings check: before the database wait, the unit runs get_settings() in a container and fails at once with Krater's message if the env file or the options are wrong.
Disk: krater-build runs docker builder prune -f --filter until=168h. SeaweedFS gets a volume limit (new option storage.maxVolumes, 12 x 1 GB on weft) so it cannot fill the 40 GB disk.
Compute: compute.labelPrefix has no default now. An assertion requires it when compute.enable is true. weft sets krater-staging-.
Fallback rev is b081a13, on Krater's main (after feat: Krater Compute, the application form and project teams krater#9).
zramSwap on weft, for image builds on 4 GB.
Secrets:
- Removed the production files krater-env.age and central-prod-krater.age (Adam's ce73de8). bin/new-krater-secrets production can make them again later.
- New entries, files not created yet: krater-staging-env.age (jsp, weft) and central-staging-krater.age (jsp, hem).
- Renamed krater-staging-vast-key.age and krater-staging-project-secrets-key.age (jsp, weft). They are not used while compute is off.
- bin/new-krater-secrets staging [db host] writes the two staging files. It prints no value. The production case is gone until production exists.
Test tests/krater-weft.nix (x86_64-linux):
- Krater is off on the real weft.
- Turned on, it has its units, the staging vhosts, loopback ports, the Weave staging issuer, the proxy count, no real_ip, an empty Quilt URL, the label prefix, the volume limit, the cache prune, the settings check, and --add-host for hem on every database client but not on SeaweedFS.
- Krater's role and schema are on hem and not on skein.
- The edge has the staging routes and no production route.

flowchart LR
  user((browser)) --> cf[Cloudflare] --> alastor["alastor: Traefik"]
  alastor -- "tailnet :80" --> nginx["weft: nginx"]
  nginx --> web["krater-web :3010"]
  nginx --> s3["krater-storage :3011"]
  web -- "tailnet :5432" --> hem["hem: patchwork_central staging"]
  worker["krater-worker"] -- "tailnet :5432" --> hem
  web -. "OIDC" .-> weave["Weave staging"]

Rollout

Secrets (Jasper): bin/new-krater-secrets staging in a shell with agenix and openssl. It writes secrets/krater-staging-env.age and secrets/central-staging-krater.age and prints no value. Commit both.
Weave staging: create Krater's app (redirect URI https://krater-staging.patchworklabs.org/auth/callback). Then:
- Put the client id in weaveClientId in hosts/weft/krater.nix.
- Put KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET in the env file with cd secrets && agenix -e krater-staging-env.age.
Merge this PR, then:
- Deploy hem (deploy .#hem), so the krater role gets its password.
- Deploy weft once with weaveClientId still empty. Krater stays off.
- Run sudo systemctl start krater-build on weft by hand. The first build takes minutes.
- Set weaveClientId and deploy weft again. If a setting is wrong, krater-web fails at once with Krater's message.
DNS in patchworklabsorg/dns: krater-staging and krater-staging-storage, proxied, to alastor.
alastor: bump patchwork-infra in jaspermayone/infra and deploy alastor (--magic-rollback false) for the edge routes.
CI deploys: set deployAuthorizedKeys, and point the Krater repo deploy secrets at weft.

Checks

nixosConfigurations.weft, skein and hem evaluate. jaspermayone/infra's alastor evaluates with this branch as patchwork-infra.
Built on warp: checks.x86_64-linux.krater-weft, quilt-batting and central-db (VM test).
Built locally: checks.aarch64-darwin.ad-org and terraform-fmt. nix flake check --no-build passes.
The rollout above.

Original description (alastor, production)

Adds Krater, the Project Ganymede portal, as a NixOS module (nixosModules.krater) and its alastor settings (nixosModules.krater-alastor). alastor doesn't import it yet: that's a change in jaspermayone/infra once this merges.

What it runs

krater-web: the portal, and the API rented machines report to. It waits for the database, runs migrations, then serves on loopback behind Traefik at krater.patchworklabs.org.
krater-worker: background jobs (Slack, pricing, events to Quilt, and the compute scheduler when compute is on).
krater-storage: SeaweedFS's S3 gateway at krater-storage.patchworklabs.org. It holds gallery screenshots, and machines' checkpoints and outputs when compute is on.
krater-build: builds the image on alastor from one commit of patchworklabsorg/krater. There's no registry. Krater's Deploy workflow writes the commit to /var/lib/krater/source-rev as the krater-deploy user, which may run only those steps.

Database: patchwork_central

Krater has no Postgres of its own. Its data is the schema krater in patchwork_central on skein, like the other central apps:

krater = { } in hosts/central-db-schema.nix. Krater stores no personal data (only Patchwork IDs) and reads people through Weave, so it gets no shared schemas.
The role's search_path is krater, so Krater's tables, its job queue and alembic_version all live there. Krater needs no code change: its migrations and 1075 tests pass on Postgres 18 set up this way, with nothing in public.
skein's nightly dump is the backup.

Krater Compute: in the module, off on alastor

With compute.enable, the worker rents Vast.ai machines for members' jobs. The Vast key (compute.vastKeyFile) goes into the worker's env alone. alastor keeps it off (compute.enable = false) until the Vast key is rotated: the current key was used on staging. Turning it on:

Put the new key in secrets/krater-vast-key.age.
Uncomment the krater-vast-key secret in hosts/alastor/krater.nix.
Set compute.enable and compute.vastKeyFile.

Project secrets (members' tokens for their jobs) work the same way with krater-project-secrets-key.age.

Secrets

bin/new-krater-secrets writes both new files and prints no value:

krater-env.age (jsp, alastor): the env file. KRATER_DATABASE_URL holds the role's password, plus KRATER_SECRET_KEY and the storage keys. It takes the Weave client secret and the Slack values from the environment when they're set.
central-prod-krater.age (jsp, skein): the role's password alone.

Both files are in this PR (ce73de8; the recipients were checked against secrets.nix). KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET are empty because the Weave and Slack apps don't exist yet. Krater refuses to start in production until they're filled in with cd secrets && agenix -e krater-env.age (only Jasper can).

Before alastor imports it

bin/new-krater-secrets: both secrets, committed
The Weave client secret and the Slack bot token and signing secret, added to krater-env.age (Jasper)
Krater's Weave app: the client id goes in hosts/alastor/krater.nix, and the module refuses to build while it's empty
skein loads central-prod-krater.age into the central DB's password directory. skein's config doesn't load any app's password file yet, so Krater's role can't log in until it does.
The tailnet ACL lets alastor reach skein's port 5432, and a container on alastor resolves skein's tailnet name (as Quilt reaches seam)
DNS: krater and krater-storage (patchworklabsorg/dns, linked below)
deployAuthorizedKeys, and the ALASTOR_HOST and KRATER_DEPLOY_SSH_KEY secrets in the Krater repo
KRATER_TRUSTED_PROXY_COUNT: 2 if alastor's Traefik trusts Cloudflare's ranges in forwardedHeaders.trustedIPs

Checked

alastor's config with this module evaluates with compute off and with compute on (a placeholder env file, a test client id). The units are krater-build, krater-network, krater-storage, krater-web and krater-worker, and there are no timers.
The database wait works: it connects when the database is up and keeps waiting when it isn't. Migrations reach head against a central-db-style schema.
bin/new-krater-secrets ran in nixos/nix with throwaway keys. Both files decrypt, and it refuses to overwrite either one.
The module hasn't run on a NixOS host yet. Expect a round of fixes on the first deploy.

The fallback rev is Krater's claude/application-form (16de89a). After patchworklabsorg/krater#9 merges, set it to the merge commit.

This replaces the branches krater-module and krater-compute-off.

Krater (the Project Ganymede portal) as Docker containers on alastor, in the style of the weave module: Postgres, SeaweedFS screenshot storage, the portal, its procrastinate worker, and the SkyPilot API server with its oauth2-proxy sign-in. There's no registry image. krater-build fetches one commit of patchworklabsorg/krater and builds it with the repo's own Dockerfile, tagged krater:<sha>, skipping commits already built. A deploy writes /var/lib/krater/source-rev, runs `systemctl start krater-build` while the old containers keep serving, then `systemctl restart krater-web`. One agenix env file is split per container at start, so the SkyPilot server never sees Krater's session key or database password. SkyPilot's bootstrap values live in host state files, so they need no infra round trip. krater-backup dumps the database nightly. Exported as nixosModules.krater and nixosModules.krater-alastor. alastor doesn't import it yet: it needs DNS, the Weave client ids and the two secrets first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- /var/lib/krater is 0755 like the weave module's state dir. At 0750 krater-deploy couldn't reach source-rev, so every CI deploy failed. backups/ and secrets/ stay 0700 through tmpfiles, outside StateDirectory (whose mode is reapplied on every start), and dumps are written with umask 077. - Weave's issuer is https://weave.patchworklabs.org, per its discovery document. - One-level hostnames (krater-storage, krater-sky): the records are proxied through Cloudflare, whose edge certificate covers one level. - Note on KRATER_TRUSTED_PROXY_COUNT behind Cloudflare plus Traefik. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`latest` has already moved past the image Krater's staging was tested on. Screenshot storage stays on self-hosted SeaweedFS for now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

SkyPilot asks oauth2-proxy's /oauth2/auth who the user is and reads X-Auth-Request-Email from the answer. oauth2-proxy sends those headers only with set-xauthrequest on. Without it, every `sky api login` ended in a 500 ("oauth2-proxy is enabled but did not return user info"). Found on the local staging run; with the flag the login completes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…kyPilot SkyPilot's Vast support can't rent a machine and copies the Vast key onto members' hosts, so Krater ships with KRATER_COMPUTE_MODE=off until Krater Compute replaces it. With skypilot.enable = false (the default) the module leaves out krater-skypilot, krater-auth-proxy, their env and secrets, the SkyPilot service token, the krater/skypilot state directory and the krater-sky Traefik route, and no longer requires skypilot.domain or weave.skypilotClientId. KRATER_SKYPILOT_MODE stays live: the budget estimator still reads SkyPilot's public GPU catalog. alastor sets skypilot.enable = false and drops skypilot.domain, skypilotClientId and the unused krater-vast-key secret. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Krater dropped SkyPilot for Krater Compute, its own Vast layer. The module loses the SkyPilot server, its sign-in proxy, their secrets, state and Traefik route, and gains compute.enable (off by default; KRATER_COMPUTE_MODE=vast when on), compute.vastKeyFile (read into the worker's env only), compute.labelPrefix (unique per deployment on a Vast account), compute.runnerUrl, projectSecretsKeyFile (Fernet keys, the portal's env only) and weave.cliClientId (the krater CLI's public Weave app). KRATER_SKYPILOT_MODE becomes KRATER_PRICING_MODE. The portal and worker now get their own env files (web.env, worker.env) on top of app.env, since the image runs as a non-root user and can't read a root-only agenix file mounted in. alastor keeps compute off; the new secret krater-project-secrets-key.age is declared but not created. Checked by nix eval (compute off, on, and on without a key) and nixfmt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The price-history run takes over a minute, and with one task at a time Krater's minute-by-minute compute scheduler waited behind it (seen on staging). The scheduler and reconciliation share a lock, so they never overlap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Krater's v1 (PR #1) no longer contains SkyPilot: it runs with compute off and Krater Compute adds machines in its own release. The module drops the skypilot.* options, the SkyPilot API server, its sign-in proxy, their secrets, state and Traefik route, and skypilot-start.sh. KRATER_COMPUTE_MODE is always off; KRATER_SKYPILOT_MODE becomes KRATER_PRICING_MODE (the budget estimator's public GPU catalog, no server). alastor drops skypilot.enable. Adapted from the krater-compute branch's 021a7f9, without its compute options (compute.*, projectSecretsKeyFile, weave.cliClientId) or the per-container env files they need. Checked by nix eval of krater-alastor (agenix stubbed): the Krater units evaluate with no failed assertions; nixfmt is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e too This branch already had the same removal with Krater Compute's options; its version stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The default -volume.max=8 is used up by the first collections; a new bucket then can't get a volume and every upload to it fails with a 500 (found on Krater staging: screenshots after compute's checkpoints). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conflicts were additive: README (repo table, the Krater section next to the Quilt ones), flake.nix (the krater modules next to edge-alastor and quilt) and secrets/secrets.nix (krater-env and krater-vast-key next to the Quilt and AD secrets). The Krater README section now says two hostnames and, with compute off, only krater-env.age. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Krater's data moves to the schema `krater` in patchwork_central on skein (modules/central-db), like the other apps. The module no longer runs a krater-db container or its own backup timer; skein's nightly dump covers it. - hosts/central-db-schema.nix: add krater (no shared schemas). - secrets: central-prod-krater.age (jsp, skein) holds the role password; krater-env.age now carries KRATER_DATABASE_URL with the same password. - bin/new-krater-secrets writes both files with new random values and prints none of them. - krater-web waits for the database with the app's own engine, and needs network-online. Checked: the alastor config evaluates; Krater's migrations reach head in a central-db-style schema on Postgres 18 with nothing in public. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…m the environment Only Jasper and alastor can decrypt krater-env.age once it exists, so whoever runs the script can pass KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET in the environment instead of leaving them for Jasper. The script names the ones it left empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…a main Brings infra main (Quilt beside Krater, central-db, lab hosts) and the move of Krater's database to the schema `krater` in patchwork_central onto the branch with Krater Compute. Conflicts: the module keeps the compute options and the per-container web/worker env files, and drops krater-db, postgresImage and the backup timer; secrets.nix keeps both sides. The fallback rev now points at claude/application-form (16de89a) instead of PR #1's 4f130ca, so a first start without a deploy doesn't build old code. Checked: the alastor config evaluates with compute off and with compute on (the worker env alone gets the Vast key). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bin/new-krater-secrets: a new database password (in both files), secret key and storage keys. krater-env.age (jsp, alastor) leaves KRATER_WEAVE_CLIENT_SECRET, KRATER_SLACK_BOT_TOKEN and KRATER_SLACK_SIGNING_SECRET empty until the Weave and Slack apps exist; Jasper fills them in with agenix -e. central-prod-krater.age is for skein. .gitattributes keeps the script LF, so it runs from a Windows checkout too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jaspermayone · 2026-10-21T18:19:33Z

+      # The `krater` CLI's public Weave app (Krater's docs/compute/cli.md).
+      # Empty keeps the CLI's API off.
+      cliClientId = "";


@Adambomb210 what is this lol

jaspermayone · 2026-10-21T18:19:50Z

+      # Not secret, but only known once the app exists in Weave. The module
+      # refuses to build while it is empty.
+      clientId = "";


this is like... prob fine to be public

… weave app exists

…ttings check, storage and cache limits

Adambomb210 and others added 15 commits October 9, 2026 21:16

fix(krater): pin SeaweedFS to 4.47, the version staging ran

4611ac5

`latest` has already moved past the image Krater's staging was tested on. Screenshot storage stays on self-hosted SeaweedFS for now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge krater-compute-off: SkyPilot removed from the compute-off modul…

67878c3

…e too This branch already had the same removal with Krater Compute's options; its version stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adambomb210 requested review from a team and jaspermayone October 17, 2026 01:29

This was referenced Oct 17, 2026

feat: add krater and krater-storage.patchworklabs.org patchworklabsorg/dns#12

Open

feat: Krater Compute, the application form and project teams patchworklabsorg/krater#9

Merged

jaspermayone mentioned this pull request Oct 21, 2026

refactor: one vm per app, weave stays on alastor #48

Merged

2 of 3 tasks

jaspermayone reviewed Oct 21, 2026

View reviewed changes

jaspermayone mentioned this pull request Oct 22, 2026

feat(weft): add the weft host key, give it ad-join, and size it for krater #60

Merged

3 tasks done

jaspermayone added 5 commits October 22, 2026 00:17

Merge main into krater-compute: hosts per app, Krater moves to weft

37d174a

feat(krater): run krater on weft behind alastor's edge, off until its…

f027b53

… weave app exists

test(krater): match the unquoted add-host argument

43b50ce

feat(krater): run only krater staging on weft, on hem and weave staging

288cb96

Merge main into krater-compute: weft bring-up (#60)

71d18ef

jaspermayone changed the title ~~feat(krater): Krater on alastor, database in patchwork_central~~ feat(krater): krater staging on weft, database in patchwork_central on hem Oct 22, 2026

jaspermayone added 2 commits October 22, 2026 00:41

fix(krater): whole proxy chain, staging-only database and secrets, se…

dce3325

…ttings check, storage and cache limits

docs(krater): use hem in the extraHosts example

b26aa67

Navigation Menu

Repository navigation

feat(krater): krater staging on weft, database in patchwork_central on hem - #32

feat(krater): krater staging on weft, database in patchwork_central on hem#32
Adambomb210 wants to merge 23 commits into
mainfrom
krater-compute

Adambomb210 commented Oct 17, 2026 •

edited by jaspermayone

Loading

Uh oh!

jaspermayone Oct 21, 2026

Uh oh!

jaspermayone Oct 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Adambomb210 commented Oct 17, 2026 • edited by jaspermayone Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Krater staging on weft (staging only for now)

Rollout

Checks

Original description (alastor, production)

What it runs

Database: patchwork_central

Krater Compute: in the module, off on alastor

Secrets

Before alastor imports it

Checked

Uh oh!

jaspermayone Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

jaspermayone Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Adambomb210 commented Oct 17, 2026 •

edited by jaspermayone

Loading