[lorenzocorallo]

Deployment side of the regulations editor (shared spec, decision 4 and the "Deployment" contract). Aligned with the final backend services in PoliNetworkOrg/backend#110–#117 (image, entrypoint, ports, env names), and verified with the real images built from #117 (d8429f9, re-verified on eca606e).

Do not merge yet. Merge order and prerequisites are at the bottom.

What changes

regulations-collab (Hocuspocus), in the backend namespace

• Runs the backend image with bun run collab.js. It sits in the same Flux Kustomization, so it always runs the backend's digest; no new image entry is needed.
• Port 4100 is the public websocket. The Ingress publishes it on collab.polinetwork.org (proposed name), and Traefik passes the Upgrade through. No Cloudflare Access: the single-use ticket authenticates each connection.
• Port 4101 is the internal API, reachable from the backend Pod only.
• NetworkPolicy:
  • Ingress: Traefik → 4100, backend → 4101.
  • Egress: DNS, PostgreSQL 5432 and latex-worker 4200 only.
• Runs as a single replica with Recreate, because Hocuspocus keeps open documents in memory.
• Env:
  • REGULATIONS_INTERNAL_SECRET is a required ref, since collab refuses to start without it.
  • COLLAB_PORT/COLLAB_INTERNAL_PORT are set explicitly.
  • REGULATIONS_COMPILE_CONCURRENCY=2, matching the worker's 2 (the default is 3, which would only queue at the worker).
• Hardened: uid 1000 (the image's bun user), read-only root, tmpfs /tmp, no capabilities, seccomp RuntimeDefault, no service account token.

latex-worker, in its own namespace

• Image ghcr.io/polinetworkorg/latex-worker, with a new latex-worker-image provider and Receiver entry, following the backend image pattern.
• Port 4200. Ingress from the backend and collab only. No egress at all, not even DNS.
• PSA restricted is enforced on the namespace, plus read-only root, no capabilities, no SA token.
• Runs as uid 10200, a uid nothing else on the node uses (a test enforces this). The worker caps each xelatex with prlimit --nproc=64, and RLIMIT_NPROC counts every task of that uid on the node. As uid 1000, which the Bun and Node apps share, every compile failed once the node had about 90 uid-1000 tasks: xelatex couldn't fork xdvipdfmx ("I can't write on file `RRI.pdf'").
• tmpfs /work is 512Mi (the warm workdirs) and /tmp is 64Mi.
• The worker's own env names: WORKER_PORT, WORKER_WORK_DIR, WORKER_TMP_DIR, WORKER_SECRET (required: an empty secret would accept any caller), WORKER_MEMORY_LIMIT_MB=1024 (the per-xelatex prlimit --as), and WORKER_MAX_WORKDIRS=40.
• The worker runs xelatex in a clean env it builds itself: no shell escape, paranoid openin_any/openout_any, caches in /tmp.
• Probes are GET /health, which needs no auth.
• Resources: requests 250m / 768Mi, limits 1500m / 1536Mi. WORKER_CONCURRENCY=2 and WORKER_TIMEOUT_MS=45000.
  • CPU stays below 2 because the node (E2ps_v6) has 2 vCPUs in total.
  • The memory limit covers a full /work tmpfs, since memory-backed volumes count against it.
• A LimitRange plus ResourceQuota caps the namespace at exactly one worker's resources, so it can't starve the node.

Backend

• New REGULATIONS_* env, including REGULATIONS_GITHUB_MODE=app and REGULATIONS_GITHUB_BASE=main. The secrets live in a separate regulations ExternalSecret and the backend reads them with optional: true, so a missing entry never blocks the backend.
• GitHub webhook via the declared /public prefix (spec, "Public backend routes"). GitHub posts to POST https://api.polinetwork.org/public/webhooks/github, and the webhook is now the main emanation trigger. The backend's polling stays only as slow reconciliation for missed deliveries.
  • Hostname: api.polinetwork.org, the backend's public host before feat(backend): close public access and allow only in-cluster callers #28. Its DNS already points at the tunnel.
  • One IngressRoute (apps/backend/public-route.yaml) is the backend's only public surface. Its rule is Host(`api.polinetwork.org`) && PathPrefix(`/public/`) && PathRegexp(`^/public(/[A-Za-z0-9_-]+)+$`).
    • The regex is a whitelist: after Traefik's own path sanitizing, only plain segments of letters, digits, - and _ match. Dots, % (any encoded character), ;, \, empty segments and a trailing slash never match.
    • Every other path, on any host, gets Traefik's 404.
  • Middlewares (the first Traefik middlewares in this repo):
    • body limit 1 MiB (buffering, 413 above it);
    • per-visitor rate limit of 1/s with a burst of 20, keyed on Cf-Connecting-Ip (the TCP peer is always cloudflared);
    • total rate limit of 5/s with a burst of 50;
    • X-PoliNetwork-Ingress: public stamped on every request (Traefik overwrites any client value). Bun resolves .. and %2e%2e before the app sees the path, so the backend uses this marker as its second layer: a stamped request may only reach a declared /public route, whatever its path became. In-cluster callers never carry it.
  • NetworkPolicy: Traefik is added to allow-callers on 3000. A NetworkPolicy can't filter paths, so the route is the guard, and the tests make it the only router that can target backend-service.
  • The backend keeps its own checks per the contract: explicit route list, HMAC with constant-time compare and delivery-id replay protection, its own body and rate limits, and rejection of escaping paths.

Tests

tests/workload-manifests.test.rb now asserts the sandbox and exposure properties:

• no worker egress, and worker ingress only from backend and collab;
• read-only root, non-root, no SA token, size-limited tmpfs, LimitRange and quota;
• the collab websocket reachable only from Traefik and its internal port only from the backend;
• collab running the backend image;
• in the backend namespace, exactly one route targets backend-service: the /public IngressRoute, with exactly that rule, a body limit, both rate limits and the X-PoliNetwork-Ingress: public marker;
• the namespace's only Ingress serves collab:4100;
• the whitelist accepts /public/webhooks/github and rejects /api/trpc, /internal/events, /public, /publicx, /PUBLIC/..., /public/../api/trpc, /public/%2e%2e/api/trpc, /public/..%2fapi, /public%2f..%2fapi, //public/a, /public//a, /public/a/, NUL, ; and backslashes.

Also: the worker's uid is used by no other workload; WORKER_WORK_DIR/WORKER_TMP_DIR are size-limited tmpfs mounts, not the read-only root; and collab never sends more parallel compiles than the worker runs.

Each assertion was mutation-checked: I broke the manifest and confirmed the test fails.

Verification

• CI-equivalent: kustomize build (v5.7.1, as in CI) passes for clusters/k3s, every apps/* and the touched infrastructure. Both Ruby test files pass.
• Schema: kubeconform -strict with the CRD catalog validates all 62 rendered resources, including ExternalSecret, ResourceSet, ResourceSetInputProvider, Receiver, IngressRoute and Middleware. None were skipped.
• Real images on k3s v1.36.3 (prod's version): the backend and worker images were built from backend #117 (d8429f9). legal-texlive is private, so the worker used a Debian TeX Live base. They ran under the rendered manifests next to a test Postgres. The backend's IdP, OAuth and Azure env were stripped so the test never contacted production.
  • Backend: migration 0033 completed, with the expected warnings (no GitHub App, no LLM).
  • Collab listens on 4100/4101 with a read-only root.
  • Health: backend → collab:4101 /health 200 and backend → worker /health 200. collab → worker 200.
  • Blocked: collab → backend:3000 and collab → internet. The worker has no DNS and no internet.
  • The real RRI went through the worker API exactly as the backend sends it (PUT /assets/<hash>, POST /compile with the Bearer secret):
    • cold: 3 xelatex runs, 120 KB PDF, 1.3–1.7 s;
    • warm: 1 run, 0.43–0.51 s;
    • 3 parallel: all OK in ≤ 1.9 s each;
    • wrong secret: 401.
  • uid finding: at about 90 uid-1000 tasks on the node, compiles failed as uid 1000 and recovered when the extra tasks were removed. As uid 10200, with 73 unrelated uid-1000 tasks, all compiles succeeded.
• Re-verified on backend #117 eca606e: a real backend, collab and worker ran with Postgres and Redis under feat(regulations): deploy regulations-collab and latex-worker #32's manifests, and the /public webhook was exercised through Traefik:
  • Webhook:
    • a signed delivery gets 202, and a replay of the same delivery id gets 200 {"duplicate":true};
    • a wrong or missing signature gets 401; a GET or an undeclared /public/... route gets 404 from the backend;
    • ?/../../api/trpc stays on the webhook handler;
    • the spoofed marker is overwritten;
    • 2 MiB gets 413 at the edge;
    • /public/../api, %2e%2e, %2f and a trailing slash get 404 at Traefik.
  • Marker, second layer: an in-cluster caller sending X-PoliNetwork-Ingress: public to /api/trpc gets 404 from the backend; without it, normal behaviour (401 without a token).
  • Worker: it no longer sets RLIMIT_NPROC ("Max processes: unlimited") and runs as 10200. With 94 uid-1000 tasks on the node, the RRI compiled cold in 1.4 s and warm in 0.43 s, and 3 parallel compiles were all OK.
• Earlier stand-in runs: I first ran the rendered manifests on a throwaway k3s v1.36.3+k3s1 (prod's version), Traefik 3.7.8, in Docker. ESO isn't installed there, so the ExternalSecrets were replaced by test Secrets. The images were stand-ins: a Debian TeX Live worker, and a node image providing bun run collab.js → ws echo on 4100 and HTTP on 4101.

  • NetworkPolicy matrix, enforced by kube-router:

    • reachable: backend→worker, collab→worker, backend→collab:4101, collab→postgres:5432, collab DNS;
    • blocked: another namespace→worker / collab:4100 / collab:4101, collab→backend:3000, collab→internet, collab→kube API, collab→postgres:5433;
    • worker: DNS, internet and kube API all blocked.

  • Sandbox: uid 1000, /etc read-only, /work and /tmp are noswap tmpfs of 512M and 64M, CapEff 0, NoNewPrivs 1, seccomp 2, no SA token. PSA rejects a privileged root Pod in latex-worker.

  • The quota is fully used by one worker, and a Recreate rollout still succeeds under it.

  • The backend Deployment starts with the regulations Secret absent, which confirms the optional refs work.

  • Real RRI compile (this repo's latex/RRI.tex, 19 pages, fonts via relative Path=) inside the worker Pod under the exact securityContext and limits:

    Run	Time
    Cold latexmk	1.75 s
    Warm single pass	0.44 s
    2 parallel warm passes	1.0 s wall
    3 parallel cold latexmk	3.6 s wall, peak 640 MiB of 1536 MiB, 0 OOM kills

    The spec's ~40-page document should roughly double these times.

  • Websocket through Traefik via the Ingress (Host: collab.polinetwork.org): upgrade 101. An active connection and a fully idle one were both held for 150 s with no disconnect, so Traefik 3.7.8's default timeouts don't cut upgraded connections. Any other host returns 404.

  • /public route on real Traefik 3.7.8, with raw request lines sent byte-for-byte as Host: api.polinetwork.org to a stand-in backend that echoes what it receives:

    • Reaches the backend: /public/webhooks/github (any method), and /public/a-b_c/D9.
    • 404:
      • other paths: /, /api/trpc, /api/trpc/regulations.list, /internal/events, /trpc, /public, /public/, /publicx, /PUBLIC/..., /x/public/a, /public/webhooks/github/;
      • dot segments: /public/../api/trpc, /public/webhooks/../../api/trpc;
      • encoded or odd characters: /public/%2e%2e/api/trpc, /public/%2E%2E/..., /public/..%2fapi%2ftrpc, /public/%2e%2e%2f..., /public%2f..%2fapi%2ftrpc, /public/webhooks%2fgithub, /public/a%00, /public/a;x, /public/a%5c..%5capi, /public/a\\..\\api;
      • absolute-form http://api.polinetwork.org/api/trpc;
      • /public/webhooks/github on backend. or admin.polinetwork.org.
    • Normalized by Traefik before routing, still inside /public: /public/./webhooks/github, /public//webhooks/github and //public/webhooks/github arrive as /public/webhooks/github; /public/%61pi arrives as /public/api.
    • Query strings pass through, e.g. ?/../../api/trpc. The backend must route on the pathname only.
    • Limits: a 900 KiB body is accepted; 1 MiB + 1 and 5 MiB get 413. 30 concurrent requests from one visitor give 20×200 and 10×429. 80 concurrent requests from 80 visitors give 50×200 and 30×429.
    • Marker: the backend receives exactly one X-PoliNetwork-Ingress: public whether the client sends nothing, a spoofed value, lowercase, duplicates, mixed case or an empty value. A direct in-cluster call gets no marker.
    • Fails closed: while the IngressRoute referenced a middleware Traefik hadn't loaded yet, it answered 400 rather than routing without limits.
    • A Pod outside the allowed callers still can't reach backend:3000 directly.

  • The real ghcr.io/polinetworkorg/backend:latest runs bun 1.4.4 as uid 1000 (bun) under a read-only root with only /tmp writable.

Not verified

• The real collab.js and latex-worker images don't exist yet. Their probes are TCP, as for the backend.
• Cloudflare's side of /public: the Access bypass below, edge URL normalization, and GitHub's deliveries actually arriving.
• Cloudflare's side of the websocket (tunnel + edge, 100 s idle limit). Hocuspocus pings keep the connection busy, but check this after cutover.
• ESO syncing the new Key Vault entries. The kv-pn-apps condition gains latex-worker.
• The Flux ResourceSet templating; it mirrors the existing files exactly.

Before merging

1. Images:

   • legal-texlive (private, under legal) must be multi-arch, including arm64, and must grant read access to PoliNetworkOrg/backend's Actions (package settings → Manage Actions access). Otherwise the backend's latex-worker job can't build.
   • After its first publish, set the latex-worker package to public. The cluster and Flux pull ghcr.io/polinetworkorg/* without credentials. Its layers are only TeX Live and the worker.

2. Merge and release the backend PRs that add collab.js and publish ghcr.io/polinetworkorg/latex-worker. The collab Deployment uses the backend image, so merging earlier would crash-loop it and fail the backend Kustomization's health check. The worker stays inert until its image exists.

3. Create these in kv-pn-apps:

   Entry	Value
   regulations-internal-secret	openssl rand -hex 32
   regulations-worker-secret	openssl rand -hex 32 (required by the worker)
   regulations-github-app-private-key	the App's PEM (az keyvault secret set --file)
   regulations-github-webhook-secret	the App's webhook secret (openssl rand -hex 32, set the same value in the App)
   regulations-llm-api-key	the LLM key

   Every entry must exist, or its ExternalSecret stays not-Ready. Use a placeholder until the App or LLM key exists.

4. Cloudflare Zero Trust:

   • Add collab.polinetwork.org to the k3s01 tunnel → Traefik, like the other hostnames. Keep WebSockets enabled on the zone and don't put Access on it.
   • api.polinetwork.org is currently behind an Access application (it answers 302 to the Access login). Add a more specific Access application for api.polinetwork.org/public/ with a Bypass / Everyone policy so GitHub can reach it, and keep the existing Access app on the rest of the host as an extra layer at the edge.
   • Make sure the tunnel route for api.polinetwork.org → Traefik still exists.

5. Fill in the TODO values in apps/backend/deployment.yaml (plain env, not secret): REGULATIONS_GITHUB_APP_ID, REGULATIONS_LLM_PROVIDER/MODEL, REGULATIONS_TELEGRAM_CHAT_ID/TOPIC_ID, and confirm REGULATIONS_EMAIL_SENDER. Empty values mean that feature is off.

6. Backend: #117 now serves POST /public/webhooks/github with the contract's checks: HMAC, replay protection by delivery id, 1 MiB body limit, its own rate limits, and 404 while the secret is unset. Polling is reconciliation every REGULATIONS_RECONCILE_SECONDS (default 300). Migration 0034_webhook_deliveries comes with it.

7. GitHub App webhook: URL https://api.polinetwork.org/public/webhooks/github, content type application/json, the secret above, and events Pull request, Check suite and Release.

Follow-ups (not in this PR)

• Optional pid cap: the worker no longer sets a per-uid process limit. A node-wide kubelet podPidsLimit (k3s kubelet-arg: pod-max-pids=…, in ansible/roles/k3s) would bound any Pod's processes. It affects every workload, so it's left for a separate change.
• collab, like the backend, uses the PostgreSQL root credentials. A dedicated DB role would narrow what a public-facing Pod can do.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.