Skip to content

Commit dcf894a

Browse files
waleedlatif1claude
andcommitted
fix: correct the Kubernetes and chart claims the audit found
The install command was the serious one: the cloud values files hardcode a placeholder domain in six places, and the documented command overrode only `NEXT_PUBLIC_APP_URL`. Following the page literally left sign-in pointed at the placeholder, realtime rejecting every socket upgrade, and the Ingress serving the wrong host — the exact failure the page warns about further down. - The PII recipe's `INTERNAL_API_BASE_URL` self-call is blocked by the NetworkPolicy the same page recommends enabling: the app policy permits Postgres, Redis, realtime, Ollama, PII, telemetry, DNS, and 443, but has no app-to-app rule. Redaction would fail closed. - The GPU device-plugin DaemonSet is not `restricted`-compliant — it sets only two of the four controls and mounts three `hostPath` volumes — so enforcing the label namespace-wide breaks a GPU install. - Under External Secrets the chart writes nothing from `app.env`; an unmapped key fails the render. The page said the two modes behave alike. - The CronJob pods have no NetworkPolicy either. - `app.env.DATABASE_URL` is silently discarded: the chart computes it and inlines it on the container, so the PgBouncer advice pointed at a key that does nothing. - The PDB also renders from `autoscaling.minReplicas`, the reference topology could not render without an `externalDatabase.password`, `copilot`'s security-context keys nest one level deeper, and `NO_PROXY` needs the app's own Service name. - Untangled a garbled `externalSecrets.apiVersion` comment that contradicted its own default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
1 parent 7ca8dec commit dcf894a

5 files changed

Lines changed: 36 additions & 11 deletions

File tree

apps/docs/content/docs/platform/self-hosting/kubernetes.mdx

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,16 @@ helm upgrade --install sim ./helm/sim \
7070
--set app.env.CRON_SECRET="$CRON_SECRET" \
7171
--set postgresql.auth.password="$POSTGRES_PASSWORD" \
7272
--set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \
73+
--set app.env.BETTER_AUTH_URL="https://sim.yourdomain.com" \
74+
--set app.env.NEXT_PUBLIC_SOCKET_URL="https://sim-ws.yourdomain.com" \
75+
--set realtime.env.ALLOWED_ORIGINS="https://sim.yourdomain.com" \
76+
--set ingress.app.host="sim.yourdomain.com" \
77+
--set ingress.realtime.host="sim-ws.yourdomain.com" \
7378
--namespace simstudio --create-namespace
7479
```
7580

81+
Every one of those overrides is required. The cloud values files hardcode a placeholder domain in all six places, and overriding only `NEXT_PUBLIC_APP_URL` leaves sign-in pointed at the placeholder, realtime rejecting every socket upgrade, and the Ingress serving the wrong host.
82+
7683
Swap the `--values` file for your cloud: `values-aws.yaml` (EKS), `values-azure.yaml` (AKS), or `values-gcp.yaml` (GKE). Everything else is identical.
7784

7885
## Key Configuration

apps/docs/content/docs/platform/self-hosting/networking.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -336,7 +336,7 @@ The practical consequence: a mandatory-egress-proxy environment can route most L
336336
# Helm — for a release named `acme`
337337
app:
338338
env:
339-
NO_PROXY: "localhost,127.0.0.1,.svc,.svc.cluster.local,acme-sim-realtime,acme-sim-pii,acme-sim-ollama"
339+
NO_PROXY: "localhost,127.0.0.1,.svc,.svc.cluster.local,acme-sim-app,acme-sim-realtime,acme-sim-pii,acme-sim-ollama"
340340
```
341341
342342
```yaml

apps/docs/content/docs/platform/self-hosting/scaling.mdx

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ The `resources` block above is the chart's default — it is shown so the number
4141

4242
**Memory is the constraint, not CPU.** Workflow executions run inside the app process in isolated-vm sandboxes, and file parsing happens in memory. Production telemetry shows 4–8 GB steady with peaks to 12 GB under heavy execution load. Under-provision memory and you get OOMKills that terminate in-flight workflow runs — so raise `limits.memory` above the 8 Gi default if your workload shows those peaks.
4343

44-
A PodDisruptionBudget is created automatically once `replicaCount > 1` (`maxUnavailable: 25%`). Tighten it with `podDisruptionBudget.minAvailable` if you need to.
44+
A PodDisruptionBudget is created automatically once `replicaCount > 1`, or once `autoscaling.enabled` with `minReplicas > 1` — for the app and realtime independently (`maxUnavailable: 25%`). Tighten it with `podDisruptionBudget.minAvailable` if you need to.
4545

4646
### Autoscaling
4747

@@ -78,7 +78,7 @@ Each app replica opens a pool. Total connections grow with replica count, and Po
7878

7979
Budget it: `replicas × pool size + realtime + cronjobs + migrations + headroom` must stay under `max_connections`.
8080

81-
For anything beyond a handful of replicas, put **PgBouncer** in transaction pooling mode in front of the database and point `DATABASE_URL` at it. This is the single highest-leverage change for a large deployment — it decouples app replica count from database connection count.
81+
For anything beyond a handful of replicas, put **PgBouncer** in transaction pooling mode in front of the database and point the chart at it with `externalDatabase.host` / `externalDatabase.port` and `postgresql.enabled: false`. Setting `app.env.DATABASE_URL` does nothing — the chart computes `DATABASE_URL` itself and inlines it on the container, where it overrides anything from the Secret. This is the single highest-leverage change for a large deployment — it decouples app replica count from database connection count.
8282

8383
### Read replicas
8484

@@ -147,6 +147,9 @@ externalDatabase:
147147
port: 6432
148148
database: simstudio
149149
sslMode: require
150+
# Required — supply it here, through externalDatabase.existingSecret, or
151+
# through External Secrets, or the chart refuses to render.
152+
password: "<password>"
150153
151154
autoscaling:
152155
enabled: true

apps/docs/content/docs/platform/self-hosting/security.mdx

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ In increasing order of production-readiness:
3232
2. **A pre-created Kubernetes Secret** — set `app.secrets.existingSecret.enabled: true` and the secret name. Works with Sealed Secrets and SOPS. The secret is consumed wholesale and must use the standard key names.
3333
3. **External Secrets Operator** — sync from Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager. Recommended.
3434

35-
In the default and External Secrets modes, the chart writes every key under `app.env` and `realtime.env` into a chart-managed Secret mounted via `envFrom`, so no value is inlined into a pod spec. (In `existingSecret` mode the pre-created Secret is the source of truth and any `app.env` values you still pass are rendered inline — supply everything through the Secret in that mode.) Either way, a secret committed to `values.yaml` is a secret in your git history.
35+
In the default mode the chart writes every key under `app.env` and `realtime.env` into a chart-managed Secret mounted via `envFrom`, so no value is inlined into a pod spec. Under External Secrets the chart renders no Secret of its own: a Secret of the same name is populated by the operator from `externalSecrets.remoteRefs.app`, and a non-empty `app.env` key that is not mapped there fails the render. (In `existingSecret` mode the pre-created Secret is the source of truth and any `app.env` values you still pass are rendered inline — supply everything through the Secret in that mode.) Either way, a secret committed to `values.yaml` is a secret in your git history.
3636

3737
## Network boundaries
3838

@@ -108,17 +108,17 @@ So a workflow can never reach instance metadata, but the pod-level block is only
108108

109109
### The `copilot` deployment ships no NetworkPolicy
110110

111-
The `copilot` and `copilot-postgresql` workloads deliberately have no policy of their own. The `copilot` service requires `REDIS_URL` on a non-443 port, and the chart cannot know your Redis host at render time — a default egress rule would block Redis on most installs. If you run `networkPolicy.enabled: true` with `copilot.enabled: true`, those two pods are unisolated until you write dedicated NetworkPolicies for them. The chart-level `networkPolicy.egress` list does not reach them — it is only rendered into the policies the chart creates, and it creates none for these workloads.
111+
The `copilot` and `copilot-postgresql` workloads deliberately have no policy of their own, and neither do the CronJob pods — nothing in the chart's policies selects them, so they are unisolated whenever `networkPolicy.enabled: true`. The `copilot` service requires `REDIS_URL` on a non-443 port, and the chart cannot know your Redis host at render time — a default egress rule would block Redis on most installs. If you run `networkPolicy.enabled: true` with `copilot.enabled: true`, those two pods are unisolated until you write dedicated NetworkPolicies for them. The chart-level `networkPolicy.egress` list does not reach them — it is only rendered into the policies the chart creates, and it creates none for these workloads.
112112

113113
### Pod Security Standards
114114

115-
All workloads set `runAsNonRoot`, drop all Linux capabilities, disable privilege escalation, and use `seccompProfile: RuntimeDefault` — the four controls the `restricted` profile requires. Label the namespace to enforce it:
115+
All workloads set `runAsNonRoot`, drop all Linux capabilities, disable privilege escalation, and use `seccompProfile: RuntimeDefault` — the four controls the `restricted` profile requires — with one exception. The NVIDIA device-plugin DaemonSet the chart renders when `ollama.gpu.enabled: true` sets only `allowPrivilegeEscalation` and `capabilities.drop`, and mounts three `hostPath` volumes that `restricted` forbids outright. Enforce it at the namespace level only when GPU Ollama is off, or exempt that DaemonSet. Label the namespace to enforce it:
116116

117117
```bash
118118
kubectl label namespace simstudio pod-security.kubernetes.io/enforce=restricted
119119
```
120120

121-
`readOnlyRootFilesystem` is not set by default: Postgres and Ollama need a writable root, and the app container writes to Next.js's `.next/cache`. It is viable on the genuinely stateless services (`realtime`, `pii`, `copilot`) — set `<component>.securityContext.readOnlyRootFilesystem: true` and mount an `emptyDir` at `/tmp` via `extraVolumes` / `extraVolumeMounts`.
121+
`readOnlyRootFilesystem` is not set by default: Postgres and Ollama need a writable root, and the app container writes to Next.js's `.next/cache`. It is viable on the genuinely stateless services — set `realtime.securityContext`, `pii.securityContext`, or `copilot.server.securityContext` (the copilot keys nest one level deeper) to `readOnlyRootFilesystem: true`, and mount an `emptyDir` at `/tmp` via the matching `extraVolumes` / `extraVolumeMounts`.
122122

123123
## Where user code runs
124124

@@ -252,6 +252,22 @@ app:
252252

253253
`INTERNAL_API_BASE_URL` must be the **in-cluster** Service URL. The redaction path calls the app's own API, and a public ingress URL is usually not hairpin-reachable from inside the cluster. Without a reachable value the path fails closed — affected fields are scrubbed to `[REDACTION_FAILED]` rather than leaking, but redaction does not actually run.
254254

255+
With `networkPolicy.enabled: true` that self-call is blocked: the chart's app policy permits Postgres, Redis, realtime, Ollama, PII, telemetry, DNS, and TCP 443, but has no app-to-app rule. Add one, or set `networkPolicy.allowExternalEgress: true`:
256+
257+
```yaml
258+
networkPolicy:
259+
egress:
260+
- to:
261+
- podSelector:
262+
matchLabels:
263+
app.kubernetes.io/name: sim
264+
app.kubernetes.io/instance: <release>
265+
app.kubernetes.io/component: app
266+
ports:
267+
- protocol: TCP
268+
port: 3000
269+
```
270+
255271
The service bundles ~2.2 GB of spaCy models, so first start takes around three minutes and it needs at least 4 GB of memory.
256272

257273
## The shipped Compose file publishes Postgres

helm/sim/values.yaml

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1892,10 +1892,9 @@ externalSecrets:
18921892
# Enable External Secrets Operator integration
18931893
enabled: false
18941894

1895-
# ESO API version. Default "v1beta1" — supported by every ESO release from
1896-
# Current ESO releases serve only external-secrets.io/v1 (the v1beta1
1897-
# compatibility path was removed in 2026). Set to "v1beta1" only for
1898-
# clusters still running ESO < 0.17.
1895+
# ESO API version. Current ESO releases serve only external-secrets.io/v1
1896+
# (the v1beta1 compatibility path was removed in 2026). Set this to
1897+
# "v1beta1" only for clusters still running ESO < 0.17.
18991898
apiVersion: "v1"
19001899

19011900
# How often to sync secrets from the external store

0 commit comments

Comments
 (0)