amccabe · Sep 23, 2026 · Sep 23, 2026 · Sep 23, 2026 · Sep 23, 2026
@@ -1,11 +1,11 @@
-## Description
+## What changes
 
 <!-- What changes, in a paragraph. -->
 
-## Testing
+## How it was tested
 
 <!-- How it was tested: unit tests, scripts, a run against a cluster. Name what rests on unit tests only. -->
 
-## Review
+## Security review
 
-<!-- Security review: run or skipped, and why. Authentication, access control, secrets, and persisted payloads qualify. -->
+<!-- Run or skipped, and why. Authentication, access control, secrets, and persisted payloads qualify. -->
@@ -1,6 +1,6 @@
 # Agent guide
 
-This repository is a small distributed system on a local k3d cluster that breaks in realistic ways on demand. Go and Rust services live in `go-api/`, `go-grpc/`, `rust-inventory/`, and `migrate/` as one Go module plus one cargo crate; `k8s/` holds the manifests (`apps/base` for the services, `infra/` for Forgejo, Flux, and monitoring); `scenarios/` holds the failure scenarios and their kustomize overlays; `scripts/` holds bring-up, teardown, build, smoke, and validation; `docs/` holds the topology, service, and scenario pages.
+This repository is a small distributed system on a local k3d cluster that breaks in realistic ways on demand. Go and Rust services live in `go-api/`, `go-grpc/`, `rust-inventory/`, and `migrate/` as one Go module plus one cargo crate; `k8s/` holds the manifests (`apps/base` for the services, `infra/` for Forgejo, Flux, monitoring, and Flagsmith); `scenarios/` holds the failure scenarios and their kustomize overlays; `scripts/` holds bring-up, teardown, build, smoke, and validation; `docs/` holds the topology, service, and scenario pages.
 
 ## Build and test
 

@@ -28,6 +28,10 @@ is self-contained after startup (see [Prerequisites](#prerequisites) and
   - node-exporter
   - Pre-configured with per-service dashboards, recording rules for error rate
     and latency, and alerting rules.
+- Flagsmith, self-hosted, for feature flags
+  - Seeded with an `example-stack` organization and project and two
+    environments, `staging` and `production`
+  - No flags are defined and no service reads one yet
 - A local image registry that the cluster pulls from.
 - Three failure scenarios that break the stack in a specific way and revert.
 
@@ -39,7 +43,7 @@ go-grpc/         Go gRPC pricing and echo service, and its protos
 rust-inventory/  Rust HTTP inventory service
 migrate/         goose migration runner and the migration SQL
 k8s/apps/base/   the services, their Ingresses, dashboards, and alert rules
-k8s/infra/       Forgejo, Flux, and the monitoring stack
+k8s/infra/       Forgejo, Flux, the monitoring stack, and Flagsmith
 scenarios/       the failure scenarios and their overlays
 scripts/         bring-up, teardown, build, smoke, validation
 docs/            topology, services, scenarios
@@ -97,6 +101,7 @@ Starting up the stack downloads from
 - `gcr.io`
 - `ghcr.io`
 - `codeberg.org`
+- `docker.flagsmith.com`, which redirects to `docker.io`
 - `proxy.golang.org`
 - `crates.io`
 - `prometheus-community.github.io`
@@ -107,6 +112,10 @@ cluster, and the third-party telemetry defaults that do so are turned off in
 the manifests
 - Grafana usage reporting and update checks
 - Forgejo avatar fetching
+- Flagsmith's telemetry heartbeat. Its dashboard's own feature flags are read
+  from the stack's Flagsmith rather than Flagsmith's hosted service, as
+  Flagsmith's docs describe; a dashboard opened during bring-up, before the
+  seed, still reads them from the hosted service
 
 Flux still re-fetches the chart index hourly, and the other components have
 not been audited. This project is not air-gapped; block egress at the podman
@@ -124,9 +133,11 @@ has more details.
 The project is intended to be run with make targets that call bash scripts.
 
 ```sh
-make up        # registry, cluster, images, GitOps, monitoring, services, smoke
+make up        # registry, cluster, images, GitOps, monitoring, services,
+               # Flagsmith, smoke
 make smoke     # exercise the service APIs
-make validate  # pods, ingress, metrics, Prometheus targets, Flux
+make validate  # pods, ingress, host port boundary, metrics, Prometheus
+               # targets, the Flagsmith seed, Flux
 make build     # registry, proto code, image builds and pushes
 make down      # delete cluster, registry, network, kubeconfig
 make lint      # shellcheck, actionlint, kustomize build, semgrep; needs no cluster
@@ -140,7 +151,9 @@ scripts take `--help` and document their supported flags.
 
 `setup.sh` calls `build.sh`, so `make up` or a bare `bash scripts/setup.sh` on
 a clean machine runs everything, and it exits non-zero if the smoke test at
-the end fails. The scripts are idempotent and can be safely rerun. Reset any
+the end fails. The scripts are idempotent and can be safely rerun, with one
+caveat about the Flagsmith password noted under
+[Reaching example-stack](#reaching-example-stack). Reset any
 applied failure scenario before rerunning `setup.sh`, because its seed push
 restores the manifests in Forgejo but not the database - so a scenario that
 changed the schema stays broken until its script is run with `--reset`.
@@ -167,9 +180,19 @@ with a flag on `setup.sh`:
 | 6551 | Kubernetes API | `--k8s-api-port` |
 | 5111 | local image registry | `--registry-port` |
 
+All three are bound to 127.0.0.1 on the host, so nothing in the stack is
+reachable from the network the machine is on. The `*.localhost` names resolve
+locally and were never the boundary; the loopback binding is.
+
+`validate-stack.sh` takes the same three port flags, `--ingress-port`,
+`--k8s-api-port`, and `--registry-port`. It checks each port for that binding,
+so if the defaults were changed on `setup.sh`, the same values have to be
+given here or the boundary check probes ports nothing is listening on and
+fails.
+
 Everything is currently HTTP. No Ingress declares TLS, so the cluster publishes
-no HTTPS port. `--ingress-port` also rewrites the links on the landing page and
-in the UI.
+no HTTPS port. `--ingress-port` also rewrites the links on the landing page,
+in the UI, and in the pages Flagsmith builds for itself.
 
 Everything HTTP is reachable at a `*.localhost` hostname. `curl`, Chrome, and
 Firefox will resolve those names to 127.0.0.1 themselves, which is all the
@@ -187,6 +210,7 @@ only under systemd-resolved or nss-myhostname. If you need those, add one
 | http://forgejo.localhost:8090/ | Forgejo |
 | http://grafana.localhost:8090/ | Grafana |
 | http://prometheus.localhost:8090/ | Prometheus |
+| http://flagsmith.localhost:8090/ | Flagsmith |
 
 `go-grpc` only speaks HTTP/2 and is not exposed through the ingress. You can
 reach it with a port-forward
@@ -209,6 +233,7 @@ way on any machine. None of it is a production secret.
 | Forgejo | user `bootstrap`, password `password` | `--forgejo-password` on `scripts/setup.sh` |
 | Grafana | `admin` / `admin` | `k8s/infra/monitoring/helmrelease.yaml` |
 | Flux Receiver webhook | shared secret `example-stack-webhook-secret` | `k8s/infra/flux/receiver.yaml` |
+| Flagsmith | `bootstrap@flagsmith.local` / `example-stack-demo` | `--flagsmith-password` on `scripts/setup.sh` |
 
 The PostgreSQL container runs with `POSTGRES_HOST_AUTH_METHOD=trust`, so the
 password is never checked and any connection from inside the cluster is
@@ -217,6 +242,14 @@ a connection string from it). The browser UI never holds the bearer token: nginx
 injects it server-side from the Secret, through the envsubst step in
 `k8s/apps/base/ui/nginx.conf.template`, so the token has one home.
 
+Flagsmith's account is created once, on the first bring-up. Later runs do not
+reset its password: they log in with the one they are given, so every later
+`setup.sh` has to be given the same value. If you pass `--flagsmith-password`
+on the first run, pass it on every run. If you change the password in the
+Flagsmith web interface, pass the new value from then on, or the seed step
+fails at login. Flagsmith applies Django's password validators, so a short or
+common value is rejected at signup.
+
 `setup.sh` sets the Forgejo bootstrap password on every run. Pass
 `--forgejo-password` to override the default or change it afterwards
 
@@ -270,13 +303,15 @@ There are four layers of testing. All exit non-zero on failure.
   run the `rust-inventory` handlers that answer before a query: authentication,
   request validation, and liveness.
 - **`scripts/smoke-test.sh`** checks service behavior on a live stack:
-  migration Jobs succeeded or already garbage-collected, health and readiness,
+  migration Jobs succeeded, health and readiness,
   authentication rejection, the seeded inventory, order creation across the
   full call graph, insufficient stock, an unknown item, and the UI proxy paths.
 - **`scripts/validate-stack.sh`** checks the infrastructure: pod health,
-  ingress resolution, `/metrics` endpoints, Prometheus scrape targets, alert
-  rules loaded, alerts not firing, and Flux reconciliation status. It does not
-  run the scenarios.
+  ingress resolution, that each of the three host ports answers on 127.0.0.1
+  and not on the machine's own network address, `/metrics` endpoints,
+  Prometheus scrape targets, alert rules loaded, alerts not firing, that both
+  seeded Flagsmith environments answer for the keys the bring-up stored, and
+  Flux reconciliation status. It does not run the scenarios.
 - **`--verify` on a scenario's `demo.sh`** checks that the scenario broke what
   it expects to break, and that `--reset` put the stack back correctly.
 

@@ -18,8 +18,13 @@ A webhook triggers a Flux reconciliation immediately.
 
 `scenarios/lib.sh` has shared code between the scenarios' `demo.sh` scripts.
 They share flags, the Forgejo token and port-forward, the clone, the overlay
-push, the Flux path switch, and a wait helper. Each `demo.sh` contains only
-what is specific to its own failure. `demo.sh` replaces its overlay directory
+push, the Flux path switch, and a wait helper. The token is read from the
+`forgejo-bootstrap` Secret the bring-up wrote and reused while Forgejo still
+accepts it, which is the "Reusing the Forgejo access token" line the scripts
+print; one is minted through the Forgejo command line when that Secret is
+missing or the token check does not answer 200, whether because Forgejo
+rejected the token or because the port-forward had not answered yet. Each
+`demo.sh` contains only what is specific to its own failure. `demo.sh` replaces its overlay directory
 in the clone, so a file dropped from a scenario does not survive in Forgejo
 and re-running the script with nothing to change pushes nothing. A wait that
 times out exits non-zero.

@@ -184,20 +184,24 @@ infrastructure interface with the ports this stack uses.
 
 ## PostgreSQL and the migrations
 
-There is a single PostgreSQL 17 Deployment that holds both databases.
-`inventory` is created by the image's own initialization; `orders` is created
-by an initialization script mounted from a ConfigMap. Data is on a 100Mi
-local-path volume, which survives pod restarts and `k3d cluster stop` and is
-lost on cluster delete.
-
-Schema and seed data are applied by Kubernetes Jobs running
-[goose](https://github.com/pressly/goose), not by services. The services
-expect the schema to exist and fail if it does not.
+There is a single PostgreSQL 17 Deployment that holds every database in the
+stack. `inventory` is created by the image's own initialization; `orders` and
+`flagsmith` are created by an initialization script mounted from a ConfigMap,
+which the image runs only against an empty data directory, so they appear on
+a fresh volume and not on a restart. Data is on a 100Mi local-path volume,
+which survives pod restarts and `k3d cluster stop` and is lost on cluster
+delete.
+
+Schema and seed data are applied by Kubernetes Jobs, not by services:
+[goose](https://github.com/pressly/goose) for the two service databases,
+Flagsmith's own Django migrations for its. The services expect the schema
+to exist and fail if it does not.
 
 | Job | Database | Description |
 |---|---|---|
 | `migrate-inventory-v1` | `inventory` | Creates the inventory table and seeds three items |
 | `migrate-orders-v1` | `orders` | Creates the orders table |
+| `migrate-flagsmith-v1` | `flagsmith` | Runs Flagsmith's own Django migrations from its image, not goose |
 
 The migration SQL lives in `migrate/migrations/inventory/` and
 `migrate/migrations/orders/`, embedded into the `migrate` binary
@@ -206,13 +210,63 @@ filesystem directory after the embedded set, against the same
 `goose_db_version` table, which is how scenario 2 adds a migration
 without a variant image. Job names carry a version suffix: when
 migration content changes, the suffix is bumped, so Flux creates a new
-Job and leaves the completed one alone. Finished Jobs are
-garbage-collected after ten minutes.
+Job and leaves the completed one alone. Completed Jobs stay: a TTL would
+delete them, and Flux would re-create and re-run them on its next
+reconcile.
 
 On a fresh cluster the services may restart once or twice while the
 migration Jobs run. Those restarts show up in
 `kube_pod_container_status_restarts_total`.
 
+## Flagsmith
+
+Flagsmith is a self-hosted feature flag server, deployed by Flux from
+`k8s/infra/flagsmith` into its own `flagsmith` namespace the way the
+monitoring stack is. It is the unified image, so one container serves both
+the API and the web interface on port 8000, reachable at
+`http://flagsmith.localhost:8090/`. Its schema lives in the `flagsmith`
+database in the shared PostgreSQL and is applied by the
+`migrate-flagsmith-v1` Job, not by the server.
+
+| Endpoint | Auth | Description |
+|---|---|---|
+| `GET /health/liveness/` | No | Liveness. Returns `{"status": "ok"}` without touching the database |
+| `GET /health/readiness/` | No | Readiness. 200 while the database is reachable and no migration is outstanding, 500 otherwise |
+| `GET /metrics/` on port 9100 | No | Prometheus metrics, from a second HTTP server in the same process. A `ServiceMonitor` registers it |
+| `/api/v1/...` | Yes | The Flagsmith REST API. `Authorization: Token <key>` for the admin API, `X-Environment-Key: <key>` for the SDK endpoints |
+
+`scripts/setup.sh` seeds it after Flux has reconciled it: a bootstrap account,
+an organization and a project both named `example-stack`, and exactly two
+environments, `staging` and `production`. No feature flags are created, and
+no service in this stack reads a flag yet.
+
+The two environments' client-side keys are written into a
+`flagsmith-bootstrap` Secret in the `example-stack` namespace, under the keys
+`staging` and `production`, alongside the bootstrap account's API token under
+`admin-token`. Flagsmith generates all three at seed time, so they cannot live
+in the manifests Flux reconciles; `setup.sh` writes the Secret with
+`kubectl`. Flux prunes only what it applied, so a reconcile leaves the Secret
+alone, and the `app.kubernetes.io/managed-by` label on it records which script
+owns it. Nothing mounts it yet.
+
+The dashboard reads its own feature flags from a Flagsmith project too, the
+vendor's hosted one by default. The seed creates a second organization and
+project, both `flagsmith-dashboard`, with one environment, `dashboard`, and
+no flags, since the dashboard has a built-in default for each. The
+organization exists because Flagsmith's free plan, the default when
+self-hosting too, allows one project per organization. Its client-side key
+goes into a
+`flagsmith-dashboard` Secret in the `flagsmith` namespace, under
+`client-key`, which the Deployment reads as `FLAGSMITH_ON_FLAGSMITH_API_KEY`;
+`setup.sh` restarts Flagsmith when the key changes, because Django reads it
+only at startup.
+
+The bootstrap login is `bootstrap@flagsmith.local` with the password
+`--flagsmith-password` sets, `example-stack-demo` by default. It is a
+demo-only credential like every other one in this stack. Flagsmith runs
+Django's password validators on signup, so a short or common password is
+rejected.
+
 ## How the services are built
 
 Every image is a multi-stage Dockerfile built by podman through