feat(db): prod techgarden data plane — cnpg cluster, backups, snapshots (WS-02 H1/H2)

[TechGardenCode]

WS-02 PR1 of 3 — data plane. Nothing here is publicly reachable and nothing depends on identity. The restore drill (H6) runs against this empty database before any Keycloak work, which is why this PR is first.

H1 — cnpg cluster + techgarden DB + techgarden_app role

New tree kubernetes/clusters/1276-prod/databases/, modelled on 1276-dev/databases/.

File	What
postgresql/cluster.yaml	cnpg Cluster postgresql in ns databases, instances: 2 (dev runs 1)
postgresql/databases.yaml	Database CR, databaseReclaimPolicy: retain, owner → techgarden_app
postgresql/externalsecret.yaml	postgresql-superuser + postgresql-app-credentials
postgresql/app-externalsecrets.yaml	techgarden-db-app-credentials (managed-role password)
_namespace/	databases namespace

The name/namespace pair is load-bearing: postgresql in databases is what produces postgresql-rw.databases.svc.cluster.local:5432, which the app repo hardcodes in projects/api/deploy/base/api-deployment.yaml. Renaming either silently breaks the API at the flip.

Three ExternalSecrets, not two. The managed-role one carries type: kubernetes.io/basic-auth + username/password keys + the cnpg.io/reload label — cnpg managed roles require all three, and dropping any one creates the role with the password never applied.

Schema-per-module (ADR-0031): one techgarden database, one techgarden_app role. LOGIN NOSUPERUSER NOBYPASSRLS is the RLS invariant. Flyway migrates as superuser postgres (accepted blast-radius tradeoff).

Deliberately not carried over from dev: init-sql.yaml and the postInitApplicationSQLRefs/postInitSQLRefs that reference it — legacy from the retired per-service-database era, unused by the monolith.

H2 — backups from the start

⚠️ This PR fixes a gap that made snapshot backups impossible estate-wide.

shared/kube-system/snapshot-controller/ shipped only the three CRDs — no controller Deployment — on all three clusters. The external-snapshotter sidecar present in every democratic-csi controller pod is not a substitute: it services VolumeSnapshotContent, and nothing was creating the VolumeSnapshotContent in the first place. A VolumeSnapshot would have been accepted and then sat unbound forever, so a ScheduledBackup would hang rather than fail.

• snapshot-controller Deployment + RBAC added next to the CRDs (upstream v8.2.0).
• VolumeSnapshotClass freenas-iscsi-ssd-snapshot — prod had zero classes.
• cnpg backup.volumeSnapshot + daily ScheduledBackup.
• pg_dump CronJob kept as a deliberately redundant logical export. A snapshot captures the block device and inherits on-disk corruption verbatim; a dump fails loudly on it. Both, on purpose.

Blast radius: clusters/shared/** auto-syncs to dev + core + prod together. The controller is inert on dev and core — both have zero VolumeSnapshotClasses and zero VolumeSnapshots, so it starts, elects a leader, and idles.

Known ceilings (documented in-file, not fixed here)

• Snapshot-only, no barmanObjectStore → restore-to-snapshot, not PITR between snapshots. WAL archiving needs S3 credentials outside this workstream's contract.
• cnpg does not enforce retentionPolicy for the volumeSnapshot method, so snapshots accumulate. ZFS snapshots are copy-on-write, so growth is delta-sized.
• Snapshots and the pg_dump PVC both land on cm-nas — same failure domain as the source volume. Neither is an off-box tier.

Backup failure signalling

The existing cluster-wide k8s-job-failed Grafana rule already covers the pg_dump CronJob, so no second alert is defined for it. The cnpg snapshot path is not covered (snapshots aren't Jobs) and gets a cnpg-backup-stale rule — that one lives in kian.sh/ansible (lgtm_config), a different repo and deploy path, so it ships separately.

Validation

• kustomize build --enable-helm green on all four changed directories.
• cnpg CRD field names verified server-side (ScheduledBackup.spec, Cluster.spec.backup.volumeSnapshot).
• All four prod BWS entries verified present in project 72a9d5cf… with the expected keys.

Prod safety gate

The dev-health-check PreToolUse hook did not fire — its regex requires the literal homelab/kubernetes/clusters/…-prod/ adjacency, and the mandated worktree path is worktrees/ws02-pr1/kubernetes/…. I ran the check manually instead: 1276-dev.databases.postgresql, .db-backups and .namespace are all Synced + Healthy, so the gate passes on the merits. No PROMOTE_OVERRIDE used or needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_015tcbrNifvkc8ryk4g2M7XK