fix(observability): pin the prometheus operator and its admission hooks to frankfurt

[ExtraToast]

Problem

metrics-stack could not complete a Helm upgrade:

pre-upgrade hooks failed: timeout waiting for:
[Job/observability/metrics-stack-admission-create status: 'InProgress']

That failure stalled apps-observability, which blocked apps-grafana-dashboards and apps-observability-rules on their dependency — three Kustomizations sat unreconciled.

Root cause

The kube-webhook-certgen create and patch Jobs run as Helm pre-upgrade hooks, so the release fails outright if they do not finish inside the timeout.

This release pinned prometheus and alertmanager to frankfurt but configured no prometheusOperator block at all, so the hook Jobs were scheduled wherever the scheduler put them. Caught in the act:

metrics-stack-admission-patch-9hmlw   ContainerCreating   enschede-pi-1
  Pulling image "ghcr.io/jkroepke/kube-webhook-certgen"

A Pi pulling that image over the home connection takes well over the 5-minute hook timeout. The same omission left metrics-stack-operator and metrics-stack-kube-state-metrics on enschede-gtx-960m-1 while the Prometheus they serve runs on frankfurt.

The upgrade eventually succeeded only because the certgen image ended up cached on that Pi. The next chart bump pulls a new tag and fails the same way, so this needs the pin rather than luck.

Change

Both the operator and the admission-hook Jobs now select personal-stack/site: frankfurt, alongside the Prometheus and Alertmanager they serve, where the image pulls in seconds.

On verifying the value path

prometheusOperator.admissionWebhooks.patch.nodeSelector was confirmed against kube-prometheus-stack 88.5.0, the pinned version, at line 3144 of its values.yaml. The chart's main branch documents this differently — it presents admissionWebhooks.deployment.nodeSelector and states the create/patch Jobs have no separate nodeSelector, which is not true of 88.5.0. Reading the pinned version rather than main is what produced a working path.

Not included

kube-state-metrics is also unpinned and currently on enschede-gtx-960m-1. It is healthy there and moving it is a placement decision rather than part of this failure, so it is left alone.

Validation

• kubectl kustomize platform/cluster/flux/clusters/production builds clean.
• cd platform/tests && node --test — 58/58 pass.
• Cluster state after the manual recovery that preceded this change: every HelmRelease and every Kustomization reports Ready.