From 90e40918601449306c86079fb44c8bb9fbb35a59 Mon Sep 17 00:00:00 2001 From: Joris Wouter Jonkers Date: Wed, 19 Aug 2026 10:13:09 +0200 Subject: [PATCH] fix(observability): pin the prometheus operator and its admission hooks to frankfurt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit metrics-stack could not complete a Helm upgrade: pre-upgrade hooks failed: timeout waiting for: [Job/observability/metrics-stack-admission-create status: 'InProgress'] The kube-webhook-certgen create and patch Jobs run as Helm pre-upgrade hooks, so the release fails outright if they do not finish inside the timeout. The release pinned prometheus and alertmanager to frankfurt but configured no prometheusOperator block at all, so the hook Jobs were scheduled anywhere. They landed on enschede-pi-1 and spent longer than the 5-minute timeout pulling ghcr.io/jkroepke/kube-webhook-certgen over the home connection. That failure stalled the apps-observability Kustomization, which in turn blocked apps-grafana-dashboards and apps-observability-rules on their dependency, so three Kustomizations sat unreconciled. The upgrade only succeeded once the certgen image happened to be cached on that Pi — the next chart bump would pull a new tag and fail the same way. Both the operator and the admission-hook Jobs now select `personal-stack/site: frankfurt`, alongside the Prometheus and Alertmanager they serve, where the image pulls in seconds. Value paths verified against kube-prometheus-stack 88.5.0 rather than chart main, where `admissionWebhooks.patch.nodeSelector` no longer appears in the same form. --- .../observability/metrics-stack/release.yaml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/platform/cluster/flux/apps/observability/metrics-stack/release.yaml b/platform/cluster/flux/apps/observability/metrics-stack/release.yaml index de9f1843..26a66f3a 100644 --- a/platform/cluster/flux/apps/observability/metrics-stack/release.yaml +++ b/platform/cluster/flux/apps/observability/metrics-stack/release.yaml @@ -65,3 +65,22 @@ spec: resources: requests: storage: 50Gi + prometheusOperator: + # The operator belongs with the Prometheus and Alertmanager it + # manages rather than wherever the scheduler happens to place it. + nodeSelector: + personal-stack/site: frankfurt + admissionWebhooks: + patch: + # The kube-webhook-certgen create/patch Jobs run as Helm + # pre-upgrade hooks, so the whole upgrade fails if they do not + # finish inside the release timeout. Unpinned, they landed on a + # Raspberry Pi and spent longer than 5 minutes pulling + # ghcr.io/jkroepke/kube-webhook-certgen over the home + # connection, which failed the upgrade with + # "pre-upgrade hooks failed: timeout waiting for: + # [Job/observability/metrics-stack-admission-create]" and left + # apps-observability — and everything depending on it — + # unreconciled. Frankfurt pulls the image in seconds. + nodeSelector: + personal-stack/site: frankfurt