From 55804161a5134ac9660b0ca270b5affbc440922a Mon Sep 17 00:00:00 2001 From: Joris Wouter Jonkers Date: Wed, 19 Aug 2026 09:28:48 +0200 Subject: [PATCH] fix(observability): stop grafana-operator losing leader election every few hours MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit grafana-operator had restarted 190 times in 70 days, roughly three times a day, always the same way: info setup Failed to renew lease {"err": "context deadline exceeded"} error setup problem running operator {"error": "leader election lost"} exit 1 It never entered CrashLoopBackOff because it stayed up two minutes or more per cycle, so the failure was invisible in pod status. The release pinned the operator to `kubernetes.io/arch: arm64`, placing it on an Enschede Pi to free Frankfurt CPU, on the stated reasoning that it only reaches Grafana when reconciling CRs and could absorb the ~30 ms cross-site hop. The lease is what that missed: controller-runtime renews leader election against the kube-apiserver every few seconds for the life of the process, and the apiserver runs on the Frankfurt VPS. Every renewal crossed the tailnet, and jitter past the 10 s renew deadline terminated the process. The operator now selects `personal-stack/site: frankfurt`, alongside both the Grafana it manages and the apiserver it renews against, which takes the WAN hop out of the renewal path. It requests 50m CPU and 96Mi, so the capacity it returns to Enschede is not worth the restart loop. Top-level `nodeSelector` is correct for this chart — the live Deployment already carried the arm64 selector from that same key, so it is read rather than silently dropped. --- .../grafana-operator/release.yaml | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/platform/cluster/flux/apps/observability/grafana-operator/release.yaml b/platform/cluster/flux/apps/observability/grafana-operator/release.yaml index 48448683..fc10ad65 100644 --- a/platform/cluster/flux/apps/observability/grafana-operator/release.yaml +++ b/platform/cluster/flux/apps/observability/grafana-operator/release.yaml @@ -13,13 +13,22 @@ spec: name: grafana-operator namespace: observability values: - # Moved to a Pi to free Frankfurt CPU. The operator only talks to - # Grafana (cross-site ~30 ms) when reconciling CRs, which is rare - # — dashboards are static once synced. Image grafana/grafana-operator - # is multi-arch on ghcr.io. Negligible runtime overhead from the - # latency hit. + # Pinned to Frankfurt, alongside the Grafana it manages and the + # kube-apiserver it holds a leader-election lease against. + # + # This previously selected `kubernetes.io/arch: arm64` to free + # Frankfurt CPU, on the reasoning that the operator only reaches + # Grafana when reconciling CRs and so could tolerate the ~30 ms + # cross-site hop. That missed the lease: controller-runtime renews + # leader election against the apiserver every few seconds for the + # life of the process, and from Enschede every renewal crossed the + # tailnet. Jitter past the 10 s renew deadline killed the operator + # with "failed to renew lease: context deadline exceeded" -> + # "leader election lost" -> exit 1, roughly three times a day and + # 190 times over 70 days. Co-locating with the apiserver removes the + # WAN hop from the renewal path. nodeSelector: - kubernetes.io/arch: arm64 + personal-stack/site: frankfurt resources: requests: cpu: 50m