fix(observability): stop grafana-operator losing leader election every few hours

[ExtraToast]

Problem

grafana-operator has restarted 190 times in 70 days — roughly three times a day, still happening. It never showed up as CrashLoopBackOff because it survives two minutes or more per cycle, so pod status looked healthy.

Every exit is identical:

info   setup  Failed to renew lease  {"lock": "observability/grafana-operator-...", "err": "context deadline exceeded"}
error  setup  problem running operator  {"error": "leader election lost"}
exit 1

Root cause

The release pinned the operator to kubernetes.io/arch: arm64, which places it on an Enschede Pi. The stated reasoning was:

Moved to a Pi to free Frankfurt CPU. The operator only talks to Grafana (cross-site ~30 ms) when reconciling CRs, which is rare — dashboards are static once synced. Negligible runtime overhead from the latency hit.

The premise holds for Grafana traffic and misses the lease. controller-runtime renews leader election against the kube-apiserver, not Grafana, every few seconds for the entire life of the process. The apiserver runs on the Frankfurt VPS, so from Enschede every single renewal crossed the tailnet. Jitter past the 10 s renew deadline terminates the operator.

Grafana itself already runs on frankfurt (nodeSelector: personal-stack/site: frankfurt) — only its operator was remote.

Change

The operator now selects personal-stack/site: frankfurt, co-located with both the Grafana it manages and the apiserver it renews against, removing the WAN hop from the renewal path.

It requests 50m CPU and 96Mi. Frankfurt sits at 46% of allocatable CPU requests, so the capacity handed back to Enschede was not worth a three-times-daily restart loop.

On values nesting

Top-level nodeSelector is correct for this chart, unlike the grafana/pyroscope trap fixed in #763. Verified against the live cluster rather than assumed — the running Deployment already carried {"kubernetes.io/arch":"arm64"}, proving the chart reads that key rather than silently discarding it.

Validation

• kubectl kustomize platform/cluster/flux/clusters/production builds clean.
• cd platform/tests && node --test — 58/58 pass.
• Post-merge check is the restart counter: it should stop advancing past 190.