From f5b70d101afeeabd862ff86aeaca04d0e39f03ae Mon Sep 17 00:00:00 2001
From: Joris Wouter Jonkers <info@jorisjonkers.dev>
Date: Tue, 18 Aug 2026 21:01:21 +0200
Subject: [PATCH] fix(observability): stop dcgm-exporter OOM-crashlooping and
 quiet pyroscope
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

dcgm-exporter had accumulated 8198 restarts on enschede-gtx-960m-1 and
9398 on enschede-t1000-1 — roughly 17,600 OOM kills. Every start died
with exit 137 about two seconds after logging "DCGM successfully
initialized!", then CrashLoopBackOff settled at its 5-minute ceiling, so
each node re-ran the cycle 247 times a day indefinitely.

The cause was `limits.memory: 256Mi` in this HelmRelease. The upstream
chart defaults to 512Mi, and a 4.x exporter's resident set sits near
450 MiB once the DCGM host engine is up, so the override was below the
floor the process needs to reach a steady state. Limits now allow 1Gi
with a 256Mi request.

The practical consequence was a fleet-wide GPU telemetry blackout: no
DCGM metric has been recorded on either GPU node for a month, which is
why idle GPU power draw on the 960M node was invisible.

Separately, the pyroscope chart ships `extraArgs.log.level: debug`. A
single-binary homelab install logs every block flush and memberlist tick
at that level and nothing reads them, so the release now pins
`log.level: info` under the chart-name key where the chart reads it.
---
 .../apps/observability/nvidia-dcgm-exporter/release.yaml | 9 +++++++--
 .../flux/apps/observability/pyroscope/release.yaml       | 5 +++++
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/platform/cluster/flux/apps/observability/nvidia-dcgm-exporter/release.yaml b/platform/cluster/flux/apps/observability/nvidia-dcgm-exporter/release.yaml
index 500c4caa..49513e4d 100644
--- a/platform/cluster/flux/apps/observability/nvidia-dcgm-exporter/release.yaml
+++ b/platform/cluster/flux/apps/observability/nvidia-dcgm-exporter/release.yaml
@@ -52,12 +52,17 @@ spec:
       enabled: true
       type: ClusterIP
       port: 9400
+    # dcgm-exporter 4.x resident set sits around 450 MiB once the DCGM
+    # host engine has initialised, so the previous 256Mi ceiling killed it
+    # ~2s after `DCGM successfully initialized!` on every start. The
+    # upstream chart default is 512Mi; 1Gi leaves headroom above the
+    # observed steady state.
     resources:
       requests:
         cpu: 50m
-        memory: 64Mi
+        memory: 256Mi
         nvidia.com/gpu: 1
       limits:
         cpu: 200m
-        memory: 256Mi
+        memory: 1Gi
         nvidia.com/gpu: 1
diff --git a/platform/cluster/flux/apps/observability/pyroscope/release.yaml b/platform/cluster/flux/apps/observability/pyroscope/release.yaml
index b8c3b92b..cce4fd41 100644
--- a/platform/cluster/flux/apps/observability/pyroscope/release.yaml
+++ b/platform/cluster/flux/apps/observability/pyroscope/release.yaml
@@ -15,6 +15,11 @@ spec:
   values:
     pyroscope:
       replicaCount: 1
+      # The chart ships `extraArgs.log.level: debug`, which makes a
+      # single-binary homelab install log every block flush and memberlist
+      # tick. Nothing consumes those lines.
+      extraArgs:
+        log.level: info
       persistence:
         enabled: true
         size: 20Gi