This repository was archived by the owner on Aug 26, 2026. It is now read-only.
fix(observability): stop dcgm-exporter OOM-crashlooping and quiet pyroscope - #760
Merged
Conversation
…oscope dcgm-exporter had accumulated 8198 restarts on enschede-gtx-960m-1 and 9398 on enschede-t1000-1 — roughly 17,600 OOM kills. Every start died with exit 137 about two seconds after logging "DCGM successfully initialized!", then CrashLoopBackOff settled at its 5-minute ceiling, so each node re-ran the cycle 247 times a day indefinitely. The cause was `limits.memory: 256Mi` in this HelmRelease. The upstream chart defaults to 512Mi, and a 4.x exporter's resident set sits near 450 MiB once the DCGM host engine is up, so the override was below the floor the process needs to reach a steady state. Limits now allow 1Gi with a 256Mi request. The practical consequence was a fleet-wide GPU telemetry blackout: no DCGM metric has been recorded on either GPU node for a month, which is why idle GPU power draw on the 960M node was invisible. Separately, the pyroscope chart ships `extraArgs.log.level: debug`. A single-binary homelab install logs every block flush and memberlist tick at that level and nothing reads them, so the release now pins `log.level: info` under the chart-name key where the chart reads it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dcgm-exporterhas been inCrashLoopBackOffon both NVIDIA nodes for a month:Roughly 17,600 OOM kills. Each container starts, logs
DCGM successfully initialized!, and is killed with exit 137 about two seconds later.CrashLoopBackOffthen settles at its 5-minute ceiling, so each node repeats the cycle 247 times a day with no end state.Root cause
limits.memory: 256Miinplatform/cluster/flux/apps/observability/nvidia-dcgm-exporter/release.yaml.The upstream chart defaults to
512Mi, so this override lowered the ceiling. A 4.x exporter's resident set settles near 450 MiB once the DCGM host engine has initialised, which is above the 256Mi cap — the process cannot reach a steady state at that limit regardless of how many times it restarts.Change
limits.memory256Mi to 1Gi,requests.memory64Mi to 256Mi. 1Gi leaves headroom above the observed steady state rather than sitting right at the chart default.extraArgs.log.level: debug. A single-binary install logs every block flush and memberlist tick at that level and nothing consumes the output, so the release now pinslog.level: infounder the chart-name key where the chart actually reads it.Impact
GPU telemetry has been absent fleet-wide for the full month — no DCGM metric exists for either GPU node. That blackout is why idle GPU power draw on
enschede-gtx-960m-1could not be measured while investigating that host's constant load.The exporter also holds
nvidia.com/gpu: 1(one time-slice) on every one of those 247 daily attempts.Validation
kubectl kustomize platform/cluster/flux/clusters/productionbuilds clean.cd platform/tests && node --test— 56/56 pass.