Skip to content
This repository was archived by the owner on Aug 26, 2026. It is now read-only.

fix(observability): schedule pyroscope on frankfurt as intended - #763

Merged
ExtraToast merged 1 commit into
mainfrom
fix/pyroscope-to-frankfurt
Aug 18, 2026
Merged

fix(observability): schedule pyroscope on frankfurt as intended#763
ExtraToast merged 1 commit into
mainfrom
fix/pyroscope-to-frankfurt

Conversation

@ExtraToast

Copy link
Copy Markdown
Owner

Replaces #761, which GitHub auto-closed when its stacked base branch was deleted on merge of #760. Same commit, now based on main.

Problem

pyroscope-0 has been running on enschede-gtx-960m-1 — a 2015 laptop — for 109 days, despite the HelmRelease declaring frankfurt.

Root cause

The selector was at the top level of values:

values:
  pyroscope:
    replicaCount: 1
    ...
  nodeSelector:              # <-- discarded
    personal-stack/site: frankfurt

The chart reads scheduling keys under its own pyroscope key. A top-level nodeSelector is silently dropped, so the pod was scheduled with no constraint at all and landed wherever the scheduler put it. This is the same nesting trap the alloy, tempo and loki releases hit.

Change

  • nodeSelector moves under pyroscope, where the chart reads it.
  • placement_intent.frankfurt_only gains a pyroscope entry so the inventory states the same intent as the release.

Why it matters here

The 960M node's entire CPU baseline is 1160m, of which only 284m is workloads — the rest is host daemons. pyroscope accounted for roughly 17m of steady CPU and 120 MiB on that node, and it was never meant to be there.

Operational note — this needs a manual step after merge

The existing 20Gi volume cannot follow the pod:

data-pyroscope-0  sc=local-path  RWO
PV nodeAffinity: kubernetes.io/hostname In [enschede-gtx-960m-1]

local-path with reclaimPolicy: Delete. The StatefulSet will stay Pending until the PVC is deleted so a fresh volume provisions on frankfurt, which discards accumulated profile history. Profiles are short-horizon debugging data, so the retention loss is accepted. Frankfurt has 236 GiB free.

Validation

  • All five render scripts run clean (no rendered-output drift).
  • kubectl kustomize platform/cluster/flux/clusters/production builds.
  • cd platform/tests && node --test — 56/56 pass.

The pyroscope HelmRelease carried `nodeSelector.personal-stack/site:
frankfurt` at the top level of `values`. The chart reads scheduling keys
under its own `pyroscope` key, so Helm discarded the top-level entry and
the pod was scheduled without any constraint — it landed on
enschede-gtx-960m-1, a 2015 laptop, and has run there for 109 days.

Nesting the selector under `pyroscope` makes the constraint take effect.
`placement_intent.frankfurt_only` gains a `pyroscope` entry so the
inventory states the same intent the release does.

This moves roughly 17m of steady CPU and 120 MiB off a node whose entire
CPU baseline is 1160m, of which only 284m was workloads.

The existing 20Gi volume is `local-path` with a `kubernetes.io/hostname`
nodeAffinity pinning it to enschede-gtx-960m-1, so it cannot follow the
pod. The PVC has to be deleted for the StatefulSet to provision a fresh
one on frankfurt, which discards accumulated profile history. Profiles
are short-horizon debugging data and the retention loss is accepted.
@ExtraToast ExtraToast added the bug Something isn't working label Aug 18, 2026
@ExtraToast ExtraToast self-assigned this Aug 18, 2026
@ExtraToast
ExtraToast merged commit e03e590 into main Aug 18, 2026
22 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant