You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
lakehouse/statefulset_checker (normal run against a live cluster)
Malformed kubectl JSON shape (e.g. [], null, object without items): confirmed get_statefulsets() now exits cleanly with UNKNOWN: unexpected kubectl JSON shape instead of raising AttributeError
Stuck scale-down (status.replicas still above spec.replicas, e.g. desired=5 / replicas=6 / ready=6): confirmed now reports CRIT ... replicas 6/5 instead of passing silently
Deployment
Deploy to: /usr/lib/check_mk_agent/local/statefulset_checker
Make executable: chmod +x /usr/lib/check_mk_agent/local/statefulset_checker
Status mapping: 0 (OK) all StatefulSets fully ready and deployed, 2 (CRIT) missing ready replicas / incomplete rollout / stuck scale-down, 3 (UNKNOWN) kubectl or parse error
Related: DEVOPS-2883 — prod MinIO outage that exposed the monitoring gap this check addresses (Service-level health checks masked a single downed StatefulSet replica).
Adds a CheckMK local check for Kubernetes StatefulSet readiness and rollout health.
Changes:
Queries StatefulSets via microk8s kubectl.
Reports readiness and rollout failures as CRIT.
Handles command, timeout, and JSON failures as UNKNOWN.
File
Summary
lakehouse/statefulset_checker
Implements StatefulSet collection, validation, health checks, and CheckMK output. Unresolved issues remain around service ID collisions, malformed JSON, scale-down detection, and executable errors.
…downs
- get_statefulsets() now validates the decoded payload is a dict with a
list-valued items field before use, instead of calling .get() on
whatever json.loads() returns (a bare list or null would previously
raise instead of hitting the advertised UNKNOWN path).
- service_line() now flags replicas != desired instead of only
replicas < desired, so a StatefulSet stuck mid-scale-down (status.replicas
still above spec.replicas) reports CRIT instead of passing silently.
Addresses Copilot review comments on PR #50.
Account for StatefulSet ordinal start when calculating rollout updates
lakehouse/statefulset_checker:67
The partition is an absolute pod ordinal, but this calculation assumes ordinals always begin at zero. For a StatefulSet with spec.ordinals.start: 10, three replicas, and partition 11, pods 11 and 12 must be updated, while this returns zero and can report the rollout OK without either update. Account for spec.ordinals.start when deriving the expected count.
…d rollout detection
Address code review findings: pipe-separate perfdata metrics so Checkmk
parses all of them, stop collapsing '-' into the service-name separator
(which could collide distinct namespace/name pairs), emit an OK roll-up
when no StatefulSets exist, catch OSError broadly around the kubectl
call, and detect incomplete rollouts via currentRevision/updateRevision
instead of hand-rolled partition math.
Validate item and nested map shapes before dereferencing
lakehouse/statefulset_checker:52
The shape check stops at the outer items list. Inputs such as {"items":[null]} or an item with "metadata": null pass this guard and then raise AttributeError in main()/service_line() instead of producing the promised UNKNOWN result. Validate each item and the nested maps that are dereferenced before returning the list.
Validate list entries before sorting to prevent AttributeError
lakehouse/statefulset_checker:50
The shape check still accepts non-object entries such as {"items":[null]}. main() then calls item.get(...) while sorting and raises AttributeError, so this unexpected JSON bypasses the promised UNKNOWN result. Validate every list entry before returning it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bio-boris
changed the title
Summary
Testing
python3 -m py_compile lakehouse/statefulset_checkerlakehouse/statefulset_checker(normal run against a live cluster)[],null, object withoutitems): confirmedget_statefulsets()now exits cleanly withUNKNOWN: unexpected kubectl JSON shapeinstead of raisingAttributeErrorstatus.replicasstill abovespec.replicas, e.g. desired=5 / replicas=6 / ready=6): confirmed now reportsCRIT ... replicas 6/5instead of passing silentlyDeployment
/usr/lib/check_mk_agent/local/statefulset_checkerchmod +x /usr/lib/check_mk_agent/local/statefulset_checker0(OK) all StatefulSets fully ready and deployed,2(CRIT) missing ready replicas / incomplete rollout / stuck scale-down,3(UNKNOWN) kubectl or parse errorRelated: DEVOPS-2883 — prod MinIO outage that exposed the monitoring gap this check addresses (Service-level health checks masked a single downed StatefulSet replica).