fix: clear compliance_status and paused_resources when asset is skipped - #291
Conversation
|
/rebase |
When an asset enters unmanaged mode (mode=unmanaged annotation) or is excluded via the disabled-resources annotation (Root Exclusion), two gauges were left stale: - compliance_status retained the last managed value (typically 1=synced), falsely implying the operator is actively verifying the asset. - paused_resources could remain stuck at 1 if the asset had been in an edit war before the transition, triggering spurious alerts. Fix: call ClearCompliance(desired) and SetPaused(desired, false) in both early-return paths. customization_info is intentionally left untouched in the Root Exclusion path — the underlying annotations (patch, ignore, unmanaged) persist on the object while excluded, so the metric accurately reflects their presence and will be re-evaluated when exclusion is lifted. Also stabilise resilience_e2e_test.go: replace the direct metric capture + post-loop Expect with touchHCO()+waitForReconcileSucceeded() to guarantee all gauges are populated before freezing the baseline snapshot. New test coverage: - user_override_test.go: compliance_status absent while unmanaged, re-emitted as 1 after resuming managed mode - controller_e2e_test.go: compliance_status cleared while excluded, paused_resources not stuck at 1; and a new It covering the unmanaged→excluded transition (customization_info persists, compliance_status is cleared) Fixes: CNV-94678 Related: CNV-92583 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
/approve |
|
/cherry-pick release-4.22 |
|
@tiraboschi: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tiraboschi The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
841fbc3
into
openshift-virtualization:main
|
@tiraboschi: new pull request created: #298 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
3 participants
Short Description
Two early-return paths in
ReconcileAssetleftcompliance_statusandpaused_resourcesstale when an asset was skipped — keeping the last managed value indefinitely until the operator pod restarted.More Details
When an asset enters unmanaged mode (
platform.kubevirt.io/mode: unmanaged) or is excluded viadisabled-resourcesannotation (Root Exclusion), the reconciler returns early without clearing metrics. This causes:compliance_statusstays at1(synced) → false positive: the operator is not verifying the asset at all.paused_resourcesstays at1if the asset had been in an edit war before the transition → spuriousVirtPlatformThrashingDetectedalert.The Root Exclusion path is the only exclusion mechanism with this problem. Assets excluded via allowlist, missing gate CRD, or unmet conditions are correctly handled by
CleanupExcludedAsset→DeleteAssetMetricsat the controller level before reachingReconcileAsset.customization_infois intentionally not cleared in the Root Exclusion path: the underlying annotations (patch,ignore,unmanaged) persist on the object while excluded, so the metric accurately reflects their presence and will be re-evaluated when exclusion is lifted.What this PR does / why we need it
ClearCompliancehelper topkg/observability/metrics.go(with unit test).ClearCompliance(desired)+SetPaused(desired, false)in both the unmanaged and Root Exclusion early-return paths ofReconcileAsset.resilience_e2e_test.go: replaces direct metric capture + post-loopExpectwithtouchHCO()+waitForReconcileSucceeded()to guarantee all gauges are populated before freezing the baseline snapshot.user_override_test.go:compliance_statusabsent while unmanaged, re-emitted as1after resuming managed mode.controller_e2e_test.go:compliance_statuscleared while excluded,paused_resourcesnot stuck at1; newItcovering the unmanaged→excluded transition (verifiescustomization_infopersists while excluded,compliance_statusis cleared).Fixes: CNV-94678
Related: CNV-92583