fix: complete the submission when the primary is missing from cache at execution time - #3525
Draft
csviri wants to merge 1 commit into
Draft
Conversation
…t execution time
`ReconcilerExecutor.run` re-reads the primary from the cache and bails out
if it is gone. On the non-triggerOnAllEvents path that bail-out just
returned:
} else {
log.debug("Skipping execution; primary resource missing from cache");
return;
}
Two consequences.
The resource stays wedged. `submitReconciliationExecution` has already
called `state.setUnderProcessing(true)` and `state.unMarkEventReceived()`,
so returning without `eventProcessingFinished` leaves the state marked as
under processing with no event pending. `isControllerUnderExecution` then
suppresses every future submission for that resource. Usually a DELETED
event drops the whole `ResourceState` and hides this, but if the resource
leaves the informer cache without a delete event reaching the processor
(a narrowed label selector, `changeNamespaces` dropping a namespace, a
re-list whose synthetic delete is filtered by an `onDeleteFilter`) the
resource is never reconciled again.
Metrics drift. The `return` is inside the `try`, so the `finally` still
reported `reconciliationFinished` even though `reconciliationStarted` was
never reached. With `MicrometerMetricsV2` that decrements
`reconciliations.active` without a matching increment, and leaves the
`reconciliations.queue` increment from `reconciliationSubmitted`
outstanding, so both gauges drift monotonically. The same applied to the
"delete event received meanwhile" branch.
Now completes the submission via `eventProcessingFinished` (as the
sibling branch already did) and only reports `reconciliationFinished` when
`reconciliationStarted` was reported.
Adds regression tests for both; they fail without this change.
Note: `reconciliations.queue` is incremented by `reconciliationSubmitted`
but only decremented by `reconciliationStarted`, so a submission that is
abandoned still leaves the queue gauge one too high. Pairing those
properly needs a decision on the `Metrics` contract (decrement in
`reconciliationFinished`, or add an explicit "submission abandoned"
callback) and is deliberately not changed here.
16 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a correctness issue in the event-processing pipeline where a reconciliation submission could be abandoned if the primary resource vanished from the informer cache before executor execution, leaving the resource stuck “under processing” and causing in-flight metrics gauges to drift.
Changes:
- Ensure skipped executions (primary missing from cache on non-
triggerOnAllEventspath) complete submission viaeventProcessingFinished(...)so theResourceStateis properly released. - Guard
metrics.reconciliationFinished(...)so it is only reported whenmetrics.reconciliationStarted(...)was actually emitted, preventing active/queue gauge drift on skipped executions. - Add regression tests covering both the “state wedged” scenario and the “finished reported without started” metrics scenario.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/EventProcessor.java | Completes skipped submissions and prevents reconciliationFinished from being reported when reconciliation never started. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/EventProcessorTest.java | Adds regression tests ensuring submissions are completed when cache entries disappear pre-execution and metrics are not misreported. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ReconcilerExecutor.runre-reads the primary from the cache and bails outif it is gone. On the non-triggerOnAllEvents path that bail-out just
returned:
Two consequences.
The resource stays wedged.
submitReconciliationExecutionhas alreadycalled
state.setUnderProcessing(true)andstate.unMarkEventReceived(),so returning without
eventProcessingFinishedleaves the state marked asunder processing with no event pending.
isControllerUnderExecutionthensuppresses every future submission for that resource. Usually a DELETED
event drops the whole
ResourceStateand hides this, but if the resourceleaves the informer cache without a delete event reaching the processor
(a narrowed label selector,
changeNamespacesdropping a namespace, are-list whose synthetic delete is filtered by an
onDeleteFilter) theresource is never reconciled again.
Metrics drift. The
returnis inside thetry, so thefinallystillreported
reconciliationFinishedeven thoughreconciliationStartedwasnever reached. With
MicrometerMetricsV2that decrementsreconciliations.activewithout a matching increment, and leaves thereconciliations.queueincrement fromreconciliationSubmittedoutstanding, so both gauges drift monotonically. The same applied to the
"delete event received meanwhile" branch.
Now completes the submission via
eventProcessingFinished(as thesibling branch already did) and only reports
reconciliationFinishedwhenreconciliationStartedwas reported.Adds regression tests for both; they fail without this change.
Note:
reconciliations.queueis incremented byreconciliationSubmittedbut only decremented by
reconciliationStarted, so a submission that isabandoned still leaves the queue gauge one too high. Pairing those
properly needs a decision on the
Metricscontract (decrement inreconciliationFinished, or add an explicit "submission abandoned"callback) and is deliberately not changed here.
Part of #3517