Skip to content

fix: complete the submission when the primary is missing from cache at execution time - #3525

Draft
csviri wants to merge 1 commit into
operator-framework:mainfrom
csviri:fix/event-processor-cache-miss-accounting
Draft

fix: complete the submission when the primary is missing from cache at execution time#3525
csviri wants to merge 1 commit into
operator-framework:mainfrom
csviri:fix/event-processor-cache-miss-accounting

Conversation

@csviri

@csviri csviri commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

ReconcilerExecutor.run re-reads the primary from the cache and bails out
if it is gone. On the non-triggerOnAllEvents path that bail-out just
returned:

} else {
  log.debug("Skipping execution; primary resource missing from cache");
  return;
}

Two consequences.

The resource stays wedged. submitReconciliationExecution has already
called state.setUnderProcessing(true) and state.unMarkEventReceived(),
so returning without eventProcessingFinished leaves the state marked as
under processing with no event pending. isControllerUnderExecution then
suppresses every future submission for that resource. Usually a DELETED
event drops the whole ResourceState and hides this, but if the resource
leaves the informer cache without a delete event reaching the processor
(a narrowed label selector, changeNamespaces dropping a namespace, a
re-list whose synthetic delete is filtered by an onDeleteFilter) the
resource is never reconciled again.

Metrics drift. The return is inside the try, so the finally still
reported reconciliationFinished even though reconciliationStarted was
never reached. With MicrometerMetricsV2 that decrements
reconciliations.active without a matching increment, and leaves the
reconciliations.queue increment from reconciliationSubmitted
outstanding, so both gauges drift monotonically. The same applied to the
"delete event received meanwhile" branch.

Now completes the submission via eventProcessingFinished (as the
sibling branch already did) and only reports reconciliationFinished when
reconciliationStarted was reported.

Adds regression tests for both; they fail without this change.

Note: reconciliations.queue is incremented by reconciliationSubmitted
but only decremented by reconciliationStarted, so a submission that is
abandoned still leaves the queue gauge one too high. Pairing those
properly needs a decision on the Metrics contract (decrement in
reconciliationFinished, or add an explicit "submission abandoned"
callback) and is deliberately not changed here.

Part of #3517

…t execution time

`ReconcilerExecutor.run` re-reads the primary from the cache and bails out
if it is gone. On the non-triggerOnAllEvents path that bail-out just
returned:

    } else {
      log.debug("Skipping execution; primary resource missing from cache");
      return;
    }

Two consequences.

The resource stays wedged. `submitReconciliationExecution` has already
called `state.setUnderProcessing(true)` and `state.unMarkEventReceived()`,
so returning without `eventProcessingFinished` leaves the state marked as
under processing with no event pending. `isControllerUnderExecution` then
suppresses every future submission for that resource. Usually a DELETED
event drops the whole `ResourceState` and hides this, but if the resource
leaves the informer cache without a delete event reaching the processor
(a narrowed label selector, `changeNamespaces` dropping a namespace, a
re-list whose synthetic delete is filtered by an `onDeleteFilter`) the
resource is never reconciled again.

Metrics drift. The `return` is inside the `try`, so the `finally` still
reported `reconciliationFinished` even though `reconciliationStarted` was
never reached. With `MicrometerMetricsV2` that decrements
`reconciliations.active` without a matching increment, and leaves the
`reconciliations.queue` increment from `reconciliationSubmitted`
outstanding, so both gauges drift monotonically. The same applied to the
"delete event received meanwhile" branch.

Now completes the submission via `eventProcessingFinished` (as the
sibling branch already did) and only reports `reconciliationFinished` when
`reconciliationStarted` was reported.

Adds regression tests for both; they fail without this change.

Note: `reconciliations.queue` is incremented by `reconciliationSubmitted`
but only decremented by `reconciliationStarted`, so a submission that is
abandoned still leaves the queue gauge one too high. Pairing those
properly needs a decision on the `Metrics` contract (decrement in
`reconciliationFinished`, or add an explicit "submission abandoned"
callback) and is deliberately not changed here.
Copilot AI review requested due to automatic review settings July 30, 2026 09:04
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a correctness issue in the event-processing pipeline where a reconciliation submission could be abandoned if the primary resource vanished from the informer cache before executor execution, leaving the resource stuck “under processing” and causing in-flight metrics gauges to drift.

Changes:

  • Ensure skipped executions (primary missing from cache on non-triggerOnAllEvents path) complete submission via eventProcessingFinished(...) so the ResourceState is properly released.
  • Guard metrics.reconciliationFinished(...) so it is only reported when metrics.reconciliationStarted(...) was actually emitted, preventing active/queue gauge drift on skipped executions.
  • Add regression tests covering both the “state wedged” scenario and the “finished reported without started” metrics scenario.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/EventProcessor.java Completes skipped submissions and prevents reconciliationFinished from being reported when reconciliation never started.
operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/EventProcessorTest.java Adds regression tests ensuring submissions are completed when cache entries disappear pre-execution and metrics are not misreported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants