improve: reconciliation counts as retry attempt only if close to retry deadline#3380
Open
csviri wants to merge 5 commits into
Open
improve: reconciliation counts as retry attempt only if close to retry deadline#3380csviri wants to merge 5 commits into
csviri wants to merge 5 commits into
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
csviri
changed the title
csviri
changed the title
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
Contributor
There was a problem hiding this comment.
Pull request overview
Adjusts retry behavior so that a failed reconciliation triggered by an external event while inside an already-armed retry window does not consume a retry attempt; instead the original retry deadline is preserved (when more than 5s remains). Previously every failure during a retry window advanced the retry counter, which could exhaust retries quickly under frequent events.
Changes:
- Added
RetryExecution#remainingDurationUntilNextRetry()(default empty) and implemented it inGenericRetryExecutionby tracking the wall-clock time of the lastnextDelay()call. - In
EventProcessor#handleRetryOnException, when no superseding event is present and the remaining retry window exceeds a 5s threshold, reschedule on the existing deadline without invokingnextDelay(). - Added unit tests for the three branches and an integration test verifying the retry counter does not advance under frequent updates; docs updated accordingly.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| operator-framework-core/.../retry/RetryExecution.java | Adds remainingDurationUntilNextRetry() default method to the interface. |
| operator-framework-core/.../retry/GenericRetryExecution.java | Records timestamp of last nextDelay() call; computes remaining time until deadline. |
| operator-framework-core/.../event/EventProcessor.java | Preserves existing retry deadline when remaining > 5s threshold, skipping retry counter advance. |
| operator-framework-core/.../event/EventProcessorTest.java | Adds tests covering preserve-deadline, consume-attempt, and first-failure branches. |
| operator-framework/.../retry/RetryTestCustomReconciler.java | Tracks max observed RetryInfo attempt count for assertions. |
| operator-framework/.../retry/RetryIntervalHonoredOnFrequentEventsIT.java | New IT verifying frequent updates within retry window do not exhaust retry counter. |
| docs/.../error-handling-retries.md | Documents the new preserve-deadline behavior. |
csviri
commented
May 27, 2026
| * window is allowed to consume a retry attempt (i.e. advance the retry counter). Above this | ||
| * threshold the existing retry deadline is preserved instead. | ||
| */ | ||
| private static final long RETRY_DEADLINE_PRESERVE_THRESHOLD_MILLIS = 5_000; |
Collaborator
Author
There was a problem hiding this comment.
It is up to debate if this should be configurable, but maybe we can for now stick KISS principle
csviri
changed the title
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
xstefank
approved these changes
May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently there is already a retry happening, and we receive an event that triggers the reconciliation counts as a retry. But this way we can fairly easily exhaust retries. So algorithm should is changed a reconciliation counts as a retry only if it is close enough to a scheduled retry.
Signed-off-by: Attila Mészáros a_meszaros@apple.com