Repository navigation
Make failure handling in EventRecorder configurable (fail-on-error option) #3687
Description
Is your feature request related to a problem? Please describe.
Currently, DefaultEventRecorder#record treats recording strictly as "best effort": any exception thrown when writing to the API server via EventSink#emit is logged as a warning and swallowed:
try {
sink.emit(toEvent(context, event), context);
} catch (Exception e) {
// recording an event must never break the caller: a controller that fails to reconcile
// because it could not write an event is strictly worse than one that records nothing
log.warn("Could not record {} event ...", ...);
}While swallowing errors is desirable for informational telemetry, in our use case (and in environments with strict audit, compliance, or state tracking requirements), emitting a Kubernetes Event is considered part of the required reconciliation side-effects. If the event cannot be persisted (e.g. due to RBAC issues, network partition, or API server failure), continuing reconciliation without failing leaves the cluster in an inconsistent or un-audited state. We want the reconciliation to fail and re-queue/retry when an event fails to be recorded.
Describe the solution you'd like
Make this behavior configurable. When enabled, failures to record an event should propagate as an exception to trigger standard error handling and reconciliation retries, while keeping the current non-failing behavior as the default.
Describe alternatives you've considered
Extending DefaultEventRecorder and overriding the record method. However, this is not clean because it requires duplicating framework logic (such as parameter validation and calling toEvent(...) / sink.emit(...)) just to bypass the try-catch block.
Additional context
None
Activity
afalhambra-hivemq commentedon Nov 10, 2026
@bertdw-1985 Have a look at #3688 where it adds a
failOnError(true)option toDefaultEventRecorder.Builder, off by default.Wire it with
withEventRecorder(DefaultEventRecorder.builder(new DefaultEventSink(client)).failOnError(true).build())and a failure to record propagates to the reconciler, so the reconciliation fails and is retried.csviri commentedon Nov 10, 2026
@afalhambra-hivemq @bertdw-1985 would it be a sin to make this even default the default, it would be a breaking behavior, but it is much more intuitive. What do you think?
afalhambra-hivemq commentedon Nov 11, 2026
@csviri I'd keep it off by default. Kubernetes calls events "best-effort, supplemental data", client-go's recorder never fails the caller and controller-runtime uses that same recorder, so we'd be the only framework failing a reconcile over an event.
It would also bite everyone upgrading from 5.6 with slightly off RBAC (for example a missing verb on events, or no access to
defaultfor cluster scoped objects): today a warning, in 5.7 any reconcile that records an event fails and gets retried. And with event rate limiting on it makes things worse: the 429 fails the reconcile, the retry records the event again, so we write more events exactly while the API server is throttling them.If the worry is a broken setup going unnoticed, I'd rather make the warning louder than fatal.
csviri commentedon Nov 13, 2026
no, makes sense this way, thank you!