[shell-operator] fix: prevent silent empty snapshots from lost kubernetes monitors 1.17#923
Merged
Merged
Conversation
Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
…ts first consumer Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
4 tasks
Glitchy-Sheep
added a commit
that referenced
this pull request
Jul 9, 2026
EnableKubernetesBindings dropped the Synchronization run after a partial failure and reused a link whose monitor was gone, so hooks read an empty snapshot while still succeeding. Make the method idempotent, repair a missing monitor, and warn when a configured binding has no live monitor. Ported from #923. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Glitchy-Sheep
added a commit
that referenced
this pull request
Jul 10, 2026
… first consumer Port of the FactoryStore part of #923 (released in v1.17.7, already running in deckhouse release-1.76): - factory contexts derive from the store's base context (the owning kubeEventsManager); stopping the consumer that created a factory no longer kills the shared informer for every other consumer - Start detects a factory whose informer goroutine has exited, drops it and builds a fresh one instead of reusing a corpse whose HasSynced() is still true and which never delivers events - AddEventHandler failure fails Start instead of a Warn with a nil registration - Reset() cancels factories before dropping references - the cache-sync poll runs outside the store lock, so one slow LIST does not serialize Start/Stop of every informer in the process On top of the port: each dead-factory detection increments the new {PREFIX}factory_informer_dead_total{gvr,namespace} counter. NewFactoryStore() becomes NewFactoryStore(ctx) - the same exported API change that already shipped upstream in v1.17.7. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes a class of failures where a hook keeps running "successfully" while one of its
kubernetesbindings silently feeds it empty snapshots, because the binding'smonitor or its shared informer was lost and never restored.
Two root causes are addressed, plus a hardening fix for the events goroutine.
What this PR does / why we need it
1.
EnableKubernetesBindingsis now idempotent and self-healing(
pkg/hook/controller/kubernetes_bindings_controller.go)Previously the method bailed out early when
len(BindingMonitorLinks) == len(KubernetesBindings),comparing only counts. A
DisableKubernetesBindingsracing withEnablecould leave abinding whose link exists but whose monitor is gone — after that, every
SnapshotsForcall for the binding returned empty while the hook continued to succeed.Now each binding is checked individually: a missing monitor is re-added and restarted
(guarded against duplicates), and a
SynchronizationBindingExecutionInfois alwaysreturned for every binding, so a repeated call after a partial failure never drops the
Synchronization run that populates hook snapshots. Additionally,
SnapshotsForlogs awarning when a configured binding has no live monitor, making the previously silent
empty-snapshot state observable.
2. Shared informer lifetime now belongs to the
FactoryStore, not its first consumer(
pkg/kube_events_manager/factory.go)Factory contexts were derived from the context of whichever consumer happened to create
the shared factory first — stopping that consumer killed the shared informer for every
other monitor using the same
(namespace, GVR)index. Worse, the dead factory stayed inthe store and was happily reused:
HasSynced()on a stopped informer returnstrue,event handlers can no longer be added, and the consumer got a snapshot that never updates.
Now:
KubeEventsManager) context;Startdetects a factory whose informer goroutine has exited, drops it, and builds afresh one;
Resetcancels informer goroutines instead of leaking them detached;AddEventHandlererrors failStartwith an error instead of being logged and ignored(a consumer without a handler would degrade silently);
serializes Start/Stop of every other informer.
3. Panic isolation for event callbacks
(
pkg/shell-operator/manager_events_handler.go)Schedule and kube-event callbacks now run with
recover()(mirroringTaskQueue.processOne), so a panicking hook dispatch logs a warning with a stack traceinstead of killing the events goroutine — and with it the whole process.