Skip to content

[shell-operator] fix: prevent silent empty snapshots from lost kubernetes monitors 1.17#923

Merged
ldmonster merged 3 commits into
release-1.17from
fix/hook-snapshot-loss-1.17
Jul 9, 2026
Merged

[shell-operator] fix: prevent silent empty snapshots from lost kubernetes monitors 1.17#923
ldmonster merged 3 commits into
release-1.17from
fix/hook-snapshot-loss-1.17

Conversation

@ldmonster

@ldmonster ldmonster commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Overview

Fixes a class of failures where a hook keeps running "successfully" while one of its
kubernetes bindings silently feeds it empty snapshots, because the binding's
monitor or its shared informer was lost and never restored.

Two root causes are addressed, plus a hardening fix for the events goroutine.

What this PR does / why we need it

1. EnableKubernetesBindings is now idempotent and self-healing
(pkg/hook/controller/kubernetes_bindings_controller.go)

Previously the method bailed out early when len(BindingMonitorLinks) == len(KubernetesBindings),
comparing only counts. A DisableKubernetesBindings racing with Enable could leave a
binding whose link exists but whose monitor is gone — after that, every
SnapshotsFor call for the binding returned empty while the hook continued to succeed.

Now each binding is checked individually: a missing monitor is re-added and restarted
(guarded against duplicates), and a Synchronization BindingExecutionInfo is always
returned for every binding, so a repeated call after a partial failure never drops the
Synchronization run that populates hook snapshots. Additionally, SnapshotsFor logs a
warning when a configured binding has no live monitor, making the previously silent
empty-snapshot state observable.

2. Shared informer lifetime now belongs to the FactoryStore, not its first consumer
(pkg/kube_events_manager/factory.go)

Factory contexts were derived from the context of whichever consumer happened to create
the shared factory first — stopping that consumer killed the shared informer for every
other monitor using the same (namespace, GVR) index. Worse, the dead factory stayed in
the store and was happily reused: HasSynced() on a stopped informer returns true,
event handlers can no longer be added, and the consumer got a snapshot that never updates.

Now:

  • factory contexts derive from the store owner's (KubeEventsManager) context;
  • Start detects a factory whose informer goroutine has exited, drops it, and builds a
    fresh one;
  • Reset cancels informer goroutines instead of leaking them detached;
  • AddEventHandler errors fail Start with an error instead of being logged and ignored
    (a consumer without a handler would degrade silently);
  • the cache-sync wait was moved outside the store lock, so one slow initial LIST no longer
    serializes Start/Stop of every other informer.

3. Panic isolation for event callbacks
(pkg/shell-operator/manager_events_handler.go)

Schedule and kube-event callbacks now run with recover() (mirroring
TaskQueue.processOne), so a panicking hook dispatch logs a warning with a stack trace
instead of killing the events goroutine — and with it the whole process.

Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
@ldmonster ldmonster self-assigned this Jul 9, 2026
@ldmonster ldmonster added the bug Something isn't working label Jul 9, 2026
@ldmonster ldmonster changed the base branch from main to release-1.17 July 9, 2026 08:41
@ldmonster ldmonster changed the title [shell-operator] fix/hook snapshot loss 1.17 [shell-operator] fшx: prevent silent empty snapshots from lost kubernetes monitors 1.17 Jul 9, 2026
@ldmonster ldmonster changed the title [shell-operator] fшx: prevent silent empty snapshots from lost kubernetes monitors 1.17 [shell-operator] fix: prevent silent empty snapshots from lost kubernetes monitors 1.17 Jul 9, 2026
ldmonster added 2 commits July 9, 2026 11:52
Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
…ts first consumer

Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
@ldmonster ldmonster merged commit e3d9d59 into release-1.17 Jul 9, 2026
6 checks passed
@ldmonster ldmonster deleted the fix/hook-snapshot-loss-1.17 branch July 9, 2026 12:35
Glitchy-Sheep added a commit that referenced this pull request Jul 9, 2026
EnableKubernetesBindings dropped the Synchronization run after a partial
failure and reused a link whose monitor was gone, so hooks read an empty
snapshot while still succeeding. Make the method idempotent, repair a
missing monitor, and warn when a configured binding has no live monitor.

Ported from #923.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Glitchy-Sheep added a commit that referenced this pull request Jul 10, 2026
… first consumer

Port of the FactoryStore part of #923 (released in v1.17.7, already
running in deckhouse release-1.76):

- factory contexts derive from the store's base context (the owning
  kubeEventsManager); stopping the consumer that created a factory no
  longer kills the shared informer for every other consumer
- Start detects a factory whose informer goroutine has exited, drops it
  and builds a fresh one instead of reusing a corpse whose HasSynced()
  is still true and which never delivers events
- AddEventHandler failure fails Start instead of a Warn with a nil
  registration
- Reset() cancels factories before dropping references
- the cache-sync poll runs outside the store lock, so one slow LIST
  does not serialize Start/Stop of every informer in the process

On top of the port: each dead-factory detection increments the new
{PREFIX}factory_informer_dead_total{gvr,namespace} counter.

NewFactoryStore() becomes NewFactoryStore(ctx) - the same exported API
change that already shipped upstream in v1.17.7.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant