Skip to content

[shell-operator] fix: protect shared informer lifecycle#917

Open
fuldaxxx wants to merge 4 commits into
mainfrom
fix/kubeevents-factorystore-ctx-leak
Open

[shell-operator] fix: protect shared informer lifecycle#917
fuldaxxx wants to merge 4 commits into
mainfrom
fix/kubeevents-factorystore-ctx-leak

Conversation

@fuldaxxx

@fuldaxxx fuldaxxx commented Jul 2, 2026

Copy link
Copy Markdown
Member

Overview

Fix a shared-informer lifecycle bug in FactoryStore (pkg/kube_events_manager/factory.go).
A shared informer is a long-lived, store-owned resource, but its context used to be
derived from the first consumer that registered it. Cancelling that consumer's
context (e.g. when its monitor is stopped during a module reload) tore down the shared
informer for every other consumer still using it, silently freezing their snapshots
until the process was restarted. Ownership of the informer lifetime is moved to the
store, plus three supporting fixes (dead-factory detection, a narrower critical section,
and an un-swallowed error).

What this PR does / why we need it

Root cause. NewFactoryStore() did not receive a context, and add() derived the
factory context via context.WithCancel(ctx) from the caller's context. The chain
mgr.ctx → monitor.ctx → resourceInformer.ei.ctx → Start(ei.ctx) → add(ei.ctx) meant
factory.ctx was a child of the ei.ctx of whichever monitor created the factory first.
Because factories are shared across monitors with the same FactoryIndex
(GVR + namespace + selectors), stopping that first monitor cancelled the shared informer
for all remaining consumers, while the refcount (handlerRegistrations) was still > 0 so
the factory stayed cached. The surviving consumers were left attached to a dead informer.

This is a classic ownership bug: the lifetime of a shared, long-lived resource must be
owned by the store, not inherited from a transient consumer.

Impact. The failure is silent (no error, HasSynced() on a stopped-but-once-synced
informer returns true) and only recoverable by restarting the process. In
surfaced as an empty DexClient snapshot in user-authn: the hook feeds the desired
object list into values, Helm prunes anything not in the list, so an empty
to the dex-client-* Secret and OAuth2Client being deleted — breaking OIDC login to
the cluster. Mass module reloads (e.g. during an upgrade) are the trigger w

Special notes for your reviewer

Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
@fuldaxxx fuldaxxx requested review from ipaqsa and ldmonster July 2, 2026 13:10
@fuldaxxx fuldaxxx self-assigned this Jul 2, 2026
@fuldaxxx fuldaxxx added bug Something isn't working go Pull requests that update Go code labels Jul 2, 2026
Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
@fuldaxxx fuldaxxx marked this pull request as ready for review July 6, 2026 14:04
Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a shared-informer lifecycle/ownership bug in FactoryStore so that shared informers are anchored to the events-manager/store lifetime rather than the first consumer’s context, preventing other consumers’ snapshots from silently freezing when that first consumer is stopped.

Changes:

  • Make FactoryStore context-aware (baseCtx) and derive factory/informer contexts from the store instead of from the first registering consumer.
  • Add dead-factory detection/recreation, narrow the store lock critical section during initial sync, and stop silently swallowing a watch error handler registration error.
  • Add targeted tests covering the shared-informer lifetime, shutdown waiting behavior, and dead-factory recreation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
pkg/kube_events_manager/monitor_test.go Updates monitor test setup to pass a context into NewFactoryStore.
pkg/kube_events_manager/kube_events_manager.go Anchors the factory store lifetime to the manager context.
pkg/kube_events_manager/factory.go Reworks shared informer lifecycle ownership and improves concurrency/liveness handling.
pkg/kube_events_manager/factory_test.go Adds regression tests for shared informer ownership, WaitStopped, and dead-factory recreation.
Comments suppressed due to low confidence (1)

pkg/kube_events_manager/factory.go:158

  • Start logs AddEventHandler errors but continues, still recording the (possibly nil) registration and returning nil to the caller. This can leave a broken handler entry and makes failures invisible to resourceInformer.start(). Consider returning the error (after releasing c.mu) so the caller can handle it and Stop won’t later try to remove a nil registration.
	registration, err := informer.AddEventHandler(handler)
	if err != nil {
		log.Warn("Factory store: couldn't add event handler to the factory's informer",
			slog.String(pkg.LogKeyNamespace, index.Namespace), slog.String(pkg.LogKeyGVR, index.GVR.String()),
			log.Err(err))

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +54 to 58
func NewFactoryStore(ctx context.Context) *FactoryStore {
fs := &FactoryStore{
data: make(map[FactoryIndex]*Factory),
stoppedCh: make(map[FactoryIndex]chan struct{}),
data: make(map[FactoryIndex]*Factory),
baseCtx: ctx,
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants