Skip to content

config-reloader: trigger reload after initial directory sync - #2528

Open
shunki-fujita wants to merge 3 commits into
VictoriaMetrics:masterfrom
shunki-fujita:config-reloader-initial-sync-reload
Open

config-reloader: trigger reload after initial directory sync#2528
shunki-fujita wants to merge 3 commits into
VictoriaMetrics:masterfrom
shunki-fujita:config-reloader-initial-sync-reload

Conversation

@shunki-fujita

@shunki-fujita shunki-fujita commented Aug 21, 2026

Copy link
Copy Markdown

Fixes #2527

Problem

Since v0.72.0, VMAlert reads its rule files from an emptyDir populated by the config-init init container and synced by the config-reloader sidecar. If the rules ConfigMap is updated between the init container's copy and the sidecar starting, the sidecar's initial sync writes the new content to the target dir but never notifies the application. The application keeps serving the snapshot it read at startup, and if the ConfigMap never changes again, no fsnotify event ever fires — the stale state is permanent.

Fix

  • dirWatcher.start() sends one reload signal after the initial directory sync. This also applies to watch-only directories (no --target-dir): the same startup race exists there — a change landing
    between the application's first read and the watch being established produces no event — just with a smaller window.
  • Failed reload calls are retried with exponential backoff (1s, doubling up to 30s) until they succeed. Without this, the startup reload could be lost when the application's reload endpoint is not listening yet, reintroducing the same permanent staleness. This also covers transient failures of event-driven reloads.
  • Directories whose initial sync failed are retried with the same capped backoff until they succeed. The retry timer is multiplexed into the event loop, so a failing directory never blocks event processing, and every retry round that makes progress triggers another reload — content that synced successfully is never held back by a still-failing directory.

Tests

  • reload retry: the reload call fails twice and succeeds on the third attempt
  • initial sync retry: a blocked target dir recovers and triggers another reload
  • partial failure: a permanently failing pair does not hold back the reload for a pair that synced successfully
  • liveness: events for other directories are processed while an initial sync retry is pending
  • no watched dirs: no startup reload
  • existing dir-watcher tests drain the startup signal, pinning the new behavior

@shunki-fujita
shunki-fujita marked this pull request as draft August 21, 2026 06:36

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@shunki-fujita
shunki-fujita force-pushed the config-reloader-initial-sync-reload branch from 3995a5a to 623729e Compare August 21, 2026 06:54
@shunki-fujita
shunki-fujita force-pushed the config-reloader-initial-sync-reload branch from 623729e to e7629bb Compare August 21, 2026 07:02
@shunki-fujita
shunki-fujita marked this pull request as ready for review August 21, 2026 07:06

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread cmd/config-reloader/file_watch.go Outdated
@vrutkovs vrutkovs self-assigned this Aug 21, 2026
Comment thread cmd/config-reloader/file_watch.go Outdated
@shunki-fujita
shunki-fujita requested a review from vrutkovs August 26, 2026 07:55

@vrutkovs vrutkovs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs CHANGELOG entry too

if !synced {
continue
}
select {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that we add the directory to the cache first time anyway, even if the sync succeeds? Perhaps we should change that assumption - and that would help us get rid of "infinite cycle to handle initial sync"?

We can update updateCache function to return newHash and set filesContentHashPath[baseDir] = newHash here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vmalert: rules stay stale forever when the rules ConfigMap is updated during pod startup

2 participants