Skip to content

Stop valkey only after cloud_controller_ng has drained - #717

Closed
kathap wants to merge 2 commits into
developfrom
valkey-drain-ordering
Closed

kathap wants to merge 2 commits into
developfrom
valkey-drain-ordering

Conversation

@kathap

@kathap kathap commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Valkey is co-located with cloud_controller_ng on api VMs and is the metrics store CC reaches over a unix socket on every request. During a rolling update BOSH could stop valkey while CC was still serving, so in-flight requests failed on the removed socket and returned 500s.

Add a drain script to the valkey job that blocks until the CC process has exited, bounded by the new valkey.drain_timeout property, so BOSH stops valkey only after CC has finished draining.

  • A short explanation of the proposed change:
    Add a drain script to the valkey job. When BOSH stops an api VM, the valkey drain now blocks until the co-located cloud_controller_ng process has exited, so valkey is stopped only after CC has finished draining. A new valkey.drain_timeout property (default 60s) bounds the wait so a stuck CC can't hang the deploy.
  • An explanation of the use cases your change solves
    Valkey runs on each api VM as CC's metrics store, reached over a unix socket that CC hits on every request. During a rolling update, BOSH could stop valkey (removing the socket) while CC was still serving, so in-flight requests failed with Redis::CannotConnectError and returned 500s to callers, on every api VM, every deploy. Draining valkey after CC closes that window at the source.

This change was developed with AI assistance; all code was reviewed and tested by me.

Valkey is co-located with cloud_controller_ng on api VMs and is the
metrics store CC reaches over a unix socket on every request. During a
rolling update BOSH could stop valkey while CC was still serving, so
in-flight requests failed on the removed socket and returned 500s.

Add a drain script to the valkey job that blocks until the CC process
has exited, bounded by the new valkey.drain_timeout property, so BOSH
stops valkey only after CC has finished draining.

@philippthun philippthun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest not to merge this.

The "valkey stops before CC drains" window does not exist. BOSH waits for all drains before any monit stop (lifecycle). The real root cause was the CC drain not running at all (Ruby 4.0 + --disable-all = LoadError), fixed by #698.

@kathap

kathap commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Closing as not needed any more, see Philipps comment

@kathap kathap closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants