Skip to content

[PR #7939/d212f0b5 backport][3.115] fix: make redis workers to know the current num of workers at startup - #7941

Merged
gerrod3 merged 1 commit into
3.115from
patchback/backports/3.115/d212f0b5737f578fef0aaedb5e799872bca7202e/pr-7939
Aug 3, 2026
Merged

[PR #7939/d212f0b5 backport][3.115] fix: make redis workers to know the current num of workers at startup#7941
gerrod3 merged 1 commit into
3.115from
patchback/backports/3.115/d212f0b5737f578fef0aaedb5e799872bca7202e/pr-7939

Conversation

@patchback

@patchback patchback Bot commented Aug 3, 2026

Copy link
Copy Markdown

This is a backport of PR #7939 as merged into main (d212f0b).

Fixes: #7912
Addresses: https://redhat.atlassian.net/browse/PULP-2149

Problem

Redis workers find work by repeatedly asking the database for waiting tasks, then sleeping briefly. Sleep is supposed to get longer as the fleet grows, so many workers don’t all pound the DB at once.

But each new worker starts life assuming the fleet has one worker. It only learns the real count on its first heartbeat (~10s later). Until then it polls about every 10 ms.

So a scale-up (e.g. 25 → 150) creates a short window of thousands of SELECTs per second against core_task. The DB saturates, heartbeat writes fail, workers exit, replacements start, and the same aggressive startup polling happens again.

Solution

workers should learn fleet size at startup before first heartbeat

📜 Checklist

  • Commits are cleanly separated with meaningful messages (simple features and bug fixes should be squashed to one commit)
  • A changelog entry or entries has been added for any significant changes
  • Follows the Pulp policy on AI Usage
  • (For new features) - User documentation and test coverage has been added

See: Pull Request Walkthrough

Signed-off-by: Carlos Feria <2582866+carlosthe19916@users.noreply.github.com>
(cherry picked from commit d212f0b)
@gerrod3
gerrod3 merged commit 30b605f into 3.115 Aug 3, 2026
14 checks passed
@gerrod3
gerrod3 deleted the patchback/backports/3.115/d212f0b5737f578fef0aaedb5e799872bca7202e/pr-7939 branch August 3, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants