Fix concurrent publish bottleneck and agent ping task leaks - #14171
Draft
sb-abhish3k wants to merge 1 commit into
Draft
sb-abhish3k wants to merge 1 commit into
sb-abhish3k wants to merge 1 commit into
Conversation
MessageBusBase: replaced exclusive Gate with ReadWriteLock so multiple publishers can run in parallel. Subscriber callbacks are now invoked outside the lock to prevent slow hypervisor callbacks from starving write lock holders (subscribe/unsubscribe). Under heavy reconnect load (pod restart with many hosts) the old design serialized all publish() callers through a single gate, blocking API threads and causing full management server unresponsiveness. DirectAgentAttache: fixed a race where PingTask could be scheduled after disconnect() had already cleared _futures, causing it to never be cancelled. Switched PingTask scheduling from scheduleAtFixedRate to scheduleWithFixedDelay so slow hypervisor responses do not pile up concurrent ping executions and exhaust the cron thread pool.
Author
|
@blueorangutan package |
|
@sb-abhish3k a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 4.20 #14171 +/- ##
============================================
+ Coverage 16.38% 16.39% +0.01%
- Complexity 13615 13622 +7
============================================
Files 5669 5669
Lines 501561 501471 -90
Branches 60922 60906 -16
============================================
+ Hits 82168 82209 +41
+ Misses 410185 410050 -135
- Partials 9208 9212 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19250 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
MessageBusBase: replaced exclusive Gate with ReadWriteLock so multiple publishers can run in parallel. Subscriber callbacks are now invoked outside the lock to prevent slow hypervisor callbacks from starving write lock holders (subscribe/unsubscribe). Under heavy reconnect load (pod restart with many hosts) the old design serialized all publish() callers through a single gate, blocking API threads and causing full management server unresponsiveness.
DirectAgentAttache: fixed a race where PingTask could be scheduled after disconnect() had already cleared _futures, causing it to never be cancelled. Switched PingTask scheduling from scheduleAtFixedRate to scheduleWithFixedDelay so slow hypervisor responses do not pile up concurrent ping executions and exhaust the cron thread pool.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?