feat(vmop): explain why a migration is queued behind concurrency limits#2656
Open
danilrwx wants to merge 1 commit into
Open
feat(vmop): explain why a migration is queued behind concurrency limits#2656danilrwx wants to merge 1 commit into
danilrwx wants to merge 1 commit into
Conversation
0522faf to
e288387
Compare
A migration that cannot start yet — because an inbound, cluster, or per-node outbound migration limit is taken — used to sit in Pending with only a generic "queued" message, indistinguishable from a stuck one. Surface the concrete reason on the pending operation: - inbound: node "X" has no free inbound migration slot (N in progress), the occupancy count now lives in the inbound limiter's own package; - cluster/outbound: taken from the MigrationConcurrencyLimitReached condition KubeVirt sets on the migration, which already carries the node and counts. No duplication of KubeVirt's limit algorithm: DVP only reads the condition it sets and reports its own inbound limiter. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
e288387 to
ac6774c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When a migration (a firmware update, a node drain, or a manual migrate) cannot start immediately because a concurrency limit is already taken, the operation used to sit in
Pendingfor a long time with only a generic "queued, waiting for the queue to be processed" message — indistinguishable from a stuck one.This surfaces the concrete reason on the pending operation for all three limits, naming the node and current counts:
node "worker-0" has no free inbound migration slot (1 in progress)— from the controller's own inbound limiter.MigrationConcurrencyLimitReachedcondition that KubeVirt now sets on the migration, e.g.the cluster live migration limit is reached (3/3 running)ornode "worker-0" has no free outbound migration slot (1/1 running).The cluster/outbound limits are enforced by KubeVirt, so the reason is reported at the source (see the paired 3p-kubevirt change) and this controller only reads the resulting condition — no duplication of KubeVirt's limit algorithm. The inbound limit is enforced by this controller's own limiter and is reported here.
Why do we need it, and what problem does it solve?
Migrations queued behind a concurrency limit looked identical to stuck ones, so users waited and escalated with nothing actionable to look at. This makes the limit, the node, and the numbers visible directly on the operation, so a queued migration is recognizable at a glance without digging into controller logs.
What is the expected result?
Start a migration while a migration limit (inbound, cluster, or per-node outbound) is already taken:
Pending(unchanged behavior).Checklist
Changelog entries