Skip to content

feat(vmop): explain why a migration is queued behind concurrency limits#2656

Open
danilrwx wants to merge 1 commit into
mainfrom
feat/vmop/surface-queued-migration-count
Open

feat(vmop): explain why a migration is queued behind concurrency limits#2656
danilrwx wants to merge 1 commit into
mainfrom
feat/vmop/surface-queued-migration-count

Conversation

@danilrwx

@danilrwx danilrwx commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Description

When a migration (a firmware update, a node drain, or a manual migrate) cannot start immediately because a concurrency limit is already taken, the operation used to sit in Pending for a long time with only a generic "queued, waiting for the queue to be processed" message — indistinguishable from a stuck one.

This surfaces the concrete reason on the pending operation for all three limits, naming the node and current counts:

  • inbound (target node): node "worker-0" has no free inbound migration slot (1 in progress) — from the controller's own inbound limiter.
  • cluster / per-node outbound: taken from a MigrationConcurrencyLimitReached condition that KubeVirt now sets on the migration, e.g. the cluster live migration limit is reached (3/3 running) or node "worker-0" has no free outbound migration slot (1/1 running).

The cluster/outbound limits are enforced by KubeVirt, so the reason is reported at the source (see the paired 3p-kubevirt change) and this controller only reads the resulting condition — no duplication of KubeVirt's limit algorithm. The inbound limit is enforced by this controller's own limiter and is reported here.

Why do we need it, and what problem does it solve?

Migrations queued behind a concurrency limit looked identical to stuck ones, so users waited and escalated with nothing actionable to look at. This makes the limit, the node, and the numbers visible directly on the operation, so a queued migration is recognizable at a glance without digging into controller logs.

What is the expected result?

Start a migration while a migration limit (inbound, cluster, or per-node outbound) is already taken:

  1. The operation stays Pending (unchanged behavior).
  2. Its message names the limit that is reached, the node, and the current counts, instead of the generic queued message.
  3. Once a slot frees up, the migration proceeds as before.

Note: the cluster/outbound reason relies on the paired 3p-kubevirt change that sets the MigrationConcurrencyLimitReached condition; without it, those cases fall back to the generic queued message. The inbound reason works independently.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: vmop
type: feature
summary: "A migration waiting for a free concurrency slot now names the limit that is reached (inbound, cluster, or node outbound) with the node and current counts, instead of a generic pending message."

@danilrwx danilrwx force-pushed the feat/vmop/surface-queued-migration-count branch 3 times, most recently from 0522faf to e288387 Compare July 13, 2026 21:22
A migration that cannot start yet — because an inbound, cluster, or per-node
outbound migration limit is taken — used to sit in Pending with only a generic
"queued" message, indistinguishable from a stuck one.

Surface the concrete reason on the pending operation:
- inbound:  node "X" has no free inbound migration slot (N in progress),
  the occupancy count now lives in the inbound limiter's own package;
- cluster/outbound: taken from the MigrationConcurrencyLimitReached condition
  KubeVirt sets on the migration, which already carries the node and counts.

No duplication of KubeVirt's limit algorithm: DVP only reads the condition it
sets and reports its own inbound limiter.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx danilrwx force-pushed the feat/vmop/surface-queued-migration-count branch from e288387 to ac6774c Compare July 13, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant