Autoscale: infinite scale-up when VMs fail to START (Stopped) — #11244 guard only counts Error state

[VishnuVuggepalli]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VM Autoscale Feature

CLOUDSTACK VERSION

4.22.0.0 (observed)
Also present in 4.22.1.0, 4.23.0.0 and main (verified by source inspection)

CONFIGURATION

Advanced zone, KVM, Ceph/RBD-only primary storage. One AutoScale VM group
(min_members=1, max_members=2, interval=30) on an isolated network with a
/24 guest CIDR.

OS / ENVIRONMENT

Linux (Debian 12), KVM hosts.

SUMMARY

The infinite-autoscaling guard added in #11244 (fixing #9318) only counts
instances in State.Error. When scale-up VMs fail to start — as opposed to
failing to be created — they land in State.Stopped, not State.Error. The
guard therefore never trips, and the group scales up on every interval
indefinitely.

In our incident this produced 2,296 VMs from a group whose max_members is 2,
over roughly 21 hours, until the guest subnet was exhausted.

Two independent counters are involved and both exclude Stopped:

1. AutoScaleVmGroupVmMapDaoImpl.getErroredInstanceCount() — the Prevent infinite autoscaling #11244 guard —
   counts State.Error only:

public int getErroredInstanceCount(long vmGroupId) {
    SearchCriteria<Integer> sc = CountBy.create();
    sc.setParameters("vmGroupId", vmGroupId);
    sc.setJoinParameters("vmSearch", "states", State.Error);   // Stopped not counted
    ...
}

2. AutoScaleVmGroupVmMapDaoImpl.countAvailableVmsByGroup() — used by every
   scaling decision in AutoScaleManagerImpl (checkConditionUp,
   checkConditionDown, checkAutoScaleVmGroup, and the group-state handlers) —
   counts only Starting, Running, Stopping, Migrating:

sc.setJoinParameters("vmSearch", "states",
        State.Starting, State.Running, State.Stopping, State.Migrating);  // Stopped not counted

So with N leaked Stopped members, currentVM == 0:

• checkAutoScaleVmGroup: if (currentVM < minMembers) -> 0 < 1 -> scale up, every interval
• checkAutoScaleVmGroup: if (currentVM > maxMembers) -> 0 > 2 -> scale-down never fires
• checkConditionDown: if (currentVM - 1 < minVm) -> -1 < 1 -> scale-down additionally blocked
• checkConditionUp: errored-instance guard -> 0 > 10 false -> guard never trips

A third defect prevents the failed VM from being cleaned up, which is what allows
the leak to accumulate in the first place. doScaleUp persists the group map row
before attempting the start, and its cleanup is guarded on ServerApiException:

// AutoScaleManagerImpl.doScaleUp
autoScaleVmGroupVmMapDao.persist(groupVmMapVO);   // persisted BEFORE the start attempt
try {
    startNewVM(vm.getId());
    ...
} catch (ServerApiException e) {
    ...
    destroyVm(vm.getId());      // never reached, see below
    break;
}

startNewVM does convert InsufficientCapacityException into ServerApiException,
but it never sees that exception, because VirtualMachineManagerImpl.start()
has already wrapped it into an unchecked CloudRuntimeException:

try {
    advanceStart(vmUuid, params, planToDeploy, planner);
} catch (ConcurrentOperationException | InsufficientCapacityException e) {
    throw new CloudRuntimeException(String.format("Unable to start a VM [%s] due to [%s].", vmUuid, e.getMessage()), e);
}

CloudRuntimeException matches none of startNewVM's typed catches and is not a
ServerApiException, so it propagates past doScaleUp's handler to
AutoScaleManagerImpl$MonitorTask, and destroyVm() is never called. The
observed log line is exactly this:

WARN  [c.c.n.a.A.MonitorTask] Caught the following exception on monitoring AutoScale Vm Group
      com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM [...]

Note that PR #9574 ("Prevent infinite retries of autoscaling"), which proposed a
one-line change to AutoScaleVmGroupVmMapDaoImpl, was closed unmerged; the merged
#11244 took the threshold approach instead, which is what leaves this variant
uncovered.

STEPS TO REPRODUCE

1. Create an AutoScale VM group (min_members=1, max_members=2, short interval).

2. Let it stabilise at 1 running VM.

3. Break VM start (not creation) in a way that returns an
   InsufficientCapacityException or ResourceUnavailableException from
   advanceStart.

   The trigger we actually hit was the group network's Virtual Router becoming
   unreachable, so VirtualRouterElement.applyDhcpEntries failed with:
   ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply dhcp entry on router.
   That was an observed failure rather than a deliberate test, so I have not
   confirmed that stopping the VR is a minimal reproducer — any start-path failure
   that surfaces as CloudRuntimeException out of VirtualMachineManagerImpl.start()
   should exhibit the same leak.

4. Observe: one new VM per interval, each landing in Stopped, each retaining its
   autoscale_vmgroup_vm_map row, indefinitely.

EXPECTED RESULTS

Scale-up stops after a bounded number of consecutive failed starts, and/or failed
instances are cleaned up, and/or Stopped members count toward max_members.

ACTUAL RESULTS

Unbounded VM creation. In our case, one VM per 30s for ~21 hours:

• 2,043 VMs in Error (created after the guest subnet was exhausted — these fail
  at IP allocation, before a NIC is assigned)
• 253 VMs in Stopped, each holding a NIC and therefore a guest IP
• The /24 guest network reached 254/254 NICs and all subsequent VM deployments —
  including unrelated, non-autoscale ones — failed with
  InsufficientVirtualNetworkCapacityException: Unable to acquire Guest IP address

autoscale.errored.instance.threshold was at its default of 10 throughout, and
getErroredInstanceCount() returned 0 the entire time, because none of the leaked
instances were in Error — they were in Stopped.

SUGGESTED FIX

Any one of these would break the loop; the first two seem most direct:

1. Include State.Stopped in getErroredInstanceCount() (or add a separate
   "failed instance" counter covering both Error and Stopped).
2. Broaden doScaleUp's catch from ServerApiException to also handle
   CloudRuntimeException, so destroyVm() runs and the member is not leaked.
3. Count Stopped members in countAvailableVmsByGroup() so that leaked members
   contribute to max_members and become eligible for scale-down.

[github-actions]

🎯 Triage report

Well-documented report of an infinite AutoScale scale-up loop: when a scale-up VM fails to start (landing in State.Stopped rather than State.Error), neither the errored-instance guard from #11244 nor the active-member counter account for Stopped VMs, so the group scales up forever. The reporter also identifies a related defect where doScaleUp's cleanup only catches ServerApiException, missing the CloudRuntimeException actually thrown by VirtualMachineManagerImpl.start(), so failed VMs are never destroyed. Real-world impact was VM/IP exhaustion (2,296 VMs from a max_members=2 group).

📊 Assessment

Dimension	Value	Reasoning
Type	type:bug	Clear functional defect with reproducible logic flaw and code-level evidence
Component	component:management-server	AutoScale scaling logic lives in AutoScaleManagerImpl/AutoScaleVmGroupVmMapDaoImpl (management server core); no dedicated component:autoscale label exists
Severity	Severity:Major	Caused resource exhaustion (guest IP/subnet, thousands of leaked VMs) impacting unrelated deployments in the reporter's environment, though it requires a specific start-failure trigger to manifest
Labels	type:bug, Severity:Major, component:management-server	See above
Coding agent	Suitable	Root cause is precisely identified with exact file/method names and code snippets, and three concrete, scoped fix options are proposed (broaden guard to include Stopped, catch CloudRuntimeException in doScaleUp, or count Stopped in countAvailableVmsByGroup)

🔗 Similar issues

• PR Prevent infinite autoscaling #11244 (referenced in issue) — merged fix for the original infinite-scale-up bug (Autoscale Group creates Infinite Number Of VMs when unable to startup #9318), but only guards against State.Error, not State.Stopped; this issue identifies the gap left by that fix.
• PR Prevent infinite retries of autoscaling #9574 (referenced in issue) — an earlier, unmerged one-line proposal that reportedly would have addressed a related variant.

No separate open GitHub issue duplicating this exact report was found via search.

💡 Notes and suggestions

• Recommend maintainers/assignee evaluate the reporter's three suggested fixes together rather than in isolation: option 1 (include Stopped in getErroredInstanceCount()) stops the infinite loop quickly, but combining it with option 2 (broadening the doScaleUp catch to CloudRuntimeException) prevents the VM leak from accumulating in the first place, which is the root cause of the exhaustion.
• Worth double-checking whether other terminal/failure VM states (e.g. Destroyed, Expunging) should also be considered in countAvailableVmsByGroup() for consistency.
• A regression/unit test simulating a start failure that raises CloudRuntimeException (not ServerApiException) from advanceStart would help confirm the fix and guard against future regressions.

Generated by Daily Issue Triage · sonnet50 102.2K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9