KVM: instance left running on the host with no record in CloudStack after an out-of-band power report during start

[bhouse-nexthop]

ISSUE TYPE

• Bug Report

COMPONENT NAME

engine-orchestration, VM power state sync, KVM

CLOUDSTACK VERSION

4.22

CONFIGURATION

KVM hosts. Any host where instances stop or crash while another instance is being deployed.

OS / ENVIRONMENT

KVM / libvirt.

SUMMARY

An instance can end up running on a KVM host with no record in CloudStack. Its IP address is released and later handed to another instance, so two instances answer for the same address. The root volume is marked for deletion but cannot be deleted while the domain holds it, so the storage is stranded too.

The trigger is the out-of-band ping the KVM agent sends when another instance on the same host shuts down or crashes.

STEPS TO REPRODUCE

Needs a host with instance churn during a deploy.

1. Deploy an instance on a KVM host.
2. While the StartCommand is still in flight, have another instance on the same host shut itself down or crash.
3. That fires LibvirtDomainListener.onLifecycleChange, which calls Agent.triggerUpdate() and sends a PingCommand with outOfBand=true.
4. Repeat as needed. On a busy host these pings can arrive every few seconds.

The window that does the damage is small: the report has to be collected before the new domain exists but processed after the start job finishes. It does not fire on every deploy, but it recurs.

EXPECTED RESULTS

The deploy succeeds, or it fails and the host is told to stop the instance.

An instance absent from a report collected before it started is not treated as missing.

ACTUAL RESULTS

Sequence, from a real occurrence (times shortened):

time	what happens
20:23:5x	DeployVM starts, instance -> Starting, StartCommand sent
20:25:14	another instance crashes; agent collects a report and sends it with force
20:25:16	the new domain is created on the host
20:25:17	StartAnswer arrives, instance -> Running
20:25:18	the 20:25:14 report is processed; instance is not in it -> PowerReportMissing
	NICs and IP released, instance -> Stopped. No StopCommand is sent
20:25:19	the deploy job finds it Stopped, logs an error, instance -> Error, job fails
20:25:28	user expunges it; advanceStop() returns at once because state is Error
20:25:39	expunged and removed. No StopCommand was ever sent
20:25:40	every later report: Unable to find matched VM in CloudStack DB. name: i-2-3-VM

The domain is still running the whole time.

CAUSE

Five separate things line up.

1. force skips the graceful period for a missing instance.

VirtualMachinePowerStateSyncImpl.processMissingVmReport():

if (!force) {
    vmsThatAreMissingReport = filterOutdatedFromMissingVmReport(vmsThatAreMissingReport);
}
...
if (force || (milliSecondsSinceLastStateUpdate > milliSecondsGracefulPeriod)) {

force comes from PingCommand.outOfBand, set only by Agent.triggerUpdate(), called only from LibvirtDomainListener on a self-shutdown or crash. It was added so a reported stop takes effect immediately. Applying it to the absence of a report is the defect: a report says nothing about an instance that did not exist yet when it was collected.

The KVM report lists only powered-on domains, so an instance that is still starting is simply absent.

The log line makes this hard to see. It prints "has passed graceful period" even when force short-circuited the check, so it reports an elapsed time far below the graceful period as having passed it.

2. The missing-report branch releases resources without stopping anything.

VirtualMachineManagerImpl.handlePowerOffReportWithNoPendingJobsOnVM():

if (PowerState.PowerOff.equals(vm.getPowerState())) {
    ...
    Pair<Boolean, String> result = sendStop(vmGuru, profile, true, true);
    if (!result.first()) { return; } else { releaseVmResources(profile, true); }
} else if (PowerState.PowerReportMissing.equals(vm.getPowerState())) {
    releaseVmResources(profile, true);   // no StopCommand
}

The IP is freed while the instance is still using it.

3. Expunge never contacts the host for an instance in Error or Stopped.

advanceStop() returns before it looks at the host id:

if (state == State.Stopped) return;
if (state == State.Destroyed || state == State.Expunging || state == State.Error) return;

vm.destroy.forcestop does not help, the early return happens first.

4. A power-on report for a destroyed instance is only logged.

handlePowerOnReportWithNoPendingJobsOnVM() has case Destroyed: case Expunging: log and break. CloudStack knows the host and the instance name at that moment and does nothing with them.

5. Unknown instances are only logged at debug.

convertVmStateReport() writes one debug line per unknown instance per report and nothing acts on it, so the condition can persist unnoticed indefinitely.

IMPACT

• an instance runs unmanaged and invisible
• its IP is reassigned, giving an address conflict between two live instances
• its root volume stays in Destroy and cannot be deleted while the domain holds it, so primary storage is stranded
• for RBD the delete fails on the image watcher, which is the only reason the disk is not deleted underneath a running guest