KVM Host HA activity check incorrectly detects activity on CLVM_NG storage after host failure

[rchicote72]

problem

During Host HA testing on a KVM cluster using CLVM_NG Primary Storage, we observed that the VM activity check performed against CLVM_NG may incorrectly report activity for a host that has been completely powered off.

When the failed host only runs HA-enabled VMs whose volumes reside on CLVM_NG, CloudStack continues reporting that the host has VM activity. As a result, Host HA may enter Degraded and repeat the activity-check cycle instead of progressing to fencing.

The other KVM hosts correctly detect that the failed host is unreachable.

As a comparison, when the same failed host also runs an HA-enabled VM whose volume resides on NFS Primary Storage, CloudStack checks both storage types and we observe:

CLVM_NG activity check -> host has activity
NFS activity check -> host does not have activity

In this case, Host HA is able to progress to fencing.

Once the host reaches Fenced, the HA-enabled VMs stored on CLVM_NG are successfully restarted on other KVM hosts.

This indicates that CLVM_NG itself works correctly as shared storage for VM HA once fencing has occurred. The problem appears to be specifically related to the Host HA VM activity detection performed against CLVM_NG storage.

As a workaround, we currently use a small shared NFS Primary Storage with one small HA-enabled NFS-backed VM per KVM host. These VMs belong to a host anti-affinity group so that one VM is normally present on each host.

With this workaround, Host HA correctly detects the absence of activity through NFS, reaches fencing, and the CLVM_NG-backed HA VMs are successfully restarted on the remaining hosts.

versions

Apache CloudStack: 4.23.0.0

Hypervisor: KVM

Operating System: Ubuntu 24.04

Cluster: 4 KVM hosts

Primary Storage:

• CLVM_NG backed by shared Fibre Channel SAN
• sanlock + lvmlockd
• Additional shared NFS Primary Storage

Host HA provider: KVMHAProvider

OOBM: IPMI

VM HA: enabled on the test VMs

Host HA activity-check configuration:

kvm.ha.activity.check.failure.ratio = 0.7
kvm.ha.activity.check.interval = 60
kvm.ha.activity.check.max.attempts = 10
kvm.ha.activity.check.timeout = 60
kvm.ha.degraded.max.period = 300

The activity-check parameters above use their default values.

The following parameter is changed from its default value:

kvm.ha.recover.failure.threshold = 0

The default value is 1.

We use 0 to skip host recovery attempts and proceed to fencing once Host HA determines that the host must be recovered.

The steps to reproduce the bug

1. Create a KVM cluster with multiple hosts.

2. Configure Host HA using KVMHAProvider and configure OOBM/IPMI for the KVM hosts.

3. Configure shared CLVM_NG Primary Storage backed by Fibre Channel SAN using sanlock and lvmlockd.

4. Deploy one or more HA-enabled VMs with their volumes on CLVM_NG.

5. Ensure that the test host only runs HA-enabled VMs whose volumes reside on CLVM_NG.

6. Completely power off the KVM host to simulate an unexpected host failure.

7. Monitor the Host HA activity checks in the CloudStack Management Server log.

8. CloudStack detects that the host is unreachable, but the CLVM_NG VM activity check reports that the host still has activity.

9. Host HA may enter Degraded and repeat the activity-check process instead of progressing to fencing.

10. Power the host back on and deploy an additional HA-enabled VM whose volume resides on shared NFS Primary Storage.

11. Repeat the hard power-off test.

12. CloudStack now performs activity checks against both CLVM_NG and NFS. The observed result is:

CLVM_NG -> host has activity
NFS -> host does not have activity

13. Host HA is now able to progress from Checking to Recovering, Fencing and finally Fenced.

14. After fencing, the HA-enabled VMs stored on CLVM_NG are successfully restarted on the remaining KVM hosts.

15. When the failed host becomes available again, CloudStack also automatically starts the pending NFS-backed auxiliary VM on that host because the host anti-affinity constraint can be satisfied again.

CLVM_NG-backed VMs can also be live migrated normally between healthy hosts. Therefore, the problem does not appear to be related to CLVM_NG volume activation, locking, or general access to the shared storage.

What to do about it?

The Host HA VM activity-check mechanism should correctly determine that a completely powered-off KVM host no longer has VM activity when the affected VM volumes reside on CLVM_NG.

Expected behavior:

Host powered off
|
v
CLVM_NG activity check
|
v
No VM activity
|
v
Fencing
|
v
Fenced
|
v
VM HA

The CLVM_NG activity-check implementation should be reviewed to determine why it reports activity for a host that has been completely powered off.

The issue appears to be limited to activity detection. Once fencing is reached, CloudStack successfully restarts the CLVM_NG-backed HA VMs on other hosts.

For comparison, an NFS-backed VM on the same failed host correctly reports no activity and allows Host HA to proceed to fencing.

It may therefore be useful to compare how VM activity is determined for CLVM_NG storage with the NFS activity-check path.

Management Server and KVM agent logs from both scenarios can be provided if required.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

cc @sureshanaparti @Pearl1594

[Pearl1594]

Host HA is not supported for CLVM pool types in the 1st release of its support. It is in pipeline and will be supported in future. Thanks for raising the issue.