Better handling for outband power operations on VR

[rajujith]

ISSUE TYPE

• Bug Report
• Improvement Request

COMPONENT NAME

VR,VMware

CLOUDSTACK VERSION

4.17.2

SUMMARY

When a VR is stopped on cloudstack then management/control IP is released.

If the VR is powered on later on the hypervisor the VR will function normally with its previous management IP but CloudStack won’t find this IP information although the power state is synced.

This situation could result in operations issues on the guest works since either :

There is a management IP on the VR but cloudstack doesn’t know about it so most of the operations can fail. 

The previously released IP could be allocated to a new VR resulting in IP conflict and potentially configuring the wrong VR with network configurations as well. 

STEPS TO REPRODUCE

    1.Deploy a VM on a new isolated guest network with a default offering.

    2.Stop the VR in cloudStack

    3. Start the VR on the hypervisor. 

A possible solution is to restart the VR once it detects an out-of-band power on.

[weizhouapache]

@rajujith
I think the solution is, reserving the management IP of stopped VRs (only on vmware)
However, it might lead to waste of management IPs. we can reserve the management IP so for few hours or 1 day.

[rajujith]

@weizhouapache there could be use cases where the VR would be started outside the proposed reservation time, in such cases it is still good to restart the VR to avoid those corner cases too.

[DaanHoogland]

I do not think starting a VR out-of bounds is a valid way of operating a cloud/handling a VR. That said I think on power on detection of a VM of type VR we should restart the network it belongs to with cleanup. A slight impediment: what if the network is a redundant one?
/me will do some experimenting.

[weizhouapache]

@weizhouapache there could be use cases where the VR would be started outside the proposed reservation time, in such cases it is still good to restart the VR to avoid those corner cases too.

@rajujith
the management IP of VR will be re-allocated, which might lead to inconsistent Ips in cloudstack database and inside the VR.

stop/start VR might be an option, but might lead some unexpected downtime of the network. (I remember there was a PR for it)

[DaanHoogland]

@rajujith , I created a possible fix. Your descrition is about how to reproduce the root cause, but this does not mean the error condition right? Can you describe how to force the error condition?

[weizhouapache]

@rajujith @DaanHoogland
I searched my mailbox, and found this PR #3915

[DaanHoogland]

Good find @weizhouapache however, I think as the router is started out of bounds, and hance was down, we shouldn't worry to much about downtime. The solution looks fine but over complicated to me. I think we can in genera just reboot (if that adequately takes care of all possible IP conflicts.

[weizhouapache]

Good find @weizhouapache however, I think as the router is started out of bounds, and hance was down, we shouldn't worry to much about downtime. The solution looks fine but over complicated to me. I think we can in genera just reboot (if that adequately takes care of all possible IP conflicts.

this might lead to downtime if VR is migrated out-of-band.
need some testing

[andrijapanicsb]

Guys, we here have a DESIGN issue - and (IMO) nothing else. Due to the design issue - we are seeing a bug in certain situations.

CloudStack should NOT be marking the management IP of the VR as free in the DB when a VR is stopped (which is what happens when you get PowerOff report from VMware during out-of-band VR operations - stopping a VR in vSphere, DRS moving VR, or even manual live migrations in vSphere).

If we don't release the IP in the DB (nor i.e. remove nics from the nics table) - there will be no bug/issue present..

It makes zero sense to remove IPs in DB when you stop the VM - the VM is still there, why remove it's IP/nic in DB?
Unless there is a very specific reason that we are unaware of.

[rajujith]

@rajujith , I created a possible fix. Your descrition is about how to reproduce the root cause, but this does not mean the error condition right? Can you describe how to force the error condition?

@DaanHoogland my concern is about Cloudstack and the infrastructure going out of sync w.r.t to the management IP in the VR. The possible scenarios are power on at vCenter, incorrect out-of-band power on/power off events detected due to DRS etc..

[alexandremattioli]

Suggestion.
New global setting "release management IP" true/false.

If true then behaviour is as is.
If false then ACS doesn't release the IP.

@DaanHoogland @weizhouapache @NuxRo @rajujith @andrijapanicsb