VPC VR can't be started on VMware due to missing routes

[andrijapanicsb]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR agent ?

CLOUDSTACK VERSION

4.17.0.0+, VMware 

CONFIGURATION

mgmt server is in Pod1/subnet1,
hypervisor (and thus VR's control NIC) is in Pod2/subnet2 - thus there is a routing process between mgmt and VR's NIC IP

OS / ENVIRONMENT

Probably affects only VMware, but has not been tested with XS/KVM (should not be a problem, but needs checking)

SUMMARY

Only VPC VR affected (VR for shared or simpler Isolated network is NOT affected).
There is a patching error of the VPC VR (this might be a problem or a consequence of missing routes) - either way, route from VR to the subnet of the mgmt server (subnet1) is missing/not configured, and thus VR can't talk to the mgmt server - VR configuration never completes. CMDLINE looks fine (all parameters look fine) but things are not applied correctly on the OS level.

STEPS TO REPRODUCE

For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.

For new features, show how the feature would be used.

mgmt server in subnet1, hypervisor (and thus VR's control NIC) in subnet2 - so that VR needs to send packet to the Pod2's gateway in order to reach Pod1/mgmt server.

EXPECTED RESULTS

VR patched up and operational

ACTUAL RESULTS

VR not patched, routes not present as needed.

CMD line of the VPC VR:

cloud.log

Routing table empty (there should be multiple routes (route to the mgmt cidr should be via 10.1.32.1 for 10.1.32.0 network - see CMDLINE output)

[weizhouapache]

it looks the route for mgmt cidr (10.0.32.0/20) is missing.
the problem is, where can we find the gateway of mgmt cidr (e.g. 10.0.32.1) ? cannot find it in cmdline.

xenserver/kvm should not be impacted, as they use link local ip.

[H-LK]

A critical export was moved from bootstrap.sh to init.sh

When the VPC is setup it asked in Line 64 for the hypvervisor

cloudstack/systemvm/debian/opt/cloud/bin/setup/vpcrouter.sh

Line 64 in 296035d

if [ "$HYPERVISOR" == "vmware" ] || [ "$HYPERVISOR" == "hyperv" ];

The vpcrouter.sh gets called by the bootstrap.sh. In Version <= 4.16 it gets exported there and is available for vpcrouter.sh

cloudstack/systemvm/debian/opt/cloud/bin/setup/bootstrap.sh

Line 215 in 77df050

export HYPERVISOR=$(hypervisor)

The comit 830f306 removes the export in the bootstrap.sh and adds it to init.sh

[weizhouapache]

A critical export was moved from bootstrap.sh to init.sh

When the VPC is setup it asked in Line 64 for the hypvervisor

cloudstack/systemvm/debian/opt/cloud/bin/setup/vpcrouter.sh

Line 64 in 296035d

if [ "$HYPERVISOR" == "vmware" ] || [ "$HYPERVISOR" == "hyperv" ];

The vpcrouter.sh gets called by the bootstrap.sh. In Version <= 4.16 it gets exported there and is available for vpcrouter.sh

cloudstack/systemvm/debian/opt/cloud/bin/setup/bootstrap.sh

Line 215 in 77df050

export HYPERVISOR=$(hypervisor)

The comit 830f306 removes the export in the bootstrap.sh and adds it to init.sh

@H-LK
thanks for the findings.

prior to 4.17, the route is added in vpcrouter.sh which is called by cloud-early-config service.
since 4.17, two steps are added before cloud-early-config

• init.sh (called by cloud-preinit service)
• patch (transfer new files/scripts from host or management server to VRs via scp command)

The scp in patch step does not work, because the route is not added (cloud-early-config is not executed yet).
The route should be added in init.sh, instead of vpcrouter.sh

[hooklee2000]

We tested current 4.18-vm-autoscaling branch 4.18.0.0-SNAPSHOT yesteday. Using kvm systemtemplate4.17.0 which modified cloudstack-release to 4.18.0，we found VR vm disappeared in ui.
Using ssh connected the VR vm under agent host, it had console proxy java process, so acted as CPVM wrongly.

[weizhouapache]

That is strange. Can you test it again ?

[hooklee2000]

We pulled 4.18-vm-autoscaling branch on 2022-09-26, it worked well. Then we pulled on 2022-11-17, it failed cause VR cannot start.
system vms

virtual route vm

view console, url is http://10.2.40.59/resource/noVNC/vnc.html?autoconnect=true&port=8080&token=F6IDkA3EnAfclsOgXqbytvUsEbKgC2jAWuP6ybG5HyqNgyR2fAwSFn9BbsCjgC28jQTR5O5v84f-T18N5Xmj-_otcsNv3I7sYLl9dN7HbM5LeVbxzaoG19e4DjEcVklBnN7_QclFRuV7NE-X9cp0YYLd9w6eWdQEjGKPvc4CxJaPqcZDn53gXVIUuj2-l7aTK7Wf4bnMnJ41oLNP39kqCaHiwJXskke7ciEcpIWzIJr3GXSGcfD67N5NOAXto0efGs2ZOKcpjkcw2B6cLj0_tepUNDdjehBq7TSHIjgWFu788xVA0Km2jY7HZbChqZ39icAfeNhL2SfVglKWg9BORTidPb51sYufJrbsv8V_MhfcMICUFZq-q5yPCtiLZTQpox_o65J5v-ed7r6Ft_rJbC3XbTwOv4P8nfNI_UZkCMfM41rLOyZRNBjhkFijXCHi7H9TX7v-93vsQq0cbK6Quw.
Correctly it may be http://10.2.8.13:8080/client/console?cmd=access&vm=7545d234-0464-401b-a283-09c8d8b3f725
register iso timeout

[hooklee2000]

we also tested main branch pulled on 2022-11-17, it had the same problem. How ever, Using virsh list it had not vrvm, it only had cpvm and ssvm.
4.18-vm-autoscaling had vrvm,but vrvm had cpvm java process.

[weizhouapache]

we also tested main branch pulled on 2022-11-17, it had the same problem. How ever, Using virsh list it had not vrvm, it only had cpvm and ssvm. 4.18-vm-autoscaling had vrvm,but vrvm had cpvm java process.

@hooklee2000 did you test with a fresh installation or upgrade existing environment ?

[hooklee2000]

We tested all with a fresh installation.
By the way, using mysql 8, we must disable mysql's validate_password :

set global validate_password.policy=0;
set global validate_password.length=1;

otherwise cloudstack-setup-database printed error :
We apologize for below error:

cloudstack-setup-database  cloud:xxxxxx@127.0.0.1 --deploy-as=root:xxxxxx
***************************************************************
Encountering an error when executing mysql script
----------------------------------------------------------------------
table:
/usr/share/cloudstack-management/setup/create-database.sql

Error:
b"mysql: [Warning] Using a password on the command line interface can be insecure.\nERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)\n"

Sql parameters:
{'user': 'root', 'passwd': 'xxxxxx', 'port': 3306, 'host': '127.0.0.1'}
----------------------------------------------------------------------
            
***************************************************************
Please run:

    cloudstack-setup-databases -h

for full help

[weizhouapache]

We tested all with a fresh installation. By the way, using mysql 8, we must disable mysql's validate_password :

set global validate_password.policy=0;
set global validate_password.length=1;

otherwise cloudstack-setup-database printed error : We apologize for below error:

cloudstack-setup-database  cloud:xxxxxx@127.0.0.1 --deploy-as=root:xxxxxx
***************************************************************
Encountering an error when executing mysql script
----------------------------------------------------------------------
table:
/usr/share/cloudstack-management/setup/create-database.sql

Error:
b"mysql: [Warning] Using a password on the command line interface can be insecure.\nERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)\n"

Sql parameters:
{'user': 'root', 'passwd': 'xxxxxx', 'port': 3306, 'host': '127.0.0.1'}
----------------------------------------------------------------------
            
***************************************************************
Please run:

    cloudstack-setup-databases -h

for full help

@hooklee2000
can you file a new github issue for this ?