KVM: DPDK live migrations by nvazquez · Pull Request #3365 · apache/cloudstack

nvazquez · 2019-05-30T12:41:19Z

Description

This feature allows administrators to perform live migrations on DPDK enabled VMs between DPDK enabled hosts in the same cluster.

A DPDK enabled host must be configured to enable the CloudStack DPDK support. Properties on agent.properties must be populated as:

openvswitch.dpdk.enabled=true
openvswitch.dpdk.ovs.path=<OVS_PATH>
network.bridge.type=openvswitch
libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver

Each time a CloudStack agent on a KVM host starts a connection to its management server, it sends a comma-separated list of capabilities available on the host, such as: 'hvm', 'snapshot'. The management server persists the information received from the agent on database ('capabilities' column on 'hosts' table)

When the property 'openvswitch.dpdk.enabled' = true, the 'dpdk' capability is included in the list of capabilities sent to the management server.
When the property 'openvswitch.dpdk.enabled' = false, the list of capabilities sent to the management server does not inclide the 'dpdk' capability.

Please note that agent restart is required after any property from agent.properties file is changed. Also having the dpdk capability, the hosts will be able and will host VMs without the dpdk configuration. In this case, openvswitch ports without DPDK support are created, just like for system VMs.

The management server is enhanced to determinate if a VM is a DPDK enabled VM when:

The VM type is 'User'
The VM details or service offering details contain additional configurations with the name:
- 'dpdk-hugepages'
- 'dpdk-numa'
The KVM host on which the VM is running contains the 'dpdk' capability.

API changes

The 'findSuitableHostsForMigration' API is enhanced to support DPDK enabled VMs.

When the VM is a DPDK enabled VM then only DPDK enabled hosts must be suitable for migration. Suitable hosts must be only the ones with the 'dpdk' capability as part of its capabilities.

The 'migrateVirtualMachine' API method is extended to support DPDK enabled VMs live migrations:

When the VM to live migrate is not a DPDK enabled VM, the logic remains unchanged.
When the VM to live migrate is a DPDK enabled VM, it can be live migrated between two hosts when:
- The KVM source host is a DPDK enabled host (the 'dpdk' capability is part of its capabilities)
- The KVM destination host is a DPDK enabled host on the same cluster as the source host
- If the destination host is not a DPDK enabled host, then the migration must fail with a descriptive message.
- Before attempting the live migration, the destination host must be prepared for the migration:
  - A new DPDK port must be created for each interface of the VM
- Before attempting the live migration, the source host must:
  - Obtain the VM XML domain as a copy of the current XML domain
  - Replace the XML parts, referencing the new DPDK ports on the destination host for each interface
- After the migration is successful, the VM interface DPDK ports on the source host must be removed.
- In case of migration failure, the DPDK ports created on the destination hosts are cleaned up.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

borisstoyanov · 2019-05-30T12:41:56Z

@blueorangutan package

blueorangutan · 2019-05-30T12:42:28Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-05-30T13:02:53Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2816

borisstoyanov · 2019-05-30T16:31:55Z

@blueorangutan package

blueorangutan · 2019-05-30T16:32:29Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-05-30T17:07:10Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2817

borisstoyanov · 2019-05-30T17:08:04Z

@blueorangutan test

blueorangutan · 2019-05-30T17:08:29Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-05-30T22:31:17Z

Trillian test result (tid-3613)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32361 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3365-t3613-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 70 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

borisstoyanov · 2019-05-31T06:39:35Z

@blueorangutan test

blueorangutan · 2019-05-31T06:40:31Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov

LGTM, based on marvin tests, code review and manual verification on DPDK enabled hosts, both covering normal and DPDK VMs. It passed tests for migration(positive and negative) on enabled and non VMs. Also cleanup is properly done as well as host capability recognition.
test results: test-results.xlsx

blueorangutan · 2019-05-31T16:34:10Z

Trillian test result (tid-3624)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33753 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3365-t3624-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 69 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_hostha_enable_ha_when_host_in_maintenance	`Error`	301.71	test_hostha_kvm.py

borisstoyanov · 2019-06-05T12:55:24Z

@nvazquez can you address the merge conflicts pls.

blueorangutan · 2019-06-10T13:25:15Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-06-10T20:56:52Z

Trillian test result (tid-3679)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 25576 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3365-t3679-kvm-centos7.zip
Smoke tests completed. 71 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland

looks good overall, to large to judge in detail. the prove of the code is in the testing.

nvazquez · 2019-06-17T13:09:45Z

Thanks @DaanHoogland, renamed classes after your comment
@blueorangutan package

blueorangutan · 2019-06-17T13:10:44Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-17T13:31:33Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2893

nvazquez · 2019-06-17T13:36:25Z

@blueorangutan test

blueorangutan · 2019-06-17T13:36:45Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-06-17T16:12:10Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2894

blueorangutan · 2019-06-17T16:44:52Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2895

blueorangutan · 2019-06-17T17:13:26Z

Packaging result: ✔centos6 ✔centos7 ✖debian. JID-2896

blueorangutan · 2019-06-18T00:06:14Z

Trillian test result (tid-3696)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36164 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3365-t3696-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 71 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2019-06-20T11:31:48Z

@blueorangutan package

blueorangutan · 2019-06-20T11:32:59Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-20T12:15:06Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2912

nvazquez · 2019-06-20T12:55:08Z

@blueorangutan test

blueorangutan · 2019-06-20T12:55:57Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-06-20T23:13:20Z

Trillian test result (tid-3712)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35287 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3365-t3712-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 69 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_rvpc_multi_tiers	`Failure`	404.25	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	430.53	test_vpc_redundant.py
test_01_redundant_vpc_site2site_vpn	`Failure`	268.41	test_vpc_vpn.py

nvazquez · 2019-06-25T03:45:00Z

@blueorangutan package

blueorangutan · 2019-06-25T03:46:09Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-25T04:17:52Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-11

nvazquez · 2019-06-25T04:48:49Z

@blueorangutan test

blueorangutan · 2019-06-25T04:49:07Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-06-25T08:27:33Z

@anuragaw @shwstppr @DaanHoogland please review, thanks.

blueorangutan · 2019-06-25T15:13:45Z

Trillian test result (tid-9)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35967 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3365-t9-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 71 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2019-06-25T15:22:47Z

Merging based on approvals and test results

nvazquez added component:openvswitch component:dpdk component:migration labels May 30, 2019

nvazquez added this to the 4.13.0.0 milestone May 30, 2019

nvazquez self-assigned this May 30, 2019

apache deleted a comment from blueorangutan May 31, 2019

borisstoyanov approved these changes May 31, 2019

View reviewed changes

nvazquez mentioned this pull request Jun 3, 2019

DPDK live migrations apache/cloudstack-documentation#45

Merged

nvazquez changed the title ~~DPDK live migrations~~ KVM: DPDK live migrations Jun 3, 2019

nvazquez added the component:kvm label Jun 3, 2019

nvazquez closed this Jun 3, 2019

nvazquez reopened this Jun 3, 2019

borisstoyanov requested review from DaanHoogland, PaulAngus, andrijapanicsb and yadvr June 5, 2019 12:54

DaanHoogland approved these changes Jun 13, 2019

View reviewed changes

Comment thread api/src/main/java/com/cloud/agent/api/to/DPDKTO.java Outdated

Rename DPDK classes lowercase

ae59f87

yadvr approved these changes Jun 25, 2019

View reviewed changes

nvazquez merged commit a75444a into apache:master Jun 25, 2019

nvazquez deleted the dpdklivemigrations branch April 6, 2020 14:41

Uh oh!

Conversation

nvazquez commented May 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

API changes

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

borisstoyanov commented May 30, 2019

Uh oh!

blueorangutan commented May 30, 2019

Uh oh!

blueorangutan commented May 30, 2019

Uh oh!

borisstoyanov commented May 30, 2019

Uh oh!

blueorangutan commented May 30, 2019

Uh oh!

blueorangutan commented May 30, 2019

Uh oh!

borisstoyanov commented May 30, 2019

Uh oh!

blueorangutan commented May 30, 2019

Uh oh!

blueorangutan commented May 30, 2019

Uh oh!

borisstoyanov commented May 31, 2019

Uh oh!

blueorangutan commented May 31, 2019

Uh oh!

borisstoyanov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented May 31, 2019

Uh oh!

borisstoyanov commented Jun 5, 2019

Uh oh!

blueorangutan commented Jun 10, 2019

Uh oh!

blueorangutan commented Jun 10, 2019

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nvazquez commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 17, 2019

Uh oh!

nvazquez commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 17, 2019

Uh oh!

blueorangutan commented Jun 18, 2019

Uh oh!

nvazquez commented Jun 20, 2019

Uh oh!

blueorangutan commented Jun 20, 2019

Uh oh!

blueorangutan commented Jun 20, 2019

Uh oh!

nvazquez commented Jun 20, 2019

Uh oh!

blueorangutan commented Jun 20, 2019

Uh oh!

blueorangutan commented Jun 20, 2019

Uh oh!

nvazquez commented Jun 25, 2019

Uh oh!

nvazquez commented May 30, 2019 •

edited

Loading

borisstoyanov left a comment •

edited

Loading