[KVM] Agent LB Fix: Connections from disabled KVM host agents are refused by nvazquez · Pull Request #3617 · apache/cloudstack

nvazquez · 2019-10-04T05:28:49Z

Description

This PR introduces a fix for a limitation on the KVM agent load balancing feature.

When a host is in 'Disabled' state and the agent is restarted, then the host is not able to connect to the management server, staying indefinitely in 'Connecting' state

2019-10-04 05:23:12,242 DEBUG [c.c.r.ResourceState] (AgentConnectTaskPool-5:ctx-e3780a57) (logid:7c4cc142) Resource state update: [id = 4; name = trl-65-k-M7-nvazquez-kvm4; old state = Disabled; event = InternalCreated; new state = Disabled]
2019-10-04 05:23:12,242 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5:ctx-e3780a57) (logid:7c4cc142) Transition:[Resource state = Disabled, Agent event = AgentConnected, Host id = 4, name = trl-65-k-M7-nvazquez-kvm4]
2019-10-04 05:23:12,248 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5:ctx-e3780a57) (logid:7c4cc142) Failed to handle host connection: 
java.lang.IndexOutOfBoundsException: fromIndex = -1
	at java.util.SubList.<init>(AbstractList.java:620)
	at java.util.RandomAccessSubList.<init>(AbstractList.java:775)
	at java.util.AbstractList.subList(AbstractList.java:484)
	at org.apache.cloudstack.agent.lb.algorithm.IndirectAgentLBRoundRobinAlgorithm.sort(IndirectAgentLBRoundRobinAlgorithm.java:44)
	at org.apache.cloudstack.agent.lb.IndirectAgentLBServiceImpl.getManagementServerList(IndirectAgentLBServiceImpl.java:93)
	at org.apache.cloudstack.agent.lb.IndirectAgentLBServiceImpl.compareManagementServerList(IndirectAgentLBServiceImpl.java:101)
	at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1094)
	at com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:126)
	at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1187)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-10-04 05:23:12,248 WARN  [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5:ctx-e3780a57) (logid:7c4cc142) Unable to create attache for agent: Seq 0-0:  { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":3,"cpus":3,"speed":1599,"memory":7277121536,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-957.12.2.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.6.1810","secured":"true","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"d609214d-020e-3727-8515-c58c792f1d1a-LibvirtComputingResource","name":"trl-65-k-M7-nvazquez-kvm4","id":0,"version":"4.14.0.0-SNAPSHOT","iqn":"iqn.1994-05.com.redhat:213e8ff8bef8","privateIpAddress":"10.2.2.79","privateMacAddress":"1e:00:56:01:07:47","privateNetmask":"255.255.0.0","storageIpAddress":"10.2.2.79","storageNetmask":"255.255.0.0","storageMacAddress":"1e:00:56:01:07:47","resourceName":"LibvirtComputingResource","gatewayIpAddress":"10.2.254.254","msHostList":"10.2.2.84,10.2.2.104@roundrobin","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"a1708750-e55c-4b15-89b3-2b9cc39b6a77","host":"10.2.2.79","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":20935868416,"availableBytes":19100352512},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"d609214d-020e-3727-8515-c58c792f1d1a-LibvirtComputingResource","name":"trl-65-k-M7-nvazquez-kvm4","id":0,"version":"4.14.0.0-SNAPSHOT","resourceName":"LibvirtComputingResource","msHostList":"10.2.2.84,10.2.2.104@roundrobin","wait":0}}] }
2019-10-04 05:23:12,263 WARN  [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Throwing away a request because it came through as the first command on a connect: Seq 0--1:  { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 111, [{"com.cloud.agent.api.ShutdownCommand":{"reason":"sig.kill","wait":0}}] }
2019-10-04 05:23:12,395 WARN  [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Throwing away a request because it came through as the first command on a connect: Seq 0-1:  { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":0,"wait":0}}] }
2019-10-04 05:23:13,320 INFO  [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Connection from /10.2.2.79 closed but no cleanup was done.

The main reason is that when the administrator changed the value of the global configurations host or indirect.agent.lb.algorithm, the LB mechanism in the management server sent the updated values of these configurations only to the hosts in 'Enabled' state. In particular, hosts in 'Disabled' state did not receive the updated management server list (and LB algorithm).

Types of changes

Bug fix (non-breaking change which fixes an issue)

Screenshots (if appropriate):

How Has This Been Tested?

In a KVM environment:
Set global settings as following:

• indirect.agent.lb.algorithm set to “roundrobin”
• indirect.agent.lb.check.interval set to 360 (must be different than 0)

Case 1:

Disable a host H
Restart agent on host H
Verify that the host is 'Up'

Case 2:

Disable a host H
Update one of the global settings: host or indirect.agent.lb.algorithm
Verify that agent in host H receives the updated values of the configurations

nvazquez · 2019-10-04T05:29:22Z

@blueorangutan package

blueorangutan · 2019-10-04T05:29:48Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-10-04T05:49:32Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-306

nvazquez · 2019-10-04T06:01:46Z

@blueorangutan test

blueorangutan · 2019-10-04T06:02:48Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-10-22T09:21:16Z

@blueorangutan package

blueorangutan · 2019-10-22T09:22:11Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-10-22T10:10:28Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-327

andrijapanicsb · 2019-11-14T09:25:06Z

@blueorangutan test

blueorangutan · 2019-11-14T09:25:54Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr

Needs testing

andrijapanicsb · 2019-11-25T12:43:53Z

@blueorangutan package

blueorangutan · 2019-11-25T12:44:33Z

@andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-11-25T13:02:48Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-385

andrijapanicsb · 2019-11-25T23:27:13Z

@blueorangutan test

blueorangutan · 2019-11-25T23:27:35Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

andrijapanicsb · 2019-11-25T23:32:17Z

LGTM.

Tested manually as following:

TestID	Test Name	Steps	Expected Result	Status
1	New configuration values are propagated to host in “Disabled” state	Disable a host. Change value of the “host” and/or “indirect.agent.lb.algorithm” global settings.	Observe the content of the /etc/cloudstack/agent/agent.properties contain the new values, even though the host is still Disabled.	Pass
2	New setting values are propagated to the host in the PrepareForMaintenance	Put host in maintenance mode. While the host is in the PrepareForMaintenance state, change the values of “host” and/or “indirect.agent.lb.algorithm”.	Observe the content of the /etc/cloudstack/agent/agent.properties contain the new values from the “host” and/or ” indirect.agent.lb.algorithm” almost immediately (i.e. no more than few seconds after the values are changed via API call)	Pass
3	New configuration values are propagated to the host that is already in Maintenance mode	Put host in maintenance mode. After the host is in the Maintenance state, change the values of “host” and/or “indirect.agent.lb.algorithm”.	Observe the content of the /etc/cloudstack/agent/agent.properties contain the new values from the “host” and/or ” indirect.agent.lb.algorithm” almost immediately (i.e. no more than few seconds after the values are changed via API call)	Pass
5	Host is able to connect back to management server, after the restart	Disable the host. Change the values of “host” and/or “indirect.agent.lb.algorithm”. Restart the cloudstack-agent on the host	Observe that host will temporarily be in the “Connecting” state and will eventually transition to the “Up” state (connected to a management server)	Pass
6	A host doesn’t require agent restart in order to exit Maintenance mode	While having "kvm.ssh.to.agent" seto to false , put the host into the Maintenance mode. Do NOT restart cloudstack-agent . Cancel the Maintenance mode for the host	Observe that host “Resource state” will transition from “Maintenance” to “Enabled”	Pass

GabrielBrascher

Code LGTM

andrijapanicsb · 2019-11-26T16:17:21Z

@blueorangutan test

blueorangutan · 2019-11-26T16:17:38Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-11-27T03:07:39Z

Trillian test result (tid-551)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37276 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3617-t551-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

andrijapanicsb · 2019-11-27T09:28:31Z

LGTM.

(Extensive manual testing)

GabrielBrascher · 2019-11-27T13:56:38Z

Just to keep track on PRs vs Issues. This fixes #3581.

nvazquez · 2019-11-27T14:59:04Z

Thanks @andrijapanicsb and @GabrielBrascher, have also added the Issue number on the description after Gabriel's comment

nvazquez · 2019-12-02T12:18:45Z

@blueorangutan package

blueorangutan · 2019-12-02T12:19:04Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-12-02T12:39:56Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-416

nvazquez · 2019-12-02T12:47:19Z

@blueorangutan test

blueorangutan · 2019-12-02T12:48:04Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-12-02T23:25:13Z

Trillian test result (tid-572)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36554 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3617-t572-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2019-12-04T13:42:58Z

Merging based on reviews and test results

…used (apache#3617) * Remove constraint for just enabled hosts receiving the ms list * Allow sending SetupMSList command when host is in maintenance

nvazquez added type:bug component:kvm labels Oct 4, 2019

nvazquez added this to the 4.14.0.0 milestone Oct 4, 2019

nvazquez self-assigned this Oct 4, 2019

andrijapanicsb self-requested a review November 12, 2019 15:12

yadvr approved these changes Nov 25, 2019

View reviewed changes

GabrielBrascher reviewed Nov 25, 2019

View reviewed changes

Comment thread server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java

andrijapanicsb approved these changes Nov 25, 2019

View reviewed changes

GabrielBrascher approved these changes Nov 26, 2019

View reviewed changes

nvazquez force-pushed the disabledhostsagentlb branch from 9fd4de9 to 9bb6477 Compare November 26, 2019 15:50

apache deleted a comment from blueorangutan Nov 26, 2019

andrijapanicsb changed the title ~~[KVM] Agent LB Fix: Connections from disabled KVM host agents are refused~~ [WIP - DO NOT MERGE] [KVM] Agent LB Fix: Connections from disabled KVM host agents are refused Nov 27, 2019

apache deleted a comment from blueorangutan Nov 27, 2019

nvazquez added 2 commits December 2, 2019 09:16

Remove constraint for just enabled hosts receiving the ms list

bfb8b30

Allow sending SetupMSList command when host is in maintenance

3023a2b

nvazquez force-pushed the disabledhostsagentlb branch from 9bb6477 to 3023a2b Compare December 2, 2019 12:17

nvazquez changed the base branch from master to 4.13 December 2, 2019 12:17

nvazquez changed the title ~~[WIP - DO NOT MERGE] [KVM] Agent LB Fix: Connections from disabled KVM host agents are refused~~ [KVM] Agent LB Fix: Connections from disabled KVM host agents are refused Dec 2, 2019

nvazquez modified the milestones: 4.14.0.0, 4.13.1.0 Dec 2, 2019

nvazquez merged commit 07d64cb into apache:4.13 Dec 4, 2019

yadvr mentioned this pull request Aug 12, 2020

Agent LB not working with disabled hosts #3581

Closed

Uh oh!

Conversation

nvazquez commented Oct 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented Oct 4, 2019

Uh oh!

blueorangutan commented Oct 4, 2019

Uh oh!

blueorangutan commented Oct 4, 2019

Uh oh!

nvazquez commented Oct 4, 2019

Uh oh!

blueorangutan commented Oct 4, 2019

Uh oh!

yadvr commented Oct 22, 2019

Uh oh!

blueorangutan commented Oct 22, 2019

Uh oh!

blueorangutan commented Oct 22, 2019

Uh oh!

andrijapanicsb commented Nov 14, 2019

Uh oh!

blueorangutan commented Nov 14, 2019

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

andrijapanicsb commented Nov 25, 2019

Uh oh!

blueorangutan commented Nov 25, 2019

Uh oh!

blueorangutan commented Nov 25, 2019

Uh oh!

Uh oh!

andrijapanicsb commented Nov 25, 2019

Uh oh!

blueorangutan commented Nov 25, 2019

Uh oh!

andrijapanicsb commented Nov 25, 2019

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

andrijapanicsb commented Nov 26, 2019

Uh oh!

blueorangutan commented Nov 26, 2019

Uh oh!

blueorangutan commented Nov 27, 2019

Uh oh!

andrijapanicsb commented Nov 27, 2019

Uh oh!

GabrielBrascher commented Nov 27, 2019

Uh oh!

nvazquez commented Nov 27, 2019

Uh oh!

nvazquez commented Dec 2, 2019

Uh oh!

blueorangutan commented Dec 2, 2019

Uh oh!

blueorangutan commented Dec 2, 2019

Uh oh!

nvazquez commented Dec 2, 2019

Uh oh!

blueorangutan commented Dec 2, 2019

Uh oh!

blueorangutan commented Dec 2, 2019

Uh oh!

nvazquez commented Dec 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

nvazquez commented Oct 4, 2019 •

edited

Loading