NioServer: retain links by address string to minimize resource leak

[yadvr]

Every time a client connects, the NioServer will retain the link against
the InetSocketAddress object. If the same agent/client reconnects, it
will grow older links over time and in case of denial of service attack
or a client/script/monitoring-service reconnecting aggressively against
port 8250 will cause the _links weak hashmap to grow over time and
very quickly.

The fix will ensure that only one Link gets weakly retained for an
incoming client based on its address string.

This is what is seen when only one kvm/agent connects to the management server:

When I restart the cloudstack-agent couple of times, I see the _links grow:

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-189

[yadvr]

@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-230)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26166 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3525-t230-kvm-centos7.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

[yadvr]

Caveat: for multiple-mgmt servers behind an LB, it will end up creating only one weak entry in the weak hash map. I'm not sure if it could cause any issues.

[svenvogel]

@rhtyd sounds intresting. LGTM. I think when anybody use a lb maybe the transparent mode can help?

[nathanejohnson]

LGTM. I've built this locally in our lab and it seems to work well

[DaanHoogland]

Caveat: for multiple-mgmt servers behind an LB, it will end up creating only one weak entry in the weak hash map. I'm not sure if it could cause any issues.

Do you mean that the two servers would share that hashmap, @rhtyd ? that seem strange to me. And if you mean on the agent side, that would only be a problem if the MSses are natted behind the same address, would it?

[DaanHoogland]

code looks good and as to your caveat @rhtyd; can we add an identifier of sorts to the key of the hashmap?

[yadvr]

In case of mgmt servers behind an LB, this will likely create a single entry in the weak hashmap. I'll revisit this towards 4.14 or 4.13.1; it's also possible to add a garbage collector to remove old/unreachable links from the heap.

[yadvr]

Sorry, reverting this PR because I remembered why it was put On-Hold, as the change would cause a single Link to be added to a Map when the mgmt server is behind a LB. So the changes are not optimal for all cases.