Skip to content

LB: per-rule keepalive and timeouts, plus three HAProxy config fixes - #14131

Open
bhouse-nexthop wants to merge 15 commits into
apache:mainfrom
bhouse-nexthop:haproxy-keepalive
Open

bhouse-nexthop wants to merge 15 commits into
apache:mainfrom
bhouse-nexthop:haproxy-keepalive

Conversation

@bhouse-nexthop

@bhouse-nexthop bhouse-nexthop commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Description

#12586 made the load balancer idle timeout globally configurable, as
network.loadbalancer.haproxy.idle.timeout. This expands that to per load balancer rule, and
does the same for keepalive, which until now could only be set on the network offering.

Where these settings live today:

setting scope set where reaches haproxy as
keepalive network offering keepaliveenabled, create-time only, no UI option httpclose in defaults
idle timeout global network.loadbalancer.haproxy.idle.timeout (#12574, #12586) timeout client, timeout server in defaults

Both land in the defaults section, so they apply to every LB rule on every network using that
offering. One rule cannot differ from another on the same router.

New parameters on createLoadBalancerRule and updateLoadBalancerRule

Each is optional, and each writes into that rule's own listen section, which overrides defaults:

parameter type haproxy directive applies to unset means
keepalive boolean option http-keep-alive, else option httpclose rules served in http mode the offering's keepaliveenabled
idletimeout long, ms timeout client and timeout server every rule, tcp and http the global from #12586
keepalivetimeout long, ms timeout http-keep-alive http mode, keepalive on idletimeout applies

idletimeout keeps the global's meaning, including 0 for infinite.

What that produces, taken from the generator - an http rule with all three set, a tcp rule with
only an idle timeout, and a rule that set nothing:

listen 10_2_0_2-80
	bind 10.2.0.2:80
	mode http
	option http-keep-alive
	timeout http-keep-alive 15000
	timeout client     600000
	timeout server     600000
	balance roundrobin
	server 10_2_0_2-80_0 10.1.10.3:80 check

listen 10_2_0_3-3306
	bind 10.2.0.3:3306
	timeout client     600000
	timeout server     600000
	balance roundrobin
	server 10_2_0_3-3306_0 10.1.10.4:3306 check

listen 10_2_0_4-8080
	bind 10.2.0.4:8080
	balance roundrobin
	server 10_2_0_4-8080_0 10.1.10.5:8080 check

The three values are returned on listLoadBalancerRules, and are settable on the load balancer
rule's edit form in the UI, where blank reads as "Use default".

A negative timeout, or one above 2147483647 ms (the largest HAProxy accepts), is rejected by the API before the rule is created or changed. Haproxy treats one as a fatal parse error, which would
leave every rule on that router serving its previous config, so the generator drops it as well.

Stored as firewall_rule_details. No schema change.

Applies to public LB rules. Application load balancers are created through a different command and
are not covered.

What this makes possible

A rule on port 80 can now have keep-alive and X-Forwarded-For. It could not before:

keepalive mode connection reuse XFF
offering flag off (the default) mode http no, Connection: close per response yes
offering flag on mode tcp yes no, forwardfor needs http mode
per rule, this PR mode http yes yes

A rule that sets nothing keeps today's behaviour exactly, including the fall back to mode tcp
when the offering flag is on.

Also fixes three bugs in the same file

1. option forceclose is not a valid keyword

With keepalive enabled, a rule that also uses HTTP-based stickiness or SSL offload emits
no option forceclose. HAProxy has rejected that keyword since 2.0, and the system VM is Debian 12
(HAProxy 2.6):

[ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not supported any more
        since HAProxy 2.0, please just remove it, or use 'option httpclose' if absolutely
        needed.
[ALERT] config : Fatal errors found in configuration.

The whole file is rejected, so haproxy keeps serving the previous config and the rule silently
never takes effect. Replaced with option http-keep-alive.

2. One rule's settings leak into every rule generated afterwards

globalSection and defaultsSection are static String[]. The generator wraps them in
Arrays.asList(...) and calls set(), which writes through to the arrays, so the value
survives the call:

set once by leaks as
keepalive on any rule no option httpclose in the defaults for every later rule, any network, any account
idleTimeout > 0 that timeout becomes the default for the next rule
idleTimeout == 0 timeouts blanked permanently

The generator now copies into a fresh list, and the sections themselves became List.of(...), so a
future set() throws instead of silently corrupting every config the agent builds afterwards.
Output is unchanged for a single rule.

3. appsession was removed from HAProxy in 1.6

An LB rule with AppCookie stickiness emitted appsession, which the shipped haproxy refuses:

[ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6.
[ALERT] config : Fatal errors found in configuration.

Same blast radius as bug 1 - the file is rejected, so every rule on that router keeps serving its
previous config. The VR advertises AppCookie as a supported stickiness method, so this is
reachable from the API.

Replaced with the documented stick table equivalent:

appsession replacement
len <n> stick-table type string len <n> size 10k
timeout <h> expire <h>
the cookie stick store-response res.cook(<name>) + stick match req.cook(<name>)
request-learn stick store-request req.cook(<name>)
prefix, mode no equivalent, logged and ignored

prefix and mode were never applied - the directive carrying them was rejected - so no working
behaviour changes.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

HAProxyConfiguratorTest Tests run: 17, Failures: 0 and LoadBalancingRulesManagerImplTest
Tests run: 16, Failures: 0.

Each bug fix has a case that fails without it:

  • keepalive with SSL offload emits no forceclose, and does emit option http-keep-alive
  • a keepalive config followed by a non-keepalive one leaves no no option httpclose behind
  • an idle timeout followed by an unset one falls back to the built-in default
  • AppCookie stickiness emits a stick table and never appsession

The rest cover the per-rule settings:

  • a rule that sets keepalive stays in http mode
  • a rule's keepalive wins over the offering, in both directions
  • a rule that sets nothing is byte for byte what it was before
  • timeout http-keep-alive is emitted, and only when keepalive is on
  • idletimeout overrides per rule on a tcp rule, leaving the defaults section alone
  • a negative timeout never reaches the config, and is rejected at the API
  • two rules in one config do not pick up each other's settings

Generated configs are checked against a real haproxy -c, which is where the alert text above
comes from. A file holding an AppCookie rule, an http rule with all three settings, a tcp rule
with a timeout, and a rule with a negative timeout is fatal before these changes and valid
after
- the three alerts above become zero.

@codecov

codecov Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 54.97382% with 86 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.80%. Comparing base (6caa33e) to head (8d3ec34).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
...a/com/cloud/network/router/CommandSetupHelper.java 0.00% 18 Missing ⚠️
...in/java/com/cloud/network/HAProxyConfigurator.java 70.73% 9 Missing and 3 partials ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java 0.00% 12 Missing ⚠️
...ork/router/VirtualNetworkApplianceManagerImpl.java 0.00% 12 Missing ⚠️
...in/java/com/cloud/agent/api/to/LoadBalancerTO.java 50.00% 9 Missing ⚠️
.../cloudstack/api/response/LoadBalancerResponse.java 0.00% 9 Missing ⚠️
...d/user/loadbalancer/UpdateLoadBalancerRuleCmd.java 33.33% 8 Missing ⚠️
...d/user/loadbalancer/CreateLoadBalancerRuleCmd.java 54.54% 5 Missing ⚠️
...loud/network/lb/LoadBalancingRulesManagerImpl.java 98.27% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14131      +/-   ##
============================================
+ Coverage     20.77%   20.80%   +0.02%     
- Complexity    20845    20905      +60     
============================================
  Files          6053     6053              
  Lines        548174   548413     +239     
  Branches      73227    73268      +41     
============================================
+ Hits         113866   114075     +209     
- Misses       420967   420977      +10     
- Partials      13341    13361      +20     
Flag Coverage Δ
uitests 3.86% <ø> (-0.01%) ⬇️
unittests 22.23% <54.97%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bhouse-nexthop
bhouse-nexthop marked this pull request as draft September 10, 2026 15:03
@bhouse-nexthop
bhouse-nexthop changed the base branch from 4.22 to main September 10, 2026 15:10
@bhouse-nexthop bhouse-nexthop changed the title Fix two bugs in the virtual router's HAProxy config LB: per-rule keepalive and timeouts, plus two HAProxy config fixes Sep 10, 2026
@bhouse-nexthop
bhouse-nexthop marked this pull request as ready for review September 10, 2026 16:04
@bhouse-nexthop bhouse-nexthop changed the title LB: per-rule keepalive and timeouts, plus two HAProxy config fixes LB: per-rule keepalive and timeouts, plus three HAProxy config fixes Sep 10, 2026
@weizhouapache weizhouapache added this to the 24.0 milestone Sep 10, 2026
@weizhouapache

Copy link
Copy Markdown
Member

@bhouse-nexthop
could you make some changes for VR health check ?
similar to #12596 which fixes #12591

@boring-cyborg boring-cyborg Bot added component:virtual-router Python Warning... Python code Ahead! labels Sep 10, 2026
@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

@weizhouapache good catch, thanks - added in the latest commit.

haproxy_check.py knew about maxconn and the global idle timeout, so the new per rule settings
could drift with nothing noticing. The health check data now carries all three, and the check
validates them in that rule's own listen section:

data checked
ruleKeepAlive option http-keep-alive, or option httpclose
ruleIdleTimeout timeout client, timeout server
ruleKeepAliveTimeout timeout http-keep-alive

The http mode test needed widening as well, otherwise a rule that sets keepalive itself stopped
being checked at all:

before   port 80 and the offering has keepalive off
after    port 80 and (the rule sets keepalive, or the offering has it off)

Compatibility: an older management server sends none of these keys, and a rule that inherits sends
them empty. Both read as nothing to check, so an updated system VM template against an older
management server still passes.

I ran the script against generated configs rather than only reading it - a matching config passes,
and each of a wrong keepalive option, a drifted timeout client and a missing
timeout http-keep-alive fails.

@Damans227 Damans227 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @bhouse-nexthop, the changes look right. CLGTM.

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19626

@RosiKyu

RosiKyu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 20002

@RosiKyu

RosiKyu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 20003

@RosiKyu

RosiKyu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 20007

@DaanHoogland

Copy link
Copy Markdown
Contributor

@bhouse-nexthop FYI:

22:05:40 [ERROR] /jenkins/workspace/acs-centos8-pkg-builder/dist/rpmbuild/BUILD/cloudstack-24.0.0-SNAPSHOT/server/src/test/java/com/cloud/network/lb/LoadBalancingRulesManagerImplTest.java:[586,28] error: cannot find symbol
22:05:40   symbol:   variable Collections
22:05:40   location: class LoadBalancingRulesManagerImplTest

Signed-off-by: Brad House <bhouse@nexthop.ai>
@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

@DaanHoogland thanks for the heads-up. #14166 removed the test's java.util.Collections import along with its last use, and this PR adds a new use, so the merged result didn't compile. I've merged main into the branch and restored the import (3f69e75). LoadBalancingRulesManagerImplTest (28), UpdateLoadBalancerTest (2) and HAProxyConfiguratorTest (17) pass on the merged code.

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20030

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-17577)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 56783 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14131-t17577-kvm-ol8.zip
Smoke tests completed. 157 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

Thanks @DaanHoogland. Trillian is clean on the build with main merged in (157 OK, 0 errors). The one Simulator CI failure is in test_affinity_groups_projects: all 16 of its tests passed, and only the suite's teardown failed, when deleting its test domain. That isn't related to this PR. From my side it's ready to merge.

@RosiKyu RosiKyu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bhouse-nexthop, nice work!

I've tested this on a KVM lab (Oracle Linux 9, advanced zone, system VM template 4.22.0, HAProxy 2.6.12): baseline 24.0 build first, then with the PR packages. I've first focused on what your report lists as not covered, plus upgrade and extra negative cases. One issue needs a fix before merge.

Issue: a create refused for a negative value still leaves a rule, which later goes live

  1. createLoadBalancerRule name=neg publicipid={ip} networkid={net} publicport=8088 privateport=80 algorithm=roundrobin keepalivetimeout=-1
  2. The API answers 431 keepalivetimeout must be 0 or greater, got [-1]. 0 means no timeout. (same with idletimeout=-1).
  3. listLoadBalancerRules still shows rule neg on port 8088 (state Add, like any new rule without instances).
  4. Creating a valid rule on 8088 is refused: The range specified, 8088-8088, conflicts with rule FirewallRule {... "state":"Add" ...}.
  5. Edit any other rule on the same network. The "refused" rule is now pushed to the router: its firewall rule turns Active and HAProxy listens on {ip}:8088 (listen ..._8088 section in haproxy.cfg).

Expected Result: when the create is refused, no rule is left behind, as with the older checks: algorithm=foo, protocol=foo and an invalid cidrlist are refused on the same build and leave nothing. Deleting the leftover rule works as a workaround.

What passed verification

  • Upgrade: rules created on the old build (http 80, tcp, ssl, tcp-proxy) render the same after the upgrade, haproxy -c passes, and traffic through them still works.
  • SSL offload rule with keepalive=true idletimeout=300000 keepalivetimeout=10000: option http-keep-alive, timeout http-keep-alive 10000 and timeout client/server 300000 in the ssl listen section, the bind line and certificate unchanged, and an HTTP/2 connection is reused.
  • tcp-proxy rule with keepalive and timeouts: the timeouts apply, send-proxy is kept and the backend still receives the PROXY header. Keepalive is ignored, with "Keepalive ignored for lb rule ..., it is served in tcp mode" in the agent log.
  • Port 80 with keepalive=false keepalivetimeout=7000: option httpclose, and "Keepalive timeout ignored ..., keepalive is off for this rule" in the log.
  • Health check: a hand-edited timeout on the tcp-proxy rule is reported ("Expected 'timeout client 120000' ... but found 120001"), and a real re-apply fixes it.
  • UI edit form: shows the current values, with empty fields as "Use default". Saving sets the values on the rule and the router. Clearing one field resets only that one (the others are kept). "Use default" plus cleared fields puts the rule back to option httpclose. -5 shows "idletimeout must be 0 or greater, got [-5]" and nothing changes.
  • idletimeout=abc is rejected, and another user can't edit the rule.

Minor, not blocking

  • idletimeout=99999999999 is accepted by the API, but the router refuses the config ("timer overflow ... maximum value is 2147483647 ms"), so the user gets only "Failed to update load balancer rule". The old value is kept. A max check in the API would give a clear message.
  • idletimeout=0 ("no timeout") renders timeout client 0, and haproxy -c warns "missing timeouts ... you will certainly encounter various problems".

Evidence

Lab mapping: {ip} = 10.0.60.223 (public IP of the network, source NAT), {net} = d31838b7-63e1-4345-88d1-cf6a3fde1de9 (isolated network qa13789-net), router r-4-VM, management server ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1.

Management server, steps 1-2: the create with keepalivetimeout=-1 is refused.

[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule name=qa14131-neg publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 publicport=8088 privateport=80 algorithm=roundrobin keepalivetimeout=-1 filter=id,name
🙈 Error: (HTTP 431, error code 9999) keepalivetimeout must be 0 or greater, got [-1]. 0 means no timeout.

Management server, step 3: the refused rule qa14131-neg exists, and its firewall rule on 8088 is there too.

[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list loadbalancerrules listall=true id=2a3b5fae-31a1-4590-88de-364ff4489c5c filter=name,publicport,privateport,state,keepalive,keepalivetimeout
{
  "count": 1,
  "loadbalancerrule": [
    {
      "name": "qa14131-neg",
      "privateport": "80",
      "publicport": "8088",
      "state": "Add"
    }
  ]
}
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list firewallrules listall=true ipaddressid=b638e93e-2728-42fc-86f2-3e772f0c01e7 filter=startport,state,purpose
{
  "count": 5,
  "firewallrule": [
    {
      "startport": 8088,
      "state": "Add"
    },
    {
      "startport": 3306,
      "state": "Active"
    },
    {
      "startport": 443,
      "state": "Active"
    },
    {
      "startport": 2222,
      "state": "Active"
    },
    {
      "startport": 80,
      "state": "Active"
    }
  ]
}

Management server, steps 2 and 4: the same with idletimeout=-1, then a valid rule on 8088 is refused because of the leftover.

[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule name=qa14131-neg2 publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 publicport=8089 privateport=80 algorithm=roundrobin idletimeout=-1 filter=id,name
🙈 Error: (HTTP 431, error code 9999) idletimeout must be 0 or greater, got [-1]. 0 means no timeout.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule name=qa14131-retry publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 publicport=8088 privateport=80 algorithm=roundrobin filter=id,name
🙈 Error: (HTTP 537, error code 9999) The range specified, 8088-8088, conflicts with rule FirewallRule {"id":9,"networkId":204,"purpose":"LoadBalancing","state":"Add","uuid":"2a3b5fae-31a1-4590-88de-364ff4489c5c"} which has 8088-8088

Management server and router, step 5: after another rule on the network was edited, the leftover's firewall rule is Active and HAProxy listens on 8088.

[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list loadbalancerrules listall=true keyword=qa14131-neg filter=id,name,publicport,state
{
  "count": 1,
  "loadbalancerrule": [
    {
      "id": "2a3b5fae-31a1-4590-88de-364ff4489c5c",
      "name": "qa14131-neg",
      "publicport": "8088",
      "state": "Add"
    }
  ]
}
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list firewallrules listall=true ipaddressid=b638e93e-2728-42fc-86f2-3e772f0c01e7 filter=startport,state
{
  "count": 5,
  "firewallrule": [
    {
      "startport": 8088,
      "state": "Active"
    },
    {
      "startport": 3306,
      "state": "Active"
    },
    {
      "startport": 443,
      "state": "Active"
    },
    {
      "startport": 2222,
      "state": "Active"
    },
    {
      "startport": 80,
      "state": "Active"
    }
  ]
}
root@r-4-VM:~# sed -n "/^listen 10_0_60_223-8088/,/^\s*$/p" /etc/haproxy/haproxy.cfg
listen 10_0_60_223-8088
	bind 10.0.60.223:8088
	balance roundrobin
	 
root@r-4-VM:~# ss -ltnp | grep 8088
LISTEN 0      4096     10.0.60.223:8088      0.0.0.0:*    users:(("haproxy",pid=9566,fd=9))                                                    

Management server: older create checks on the same build refuse and leave nothing behind.

[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 privateport=80 filter=id,name name=qa14131-old1 publicport=8090 algorithm=foo
🙈 Error: (HTTP 431, error code 9999) Invalid algorithm: foo
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 privateport=80 filter=id,name name=qa14131-old2 publicport=8091 algorithm=roundrobin protocol=foo
🙈 Error: (HTTP 431, error code 9999) protocol foo is not in valid protocols [tcp, udp, tcp-proxy, ssl]
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 privateport=80 filter=id,name name=qa14131-old3 publicport=8092 algorithm=roundrobin cidrlist=1.2.3.400/33
🙈 Error: (HTTP 431, error code 9999) CIDR [1.2.3.400/33] is invalid.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# for n in qa14131-old1 qa14131-old2 qa14131-old3; do cmk list loadbalancerrules listall=true name=$n filter=name,publicport,state; echo "$n: $(cmk list loadbalancerrules listall=true name=$n | grep -c "\"name\"") rule(s)"; done
qa14131-old1: 0 rule(s)
qa14131-old2: 0 rule(s)
qa14131-old3: 0 rule(s)

createLoadBalancerRule stored the per rule settings only after the
rule had been created, so a refused keepalivetimeout or idletimeout
left the rule behind in state Add. It held its port, and the next
apply on the network pushed it to the router.

Check the timeouts first, so a refused create leaves nothing. Also
refuse a timeout above 2147483647 ms, which HAProxy rejects as a timer
overflow; it used to reach the router and fail the whole apply.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

@RosiKyu thanks for the thorough test, and for the clear repro. You're right about the leftover rule. Fixed in 8d3ec34.

The leftover rule. createLoadBalancerRule created the rule first and stored the per rule settings after it, so the timeout check ran too late. The command now checks idletimeout and keepalivetimeout before the rule is created. A refused create leaves no rule and no firewall rule, the same as algorithm=foo. A new unit test (CreateLoadBalancerRuleCmdTest) checks that a refused value never reaches createPublicLoadBalancerRule. Any rule already left behind by the old build still needs deleting by hand, as you did.

idletimeout=99999999999. Taken up: the API now refuses a timeout above 2147483647 ms, the largest HAProxy accepts. The error says so, so the user no longer gets only "Failed to update load balancer rule". The check covers both timeouts, on create and on update.

idletimeout=0. I've left this one as it is. 0 means no timeout, the same as for the global network.loadbalancer.haproxy.idle.timeout from #12586. When the global is 0, the generator removes timeout client and timeout server from defaults, and haproxy -c gives the same "missing timeouts" warning. The per rule value matches that meaning. The warning is HAProxy's advice against unlimited timeouts, and the parameter description already says what 0 does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants