Repository navigation
LB: per-rule keepalive and timeouts, plus three HAProxy config fixes - #14131
bhouse-nexthop wants to merge 15 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14131 +/- ##
============================================
+ Coverage 20.77% 20.80% +0.02%
- Complexity 20845 20905 +60
============================================
Files 6053 6053
Lines 548174 548413 +239
Branches 73227 73268 +41
============================================
+ Hits 113866 114075 +209
- Misses 420967 420977 +10
- Partials 13341 13361 +20
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
a8f56df to
4eed46e
Compare
4eed46e to
6a432c2
Compare
6a432c2 to
08327da
Compare
|
@bhouse-nexthop |
|
@weizhouapache good catch, thanks - added in the latest commit.
The http mode test needed widening as well, otherwise a rule that sets keepalive itself stopped Compatibility: an older management server sends none of these keys, and a rule that inherits sends I ran the script against generated configs rather than only reading it - a matching config passes, |
There was a problem hiding this comment.
thanks @bhouse-nexthop, the changes look right. CLGTM.
|
@blueorangutan package |
|
@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19626 |
|
@blueorangutan package |
|
@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 20002 |
|
@blueorangutan package |
|
@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 20003 |
|
@blueorangutan package |
|
@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 20007 |
|
@bhouse-nexthop FYI: |
Signed-off-by: Brad House <bhouse@nexthop.ai>
|
@DaanHoogland thanks for the heads-up. #14166 removed the test's |
|
@blueorangutan package |
|
@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20030 |
|
@blueorangutan test |
|
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17577)
|
|
Thanks @DaanHoogland. Trillian is clean on the build with main merged in (157 OK, 0 errors). The one Simulator CI failure is in |
RosiKyu
left a comment
There was a problem hiding this comment.
Thanks @bhouse-nexthop, nice work!
I've tested this on a KVM lab (Oracle Linux 9, advanced zone, system VM template 4.22.0, HAProxy 2.6.12): baseline 24.0 build first, then with the PR packages. I've first focused on what your report lists as not covered, plus upgrade and extra negative cases. One issue needs a fix before merge.
Issue: a create refused for a negative value still leaves a rule, which later goes live
createLoadBalancerRule name=neg publicipid={ip} networkid={net} publicport=8088 privateport=80 algorithm=roundrobin keepalivetimeout=-1- The API answers
431 keepalivetimeout must be 0 or greater, got [-1]. 0 means no timeout.(same withidletimeout=-1). listLoadBalancerRulesstill shows rulenegon port 8088 (stateAdd, like any new rule without instances).- Creating a valid rule on 8088 is refused:
The range specified, 8088-8088, conflicts with rule FirewallRule {... "state":"Add" ...}. - Edit any other rule on the same network. The "refused" rule is now pushed to the router: its firewall rule turns
Activeand HAProxy listens on{ip}:8088(listen ..._8088section inhaproxy.cfg).
Expected Result: when the create is refused, no rule is left behind, as with the older checks: algorithm=foo, protocol=foo and an invalid cidrlist are refused on the same build and leave nothing. Deleting the leftover rule works as a workaround.
What passed verification
- Upgrade: rules created on the old build (http 80, tcp, ssl, tcp-proxy) render the same after the upgrade,
haproxy -cpasses, and traffic through them still works. - SSL offload rule with
keepalive=true idletimeout=300000 keepalivetimeout=10000:option http-keep-alive,timeout http-keep-alive 10000andtimeout client/server 300000in the ssl listen section, the bind line and certificate unchanged, and an HTTP/2 connection is reused. - tcp-proxy rule with keepalive and timeouts: the timeouts apply,
send-proxyis kept and the backend still receives the PROXY header. Keepalive is ignored, with "Keepalive ignored for lb rule ..., it is served in tcp mode" in the agent log. - Port 80 with
keepalive=false keepalivetimeout=7000:option httpclose, and "Keepalive timeout ignored ..., keepalive is off for this rule" in the log. - Health check: a hand-edited timeout on the tcp-proxy rule is reported ("Expected 'timeout client 120000' ... but found 120001"), and a real re-apply fixes it.
- UI edit form: shows the current values, with empty fields as "Use default". Saving sets the values on the rule and the router. Clearing one field resets only that one (the others are kept). "Use default" plus cleared fields puts the rule back to
option httpclose.-5shows "idletimeout must be 0 or greater, got [-5]" and nothing changes. idletimeout=abcis rejected, and another user can't edit the rule.
Minor, not blocking
idletimeout=99999999999is accepted by the API, but the router refuses the config ("timer overflow ... maximum value is 2147483647 ms"), so the user gets only "Failed to update load balancer rule". The old value is kept. A max check in the API would give a clear message.idletimeout=0("no timeout") renderstimeout client 0, andhaproxy -cwarns "missing timeouts ... you will certainly encounter various problems".
Evidence
Lab mapping: {ip} = 10.0.60.223 (public IP of the network, source NAT), {net} = d31838b7-63e1-4345-88d1-cf6a3fde1de9 (isolated network qa13789-net), router r-4-VM, management server ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1.
Management server, steps 1-2: the create with keepalivetimeout=-1 is refused.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule name=qa14131-neg publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 publicport=8088 privateport=80 algorithm=roundrobin keepalivetimeout=-1 filter=id,name
🙈 Error: (HTTP 431, error code 9999) keepalivetimeout must be 0 or greater, got [-1]. 0 means no timeout.
Management server, step 3: the refused rule qa14131-neg exists, and its firewall rule on 8088 is there too.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list loadbalancerrules listall=true id=2a3b5fae-31a1-4590-88de-364ff4489c5c filter=name,publicport,privateport,state,keepalive,keepalivetimeout
{
"count": 1,
"loadbalancerrule": [
{
"name": "qa14131-neg",
"privateport": "80",
"publicport": "8088",
"state": "Add"
}
]
}
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list firewallrules listall=true ipaddressid=b638e93e-2728-42fc-86f2-3e772f0c01e7 filter=startport,state,purpose
{
"count": 5,
"firewallrule": [
{
"startport": 8088,
"state": "Add"
},
{
"startport": 3306,
"state": "Active"
},
{
"startport": 443,
"state": "Active"
},
{
"startport": 2222,
"state": "Active"
},
{
"startport": 80,
"state": "Active"
}
]
}
Management server, steps 2 and 4: the same with idletimeout=-1, then a valid rule on 8088 is refused because of the leftover.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule name=qa14131-neg2 publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 publicport=8089 privateport=80 algorithm=roundrobin idletimeout=-1 filter=id,name
🙈 Error: (HTTP 431, error code 9999) idletimeout must be 0 or greater, got [-1]. 0 means no timeout.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule name=qa14131-retry publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 publicport=8088 privateport=80 algorithm=roundrobin filter=id,name
🙈 Error: (HTTP 537, error code 9999) The range specified, 8088-8088, conflicts with rule FirewallRule {"id":9,"networkId":204,"purpose":"LoadBalancing","state":"Add","uuid":"2a3b5fae-31a1-4590-88de-364ff4489c5c"} which has 8088-8088
Management server and router, step 5: after another rule on the network was edited, the leftover's firewall rule is Active and HAProxy listens on 8088.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list loadbalancerrules listall=true keyword=qa14131-neg filter=id,name,publicport,state
{
"count": 1,
"loadbalancerrule": [
{
"id": "2a3b5fae-31a1-4590-88de-364ff4489c5c",
"name": "qa14131-neg",
"publicport": "8088",
"state": "Add"
}
]
}
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list firewallrules listall=true ipaddressid=b638e93e-2728-42fc-86f2-3e772f0c01e7 filter=startport,state
{
"count": 5,
"firewallrule": [
{
"startport": 8088,
"state": "Active"
},
{
"startport": 3306,
"state": "Active"
},
{
"startport": 443,
"state": "Active"
},
{
"startport": 2222,
"state": "Active"
},
{
"startport": 80,
"state": "Active"
}
]
}
root@r-4-VM:~# sed -n "/^listen 10_0_60_223-8088/,/^\s*$/p" /etc/haproxy/haproxy.cfg
listen 10_0_60_223-8088
bind 10.0.60.223:8088
balance roundrobin
root@r-4-VM:~# ss -ltnp | grep 8088
LISTEN 0 4096 10.0.60.223:8088 0.0.0.0:* users:(("haproxy",pid=9566,fd=9))
Management server: older create checks on the same build refuse and leave nothing behind.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 privateport=80 filter=id,name name=qa14131-old1 publicport=8090 algorithm=foo
🙈 Error: (HTTP 431, error code 9999) Invalid algorithm: foo
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 privateport=80 filter=id,name name=qa14131-old2 publicport=8091 algorithm=roundrobin protocol=foo
🙈 Error: (HTTP 431, error code 9999) protocol foo is not in valid protocols [tcp, udp, tcp-proxy, ssl]
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk create loadbalancerrule publicipid=b638e93e-2728-42fc-86f2-3e772f0c01e7 networkid=d31838b7-63e1-4345-88d1-cf6a3fde1de9 privateport=80 filter=id,name name=qa14131-old3 publicport=8092 algorithm=roundrobin cidrlist=1.2.3.400/33
🙈 Error: (HTTP 431, error code 9999) CIDR [1.2.3.400/33] is invalid.
[root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# for n in qa14131-old1 qa14131-old2 qa14131-old3; do cmk list loadbalancerrules listall=true name=$n filter=name,publicport,state; echo "$n: $(cmk list loadbalancerrules listall=true name=$n | grep -c "\"name\"") rule(s)"; done
qa14131-old1: 0 rule(s)
qa14131-old2: 0 rule(s)
qa14131-old3: 0 rule(s)
createLoadBalancerRule stored the per rule settings only after the rule had been created, so a refused keepalivetimeout or idletimeout left the rule behind in state Add. It held its port, and the next apply on the network pushed it to the router. Check the timeouts first, so a refused create leaves nothing. Also refuse a timeout above 2147483647 ms, which HAProxy rejects as a timer overflow; it used to reach the router and fail the whole apply. Signed-off-by: Brad House <bhouse@nexthop.ai>
|
@RosiKyu thanks for the thorough test, and for the clear repro. You're right about the leftover rule. Fixed in 8d3ec34. The leftover rule.
|
Description
#12586 made the load balancer idle timeout globally configurable, as
network.loadbalancer.haproxy.idle.timeout. This expands that to per load balancer rule, anddoes the same for keepalive, which until now could only be set on the network offering.
Where these settings live today:
keepaliveenabled, create-time only, no UIoption httpcloseindefaultsnetwork.loadbalancer.haproxy.idle.timeout(#12574, #12586)timeout client,timeout serverindefaultsBoth land in the
defaultssection, so they apply to every LB rule on every network using thatoffering. One rule cannot differ from another on the same router.
New parameters on
createLoadBalancerRuleandupdateLoadBalancerRuleEach is optional, and each writes into that rule's own
listensection, which overridesdefaults:keepaliveoption http-keep-alive, elseoption httpclosekeepaliveenabledidletimeouttimeout clientandtimeout serverkeepalivetimeouttimeout http-keep-aliveidletimeoutappliesidletimeoutkeeps the global's meaning, including0for infinite.What that produces, taken from the generator - an http rule with all three set, a tcp rule with
only an idle timeout, and a rule that set nothing:
The three values are returned on
listLoadBalancerRules, and are settable on the load balancerrule's edit form in the UI, where blank reads as "Use default".
A negative timeout, or one above 2147483647 ms (the largest HAProxy accepts), is rejected by the API before the rule is created or changed. Haproxy treats one as a fatal parse error, which would
leave every rule on that router serving its previous config, so the generator drops it as well.
Stored as
firewall_rule_details. No schema change.Applies to public LB rules. Application load balancers are created through a different command and
are not covered.
What this makes possible
A rule on port 80 can now have keep-alive and
X-Forwarded-For. It could not before:mode httpConnection: closeper responsemode tcpforwardforneeds http modemode httpA rule that sets nothing keeps today's behaviour exactly, including the fall back to
mode tcpwhen the offering flag is on.
Also fixes three bugs in the same file
1.
option forcecloseis not a valid keywordWith keepalive enabled, a rule that also uses HTTP-based stickiness or SSL offload emits
no option forceclose. HAProxy has rejected that keyword since 2.0, and the system VM is Debian 12(HAProxy 2.6):
The whole file is rejected, so haproxy keeps serving the previous config and the rule silently
never takes effect. Replaced with
option http-keep-alive.2. One rule's settings leak into every rule generated afterwards
globalSectionanddefaultsSectionarestatic String[]. The generator wraps them inArrays.asList(...)and callsset(), which writes through to the arrays, so the valuesurvives the call:
no option httpclosein the defaults for every later rule, any network, any accountidleTimeout> 0idleTimeout== 0The generator now copies into a fresh list, and the sections themselves became
List.of(...), so afuture
set()throws instead of silently corrupting every config the agent builds afterwards.Output is unchanged for a single rule.
3.
appsessionwas removed from HAProxy in 1.6An LB rule with
AppCookiestickiness emittedappsession, which the shipped haproxy refuses:Same blast radius as bug 1 - the file is rejected, so every rule on that router keeps serving its
previous config. The VR advertises
AppCookieas a supported stickiness method, so this isreachable from the API.
Replaced with the documented stick table equivalent:
len <n>stick-table type string len <n> size 10ktimeout <h>expire <h>stick store-response res.cook(<name>)+stick match req.cook(<name>)request-learnstick store-request req.cook(<name>)prefix,modeprefixandmodewere never applied - the directive carrying them was rejected - so no workingbehaviour changes.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
How Has This Been Tested?
HAProxyConfiguratorTestTests run: 17, Failures: 0andLoadBalancingRulesManagerImplTestTests run: 16, Failures: 0.Each bug fix has a case that fails without it:
forceclose, and does emitoption http-keep-aliveno option httpclosebehindAppCookiestickiness emits a stick table and neverappsessionThe rest cover the per-rule settings:
timeout http-keep-aliveis emitted, and only when keepalive is onidletimeoutoverrides per rule on a tcp rule, leaving the defaults section aloneGenerated configs are checked against a real
haproxy -c, which is where the alert text abovecomes from. A file holding an
AppCookierule, an http rule with all three settings, a tcp rulewith a timeout, and a rule with a negative timeout is fatal before these changes and valid
after - the three alerts above become zero.