Skip to content

fix(lb): bannable targets should not affect RR distribution - #2077

Merged
meskill merged 2 commits into
mainfrom
meskill-2026-11-19-fix-lb---vpm
Nov 19, 2026
Merged

meskill merged 2 commits into
mainfrom
meskill-2026-11-19-fix-lb---vpm

Conversation

@meskill

@meskill meskill commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

With round-robin (and weighted-round-robin) strategy if the target is banned the next target will get twice as much load because it will handle both load of itself and banned target.

The fix creates the list of healthy replicas instead that is used to distribute the load.

@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.68847% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pgdog/src/backend/pool/lb/mod.rs 97.22% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

Metal bench (arm64) failed. 5 of 520 measures outside bounds.

suite throughput cpu/txn user cycles/txn user instr/txn kernel cycles/txn kernel instr/txn ctx/txn
pooler 19.63 (-0.7% ± 1.4%) 19,361 (-0.4% ± 0.3%) 31,417 (-0.1% ± 0.05%) 31,237 (-0.8% ± 2.8%) 51,035 (+0.2% ± 0.3%)
lb 20.4 (+0.4% ± 0.9%) 20,378 (+0.9% ± 0.6%) 32,540 (+0.6% ± 0.03%), bound +0.5% 31,991 (+0.2% ± 1.9%) 51,973 (-0.2% ± 0.4%)
sharding-w2 33.62 (+1.2% ± 1.6%) 46,311 (+1.3% ± 2.0%) 95,076 (+0.1% ± 0.00%) 41,352 (+1.0% ± 2.9%) 53,718 (-0.1% ± 0.5%) 1.542 (-0.0% ± 0.1%)
sharding-w8 245,669 (+0.8% ± 1.0%) 29.2 (-0.7% ± 0.8%) 45,684 (+0.2% ± 1.5%) 50,635 (+0.2% ± 0.01%) 35,005 (-1.8% ± 1.5%) 42,360 (-0.0% ± 0.2%) 0.1673 (-0.3% ± 2.9%)
Counters per txn
suite frontend stall l1i refill backend stall
pooler 9,423 (-1.1% ± 0.8%) 2,908 (-0.6% ± 0.5%)
lb 10,166 (+1.1% ± 1.7%) 3,006 (+0.1% ± 0.9%)
sharding-w2 20,882 (+2.4% ± 5.2%) 4,594 (+1.8% ± 3.5%)
sharding-w8 21,558 (-0.6% ± 2.7%) 4,366 (+0.4% ± 2.0%)
benchmark measure result baseline change bound
extended_cached::cte instructions 24,235 23,959 +1.15% ± 0.00% +1%
full_extended_cached::aggregate instructions 146,231 144,115 +1.47% ± 0.00% +1%
full_simple::order_limit_offset instructions 195,476 193,049 +1.26% ± 0.00% +1%
simple_cold::select_omni instructions 20,673 20,461 +1.04% ± 0.00% +1%

Microbench: 503 of 507 measures within bounds.

Hot code per txn
path base head change
pooler 114.2 KB 113.8 KB -0.4 KB
sharding-w2 168.1 KB 168.8 KB +0.7 KB

pooler:

crate base head change
pgdog::frontend::client::query_engine 20.5 KB 20.4 KB -0.1 KB
tokio 18.4 KB 18.4 KB +0.0 KB
pgdog::backend::pool 17.5 KB 17.5 KB +0.0 KB
aws-lc (TLS crypto) 10.1 KB 10.2 KB +0.1 KB
rustls (TLS) 8.4 KB 8.4 KB -0.1 KB
pgdog::frontend (other) 7.9 KB 7.9 KB -0.1 KB
pgdog::backend::server 7.9 KB 7.6 KB -0.2 KB
other 6.0 KB 6.1 KB +0.1 KB
everything else 17.5 KB 17.3 KB -0.2 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 5.6 KB +0.0 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 5.4 KB +0.1 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.2 KB -0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 3.4 KB -0.1 KB
tokio::runtime::task::raw::poll 3.1 KB -0.1 KB

sharding-w2:

crate base head change
pgdog::frontend::router 29.3 KB 29.6 KB +0.3 KB
pgdog::frontend::client::query_engine 21.6 KB 21.8 KB +0.2 KB
tokio 18.6 KB 18.8 KB +0.2 KB
pgdog::backend::pool 18.0 KB 18.6 KB +0.6 KB
aws-lc (TLS crypto) 12.7 KB 12.8 KB +0.1 KB
other 10.0 KB 9.7 KB -0.3 KB
pgdog::frontend (other) 8.9 KB 8.9 KB +0.0 KB
pgdog::backend::server 8.1 KB 8.2 KB +0.1 KB
everything else 41.1 KB 40.5 KB -0.6 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 6.2 KB -0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 5.4 KB +0.1 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.1 KB +0.0 KB
pgdog::frontend::router::parser::query::select::<impl pgdog::frontend::router::parser::que 3.9 KB +0.0 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 3.4 KB +0.1 KB
lb flamegraph vs 61e9a90

lb

Open interactive for hover, zoom and search.

lb removed

Open interactive.

microbench flamegraph vs 61e9a90

microbench

Open interactive for hover, zoom and search.

microbench removed

Open interactive.

pooler flamegraph vs 61e9a90

pooler

Open interactive for hover, zoom and search.

pooler removed

Open interactive.

sharding-w2 flamegraph vs 61e9a90

sharding-w2

Open interactive for hover, zoom and search.

sharding-w2 removed

Open interactive.

sharding-w8 flamegraph vs 61e9a90

sharding-w8

Open interactive for hover, zoom and search.

sharding-w8 removed

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 50078d0 with 61e9a90 on m8g.metal-24xl (spot).

@github-actions

Copy link
Copy Markdown
Contributor

Metal bench (x86) failed. 6 of 520 measures outside bounds.

suite throughput cpu/txn user cycles/txn user instr/txn kernel cycles/txn kernel instr/txn ctx/txn
pooler 35.11 (-0.0% ± 0.2%) 23,756 (-0.1% ± 0.7%) 32,749 (-0.1% ± 0.01%) 58,100 (-0.0% ± 0.3%) 38,515 (-0.0% ± 0.1%)
lb 36.04 (+0.0% ± 0.5%) 24,756 (-0.2% ± 0.8%) 33,944 (+1.0% ± 0.9%), bound +0.5% 58,989 (+0.2% ± 0.4%) 39,171 (-0.0% ± 0.1%)
sharding-w2 56.52 (-0.1% ± 0.5%) 53,358 (-0.2% ± 1.5%) 98,829 (+0.1% ± 0.07%) 72,789 (+0.2% ± 0.7%) 43,072 (-0.0% ± 0.1%) 1.392 (-0.0% ± 0.04%)
sharding-w8 147,927 (-0.3% ± 2.2%) 48.27 (+0.7% ± 0.7%) 57,967 (+0.9% ± 2.7%) 54,263 (+0.2% ± 0.10%) 64,550 (+0.5% ± 1.5%) 34,221 (+0.0% ± 0.2%) 0.1696 (-1.0% ± 4.4%)
Counters per txn
suite frontend stall l1i refill backend stall
pooler 79,268 (-1.3% ± 0.7%) 3,835 (-0.5% ± 0.5%)
lb 81,498 (-1.2% ± 1.0%) 3,963 (+0.5% ± 0.2%)
sharding-w2 163,672 (-1.2% ± 3.3%) 6,229 (-0.4% ± 0.7%)
sharding-w8 124,873 (-1.9% ± 3.0%) 5,088 (-0.1% ± 0.6%)
benchmark measure result baseline change bound
extended_cached::aggregate instructions 17,705 17,437 +1.54% ± 0.00% +1%
extended_evicting::set instructions 15,325 15,051 +1.82% ± 0.00% +1%
full_named_cached::aggregate instructions 142,964 141,385 +1.12% ± 0.00% +1%
full_named_cached::update_by_key instructions 65,027 63,734 +2.03% ± 0.00% +1%
simple_cold::set instructions 13,664 13,387 +2.07% ± 0.00% +1%

Microbench: 502 of 507 measures within bounds.

Microbench latency: 33 benches changed more than 3%
benchmark latency baseline change instructions change
extended_evicting::begin 2,233 ns 2,123 ns +5.18% ± 0.40% +0.00% ± 0.00%
extended_cold::update_all_shards 3,639 ns 3,461 ns +5.14% ± 0.63% +0.00% ± 0.00%
simple_cold::begin 1,955 ns 1,861 ns +5.05% ± 0.76% +0.00% ± 0.00%
simple_cold::set 2,523 ns 2,404 ns +4.97% ± 1.50% +2.07% ± 0.00%
extended_cold::delete_by_key 3,916 ns 3,742 ns +4.65% ± 1.24% +0.00% ± 0.00%
extended_evicting::delete_by_key 4,074 ns 3,896 ns +4.56% ± 0.51% +0.00% ± 0.00%
extended_evicting::insert 6,552 ns 6,270 ns +4.50% ± 0.38% +0.00% ± 0.00%
extended_evicting::select_all_shards 4,285 ns 4,105 ns +4.39% ± 1.18% +0.00% ± 0.00%
extended_evicting::update_all_shards 3,779 ns 3,623 ns +4.30% ± 0.92% +0.00% ± 0.00%
extended_cold::select_by_key 4,947 ns 4,743 ns +4.30% ± 0.50% +0.00% ± 0.00%
extended_cold::set 2,642 ns 2,534 ns +4.27% ± 1.29% +0.00% ± 0.00%
extended_cold::begin 2,083 ns 1,999 ns +4.21% ± 0.75% +0.00% ± 0.00%
extended_cold::select_omni 3,689 ns 3,541 ns +4.18% ± 1.14% +0.00% ± 0.00%
extended_evicting::update_by_key 4,686 ns 4,500 ns +4.13% ± 0.57% +0.01% ± 0.00%
extended_cold::update_by_key 4,522 ns 4,343 ns +4.13% ± 0.55% +0.00% ± 0.00%
extended_evicting::delete_all_shards 3,101 ns 2,979 ns +4.10% ± 0.58% +0.00% ± 0.00%
extended_cold::distinct 4,362 ns 4,194 ns +4.00% ± 0.73% +0.00% ± 0.00%
extended_evicting::set 2,792 ns 2,685 ns +3.98% ± 1.01% +1.82% ± 0.00%
extended_evicting::select_omni 3,895 ns 3,749 ns +3.90% ± 0.88% +0.00% ± 0.00%
extended_cold::select_all_shards 4,068 ns 3,916 ns +3.89% ± 1.16% +0.00% ± 0.00%
simple_cold::update_all_shards 3,497 ns 3,367 ns +3.87% ± 1.40% +0.00% ± 0.00%
simple_cold::delete_all_shards 2,807 ns 2,707 ns +3.69% ± 0.57% +0.00% ± 0.00%
extended_cold::delete_all_shards 2,929 ns 2,826 ns +3.64% ± 0.30% +0.00% ± 0.00%
extended_cold::select_in_list 5,901 ns 5,695 ns +3.62% ± 0.66% +0.00% ± 0.00%
simple_cold::delete_by_key 3,750 ns 3,620 ns +3.59% ± 0.67% +0.00% ± 0.00%
extended_evicting::lookup 5,622 ns 5,430 ns +3.54% ± 0.97% +0.93% ± 0.00%
extended_evicting::select_by_key 5,133 ns 4,962 ns +3.45% ± 1.01% +0.00% ± 0.00%
extended_evicting::insert_returning 7,386 ns 7,140 ns +3.45% ± 0.85% +0.00% ± 0.00%
extended_evicting::distinct 4,517 ns 4,367 ns +3.44% ± 0.89% +0.00% ± 0.00%
extended_cold::subquery 6,050 ns 5,855 ns +3.33% ± 0.46% +0.66% ± 0.00%
extended_cold::insert 6,307 ns 6,109 ns +3.24% ± 0.72% +0.00% ± 0.00%
extended_cold::insert_returning 7,165 ns 6,947 ns +3.14% ± 0.78% +0.33% ± 0.00%
pooler_simple::select_1 7,461 ns 7,697 ns -3.05% ± 2.00% -0.78% ± 0.00%
Hot code per txn
path base head change
pooler 111.9 KB 113.5 KB +1.6 KB
sharding-w2 168.5 KB 168.2 KB -0.2 KB

pooler:

crate base head change
pgdog::frontend::client::query_engine 21.2 KB 22.1 KB +0.9 KB
tokio 17.4 KB 18.0 KB +0.6 KB
pgdog::backend::pool 17.4 KB 16.9 KB -0.5 KB
pgdog::frontend (other) 9.4 KB 9.4 KB +0.0 KB
rustls (TLS) 7.1 KB 7.8 KB +0.7 KB
pgdog::backend::server 7.4 KB 7.6 KB +0.2 KB
aws-lc (TLS crypto) 6.6 KB 7.1 KB +0.5 KB
jemalloc 7.2 KB 6.8 KB -0.4 KB
everything else 18.1 KB 17.8 KB -0.3 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 7.1 KB +0.0 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 6.4 KB +0.1 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.6 KB +0.4 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 3.8 KB +0.2 KB
tokio::runtime::task::raw::poll 3.7 KB +0.1 KB

sharding-w2:

crate base head change
pgdog::frontend::router 27.8 KB 27.1 KB -0.8 KB
pgdog::frontend::client::query_engine 23.9 KB 23.4 KB -0.5 KB
pgdog::backend::pool 17.8 KB 17.9 KB +0.1 KB
tokio 18.1 KB 17.6 KB -0.5 KB
other 10.8 KB 11.2 KB +0.5 KB
jemalloc 9.8 KB 10.4 KB +0.6 KB
aws-lc (TLS crypto) 10.2 KB 10.1 KB -0.1 KB
pgdog::frontend (other) 10.5 KB 10.1 KB -0.4 KB
everything else 39.6 KB 40.4 KB +0.8 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 7.6 KB -0.2 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 6.1 KB -0.2 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.6 KB -0.1 KB
pgdog::frontend::router::parser::query::select::<impl pgdog::frontend::router::parser::que 4.5 KB +0.1 KB
tokio::runtime::task::raw::poll 3.8 KB -0.1 KB
lb flamegraph vs 61e9a90

lb

Open interactive for hover, zoom and search.

lb removed

Open interactive.

microbench flamegraph vs 61e9a90

microbench

Open interactive for hover, zoom and search.

microbench removed

Open interactive.

pooler flamegraph vs 61e9a90

pooler

Open interactive for hover, zoom and search.

pooler removed

Open interactive.

sharding-w2 flamegraph vs 61e9a90

sharding-w2

Open interactive for hover, zoom and search.

sharding-w2 removed

Open interactive.

sharding-w8 flamegraph vs 61e9a90

sharding-w8

Open interactive for hover, zoom and search.

sharding-w8 removed

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 50078d0 with 61e9a90 on c7a.metal-48xl (spot).

@meskill
meskill merged commit 5bc4e1c into main Nov 19, 2026
50 of 52 checks passed
@meskill
meskill deleted the meskill-2026-11-19-fix-lb---vpm branch November 19, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants