Skip to content

feat: support two-table cross-shard JOINs (JOIN, LEFT JOIN, CROSS JOIN, NATURAL JOIN) - #2070

Draft
jkaczman wants to merge 1 commit into
mainfrom
jk-cross-shard-joins-core-v2
Draft

jkaczman wants to merge 1 commit into
mainfrom
jk-cross-shard-joins-core-v2

Conversation

@jkaczman

Copy link
Copy Markdown
Contributor

Building off of #2042, #2060, #2061, #2064 and #2066, this PR now executes and re-writes JOINs of two sharded tables (if subqueries are set to be rewritten in pgdog.toml)

This is the start of the foundation for cross-shard JOINs (re #94) giving partial support for JOIN, LEFT JOIN, CROSS JOIN and NATURAL JOIN

Say we have a statement such as

SELECT o.id, s.username FROM orders o JOIN users s ON s.id = o.seller_id

where orders is sharded by buyer_id (the user who bought the order) and users by id.

Say order 10 is on Shard A (with the buyer of order 10), but the seller of order 10, user 3, is on Shard B.

Previously, we sent the statement to all shards as-is. Neither shard is fundamentally capable of joining order 10 with user 3 as order 10 is on Shard A but user 3 is on Shard B. Therefore, the client didn't receive (10, 'user3').

Now, we first fetch all rows of the table after JOIN (s) from all shards with SELECT * FROM users s (will change; see future work). Then, we re-write the JOIN to use the fetched rows as unnest(..) arrays (as we do for subqueries in #2003 and #2023):

SELECT o.id, s.username FROM orders o JOIN (SELECT u.c1, u.c2, u.c3 FROM unnest($1::pg_catalog._int8, $2::pg_catalog._text, $3::pg_catalog._text) u(c1, c2, c3)) s(id, username, status) ON s.id = o.seller_id

Shard A now is capable of joining the forementioned order 10 with user 3 (from the fetched rows), so the client now correctly receives (10, 'user3')!

If JOINs are on the sharding keys of both tables (e.g., ON s.id = o.buyer_id) and both tables are sharded the same way, we correctly don't rewrite these as they can be run as-is (the rows that those JOINs join fundamentally align to the same shard).

Some other cases are handled:

  • Tables without aliases after JOIN (as Rails writes JOINs) (e.g., INNER JOIN "users" ON "users"."id" = "orders"."seller_id")
  • INSERT INTO .. SELECT .. JOIN ..
  • JOINs next to cross-shard subqueries in WHERE (e.g., .. JOIN users u ON .. WHERE o.id IN (SELECT order_id FROM packages))

Planned future work (for JOINs):

  • Three or more tables, RIGHT JOIN and FULL JOIN (sent as-is for now; same as before)
  • LEFT JOINs from tables that aren't sharded (e.g., FROM states s LEFT JOIN users u ON ..) (sent as-is for now; same as before)
  • Fetching only the columns and rows that statements actually use instead of whole tables
  • Returning errors if the fetched rows are larger than a limit (e.g., 100 MB)

Unrelated fixes:

  • Errors that Postgres returns for fetches are now returned to the client instead of closing the client's connection

@jkaczman
jkaczman added this pull request to stack #2041 November 19, 2026 12:32
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Metal bench (arm64) failed. 3 of 520 measures outside bounds.

suite throughput cpu/txn user cycles/txn user instr/txn kernel cycles/txn kernel instr/txn ctx/txn
pooler 19.04 (+0.1% ± 0.8%) 18,345 (+0.6% ± 0.5%) 31,429 (+0.0% ± 0.2%) 30,439 (-0.2% ± 1.5%) 51,064 (-0.1% ± 0.4%)
lb 19.53 (+0.3% ± 0.2%) 18,803 (+0.9% ± 1.2%) 32,326 (-0.1% ± 0.1%) 30,930 (-0.0% ± 1.0%) 52,175 (-0.1% ± 0.5%)
sharding-w2 34.49 (+2.2% ± 0.7%) 46,548 (+1.8% ± 1.4%) 94,982 (+0.0% ± 0.02%) 44,086 (+4.1% ± 2.9%) 53,628 (-0.1% ± 0.3%) 1.534 (-0.6% ± 0.5%)
sharding-w8 237,441 (-1.1% ± 1.3%) 30.29 (+1.4% ± 1.4%) 45,994 (+0.9% ± 1.5%) 50,572 (+0.1% ± 0.05%) 37,709 (+2.1% ± 1.4%) 42,403 (-0.0% ± 0.1%) 0.1614 (-2.2% ± 1.9%)
Counters per txn
suite frontend stall l1i refill backend stall
pooler 7,940 (+1.1% ± 1.1%) 2,727 (+0.7% ± 0.6%)
lb 8,113 (+1.8% ± 2.9%) 2,758 (+1.0% ± 1.1%)
sharding-w2 21,536 (+5.0% ± 4.0%) 4,500 (+2.1% ± 2.9%)
sharding-w8 22,732 (+3.2% ± 4.5%) 4,360 (+0.8% ± 2.3%)
benchmark measure result baseline change bound
full_extended_cached::aggregate instructions 145,762 144,115 +1.14% ± 0.00% +1%
full_extended_cached::cte instructions 101,085 99,590 +1.50% ± 0.00% +1%
full_named_cached::aggregate instructions 137,191 134,706 +1.84% ± 0.00% +1%

Microbench: 504 of 507 measures within bounds.

Hot code per txn
path base head change
pooler 115.8 KB 114.3 KB -1.5 KB
sharding-w2 168.9 KB 169.9 KB +1.0 KB

pooler:

crate base head change
pgdog::frontend::client::query_engine 20.3 KB 20.2 KB -0.1 KB
tokio 18.6 KB 18.3 KB -0.2 KB
pgdog::backend::pool 17.7 KB 17.4 KB -0.2 KB
aws-lc (TLS crypto) 10.4 KB 10.2 KB -0.2 KB
rustls (TLS) 8.4 KB 8.4 KB +0.0 KB
pgdog::frontend (other) 8.0 KB 8.0 KB +0.0 KB
pgdog::backend::server 7.8 KB 7.9 KB +0.1 KB
other 6.6 KB 6.3 KB -0.2 KB
everything else 18.1 KB 17.5 KB -0.6 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 5.7 KB +0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 5.4 KB +0.0 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.3 KB +0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 3.4 KB +0.0 KB
tokio::runtime::task::raw::poll 3.2 KB +0.1 KB

sharding-w2:

crate base head change
pgdog::frontend::router 29.2 KB 29.2 KB -0.1 KB
pgdog::frontend::client::query_engine 21.6 KB 21.6 KB +0.0 KB
tokio 18.5 KB 18.9 KB +0.4 KB
pgdog::backend::pool 18.1 KB 18.2 KB +0.2 KB
aws-lc (TLS crypto) 13.3 KB 13.4 KB +0.1 KB
other 10.3 KB 10.1 KB -0.2 KB
pgdog::frontend (other) 8.9 KB 8.9 KB +0.0 KB
rustls (TLS) 8.2 KB 8.2 KB -0.1 KB
everything else 40.8 KB 41.4 KB +0.6 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 6.3 KB +0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 5.2 KB -0.1 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.1 KB +0.0 KB
pgdog::frontend::router::parser::query::select::<impl pgdog::frontend::router::parser::que 3.8 KB -0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 3.3 KB -0.1 KB
lb flamegraph vs 61e9a90

lb

Open interactive for hover, zoom and search.

lb removed

Open interactive.

microbench flamegraph vs 61e9a90

microbench

Open interactive for hover, zoom and search.

microbench removed

Open interactive.

pooler flamegraph vs 61e9a90

pooler

Open interactive for hover, zoom and search.

pooler removed

Open interactive.

sharding-w2 flamegraph vs 61e9a90

sharding-w2

Open interactive for hover, zoom and search.

sharding-w2 removed

Open interactive.

sharding-w8 flamegraph vs 61e9a90

sharding-w8

Open interactive for hover, zoom and search.

sharding-w8 removed

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 0b9dec6 with 61e9a90 on m8gd.metal-24xl (spot).

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Metal bench (x86) failed. 2 of 520 measures outside bounds.

suite throughput cpu/txn user cycles/txn user instr/txn kernel cycles/txn kernel instr/txn ctx/txn
pooler 25.8 (+0.1% ± 0.3%) 18,685 (+0.5% ± 0.2%) 32,752 (-0.0% ± 0.03%) 33,577 (-0.1% ± 0.8%) 36,478 (-0.0% ± 0.6%)
lb 26.17 (+0.0% ± 0.3%) 19,172 (-0.0% ± 0.5%) 33,627 (-0.3% ± 0.9%) 33,691 (+0.1% ± 0.5%) 37,332 (+0.3% ± 0.4%)
sharding-w2 42.15 (+0.2% ± 1.4%) 47,346 (+0.3% ± 0.7%) 98,722 (+0.0% ± 0.01%) 38,615 (-0.2% ± 2.6%) 40,132 (-0.0% ± 0.2%) 1.525 (-0.1% ± 0.3%)
sharding-w8 205,422 (-0.0% ± 1.0%) 33.14 (+0.1% ± 0.2%) 46,393 (+0.2% ± 0.5%) 53,069 (-0.0% ± 0.07%) 31,437 (-0.0% ± 0.1%) 32,510 (+0.0% ± 0.3%) 0.1717 (-0.1% ± 3.1%)
Counters per txn
suite frontend stall l1i refill backend stall
pooler 3,184 (+1.0% ± 0.9%) 1,081 (+0.6% ± 1.0%) 40,422 (+0.8% ± 0.5%)
lb 3,307 (-2.7% ± 0.5%) 1,105 (-2.8% ± 0.9%) 41,240 (+1.9% ± 0.4%)
sharding-w2 6,250 (-0.9% ± 1.9%) 1,486 (-2.5% ± 1.8%) 116,867 (+1.3% ± 1.9%)
sharding-w8 3,476 (-0.6% ± 1.6%) 850 (-0.7% ± 1.1%) 179,906 (+0.4% ± 0.9%)
benchmark measure result baseline change bound
full_named_cached::aggregate instructions 143,612 141,384 +1.58% ± 0.00% +1%
full_named_cached::delete_by_key instructions 63,426 62,195 +1.98% ± 0.00% +1%

Microbench: 505 of 507 measures within bounds.

Microbench latency: 6 benches changed more than 3%
benchmark latency baseline change instructions change
extended_cached::group_by 3,004 ns 2,795 ns +7.48% ± 0.55% +0.10% ± 0.00%
extended_cached::having 3,234 ns 3,012 ns +7.38% ± 1.01% +0.09% ± 0.00%
extended_cached::having_distinct_limit 3,587 ns 3,388 ns +5.88% ± 0.50% +0.08% ± 0.00%
extended_evicting::group_by 7,116 ns 6,753 ns +5.38% ± 0.26% +0.05% ± 0.00%
extended_cold::group_by 6,900 ns 6,582 ns +4.84% ± 0.60% +0.05% ± 0.00%
simple_cold::group_by 6,701 ns 6,396 ns +4.78% ± 0.67% +0.05% ± 0.00%
Hot code per txn
path base head change
pooler 116.3 KB 115.1 KB -1.2 KB
sharding-w2 171.0 KB 170.6 KB -0.4 KB

pooler:

crate base head change
pgdog::frontend::client::query_engine 22.8 KB 22.8 KB -0.1 KB
pgdog::backend::pool 18.3 KB 18.4 KB +0.1 KB
tokio 18.6 KB 18.4 KB -0.2 KB
pgdog::frontend (other) 9.6 KB 9.6 KB -0.1 KB
aws-lc (TLS crypto) 8.8 KB 8.6 KB -0.1 KB
pgdog::backend::server 8.2 KB 8.0 KB -0.2 KB
rustls (TLS) 7.8 KB 7.8 KB +0.0 KB
other 6.1 KB 5.8 KB -0.3 KB
everything else 16.2 KB 15.8 KB -0.4 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 7.3 KB -0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 6.6 KB +0.1 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.9 KB -0.1 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 3.9 KB -0.1 KB
tokio::runtime::task::raw::poll 3.5 KB -0.1 KB

sharding-w2:

crate base head change
pgdog::frontend::router 28.8 KB 28.8 KB +0.1 KB
pgdog::frontend::client::query_engine 23.8 KB 23.7 KB -0.1 KB
pgdog::backend::pool 18.9 KB 18.9 KB -0.1 KB
tokio 18.1 KB 18.4 KB +0.3 KB
other 11.4 KB 10.8 KB -0.7 KB
pgdog::frontend (other) 10.8 KB 10.8 KB +0.0 KB
aws-lc (TLS crypto) 10.1 KB 10.6 KB +0.4 KB
pgdog::backend::server 8.8 KB 8.8 KB -0.1 KB
everything else 40.4 KB 40.1 KB -0.3 KB
function head change
pgdog::frontend::client::Client::run::{{closure}} 8.1 KB +0.0 KB
pgdog::frontend::client::query_engine::query::<impl pgdog::frontend::client::query_engine: 6.2 KB -0.1 KB
pgdog::frontend::router::parser::query::select::<impl pgdog::frontend::router::parser::que 4.9 KB +0.1 KB
pgdog::frontend::client::query_engine::QueryEngine::handle::{{closure}} 4.6 KB -0.1 KB
tokio::runtime::task::raw::poll 3.9 KB +0.1 KB
lb flamegraph vs 61e9a90

lb

Open interactive for hover, zoom and search.

lb removed

Open interactive.

microbench flamegraph vs 61e9a90

microbench

Open interactive for hover, zoom and search.

microbench removed

Open interactive.

pooler flamegraph vs 61e9a90

pooler

Open interactive for hover, zoom and search.

pooler removed

Open interactive.

sharding-w2 flamegraph vs 61e9a90

sharding-w2

Open interactive for hover, zoom and search.

sharding-w2 removed

Open interactive.

sharding-w8 flamegraph vs 61e9a90

sharding-w8

Open interactive for hover, zoom and search.

sharding-w8 removed

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 0b9dec6 with 61e9a90 on c7i.metal-48xl (on-demand).

@jkaczman
jkaczman force-pushed the jk-cross-shard-joins-core-v2 branch from 9919c18 to 1b60ebe Compare November 19, 2026 13:56
@jkaczman
jkaczman marked this pull request as draft November 19, 2026 14:35
Base automatically changed from jk-rails-limit-count-const-return-fix to main November 19, 2026 18:26
@jkaczman
jkaczman force-pushed the jk-cross-shard-joins-core-v2 branch from 1b60ebe to 0b9dec6 Compare November 19, 2026 18:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant