Repository navigation
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
|
Metal bench (arm64) failed. 3 of 520 measures outside bounds.
Counters per txn
Microbench: 504 of 507 measures within bounds. Hot code per txn
pooler:
sharding-w2:
Logslb, microbench, pooler, sharding-w2, sharding-w8, verdict.json |
Contributor
|
Metal bench (x86) failed. 2 of 520 measures outside bounds.
Counters per txn
Microbench: 505 of 507 measures within bounds. Microbench latency: 6 benches changed more than 3%
Hot code per txn
pooler:
sharding-w2:
Logslb, microbench, pooler, sharding-w2, sharding-w8, verdict.json Compared 0b9dec6 with 61e9a90 on c7i.metal-48xl (on-demand). |
jkaczman
force-pushed
the
jk-cross-shard-joins-core-v2
branch
from
November 19, 2026 13:56
9919c18 to
1b60ebe
Compare
jkaczman
marked this pull request as draft
November 19, 2026 14:35
Base automatically changed from
jk-rails-limit-count-const-return-fix
to
main
November 19, 2026 18:26
jkaczman
force-pushed
the
jk-cross-shard-joins-core-v2
branch
from
November 19, 2026 18:26
1b60ebe to
0b9dec6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Building off of #2042, #2060, #2061, #2064 and #2066, this PR now executes and re-writes JOINs of two sharded tables (if subqueries are set to be rewritten in pgdog.toml)
This is the start of the foundation for cross-shard JOINs (re #94) giving partial support for
JOIN,LEFT JOIN,CROSS JOINandNATURAL JOINSay we have a statement such as
where
ordersis sharded bybuyer_id(the user who bought the order) andusersbyid.Say order 10 is on Shard A (with the buyer of order 10), but the seller of order 10, user 3, is on Shard B.
Previously, we sent the statement to all shards as-is. Neither shard is fundamentally capable of joining order 10 with user 3 as order 10 is on Shard A but user 3 is on Shard B. Therefore, the client didn't receive
(10, 'user3').Now, we first fetch all rows of the table after
JOIN(s) from all shards withSELECT * FROM users s(will change; see future work). Then, we re-write the JOIN to use the fetched rows asunnest(..)arrays (as we do for subqueries in #2003 and #2023):Shard A now is capable of joining the forementioned order 10 with user 3 (from the fetched rows), so the client now correctly receives
(10, 'user3')!If JOINs are on the sharding keys of both tables (e.g.,
ON s.id = o.buyer_id) and both tables are sharded the same way, we correctly don't rewrite these as they can be run as-is (the rows that those JOINs join fundamentally align to the same shard).Some other cases are handled:
JOIN(as Rails writes JOINs) (e.g.,INNER JOIN "users" ON "users"."id" = "orders"."seller_id")INSERT INTO .. SELECT .. JOIN ..WHERE(e.g.,.. JOIN users u ON .. WHERE o.id IN (SELECT order_id FROM packages))Planned future work (for JOINs):
RIGHT JOINandFULL JOIN(sent as-is for now; same as before)LEFT JOINs from tables that aren't sharded (e.g.,FROM states s LEFT JOIN users u ON ..) (sent as-is for now; same as before)Unrelated fixes: