Repository navigation
DRAFT Bench 26264 - #27154
Draft
kosiew wants to merge 4 commits into
Draft
DRAFT Bench 26264#27154kosiew wants to merge 4 commits into
kosiew wants to merge 4 commits into
Conversation
- Four lone-sort configurations; reports spill counts, rows, bytes. - Validates sorted output and reservation cleanup. - Debug/CI-profile smoke, Clippy, formatting pass. - Full suites timed out. Four-partition case excluded: memory exhaustion.
The final merge of a spilled sort seated every run the memory pool would admit, and held those buffers until its output was consumed. Partitions that finished their input early could take the whole pool this way, and a partition still sorting then failed with ResourcesExhausted (apache#25804, finding 9). Seat runs beyond the minimum two only while the pool can admit what the merge takes beyond its own workspace plus as much again, and return the extra before merging, as apache#25383 does for aggregate replay. The minimum merge is admitted as before, so a sort under tight memory keeps its batch sizes and does not re-spill runs to leave headroom.
Check at the builder that runs beyond the minimum two are seated only with equal headroom, and that a headroom request beyond usize::MAX stops seating runs instead of failing the merge.
The fan-in headroom made a lone sort partition that has the pool to itself rewrite runs in an extra merge pass, for no one: with 60 x 8192 rows of Int64 plus a 120-byte string, 1 MB sort_spill_reservation_bytes and a 16 MB GreedyMemoryPool, it wrote 1,589,248 rows to spill instead of 983,040. Keep the headroom only while the sort has other partitions sharing the pool, or another consumer holds memory in it. Sibling partitions count even when they hold nothing at the moment the merge seats its runs. The operator consuming the output of a lone partition is not counted: it reserves nothing before its first batch, and keeping headroom for it would cost the extra pass whether or not it ever needs the memory. With one partition, the case above writes the same rows as without headroom in all four configurations measured. With four partitions in a 32 MB pool and 1 MB reservations, the sort fails with "Not enough memory to continue external sort" without headroom and succeeds with it. Also cover the minimum two runs under a budget below their headroom, and seating against a retained merge workspace on both sides of 2 * (T - W).
Contributor
Author
|
run benchmark sort_spill baseline:
ref: 6194e50c12 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing bench-26264 (217bd1f) to 6194e50 diff Run configurationrun benchmark sort_spill
baseline:
ref: "6194e50c12"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing bench-26264 (217bd1f) to 6194e50 diff Run configurationrun benchmark sort_spill
baseline:
ref: "6194e50c12"CPU Details (lscpu)Details
Resource Usagesort_spill — base (merge-base)
sort_spill — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bench 26264