Repository navigation
[experiment] V2 scan with one segment source per partition - #10765
Closed
joseph-isaacs wants to merge 46 commits into
Closed
joseph-isaacs wants to merge 46 commits into
joseph-isaacs wants to merge 46 commits into
Conversation
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> (cherry picked from commit 6e27e30)
The V2 opener gave every partition its own VortexFile, so each partition read and decoded chunks straddling its boundaries and the file's metadata segments on its own. V1 avoids this through the shared layout reader. Keep a weak per-path map of V2 segment sources beside `layout_readers` so concurrent partitions read through one source. TPC-H SF1 q4 went from 196 segment reads (52.4 MB) to 115 (34.8 MB) per query, matching V1. Hot SF10 geomean against V1 moved from 1.01 to about 0.96 on this machine. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit bec59fe)
Prefetch needs the segments a stage will read before the stage runs. Compiling already registers every segment scan and does no IO, so `Scan::segments` compiles a plan over rows under a mask and lists them, and the per-plan `reach` walk that re-derived the same answer goes, with the `Reach` coordinates it carried. `Shares` used the walk to decode a segment once when several splits or readers of one scan read it, and goes with it. The V2 executor runs one split per scan and shares reads in its IO session, so a scan's splits now each read the segments they overlap, and a plan reading one segment through two paths reads it through each. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A `PreparedScan` holds a scan's plans over one file, built once from the footer, and makes its planners; nothing in it touches a segment and no planner plans again. The planners form a chain, each emitting the next through the planning protocol: - `PrunePlanner` proves the filter's zone maps over the scan's rows, prefetching the zone tables together, and emits one chunk planner for the rows no proof excludes. Rows a proof excludes are never planned. - `ChunkPlanner` cuts those rows into chunks and emits a filter or a projection planner per chunk, skipping chunks nothing is selected or kept of. The run drives an emitted chunk ahead of the next, so chunks waiting on reads overlap. - `FilterPlanner` prefetches its chunk's conjunct segments and announces the projection's at its first compute, runs the conjuncts in the adaptive order over the surviving rows, and emits the projection. - `ProjectionPlanner` emits one morsel per piece of the rows. A stage with nothing to do is never made: a filter without proofs starts at the chunk planner, a scan without a filter at the projection. A scan runs as one task owning one planning run, instead of one task and run per split plus a private run for the proofs; `with_concurrency` sets nothing. The IO session keeps every read it starts, so a later stage or morsel of a chunk that reads the segment again shares the read. The segments a stage prefetches come from compiling its plans. DataFusion attaches each partition's rows from the cached footer when it repartitions a file, so the V2 opener reads exactly those rows instead of translating a byte range through the layout. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A scan reads a zoned column through the zoned node's data child and never its zone table, yet the node and its zone table stayed in every optimized data plan. A new `ZonedDataRule` replaces a zoned data plan by its data, so an optimized plan carries a zone table only where a pruning proof reads it. Legacy stats get the same treatment through `LegacyStatsRule`; the rule that dropped legacy stats beneath a zoned plan is subsumed. A zone proof over legacy stats depended on the converted zoned node surviving child optimization. `ExpressionLegacyStatsRule` now converts the legacy child under a proof directly, through the shared `is_zone_proof` check, so the pruning rewrite still meets a zoned plan. With zone tables gone from data plans, the identity projection is optimized and cached like every other projection: chunked columns stay lazy through `Concat::map_chunks`, and a projection-only scan no longer lowers the zone tables it never reads. (cherry picked from commit 91be25d) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
DataFusion opens a file once per partition, so each partition lowered the layout, planned the filter and proved the zone maps on its own. A scan cache now keeps each distinct filter's plans along with the zones their proofs excluded, and the opener keeps one cache per file that concurrent partitions find through a weak handle and that the last scan holding it frees. A prune planner skips the zone maps of a proof an earlier scan of the file already proved: they are not read again. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The stream a scan returns now owns the planning run and drives it as it is polled. Every CPU step of the run, the planners' and morsels' compute, is a `spawn_cpu` task; every read is already a `spawn_io` task of the segment source; the stream's poll awaits one or the other and does no scan work on the engine's thread. The scan advances only as fast as its consumer pulls, so the task and channel that ran ahead of it go. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A prune planner no longer knows about chunks or selections. It is built with the planner it continues into, as a function of the rows its proofs keep, and emits that planner with the kept mask once every proof has run. `PreparedScan::planner` supplies the chunk planner. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…h other A planner may now emit a child with `PlannerOutput::Spawn`: the run does not queue it but hands it to its owner as `Progress::Spawn`, and the planner continues at once. The owner runs it anywhere; the blocking driver admits it as another root of the same run. The chunk planner spawns every chunk. The scan's stream runs each on a run and a session of its own, stepped on `spawn_cpu`, up to `concurrency` chunks per worker thread at once, merging their batches as they finish. A scan with fewer engine partitions than cores had lost the parallelism of one task per split, 2 to 4 times slower on selective queries with one partition; it is back, without a planner knowing about threads. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Hints were deduplicated by scanning the batch for each segment, which is quadratic in a chunk's segments: a wide projection over a filtered chunk announces hundreds. A set of the segments seen replaces the scan. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…es not A spawned chunk runs on one task of the runtime that drives its run to the end and awaits only its reads, instead of a `spawn_cpu` task per step: a chunk's run waits for reads many times, and a hop at each wait was latency on its critical path that no other chunk's work filled. At a concurrency of one, chunks run on the scan's own run as further roots: the run's steps already interleave their work, and a run, a session and a task per chunk only added a hand-off per chunk, measured at 2 to 3 percent on TPC-H, for parallelism the engine provides. DataFusion sets the concurrency, unless configured, to the cores per partition, so a scan over one partition per core runs its chunks inline and a single-partition scan runs them beside each other. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A planner says what it needs, compute or reads. Where either runs is an engine's decision, so this crate stops driving: `ScanBuilder::prepare` lowers and plans a scan without IO into a `PreparedScan`, which exposes its rows, selection, filter plans and IO service, and `PrunePlanner` and `ChunkPlanner` are public for an engine to compose, the prune planner continuing into whatever it is built with and passing `None` when no proof ran. The builder's ordering, concurrency, mapping and streams go with the driver, and the layout's tests drive their scans on a run of their own. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener composes a scan's planners, a prune planner into a chunk planner when the scan has a filter and the chunk planner alone when it does not, and a driver of this crate runs them: the root run stepped on `spawn_cpu` tasks as DataFusion polls the stream, each spawned chunk on a task of its own up to the concurrency, or on the root run when the engine's partitions already fill the cores, with the concurrency derived from cores per partition unless configured. Batches are mapped to Arrow on the task that produced them. Every spawning decision of a V2 scan under DataFusion is in this crate. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`vortex_file::v2::ScanDriver` composes a scan's planners and runs them on the session's runtime, the way an engine's driver does, for the file API's callers: the compress and random-access benchmarks move to it. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener prepares the scan where compute runs, composes its planners itself, a prune planner continuing into a chunk planner when there is a filter and the chunk planner alone when there is not, and hands the driver the first planner. The driver only drives. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The prune planner skipped prefetching the zone maps of proofs an earlier scan of the file had proved. The segment list now comes from compiling the proof, and a proved proof compiles to its kept result without a segment scan, so the check repeated what compile already knew. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The scan code named the filter's zone-map plans proofs. They are the filter's pruning expressions evaluated over the file's zone map, so the layout, file and DataFusion code now says pruning, matching prune_zones, PrunedZones and the pruning field they already used. Behaviour is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…steps A planner that waits can now say what it will read later: the run asks every parked planner to speculate when nothing else can compute, and registers the answer with its root's session as reads that are never delivered. The prune planner speculates the predicate columns of the next chunk of its partition on each ask, so they are read while the zone maps are, as its filter planner would read them once pruning keeps the chunk; a scan selecting rows by index speculates nothing. The segment IO service keeps speculated reads for whichever session fetches a segment first, so a chunk on its own session takes the read the prune planner started. The root run's CPU step now runs until the run has a batch, waits, or is done, collecting every chunk it spawns, instead of one spawn_cpu hop per chunk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The prune planner ran its pruning plans one after another, each over the rows the previous one kept, so a filter with three plans waited for three reads in turn, each wait costing a round trip and a hop of the root run. It now builds every plan's adaptor in its first compute, publishes all their zone-map reads as one batch, waits once, and intersects the kept rows. Each adaptor numbers its requests from its own range, so the planner routes deliveries by request id. The separate compile of each plan to prefetch its segments is gone: the adaptors' fetches are the reads. Debug events mark when the pruning reads go out, when a chunk's reads are speculated, and when a chunk's run starts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The root run was stepped on a spawn_cpu task per step, so a partition paid a hop to take its zone-map completion, another to step the run, and a task spawn for its chunk, each a few hundred microseconds when every core is busy. The root run now runs on one task that awaits its reads in place, like a chunk's, and the first chunk a partition spawns runs on that run as another root. Later chunks are spawned as before, so one partition with many chunks stays parallel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Planning cuts a partition's rows from the footer's row count on the writer's 8192-row grid, but the writer buffers chunks to megabytes, so at fourteen partitions nearly every boundary fell inside a chunk and both neighbours decoded it: q19 at SF1 retired 17% more instructions than the stacked V2 scan, which aligned partitions to natural splits, and none more at one partition. The opener now snaps the planning-time rows up to the natural split boundaries of the columns the scan reads, computed once per file as the byte-range fallback already does, so the partitions still tile the file and no chunk is decoded twice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…reads Speculation prefetched only the predicate columns of the next chunk, so the projection columns interleaved between them in the file were read as gap bytes and then read again on their own: q6 at SF1 issued 23 physical reads for 164 requests where the stacked scan issued 14, and 14% more bytes. The prune planner now speculates the batch the chunk's filter planner would publish, predicate prefetches and projection announcements together, and the segment IO service keeps speculated announcements, requested and never polled, for whichever session fetches the segment first. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…erlap The session polled a fetch as it was submitted, which marked it wanted and let the read driver dispatch the zone map alone before the run had speculated the chunk's reads microseconds later: q6 at SF1 still issued 25 physical reads for the stacked scan's 14. The run now speculates before it polls its sources, and the session registers a fetch on submit and polls it when the run polls, so what a planner speculates is registered beside the reads it waits for and a coalescing source folds them into one read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A chunk's filter planner announced its projection through its own session, which registered a second request for a segment speculation had already announced beside the chunk's reads. The second request was dispatched alone when the projection fetched, after the chunk's read had gone: two of each partition's eleven requests came back as a separate read inside the same range. A session announcement now takes the speculated announcement, and a fetch prefers it too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A chunk straddling two partitions is fetched by both. A partition's chunk took its speculated read out of the service's map, so when the neighbouring partition speculated the same segment a moment later it started another read, and once the first had completed the file's shared source no longer folded them: on q6 at SF1, 26 segments were read twice. A speculated read now stays in the service's map for as long as the service lives, cloned by every session that fetches it. Trace events name each new segment request by the path that made it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Filtered chunks were cut on a fixed 1M-row grid over the file, so a partition of 430K rows straddling a grid line got a small second chunk, speculated only on the next idle wait, after its neighbour's read had gone: on q6 at SF1 those tails came back as separate reads of ranges already read. Chunks now start at the partition's own first row, so a partition shorter than a chunk is one chunk and its reads go out together. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
One chunk per idle wait left a partition of five chunks at SF10 reading its last three only once their tasks started, where the stacked scan had every split prefetching from the start. The prune planner now speculates every chunk's predicate prefetches and projection announcements in one batch on the first ask, registered together so the partition's reads coalesce as one scan's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Two chunks of one partition can straddle a writer chunk. The first chunk's session took the speculated announcement for it and read it; the second found nothing speculated, announced a request of its own and read the segment again at the end of the scan: q12 at SF10 issued 30 single-request reads of ranges already read. A speculated announcement now stays with the service, a session's own announcement of the segment adds nothing, and the first fetch turns it into the shared read every later session finds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A projection-only partition of seventeen 64K-row chunks handed each chunk through the channel on its own, each needing a poll, a hop and a task spawn, so its chunks started their first reads a few hundred microseconds apart and the last ones no longer coalesced: q12 at SF10 issued 27 single-request reads of the orders table at the end of the scan, where the stacked scan spawned every split of a partition within microseconds. The root task now collects every chunk the run spawns until it has a batch, waits, or finishes, and sends them as one group the consumer spawns at once. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ments A read only announced is not started until the projection fetches it, after the filter, with nothing to overlap, and in a file whose columns are written in bursts a wide projection column shares no read with the narrow predicate beside it. On q14 at SF1 the projection waited 2.4 to 4 ms on reads of its own while its CPU was under 0.1 ms, the whole gap to V1, which polls all of a split's reads at open. The prune planner now speculates the projection's prefetches after the predicates' in the same batch, and the session registers a batch's announcements before it starts any read, so what is announced is dispatched with the reads it coalesces with. Debug events mark the projection's start, reads and finish, and each batch handed to the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A filtered partition of fewer rows than a chunk ran as one chunk, on one task from its first read to its last batch, and an engine's partitions started together straggled on uneven cores: identical 430K-row filter chunks took 0.5 to 5 ms at SF1, and the query waited for the slowest while V1's finer futures were stolen by the idle cores. A filtered chunk is now a quarter of the partition's rows on the row grid, between 256K and 1M rows, so a partition's chunks after its first are spawned where the runtime has room. SF1 partitions get two chunks; SF10 partitions keep their 1M-row chunks. A fixed 256K cost SF10 q1 8%. Local, 14 partitions, against the previous head: SF1 0.993 median (q15 0.976, q19 0.982), SF10 1.008. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Interning hashes an expression's whole tree, and under a mutex the partitions opening a file together serialized on it: 3 ms of lock spinning over 15 runs of q15. Readers probe concurrently; only a miss takes the write lock, and checks again under it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
… beside A zone-map fetch is the read everything else in a partition waits on, yet the file's read driver coalesced it with the chunk reads speculated beside it, so pruning waited for a 3 MB read from the page cache and, over an object store, for a 9.5 MB one behind 150 others. An urgent intent reads a segment on its own as soon as it is published, coalesced only with other urgent reads: a segment source may request a segment alone, the read driver keeps such requests apart, and the prune planner's adaptors publish their reads urgent. Over MinIO with 20 ms of round trip, SF10 q6 pruning finished at 25 to 50 ms instead of 320 to 390. Local: neutral (SF1 0.996, SF10 1.005 against the previous commit). MinIO with 20 ms round trip and 100 MB/s per connection: SF1 0.60 of V1, SF10 1.2 (the SF10 loss is addressed next). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A partition speculated every chunk's reads at once, and the file's read driver dispatches in registration order, so with an engine's partitions started together the last partitions' first chunks queued behind the first partitions' last: over MinIO at SF10 every partition's zone maps and first batches came at once, and the batches of the rest tailed for 300 ms while V1, reading one split at a time per partition, shared the bandwidth evenly. Each partition now reads ahead only its first chunks when the store is not a local file, two by default; the later chunks fetch their own reads as their tasks start, interleaved across partitions. A local file keeps reading every chunk ahead: reading two ahead there cost SF10 6%. MinIO, 20 ms round trip, 100 MB/s per connection, 10 partitions at SF10: 0.90 of V1 at the median (q1 0.79, q19 0.66, q6 1.23 bimodal), from 1.27 with everything read ahead; SF1 at 14 partitions 0.61 of V1. Local unchanged (SF1 1.001, SF10 1.001). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A V2 scan cache lived only as long as the scans of one query holding it, so every query lowered the file's layout, optimized its projection and filter plans, proved its zones and decoded its dictionary values again: q2 at SF1 ran 35 prepares a query, 1.6 ms of CPU, with the first per table on the critical path of a five-table join chain. The cache now lives beside the footer in DataFusion's session-level file metadata cache, so a later query of the file plans nothing again; partitions opening a file the cache misses still share one cache, put with the entry the first of them writes, and a rewritten file invalidates both. The entry's reported size does not count the cache, which grows after the entry is put. Local, 14 partitions, against the previous commit: SF1 0.965 (q19 0.90, q6 0.93, q2 0.96, q13 0.96), SF10 0.962 (q6 0.95, q12 0.95, q19 0.94). Against V1: SF1 0.955, SF10 0.81 on the targeted sets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
… four and a sliver A SF10 orders partition of 1,064,960 rows was cut into four 262,144-row chunks and one of 16,384 rows, each sliver with its own reads, filter evaluation and batches. Rounding the quarter up to the row grid cuts it into three chunks of 270,336 rows and one of 253,952. The prepare diagnostic now reports the chunk rows a scan was given. Timing is neutral on TPC-H SF10 (q1, q3, q10, q12 within noise of the previous rule). A byte-aware rule that cut q13's wide o_comment predicate into 106,496-row chunks was tried and measured within the 2% noise floor of the old rule across four sessions, so it was not kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The run clears a root's session when the root finishes. The first chunk a partition spawns, and every chunk when the driver runs chunks in the partition's own run, were admitted with the root planner's session, so a root finishing dropped the in-flight reads of the chunks still running: with the scan concurrency at zero every TPC-H query failed with "The split's driver is waiting with no reads in flight". Each admitted root now gets a session from the scan's IO service, as a chunk on a task of its own already did. A zero scan concurrency, which for the V2 driver means chunks run in the partition's own run, no longer reaches the V1 builder's assertion that it is above zero. The regression test reads a wide column from a throttled store whose reads take longer the larger they are, so a chunk's reads are still in flight when another chunk finishes; it fails with that error before this change. Running chunks in the partition's own run is not faster: against V1 at SF1 it measured q8 0.966 (default 0.992), q9 1.015 (0.999), q21 1.014 (1.025), q6 0.812 (0.847), q1 0.960 (0.986), and at SF10 q13 1.052 (1.055), q22 1.023 (1.072), q6 0.825 (0.778), q1 0.857 (0.857), so the default stays. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
… fast mode A V2 scan's burst of reads opens a few hundred connections to the store (V1 about 160), and with object_store's unbounded idle pool every later query reused them all and alternated between a fast and a slow mode: q6 at SF10 on MinIO behind toxiproxy (20 ms RTT, 4 ms jitter, 100 MB/s a connection, 10 partitions) ran 404, 547, 268, 506, 269, 273 ms. With the idle pool bounded to 32 connections about 85 are in use and every query is fast: q6 235 ms against V1's 340, q1 420 against 653, q12 293 against 536. V1 is unchanged by the bound. The same key set through `AWS_POOL_MAX_IDLE_PER_HOST` still overrides it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A file's DataFusion partitions share one FileSegmentSource, so one read driver with at most 32 reads in flight serves every partition of the file. On the 96-core CI runner the full branch measures 1.003 of V1 at SF1 while 14 local cores measure 0.963 and 96 time-shared local partitions 0.876: the suspect is that shared driver. This builds the V2 scan over the file's own segment source per partition, as V1 does, to measure it on CI. Not for merging. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
action/bench-sql
joseph-isaacs
added
action/bench-sql
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.955x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.029x ➖, 1↑ 3↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 3↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.984x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.838x ➖, 6↑ 3↓)
datafusion / parquet / ns (0.941x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.943x ➖, 1↑ 2↓)
|
Contributor
Author
|
Result (manual dispatch of the SQL benchmark, run 40047049933, vs develop 31027fc): TPC-H SF1 NVMe hot 1.018 (full branch #10694: 1.003), SF10 NVMe 0.995 (0.987), SF1 on S3 0.913 (0.780). One segment source per partition does not close the SF1 gap on the 96-core runner and costs most of the S3 win, which comes from sharing reads across partitions. Hypothesis refuted; the shared source stays. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI experiment, not for merging. Measures whether the one
FileSegmentSource(and so one read driver, at most 32 reads in flight) that V2 shares across a file's DataFusion partitions is why the full branch (#10694) is 1.003 of V1 at SF1 on the 96-core runner while 14 local cores measure 0.963 and 96 time-shared local partitions 0.876. This head is #10694 plus one commit giving each partition the file's own segment source, as V1 has. Compare its TPC-H SF1 / SF10 NVMe comments with #10694's (SF1 hot 1.003, SF10 0.987).🤖 Generated with Claude Code
https://claude.ai/code/session_01Vr65ZwPZz4LPjkvDA6ZUuE