Repository navigation

[experiment] V2 scan with one segment source per partition - #10765

Closed

joseph-isaacs wants to merge 46 commits into

developfrom

ji/scan-v2-exp-noshare

Closed

[experiment] V2 scan with one segment source per partition#10765
joseph-isaacs wants to merge 46 commits into
developfrom
ji/scan-v2-exp-noshare

Commits on Oct 22, 2026
- Add a switchable planner-driven scan v2 path for DataFusion
  joseph-isaacs
  committed
  8b80703
- Share one V2 segment source per file across DataFusion partitions
  
  joseph-isaacs
  and
  claude
  committed
  6b7b44d
- List a compiled scan's segments and drop the reach walk
  
  joseph-isaacs
  and
  claude
  committed
  3ba184e
- Plan a V2 scan as a chain of planners from one prepared scan
  
  joseph-isaacs
  and
  claude
  committed
  3aa00b8
- Optimize a zoned data plan to its data child
  
  joseph-isaacs
  and
  claude
  committed
  8b17f7b
- Share one V2 scan cache per file across DataFusion partitions
  
  joseph-isaacs
  and
  claude
  committed
  40ce998
- Drive a V2 scan's run from its stream, with compute on spawn_cpu
  
  joseph-isaacs
  and
  claude
  committed
  cbde02c
- Build a prune planner with the planner it continues into
  
  joseph-isaacs
  and
  claude
  committed
  460725f
- Spawn a scan's chunks as independent planners and run them beside each other
  
  joseph-isaacs
  and
  claude
  committed
  4797a34
- Dedup a planner's IO hints with a set
  
  joseph-isaacs
  and
  claude
  committed
  22aa027
- Run a V2 scan's chunks beside each other only as far as the engine does not
  
  joseph-isaacs
  and
  claude
  committed
  bb10f65
- Expose a V2 scan's planners and prepared plans; drive them elsewhere
  
  joseph-isaacs
  and
  claude
  committed
  d091576
- Compose and drive a V2 scan in DataFusion
  
  joseph-isaacs
  and
  claude
  committed
  25e5f42
- Drive a V2 scan of a file for callers without an engine
  
  joseph-isaacs
  and
  claude
  committed
  d858c4b
- Compose a V2 scan's planners in the DataFusion opener
  
  joseph-isaacs
  and
  claude
  committed
  af81c35
- Drop is_proved: a proved proof lists no segment
  
  joseph-isaacs
  and
  claude
  committed
  4cad9a8
- Call the zone-map plans pruning plans, not proofs
  
  joseph-isaacs
  and
  claude
  committed
  da8a7c8
- Speculate the next chunk's reads while pruning waits; batch root-run steps
  
  joseph-isaacs
  and
  claude
  committed
  5c130fb
- Evaluate every pruning plan at once
  
  joseph-isaacs
  and
  claude
  committed
  cf7117f
- Drive the root run on one task and run the first chunk on it
  
  joseph-isaacs
  and
  claude
  committed
  c6b2ad9
- Snap a partition's planning-time rows to the file's natural splits
  
  joseph-isaacs
  and
  claude
  committed
  68c08ec
- Speculate a chunk's projection as announcements beside its predicate reads
  
  joseph-isaacs
  and
  claude
  committed
  3905421
- Register a chunk's speculated reads beside the zone-map fetch they overlap
  
  joseph-isaacs
  and
  claude
  committed
  e26781b
- Let a session's announcement take the one speculated for its segment
  
  joseph-isaacs
  and
  claude
  committed
  b9350ec
- Share a speculated read with every session that fetches its segment
  
  joseph-isaacs
  and
  claude
  committed
  b73d248
- Cut a partition's chunks from its own start, not a grid over the file
  
  joseph-isaacs
  and
  claude
  committed
  a8700b4
- Speculate every chunk of the partition while its zone maps are read
  
  joseph-isaacs
  and
  claude
  committed
  3294ef5
- Share a speculated announcement with every session once fetched
  
  joseph-isaacs
  and
  claude
  committed
  816ce94
- Hand a run's spawned chunks to the consumer together
  
  joseph-isaacs
  and
  claude
  committed
  17f46e6
- Speculate the projection's reads beside the predicates', not announcements
  
  joseph-isaacs
  and
  claude
  committed
  a203827
- Report what a V2 scan's stages did as metrics
  
  joseph-isaacs
  and
  claude
  committed
  44e8d1b
- Read a projection-only partition's segments up front, shared by its chunks
  
  joseph-isaacs
  and
  claude
  committed
  ec78689
- Mark the start and end of every phase of a partition
  
  joseph-isaacs
  and
  claude
  committed
  19b1f7d
- Record what the DataFusion driver runs, beside the scan's stage metrics
  
  joseph-isaacs
  and
  claude
  committed
  79d2062
- Build each projection and filter plan once per file, the other partitions waiting for it
  
  joseph-isaacs
  and
  claude
  committed
  629b0e7
- Intern a scan's expressions per file, so partitions share its plans and values
  
  joseph-isaacs
  and
  claude
  committed
  141570f
Commits on Oct 23, 2026
- Cut a filtered partition into a few chunks, a quarter of its rows each
  
  joseph-isaacs
  and
  claude
  committed
  4765068
- Probe interned expressions under a read lock
  
  joseph-isaacs
  and
  claude
  committed
  6ea72d8
- Read a partition's zone maps on their own, ahead of the data they sit beside
  
  joseph-isaacs
  and
  claude
  committed
  4ca88fa
- Read ahead only a few chunks of a file in an object store
  
  joseph-isaacs
  and
  claude
  committed
  b817434
- Keep a file's scan cache with its cached footer, across queries
  
  joseph-isaacs
  and
  claude
  committed
  28e7298
- Round a filtered partition's quarter up, so it is four chunks and not four and a sliver
  
  joseph-isaacs
  and
  claude
  committed
  48cbf38
- Give every root admitted to a partition's run an IO session of its own
  
  joseph-isaacs
  and
  claude
  committed
  d735965
- Bound the bench's S3 idle connection pool, so every query runs in the fast mode
  
  joseph-isaacs
  and
  claude
  committed
  0c42ba4
- EXPERIMENT: one segment source per V2 partition, as V1
  
  joseph-isaacs
  and
  claude
  committed
  d389a2b
Commits on Oct 24, 2026
- EXPERIMENT: let a manually dispatched SQL benchmark comment on #10765
  
  joseph-isaacs
  and
  claude
  committed
  e922dfc

Commits

Commits on Oct 22, 2026

Commits on Oct 23, 2026

Commits on Oct 24, 2026