Skip to content

perf: start Parquet prefetch before predicate evaluation - #7

Draft
peterxcli wants to merge 1 commit into
codex/parquet-io-2-prefetchfrom
codex/parquet-io-3-early-prefetch
Draft

peterxcli wants to merge 1 commit into
codex/parquet-io-2-prefetchfrom
codex/parquet-io-3-early-prefetch

Conversation

@peterxcli

@peterxcli peterxcli commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Which issue does this PR close?

Related to apache#24393. Stage 3 of the Parquet I/O stack, based on #5.

Stack, with each PR targeting its predecessor: #4#5#7#8#9.

Rationale for this change

Starting the next read after predicate evaluation misses an opportunity to overlap substantial scan CPU work with I/O.

What changes are included in this PR?

For upfront reads with prefetch enabled, start the successor read before evaluating the current row group's predicates. Keep pending work aligned with the decoder when filters silently eliminate a group; retain cancellation and demand-read fallback.

What is the testing strategy for this PR?

The predicate-overlap regression checks that the next read starts before predicate CPU work. Existing cases cover skipped groups, dynamic filters, ordering, cancellation, and memory release.

Workspace Clippy with all targets/features, the extended test suite, and the focused Parquet/queue checks passed on lsa-cupid1. Tests use a private PID namespace, CPUs 96–111, and a 65,536 file limit.

Full partitioned ClickBench on lsa-cupid1, /work/peterxcli NVMe: 100 files, 99,997,497 rows, all 43 queries, eight workers/partitions. Versus the immediate parent, the sum of query medians changes -0.16% / -0.54% warm and -2.03% with evicted files (negative is faster). Warm values are two reversed-order passes; evicted files use three runs per query.

Keep. Q21 with 8 ms added per read improves 24.09% versus the parent. Results and methodology · 43 per-query comparisons and CPU charts. Result-ordering limitations and synthetic-latency controls are documented in the report.

Are there any user-facing changes?

The existing opt-in prefetch starts earlier. No new setting or default change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant