test(parquet): add I/O tracking test for Bloom filter loading baseline by mohitgurav20 · Pull Request #26897 · apache/datafusion

mohitgurav20 · 2026-10-29T19:16:38Z

Which issue does this PR close?

Part of #25797. Baseline for #25811.

Rationale for this change

In #25811 we evaluate Parquet bloom filters incrementally during row group pruning to avoid fetching filters for later columns when an earlier column already rejects the row group.

Per review discussion on #25811, this PR adds a baseline test to main using a tracking AsyncFileReader (similar to arrow-rs async reader tests). It records that main currently fetches all candidate column bloom filters upfront (2 requests for a 2-column query where the first column is absent).

Once this merges, #25811 will rebase on top and show the skipped fetch directly in the test diff.

What changes are included in this PR?

Adds RangeTracker, TrackingAsyncReader, and TrackingReaderFactory in datafusion/datasource-parquet/src/opener/mod.rs to record requested byte ranges.
Adds test_bloom_filter_io_pattern_baseline asserting current upfront fetch behavior on main.

Are these changes tested?

Yes, via test_bloom_filter_io_pattern_baseline.

Are there any user-facing changes?

No.

alamb

Thanks for this @mohitgurav20

Can you please:

Move this to a "end to end" test that uses datafusion public APIs (dataframe or SQL, preferably SQL)
Move to using insta snapshots

The test hould do something like:

Create a test parquet file
REad the test parquet file using an instrumented object store
Report the series of events

Something like

Read data page for column 1 (offsets...)
Read PageInded for column 1 (ofsets...)
Produce batch (1024 rows)
Read Bloom Filter for column 1 (offsets...)

Then the log should be rendered as text and compared to a snapshot using insta snapshot

…ng baseline - Add InstrumentedObjectStore that correlates requested byte ranges to Parquet file structures (footers, page indexes, bloom filters, data pages) - Add test_parquet_bloom_filter_io_baseline using public SessionContext SQL APIs - Use insta snapshot to capture and compare the storage I/O trace - Documents baseline on main where all candidate column Bloom filters are loaded upfront, establishing the target diff for incremental pruning (apache#25811)

mohitgurav20 · 2026-10-31T07:58:25Z

hiii @alamb
Done — switched this to an end-to-end SQL test in datafusion/core/tests/parquet/bloom_filter_io.rs with an instrumented object store and insta snapshot.

The snapshot records that both bloom filters are loaded upfront on main. Once this lands, #25811's diff on the snapshot will cleanly show the second fetch skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Split the Bloom filter I/O test into reusable setup code in tests/parquet/io/mod.rs and the test itself in tests/parquet/io/bloom_filter_io.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…actor

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

alamb

Thank you very much @mohitgurav20 -- this is great.

I took the liberty to rework the tests a little so they are structured to accept other behaviors as well (specifically with predicate pushdown)

let me know what you think

cc @haohuaijin

alamb · 2026-11-02T20:33:58Z

 use std::sync::Arc;
 use tempfile::NamedTempFile;

+mod bloom_filter_io;


👍 Looks great. Thank you

alamb · 2026-11-02T20:58:09Z

+    // Loads the Bloom filters, finds a match, then reads the data pages and
+    // produces a batch.
+    assert_snapshot!(test.query("SELECT * FROM t WHERE a = 10 AND b = 100").await, @r"
+    --- Query: SELECT * FROM t WHERE a = 10 AND b = 100 (1 rows) ---


I am biased, but quite pleased with this test - it shows clearly the IO pattern of the reader

mohitgurav20 · 2026-11-03T08:27:16Z

Thanks @alamb, the refactor looks great! Splitting the test fixture into io/mod.rs makes the setup much cleaner and reusable across other scan tests, and inline snapshots make reviewing the I/O traces much simpler.

Happy with these changes as-is. Once this lands in main, I will rebase #25811 on top so the incremental pruning PR shows the skipped Bloom filter read directly in test_parquet_bloom_filter_io_pruned.

haohuaijin

LGTM, Thanks @alamb @mohitgurav20

mohitgurav20 · 2026-11-03T10:43:51Z

thanks for you review,time and guidance @alamb @haohuaijin

alamb · 2026-11-03T11:45:00Z

Thanks everyone

@r

…own and TopK (apache#27022) ## Which issue does this PR close? - Part of apache#26682 - Follow on to apache#26897 ## Rationale for this change We have several issues / PRs open that contemplate changing the I/O pattern such as - apache#24393 - apache#20324 - apache#26387 - apache#26418 - apache#23492 - apache#24086 - apache#25811 I want to have a test that clearly shows how those proposals effect the scan ## What changes are included in this PR? Add some new tests that show the Parquet I/O patterns. For example I think this is quite clear now the difference in I/O between filter pushdown and not ```rust /// Without filter pushdown, the scan fetches every column of every row group #[tokio::test] async fn test_filter_pushdown_io_disabled() { let test = filter_pushdown_test(false).await; assert_snapshot!(test.query("SELECT * FROM t WHERE val = 7").await, @r" --- Query: SELECT * FROM t WHERE val = 7 (1 rows) --- Request 1 Read Parquet footer (offsets 0..1402) Request 2 Read dictionary page and data page 0 for column 'id' (rg 0) (offsets 4..51) Read dictionary page and data page 0 for column 'val' (rg 0) (offsets 51..98) Request 3 Read dictionary page and data page 0 for column 'id' (rg 1) (offsets 98..145) Read dictionary page and data page 0 for column 'val' (rg 1) (offsets 145..192) Request 4 Read dictionary page and data page 0 for column 'id' (rg 2) (offsets 192..239) Read dictionary page and data page 0 for column 'val' (rg 2) (offsets 239..286) Produce batch (1 rows) "); } /// **With** filter pushdown, the scan first fetches and evaluates the filter /// column (`val`) for each row group, and then fetches the remaining columns /// (`id`) only for the row groups with matching rows (row group 1). #[tokio::test] async fn test_filter_pushdown_io_enabled() { let test = filter_pushdown_test(true).await; assert_snapshot!(test.query("SELECT * FROM t WHERE val = 7").await, @r" --- Query: SELECT * FROM t WHERE val = 7 (1 rows) --- Request 1 Read Parquet footer (offsets 0..1402) Request 2 Read dictionary page and data page 0 for column 'val' (rg 0) (offsets 51..98) Request 3 Read dictionary page and data page 0 for column 'val' (rg 1) (offsets 145..192) Request 4 Read dictionary page for column 'id' (rg 1) (offsets 98..124) Read data page 0 for column 'id' (rg 1) (offsets 124..145) Produce batch (1 rows) Request 5 Read dictionary page and data page 0 for column 'val' (rg 2) (offsets 239..286) "); } ``` ## What is the testing strategy for this PR? CI ## Are there any user-facing changes? No --------- Co-authored-by: mohitgurav20 <mohitgurav20@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added the datasource Changes to the datasource crate label Oct 29, 2026

mohitgurav20 force-pushed the test/parquet-bloom-filter-io-baseline branch from aacf4dd to 587ce90 Compare October 29, 2026 19:36

mohitgurav20 mentioned this pull request Oct 29, 2026

perf: evaluate Parquet Bloom filters incrementally during row group pruning #25811

Open

mohitgurav20 marked this pull request as ready for review October 29, 2026 19:56

alamb reviewed Oct 30, 2026

View reviewed changes

mohitgurav20 force-pushed the test/parquet-bloom-filter-io-baseline branch from 5788ada to c0a70fb Compare October 31, 2026 07:25

github-actions Bot added core Core DataFusion crate and removed datasource Changes to the datasource crate labels Oct 31, 2026

mohitgurav20 requested a review from alamb November 1, 2026 20:05

alamb and others added 6 commits November 2, 2026 15:37

test(parquet): use inline snapshot for Bloom filter I/O test

73ba64e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(parquet): move I/O tracing setup into parquet/io module

37110ee

Split the Bloom filter I/O test into reusable setup code in tests/parquet/io/mod.rs and the test itself in tests/parquet/io/bloom_filter_io.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rework tests

a6a9426

Merge remote-tracking branch 'apache/main' into alamb/pr-26897-io-ref…

61f0e9d

…actor

test(parquet): complete comment on pruned Bloom filter I/O test

1522d2c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

comments

f1517f6

alamb approved these changes Nov 2, 2026

View reviewed changes

alamb reviewed Nov 2, 2026

View reviewed changes

note where similar tests are

2fa72b0

alamb mentioned this pull request Nov 2, 2026

test(parquet): add I/O access tests for parquet pruning, filter pushdown and TopK #27022

Merged

Merge branch 'main' into test/parquet-bloom-filter-io-baseline

606ab3f

haohuaijin approved these changes Nov 3, 2026

View reviewed changes

alamb added this pull request to the merge queue Nov 3, 2026

Merged via the queue into apache:main with commit 39a66b1 Nov 3, 2026
44 checks passed

github-actions Bot added the v56.0.0 label Nov 3, 2026

Navigation Menu

Repository navigation

test(parquet): add I/O tracking test for Bloom filter loading baseline - #26897

test(parquet): add I/O tracking test for Bloom filter loading baseline#26897
alamb merged 9 commits into
apache:mainfrom
mohitgurav20:test/parquet-bloom-filter-io-baseline

mohitgurav20 commented Oct 29, 2026 •

edited

Loading

Uh oh!

alamb left a comment

Uh oh!

mohitgurav20 commented Oct 31, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Nov 2, 2026

Uh oh!

alamb Nov 2, 2026

Uh oh!

mohitgurav20 commented Nov 3, 2026

Uh oh!

haohuaijin left a comment

Uh oh!

mohitgurav20 commented Nov 3, 2026

Uh oh!

alamb commented Nov 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mohitgurav20 commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

mohitgurav20 commented Oct 31, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

mohitgurav20 commented Nov 3, 2026

Uh oh!

haohuaijin left a comment

Choose a reason for hiding this comment

Uh oh!

mohitgurav20 commented Nov 3, 2026

Uh oh!

alamb commented Nov 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mohitgurav20 commented Oct 29, 2026 •

edited

Loading