perf(parquet): scope exact metadata reads by storage capability

[JunRuiLee]

Purpose

Linked issue: close #687

Use an explicit range-read capability to choose the Parquet metadata loading strategy:

• Built-in HDFS, local filesystem, and memory readers use the seek-based path: read the 8-byte footer first, then fetch the exact footer metadata and requested page-index ranges.
• Object-store readers and caller-provided filesystem operators retain the 512 KiB suffix prefetch by default, avoiding additional serialized network round trips when their range-read cost is unknown.

OpenDAL's HDFS native service keeps a file-scoped positioned-read handle, so exact range reads reuse the same underlying HDFS file reader. The capability is determined in the IO/storage layer and propagated through InputFileReader; the Parquet layer does not infer the backend from URI strings. Custom FileRead implementations may explicitly opt in, while the default remains conservative.

Performance validation

A local HDFS comparison showed better read performance with exact range reads than with the fixed 512 KiB prefetch. Object stores and caller-provided filesystem operators have not been benchmarked, so this change deliberately preserves their existing prefetch behavior rather than assuming the HDFS result transfers to them.

This PR remains Draft until the reproducible HDFS measurements are added. The benchmark record should include the same dataset/query for both variants, file count, representative footer-metadata sizes, repeated wall-clock results, and bytes-read or request counts when available; #687 tracks that evidence.

Brief change log

• Add a defaulted FileRead::supports_cheap_range_reads capability with a conservative false default.
• Enable exact metadata/page-index reads for built-in HDFS, local filesystem, and memory backends.
• Keep the fixed 512 KiB metadata prefetch for OSS, S3, COS, AzDLS, OBS, GCS, caller-provided filesystem operators, and other readers that do not explicitly opt in.
• Avoid requesting OffsetIndex solely because an external row selection is present but empty.
• Add range-tracking tests for both exact-read and prefetch paths.
• Add storage classification and FileRead capability-propagation tests.

Tests

• cargo fmt --all -- --check
• cargo test --locked -p paimon --lib arrow::format::parquet::tests (49 passed)
• cargo test --locked -p paimon --lib scheme_tests --features storage-all (10 passed)
• cargo test --locked -p paimon --lib custom_fs_operator --features storage-all (3 passed)
• Targeted memory and caller-provided filesystem capability-propagation tests passed.
• cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
• git diff --check

API and Format

Adds a defaulted FileRead::supports_cheap_range_reads method. Existing implementations do not need to implement the new method; they receive the conservative false behavior. No storage-format changes.

Documentation

No user-facing documentation changes required. Benchmark evidence is tracked in #687 before the PR is marked ready.

[JingsongLi]

How about object store?

[JunRuiLee]

How about object store?

Good point. I revised the implementation so exact metadata reads are opt-in through an explicit FileRead capability, rather than inferred from URI or path semantics.

• Built-in HDFS, local filesystem, and memory readers advertise cheap range reads and use exact footer, metadata, and page-index reads.
• OSS, S3, COS, AzDLS, OBS, and GCS keep the existing 512 KiB metadata prefetch.
• Caller-provided filesystem operators (CustomFs) also keep prefetch by default, because filesystem semantics alone do not guarantee cheap range reads.
• Other custom FileRead implementations default to the conservative prefetch behavior, but can explicitly opt in.

The capability is decided in the storage/IO layer and propagated to the Parquet reader. Tests cover backend classification, capability propagation, exact reads, and retained prefetch.

We have only benchmarked HDFS so far, so preserving the existing behavior for object stores and unknown custom backends is intentional.

[JingsongLi]

Thanks for revising the object-store behavior. I traced the change from Storage through FileIO and FileRead into the Parquet metadata reader. I did not find a correctness issue, but I think two points should be addressed before merging.

First, supports_cheap_range_reads = true does not necessarily mean that exact metadata reads are faster. The new tests show the request trade-off clearly: a small file on the prefetch path can be loaded with one read, while the exact path uses two serialized reads without page indexes and three when OffsetIndex is requested. The HDFS native reader reuses a file-scoped handle and block locations, but each range still creates a block stream and sends another read-block operation. This may be beneficial because it reads fewer bytes, but it can also increase per-file latency, especially when scanning many small Parquet files. Issue #687 says the reproducible HDFS measurements should be available before this PR is ready, but those numbers are not present yet. Could we add the file count, footer/index size distribution, request counts, and repeated wall-clock results before deciding whether this should be enabled unconditionally (or needs a file-size threshold)?

Second, the policy currently expands a public, format-agnostic API for a single private Parquet consumer. FileRead::supports_cheap_range_reads is only consumed by ArrowFileReader, but propagating the value requires changes across Storage -> FileIO -> InputFile/OutputFile -> InputFileReader -> FileRead. Storing a read policy on OutputFile is a sign that the policy has crossed its natural boundary.

A smaller design would keep the classification crate-private:

1. Keep Storage::supports_cheap_range_reads() as the backend decision point.
2. Expose it through a crate-private method on FileIO.
3. Have DataFileReader pass the policy to create_format_reader_with_budget.
4. Store the resulting metadata prefetch policy on ParquetFormatReader / ArrowFileReader.
5. Remove the public FileRead method and the capability fields on InputFile, OutputFile, and InputFileReader.

DataFileReader already owns the FileIO and is the only production caller of create_format_reader_with_budget, so this keeps the same behavior for HDFS/local/memory, object stores, and CustomFs without widening the public I/O contract.