parquet 59 deprecates ParquetObjectReader. Upstream's replacement is not a
rename — you implement AsyncFileReader yourself
(apache/arrow-rs#10308).
Three #[allow(deprecated)] currently hold the line:
rust/timscentroid/src/storage.rs — read_parquet_peaks
rust/timscentroid/src/lazy/query.rs — ParquetQuerier::new_async, ParquetQuerier::query_async
What the replacement has to cover
AsyncFileReader requires get_bytes and get_metadata. The catch is
get_byte_ranges: its default implementation is serial —
for range in ranges {
let data = self.get_bytes(range).await?;
result.push(data);
}
ParquetObjectReader overrides it with ObjectStore::get_ranges, which fetches
concurrently. Taking the default would serialize every column-chunk fetch, so
the override is required, not optional. ParquetObjectReader also does metadata
suffix-prefetch and caching worth matching.
So the work is roughly:
get_bytes → ObjectStore::get_range
get_byte_ranges → ObjectStore::get_ranges (must stay concurrent)
get_metadata → ParquetMetaDataReader with a suffix prefetch hint
Verifying it
Parquet reads here go through InstrumentedStore, whose counters now have test
coverage in rust/timscentroid/src/instrumentation.rs. A regression in fetch
batching shows up as a higher get_count for the same read, which is worth
asserting so a serial get_byte_ranges cannot land unnoticed.
parquet59 deprecatesParquetObjectReader. Upstream's replacement is not arename — you implement
AsyncFileReaderyourself(apache/arrow-rs#10308).
Three
#[allow(deprecated)]currently hold the line:rust/timscentroid/src/storage.rs—read_parquet_peaksrust/timscentroid/src/lazy/query.rs—ParquetQuerier::new_async,ParquetQuerier::query_asyncWhat the replacement has to cover
AsyncFileReaderrequiresget_bytesandget_metadata. The catch isget_byte_ranges: its default implementation is serial —ParquetObjectReaderoverrides it withObjectStore::get_ranges, which fetchesconcurrently. Taking the default would serialize every column-chunk fetch, so
the override is required, not optional.
ParquetObjectReaderalso does metadatasuffix-prefetch and caching worth matching.
So the work is roughly:
get_bytes→ObjectStore::get_rangeget_byte_ranges→ObjectStore::get_ranges(must stay concurrent)get_metadata→ParquetMetaDataReaderwith a suffix prefetch hintVerifying it
Parquet reads here go through
InstrumentedStore, whose counters now have testcoverage in
rust/timscentroid/src/instrumentation.rs. A regression in fetchbatching shows up as a higher
get_countfor the same read, which is worthasserting so a serial
get_byte_rangescannot land unnoticed.