[core] Reduce OSS requests for dense manifest sidecar selections - #990
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
LGTM. Reviewed e57d695 and found no blocking issues. Checked that the estimate and selected read share the same range planner, the full-read policy respects the request-count threshold and 4x amplification cap, empty selections remain header-only, and the scan callers still apply entry-level filtering after a full-manifest fallback.
Local validation: all 8 existing manifest-sidecar tests and 109 table-scan tests passed. Three additional review-only tests also passed: exact threshold/4x-cap boundaries; request counting through the OSS backend against a local HTTP endpoint (one manifest-body GET for dense or large-contiguous selections, two range GETs for the sparse case, zero body requests for an empty selection); and scan-result equivalence with sidecars disabled across dense, sparse and empty row-range selections. All 14 GitHub checks were successful for this head. The extra tests were kept in the isolated review checkout; no changes were pushed.
Purpose
Follow up on #988: a scan that uses a manifest sidecar can issue several range GETs for one manifest even when most of its blocks are selected. For an approximately 8 MiB manifest, this can turn one full GET into a sidecar GET plus two or more range GETs.
Changes
For an 8 MiB manifest with two planned ranges and at least 4 MiB selected, the manifest body now uses one full read rather than two range reads.
Tests
cargo test -p paimon spec::manifest_sidecar::tests --lib(8 passed)cargo test -p paimon table::table_scan::tests --lib(109 passed)cargo clippy -p paimon --lib --tests -- -D warningscargo fmt --all -- --checkNo storage format change.