Skip to content

[core] Reduce OSS requests for dense manifest sidecar selections - #990

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/sidecar-qps
Sep 29, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/sidecar-qps

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Follow up on #988: a scan that uses a manifest sidecar can issue several range GETs for one manifest even when most of its blocks are selected. For an approximately 8 MiB manifest, this can turn one full GET into a sidecar GET plus two or more range GETs.

Changes

  • Estimate selected manifest bytes and the number of range reads after the existing 4 MiB coalescing rule.
  • Read the full manifest when doing so saves range requests and the transfer increase is bounded by the range count, capped at 4x. Keep sparse selections on the range-read path and empty selections header-only.
  • Use the same range planner for the estimate and the actual selected read.

For an 8 MiB manifest with two planned ranges and at least 4 MiB selected, the manifest body now uses one full read rather than two range reads.

Tests

  • cargo test -p paimon spec::manifest_sidecar::tests --lib (8 passed)
  • cargo test -p paimon table::table_scan::tests --lib (109 passed)
  • cargo clippy -p paimon --lib --tests -- -D warnings
  • cargo fmt --all -- --check

No storage format change.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Reviewed e57d695 and found no blocking issues. Checked that the estimate and selected read share the same range planner, the full-read policy respects the request-count threshold and 4x amplification cap, empty selections remain header-only, and the scan callers still apply entry-level filtering after a full-manifest fallback.

Local validation: all 8 existing manifest-sidecar tests and 109 table-scan tests passed. Three additional review-only tests also passed: exact threshold/4x-cap boundaries; request counting through the OSS backend against a local HTTP endpoint (one manifest-body GET for dense or large-contiguous selections, two range GETs for the sparse case, zero body requests for an empty selection); and scan-result equivalence with sidecars disabled across dense, sparse and empty row-range selections. All 14 GitHub checks were successful for this head. The extra tests were kept in the isolated review checkout; no changes were pushed.

@JingsongLi
JingsongLi merged commit 91a8b21 into apache:main Sep 29, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants