Skip to content

[core] Reduce global index planning IO - #699

Merged
JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/reduce-global-index-planning-io
Aug 10, 2026
Merged

[core] Reduce global index planning IO#699
JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/reduce-global-index-planning-io

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Purpose

Align Rust global-index planning with Java and PyPaimon to avoid unnecessary object-store IO and unrelated index candidates.

Changes

  • prune sorted global-index manifest entries with the scan partition filter before evaluating index shards, matching Java and PyPaimon
  • use the file size already stored in IndexFileMeta, falling back to metadata lookup for legacy or invalid sizes, matching GlobalIndexIOMeta in Java and PyPaimon
  • keep deletion-vector selection and reader cache behavior unchanged

Performance

On a partitioned data-evolution table with an equality lookup and a partition predicate:

  • native FAST planning: 8.18s -> 2.93s
  • retained index entries: 762 -> 176
  • indexed candidate ranges: 4 -> 1
  • peak RSS in a plan-plus-explain benchmark: about 2.45 GiB -> 586 MiB

The result split and row range were unchanged.

Tests

  • cargo test -p paimon global_index_scanner --lib (46 passed)
  • cargo test -p paimon table_scan::tests --lib (84 passed)
  • cargo fmt --all -- --check
  • git diff --check

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 10, 2026 01:31

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit c120ac3 into apache:main Aug 10, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants