Skip to content

[rust] Reuse BTree data blocks across index lookups - #995

Merged
JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/cache-btree-data-blocks
Sep 30, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/cache-btree-data-blocks

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Global-index predicates such as an OR of precise AND branches can issue multiple equality lookups against the same BTree file. The reader already reuses the footer and index block, but each lookup reads its data block again.

This PR adds a scan-scoped data-block cache so lookups that resolve to the same block reuse its decoded contents without changing predicate or residual-filter semantics.

Changes

  • Cache decoded BTree data blocks by file path, offset, and size.
  • Bound the cache by decoded bytes with LRU eviction across all BTree files in one scan.
  • Treat btree-index.cache-size as the total Java-compatible BTree cache budget. The data-block cache receives the data-pool share selected by btree-index.high-priority-pool-ratio (default 0.1).
  • Share the cache across BTree readers within one global-index scan.
  • Keep standalone BTreeIndexReader::open behavior unchanged.

The cache is scoped to one scan and storage context. It does not add an unbounded global cache or share mutable reader state. Concurrent cold misses are unchanged. Rust does not yet cache high-priority BTree metadata across readers; that reserved share is therefore not allocated to data blocks.

Verification

A counting FileRead test performs three equality lookups whose keys are in one data block:

Configuration Data-block reads
Cache disabled 3
Cache enabled 1

The test also verifies the exact row IDs. Additional tests cover file isolation, disabling the cache, byte-budget eviction across multiple files, and Java-compatible pool-ratio validation.

  • cargo test -p paimon --lib btree::tests (40 passed, 1 ignored)
  • cargo test -p paimon --lib table::global_index_scanner::tests (58 passed, 1 ignored)
  • cargo test -p paimon --lib spec::core_options::tests::test_btree_index_data_block_cache_size_matches_java_pool_split
  • cargo clippy -p paimon --lib -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

This PR does not claim production OSS request or end-to-end workload results; those require environment-specific validation.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 30, 2026 06:48
@JingsongLi
JingsongLi merged commit 6267e9c into apache:main Sep 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants