[rust] Reuse BTree data blocks across index lookups - #995
Merged
JingsongLi merged 2 commits intoSep 30, 2026
Merged
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
September 30, 2026 06:48
JingsongLi
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Global-index predicates such as an OR of precise AND branches can issue multiple equality lookups against the same BTree file. The reader already reuses the footer and index block, but each lookup reads its data block again.
This PR adds a scan-scoped data-block cache so lookups that resolve to the same block reuse its decoded contents without changing predicate or residual-filter semantics.
Changes
btree-index.cache-sizeas the total Java-compatible BTree cache budget. The data-block cache receives the data-pool share selected bybtree-index.high-priority-pool-ratio(default0.1).BTreeIndexReader::openbehavior unchanged.The cache is scoped to one scan and storage context. It does not add an unbounded global cache or share mutable reader state. Concurrent cold misses are unchanged. Rust does not yet cache high-priority BTree metadata across readers; that reserved share is therefore not allocated to data blocks.
Verification
A counting
FileReadtest performs three equality lookups whose keys are in one data block:The test also verifies the exact row IDs. Additional tests cover file isolation, disabling the cache, byte-budget eviction across multiple files, and Java-compatible pool-ratio validation.
cargo test -p paimon --lib btree::tests(40 passed, 1 ignored)cargo test -p paimon --lib table::global_index_scanner::tests(58 passed, 1 ignored)cargo test -p paimon --lib spec::core_options::tests::test_btree_index_data_block_cache_size_matches_java_pool_splitcargo clippy -p paimon --lib -- -D warningscargo fmt --all -- --checkgit diff --checkThis PR does not claim production OSS request or end-to-end workload results; those require environment-specific validation.