Skip to content

fix(scan): prune primary-key files with key stats - #698

Merged
JingsongLi merged 1 commit into
apache:mainfrom
QuakeWang:verify-pk-stats
Aug 10, 2026
Merged

fix(scan): prune primary-key files with key stats#698
JingsongLi merged 1 commit into
apache:mainfrom
QuakeWang:verify-pk-stats

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Rust key-value files store primary-key statistics in key_stats, while scan pruning only evaluated value_stats. As a result, Rust-written primary-key files with empty value stats could not be pruned by primary-key predicates.

Brief change log

  • Evaluate file-safe primary-key predicates against key_stats.
  • Resolve key-stat layouts using trimmed primary-key order across schema evolution.
  • Preserve fail-open behavior for missing or invalid statistics.
  • Add regression tests for Rust-written files, composite keys, schema evolution, and corrupted stats.

Tests

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo test --locked -p paimon --all-targets --features fulltext,vortex
  • cargo test --locked -p paimon-rest-server --all-targets

API and Format

Documentation

Rust key-value files store primary-key statistics in key_stats while value_stats may be empty, so scan planning cannot prune them with primary-key predicates.

Evaluate file-safe key predicates against key_stats, including schema-evolution mapping, while preserving fail-open behavior.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit c58be1f into apache:main Aug 10, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants