Skip to content

feat: use exact integer statistics for ordering inference - #26776

Open
haohuaijin wants to merge 8 commits into
apache:mainfrom
haohuaijin:integer-statistics-ordering
Open

haohuaijin wants to merge 8 commits into
apache:mainfrom
haohuaijin:integer-statistics-ordering

Conversation

@haohuaijin

@haohuaijin haohuaijin commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

An ordered integer column with exact extrema can still require a redundant sort for expressions such as a + 1, because ordering analysis starts with an unbounded interval. When a is ascending and its exact bounds are [1, 3], those bounds prove that adding one cannot overflow and preserves the input ordering.

What changes are included in this PR?

  • Seed trusted non-null integer bounds from original file-scan statistics into EquivalenceProperties.
  • Propagate bounds through computed projections, preserve them through window schema extension, and retain union bounds only when they cover every branch.
  • Preserve original scan statistics across protobuf round-trips so decoding reconstructs the same bounds. Public statistics retain the existing filter-precision policy.
  • Keep range diagnostics deterministic across plan reconstruction.

What is the testing strategy for this PR?

Extend the existing parquet_sorted_statistics.slt with plan and result checks for safe arithmetic, signed overflow and unsigned underflow, consecutive and reordered projections, grouping, windows, filters and dynamic filters, and unions with bounded or unknown branches. The tests use the SLT runner's four target partitions and default sort pushdown.

Strengthen the existing filtered Parquet scan round-trip test with exact integer extrema. The existing TPC-H round-trip test also verifies plans with multiple bounded columns.

Validation passed:

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • The complete parquet_sorted_statistics.slt file
  • 43 equivalence-property tests and two range-partition projection tests
  • 30 protobuf source integration tests with JSON enabled
  • TPC-H plan round-trips for all 22 queries

Are there any user-facing changes?

Ordered integer arithmetic can avoid redundant SortExec operators when exact bounds prove it safe. Bounds do not establish ordering on their own.

The additive EquivalenceProperties::with_column_statistics API accepts exact integer extrema. Range propagation adds planning metadata work; it does not calculate bounds per row.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Oct 26, 2026
@haohuaijin haohuaijin changed the title perf: use exact integer statistics for ordering inference feat: use exact integer statistics for ordering inference Oct 26, 2026
@github-actions github-actions Bot added the proto Related to proto crate label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate physical-expr Changes to the physical-expr crates proto Related to proto crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use exact column statistics to prove ordering through integer arithmetic

1 participant