[shrivasshankar]

Which issue does this PR close?

• Closes Re-land Decimal filter selectivity estimation (#25790) without the TPC-H Q18 regression #27146.
• Re-lands feat: estimate Decimal filter selectivity with interval analysis #25790 (reverted in Revert #25790 (Decimal filter selectivity) due to TPC-H Q18 regression #27145 because of TPC-H Q18 regression #27108).

Rationale for this change

#25790 gave two kinds of decimal filters bad estimates:

• No min/max stats, like Q18's HAVING sum(l_quantity) > 300: estimated at
  100% of rows instead of 20%, which flipped the Q18 semi join.
• Equality, like TPC-DS ca_gmt_offset = -5 on DECIMAL(5, 2): interval
  analysis counts every 0.01 step, so it estimated 100 rows instead of 18,246.

What changes are included in this PR?

• Commit 1 is feat: estimate Decimal filter selectivity with interval analysis #25790 unchanged.
• Commits 2 and 3 make those two cases keep the estimate they had before
  feat: estimate Decimal filter selectivity with interval analysis #25790. Range filters on decimals still use interval analysis.

What is the testing strategy for this PR?

• Unit tests for both cases, and a Q18 plan check in
  filter_statistics_decimal.slt.
• All 22 TPC-H plans (SF1, SF10) and 99 TPC-DS plans (SF1) match main. With
  feat: estimate Decimal filter selectivity with interval analysis #25790 alone, TPC-H Q18 and 7 TPC-DS plans changed.
• The Q6-style query from Estimate Decimal filter selectivity #25613 still improves: 111,291 estimated rows
  (actual 114,160) vs 1,200,243 on main.

Are there any user-facing changes?

Better estimates for range filters on decimal columns with min/max stats. One
new public function, check_statistics_support.

[shrivasshankar]

cc @asolimando @devanbenz @kosiew, this re-lands apache/datafusion#25790 · feat: estimate Decimal filter… with the Q18 fix. The Q18 plan check is in filter_statistics_decimal.slt.

[shrivasshankar]

Update: while checking TPC-DS, I found that = filters on decimal columns were also badly underestimated (100 rows instead of 18,246 for ca_gmt_offset = -5), which flipped 5 TPC-DS plans. I added a commit so decimal equality keeps its old estimate. All TPC-H and TPC-DS plans now match main.

[devanbenz]

@kosiew @xudong963 Are you able to kick off the automated benchmarks? We should run all available benchmarks both with pushdown filtering on and off. If we can think of any other configurations which may be impacted by this as well please run those as well.

@shrivasshankar

Better estimates for range filters on decimal columns with min/max stats. One new public function, check_statistics_support.

Better estimates is nice! But also, it would be cool to see metrics associated with performance goodness. If you have them please add them to the PR description :)

[shrivasshankar]

@kosiew @xudong963 Are you able to kick off the automated benchmarks? We should run all available benchmarks both with pushdown filtering on and off. If we can think of any other configurations which may be impacted by this as well please run those as well.

@shrivasshankar

Better estimates for range filters on decimal columns with min/max stats. One new public function, check_statistics_support.

Better estimates is nice! But also, it would be cool to see metrics associated with performance goodness. If you have them please add them to the PR description :)

Thanks! With this PR, every TPC-H (SF1, SF10) and TPC-DS (SF1) plan is identical to main, with pushdown_filters both on and off, so I wouldn’t expect runtime changes on those benchmarks. The gain for now is in the estimates themselves. For example, the Q6-style filter goes from 1.2M estimated rows to 111K (actual 114K). I’ll add the benchmark results to the description once they’ve run.