Repository navigation
Pass 1: push down filters and HAVING across levels to create more candidates (and reuse DataFusion's filter push-down) #664
Description
Follows the #561 decision on filters and HAVING.
Context
#561 keeps summary candidates for filtered measures and HAVING (they no longer fall back to the exact option only):
- a per-measure filter (
FILTER (WHERE …),COUNT(y)over a nullabley) is applied to the state asSummaryAgg.filter(realized in feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575, as Pass 1: single-measure filtered aggregates #608 already does); - a HAVING that reads only group keys is pushed below the aggregate;
- a HAVING that reads aggregate values stays a
Filterabove the output, withguarantee = None.
This handles one level: a filter or HAVING directly on one eligible aggregate.
Idea
Filters and HAVING at other levels may also be pushed down, and each push-down can create new candidates:
- a
Filterseveral operators above an aggregate (throughProject,Sort,Limit, joins on the key, a set operation), or a HAVING over a nested aggregate (e.g.sum by (job) (rate(...))with a predicate onjobabove the outer sum), can move below the inner aggregation, so the inner target's state reads fewer rows and its coverage selection (docs: ASAP primitive schema and summary coverage (definition + selection) #573 §4.2.2) gets the predicate; - a predicate on group keys that survives through a nested aggregation can be pushed into both levels;
- pushing a predicate into a
SummaryAgg.filteror below the scan changes coverage, so two queries that differ only in such predicates may share one state with different selections (docs: ASAP primitive schema and summary coverage (definition + selection) #573) instead of two states.
Each push-down is a rewrite rule in Pass 1 (#509: "applies rewrite rules"); the rewritten and original forms are both candidates, and Stage 3 chooses.
DataFusion
DataFusion's optimizer already has these rules for SQL logical plans, e.g. PushDownFilter (moves filters below projections, aggregates on group keys, joins, unions), plus SimplifyExpressions and EliminateFilter. Today the SQL frontend lowers DataFusion's unoptimized plan (only type coercion runs). Options:
- run DataFusion's filter push-down before lowering, for SQL only (it does not help PromQL, which does not go through DataFusion);
- port the relevant rules to the unified IR, so they apply to every frontend and can be offered as candidates rather than applied unconditionally.
Shared aggregates
When one aggregate is shared by several queries (one node in the workload DAG, #521 / Pass 2), a predicate that one query applies above it must not be pushed below it: the other queries read the aggregate without that predicate. #561 records the HAVING placement per target, not per consumer, so a key-only Filter from one consumer shows as pushable even when another consumer of the same aggregate has no filter. Composition (#575) must only push down a predicate that every consumer of the aggregate applies (or copy the aggregate for the filtering consumer, which is then a different, unshared candidate).
Questions
- Which push-downs are always beneficial (apply in the frontend) and which should be candidates (Pass 1 rules)?
- Push-down through joins and set operations: which key/coverage conditions make it valid in the IR?
- Interaction with sharing (feat(planner): Pass 2 identical-expression rule (shared-input variants) #588 / Pass 2 summary-capability rule: share one summary sized for the strictest consumer #595): when does pushing a predicate into a shared state's selection beat keeping two states?
Activity
zzylol commentedon Oct 12, 2026
More shapes found in the #561 review where a HAVING-like value predicate is not directly above the
Aggregate. Pass 1 (#561) only records the adjacent form; these are candidates for multi-level push-down here:SELECT * FROM (SELECT src, count(*) c FROM flows GROUP BY src) t WHERE c > 10lowers to Filter → Project → Aggregate.lower_filter(frontend-sql/src/sql/mod.rs):... HAVING src='a' AND count(*) > (SELECT count(*) FROM flows)becomes Filter(subquery) over Filter(src='a') over the Aggregate.GROUP BY ROLLUP(src) HAVING count(*) > 3lowers to Filter → Concat → Project → Aggregate.IN/EXISTSbecomes a semi-join above the aggregate.Pushing a key-only predicate below a shared aggregate is only valid when every consumer applies it (#561 now records each consumer's Filter). Separately, a value predicate over an approximate output gets guarantee None through guarantee propagation in #575, whatever its depth.
fix(planner): per-consumer HAVING entries, global HAVING above, stric…
fix(planner): per-consumer HAVING entries, global HAVING above, stric…
fix(planner): per-consumer HAVING entries, global HAVING above, stric…
fix(planner): per-consumer HAVING entries, global HAVING above, stric…
feat(planner): Stage 1 Pass 1 per-computation candidate generation (#561