Skip to content

Pass 1: push down filters and HAVING across levels to create more candidates (and reuse DataFusion's filter push-down) #664

Open

Description

@zzylol

Follows the #561 decision on filters and HAVING.

Context

#561 keeps summary candidates for filtered measures and HAVING (they no longer fall back to the exact option only):

This handles one level: a filter or HAVING directly on one eligible aggregate.

Idea

Filters and HAVING at other levels may also be pushed down, and each push-down can create new candidates:

  • a Filter several operators above an aggregate (through Project, Sort, Limit, joins on the key, a set operation), or a HAVING over a nested aggregate (e.g. sum by (job) (rate(...)) with a predicate on job above the outer sum), can move below the inner aggregation, so the inner target's state reads fewer rows and its coverage selection (docs: ASAP primitive schema and summary coverage (definition + selection) #573 §4.2.2) gets the predicate;
  • a predicate on group keys that survives through a nested aggregation can be pushed into both levels;
  • pushing a predicate into a SummaryAgg.filter or below the scan changes coverage, so two queries that differ only in such predicates may share one state with different selections (docs: ASAP primitive schema and summary coverage (definition + selection) #573) instead of two states.

Each push-down is a rewrite rule in Pass 1 (#509: "applies rewrite rules"); the rewritten and original forms are both candidates, and Stage 3 chooses.

DataFusion

DataFusion's optimizer already has these rules for SQL logical plans, e.g. PushDownFilter (moves filters below projections, aggregates on group keys, joins, unions), plus SimplifyExpressions and EliminateFilter. Today the SQL frontend lowers DataFusion's unoptimized plan (only type coercion runs). Options:

  • run DataFusion's filter push-down before lowering, for SQL only (it does not help PromQL, which does not go through DataFusion);
  • port the relevant rules to the unified IR, so they apply to every frontend and can be offered as candidates rather than applied unconditionally.

Shared aggregates

When one aggregate is shared by several queries (one node in the workload DAG, #521 / Pass 2), a predicate that one query applies above it must not be pushed below it: the other queries read the aggregate without that predicate. #561 records the HAVING placement per target, not per consumer, so a key-only Filter from one consumer shows as pushable even when another consumer of the same aggregate has no filter. Composition (#575) must only push down a predicate that every consumer of the aggregate applies (or copy the aggregate for the filtering consumer, which is then a different, unshared candidate).

Questions

Activity

  1. zzylol commented on Oct 12, 2026

    @zzylol
    ContributorAuthor

    More shapes found in the #561 review where a HAVING-like value predicate is not directly above the Aggregate. Pass 1 (#561) only records the adjacent form; these are candidates for multi-level push-down here:

    • Derived-table WHERE: SELECT * FROM (SELECT src, count(*) c FROM flows GROUP BY src) t WHERE c > 10 lowers to Filter → Project → Aggregate.
    • HAVING split by lower_filter (frontend-sql/src/sql/mod.rs): ... HAVING src='a' AND count(*) > (SELECT count(*) FROM flows) becomes Filter(subquery) over Filter(src='a') over the Aggregate.
    • GROUP BY ROLLUP(src) HAVING count(*) > 3 lowers to Filter → Concat → Project → Aggregate.
    • Possibly: after canonicalization, a HAVING with IN/EXISTS becomes a semi-join above the aggregate.

    Pushing a key-only predicate below a shared aggregate is only valid when every consumer applies it (#561 now records each consumer's Filter). Separately, a value predicate over an approximate output gets guarantee None through guarantee propagation in #575, whatever its depth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions