refactor(planner): plan the workload on the unified operator IR

[zzylol]

Why

#539–#541 (merged) add the unified frontends, the timing pass, the physical export and a runtime that compiles it, but the planner still lowers queries to the legacy QueryExpr graph and runs them through the legacy PostAsapDAG runtime. Until the planner switches, nothing in the product uses the #511 IR.

What

The planner runs end to end on the unified OperatorNode DAG. No new #509 stage logic is added here: this PR ports the legacy cost-ranked candidate search (search_workload with the ASAPStrategies), which was then the planner. #561 onward added the #509 stages, and #674 replaced the legacy search with them: on main the planner is the planner pipeline, run_stage1_to_stage3 / Stage1ToStage3Pass (Stage 1 logical ASAP-aware optimization → Stage 2 physical ASAP-aware optimization → Stage 3 plan selection).

• Stage 0 (query-language frontends) to Stage 1. The planner lowers through the unified frontends (now the crate roots of asap_frontend_{sql,promql,metricsql}, refactor: move the unified frontends and physical planner to the crate roots #665) and gets QueryRoots; ParsedWorkload keeps scalar roots. The MetricsQL frontend input carries the planning clock and lowers through lower_metricsql_query_workload, so bare selectors get the ingestion-interval lookback (feat(promql): lower queries to unified operator and scalar IR #540).
• The planner runs on OperatorNode. Candidate generation, selection and DAG assembly run on OperatorNode. PlanOutput exposes the whole workload DAG. Planned summary states derive their coverage from the node (feat(ir): derive summary coverage as definition + selection #646).
• Standalone scalar queries are planned. A workload entry whose root is a scalar (scalar(quantile(0.99, rate(x[5m]))), scalar(sum(up)) + 1) used to be copied through unplanned. Per docs: define unified operators and SQL/PromQL scalar boundaries #511 §2.3 a scalar query references operator sub-DAGs; those sub-DAGs now go through the same planning as operator roots, so they can get summary candidates, and the scalar is rebuilt over the selected sub-DAGs (PlanOutput.scalar_plans).
  • Each reference is planned against the entry's accuracy target, so combinations of approximate values such as scalar(quantile(0.99, …)) - scalar(quantile(0.5, …)) keep summary candidates. The guarantee of such a combination is None until Compose accuracy guarantees through arithmetic and comparisons over approximate values #669; Stage 3 plan selection rejects it without output-level evidence. References inside Project/Filter expressions follow the same rule.
  • PlanOutput.scalar_plans holds one plan per distinct referenced node, and the export emits one ScalarRef edge per distinct referenced node. Structurally identical references (scalar(rate(x[5m])) - scalar(rate(x[5m]))) are separate nodes and are planned separately until Pass 2 (ASAP-aware CSE, Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580).
  • vector op scalar(v) in a Project or Filter (a / scalar(b), a > scalar(b)) is lowered with one PromqlVectorToScalar input per distinct scalar reference, after the row input; its statistics carry one scalar_inputs edge per such input. These queries previously failed costing with UnsupportedQueryOperator.
  • PhysicalASAPDAG gains scalar_roots: Vec<(usize, ScalarExpr<PhysicalASAPNodeId>)>: entry i of the batch is the scalar root keyed i if there is one, otherwise the next entry of roots. Validation checks the keys and treats scalar references as roots for existence, cycles and reachability (PHYSICAL_ASAP_DAG_WIRE_VERSION 9 here; 10 on main since refactor(planner): replace the summary lifecycle with #509 materialization options #667). The workload export that emits them is added in feat(planner): export a planned workload as one physical ASAP DAG #666.
  • check_contract checks that scalar_plans holds one plan per distinct reference of each scalar root, in order and keyed by its entry. frontier_from_timing and the explanation report walk scalar references like operator inputs.
• Candidate generation fixes found during the port (in the legacy search strategies, which stay in asap-aware-mapping for the explanation entry points and dag_export):
  • Top-k reuse (TopKLimitReuseStrategy) requires equal partition_by; a global top-10 is no longer offered as the source of a per-job top-5. Heavy-hitter promotion and the maintained-population (current-series heap) candidates require the Limit to partition like its Sort.
  • Logical rewrites are told apart by the candidate's provenance (ReplacementSubDAG::is_logical_rewrite: LogicalRewrite, CseShare, CseRecompute, AccuracyReconciliation) instead of by the replacement's shape.
  • Filtered COUNT/SUM rankings (COUNT(*) FILTER (WHERE ...), COUNT(col) over a nullable column, ordered and limited) are no longer promoted to a heavy-hitter TopK, either in IR canonicalize or in realize_keyed_additive_summary_input. This bug was on main since feat(ir): add canonicalization, sub-DAG sharing and flat DAGs on the unified IR #645.
• Query roots are finalized. A root whose selected candidate is an exact accumulator state (sum_over_time(m[1m]), sum(rate(x[5m]))) is read through FinalizeExactAccumulator, so the exported root yields values, not a bare SummaryAgg. Scalar references are finalized the same way, since scalar(v) reads values. On main, Stage 1 composition (feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575) builds the finalize into every exact-accumulator candidate.
• Search errors instead of panics. The legacy candidate search on input that already contains ASAP operators returns SearchError::AsapInput, and the explanation entry points pass it on. On main the planner pipeline rejects such input in Stage 1 Pass 1 (PerComputationCandidateError::ContainsAsap, reported as OptimizeError::PerComputationCandidates).
• The legacy physical planner is removed. It read the planner's QueryExpr output, so it is deleted with its expression compiler, its PromQL fallback path and its raw sources. Every caller uses feat(runtime): export timed plans as PhysicalASAPDAG and compile them #541's physical planner and sources (physical_planner / sources since refactor: move the unified frontends and physical planner to the crate roots #665).
• Evaluation vocabulary (docs: ASAP primitive schema and summary coverage (definition + selection) #573). readout names become evaluation: the readout module becomes evaluation, ReadoutQuery → SummaryEvaluation, ExactReadout → ExactEvaluation, GuaranteeSource::SketchReadout → SketchEvaluation, EstimatorContract::ClassicHll { max_distinct_per_evaluation }.
• Dead code removed: the never-constructed VectorScalar / ScalarVector operand modes and the unused ParsedWorkload::new / entries.

How

The change is a port: every planner module that took QueryExpr / SummaryNode now takes Rc<OperatorNode>, using the same strategies and cost model. Tests that drive the planner (frontend planning tests, physical-operator tests that plan before executing, integration tests, devtools) are ported to the unified entry points. The legacy frontend modules and the QueryExpr / SummaryNode IRs stayed compiled in this PR; #665 moved the unified modules to the crate roots, and #543 removed the legacy IRs.

Before this PR

e2e_plan lowered to QueryExpr, planned and costed SummaryNodes, and the runtime compiled a PostAsapDAG. The unified IR from #539–#541 was only reachable from its own tests. Standalone scalar queries were copied through unplanned, an exact-state root was exported as a bare SummaryAgg, and a filtered COUNT/SUM ranking could be planned as a heavy-hitter sketch over every row.

After this PR

e2e_plan and the planner produce and run unified OperatorNode plans; the runtime path is PhysicalASAPDAG → the physical planner. Public API changes for planner callers:

• Frontends return OperatorNode / QueryRoot; FrontendInput::Metricsql carries now_ms.
• search_workload, search_workload_with, search_workload_with_targets, explain_replacements and explain_replacements_with return Result (SearchError).
• is_logical_rewrite is now the method ReplacementSubDAG::is_logical_rewrite.
• PlanOutput.scalar_plans and PhysicalASAPDAG.scalar_roots are new.
• The readout → evaluation renames listed above.
• The legacy physical planner, its sources and its PromQL fallback path are gone.

Later changes: #665 moved the unified frontends and physical planner to the crate roots; #667 removed the summary maintenance lifecycle (wire version 10); #674 retired the legacy search as the planner in favor of run_stage1_to_stage3 / Stage1ToStage3Pass (#675 names), which does not merge identical producers or scalar references until Pass 2 (#580); the scalar tests were renamed accordingly.

Evidence

Execution examples: the planner, integration and devtools tests run on the unified IR. crates/integration-tests/tests/operator_design_examples.rs (new) plans, exports and executes the #511 worked examples; operator_sharing.rs (new) checks that non-ASAP operators above and below summary operators can share inputs with them (#468). Screenshots and measurements: not applicable.

Verification

• Unit and integration tests: ported to the unified entry points; subquery_time_shift_is_retained (a subquery's offset stays a TimeShift) and does_not_promote_a_filtered_count_ranking were carried over from the legacy tests.
• Scalar queries (names on main): scalar_queries_read_planned_sub_dags (["1", "scalar(quantile(0.99, rate(x[5m])))", "quantile(0.99, rate(x[5m]))"] keeps entry order and the scalar reads its planned sub-DAG), combined_scalar_references_keep_summary_candidates, identical_scalar_references_satisfy_the_plan_contract, scalar_reference_in_an_expression_is_a_scalar_input, scalar_references_are_one_input_per_distinct_reference, contract_rejects_a_scalar_reference_without_its_plan, contract_checks_scalar_plan_order_entry_and_count, frontier_from_timing_includes_ingestion_scalar_reference, a_quantile_read_by_a_scalar_expression_is_located, and scalar_roots_export_in_batch_order_and_reference_nodes (also rejects duplicate and descending keys).
• Finalized roots: exact_state_roots_are_finalized (summary_sharing.rs) and assemble_selected_query_finalizes_only_an_exact_state_root (legacy search).
• Candidate generation and search fixes: global_limit_is_not_reused_for_a_per_group_limit, search_rejects_a_root_with_an_asap_operator, filtered_count_ranking_is_not_a_heavy_hitter (through the SQL frontend).
• End-to-end tests: crates/planner/tests/e2e_plan.rs, summary_sharing.rs.
• Other checks on the PR head: cargo fmt --check; workspace all-target Clippy with -D warnings; cargo test --workspace (2,010 passed, 0 failed, 2 ignored); dag-viewer test_render (31 passed); tools/verify_metricsql_external_consumer.sh OK.

Architectural decisions

• Cut over first, move second. This PR called the unified paths; moving them to the crate roots was refactor: move the unified frontends and physical planner to the crate roots #665, a pure move, so this diff carries only planner logic.
• The legacy physical planner is removed here, not in the move PR, because it consumed the planner's QueryExpr output and cannot compile after the cutover.
• No execution fallback. feat(runtime): export timed plans as PhysicalASAPDAG and compile them #541 ships no PromQL evaluator, and this PR deletes the legacy planner's PromQL fallback path; the runtime callers only compile what the native operators support. Executing anything else is the deployment runtime's job (Stage 4).
• Scalar roots in the export, not a new query-entry structure (docs: define unified operators and SQL/PromQL scalar boundaries #511 §2.3): the scalar keeps its ScalarExpr form with references to exported nodes.
• Approximate combinations keep their summary candidates. Candidate generation never prunes because of how an output is combined. The combined guarantee is Compose accuracy guarantees through arithmetic and comparisons over approximate values #669; in the MVP a rank-interval guarantee counts as None, and Stage 3 plan selection accepts a combination only with output-level evidence.

Limitations and follow-up

• refactor: move the unified frontends and physical planner to the crate roots #665: move the unified frontends and physical planner to the crate roots (merged).
• refactor(ir): remove the legacy QueryExpr and SummaryExpr IRs; move docs and DAG viewer to the unified IR #543: remove the legacy QueryExpr / SummaryExpr IRs (merged).
• refactor(planner): replace the summary lifecycle with #509 materialization options #667: remove the summary maintenance lifecycle in favor of the docs: propose workload-wide planning, summary sharing, and materialization #509 Stage 2 materialization options (merged).
• Compose accuracy guarantees through arithmetic and comparisons over approximate values #669: compose accuracy guarantees through arithmetic and comparisons over approximate values.
• Execute scalar operator references in Project/Filter expressions of the physical DAG #668: the physical operators cannot yet execute scalar references inside Project/Filter expressions.
• Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580: Pass 2 (ASAP-aware CSE) merges identical producers and scalar references.

Stack: tracker #528 · main → #542 → #665 → #666 → #543 → #667 → #561.

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW