Reorganize crates and modules by #509 stages and the #511 unified IR #572

Problem

The planning stages (#509) and the unified operator DAG (#511) are settled, and #514/#544 renamed the DAG types to match. The code layout still follows the old pre-ASAP / post-ASAP split:

• asap-types has pre_asap and post_asap modules. The names no longer describe their contents. pre_asap::schema is the one schema for every DAG, and it already depends on post_asap::sketch. post_asap::sketch is shared vocabulary that the logical stage uses. post_asap::execution_data_state::ExecutionTiming is physical-stage data. The new ir/node.rs imports from both modules.
• asap-aware-mapping (45k lines) mixes three docs: propose workload-wide planning, summary sharing, and materialization #509 stages: Stage 1 rewriting (replacement, rewrite, rollup, …), Stage 2 materialization and lowering (summary_maintenance_lifecycle, query_physical_lowering, …), and Stage 3 cost and accuracy (cost_model, analytical_cost, erp, …). Its crate doc still describes it as "the cost-aware optimizer layer over the pre-ASAP intent algebra".
• asap-physical-operators mixes Stage 2 planning (plan, physical_planner) with Stage 4 execution (runtime, operators, sources, summary_kernels).

Proposed layout

One crate per #509 stage, so Cargo enforces the one-way stage flow. Shared types are arranged by #511 section.

crates/
  types/                     shared vocabulary; every stage depends on it
    ir/                      #511: one operator DAG for logical and physical plans
      operator/              §1   node, non_asap, asap, operator_properties (+ Source, GroupKeys,
                                  Reduction), agg_intent, maintained_population
      scalar/                §2.2 scalar, ColumnRef / ScalarValue / Compare+ArithmeticOpKind,
                                  scalar type rules, column resolution
      schema/                §2.1 Schema, Field, FieldDataType, DataType + state types
                                  (SketchKind, ExactKind, GroupingStrategy, SummaryUpdate)
      properties/            §2.2–2.3: guarantee, timing (ExecutionTiming), summary_coverage (#567)
      export, wire, cse, canonicalize
    workload/                #509 planner inputs: query/data workload, resources
    physical/                #509 Stage 2 decision data: execution data state (materialization),
                             summary maintenance, window summaries (panes, EH)
  frontend-common, -sql,     Stage 0  → CandidateLogicalDAGs (unchanged)
  -promql, -metricsql
  logical-optimizer/         Stage 1  → CandidateLogicalASAPDAGs
    pass1/                   replacement, rewrite, rollup, grouping, exact_composition,
                             function_rules, maintained_population, logical_candidates, explanation
    pass2/                   ASAP-aware CSE: accuracy reconciliation for shared summaries, topk_reuse
    accuracy/                analytical model: estimators, composition, allocation, evidence
  physical-optimizer/        Stage 2  → CandidatePhysicalASAPDAGs
    materialization/         summary_maintenance_lifecycle, pane_sharing, storage_io
    implementation/          query_physical_lowering + asap-physical-operators::{plan, physical_planner}
  plan-selection/            Stage 3  → one PhysicalASAPDAG
    cost/                    cost_model, analytical_cost, physical_plan_cost_model, empirical_cost,
                             physical_operator_statistics, physical_handoff_cost, recurrence,
                             summary_maintenance_cost
    accuracy/                empirical model: erp, empirical_comparison
  executor/                  Stage 4 reference executor: operators, runtime, sources,
                             summary_kernels, expressions, values
  planner/                   facade running Stages 0–3 (absorbs today's pass/ MajorPass driver)
  devtools/, integration-tests/

pre_asap, post_asap, asap-aware-mapping and asap-physical-operators disappear. Logical and physical candidates are the same OperatorNode DAG under #511, differing only in timing, so LogicalASAPDAG / PhysicalASAPDAG stay in types::ir. Leftover pre-#509 names are renamed in the same step: PostAsapDAG, the pre_asap_sub_dag export key, and tools/dag-viewer/post_asap_fixture.json.

Accuracy placement

#509 splits accuracy by stage:

• Stage 1 (Pass 1) keeps every candidate "not provably unable to meet its accuracy requirement". That proof is analytical: the per-family error bounds and parameter sizing in accuracy/estimators, propagation through exact functions (composition), and error-budget allocation (allocation). Today replacement.rs is the main consumer.
• Stage 1 (Pass 2): "a shared summary must meet the strictest accuracy requirement". That is accuracy/reconciliation.
• Stage 3 is "the only stage that uses the deployment's cost and accuracy models… accuracy is estimated by the deployment's accuracy model, not assumed from a summary's nominal bound". That is the empirical side: erp and empirical_comparison.
• Stage 2 makes no accuracy decisions. It preserves OperatorNode.guarantee and records window-summary evidence, such as an EH boundary falling inside a bucket (summary_maintenance_cost/window.rs), for Stage 3.

Stage 3 reads the analytical result from OperatorNode.guarantee, which Stage 1 establishes (#511 §2.2), so plan-selection does not need to call the analytical model.

The deployment's accuracy model plugs into Stage 3 only. Today the pluggable AccuracyModel trait is passed into Pass 1 (SketchAlgorithmStrategy::new_with_planning_inputs), so a deployment can supply composition proofs there. That conflicts with #509. After the move:

• Stage 1 uses only the planner's built-in analytical rules. It does not use AccuracyModel.
• "No analytical rule" is not a proof of infeasibility. Today a missing rule makes callers fail closed for approximate input, so Pass 1 prunes the candidate. Under docs: propose workload-wide planning, summary sharing, and materialization #509, Pass 1 may prune only when a candidate is provably invalid. A candidate with no rule is therefore kept with guarantee = None, meaning unknown, never treated as exact (docs: define unified operators and SQL/PromQL scalar boundaries #511 §2.2).
• Stage 3 accepts or rejects those candidates using the deployment's accuracy model, the trait's new home in plan-selection/accuracy/.

Decisions

• Timing: after refactor(ir): remove the legacy QueryExpr and SummaryExpr IRs; move docs and DAG viewer to the unified IR #543 removes the legacy graphs, so far less code moves.
• Shape: crates per stage, not modules inside today's crates.
• asap-types keeps its crate name; the module names carry the meaning.
• Accuracy model: analytical in Stage 1, the deployment's (empirical, pluggable) in Stage 3 only. Stage 1 keeps candidates that have no analytical rule, with guarantee = None.
• Executor: stays in this repo as a separate crate. No stage crate depends on it, only integration-tests and devtools. Today asap-aware-mapping already has no dependency on asap-physical-operators.

Plan

One PR per step, each a pure move plus import updates:

1. types modules (ir/, workload/, physical/; remove pre_asap / post_asap)
2. logical-optimizer
3. physical-optimizer
4. plan-selection
5. executor, then retire asap-aware-mapping / asap-physical-operators

The module mapping above comes from module docs and call sites. Each step re-checks it file by file.

Follows #543 · Stages: #509 · IR: #511 · Tracker: #528

Update (2026-10-03): verified move manifest and decisions

A file-by-file check against call sites on #541' found that this is not a pure move. Decisions:

• Placement corrections:
  • asap-physical-operators::{plan, physical_planner} go to executor, not physical-optimizer. They compile plans into runtime operators, and plan and runtime import each other.
  • query_physical_lowering and storage_io go to plan-selection. Only cost code calls them, and they import analytical_cost.
  • Split execution_data_state: ExecutionTiming / ExecutionDataState go to types::ir::properties, because ir/node, ir/timing and ir/physical_export use them. The rest goes to types::physical.
  • Delete modules with no callers: pane_sharing, erp, empirical_comparison and post_asap::query_time.
• Maintenance lifecycle is deleted, not moved. Stage 2 materialization (docs: propose workload-wide planning, summary sharing, and materialization #509) decides per sub-DAG whether to materialize, and at ingestion or query time. Done in the rebuilt refactor(ir): remove the legacy QueryExpr and SummaryExpr IRs; move docs and DAG viewer to the unified IR #543:
  • Deleted (about 11.6k lines): summary_maintenance_lifecycle, summary_maintenance_cost/, summary_maintenance_dag_export, post_asap::{summary_maintenance, summary_maintenance_lifecycle}, SummaryWindowFramework, the lifecycle e2e test, the viewer's lifecycle UI and its proposal doc.
  • Kept: ir/timing (LifecycleAssignment is renamed MaterializationAssignment), and the pane primitives for Pass 2 window composition.
  • Until Stage 2 materialization exists, every node defaults to query time.
• Backward dependencies:
  • Deleting the lifecycle removes the Stage 2 ↔ Stage 3 cycle.
  • The remaining cycle is {cost_model, replacement, recurrence, exact_composition}. Phase B breaks it by moving only the CostModel trait and the recurrence types into types (types only).
  • Removing cost-based ranking from Stage 1 (docs: propose workload-wide planning, summary sharing, and materialization #509: only Stage 3 uses the cost model) comes with Phase C, when the stage pipeline replaces MajorPass.
• Phase C runs in parallel with Phase B. Stage 2 (physical_candidates) and Stage 3 (plan_selection) start as new modules and move with the rest.
• Import rewrite volume: about 686 use lines in 228 files for the types step; the other steps are small.

Update (2026-10-04): Stage 1 stops using the cost model

B0 ("move only the CostModel trait into types") is not possible. The trait signature needs Stage 1 types that carry behavior, and two default methods call Stage 1 sizing and Stage 3 recurrence logic. Decisions:

• Q36 (a). Following docs: propose workload-wide planning, summary sharing, and materialization #509, Stage 1 does not use the cost model; only Stage 3 does. This is the same rule as Q22 for accuracy.
  • MajorPass (cost-ranked Pass 1) is retired, and the planner facade runs the stage pipeline: Stage 1 (feat(planner): Stage 1 Pass 1 per-computation candidate generation #561/feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575) → Stage 2 → Stage 3 (feat(planner): physical ASAP-aware optimization and plan selection for candidate logical ASAP DAGs #576).
  • The cost-dependent parts of replacement, exact_composition and grouping move to plan-selection or are deleted. This changes planner behavior and lands as its own PR before B2.
• Q37. PlanningModels, the bundled cost and accuracy models, moves to plan-selection. The facade imports it from there.