From 0cde14c66ae101cda81b30e659b9a5d3aec3df11 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 12 Oct 2026 19:53:07 +0000 Subject: [PATCH 1/4] docs: align #509/#511/#573 docs with the decided vocabulary and Pass 1 candidates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - operator-sharing.md: ReadPopulation { readout } -> EvaluatePopulation { evaluation } (matches crates/types/src/ir/asap.rs); "readout" -> "evaluation" where it names reading summary state; add the missing `filter` field to the SummaryAgg sketch. - asap-primitive-schema.md: "readout" -> "evaluation" where it names the operation (intent table header, §6.5 heading and its anchor/table row, top-k evaluation, guarantee of an evaluation). Verb "read out" kept, as in the code comments. Coverage cache text unchanged. - planner-layering.md: define an eligible sub-DAG at its first use; add the exact accumulator state candidate to the Pass 1 table and text; recompute Example 1 (Pass 1: 2 x 2 x 2 x 3 = 24, Pass 2: 24 x 2 x 3 x 3 = 432, Stage 2: 24 x 52 = 1,248). - planner-layering.md: every aggregation is its own Pass 1 target, including nested ones (Q1's per-series rate and per-job sum, Q2's per-series sum_over_time and per-job top-k), as in #561's enumerate_local_logical_candidates; the Algorithm 1 example and diagram show the four sub-DAGs, and Example 1 lists options per target. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --- .../proposals/asap-primitive-schema.md | 22 ++--- .../design_docs/proposals/operator-sharing.md | 22 ++--- .../design_docs/proposals/planner-layering.md | 95 +++++++++++++------ 3 files changed, 89 insertions(+), 50 deletions(-) diff --git a/docs/design_docs/proposals/asap-primitive-schema.md b/docs/design_docs/proposals/asap-primitive-schema.md index 6e51cf27e..7a7ffb84d 100644 --- a/docs/design_docs/proposals/asap-primitive-schema.md +++ b/docs/design_docs/proposals/asap-primitive-schema.md @@ -84,7 +84,7 @@ Based on our requirement, each field should contain the following information. - **Statistical aggregation intents** (`AggIntent`, `crates/types/src/pre_asap/agg_intent.rs`): which aggregate the state can answer, and how it is read out. - | Intent | Candidate types | Readout | + | Intent | Candidate types | Evaluation | |---|---|---| | `Quantile` | `Sketch`: `Kll`, `DDSketch` | `SummaryEstimate(Quantile { q })` | | `Cardinality` | `Sketch`: `Hll`, `Theta`, `Kmv`, `UnivMon` (one column only) | `SummaryEstimate(Cardinality)` | @@ -93,7 +93,7 @@ Based on our requirement, each field should contain the following information. | `FrequencyL2`, `FrequencyEntropy` | `Sketch`: `UnivMon` | `SummaryEstimate(FrequencyL2 \| FrequencyEntropy)` | | `Sum`, `Count`, `Min`, `Max`, `Rate`, `IRate`, `Increase` (exact) | `ExactAggregate(ExactKind, …)` | `FinalizeExactAccumulator` | - The sketch candidates are `summary_candidates(intent)` in `crates/asap-aware-mapping/src/replacement.rs`; exact accumulators come from `exact_realization` there (`Count`) and from `function_rules.rs` (the others); the readouts are `SketchStatistic` ([§6.5](#65-update-input-and-readouts-post_asapsketchrs-post_asapmaintained_populationrs)). + The sketch candidates are `summary_candidates(intent)` in `crates/asap-aware-mapping/src/replacement.rs`; exact accumulators come from `exact_realization` there (`Count`) and from `function_rules.rs` (the others); the evaluations are `SketchStatistic` ([§6.5](#65-update-input-and-evaluations-post_asapsketchrs-post_asapmaintained_populationrs)). - **Time window aggregation intents**: whether states built over smaller windows can answer a larger one. This depends on how the family combines states: - **Merge** (`SummaryMerge`, §5.6): states over disjoint panes combine into the state of their union, e.g. two 1-minute KLL states answer a 2-minute quantile. Requires a mergeable family. @@ -543,10 +543,10 @@ Both need the same KLL state, so the planner builds it once (sub-DAG sharing, [P | Question | Answer | |---|---| -| What comes out? | the same columns, with `state` replaced by the answer: `quantile` Float64 here. Counts and cardinalities are Int64 (Float64 when the state was built per series, `PerEntity`). A top-k readout is a `topk` Utf8 field today; #579 changes it to return the selected rows: the partition keys, the item identity columns and a Float64 `value` score | +| What comes out? | the same columns, with `state` replaced by the answer: `quantile` Float64 here. Counts and cardinalities are Int64 (Float64 when the state was built per series, `PerEntity`). A top-k evaluation is a `topk` Utf8 field today; #579 changes it to return the selected rows: the partition keys, the item identity columns and a Float64 `value` score | | When is it rejected? | the input is not a sketch, or the sketch cannot answer the question. For example, asking a KLL for a cardinality | | What is its coverage? | none: the output is a value | -| State or value? | value. The node carries the readout's error bound | +| State or value? | value. The node carries the evaluation's error bound | ### 5.3 `FinalizeExactAccumulator`: exact state → value @@ -870,7 +870,7 @@ OperatorNode §6.1 │ └── fields[i].dtype: FieldDataType ◀───────┘ §6.3 (the same type: the output │ └── Sketch(SketchKind, GroupingStrategy) `state` field has type `family`) │ └── category, algorithm, params §6.4 -├── guarantee: ResultGuarantee §6.5 (accuracy of a readout) +├── guarantee: ResultGuarantee §6.5 (accuracy of an evaluation) └── coverage() → SummaryCoverage §6.6 └── definition: OperatorNode (again a node, so the same structure) ``` @@ -881,10 +881,10 @@ OperatorNode §6.1 | §6.2 ASAP operators | `ASAPOp` | `OperatorNode.operator` | | §6.3 Schema and field types | `Schema`, `Field`, `FieldDataType` | every node's `schema`; `SummaryAgg.family` | | §6.4 Summary-family parameters | `SketchKind`, `SketchParams`, `GroupingStrategy`, exact/sample/wavelet/model params | the payload of each non-plain `FieldDataType` variant | -| §6.5 Update input and readouts | `SummaryUpdate`, `SketchStatistic`, `MaintainedPopulation`, `PopulationStatistic` | fields of the `ASAPOp` variants in §6.2 | +| §6.5 Update input and evaluations | `SummaryUpdate`, `SketchStatistic`, `MaintainedPopulation`, `PopulationStatistic` | fields of the `ASAPOp` variants in §6.2 | | §6.6 Summary coverage | `SummaryCoverage`, `SelectionBox`, `Constraint` | `OperatorNode::coverage()` on summary nodes | -**Example: the p99 latency state of §5.1.** The `SummaryAgg` node (§6.1) holds `ASAPOp::SummaryAgg` (§6.2) with `family = Sketch(SketchKind::new(Kll, Kll { k: 200 }), PerSubpopulationInstance)` (§6.3, §6.4) and `input = SummaryUpdate::column(latency)` (§6.5). Its `schema` (§6.3) is `(job Utf8, state Sketch(KLL k=200))`: the `state` field has the same type as `family`. A `SummaryEstimate` above it holds `query = Quantile { q: 0.99 }` (§6.5), and its node carries the `guarantee` of the readout. `coverage()` on the `SummaryAgg` node returns a `SummaryCoverage` (§6.6). +**Example: the p99 latency state of §5.1.** The `SummaryAgg` node (§6.1) holds `ASAPOp::SummaryAgg` (§6.2) with `family = Sketch(SketchKind::new(Kll, Kll { k: 200 }), PerSubpopulationInstance)` (§6.3, §6.4) and `input = SummaryUpdate::column(latency)` (§6.5). Its `schema` (§6.3) is `(job Utf8, state Sketch(KLL k=200))`: the `state` field has the same type as `family`. A `SummaryEstimate` above it holds `query = Quantile { q: 0.99 }` (§6.5), and its node carries the `guarantee` of the evaluation. `coverage()` on the `SummaryAgg` node returns a `SummaryCoverage` (§6.6). ### 6.1 DAG node (`crates/types/src/ir/node.rs`) @@ -1220,12 +1220,12 @@ pub enum WaveletKind { Haar } pub enum WaveletParams { Haar { coeffi pub enum StatModelKind { Parametric } pub enum StatModelParams { Parametric { family: String } } ``` -### 6.5 Update input and readouts (`post_asap/sketch.rs`, `post_asap/maintained_population.rs`) +### 6.5 Update input and evaluations (`post_asap/sketch.rs`, `post_asap/maintained_population.rs`) This part of the code answers two questions about summary state: 1. **Update input** (`SummaryUpdate`): when a `SummaryAgg` reads one input row, what does it add to the state? It adds a **weight**, optionally under an **item** key. -2. **Readouts**: once the state is built, what can be asked of it? `SketchStatistic` is what `SummaryEstimate` asks a sketch (§5.2). `MaintainedPopulation` and `PopulationStatistic` are the exact-population counterpart, used by `MaintainPopulation` and `EvaluatePopulation` (§5.4, §5.5). +2. **Evaluations**: once the state is built, what can be asked of it? `SketchStatistic` is what `SummaryEstimate` asks a sketch (§5.2). `MaintainedPopulation` and `PopulationStatistic` are the exact-population counterpart, used by `MaintainPopulation` and `EvaluatePopulation` (§5.4, §5.5). **Update input examples** (as the planner builds them in `asap-aware-mapping`): @@ -1237,7 +1237,7 @@ This part of the code answers two questions about summary state: `weight_domain` matters because some sketches (e.g. Count-Min) are only accurate when weights are never negative. The planner records why a weight is non-negative; if it cannot prove it, the weight counts as possibly negative. -**Readout examples:** `Quantile { q: 0.99 }` reads p99 from a KLL; `PointCount { key: endpoint, value: Some("checkout") }` reads how often `checkout` occurred from a Count-Min; `TopK { k: 5 }` reads the 5 heaviest items. +**Evaluation examples:** `Quantile { q: 0.99 }` reads p99 from a KLL; `PointCount { key: endpoint, value: Some("checkout") }` reads how often `checkout` occurred from a Count-Min; `TopK { k: 5 }` reads the 5 heaviest items. ```rust /// What one input row adds to the state. @@ -1324,7 +1324,7 @@ pub struct CurrentSeriesInput { } ``` -A finalized value's accuracy statement is `ResultGuarantee` (`post_asap/guarantee.rs`): `metric` (which error is measured, e.g. rank error), `bound` (the error bound), `failure_probability` (the chance the bound does not hold) and `provenance` (which estimates the bound came from). It is attached to readout and finalized nodes, and to `MaintainPopulation` nodes (whose population is exact); never to sketch or accumulator state. +A finalized value's accuracy statement is `ResultGuarantee` (`post_asap/guarantee.rs`): `metric` (which error is measured, e.g. rank error), `bound` (the error bound), `failure_probability` (the chance the bound does not hold) and `provenance` (which estimates the bound came from). It is attached to evaluation and finalized nodes, and to `MaintainPopulation` nodes (whose population is exact); never to sketch or accumulator state. ### 6.6 Summary coverage (`crates/types/src/ir/summary_coverage.rs`) diff --git a/docs/design_docs/proposals/operator-sharing.md b/docs/design_docs/proposals/operator-sharing.md index c9ab193a4..8bd8d499a 100644 --- a/docs/design_docs/proposals/operator-sharing.md +++ b/docs/design_docs/proposals/operator-sharing.md @@ -80,12 +80,12 @@ An operator owns its scalar expressions and references input nodes through `Rc`. Either operation category can consume the other's outputs when the input contract permits it. `NonASAP` describes one operation, not its entire sub-DAG. Frontend DAGs contain only NonASAP operations; ASAP optimization may -introduce state construction and readout. +introduce state construction and evaluation. | Category | Meaning | All operations | |---|---|---| | `Operator::NonASAP(NonASAPOp)` | Ordinary query operations that transform, combine or aggregate data | `Scan`, `Values`, `Filter`, `Project`, `Aggregate`, `Join`, `SetOp`, `Concat`, `Dedup`, `Sort`, `Limit`, `BinaryOp`, `SQLWindowFunc`, `TimeRange`, `TimeShift`, `PromqlVectorFromScalar`, `PromqlRelabel`, `PromqlInfoEnrich`, `PromqlSeriesSample`, `PromqlSubquery` | -| `Operator::ASAP(ASAPOp)` | Operations on summary state and its results, including reserved operations | `SummaryAgg`, `SummaryEstimate`, `SummaryMerge`, `SummarySubtract`, `SummaryDelete`, `SummaryJoin`, `FinalizeExactAccumulator`, `MaintainPopulation`, `ReadPopulation`, `Extension` | +| `Operator::ASAP(ASAPOp)` | Operations on summary state and its results, including reserved operations | `SummaryAgg`, `SummaryEstimate`, `SummaryMerge`, `SummarySubtract`, `SummaryDelete`, `SummaryJoin`, `FinalizeExactAccumulator`, `MaintainPopulation`, `EvaluatePopulation`, `Extension` | `CurrentTimestamp`, `EvalTimestamp` and `PromqlScalarFromVector` belong to `ScalarExpr`, defined in the [companion proposal](decoupling_op_and_expr.md#22-scalar-expressions). @@ -148,7 +148,7 @@ Ordinary payload fields have these roles: survive optimization. The optional concatenation key records a discriminator that distinguishes branches together with their within-branch key. -`ASAPOp` describes state construction, state operations and readout separately. +`ASAPOp` describes state construction, state operations and evaluation separately. `FieldDataType` (§2.1) types every output field; state-producing operations use its summary or exact-accumulator cases, never its `Plain` case. @@ -156,14 +156,14 @@ summary or exact-accumulator cases, never its `Plain` case. enum ASAPOp { SummaryAgg { child: Rc, family: FieldDataType, input: SummaryUpdate, - reduction: Reduction, grouping: GroupingStrategy, + reduction: Reduction, grouping: GroupingStrategy, filter: Option, }, SummaryEstimate { summary_input: Rc, query: SketchStatistic, }, FinalizeExactAccumulator { child: Rc }, MaintainPopulation { child: Rc, population: MaintainedPopulation }, - ReadPopulation { child: Rc, readout: PopulationStatistic }, + EvaluatePopulation { child: Rc, evaluation: PopulationStatistic }, // Reserved operations; semantics and support require further design. SummaryMerge { children: Vec> }, @@ -176,7 +176,7 @@ enum ASAPOp { } ``` -The summary fields distinguish state construction and readout: +The summary fields distinguish state construction and evaluation: | Field | Design meaning | |---|---| @@ -184,7 +184,7 @@ The summary fields distinguish state construction and readout: | `input` | The item identity and observation or weight supplied to a state update | | `reduction` | Which input entities contribute to each logical result | | `grouping` | Whether those groups use separate state instances or a supported shared structure | -| `query` / `readout` | The result requested from summary or maintained-population state | +| `query` / `evaluation` | The result requested from summary or maintained-population state | | `population` | The population whose membership and values are maintained | The common `OperatorNode` fields are declared in the overview above and explained @@ -212,7 +212,7 @@ function, explicitly written by the query author. This proposal does not insert it automatically: `sum(up)` alone is a valid query returning an instant vector. The scalar expression `PromqlScalarFromVector` represents that function and references its result to obtain one number. A valid ASAP rewrite may replace that producer with -a summary readout, preserving the required vector and accuracy semantics; it cannot +a summary evaluation, preserving the required vector and accuracy semantics; it cannot substitute raw summary state. Ordinary expressions such as `price * 2` reference columns and literals, not a query sub-DAG. @@ -369,8 +369,8 @@ Every operator output, before and after optimization, uses one `Schema` whose `Field`s are typed by `FieldDataType`: `Plain(DataType)` for a readable value, or the family, algorithm and parameters of summary or exact-accumulator state. `OperatorResultKind` marks state outputs, and state becomes a value only through -an explicit readout. The schema model, `ColumnRef` versus `ColumnId`, the -validation entry points and the readout boundary are specified in +an explicit evaluation. The schema model, `ColumnRef` versus `ColumnId`, the +validation entry points and the evaluation boundary are specified in [Schema and physical data for ASAP primitives](asap-primitive-schema.md). ### 2.2 Preserve existing accuracy semantics @@ -383,7 +383,7 @@ not make it exact; unknown accuracy must not be treated as exactness. The common node reuses `guarantee: Option` from `SummaryNode`. `Some` records an established guarantee; `None` covers an unassessed or unknown -result, or state whose accuracy is only established at readout. Exactness must be +result, or state whose accuracy is only established at evaluation. Exactness must be explicitly established using the existing model. This proposal introduces no new accuracy metric or guarantee-calculation workflow. diff --git a/docs/design_docs/proposals/planner-layering.md b/docs/design_docs/proposals/planner-layering.md index 3102e8b3a..c9565e452 100644 --- a/docs/design_docs/proposals/planner-layering.md +++ b/docs/design_docs/proposals/planner-layering.md @@ -297,17 +297,29 @@ For each eligible sub-DAG, Pass 1 identifies its computation semantics, applies rewrite rules, and generates every candidate that is not provably unable to meet its accuracy requirement. +A sub-DAG is **eligible** when its root is an aggregation (an `Aggregate` +operator) whose intent has at least one realization in the summary-family +capabilities `F`: exact accumulator state or a summary family. Each measure of +a multi-measure aggregate is its own target. Nested aggregations are separate +targets: in `sum by (job) (rate(x[1m]))`, the per-series `rate` and the +per-`job` `sum` are each a target, and their options multiply. + Example for summary candidates: | Original computation | Per-computation candidates | |---|---| -| `Sum(x) by (g)` | Exact grouped sum | +| `Sum(x) by (g)` | Exact grouped sum, exact Sum accumulator | | `TopK(k, x) by (g)` | Exact sort and limit per group, Count-Min Sketch with a top-*k* heap per group, Hydra over all groups | | `Distinct(x)` | Exact distinct, a specialized distinct summary, UnivMon | | `Entropy(x)` | Exact entropy, a specialized entropy summary, UnivMon | | `L2(x)` | Exact L2 norm, a specialized norm summary, UnivMon | | `Quantile(x, window)` | Exact quantile, KLL over the requested window | +For an intent with an exact mergeable accumulator (Count, Sum, Min, Max, Rate, +IRate, Increase), Pass 1 also offers **exact accumulator state**: a `SummaryAgg` +with an exact family → `FinalizeExactAccumulator`, in addition to the plain exact +computation and the summary families. + Each candidate records its input expression, filter, grouping, window, supported estimates and accuracy requirement. Pass 2 uses these to decide whether candidates can share a summary node. @@ -362,37 +374,59 @@ Output: C1 — CandidateLogicalASAPDAGs with each sub-DAG replaced independentl *Example for Algorithm 1* (the workload of [Example 1](#example-1-aggregation-over-dimensions--the-candidate-set-through-every-stage)). Q1 needs an exact answer, so every summary replacement for it is pruned and only the exact -computation remains. Q2 tolerates error and gets three replacements. The -Cartesian product gives 1 × 3 = 3 whole-workload candidates. +computation and the exact accumulator state remain for each of its two +sub-DAGs, the per-series rate (s₁) and the per-`job` sum (s₂). Q2's per-series +`sum_over_time` (s₃) has no summary family and gets the same two. Q2's top-*k* +(s₄) tolerates error and gets three replacements. The +Cartesian product gives 2 × 2 × 2 × 3 = 24 whole-workload candidates. ```mermaid flowchart LR subgraph IN["d: one workload LogicalDAG"] direction TB - s1["s₁ = Q1 · sum by (job) (rate(…[1m]))
accuracy: exact"]:::exact - s2["s₂ = Q2 · topk by (job) (10, sum_over_time(…[1m]))
accuracy: ε = 0.01"]:::exact + s1["s₁ = Q1 · rate(…[1m]) per series
accuracy: exact"]:::exact + s2["s₂ = Q1 · sum by (job) (s₁)
accuracy: exact"]:::exact + s3["s₃ = Q2 · sum_over_time(…[1m]) per series
accuracy: ε = 0.01"]:::exact + s4["s₄ = Q2 · topk by (job) (10, s₃)
accuracy: ε = 0.01"]:::exact end subgraph R1["Replacements(s₁)"] direction TB - a1["Exact rate + sum"]:::exact - a2["summary candidates
✗ pruned: cannot be exact"]:::pruned + a1["Exact rate"]:::exact + a3["Exact Rate accumulator
per series"]:::exact end subgraph R2["Replacements(s₂)"] + direction TB + d1["Exact sum"]:::exact + d3["Exact Sum accumulator
per job"]:::exact + a2["summary candidates
✗ pruned: cannot be exact"]:::pruned + end + subgraph R3["Replacements(s₃)"] + direction TB + e1["Exact sum_over_time"]:::exact + e3["Exact Sum accumulator
per series"]:::exact + end + subgraph R4["Replacements(s₄)"] direction TB b1["Exact sort + limit per job"]:::exact b2["CMS + top-10 heap per job"]:::summary b3["Hydra over all jobs"]:::summary end - subgraph OUT["C1 = Replacements(s₁) × Replacements(s₂) = 3 candidates"] + subgraph OUT["C1 = Replacements(s₁) × … × Replacements(s₄) = 2 × 2 × 2 × 3 = 24 candidates"] direction TB - c1["① exact Q1 + exact Q2"]:::ok - c2["② exact Q1 + CMS Q2"]:::ok - c3["③ exact Q1 + Hydra Q2"]:::ok + c1["① exact s₁, s₂, s₃ + exact s₄"]:::ok + c2["② exact s₁, s₂, s₃ + CMS s₄"]:::ok + c3["③ exact s₁, s₂, s₃ + Hydra s₄"]:::ok + c4["…"]:::ok + c24["㉔ accumulators s₁, s₂, s₃ + Hydra s₄"]:::ok end s1 --> R1 s2 --> R2 + s3 --> R3 + s4 --> R4 R1 --> OUT R2 --> OUT + R3 --> OUT + R4 --> OUT classDef data fill:#f1f3f4,stroke:#5f6368,color:#000; classDef exact fill:#fff,stroke:#5f6368,color:#000; classDef summary fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#000; @@ -916,11 +950,15 @@ flowchart TB classDef estimate fill:#e6f4ea,stroke:#188038,color:#000; ``` -**Stage 1, Pass 1: 3 candidates.** Pass 1 finds per-computation options for each query: +**Stage 1, Pass 1: 24 candidates.** Pass 1 finds per-computation options for each query: -* **Q1** has one option, the exact per-series rate and per-`job` sum. Its - accuracy requirement is exact, so no summary qualifies. -* **Q2** has three options. The exact one keeps a per-series sum and sorts +* **Q1** has two targets, the per-series `rate` and the per-`job` `sum`. Each + has two options: the exact computation, and **exact accumulator state** + (`SummaryAgg` with an exact family → `FinalizeExactAccumulator`), so Q1 has + 2 × 2 = 4 options. Its accuracy requirement is exact, so no summary qualifies. +* **Q2** has two targets. The per-series `sum_over_time` has the same two + options, exact and exact Sum accumulator state. The per-`job` top-*k* has + three options, so Q2 has 2 × 3 = 6 options. The exact one keeps a per-series sum and sorts within each `job`, which is costly at one million Zipf-distributed series. The `EpsilonDelta` target also admits a **Count-Min Sketch with a top-*k* heap per `job`**, and **Hydra over the whole `job` column**, where one sketch @@ -928,7 +966,7 @@ flowchart TB ```mermaid flowchart LR - subgraph C["Q2's three per-computation options"] + subgraph C["Q2 top-k's three per-computation options"] direction TB subgraph E["Exact"] direction LR @@ -952,16 +990,17 @@ flowchart LR classDef estimate fill:#e6f4ea,stroke:#188038,color:#000; ``` -Combining them gives 1 × 3 = 3 workload candidates: +Combining them gives 4 × 6 = 24 workload candidates: -| Logical candidate | Q1 | Q2 | +| Query | Target | Options | |---|---|---| -| Exact | exact | exact | -| Count-Min | exact | Count-Min + heap | -| Hydra | exact | Hydra | +| Q1 | per-series `rate` | exact, exact Rate accumulator | +| Q1 | per-`job` `sum` | exact, exact Sum accumulator | +| Q2 | per-series `sum_over_time` | exact, exact Sum accumulator | +| Q2 | per-`job` top-*k* | exact, Count-Min + heap, Hydra | -**Stage 1, Pass 2: 54 candidates.** Pass 2 applies two ASAP-aware CSE rules -to each of the 3 candidates, and keeps every original: +**Stage 1, Pass 2: 432 candidates.** Pass 2 applies two ASAP-aware CSE rules +to each of the 24 candidates, and keeps every original: * **Identical-expression rule.** Both queries read the same range selector, `http_requests_total[1m]`, so Pass 2 adds a variant in which Q1 and Q2 share @@ -981,7 +1020,7 @@ to each of the 3 candidates, and keeps every original: Each Pass 1 candidate therefore has 2 input choices (separate or shared) × 3 window forms for Q1 (none, sliding, tumbling) × 3 for Q2, so stage 1 outputs -3 × 2 × 3 × 3 = 54 logical candidates. +24 × 2 × 3 × 3 = 432 logical candidates. The window-composition rule applied to Q2 in the Hydra candidate: @@ -1005,7 +1044,7 @@ flowchart LR classDef estimate fill:#e6f4ea,stroke:#188038,color:#000; ``` -**Stage 2: 156 candidates.** Stage 2 picks, for each query, when its state is +**Stage 2: 1,248 candidates.** Stage 2 picks, for each query, when its state is computed and whether it is kept. What it can choose depends on the query's window form from Pass 2: @@ -1082,11 +1121,11 @@ flowchart LR ``` Across the three window forms, each query has 1 + 2 + 3 = 6 physical options, -so each Q2 option gives 6 × 6 = 36 combinations. The shared-input variant only +so each Pass 1 candidate gives 6 × 6 = 36 combinations. The shared-input variant only changes the plan when both queries read raw samples at query time, which 4 of the 6 options do (all except the two ingestion-time ones). That adds -4 × 4 = 16 shared-input plans, for 52 plans per Q2 option and -3 × 52 = 156 physical candidates. +4 × 4 = 16 shared-input plans, for 52 plans per Pass 1 candidate and +24 × 52 = 1,248 physical candidates. **Stage 3: 1 plan.** Selection first rejects invalid candidates. The deployment's accuracy model checks the summary candidates against ε = 0.01, From 72fa55546c41f01540d670736b2db897a1eb0bb8 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 12 Oct 2026 20:04:55 +0000 Subject: [PATCH 2/4] docs: leave multi-measure targets out of the eligible sub-DAG definition Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --- docs/design_docs/proposals/planner-layering.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/proposals/planner-layering.md b/docs/design_docs/proposals/planner-layering.md index c9565e452..e1d80436b 100644 --- a/docs/design_docs/proposals/planner-layering.md +++ b/docs/design_docs/proposals/planner-layering.md @@ -299,8 +299,8 @@ meet its accuracy requirement. A sub-DAG is **eligible** when its root is an aggregation (an `Aggregate` operator) whose intent has at least one realization in the summary-family -capabilities `F`: exact accumulator state or a summary family. Each measure of -a multi-measure aggregate is its own target. Nested aggregations are separate +capabilities `F`: exact accumulator state or a summary family. Nested +aggregations are separate targets: in `sum by (job) (rate(x[1m]))`, the per-series `rate` and the per-`job` `sum` are each a target, and their options multiply. From cb3e45de7db1a6bc1e0e253e98f4228cd028f7dd Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 12 Oct 2026 20:08:33 +0000 Subject: [PATCH 3/4] docs: one exact option per Pass 1 target Each Pass 1 target gets a single exact option: exact accumulator state when the intent has a buildable mergeable exact accumulator, otherwise the plain exact computation. Example 1 returns to 3 / 54 / 156 with four targets. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --- .../design_docs/proposals/planner-layering.md | 73 +++++++++---------- 1 file changed, 36 insertions(+), 37 deletions(-) diff --git a/docs/design_docs/proposals/planner-layering.md b/docs/design_docs/proposals/planner-layering.md index e1d80436b..504da8190 100644 --- a/docs/design_docs/proposals/planner-layering.md +++ b/docs/design_docs/proposals/planner-layering.md @@ -308,17 +308,20 @@ Example for summary candidates: | Original computation | Per-computation candidates | |---|---| -| `Sum(x) by (g)` | Exact grouped sum, exact Sum accumulator | +| `Sum(x) by (g)` | Exact grouped sum (exact Sum accumulator state) | | `TopK(k, x) by (g)` | Exact sort and limit per group, Count-Min Sketch with a top-*k* heap per group, Hydra over all groups | | `Distinct(x)` | Exact distinct, a specialized distinct summary, UnivMon | | `Entropy(x)` | Exact entropy, a specialized entropy summary, UnivMon | | `L2(x)` | Exact L2 norm, a specialized norm summary, UnivMon | | `Quantile(x, window)` | Exact quantile, KLL over the requested window | -For an intent with an exact mergeable accumulator (Count, Sum, Min, Max, Rate, -IRate, Increase), Pass 1 also offers **exact accumulator state**: a `SummaryAgg` -with an exact family → `FinalizeExactAccumulator`, in addition to the plain exact -computation and the summary families. +Each target gets one exact option. If its intent has an exact mergeable +accumulator (Count, Sum, Min, Max, Rate, IRate, Increase; avg via exact sum and +count accumulator states) and the accumulator can be built, the exact option is +**exact accumulator state**: a `SummaryAgg` with an exact family → +`FinalizeExactAccumulator`. Otherwise, for example when a filter or HAVING +prevents the accumulator or the intent has none, it is the plain exact +computation. Whether the accumulator is materialized is decided in Stage 2. Each candidate records its input expression, filter, grouping, window, supported estimates and accuracy requirement. Pass 2 uses these to decide @@ -374,11 +377,12 @@ Output: C1 — CandidateLogicalASAPDAGs with each sub-DAG replaced independentl *Example for Algorithm 1* (the workload of [Example 1](#example-1-aggregation-over-dimensions--the-candidate-set-through-every-stage)). Q1 needs an exact answer, so every summary replacement for it is pruned and only the exact -computation and the exact accumulator state remain for each of its two -sub-DAGs, the per-series rate (s₁) and the per-`job` sum (s₂). Q2's per-series -`sum_over_time` (s₃) has no summary family and gets the same two. Q2's top-*k* -(s₄) tolerates error and gets three replacements. The -Cartesian product gives 2 × 2 × 2 × 3 = 24 whole-workload candidates. +option remains for each of its two sub-DAGs, the per-series rate (s₁) and the +per-`job` sum (s₂): its exact accumulator state. Q2's per-series +`sum_over_time` (s₃) has no summary family and gets one option, its exact Sum +accumulator state. Q2's top-*k* (s₄) tolerates error and gets three +replacements. The Cartesian product gives 1 × 1 × 1 × 3 = 3 whole-workload +candidates. ```mermaid flowchart LR @@ -391,18 +395,15 @@ flowchart LR end subgraph R1["Replacements(s₁)"] direction TB - a1["Exact rate"]:::exact a3["Exact Rate accumulator
per series"]:::exact end subgraph R2["Replacements(s₂)"] direction TB - d1["Exact sum"]:::exact d3["Exact Sum accumulator
per job"]:::exact a2["summary candidates
✗ pruned: cannot be exact"]:::pruned end subgraph R3["Replacements(s₃)"] direction TB - e1["Exact sum_over_time"]:::exact e3["Exact Sum accumulator
per series"]:::exact end subgraph R4["Replacements(s₄)"] @@ -411,13 +412,11 @@ flowchart LR b2["CMS + top-10 heap per job"]:::summary b3["Hydra over all jobs"]:::summary end - subgraph OUT["C1 = Replacements(s₁) × … × Replacements(s₄) = 2 × 2 × 2 × 3 = 24 candidates"] + subgraph OUT["C1 = Replacements(s₁) × … × Replacements(s₄) = 1 × 1 × 1 × 3 = 3 candidates"] direction TB - c1["① exact s₁, s₂, s₃ + exact s₄"]:::ok - c2["② exact s₁, s₂, s₃ + CMS s₄"]:::ok - c3["③ exact s₁, s₂, s₃ + Hydra s₄"]:::ok - c4["…"]:::ok - c24["㉔ accumulators s₁, s₂, s₃ + Hydra s₄"]:::ok + c1["① accumulators s₁, s₂, s₃ + exact s₄"]:::ok + c2["② accumulators s₁, s₂, s₃ + CMS s₄"]:::ok + c3["③ accumulators s₁, s₂, s₃ + Hydra s₄"]:::ok end s1 --> R1 s2 --> R2 @@ -950,15 +949,15 @@ flowchart TB classDef estimate fill:#e6f4ea,stroke:#188038,color:#000; ``` -**Stage 1, Pass 1: 24 candidates.** Pass 1 finds per-computation options for each query: +**Stage 1, Pass 1: 3 candidates.** Pass 1 finds per-computation options for each query: * **Q1** has two targets, the per-series `rate` and the per-`job` `sum`. Each - has two options: the exact computation, and **exact accumulator state** - (`SummaryAgg` with an exact family → `FinalizeExactAccumulator`), so Q1 has - 2 × 2 = 4 options. Its accuracy requirement is exact, so no summary qualifies. -* **Q2** has two targets. The per-series `sum_over_time` has the same two - options, exact and exact Sum accumulator state. The per-`job` top-*k* has - three options, so Q2 has 2 × 3 = 6 options. The exact one keeps a per-series sum and sorts + has one option, its **exact accumulator state** (`SummaryAgg` with an exact + family → `FinalizeExactAccumulator`), so Q1 has one option. Its accuracy + requirement is exact, so no summary qualifies. +* **Q2** has two targets. The per-series `sum_over_time` has one option, its + exact Sum accumulator state. The per-`job` top-*k* has three options, so Q2 + has three options. The exact one keeps a per-series sum and sorts within each `job`, which is costly at one million Zipf-distributed series. The `EpsilonDelta` target also admits a **Count-Min Sketch with a top-*k* heap per `job`**, and **Hydra over the whole `job` column**, where one sketch @@ -990,17 +989,17 @@ flowchart LR classDef estimate fill:#e6f4ea,stroke:#188038,color:#000; ``` -Combining them gives 4 × 6 = 24 workload candidates: +Combining them gives 1 × 3 = 3 workload candidates: | Query | Target | Options | |---|---|---| -| Q1 | per-series `rate` | exact, exact Rate accumulator | -| Q1 | per-`job` `sum` | exact, exact Sum accumulator | -| Q2 | per-series `sum_over_time` | exact, exact Sum accumulator | +| Q1 | per-series `rate` | exact Rate accumulator | +| Q1 | per-`job` `sum` | exact Sum accumulator | +| Q2 | per-series `sum_over_time` | exact Sum accumulator | | Q2 | per-`job` top-*k* | exact, Count-Min + heap, Hydra | -**Stage 1, Pass 2: 432 candidates.** Pass 2 applies two ASAP-aware CSE rules -to each of the 24 candidates, and keeps every original: +**Stage 1, Pass 2: 54 candidates.** Pass 2 applies two ASAP-aware CSE rules +to each of the 3 candidates, and keeps every original: * **Identical-expression rule.** Both queries read the same range selector, `http_requests_total[1m]`, so Pass 2 adds a variant in which Q1 and Q2 share @@ -1020,7 +1019,7 @@ to each of the 24 candidates, and keeps every original: Each Pass 1 candidate therefore has 2 input choices (separate or shared) × 3 window forms for Q1 (none, sliding, tumbling) × 3 for Q2, so stage 1 outputs -24 × 2 × 3 × 3 = 432 logical candidates. +3 × 2 × 3 × 3 = 54 logical candidates. The window-composition rule applied to Q2 in the Hydra candidate: @@ -1044,7 +1043,7 @@ flowchart LR classDef estimate fill:#e6f4ea,stroke:#188038,color:#000; ``` -**Stage 2: 1,248 candidates.** Stage 2 picks, for each query, when its state is +**Stage 2: 156 candidates.** Stage 2 picks, for each query, when its state is computed and whether it is kept. What it can choose depends on the query's window form from Pass 2: @@ -1121,11 +1120,11 @@ flowchart LR ``` Across the three window forms, each query has 1 + 2 + 3 = 6 physical options, -so each Pass 1 candidate gives 6 × 6 = 36 combinations. The shared-input variant only +so each Q2 option gives 6 × 6 = 36 combinations. The shared-input variant only changes the plan when both queries read raw samples at query time, which 4 of the 6 options do (all except the two ingestion-time ones). That adds -4 × 4 = 16 shared-input plans, for 52 plans per Pass 1 candidate and -24 × 52 = 1,248 physical candidates. +4 × 4 = 16 shared-input plans, for 52 plans per Q2 option and +3 × 52 = 156 physical candidates. **Stage 3: 1 plan.** Selection first rejects invalid candidates. The deployment's accuracy model checks the summary candidates against ε = 0.01, From 5362d7e313483c642684764db464f9df3e217378 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 12 Oct 2026 20:08:57 +0000 Subject: [PATCH 4/4] docs: reflow the eligible sub-DAG paragraph Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --- docs/design_docs/proposals/planner-layering.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design_docs/proposals/planner-layering.md b/docs/design_docs/proposals/planner-layering.md index 504da8190..9f2395996 100644 --- a/docs/design_docs/proposals/planner-layering.md +++ b/docs/design_docs/proposals/planner-layering.md @@ -300,9 +300,9 @@ meet its accuracy requirement. A sub-DAG is **eligible** when its root is an aggregation (an `Aggregate` operator) whose intent has at least one realization in the summary-family capabilities `F`: exact accumulator state or a summary family. Nested -aggregations are separate -targets: in `sum by (job) (rate(x[1m]))`, the per-series `rate` and the -per-`job` `sum` are each a target, and their options multiply. +aggregations are separate targets: in `sum by (job) (rate(x[1m]))`, the +per-series `rate` and the per-`job` `sum` are each a target, and their options +multiply. Example for summary candidates: