diff --git a/docs/design_docs/proposals/asap-primitive-schema.md b/docs/design_docs/proposals/asap-primitive-schema.md index 6e51cf27..7a7ffb84 100644 --- a/docs/design_docs/proposals/asap-primitive-schema.md +++ b/docs/design_docs/proposals/asap-primitive-schema.md @@ -84,7 +84,7 @@ Based on our requirement, each field should contain the following information. - **Statistical aggregation intents** (`AggIntent`, `crates/types/src/pre_asap/agg_intent.rs`): which aggregate the state can answer, and how it is read out. - | Intent | Candidate types | Readout | + | Intent | Candidate types | Evaluation | |---|---|---| | `Quantile` | `Sketch`: `Kll`, `DDSketch` | `SummaryEstimate(Quantile { q })` | | `Cardinality` | `Sketch`: `Hll`, `Theta`, `Kmv`, `UnivMon` (one column only) | `SummaryEstimate(Cardinality)` | @@ -93,7 +93,7 @@ Based on our requirement, each field should contain the following information. | `FrequencyL2`, `FrequencyEntropy` | `Sketch`: `UnivMon` | `SummaryEstimate(FrequencyL2 \| FrequencyEntropy)` | | `Sum`, `Count`, `Min`, `Max`, `Rate`, `IRate`, `Increase` (exact) | `ExactAggregate(ExactKind, …)` | `FinalizeExactAccumulator` | - The sketch candidates are `summary_candidates(intent)` in `crates/asap-aware-mapping/src/replacement.rs`; exact accumulators come from `exact_realization` there (`Count`) and from `function_rules.rs` (the others); the readouts are `SketchStatistic` ([§6.5](#65-update-input-and-readouts-post_asapsketchrs-post_asapmaintained_populationrs)). + The sketch candidates are `summary_candidates(intent)` in `crates/asap-aware-mapping/src/replacement.rs`; exact accumulators come from `exact_realization` there (`Count`) and from `function_rules.rs` (the others); the evaluations are `SketchStatistic` ([§6.5](#65-update-input-and-evaluations-post_asapsketchrs-post_asapmaintained_populationrs)). - **Time window aggregation intents**: whether states built over smaller windows can answer a larger one. This depends on how the family combines states: - **Merge** (`SummaryMerge`, §5.6): states over disjoint panes combine into the state of their union, e.g. two 1-minute KLL states answer a 2-minute quantile. Requires a mergeable family. @@ -543,10 +543,10 @@ Both need the same KLL state, so the planner builds it once (sub-DAG sharing, [P | Question | Answer | |---|---| -| What comes out? | the same columns, with `state` replaced by the answer: `quantile` Float64 here. Counts and cardinalities are Int64 (Float64 when the state was built per series, `PerEntity`). A top-k readout is a `topk` Utf8 field today; #579 changes it to return the selected rows: the partition keys, the item identity columns and a Float64 `value` score | +| What comes out? | the same columns, with `state` replaced by the answer: `quantile` Float64 here. Counts and cardinalities are Int64 (Float64 when the state was built per series, `PerEntity`). A top-k evaluation is a `topk` Utf8 field today; #579 changes it to return the selected rows: the partition keys, the item identity columns and a Float64 `value` score | | When is it rejected? | the input is not a sketch, or the sketch cannot answer the question. For example, asking a KLL for a cardinality | | What is its coverage? | none: the output is a value | -| State or value? | value. The node carries the readout's error bound | +| State or value? | value. The node carries the evaluation's error bound | ### 5.3 `FinalizeExactAccumulator`: exact state → value @@ -870,7 +870,7 @@ OperatorNode §6.1 │ └── fields[i].dtype: FieldDataType ◀───────┘ §6.3 (the same type: the output │ └── Sketch(SketchKind, GroupingStrategy) `state` field has type `family`) │ └── category, algorithm, params §6.4 -├── guarantee: ResultGuarantee §6.5 (accuracy of a readout) +├── guarantee: ResultGuarantee §6.5 (accuracy of an evaluation) └── coverage() → SummaryCoverage §6.6 └── definition: OperatorNode (again a node, so the same structure) ``` @@ -881,10 +881,10 @@ OperatorNode §6.1 | §6.2 ASAP operators | `ASAPOp` | `OperatorNode.operator` | | §6.3 Schema and field types | `Schema`, `Field`, `FieldDataType` | every node's `schema`; `SummaryAgg.family` | | §6.4 Summary-family parameters | `SketchKind`, `SketchParams`, `GroupingStrategy`, exact/sample/wavelet/model params | the payload of each non-plain `FieldDataType` variant | -| §6.5 Update input and readouts | `SummaryUpdate`, `SketchStatistic`, `MaintainedPopulation`, `PopulationStatistic` | fields of the `ASAPOp` variants in §6.2 | +| §6.5 Update input and evaluations | `SummaryUpdate`, `SketchStatistic`, `MaintainedPopulation`, `PopulationStatistic` | fields of the `ASAPOp` variants in §6.2 | | §6.6 Summary coverage | `SummaryCoverage`, `SelectionBox`, `Constraint` | `OperatorNode::coverage()` on summary nodes | -**Example: the p99 latency state of §5.1.** The `SummaryAgg` node (§6.1) holds `ASAPOp::SummaryAgg` (§6.2) with `family = Sketch(SketchKind::new(Kll, Kll { k: 200 }), PerSubpopulationInstance)` (§6.3, §6.4) and `input = SummaryUpdate::column(latency)` (§6.5). Its `schema` (§6.3) is `(job Utf8, state Sketch(KLL k=200))`: the `state` field has the same type as `family`. A `SummaryEstimate` above it holds `query = Quantile { q: 0.99 }` (§6.5), and its node carries the `guarantee` of the readout. `coverage()` on the `SummaryAgg` node returns a `SummaryCoverage` (§6.6). +**Example: the p99 latency state of §5.1.** The `SummaryAgg` node (§6.1) holds `ASAPOp::SummaryAgg` (§6.2) with `family = Sketch(SketchKind::new(Kll, Kll { k: 200 }), PerSubpopulationInstance)` (§6.3, §6.4) and `input = SummaryUpdate::column(latency)` (§6.5). Its `schema` (§6.3) is `(job Utf8, state Sketch(KLL k=200))`: the `state` field has the same type as `family`. A `SummaryEstimate` above it holds `query = Quantile { q: 0.99 }` (§6.5), and its node carries the `guarantee` of the evaluation. `coverage()` on the `SummaryAgg` node returns a `SummaryCoverage` (§6.6). ### 6.1 DAG node (`crates/types/src/ir/node.rs`) @@ -1220,12 +1220,12 @@ pub enum WaveletKind { Haar } pub enum WaveletParams { Haar { coeffi pub enum StatModelKind { Parametric } pub enum StatModelParams { Parametric { family: String } } ``` -### 6.5 Update input and readouts (`post_asap/sketch.rs`, `post_asap/maintained_population.rs`) +### 6.5 Update input and evaluations (`post_asap/sketch.rs`, `post_asap/maintained_population.rs`) This part of the code answers two questions about summary state: 1. **Update input** (`SummaryUpdate`): when a `SummaryAgg` reads one input row, what does it add to the state? It adds a **weight**, optionally under an **item** key. -2. **Readouts**: once the state is built, what can be asked of it? `SketchStatistic` is what `SummaryEstimate` asks a sketch (§5.2). `MaintainedPopulation` and `PopulationStatistic` are the exact-population counterpart, used by `MaintainPopulation` and `EvaluatePopulation` (§5.4, §5.5). +2. **Evaluations**: once the state is built, what can be asked of it? `SketchStatistic` is what `SummaryEstimate` asks a sketch (§5.2). `MaintainedPopulation` and `PopulationStatistic` are the exact-population counterpart, used by `MaintainPopulation` and `EvaluatePopulation` (§5.4, §5.5). **Update input examples** (as the planner builds them in `asap-aware-mapping`): @@ -1237,7 +1237,7 @@ This part of the code answers two questions about summary state: `weight_domain` matters because some sketches (e.g. Count-Min) are only accurate when weights are never negative. The planner records why a weight is non-negative; if it cannot prove it, the weight counts as possibly negative. -**Readout examples:** `Quantile { q: 0.99 }` reads p99 from a KLL; `PointCount { key: endpoint, value: Some("checkout") }` reads how often `checkout` occurred from a Count-Min; `TopK { k: 5 }` reads the 5 heaviest items. +**Evaluation examples:** `Quantile { q: 0.99 }` reads p99 from a KLL; `PointCount { key: endpoint, value: Some("checkout") }` reads how often `checkout` occurred from a Count-Min; `TopK { k: 5 }` reads the 5 heaviest items. ```rust /// What one input row adds to the state. @@ -1324,7 +1324,7 @@ pub struct CurrentSeriesInput { } ``` -A finalized value's accuracy statement is `ResultGuarantee` (`post_asap/guarantee.rs`): `metric` (which error is measured, e.g. rank error), `bound` (the error bound), `failure_probability` (the chance the bound does not hold) and `provenance` (which estimates the bound came from). It is attached to readout and finalized nodes, and to `MaintainPopulation` nodes (whose population is exact); never to sketch or accumulator state. +A finalized value's accuracy statement is `ResultGuarantee` (`post_asap/guarantee.rs`): `metric` (which error is measured, e.g. rank error), `bound` (the error bound), `failure_probability` (the chance the bound does not hold) and `provenance` (which estimates the bound came from). It is attached to evaluation and finalized nodes, and to `MaintainPopulation` nodes (whose population is exact); never to sketch or accumulator state. ### 6.6 Summary coverage (`crates/types/src/ir/summary_coverage.rs`) diff --git a/docs/design_docs/proposals/operator-sharing.md b/docs/design_docs/proposals/operator-sharing.md index c9ab193a..8bd8d499 100644 --- a/docs/design_docs/proposals/operator-sharing.md +++ b/docs/design_docs/proposals/operator-sharing.md @@ -80,12 +80,12 @@ An operator owns its scalar expressions and references input nodes through `Rc`. Either operation category can consume the other's outputs when the input contract permits it. `NonASAP` describes one operation, not its entire sub-DAG. Frontend DAGs contain only NonASAP operations; ASAP optimization may -introduce state construction and readout. +introduce state construction and evaluation. | Category | Meaning | All operations | |---|---|---| | `Operator::NonASAP(NonASAPOp)` | Ordinary query operations that transform, combine or aggregate data | `Scan`, `Values`, `Filter`, `Project`, `Aggregate`, `Join`, `SetOp`, `Concat`, `Dedup`, `Sort`, `Limit`, `BinaryOp`, `SQLWindowFunc`, `TimeRange`, `TimeShift`, `PromqlVectorFromScalar`, `PromqlRelabel`, `PromqlInfoEnrich`, `PromqlSeriesSample`, `PromqlSubquery` | -| `Operator::ASAP(ASAPOp)` | Operations on summary state and its results, including reserved operations | `SummaryAgg`, `SummaryEstimate`, `SummaryMerge`, `SummarySubtract`, `SummaryDelete`, `SummaryJoin`, `FinalizeExactAccumulator`, `MaintainPopulation`, `ReadPopulation`, `Extension` | +| `Operator::ASAP(ASAPOp)` | Operations on summary state and its results, including reserved operations | `SummaryAgg`, `SummaryEstimate`, `SummaryMerge`, `SummarySubtract`, `SummaryDelete`, `SummaryJoin`, `FinalizeExactAccumulator`, `MaintainPopulation`, `EvaluatePopulation`, `Extension` | `CurrentTimestamp`, `EvalTimestamp` and `PromqlScalarFromVector` belong to `ScalarExpr`, defined in the [companion proposal](decoupling_op_and_expr.md#22-scalar-expressions). @@ -148,7 +148,7 @@ Ordinary payload fields have these roles: survive optimization. The optional concatenation key records a discriminator that distinguishes branches together with their within-branch key. -`ASAPOp` describes state construction, state operations and readout separately. +`ASAPOp` describes state construction, state operations and evaluation separately. `FieldDataType` (§2.1) types every output field; state-producing operations use its summary or exact-accumulator cases, never its `Plain` case. @@ -156,14 +156,14 @@ summary or exact-accumulator cases, never its `Plain` case. enum ASAPOp { SummaryAgg { child: Rc, family: FieldDataType, input: SummaryUpdate, - reduction: Reduction, grouping: GroupingStrategy, + reduction: Reduction, grouping: GroupingStrategy, filter: Option, }, SummaryEstimate { summary_input: Rc, query: SketchStatistic, }, FinalizeExactAccumulator { child: Rc }, MaintainPopulation { child: Rc, population: MaintainedPopulation }, - ReadPopulation { child: Rc, readout: PopulationStatistic }, + EvaluatePopulation { child: Rc, evaluation: PopulationStatistic }, // Reserved operations; semantics and support require further design. SummaryMerge { children: Vec> }, @@ -176,7 +176,7 @@ enum ASAPOp { } ``` -The summary fields distinguish state construction and readout: +The summary fields distinguish state construction and evaluation: | Field | Design meaning | |---|---| @@ -184,7 +184,7 @@ The summary fields distinguish state construction and readout: | `input` | The item identity and observation or weight supplied to a state update | | `reduction` | Which input entities contribute to each logical result | | `grouping` | Whether those groups use separate state instances or a supported shared structure | -| `query` / `readout` | The result requested from summary or maintained-population state | +| `query` / `evaluation` | The result requested from summary or maintained-population state | | `population` | The population whose membership and values are maintained | The common `OperatorNode` fields are declared in the overview above and explained @@ -212,7 +212,7 @@ function, explicitly written by the query author. This proposal does not insert it automatically: `sum(up)` alone is a valid query returning an instant vector. The scalar expression `PromqlScalarFromVector` represents that function and references its result to obtain one number. A valid ASAP rewrite may replace that producer with -a summary readout, preserving the required vector and accuracy semantics; it cannot +a summary evaluation, preserving the required vector and accuracy semantics; it cannot substitute raw summary state. Ordinary expressions such as `price * 2` reference columns and literals, not a query sub-DAG. @@ -369,8 +369,8 @@ Every operator output, before and after optimization, uses one `Schema` whose `Field`s are typed by `FieldDataType`: `Plain(DataType)` for a readable value, or the family, algorithm and parameters of summary or exact-accumulator state. `OperatorResultKind` marks state outputs, and state becomes a value only through -an explicit readout. The schema model, `ColumnRef` versus `ColumnId`, the -validation entry points and the readout boundary are specified in +an explicit evaluation. The schema model, `ColumnRef` versus `ColumnId`, the +validation entry points and the evaluation boundary are specified in [Schema and physical data for ASAP primitives](asap-primitive-schema.md). ### 2.2 Preserve existing accuracy semantics @@ -383,7 +383,7 @@ not make it exact; unknown accuracy must not be treated as exactness. The common node reuses `guarantee: Option` from `SummaryNode`. `Some` records an established guarantee; `None` covers an unassessed or unknown -result, or state whose accuracy is only established at readout. Exactness must be +result, or state whose accuracy is only established at evaluation. Exactness must be explicitly established using the existing model. This proposal introduces no new accuracy metric or guarantee-calculation workflow. diff --git a/docs/design_docs/proposals/planner-layering.md b/docs/design_docs/proposals/planner-layering.md index 3102e8b3..9f239599 100644 --- a/docs/design_docs/proposals/planner-layering.md +++ b/docs/design_docs/proposals/planner-layering.md @@ -297,17 +297,32 @@ For each eligible sub-DAG, Pass 1 identifies its computation semantics, applies rewrite rules, and generates every candidate that is not provably unable to meet its accuracy requirement. +A sub-DAG is **eligible** when its root is an aggregation (an `Aggregate` +operator) whose intent has at least one realization in the summary-family +capabilities `F`: exact accumulator state or a summary family. Nested +aggregations are separate targets: in `sum by (job) (rate(x[1m]))`, the +per-series `rate` and the per-`job` `sum` are each a target, and their options +multiply. + Example for summary candidates: | Original computation | Per-computation candidates | |---|---| -| `Sum(x) by (g)` | Exact grouped sum | +| `Sum(x) by (g)` | Exact grouped sum (exact Sum accumulator state) | | `TopK(k, x) by (g)` | Exact sort and limit per group, Count-Min Sketch with a top-*k* heap per group, Hydra over all groups | | `Distinct(x)` | Exact distinct, a specialized distinct summary, UnivMon | | `Entropy(x)` | Exact entropy, a specialized entropy summary, UnivMon | | `L2(x)` | Exact L2 norm, a specialized norm summary, UnivMon | | `Quantile(x, window)` | Exact quantile, KLL over the requested window | +Each target gets one exact option. If its intent has an exact mergeable +accumulator (Count, Sum, Min, Max, Rate, IRate, Increase; avg via exact sum and +count accumulator states) and the accumulator can be built, the exact option is +**exact accumulator state**: a `SummaryAgg` with an exact family → +`FinalizeExactAccumulator`. Otherwise, for example when a filter or HAVING +prevents the accumulator or the intent has none, it is the plain exact +computation. Whether the accumulator is materialized is decided in Stage 2. + Each candidate records its input expression, filter, grouping, window, supported estimates and accuracy requirement. Pass 2 uses these to decide whether candidates can share a summary node. @@ -362,37 +377,55 @@ Output: C1 — CandidateLogicalASAPDAGs with each sub-DAG replaced independentl *Example for Algorithm 1* (the workload of [Example 1](#example-1-aggregation-over-dimensions--the-candidate-set-through-every-stage)). Q1 needs an exact answer, so every summary replacement for it is pruned and only the exact -computation remains. Q2 tolerates error and gets three replacements. The -Cartesian product gives 1 × 3 = 3 whole-workload candidates. +option remains for each of its two sub-DAGs, the per-series rate (s₁) and the +per-`job` sum (s₂): its exact accumulator state. Q2's per-series +`sum_over_time` (s₃) has no summary family and gets one option, its exact Sum +accumulator state. Q2's top-*k* (s₄) tolerates error and gets three +replacements. The Cartesian product gives 1 × 1 × 1 × 3 = 3 whole-workload +candidates. ```mermaid flowchart LR subgraph IN["d: one workload LogicalDAG"] direction TB - s1["s₁ = Q1 · sum by (job) (rate(…[1m]))
accuracy: exact"]:::exact - s2["s₂ = Q2 · topk by (job) (10, sum_over_time(…[1m]))
accuracy: ε = 0.01"]:::exact + s1["s₁ = Q1 · rate(…[1m]) per series
accuracy: exact"]:::exact + s2["s₂ = Q1 · sum by (job) (s₁)
accuracy: exact"]:::exact + s3["s₃ = Q2 · sum_over_time(…[1m]) per series
accuracy: ε = 0.01"]:::exact + s4["s₄ = Q2 · topk by (job) (10, s₃)
accuracy: ε = 0.01"]:::exact end subgraph R1["Replacements(s₁)"] direction TB - a1["Exact rate + sum"]:::exact - a2["summary candidates
✗ pruned: cannot be exact"]:::pruned + a3["Exact Rate accumulator
per series"]:::exact end subgraph R2["Replacements(s₂)"] + direction TB + d3["Exact Sum accumulator
per job"]:::exact + a2["summary candidates
✗ pruned: cannot be exact"]:::pruned + end + subgraph R3["Replacements(s₃)"] + direction TB + e3["Exact Sum accumulator
per series"]:::exact + end + subgraph R4["Replacements(s₄)"] direction TB b1["Exact sort + limit per job"]:::exact b2["CMS + top-10 heap per job"]:::summary b3["Hydra over all jobs"]:::summary end - subgraph OUT["C1 = Replacements(s₁) × Replacements(s₂) = 3 candidates"] + subgraph OUT["C1 = Replacements(s₁) × … × Replacements(s₄) = 1 × 1 × 1 × 3 = 3 candidates"] direction TB - c1["① exact Q1 + exact Q2"]:::ok - c2["② exact Q1 + CMS Q2"]:::ok - c3["③ exact Q1 + Hydra Q2"]:::ok + c1["① accumulators s₁, s₂, s₃ + exact s₄"]:::ok + c2["② accumulators s₁, s₂, s₃ + CMS s₄"]:::ok + c3["③ accumulators s₁, s₂, s₃ + Hydra s₄"]:::ok end s1 --> R1 s2 --> R2 + s3 --> R3 + s4 --> R4 R1 --> OUT R2 --> OUT + R3 --> OUT + R4 --> OUT classDef data fill:#f1f3f4,stroke:#5f6368,color:#000; classDef exact fill:#fff,stroke:#5f6368,color:#000; classDef summary fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#000; @@ -918,9 +951,13 @@ flowchart TB **Stage 1, Pass 1: 3 candidates.** Pass 1 finds per-computation options for each query: -* **Q1** has one option, the exact per-series rate and per-`job` sum. Its - accuracy requirement is exact, so no summary qualifies. -* **Q2** has three options. The exact one keeps a per-series sum and sorts +* **Q1** has two targets, the per-series `rate` and the per-`job` `sum`. Each + has one option, its **exact accumulator state** (`SummaryAgg` with an exact + family → `FinalizeExactAccumulator`), so Q1 has one option. Its accuracy + requirement is exact, so no summary qualifies. +* **Q2** has two targets. The per-series `sum_over_time` has one option, its + exact Sum accumulator state. The per-`job` top-*k* has three options, so Q2 + has three options. The exact one keeps a per-series sum and sorts within each `job`, which is costly at one million Zipf-distributed series. The `EpsilonDelta` target also admits a **Count-Min Sketch with a top-*k* heap per `job`**, and **Hydra over the whole `job` column**, where one sketch @@ -928,7 +965,7 @@ flowchart TB ```mermaid flowchart LR - subgraph C["Q2's three per-computation options"] + subgraph C["Q2 top-k's three per-computation options"] direction TB subgraph E["Exact"] direction LR @@ -954,11 +991,12 @@ flowchart LR Combining them gives 1 × 3 = 3 workload candidates: -| Logical candidate | Q1 | Q2 | +| Query | Target | Options | |---|---|---| -| Exact | exact | exact | -| Count-Min | exact | Count-Min + heap | -| Hydra | exact | Hydra | +| Q1 | per-series `rate` | exact Rate accumulator | +| Q1 | per-`job` `sum` | exact Sum accumulator | +| Q2 | per-series `sum_over_time` | exact Sum accumulator | +| Q2 | per-`job` top-*k* | exact, Count-Min + heap, Hydra | **Stage 1, Pass 2: 54 candidates.** Pass 2 applies two ASAP-aware CSE rules to each of the 3 candidates, and keeps every original: