Skip to content

docs: align #509/#511/#573 docs with the decided vocabulary and Pass 1 candidates #663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zzylol merged 4 commits into from
Oct 12, 2026
Merged

docs: align #509/#511/#573 docs with the decided vocabulary and Pass 1 candidates #663

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0cde14c
docs: align #509/#511/#573 docs with the decided vocabulary and Pass …
zzylol Oct 12, 2026
72fa555
docs: leave multi-measure targets out of the eligible sub-DAG definition
zzylol Oct 12, 2026
cb3e45d
docs: one exact option per Pass 1 target
zzylol Oct 12, 2026
5362d7e
docs: reflow the eligible sub-DAG paragraph
zzylol Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions docs/design_docs/proposals/asap-primitive-schema.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Based on our requirement, each field should contain the following information.

- **Statistical aggregation intents** (`AggIntent`, `crates/types/src/pre_asap/agg_intent.rs`): which aggregate the state can answer, and how it is read out.

| Intent | Candidate types | Readout |
| Intent | Candidate types | Evaluation |
|---|---|---|
| `Quantile` | `Sketch`: `Kll`, `DDSketch` | `SummaryEstimate(Quantile { q })` |
| `Cardinality` | `Sketch`: `Hll`, `Theta`, `Kmv`, `UnivMon` (one column only) | `SummaryEstimate(Cardinality)` |
Expand All @@ -93,7 +93,7 @@ Based on our requirement, each field should contain the following information.
| `FrequencyL2`, `FrequencyEntropy` | `Sketch`: `UnivMon` | `SummaryEstimate(FrequencyL2 \| FrequencyEntropy)` |
| `Sum`, `Count`, `Min`, `Max`, `Rate`, `IRate`, `Increase` (exact) | `ExactAggregate(ExactKind, …)` | `FinalizeExactAccumulator` |

The sketch candidates are `summary_candidates(intent)` in `crates/asap-aware-mapping/src/replacement.rs`; exact accumulators come from `exact_realization` there (`Count`) and from `function_rules.rs` (the others); the readouts are `SketchStatistic` ([§6.5](#65-update-input-and-readouts-post_asapsketchrs-post_asapmaintained_populationrs)).
The sketch candidates are `summary_candidates(intent)` in `crates/asap-aware-mapping/src/replacement.rs`; exact accumulators come from `exact_realization` there (`Count`) and from `function_rules.rs` (the others); the evaluations are `SketchStatistic` ([§6.5](#65-update-input-and-evaluations-post_asapsketchrs-post_asapmaintained_populationrs)).

- **Time window aggregation intents**: whether states built over smaller windows can answer a larger one. This depends on how the family combines states:
- **Merge** (`SummaryMerge`, §5.6): states over disjoint panes combine into the state of their union, e.g. two 1-minute KLL states answer a 2-minute quantile. Requires a mergeable family.
Expand Down Expand Up @@ -543,10 +543,10 @@ Both need the same KLL state, so the planner builds it once (sub-DAG sharing, [P

| Question | Answer |
|---|---|
| What comes out? | the same columns, with `state` replaced by the answer: `quantile` Float64 here. Counts and cardinalities are Int64 (Float64 when the state was built per series, `PerEntity`). A top-k readout is a `topk` Utf8 field today; #579 changes it to return the selected rows: the partition keys, the item identity columns and a Float64 `value` score |
| What comes out? | the same columns, with `state` replaced by the answer: `quantile` Float64 here. Counts and cardinalities are Int64 (Float64 when the state was built per series, `PerEntity`). A top-k evaluation is a `topk` Utf8 field today; #579 changes it to return the selected rows: the partition keys, the item identity columns and a Float64 `value` score |
| When is it rejected? | the input is not a sketch, or the sketch cannot answer the question. For example, asking a KLL for a cardinality |
| What is its coverage? | none: the output is a value |
| State or value? | value. The node carries the readout's error bound |
| State or value? | value. The node carries the evaluation's error bound |

### 5.3 `FinalizeExactAccumulator`: exact state → value

Expand Down Expand Up @@ -870,7 +870,7 @@ OperatorNode §6.1
│ └── fields[i].dtype: FieldDataType ◀───────┘ §6.3 (the same type: the output
│ └── Sketch(SketchKind, GroupingStrategy) `state` field has type `family`)
│ └── category, algorithm, params §6.4
├── guarantee: ResultGuarantee §6.5 (accuracy of a readout)
├── guarantee: ResultGuarantee §6.5 (accuracy of an evaluation)
└── coverage() → SummaryCoverage §6.6
└── definition: OperatorNode (again a node, so the same structure)
```
Expand All @@ -881,10 +881,10 @@ OperatorNode §6.1
| §6.2 ASAP operators | `ASAPOp` | `OperatorNode.operator` |
| §6.3 Schema and field types | `Schema`, `Field`, `FieldDataType` | every node's `schema`; `SummaryAgg.family` |
| §6.4 Summary-family parameters | `SketchKind`, `SketchParams`, `GroupingStrategy`, exact/sample/wavelet/model params | the payload of each non-plain `FieldDataType` variant |
| §6.5 Update input and readouts | `SummaryUpdate`, `SketchStatistic`, `MaintainedPopulation`, `PopulationStatistic` | fields of the `ASAPOp` variants in §6.2 |
| §6.5 Update input and evaluations | `SummaryUpdate`, `SketchStatistic`, `MaintainedPopulation`, `PopulationStatistic` | fields of the `ASAPOp` variants in §6.2 |
| §6.6 Summary coverage | `SummaryCoverage`, `SelectionBox`, `Constraint` | `OperatorNode::coverage()` on summary nodes |

**Example: the p99 latency state of §5.1.** The `SummaryAgg` node (§6.1) holds `ASAPOp::SummaryAgg` (§6.2) with `family = Sketch(SketchKind::new(Kll, Kll { k: 200 }), PerSubpopulationInstance)` (§6.3, §6.4) and `input = SummaryUpdate::column(latency)` (§6.5). Its `schema` (§6.3) is `(job Utf8, state Sketch(KLL k=200))`: the `state` field has the same type as `family`. A `SummaryEstimate` above it holds `query = Quantile { q: 0.99 }` (§6.5), and its node carries the `guarantee` of the readout. `coverage()` on the `SummaryAgg` node returns a `SummaryCoverage` (§6.6).
**Example: the p99 latency state of §5.1.** The `SummaryAgg` node (§6.1) holds `ASAPOp::SummaryAgg` (§6.2) with `family = Sketch(SketchKind::new(Kll, Kll { k: 200 }), PerSubpopulationInstance)` (§6.3, §6.4) and `input = SummaryUpdate::column(latency)` (§6.5). Its `schema` (§6.3) is `(job Utf8, state Sketch(KLL k=200))`: the `state` field has the same type as `family`. A `SummaryEstimate` above it holds `query = Quantile { q: 0.99 }` (§6.5), and its node carries the `guarantee` of the evaluation. `coverage()` on the `SummaryAgg` node returns a `SummaryCoverage` (§6.6).

### 6.1 DAG node (`crates/types/src/ir/node.rs`)

Expand Down Expand Up @@ -1220,12 +1220,12 @@ pub enum WaveletKind { Haar } pub enum WaveletParams { Haar { coeffi
pub enum StatModelKind { Parametric } pub enum StatModelParams { Parametric { family: String } }
```

### 6.5 Update input and readouts (`post_asap/sketch.rs`, `post_asap/maintained_population.rs`)
### 6.5 Update input and evaluations (`post_asap/sketch.rs`, `post_asap/maintained_population.rs`)

This part of the code answers two questions about summary state:

1. **Update input** (`SummaryUpdate`): when a `SummaryAgg` reads one input row, what does it add to the state? It adds a **weight**, optionally under an **item** key.
2. **Readouts**: once the state is built, what can be asked of it? `SketchStatistic` is what `SummaryEstimate` asks a sketch (§5.2). `MaintainedPopulation` and `PopulationStatistic` are the exact-population counterpart, used by `MaintainPopulation` and `EvaluatePopulation` (§5.4, §5.5).
2. **Evaluations**: once the state is built, what can be asked of it? `SketchStatistic` is what `SummaryEstimate` asks a sketch (§5.2). `MaintainedPopulation` and `PopulationStatistic` are the exact-population counterpart, used by `MaintainPopulation` and `EvaluatePopulation` (§5.4, §5.5).

**Update input examples** (as the planner builds them in `asap-aware-mapping`):

Expand All @@ -1237,7 +1237,7 @@ This part of the code answers two questions about summary state:

`weight_domain` matters because some sketches (e.g. Count-Min) are only accurate when weights are never negative. The planner records why a weight is non-negative; if it cannot prove it, the weight counts as possibly negative.

**Readout examples:** `Quantile { q: 0.99 }` reads p99 from a KLL; `PointCount { key: endpoint, value: Some("checkout") }` reads how often `checkout` occurred from a Count-Min; `TopK { k: 5 }` reads the 5 heaviest items.
**Evaluation examples:** `Quantile { q: 0.99 }` reads p99 from a KLL; `PointCount { key: endpoint, value: Some("checkout") }` reads how often `checkout` occurred from a Count-Min; `TopK { k: 5 }` reads the 5 heaviest items.

```rust
/// What one input row adds to the state.
Expand Down Expand Up @@ -1324,7 +1324,7 @@ pub struct CurrentSeriesInput {
}
```

A finalized value's accuracy statement is `ResultGuarantee` (`post_asap/guarantee.rs`): `metric` (which error is measured, e.g. rank error), `bound` (the error bound), `failure_probability` (the chance the bound does not hold) and `provenance` (which estimates the bound came from). It is attached to readout and finalized nodes, and to `MaintainPopulation` nodes (whose population is exact); never to sketch or accumulator state.
A finalized value's accuracy statement is `ResultGuarantee` (`post_asap/guarantee.rs`): `metric` (which error is measured, e.g. rank error), `bound` (the error bound), `failure_probability` (the chance the bound does not hold) and `provenance` (which estimates the bound came from). It is attached to evaluation and finalized nodes, and to `MaintainPopulation` nodes (whose population is exact); never to sketch or accumulator state.

### 6.6 Summary coverage (`crates/types/src/ir/summary_coverage.rs`)

Expand Down
22 changes: 11 additions & 11 deletions docs/design_docs/proposals/operator-sharing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,12 +80,12 @@ An operator owns its scalar expressions and references input nodes through
`Rc<OperatorNode>`. Either operation category can consume the other's outputs when
the input contract permits it. `NonASAP` describes one operation, not its entire
sub-DAG. Frontend DAGs contain only NonASAP operations; ASAP optimization may
introduce state construction and readout.
introduce state construction and evaluation.

| Category | Meaning | All operations |
|---|---|---|
| `Operator::NonASAP(NonASAPOp)` | Ordinary query operations that transform, combine or aggregate data | `Scan`, `Values`, `Filter`, `Project`, `Aggregate`, `Join`, `SetOp`, `Concat`, `Dedup`, `Sort`, `Limit`, `BinaryOp`, `SQLWindowFunc`, `TimeRange`, `TimeShift`, `PromqlVectorFromScalar`, `PromqlRelabel`, `PromqlInfoEnrich`, `PromqlSeriesSample`, `PromqlSubquery` |
| `Operator::ASAP(ASAPOp)` | Operations on summary state and its results, including reserved operations | `SummaryAgg`, `SummaryEstimate`, `SummaryMerge`, `SummarySubtract`, `SummaryDelete`, `SummaryJoin`, `FinalizeExactAccumulator`, `MaintainPopulation`, `ReadPopulation`, `Extension` |
| `Operator::ASAP(ASAPOp)` | Operations on summary state and its results, including reserved operations | `SummaryAgg`, `SummaryEstimate`, `SummaryMerge`, `SummarySubtract`, `SummaryDelete`, `SummaryJoin`, `FinalizeExactAccumulator`, `MaintainPopulation`, `EvaluatePopulation`, `Extension` |

`CurrentTimestamp`, `EvalTimestamp` and `PromqlScalarFromVector` belong to
`ScalarExpr`, defined in the [companion proposal](decoupling_op_and_expr.md#22-scalar-expressions).
Expand Down Expand Up @@ -148,22 +148,22 @@ Ordinary payload fields have these roles:
survive optimization. The optional concatenation key records a discriminator
that distinguishes branches together with their within-branch key.

`ASAPOp` describes state construction, state operations and readout separately.
`ASAPOp` describes state construction, state operations and evaluation separately.
`FieldDataType` (§2.1) types every output field; state-producing operations use its
summary or exact-accumulator cases, never its `Plain` case.

```rust
enum ASAPOp {
SummaryAgg {
child: Rc<OperatorNode>, family: FieldDataType, input: SummaryUpdate,
reduction: Reduction, grouping: GroupingStrategy,
reduction: Reduction, grouping: GroupingStrategy, filter: Option<Predicate>,
},
SummaryEstimate {
summary_input: Rc<OperatorNode>, query: SketchStatistic,
},
FinalizeExactAccumulator { child: Rc<OperatorNode> },
MaintainPopulation { child: Rc<OperatorNode>, population: MaintainedPopulation },
ReadPopulation { child: Rc<OperatorNode>, readout: PopulationStatistic },
EvaluatePopulation { child: Rc<OperatorNode>, evaluation: PopulationStatistic },

// Reserved operations; semantics and support require further design.
SummaryMerge { children: Vec<Rc<OperatorNode>> },
Expand All @@ -176,15 +176,15 @@ enum ASAPOp {
}
```

The summary fields distinguish state construction and readout:
The summary fields distinguish state construction and evaluation:

| Field | Design meaning |
|---|---|
| `family` | The summary or exact accumulator chosen, including its family-specific parameters |
| `input` | The item identity and observation or weight supplied to a state update |
| `reduction` | Which input entities contribute to each logical result |
| `grouping` | Whether those groups use separate state instances or a supported shared structure |
| `query` / `readout` | The result requested from summary or maintained-population state |
| `query` / `evaluation` | The result requested from summary or maintained-population state |
| `population` | The population whose membership and values are maintained |

The common `OperatorNode` fields are declared in the overview above and explained
Expand Down Expand Up @@ -212,7 +212,7 @@ function, explicitly written by the query author. This proposal does not insert
it automatically: `sum(up)` alone is a valid query returning an instant vector.
The scalar expression `PromqlScalarFromVector` represents that function and references its
result to obtain one number. A valid ASAP rewrite may replace that producer with
a summary readout, preserving the required vector and accuracy semantics; it cannot
a summary evaluation, preserving the required vector and accuracy semantics; it cannot
substitute raw summary state. Ordinary expressions such as `price * 2` reference
columns and literals, not a query sub-DAG.

Expand Down Expand Up @@ -369,8 +369,8 @@ Every operator output, before and after optimization, uses one `Schema` whose
`Field`s are typed by `FieldDataType`: `Plain(DataType)` for a readable value, or
the family, algorithm and parameters of summary or exact-accumulator state.
`OperatorResultKind` marks state outputs, and state becomes a value only through
an explicit readout. The schema model, `ColumnRef` versus `ColumnId`, the
validation entry points and the readout boundary are specified in
an explicit evaluation. The schema model, `ColumnRef` versus `ColumnId`, the
validation entry points and the evaluation boundary are specified in
[Schema and physical data for ASAP primitives](asap-primitive-schema.md).

### 2.2 Preserve existing accuracy semantics
Expand All @@ -383,7 +383,7 @@ not make it exact; unknown accuracy must not be treated as exactness.

The common node reuses `guarantee: Option<ResultGuarantee>` from `SummaryNode`.
`Some` records an established guarantee; `None` covers an unassessed or unknown
result, or state whose accuracy is only established at readout. Exactness must be
result, or state whose accuracy is only established at evaluation. Exactness must be
explicitly established using the existing model. This proposal introduces no new
accuracy metric or guarantee-calculation workflow.

Expand Down
Loading
Loading