Skip to content

types: evaluation-relative coverage time and mergeable summary families - #592

Draft
zzylol wants to merge 3 commits into
stack/509-w6-counter-prooffrom
stack/509-w3-relative-coverage
Draft

zzylol wants to merge 3 commits into
stack/509-w6-counter-prooffrom
stack/509-w3-relative-coverage

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Restacked on #646 (coverage = definition + selection, derived). Chain: … #594 → #593 → #592 → #591 → … (Phase C tail, #528).

Step 3 of the #580 plan (item A prerequisites). Part of #509.

Why

Stage 1 tumbling windows (W1) compose N SummaryAgg panes with one SummaryMerge. #646 already derives where a pane sits in time: TimeRange(w) over TimeShift(s) gives the selection's relative time (-(s+w), -s], and a merge is accepted only for equal definitions and disjoint selections. Two facts are still missing:

  • Whether a family may merge. SummaryMerge accepted any state family with matching schemas, including Rate/Increase accumulators and CMS/CountSketch heaps. W6 says merged heaps are rejected until there is an accuracy model.
  • How the inputs' selections may relate. Disjointness is right for counting families, but idempotent families (HLL, exact Min/Max) give the same state when a row is seen twice, so overlapping inputs are fine (docs: ASAP primitive schema and summary coverage (definition + selection) #573 §4.4).

What

  • FieldDataType::family_merges() returns true for exact Sum/Count/Min/Max, KLL, DDSketch, HLL, CMS, CountSketch and UnivMon. Everything else fails closed: Rate/Increase/IRate, CmsWithHeap, CountSketchWithHeap, KMV/Theta, Sample/Wavelet/StatModel and plain values. SummaryMerge::validate_inputs rejects a family that does not merge.
  • FieldDataType::merge_relation() -> Option<SelectionRelation>, next to family_merges: Disjoint for counting families, OverlapAllowed for HLL and exact Min/Max, None when the family does not merge. SelectionRelation::Contained names what the reserved SummarySubtract will require; nothing produces it yet.
  • SummaryCoverage::derive skips the disjointness check for OverlapAllowed families, and the union of identical selection boxes is one box.

The earlier version of this PR added CoverageTime::{Absolute, RelativeToEvaluation} to declared coverage; that commit is dropped because #646 derives relative time from the sub-DAG.

SummaryMerge(KLL us, KLL eu)          -> selection {region ∈ {us}} ∪ {region ∈ {eu}}
SummaryMerge(KLL us, KLL us)          -> PossibleOverlap
SummaryMerge(HLL us, HLL us)          -> accepted, selection {region ∈ {us}}
SummaryMerge over CmsWithHeap states  -> "summary merge over Sketch(CmsWithHeap…) is unsupported: the family has no sound merge"

Tests

Written first; idempotent_families_merge_overlapping_states and family_merge_selection_relation fail without the change.

  • summary_merge_structure.rs: heap states are rejected and KLL states accepted (summary_merge_requires_a_mergeable_family); table tests of family_merges and merge_relation; HLL/Max states over the same rows merge and Sum/KLL ones do not.

The doc comments on SelectionRelation and merge_relation cite #573 §5.6, the doc's SummaryMerge section.

Open questions for review

  • KMV and Theta merge soundly in the literature but were not on the W6 list, so they currently fail closed.

Gate

cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo test --workspace pass at this head.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 390d792 to 0f9db83 Compare October 4, 2026 18:25
zzylol added a commit that referenced this pull request Oct 4, 2026
Pane coverage time is a CoverageTime since #592; the tumbling pane-merge test
keeps absolute time ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol changed the base branch from stack/509-e2-keyed-additive-topk to stack/509-w6-counter-proof October 4, 2026 18:25
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from ee6f881 to 761b2ce Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 0f9db83 to 6b31d2d Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from 761b2ce to f6a38fb Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 6b31d2d to 8e9b5f9 Compare October 8, 2026 16:47
zzylol added a commit that referenced this pull request Oct 9, 2026
Column identity follows the SummaryAgg's input schema (a Project renames
and drops the table); mark §6.1/§6.6 as #646 and per-family overlap as #592;
panes come from window composition (Pass 2); add IRate; three selection
shapes; current SummaryEstimate top-k output; time row of the definition
table; drop unchecked paper section numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 9, 2026
Merge coverage is cached at construction (#646); full per-type overlap
table from #592; k mismatch is a schema rejection; #579 top-k shape;
tighter cost bounds; walk-order caveat for combining conditions; Project
qualifier; absolute time marked later; Example B scope; one-walk wording;
define by[job] and closed schema; link Pass 2; window notation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from f6a38fb to 6fd1d74 Compare October 10, 2026 14:32
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 8e9b5f9 to 64b46c1 Compare October 10, 2026 14:32
zzylol and others added 3 commits October 11, 2026 16:22
FieldDataType::family_merges() is true for exact Sum/Count/Min/Max,
KLL, DDSketch, HLL, CMS, CountSketch and UnivMon; every other family
fails closed (Rate/Increase accumulators and heap top-k sketches
included, per W6). SummaryMerge::validate_inputs rejects the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FieldDataType::merge_relation() returns the SelectionRelation a merge
needs (#573 §4.4): Disjoint for counting families (exact Sum/Count, KLL,
DDSketch, CMS, CountSketch, UnivMon), OverlapAllowed for idempotent ones
(exact Min/Max, HLL), and None when the family does not merge. Contained
names what the reserved SummarySubtract will require.

SummaryCoverage::derive skips the disjointness check for OverlapAllowed
families, and the union of identical selection boxes is one box.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
The design doc moved the SummaryMerge section from §4.4 to §5.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from 6fd1d74 to da28717 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 64b46c1 to 192311f Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 12, 2026
…ion) (#573)

* docs: design for ASAP primitive schema and summary semantics

One design document for the edge schema of ASAP primitives: fields that carry
summary state, the summary semantics a schema must preserve, coverage, and
per-operator examples. Related docs keep a summary and link to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: restore the hand-written sections before the examples

Keep the author's sections 1–4 as written; the operator examples (§5) and
key code interfaces (§6) stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: a top-k readout returns selected rows

Follows #579: the top-k sketch readout derives one row per selected item
instead of a packed Utf8 column, matching the exact Sort → Limit shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: explain why summary coverage is a node field, not part of the schema

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: summary coverage records rows and columns

SummaryCoverage records which columns a state summarizes (input,
group_by) as well as which rows. Update the SummaryAgg and SummaryMerge
examples to #560: merges compare coverage columns, carry no coverage
until #646, and merged_coverage is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: summary coverage as definition + selection, based on Goldstein-Larson view matching

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: the 5.1 example's time index is a Timestamp column

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: absolute time is a timestamp-column interval; match the coverage interface in #646

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: ambiguous column names and literal types in selections

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: fill §1 links and operator list, §2 terminology, §3 summary type terms

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: point links into asap-primitive-schema.md at its current sections

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: list the levels of a summary type as a table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: list supported query intents per summary type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: structure §4 with summary table, rule tables and bullet lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: rename §4.2 heading

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: explain the residual condition of view matching

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: define the three parts of a view's WHERE with examples

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: define equivalence classes, ranges and residuals

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: move the Goldstein & Larson mapping after §4.2 as §4.3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: say which paper requirement unions of views relax

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: SummaryMerge is implemented on main since #560

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: §4.7 leaves the deployment runtime, such as SDS, out of coverage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: rewrite §4.1 as a worked G&L example and §4.2–§4.5 in plain terms

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: worked examples for §4.4 and §4.5; drop the §5.8 recap table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: draw the §5 examples as diagrams and timelines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: reword how coverage is computed in §4.4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: explain the selection derivation in §4.4 as goal and steps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: move the Goldstein & Larson mapping into a footnote of §4.2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: clearer wording for Rule 1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: draw §5 examples as bottom-up operator trees

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: draw §4.4 and §5 examples as DAGs with shared nodes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: rename step 3 in §4.3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: phrase step 3 outcomes as conditions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: plainer Rule 1 and Rule 2 details with reasons and examples

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: §4.3 derives the definition first, then the selection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: name the code interface in the §4.5 heading

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: move the coverage code interface to §6.5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: §6 intro points to §6.5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: merge §4.2 and §4.3 (why two parts, then derivation); fold §4.4 into §5.6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: refer to the downstream deployment runtime, not a specific backend

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: same wording in the §2 layout row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: leave the SDS definition as a TODO for a separate doc

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: make §5 easier to read with one question table per operator

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: explain every struct and variant field in §6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: label §5 operator descriptions as operator definitions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: explain every function in §6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: describe SketchKind.category as the aggregation intents it answers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: explain what §6.3 update input and readouts are for, with examples

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: rename §6.2 to summary-family parameters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: show how the §6 code interfaces connect

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: order §6 subsections from the DAG node inward

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: add the cost of deriving coverage (§4.2.3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: fix correctness issues found in review

Column identity follows the SummaryAgg's input schema (a Project renames
and drops the table); mark §6.1/§6.6 as #646 and per-family overlap as #592;
panes come from window composition (Pass 2); add IRate; three selection
shapes; current SummaryEstimate top-k output; time row of the definition
table; drop unchecked paper section numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: fix second-review findings

Merge coverage is cached at construction (#646); full per-type overlap
table from #592; k mismatch is a schema rejection; #579 top-k shape;
tighter cost bounds; walk-order caveat for combining conditions; Project
qualifier; absolute time marked later; Example B scope; one-walk wording;
define by[job] and closed schema; link Pass 2; window notation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: fix §1 grammar; start every example from its PromQL or SQL query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: restore the author's wording in §1, §3 and §4 openings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: show ASAPOp generic over its child reference, as on main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: describe the union and instant-selector rules as #646 implements them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: touching value ranges join; instant selectors join the later lifting rules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: PromQL comparisons by lowering; constants from folded casts

A vector/scalar comparison lowers to a Filter above rate and moves to the
selection; a bool comparison is a Project and a vector/vector comparison a
BinaryOp, both kept in the definition. The SQL front end folds numeric casts
of literals as DataFusion does, so `latency < 100` reads as a constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: only a Project may rename on the way to the selection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant