Skip to content

Pass 2 window composition: tumbling panes - #601

Draft
zzylol wants to merge 4 commits into
stack/509-w7b-sql-frequencyfrom
stack/509-x1-stage1-tumbling
Draft

zzylol wants to merge 4 commits into
stack/509-w7b-sql-frequencyfrom
stack/509-x1-stage1-tumbling

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Stacked on #599 (wave 1 chain). Part of #580 item A, step 4; #509 Pass 2 window-composition rule.

Why

#509 Example 3, Pattern B, answers a 5-min p99 that runs every minute by merging 1-min tumbling panes. #591 made the executor run that shape, but the planner never generated it. apply_materialization_timings also rejected SummaryMerge, so a pane plan could not be exported.

What

  1. Timing. A SummaryMerge runs at its consumer's timing. Its inputs must be summary state, and a merge at ingestion time rejects query-time inputs.
  2. Window form as a Pass 1 axis (W3, W4). WindowForm::{Whole, Tumbling { pane_ms }} is set per alternative (LocalLogicalTarget::windows). A query over a range window that repeats on a cadence gets a tumbling copy of every alternative whose family merges (family_merges()), with pane_ms = gcd(lookback, cadence). The cadence comes from RootDemand. A form needs 2 to 64 panes. stage1_logical_candidates now takes demand. Labels read "Q1 Kll · tumbling 1m panes".
  3. Realization (W1, W2). Pane i is SummaryAgg over TimeRange(w) over TimeShift(o + i·w) over the one Scan, where o is the query's own offset. Its coverage is derived from that sub-DAG (feat(ir): derive summary coverage as definition + selection #646): relative time (-(o+(i+1)w), -(o+iw)]. A SummaryMerge combines the panes before SummaryEstimate or FinalizeExactAccumulator. Stage 1 uses the form only if the panes' merged coverage equals the whole-window state's, (-(o+lookback), -o]; derivation rejects overlapping panes. The site detection is ported from Generate exact window composition and materialization candidates #566 compose().
  4. Stage 3. accuracy_violation looks through a merge and checks each pane build against the family's analytical guarantee. A merge is priced at one operation per input state. Its output is the union of the inputs' groups, bounded the way one build is. This matters: without it, a reader of the merge was priced on one pane's groups, and the DP saw a spurious coupling.
  5. DP. In a sharing variant, targets whose panes read one scan are also checked for coupling.
  6. Fix. FinalizeExactAccumulator over a SummaryMerge named its value column state, so readers such as a heap sketch over sum_over_time failed to compile. It now looks through the merge.

Before / After

Before After
Example 3B candidates 3: exact 2.400, KLL 2.083, DDSketch 2.083 cost/s 5: adds KLL and DDSketch in 1-min tumbling panes, 5.167 cost/s each
Example 3B selected KLL (2.083) KLL (2.083)
Example 1 candidates 64 88: Q2's exact sum_over_time also comes in 10-s panes; rates and heaps do not merge
Example 1 selected P60, 4.620 the same plan, now P82, 4.620; the cheapest tumbling candidate is 9.820
Example 3A 486 486 (one-off batch, so no cadence)

Tumbling costs more for now because everything runs at query time, so every pane is rebuilt at each evaluation. In 3B, the five pane builds together cost the same as the whole build (5 × 0.067 = 0.333). The extra +3.08 comes from each pane's TimeShift and TimeRange passing the full 5-min scan, 20M rows (5 × 0.667 against 0.333), plus 0.083 for the merge. This is left untuned, as asked.

The Example 1 fixture is now 9.3 MB (was 4.1 MB). It has 88 candidates, above the devtool's default display cap of 64, so it is generated with --max-candidates 128. The selection fallback cap stays 64, and the facade's DP (#572) still returns TreeDp on Example 1.

Tests

  • Unit tests: the gcd and cadence (including scheduled times); KLL and DDSketch get a form and rate/increase get none; pane construction, shifts and coverage, also with an offset; the coverage proof rejects a width that does not tile; tumbling and whole forms have one output schema (regression for fix 6); SummaryMerge timing (regression for fix 1).
  • DP equals exhaustive: Example 1 (88 combinations), and Pattern B with window forms.
  • Pass 2 sharing: a 5-min and a 3-min p99 every minute build 8 panes independently and 5 in the shared variant, through the existing identical-expression rule.
  • Runtime: the planner's tumbling KLL plan for Pattern B compiles in the executor. Over 2 series × 20 samples it returns the same per-series p99 as the whole-window plan (19 and 119 at ts = 300 000).
  • Example 3: ported the test: #509 Examples 2–4 acceptance specs and end-to-end tests #590 tests and planner_layering_common (window_form reads a merged build as a pane). The tumbling-only tests pass. stage3_a_shared_scan_is_not_costlier now passes and is un-ignored. Five tests stay ignored: EH ×2, partial groupings, and sliding windows ×2; the sliding ones name the tumbling test that passes. The spec doc stays in test: #509 Examples 2–4 acceptance specs and end-to-end tests #590.
  • stage_pipeline --example planner-layering-3a|3b added (from test: #509 Examples 2–4 acceptance specs and end-to-end tests #590). The Example 1 fixture is regenerated.

Gate: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo test --workspace: 1,609 passed / 12 ignored (#599: 1,580 / 7). Viewer tests: 29 OK.

What Stage 2 materialization needs from this PR

Panes are separate SummaryAgg nodes, each with evaluation-relative coverage, so Stage 2 can assign each one ingestion time (B1) or query time kept (B3) through MaterializationAssignment::set. The merge then follows its consumer. Retention is lookback per pane width. Nothing marks yet which pane is the newest one to build at query time (B3); its coverage [-w, 0) identifies it.

Links: #509, #580, #566 (compose() origin), #591 (executor pane merges), #590 (Example 3 acceptance tests).

🤖 Generated with Claude Code

zzylol and others added 4 commits October 11, 2026 16:22
A SummaryMerge now runs when its consumer runs, like other state
operators, and reads summary state from its inputs; a merge at ingestion
time rejects query-time inputs. apply_materialization_timings previously
rejected it as UnimplementedOperator, so planner-emitted tumbling panes
(#580) could not be exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The #509 window-composition rule's tumbling windows, as an axis on Pass 1
alternatives (#580 W1-W4). For a repeating query over a range window, each
alternative whose family merges also comes in panes of width
gcd(lookback, cadence), the cadence taken from the root's RootDemand:
N pane SummaryAggs, pane i over TimeRange(w) over TimeShift(o + i*w) over
the scan with coverage RelativeToEvaluation(-(o+(i+1)w)..-(o+iw)), merged
by a SummaryMerge before the estimate or finalize. Stage 1 offers the form
only when the panes' merged coverage is exactly the whole window.

- Stage 3 checks accuracy through a merge on each pane build (merged
  KLL/DDSketch keep the family's analytical guarantee) and prices a merge
  per input state, its output as the union of the inputs' groups.
- The DP also checks targets whose panes read one scan for coupling.
- FinalizeExactAccumulator over a SummaryMerge keeps the finalized
  value's column name (it was "state", so readers could not resolve it).
- Example 1 gains Q2's exact sum in 10-s panes: 64 -> 88 candidates, same
  selected plan (now P82) and cost; the fixture is generated with
  --max-candidates 128. The devtool labels window forms ("Kll · tumbling
  1m panes") and adds --example planner-layering-3a and -3b (#590).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the #590 Example 3 acceptance tests and their common module onto the
stage pipeline with root demand. window_form reads a pane (a build merged
by a SummaryMerge) as Tumbling of its TimeRange width. Adds Pattern B
tests for the tumbling forms, their five panes and coverage, Stage 3
validity, and a runtime check that the planner's tumbling KLL plan
executes and matches the whole-window plan. The shared-scan cost test now
passes and is un-ignored; EH, partial groupings and sliding windows stay
ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sts read flat DAGs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/509-w7b-sql-frequency branch from 166d20e to c94fbb7 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/509-x1-stage1-tumbling branch from 5190504 to 423b56b Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…stages (#577)

* test: acceptance spec and pending tests for planner-layering Example 1

Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Example 1 acceptance tests against the real stages

Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read Example 1 logical stages as flat DAGs after the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1)

The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: port #590's Example 2-4 acceptance specs and tests

The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(devtools): stage_pipeline examples for #509 Example 3

Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of
five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99),
the Example 3 acceptance tests' workloads. Example 1's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: follow #576's per-consumer charging and exact top-k columns

Plan selection charges a NotMaterialized node once per consuming query,
so the charging helper checks NodeCost.multiplier and the Example 4 A3
ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by
the ranked value and returns [job, topk_10]. The stage viewer guide's
tab 1 row now describes the actual all-exact result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests

run_planning_stages and the #509 example workloads move into the
asap-devtools library; stage_pipeline serializes the run and the
planner-layering tests adapt it. The shared data workload drops
ingestion_rate for data at rest, which validate() requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(devtools): select over every candidate; --max-displayed limits only the document

stage_pipeline capped enumeration at 64, so Example 3a (243 candidates)
selected P41 instead of the cheapest P122 (KLL for all five queries).
The pipeline now enumerates, optimizes and prices every candidate; the
document shows the first --max-displayed (default 64) plus the selected
one, with `combinations` and `capped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 2's SQL candidates compose; the SQL test waits only on #598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: name stages by number and name; say query-language frontends

The frontends' Run field, id and locals become stage0_query_language_frontends,
and test names carry stage0..stage3 with the stage name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: charging helper follows plan selection's per-consumer rule

With a materialized node the helper accepted any multiplier >= 1. It now
computes each node's computations as plan selection does (walks from
query roots and materialized nodes' inputs, stopping at materialized
nodes), checked on a hand-built candidate with a materialized node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 picks Q2 candidates by (option, no shared input)

physical_with and the invalid-candidate check no longer assume one
candidate per Q2 option, so #588's shared-input variants need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 3 Pattern B charges each node once per consuming query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output

Read from uncapped runs: Example 2's sketch plans are rejected for having
no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans,
nothing shared yet) and 3b selects query-time KLL; panes and the latency
check come with the window-summary PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 counts only unshared candidates; stage3_ charging-helper names

The 3-logical and 3-physical counts now hold when #588 adds shared-input
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): PlanningStagesRun fields carry stage number and name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(devtools): unique temp file per stage_pipeline run

cargo test runs the tests as threads of one process, so two runs of the same
example shared a file name and one deleted the other's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant