Skip to content

Commit 25d54e9

Browse files
zzylolclaude
andcommitted
docs: match the DAG API docs to the implemented collection boundaries
Describe the two plain candidate collections, the view-based compile, the caller-typed physical candidate errors, lifecycle_guarantee, and PhysicalExecution as an execution handle. Remove APIs the docs said were removed but never existed (compile_timed_candidates, PhysicalDAGCandidate), the agent instructions in the alignment proposal's baseline, and the ingestion-time Binary exception from design docs, where it is an implementation detail (the developer migration guide keeps it). Restore #485's statement that candidates do not choose placement and #508's CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the split test table in physical-planning-and-deployment.md, and take planner-backend-layering.md verbatim from #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 0f15b3b commit 25d54e9

6 files changed

Lines changed: 205 additions & 226 deletions

File tree

‎docs/design_docs/architecture/input-output-workflow.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ fields and [frontend dependencies](#frontend-specific-dependencies).
3030

3131
| Output | Fields or contents | Meaning |
3232
|---|---|---|
33-
| `CandidatePostASAPDAGs` | All legal logical candidates for the workload, represented compactly as canonical roots, one candidate set per target sub-DAG, and cross-target composition information | The candidate space: nothing is selected yet |
33+
| `CandidatePostASAPDAGs<Id>` | All legal logical candidates for the workload, represented compactly as canonical roots, one candidate set per target sub-DAG, and cross-target composition information | The candidate space: nothing is selected yet |
3434
| `PlanOutput` | One selected PostASAPDAG root (`Rc<PostASAPNode>`) per workload entry, optionally with summary-maintenance lifecycle decisions | One optimization pass's selection, returned by `e2e_plan` and `optimize` |
3535

3636
`PlanOutput` is selected from the candidate space, not a second output beside
@@ -41,7 +41,9 @@ it. [Output layers](#output-layers) places both in the `PreASAPDAG` →
4141
DAG assembly](#selection-and-dag-assembly), and
4242
[summary-maintenance lifecycle](#summary-maintenance-lifecycle-aware-helper) APIs operate on this candidate collection.
4343
These are different uses of the candidate space, not mandatory sequential
44-
stages. The logical candidates have no selected summary-maintenance lifecycle.
44+
stages. `CandidatePostASAPDAGs` itself has no selected summary-maintenance lifecycle, and
45+
its candidates do not choose precompute versus query-time placement: a chosen
46+
lifecycle assignment sets each node's execution timing.
4547

4648
These candidates are logical `PostASAPDAG`s. Planner also compiles
4749
physical candidates. Downstream systems price lifecycle assignments, bind
@@ -325,7 +327,7 @@ Additional inputs for a Planner-owned maintenance decision are listed with the
325327

326328
## Output
327329

328-
### CandidatePostASAPDAGs
330+
### `CandidatePostASAPDAGs<Id>`
329331

330332
The logical layer outputs `CandidatePostASAPDAGs`. The current
331333
Rust API represents this collection compactly as `CandidatePostASAPDAGs<Id>`; this is an
@@ -390,8 +392,8 @@ timing assignments, physical candidate generation and explicit transport exports
390392
|---|---|---|
391393
| 0. Frontends | `CandidatePreASAPDAGs` | Parse and lower PromQL, SQL, or MetricsQL; reject unsupported semantics such as PromQL `fill`. |
392394
| 1. Logical Post-ASAP | `CandidatePostASAPDAGs` | What to compute: summary families, rewrites, and exact candidates. No placement. `PlanOutput` is an optional library-selected logical result; deployments can enumerate candidates themselves. |
393-
| 2. Summary maintenance lifecycle | Lifecycle choices per unique summary state and maintained population → `CandidatePostASAPDAGs` with timing | `Ephemeral`, `Prepared`, `Shared`, or `ContinuouslyMaintained`. Each admissible assignment produces a candidate with node timing, window framework, and retention; this layer is the only source of timing. |
394-
| 3. Physical compilation | `CandidatePhysicalDAGs` | Compile operators and kernels once, then cut by timing into precompute and query DAGs with typed `InputContracts`. Ingestion-time Binary lowering requires matching compile-time timing. |
395+
| 2. Summary maintenance lifecycle | Lifecycle choices per unique summary state and maintained population → `CandidatePostASAPDAGs` with timing (`CandidatePostASAPDAGsWithTiming`) | `Ephemeral`, `Prepared`, `Shared`, or `ContinuouslyMaintained`. Each admissible assignment produces a candidate with node timing, window framework, and retention; this layer is the only source of timing. |
396+
| 3. Physical compilation | `CandidatePhysicalDAGs` | Compile operators and kernels once, then cut by timing into precompute and query DAGs with typed `InputContracts`. |
395397
| 4–5. Deployment | Deployment-owned | Prices lifecycle assignments with its own costs, including summary store cost; selects, binds, and executes. |
396398

397399
Each layer preserves all legal candidates under the supplied constraints;

‎docs/design_docs/physical-planning-and-deployment.md‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -434,15 +434,14 @@ The lifecycle layer decides timing; physical compilation reads it. Lowering a
434434
node does not depend on the frontier, so each query DAG is lowered once and
435435
different lifecycle assignments are different cuts of that lowering.
436436
`compile(dag, inputs, roots)` yields the complete `PhysicalDAG`.
437-
`frontier_from_timing(&timed_dag)` reads an assignment's timed DAG (from
438-
`export_timed_dag`) and returns its frontier: ingestion-time nodes read by
437+
`frontier_from_timing` reads an assignment's timing (a lifecycle assignment's
438+
view, or a DAG from `export_timed_dag`) and returns its frontier: ingestion-time nodes read by
439439
query-time nodes, or an ingestion-time root; a query-time node feeding an
440440
ingestion-time node is rejected. `cut_candidate(&compiled, &frontier)` then
441441
partitions the lowered operators: the frontier's ancestors form the precompute
442442
DAG and the rest form the query DAG. Helper operators are numbered by their
443443
Planner node (`u64::MAX - (node_id << 16) - index`), so a cut is byte-identical to
444-
`compile_candidate` for that frontier. One exception: an ingestion-time
445-
`Binary` lowers differently, so its timing must match at compile time.
444+
`compile_candidate` for that frontier.
446445
`compile_candidate(s)` and `enumerate_frontiers` wrap the same path. Temporal
447446
pane candidates remain a separate lowering.
448447

@@ -572,7 +571,6 @@ operator/runtime fixtures:
572571
| `summary_maintenance_lifecycle_e2e::continuous_lifecycle_compiles_and_executes_spatial_kll` | PromQL workload → selected continuous lifecycle → logical DAG → compiled precompute/query candidate → results in independent revisions; an unbounded candidate fails before pricing, and a bounded request candidate summarizes the same input samples |
573572
| `summary_maintenance_lifecycle_e2e::chosen_lifecycle_timing_decides_precompute_contents` | PromQL workload → enumerated lifecycles → explicit choice → timed DAG → compiled candidate; ContinuouslyMaintained stores the state in precompute, Ephemeral leaves precompute empty and reads the raw source at query time; both return the same p99 |
574573
| `summary_maintenance_lifecycle_e2e::lifecycle_timing_cuts_one_compilation` | KLL quantile and grouped Rate→Sum: one compilation cut by the ContinuouslyMaintained and Ephemeral timed DAGs equals `compile_candidate` for each; the frontier is the retained state or empty |
575-
576574
| `summary_maintenance_lifecycle_e2e::chosen_population_lifecycle_decides_precompute_contents` | PromQL `topk by(job)` over a maintained population → explicit choice → timed DAG → compiled candidate; ContinuouslyMaintained stores the population in precompute, Ephemeral rebuilds it from raw samples at query time; both rank alike |
577575
| `summary_maintenance_lifecycle_e2e::planner_lifecycle_selection_reproduces_strategy_timing` | For PromQL summary fixtures, the timed DAG from Planner's retained selection equals the DAG realization strategies produce |
578576
| `kll_pane_execution::five_panes_roundtrip_and_shared_merge_runs_once` | Explicit one-minute precompute DAGs → real MessagePack state bytes → five required query inputs → shared native merge → p50/p99; counts every sample once, checks adjacent aligned windows and instruments one merge start per run |

‎docs/design_docs/proposals/dag-api-alignment.md‎

Lines changed: 42 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,8 @@ lifecycle assignments; generation does not implicitly select a winner.
2525

2626
## Baseline
2727

28-
The complete implementation depends on the physical compilation and lifecycle
29-
integration underlying #508. Main and the original #480 documentation branch
30-
do not yet contain those implementations. Apply the complete refactor on that
31-
integration, preserving #480's documentation changes; do not merge main as part
32-
of this task.
28+
The implementation builds on the physical compilation and lifecycle APIs of the
29+
stack under #508; main does not contain them yet.
3330

3431
## Changes
3532

@@ -61,8 +58,8 @@ of this task.
6158
identifying their timing cuts, contracts and lifecycle metadata. Reuse the
6259
existing cut and validation implementation. Materialize precompute/query
6360
execution graphs on demand, including after deployment selection. Preserve
64-
separate compilations when timing changes operator lowering, notably an
65-
ingestion-time Binary. Keep failures attributable to their candidates.
61+
separate compilations when timing changes operator lowering. Keep failures
62+
attributable to their candidates.
6663
6. **Update the public boundary and documentation.** Route collection APIs
6764
through the existing lowering, search, lifecycle and compilation algorithms.
6865
Preserve explicit opt-in selection helpers. Update the DAG naming table to
@@ -77,8 +74,8 @@ of this task.
7774
- Shared logical subgraphs retain identity through enumeration and compilation.
7875
- Lifecycle alternatives reuse the logical graph and assign all required node
7976
timing, window and retention information. Illegal timing edges are rejected.
80-
- Equivalent timing cuts share one compilation. Timing-dependent Binary
81-
lowering remains correct and cannot reuse an incompatible realization.
77+
- Equivalent timing cuts share one compilation; a realization is never reused
78+
for timing that lowers differently.
8279
- Candidate generation preserves valid alternatives and visible rejection
8380
reasons; candidate inspection never silently invokes winner selection.
8481
- Selected physical execution graphs preserve their typed producer/reader
@@ -96,49 +93,52 @@ rather than treating a rename as completion of the structural work.
9693

9794
- Named the existing node types and shared root aliases; added frontend
9895
`lower_pre_asap_dag_candidates` and the ID-preserving `CandidatePreASAPDAGs` collection.
99-
- Added `PostASAPDAGIndex`, retaining shared node references and edge metadata.
100-
Physical compilation accepts the root, index or timing assignment through a
101-
borrowed projection. Transport import and direct compilation share validation
102-
and operator lowering; no second rewrite implementation was introduced.
103-
- Added a lazy, budget-checked timed stage of `CandidatePostASAPDAGs`. Unpriced legal choices
104-
retain unknown cost, and rejected combinations retain their choices and errors.
105-
`execution_assignment` attaches timing to the shared index. Window and
106-
retention metadata stay on the accompanying lifecycle plan; unresolved window
107-
evidence remains explicit and must be supplied before deployment installation.
108-
- Added `compile_physical_dag_candidates` and `CandidatePhysicalDAGs<Id>`. Compatible
109-
assignments share `Arc<PhysicalDAG>`; Binary timing changes produce a separate
110-
compilation. Cuts materialize on demand through the existing implementation.
96+
- Added `PostASAPDAGIndex`, retaining shared node references and projecting
97+
node and edge records once. Physical compilation takes a borrowed
98+
`PostASAPDAGView` of a transport document, an index, or a lifecycle
99+
assignment; an assignment's view overlays its timing on the index's records
100+
instead of copying them. Transport import and direct compilation share
101+
validation and operator lowering; no second rewrite implementation was introduced.
102+
- Added the lazy, budget-checked timed collection `CandidatePostASAPDAGsWithTiming`.
103+
Unpriced legal choices retain unknown cost, and rejected combinations retain
104+
their choices and errors. `execution_assignment` attaches timing to the shared
105+
index. Window and retention metadata stay on the accompanying lifecycle plan;
106+
unresolved window evidence remains explicit and must be supplied before
107+
deployment installation.
108+
- Added `compile_physical_dag_candidates` and `CandidatePhysicalDAGs`. Compatible
109+
assignments share `Arc<PhysicalDAG>`; timing that changes lowering produces a
110+
separate compilation. Cuts materialize on demand through the existing implementation.
111111
- Kept explicit eager cut and winner-selection helpers for callers requesting
112112
them. These helpers are not the candidate-preserving generation pipeline.
113113
- Added regression coverage for workload entry IDs, shared logical identity,
114-
assignment budgets and unknown costs, physical compilation sharing,
115-
timing-dependent Binary separation, transport equivalence and rejected timing.
114+
assignment budgets and unknown costs, physical compilation sharing, agreement
115+
with independent compilation of each assignment's transport, and rejected timing.
116116

117117
The Rust API migration is documented in
118118
[dag-api-migration.md](../../develop_docs/dag-api-migration.md). Downstream
119119
consumers must adopt the breaking names before repinning to this branch.
120120

121121
## Collection boundary completion
122122

123-
The logical collection now has a timed stage,
124-
`CandidatePostASAPDAGs<Id, WithTiming<'a, Id>>`, which encapsulates the existing
125-
lifecycle enumerator. `with_timing_for_root` handles logical realization,
126-
index sharing, assignment budgets and lazy generation. Single already-assembled
127-
graphs use `from_post_asap_dag`. The old lifecycle enumerator is crate-private.
128-
129-
`compile_physical_dag_candidates` consumes this timed collection directly and
130-
returns `CandidatePhysicalDAGs<Id>`. The physical collection owns shared graphs
131-
and cut descriptors; the old public `PhysicalDAGCandidate` wrapper is removed.
132-
The physical crate depends on the mapping crate to consume this typed boundary;
133-
logical search does not depend on physical compilation. Both transitions retain
134-
IDs, lifecycle metadata and errors. Reuse also checks contracts and requested
135-
roots, preventing one candidate's input boundary from contaminating another.
123+
The logical collection `CandidatePostASAPDAGs<Id>` and the timed collection
124+
`CandidatePostASAPDAGsWithTiming<'a, Id>` are separate plain types. The timed
125+
collection encapsulates the existing lifecycle enumerator: `with_timing_for_root`
126+
handles logical realization, index sharing, assignment budgets and lazy
127+
generation, and single already-assembled graphs use `from_post_asap_dag`. It
128+
also answers `lifecycle_guarantee`, so a deployment can price an alternative
129+
before binding it. A state without lifecycle alternatives yields a diagnostic
130+
entry rather than disappearing.
131+
132+
`compile_physical_dag_candidates` consumes the timed collection's entries and
133+
returns `CandidatePhysicalDAGs<M, E>`, which owns shared graphs and cut
134+
descriptors. The metadata type `M` and the timing error type `E` belong to the
135+
caller, so the physical crate does not depend on lifecycle planning; a timing
136+
failure stays typed as `PhysicalCandidateError::Timing`, separate from
137+
`PhysicalCandidateError::Compile`. Both transitions retain IDs, lifecycle
138+
metadata and errors. Reuse also checks contracts and requested roots,
139+
preventing one candidate's input boundary from contaminating another.
136140

137141
Existing explicit lifecycle selection and cut APIs delegate to the same
138142
implementation. No second lifecycle enumeration or operator compiler was added.
139-
140-
Validation: `cargo test --workspace` passed 1509 tests across 91 test groups.
141-
`cargo clippy --workspace --all-targets -- -D warnings`, formatting and diff
142-
checks passed. Integration coverage exercises the two named collection
143-
transitions, lazy assignment generation, total budgets, retained IDs and errors,
144-
graph sharing, and compilation reuse across contract/root changes.
143+
Binding runtime sources is an execution step: `PhysicalDAG::instantiate`
144+
returns a `PhysicalExecution` handle for one run, not another DAG.

0 commit comments

Comments
 (0)