diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 721bd0aa8..9a0e3e1b5 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -10,6 +10,9 @@ internals and are subordinate to the shared system contracts. Proposals for shared-contract review: +- [Offline ERP planning: physical selection to deployment](offline-erp-plan-selection.md) + proposes one-shot planning from offline profiles and workload inputs, with + global physical selection and existing deployment artifacts, without replanning. - [ASAPPlanner integration architecture](asapplanner-integration.md) proposes the Planner/backend responsibility boundary, shared semantic DAG workflow, and high-level consolidation milestones. diff --git a/docs/design_docs/offline-erp-plan-selection.md b/docs/design_docs/offline-erp-plan-selection.md new file mode 100644 index 000000000..973c1d6d3 --- /dev/null +++ b/docs/design_docs/offline-erp-plan-selection.md @@ -0,0 +1,268 @@ +# Offline ERP planning: physical selection to deployment + +Status: proposed. Audience: Planner and backend designers and implementers. +This proposal describes a new planning path; it does not claim that the current +backend implements it or replace the canonical cross-component contracts. + +## Decision and scope + +Accept a fixed query workload, data workload, offline error–resource–performance +profiles, and deployment constraints. Select one executable physical plan, then +produce its deployment plan through the existing publication contracts. + +ERP retains its existing Error–Resource Profile name. Its measurements include +accuracy, memory, and operation performance. Profiles are produced offline; +planning neither benchmarks sketches nor requires live runtime observations. + +There is no automatic replanning, drift-triggered selection, runtime profile +refresh, migration optimizer, or feedback loop in this version. A later explicit +planning request is a separate invocation. Existing observation and replanning +paths are not removed, but are not dependencies of this path. Ordinary query +execution and maintenance continue after installation. + +The result is one plan, not a Pareto frontier. The initial objective is minimum +modeled steady-state CPU under explicit accuracy and deployment constraints. + +```mermaid +flowchart TD + Q[QueryWorkload and DataWorkload] --> C[Control plane: validate immutable inputs] + E[Offline ERP profiles] --> C + D[Deployment constraints and capabilities] --> C + C --> P[Planner: legal semantic alternatives] + P --> B[Backend: executable physical realizations and cost coefficients] + B --> M[Global selection: configurations, windows, sharing] + M --> F[One validated physical plan] + F --> A[Deployment artifacts through existing publication contracts] + A --> I[Explicit installation and activation] +``` + +Candidate generation and feasibility checking may involve both Planner and +backend. The diagram is a responsibility boundary, not a requirement to fully +enumerate complete plans or commit to one semantic alternative before costing. + +## Current behavior and proposed change + +The backend already accepts ERP inputs, selects empirical parameters with +theoretical/exact fallback, compiles physical alternatives, and compares priced +workload candidates. Relevant implementation anchors are +[the ERP adapter](../../control_plane/src/physical/erp.rs), +[physical compilation](../../control_plane/src/physical/compiler.rs), +[workload pricing and selection](../../control_plane/src/physical/workload_cost.rs), +and [publication conversion](../../control_plane/src/physical/publication.rs). + +Today, ERP parameter selection returns one preferred configuration for a +selection context. The current backend-local path also requires workload cost +evidence and compares a bounded inventory of compiled alternatives. Supplying +atomic ERP measurements alone does not price every physical component. + +[sketch-bench PR #129](https://github.com/ProjectASAP/sketch-bench/pull/129) +provides a useful MILP starting point: choose one deployment per repeating query, +charge a shared deployment's ingestion once, and avoid enumerating all query to +deployment assignments. The reviewed implementation is revision `82ca7a5`. + +This proposal preserves multiple admissible configurations and physical +realizations until global selection. The MILP is a selection mechanism within +the existing Planner/backend boundary, not a replacement for semantic legality, +accuracy propagation, executable binding, or publication validation. It does +not become a second authoritative selector after another stage has discarded +the alternatives it needs. + +## Immutable inputs and outputs + +Each invocation binds the following inputs to a reproducible planning snapshot: + +| Input | Required meaning | +| --- | --- | +| QueryWorkload | Query semantics, accuracy requirements, recurrence, lookback, and evaluation phase | +| DataWorkload | Source identity, population partitioning, aggregate arrival rates, group counts, and offline distribution descriptors or supported shape evidence | +| Offline profiles | Artifact/version and record IDs, algorithm/implementation/configuration, accuracy metric and measurement scope, resource costs, units, and measurement provenance | +| Deployment context | Backend-local or collector target, runtime capabilities, retention policy, exact-execution availability, and requested resource bounds | +| Planning policy | Accuracy/fallback mode, CPU objective, common costing horizon, solver limits, and evidence applicability policy | + +Missing rates, cardinalities, or component prices remain unknown; they are not +silently zero. Group count means physical populations, not distinct sketch item +keys. Source descriptors and shape evidence are supplied offline and remain +fixed for the invocation. Profile applicability includes implementation and +measurement environment compatibility, trial sufficiency, data shape/volume, +and any declared validity constraints. Offline does not mean universally valid. + +Output contains the selected physical plan, validated deployment artifacts, and +a selection report. The report records snapshot/profile IDs, input and model +versions, configuration and realization identities, sharing, CPU breakdown, +memory scope, accuracy evidence, fallback reasons, and solver status/gap. Store +the selected mapping so a tied optimum can be audited without assuming identical +solver tie-breaking across versions. + +This is a logical contract, not a new HTTP schema or a replacement workload +type. Reuse the existing canonical workload and publication types; specify +necessary request extensions during implementation. + +## Candidate construction and evidence + +1. Planner produces legal semantic alternatives, including permitted exact or + mixed execution. Do not map arbitrary query text directly to the PR's four + capability enum values. +2. The ERP adapter exposes all admissible configurations for the relevant + readout, reusing its existing matching and runtime checks. Retain typed + parameters and evidence IDs through selection. Choosing the cheapest + configuration separately for each query can lose a better shared solution. +3. Backend derives executable window/layout alternatives from recurrence, + evaluation phase, retention requirements, and deployment capabilities. +4. Construct assignment eligibility and resource coefficients for these + alternatives. Prune only when replacement preserves every consumer's + semantics, evidence, feasibility, and all modeled costs and constraints. + +ERP resource fields map naturally to the prototype's atomic costs: + +| ERP field | Prototype field | +| --- | --- | +| `memory_bytes` | `mem_bytes_per_instance` | +| `update_cpu_seconds` | `insert_cpu_secs` | +| `merge_cpu_seconds` | `merge_cpu_secs` | +| `query_cpu_seconds` | `query_cpu_secs` | + +This projection does not replace the ERP record. Its distribution, implementation, +readout units, trial counts, and evidence provenance remain authoritative. +For example, the prototype's KLL example checks `mean_rank_err`; the backend's +KLL ERP readout uses `max_rank_err`. These are not interchangeable. Empirical +error magnitudes do not establish an explicit failure probability. + +Accuracy must apply to the final readout, including the selected data volume and +merge topology. Single-instance measurements are insufficient evidence for an +unvalidated merged result. Use applicable merged-result evidence or a supported +compositional/theoretical contract; otherwise exclude that approximate candidate. + +Sharing requires compatible source, filters/update expression, population +partitioning, retained labels, sketch configuration, window alignment and phase, +and runtime policy. Reuse backend semantic/materialization identities. Equal +capability and label names alone, as in the prototype's grouping, are insufficient. + +## Global physical selection + +For the initial supported independent-readout subset, let `z[i,d]` choose an +eligible deployment for readout `i` and `u[d]` activate deployment `d`: + +```text +sum_d z[i,d] = 1 +z[i,d] <= u[d] +u[d] <= sum_i z[i,d] +minimize sum_d u[d] * ingest_cpu[d] + + sum_i,d z[i,d] * read_and_merge_cpu[i,d] + + CPU of other selected required physical components +``` + +Steady-state terms use CPU-seconds per wall-second. Multiply by a common horizon +only when adapting to horizon-based quotes. Ingestion is charged once per +activated physical deployment; query work is charged at each consumer's cadence. +Do not reuse ERP's already aggregated selection cost and charge the same work +again. Exact/residual costs must use the same workload and units. Query-result +reuse is out of scope; sharing sketch state does not waive readout work. + +For general Planner DAGs, assignment variables must also preserve alternative +exclusivity and component dependencies. The prototype's one-deployment-per-RQE +constraint is not sufficient for arbitrary mixed DAGs. Unsupported structures +remain on an explicitly reported existing planning path, or are rejected under +a strict policy; do not claim a global optimum over them. + +All hard constraints apply before a plan is called selected. A requested total +deployment memory cap must include retained states, overlap, merge buffers, and +the specified concurrency model. The prototype's maximum single-query working +set is not this cap. CPU-derived query latency is labeled as a modeled bound, +not measured end-to-end latency. If evidence cannot establish a requested bound, +the candidate is unavailable under that bound. + +The initial objective does not optimize network/storage or deployment build +cost. Account for their feasibility and any explicit limits using backend +evidence, and report them separately. Missing required evidence is not a free +component. Preserve workload manifest/pricing checks; adapting ERP costs must +not bypass them or cause another selector to silently choose a different plan. + +Solver infeasibility produces a structured no-plan result. On a solver limit, +an explicitly allowed feasible incumbent may be returned with its gap and +non-optimal status; otherwise return no plan. Report candidate coverage and +claim optimality only within the encoded feasible inventory. Final compilation +failure is a planning failure, not permission to publish or silently substitute. + +## Window realization and deployment compilation + +The prototype models sketch window `x`, slide `y`, query lookback `S`, and +interval `T`, with `x % y == 0`, `S % x == 0`, and `T % y == 0`. +An overlapping deployment updates `x / y` states per arrival and merges `S / x` +non-overlapping instances per query. Backend eligibility must additionally +validate time units, phase, coverage, supported layout, and target capabilities. + +Do not copy `x` into `WindowRealizationCandidate.window_secs`: current backend +validation requires that field to equal the query lookback. A shorter sketch +window must be represented through a supported physical layout and its read +composition. Initially admit only layouts that the current executor can realize; +the prototype's overlap support does not prove runtime support. + +After selection, physical compilation preserves the chosen parameters, layout, +sharing, and evidence identities. Validation may reject a mismatch, but must not +rerun local ERP selection and overwrite the winner. + +Here, deployment plan means the existing publication/install artifacts: +`SummaryCatalog`, `PrecomputePlan`, target-specific `CollectorPlan` entries, +`TransmissionPlan`, and `QueryPlan`, with storage routing handled by the existing +install contract. Backend-local deployments do not invent collector plans. +Use `CompiledPhysicalPlan::to_publication_artifact()` and existing installation +validation rather than introducing a parallel deployment schema. + +Planning returns artifacts; installation is an explicit subsequent action. +Activation, materialization readiness, completeness, and exact routing retain +their existing runtime contracts. Neither planning nor installation starts an +automatic optimization feedback loop. + +## Fallback policy + +Fallback is decided during this invocation. In empirical mode an ERP miss may +choose permitted exact execution. Hybrid mode may also admit supported +theoretical configurations. Alternatives require valid cost evidence and must +satisfy the query and deployment constraints. If no allowed alternative exists, +return no plan with the failed requirements. No mode labels theoretical sizing +as empirical evidence. Runtime exact routing already encoded in the chosen plan +is execution behavior, not replanning. + +## Example and acceptance criteria + +Consider same-source, same-filter KLL queries over one hour and one day, each +evaluated every minute at a common phase. Offline profiles offer several KLL +configurations; offline data facts supply rates and population counts. Planner +validates readout semantics, ERP filters applicable evidence, and backend offers +supported shared and separate layouts. Global selection chooses the lowest-CPU +feasible combination and compilation emits the corresponding maintenance and +query plans. Without evidence for the merged readouts, that shared alternative +is excluded. No live observation is needed and no later rate change triggers +replanning. + +Implementation acceptance requires: + +- A fixed offline snapshot produces one physical plan, deployment artifacts, + and an auditable report without any runtime-samples/observer dependency. +- On a small inventory, MILP cost agrees with exhaustive search, including a + case where global sharing wins over individually cheapest configurations. +- Incompatible sources, filters, partitioning, or phases cannot share state. +- Accuracy metric mismatch, unsupported merge evidence, explicit delta without + sufficient evidence, and ERP misses follow the declared fallback policy. +- Requested bounds and missing component prices fail closed; unknown costs do + not win. Solver-limit and infeasible outcomes are reported distinctly. +- Selected parameters and sharing survive compilation/publication unchanged. + An installed finite-input test checks results against an exact oracle and + verifies the selected maintenance/read paths, including window boundaries. +- Deployment does not register a replan trigger; a subsequent runtime observation + does not alter the selected generation through this path. + +Land the implementation in stages: preserve ERP alternatives and provenance; +adapt a restricted aligned recurring-query subset to executable realizations; +integrate MILP and physical cost validation; then validate publication and +execution end to end. An offline comparison harness can validate the selector +during development, but the intended deliverable includes deployment artifacts. + +## Related contracts + +- [Planner integration architecture](asapplanner-integration.md) +- [Summary Catalog and SDS architecture](summary-catalog-sds-architecture.md) +- [Existing ERP deployment adapter](../../control_plane/docs/design-erp-deployment.md) +- [Planning terminology and architecture](../developer_docs/control-plane/planning-terminology.md) +- [Shape-aware ERP](shape-aware-erp-v1.md), an existing live-observation path that + this offline proposal does not require