Repository navigation
Conversation
…ecution stack The backend now executes ASAPPlanner's selected PhysicalASAPDAG as given (#509): the Planner runs Stages 0-3, the backend supplies deployment inputs. - Pin every Planner crate the backend uses to ASAPPlanner main (58999df2): asap-types, asap-frontend-promql, asap-logical-asap-optimization, asap-plan-selection, asap-executor. The sync script updates all five. - control_plane: lower a PromQL PlanningWorkload, run Stages 1-3 (run_stage1_to_stage3) under DeploymentInputs (DeploymentCapabilities and PlanSelectionSettings), and write the selected candidate's DAG, with each PaneChain expanded into its panes, plus the query each root answers (plan.json). - data_plane: load and validate plan.json, accept Remote Write, forward queries to Prometheus until the query engine lands. - Remove the backend's own plan selection, placement pricing, ERP, SDS, precompute and query engines, ClickHouse, OTel and OpAMP paths, which were built on the removed Planner APIs.
…puts Each MatAtIngestion output the query side reads (the executor's materialization frontier) reads one tumbling pane of raw samples. Outputs that differ only in which pane they read share one SummaryDefinition, identified by a SHA-256 of their sub-DAG with node ids blanked and offset TimeShifts skipped. Each output binds to its definition with an offset: at evaluation time T it reads the pane ending at T - offset. SummaryStore keeps built panes by definition and end. A plan with a node kept across evaluations is rejected: the backend declares it cannot keep one. #509 Example 4b on a deployment without raw data: its six 10-minute KLL panes become one definition with offsets 0..50 min.
An engine thread owns the installed plan (Planner IR holds Rcs) and receives Remote Write samples from the HTTP server. For each stored summary definition it buffers its Scan metric's samples by tumbling pane (end - width, end], like PromQL ranges. When the watermark (newest sample minus --allowed-lateness-ms) passes a pane's end, it runs the plan's Scan over them (applying label matchers) and the definition's build from its TimeRange input, then stores the output. Panes without samples are stored empty; the first pane seen is skipped because it may be partial; samples for built panes are dropped and counted. A full engine queue answers Remote Write with 503 so senders retry.
Each plan root is compiled once with two kinds of inputs: stored outputs, bound at evaluation time T to their definition's pane ending at T - offset, and query-time raw ranges (TimeRange <- TimeShift* <- Scan), bound to the metric's samples in (T - offset - range, T - offset] fetched from Prometheus and filtered by the plan's Scan. An instant query whose text is a planned query is answered with a Prometheus vector and x-asap-execution: plan. Unplanned queries are forwarded; planned ones that cannot be answered (pane missing or off the pane grid, fetch or execution failure) are forwarded with x-asap-forward-reason.
For Examples 1, 3a, 3b, 4a and 4b: plan the workload, start the data_plane binary against a Prometheus holding generated samples, Remote Write the same samples, and require every planned query to be answered by the plan with the Planner executor's result for the selected logical candidate with nothing materialized (control_plane::plan_selected_at_query_time). 3b also runs, and 4b only runs, on a deployment without raw data, so their panes are built from Remote Write. Exact results must match exactly; Example 3a/4a's five-year KLL windows exceed k and KLL compaction is randomized, so they must agree within twice the 0.005 rank target.
A plan node materialized `MatAtQueryKeptAcrossEvaluations` (#509 Example 4's B3) is now executed instead of rejected. Its output is kept in memory by (definition, pane end), with the SDS identity rule, built from its pane's raw samples the first time an evaluation reads it, reused by later evaluations, and evicted like stored panes (older than the largest read offset plus two pane widths, measured from the newest evaluation). The control plane's default DeploymentCapabilities now set query_time_results_retained. examples/planner-layering-example4-b3.json is a workload for which the Planner selects B3 under the default cost model.
A pane computed at query time is kept across evaluations only once its end is at or before the precompute engine's watermark (newest sample minus the allowed lateness). A pane read while still open is computed for that evaluation but not kept, so a later evaluation computes it again, including samples that arrived late, and keeps it once it is closed.
A planned /api/v1/query_range runs one evaluation per step, start + k*step through end, and returns a Prometheus matrix. Each query-time raw range is fetched from Prometheus once for all steps and sliced per step; each kept definition's missing panes are fetched once, so a kept pane built at one step is reused by the later ones. Stored panes are read per step. If any step cannot be answered, the whole request is forwarded with the reason. start/end accept Unix seconds or RFC 3339 (now also for instant queries), step float seconds or a Prometheus duration; requests Prometheus would reject, and requests with a nonzero limit, are forwarded unchanged.
…and asap-executor only Re-pin ASAPPlanner to main f0dc1825 and use its handoff API. - The control plane calls `asap_planner::plan_installable` once and writes the returned `InstallablePlan` as plan.json. The backend's own pane-chain expansion, Stage 2/3 plumbing and query-time reference rebuild are gone; tests use `InstallablePlan::query_time_reference`. - The data plane reads `stored_outputs` (built_at, pane window, retention_ms) and `query_time_raw_windows`, and selects every pane's and raw window's rows with `asap_executor::RawSampleSelector`. It no longer uses `frontier_from_timing` or `promql_rows`. - The SDS keeps its own identity (sub-DAG hash, offset-only TimeShifts transparent) and keeps each definition for the largest `retention_ms` among its outputs, measured from the newest evaluation that can read it, instead of "largest offset plus two pane widths". - Among Planner crates the workspace depends on `asap-planner` and `asap-executor` only; IR types come through `asap_executor::planner`. - `--plan-selection-settings` on the control plane and `--execution-memory-limit-mib` on the data plane, which the benefit harness passes.
…a shared pane store - Ingestion: N workers (`--ingestion-workers`, default one per core). Remote Write series are sharded by a hash of their label set; each worker owns its shard's open panes. The global watermark is the minimum over the workers that have seen a sample; when it passes a pane, every worker builds its part, and the last part completes the pane: parts are joined and the states of a group several workers saw are merged with the family's merge. A request is queued to every worker or refused whole. - Queries: a pool of `--query-threads` threads, one query (and every step of a range query) per thread. The executor's compiled plans hold `Rc`s, so each thread reads plan.json and compiles its own. - SDS: `RwLock<HashMap<(definition, pane end), Arc<pane>>>`. Panes are inserted once and never rewritten; readers clone the `Arc`. Kept query-time results use the same store, shared by the query threads. A stored pane binds to a root as a shared source, copied once per run (the executor's sources emit owned batches) instead of twice. - jemalloc is the data plane's global allocator.
…he watermark it announces Two races of the sharded engine, both found by a test that writes a pane's samples and queries it at once: - A planned query ran on the query pool while the ingestion workers could still be ingesting the Remote Write requests answered before it, so the newest pane was missing and the query was forwarded. A query now waits until every worker has run what was queued before it (`sync`), as the single engine thread's queue ordered it. - A worker closed its panes at the watermark it read before another worker moved it, then announced the newer watermark to the other workers but not to itself, so its part of the newest pane never arrived. It now closes and announces the same watermark.
The ingestion side of a stored definition, Scan -> TimeShift* -> TimeRange(pane) -> SummaryAgg, runs per sample instead of over each pane's buffered rows: - per series, once: the executor's RawSampleSelector turns the series' first sample into its input row (applying the Scan's matchers), and the row's group columns give its group; cached by label set; - per sample: the open pane's state for that group is updated in place with the updater the executor's batch build uses for the family (create_planner_accumulator), with the same value and timestamp; - per group, once: the definition's compiled build runs over the group's first row, and its output row is the template each closed pane's row is made from (pane timestamp and state replaced). A build of another shape (row filter, keyed or shared-grid family, operators between the range and the summary, matchers reading the value or time) keeps the batch build. A parity test checks every family the examples store (counter rate, sum, KLL at two pane widths) against the executor's batch build of the same samples: exact states byte for byte, KLL by quantile (KLL seeds its compaction randomly per sketch).
… and Prometheus remote read - A local raw buffer keeps the recent samples Remote Write delivered, sharded like ingestion: each ingestion worker appends its shard. It keeps the plan's longest query-time raw read back from the newest sample: the largest `end_offset + width` of `query_time_raw_windows` and of the panes built at query time and kept (their retention, which covers range evaluations). `--raw-buffer-ms` overrides it; 0 keeps none. - A raw window is read from the buffer; the part before the buffer's cut (its first sample, or the trimmed horizon of any shard) is read with Prometheus remote read (protobuf, `/api/v1/read`). The JSON query API is no longer used for raw samples. Forwarding of unplanned queries is unchanged. - The data plane logs the buffer's series and samples every 30 s.
…est query-time read The buffer kept every metric for the plan's single longest raw read: in the benefit plan, 60 minutes of the counter whose raw reads span one minute, and of metrics no raw window reads. It now keeps only the metrics the plan reads at query time, each for its own longest read (query-time raw windows, and panes built at query time and kept).
…ompile once; per-sample panes from columns Re-pin ASAPPlanner to main 92bc1296: the executor runs every operator on Arrow batches, compiled plans and RawSampleSelector are Send + Sync, and StateColumnUpdater builds a state column sample by sample. - Compile once: the plan's ingestion side (Ingestion) and its query roots (QueryEngine) are compiled when the plan is installed and shared by every ingestion worker and query thread through Arc. - Raw samples to columns: a raw-sample window's input batch is built from the samples' times and values and each series' cached cells (series identity and label columns, dictionary-encoded), with the Scan's matchers applied once per series through RawSampleSelector. Query-time raw windows, kept panes and the batch-build fallback all use it. - Ingestion per sample, no rows: a definition whose group is a series runs StateColumnUpdater::update in place per sample; at pane close each worker seals its states and builds its part with Batch::try_from_columns. Workers own disjoint series, so a pane is the workers' batches, shared by Arc in the SDS. Another shape keeps the plan's build, now run once over every worker's samples of the pane. - Stored panes bind through Operator::source, which treats them as held outside the run. - Stored panes are kept one pane width longer than the plan's retention: an evaluation at T runs once samples just after T have arrived, so the newest stored pane can be past T. - Rows are read only to encode answers as Prometheus JSON.
…ture by id Each label set is interned once, when Remote Write or remote read delivers it: an ahash (fixed seeds) of its label bytes finds or creates one shared `Series` with a dense `u32` id, the hash and the metric. Samples carry the `Arc<Series>`; the router shards by the hash, and the per-series cache, the per-sample group numbers, the raw buffer and the instant-selector latest-sample map key by the id. The per-request label map build and the per-sample SipHash and label-map comparisons are gone; text columns' dictionaries use ahash.
…o the Planner - `measure_costs` runs the executor's own summary operators as the data plane runs them, on one core: the `SummaryAgg` build per input row, `SummaryMerge` per input state and `SummaryEstimate` per state, for the sketch configurations of given plans and of a base table, and writes them as `MeasuredOperationCost` entries (base entries it does not measure are kept). Costs the table cannot hold go to `--report`: exact accumulators, the per-sample ingestion update, and each raw-reading query of a plan evaluated end to end per raw sample. - `control_plane --prediction <file>` writes Stage 3's predicted CPU, memory and price per second for the selected plan, and how many of its nodes were priced from measured costs. - `scripts/bench/cost_feedback.sh` measures, writes the settings with the measured table, and plans before and after with predictions.
zzylol
added
enhancement
zzylol
force-pushed
the
feat/intern-series
branch
3 times, most recently
from
October 21, 2026 00:27
132d418 to
244e56d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 7/7. Base: #834.
Why
Stage 3 predicted 0.07 core-seconds for the plan it selects at s10 with raw data, and the data plane measured 14–16. Its cost inputs price the sketch kernels alone, from sketch-bench, while the runtime pays the operators around them. Design doc, Stage E: feed the runtime's measured costs back to the Planner, and check its predictions against measured runs.
What
measure_costs(data_plane binary) measures the data plane's operation costs on this machine, on one core, running the executor's own operators as the data plane runs them:update: theSummaryAggbuild per input row (the query-time build);merge:SummaryMergeper input state;estimate:SummaryEstimateper state (p99, or a distinct count).It writes them as ASAPPlanner
MeasuredOperationCostentries for the sketch configurations of the given plans and of a base table; base entries it does not measure (Count-Min with heap) are kept. Costs the table cannot hold go to--report.control_plane --prediction <file>writes Stage 3's predicted CPU core-seconds, memory and price per second for the selected plan, and how many of its nodes were priced from measured costs.InstallablePlan.selectioncarries only the price.scripts/bench/cost_feedback.shmeasures, writes the deployment'sPlanSelectionSettingswith the measured table, and plans before and after with predictions.How
PhysicalDAG(a source of prepared batches and the operator) through the executor's public API, timed over 1,000 series × 40 samples (6 panes for merges), median of 5 repetitions.QueryEngine, a fresh engine each time, so no kept pane is reused.asap_planner::e2e_planwith the same inputs asplan_installable, the executor's summary families included, and reads the selected candidate's usage.Verification
cargo test --workspacepasses, includinge2e_examples(10). Newplan_workload::predicts_the_installed_plan_and_prices_from_measured_costs: the prediction names the installed candidate, and a measured cost for its KLL configuration moves its nodes from formula to measured prices.Measured on this machine (one core, median of 5)
KLL
updatecame out 4–7× the sketch-bench figure (117–201 ns against 29–31 ns). The kernel matches sketch-bench; the rest is the operator around it: batching, grouping, a state per group. KLLestimateat k ≥ 200 came out 3–20× below sketch-bench, because a merged state is estimated once rather than re-sorted per call.Raw-reading queries end to end, per raw sample: 0.16 µs at 240,000 samples (p99 over 60 min), 0.61 µs at 20,000 (p99 over 5 min), 3.6 µs at 4,000 (counter queries; mostly the fixed cost of each evaluation).
Stage 3's predictions against the measured run
Predicted data-plane CPU over the 90-minute run, in core-seconds. "Measured" is the data plane on #834 before this PR, running the "before" plan.
Runs with the re-planned settings
Same 90-minute harness and data-plane binary (#834); only the plan changes. Every evaluation met its accuracy target.
The plan Stage 3 now picks for raw-kept configs cannot run in the data plane. P15-m8's exact
quantile_over_timeplan fails executor validation withper-entity aggregate requires an explicit entity binding, so the data plane forwarded all 90 quantile evaluations to Prometheus ("forwarded" column). Its lower data-plane CPU (s10 13.4 to 11.2, s50 64.8 to 52.0) is work moved to Prometheus (6.0 to 8.6 and 29.0 to 41.7 core-s). It is not a saving: the total is 18.9 against 19.8 core-s at s10, and 93.8 against 93.7 at s50. The noraw configs keep their plan, and their runs match #834 within noise.s10
s10-noraw
s50
s50-noraw
What the measured-cost schema cannot express (for an ASAPPlanner issue)
SketchParamsand one ofupdate,merge,estimate. Exact accumulators (rate, increase, sum, count, min, max) have noSketchParams, so their measured costs cannot be given: update 49–51 ns/row at query time and 18 ns/sample at ingestion; merge 175–194 ns/state. Every exact node stays priced by formula.Scan/TimeRangeselection, exactquantile_over_timeand other range functions over raw rows,Sort/Limit/Project, joins, the group tables of exact aggregates. Measured end to end per raw sample (raw window plus everything the query does over it): 0.16 µs at 240,000 samples (p99 over 60 min), 0.61 µs at 20,000 (p99 over 5 min), 3.6 µs at 4,000 (the counter queries, which build six kept panes).updateper sketch, whatever the materialization. The data plane pays very different costs for the same sketch update: KLL 170 ns/row in the query-time batch build against 28 ns/sample in the ingestion updater; DDSketch 208 against 173. Stage 3 must use one number for both, so it misprices either ingestion-time or query-time plans. The table here uses the query-time build cost.SketchParamsequality. Stage 3 sizes KLL'skfrom the accuracy target and the data (469 at 1,000 series, 506 at 5,000), so a deployment cannot know beforehand whichkits candidates use. The sketch-bench table (k ∈ {25, …, 800}) priced none of them. Either a cost per parameter (e.g. update and merge as functions ofk), or the installable plan listing the configurations its candidates use, would let a runtime measure what matters.per-entity aggregate requires an explicit entity binding): Stage 3 selected a candidate the runtime cannot execute.InstallablePlan.selectioncarries the price per second but not the predicted usage (CPU, memory), so the runtime callse2e_plana second time to compare prediction with measurement.