Repository navigation
Stage 3 under-predicts CPU ~740×: measured costs price only sketch kernels, not scans, per-row work or merges #761
Description
Problem, by example
ASAPQuery-backend ran the planner-layering PromQL examples on 2,000 series (90 minutes of data, 15 s samples, raw data retained). It ran Stage 3's selected plan with AWS Fargate prices and sketch-bench's measured operation costs, using a release build and perf sampling.
| Stage 3 predicted | Measured | |
|---|---|---|
| Data plane CPU for the selected plan | 0.072 core-seconds | 53 core-seconds (about 740× more) |
| Peak memory, 10,000-series plan with no raw data | 1.06 GiB | 1.69 GiB |
The sketch kernels that measured_operation_costs prices are only 3–7% of the measured CPU. perf breaks down the rest:
- Row handling in the executor: allocation and free 32%, row clones 11%, row drops 5%, grouping keys 9%, per-row memory accounting 9%, batch validation 2%.
- Raw-sample transfer: reading raw ranges back from Prometheus as JSON is about 15.5%, about 0.89 µs per sample.
- Scans and merges: Scan rows 5.6%, building kept panes 7.4%, merging and copying stored panes about 4%.
Because those costs are missing, Stage 3 underprices query-time plans that rebuild state from raw samples. With raw data retained it selected no ingestion-time outputs at all. The two quantile queries then rebuild 1,000 per-series KLLs at every evaluation, taking 57 and 313 ms in-process, against 31 and 62 ms when they read stored panes.
Cause
Stage 3's CPU estimate prices each operator by its kernel operations: sketch updates, merges and estimates, through measured or analytical per-operation costs. It does not price:
- Scanning raw samples: per-sample decode and transfer from the data source, with the cost depending on where the raw data lives (local vs remote, JSON vs binary).
- Per-row operator overhead: materializing, grouping, hashing and moving rows between operators, which is independent of the kernel.
- Merging and copying stored state: reading N panes and merging them per evaluation, beyond the kernel's own merge cost.
- Memory overheads: row layout, label strings and decoded buffers on top of the summary state.
Proposal
- Add per-sample and per-row cost terms to
MeasuredOperationCost: scan per sample, per-row operator overhead, and merge or copy per stored pane. A runtime measures these and supplies them, so the Planner stays deployment-agnostic and only consumes numbers. - Keep the analytical defaults for these terms, but make them realistic: derive them from a reference run instead of zero.
- Report predicted vs measured per plan in the stage_pipeline output, so the gap stays visible.
Acceptance
- With the runtime-measured terms supplied, Stage 3's predicted CPU for the example above is within 2× of the measured 53 core-seconds.
- A test pins that the scan and per-row terms change the choice between rebuilding from raw samples and reading stored panes, in the direction the measurements show.
Related
#754 (kept results almost never selected), #726 (sketch memory sized by full capacity), #734 (measured operation costs). ASAPQuery-backend design-doc PR #826 has the full measurement ("Overhead of the data plane" section).
Activity
zzylol commentedon Oct 21, 2026
Closing as covered by #789. The Planner side of this issue is implemented there; the rest is tracked in the issues below.
Status on
14394adaScanentry per phase, priced per sample. The runtime supplies the number for where its raw data lives (local or remote, JSON or binary).OperatorKind::Scan,OperatorCost.phase; testexternal_raw_reads_make_the_query_time_raw_plan_costlierns_per_unitper input row,ns_per_invocationper computation), including its row handling. Per-request overhead isns_per_evaluation.OperatorKind,MeasuredOperationCostsSummaryMergeis priced per state read,PaneChainper row, at each phase.compaction_prices_merges_from_the_measured_mergeSelectedPlanReport.predicted_usage: totals, per node, per query). stage_pipeline doesn't print per-node CPU yet. The measured side is the runtime's.external_raw_reads_make_the_query_time_raw_plan_costlier,calibration_keeps_unmeasured_nodes_on_the_measured_scaleFollow-ups
Scan,TimeRangeorAggregateentry. Without a table, a raw-sample scan is priced at about 4 ns, against about 890 ns measured here.cpu_core_seconds_per_second), operator and timing in stage_pipeline's Stage 3 cost output.