Repository navigation
feat(planner): serve the #777 eval's sum-of-rate and nested topk queries #827
Description
Goal
Serve the three query shapes from the #777 eval's classic workload that ASAPQuery rejects today, so its saved plans can run end to end (#824).
| Shape | Queries | Eval plan's sketch |
|---|---|---|
sum by (job) (rate(data_total[w])) |
4 (w = 15m, 1h, 6h, 24h) | exact-increase grouped by job (10 groups), 60s tumbling |
topk(32, sum by (label_0) (rate(data_total[w]))) |
4 | cms-heap-topk-fastpath-vector2d keyed by label_0 (rows 3, cols 1024, heap 46080), 60s tumbling |
topk(32, sum by (label_0) (sum_over_time(data[w]))) |
4 | the same CMS + heap |
These are 12 of the 54 classic queries. asap-planner --planner saved refuses them, so the e2e run is blocked on this issue.
What is missing
- Planner query patterns (
build_patterns,get_is_collapsable):sum(rate)is not collapsable, and nothing matches three levels (topk(k, sum by (l) (f(m[w])))). The MILP extractor and--planner savedboth fail withno supported query pattern matched. - Engine: for
sum by (job) (rate(...)), Nested queries: outer aggregation is silently dropped for planner-generated (non-collapsable)agg(fn(m[w]))queries #496 shows the outer aggregation is dropped at query time. The nestedtopkneeds feat(planner): support nested temporal aggregation pipelines #772, and serving it from one CMS with heap is perf(planner): collapse topk(sum by ...) into CMS-with-heap #744. - Counter semantics: the eval treats
data_totalas a stream of increments (stream: job/increment), so one increase perjobis valid there. In the engine,data_totalis cumulative per series, and Nested queries: outer aggregation is silently dropped for planner-generated (non-collapsable)agg(fn(m[w]))queries #496 explains why a group-keyed increase is wrong (counter resets interleave across series). Matching the eval's plan needs a per-series delta at ingest that feeds the per-jobsum or the CMS. Without it, the deployments differ from the plan (per-series, 1e4 keys).
Done when
asap-planner --planner savedconverts sketch-bench's classic base table (synthetic-templatesclassic-shared1, ASAP cheapest at CPU weights) with all 54 queries, into 8 aggregations, one per plan deployment.- These queries' results from the engine match Prometheus on the e2e workload.
Activity
milindsrivastava1997 commentedon Oct 13, 2026
Narrowed scope
After a scoping discussion, this issue is now planner only, and covers one of the three shapes.
In scope:
topk(k, sum by (l) (sum_over_time(m[w]))), 4 queries. The planner gets one three-level pattern that collapses this shape into a singleCountMinSketchWithHeap, keyed bylwith sum weighting. That matches the eval plan's CMS+heap deployment.sum_over_timeis linear, so a CMS keyed by group gives the per-group sum up to sketch error. The pattern goes only into the planner'sbuild_patterns. The sharedget_is_collapsablestays as it is, so the engine's query-time matching doesn't change.Out of scope, deferred to a per-series delta-at-ingest follow-up:
sum by (job) (rate(data_total[w]))topk(32, sum by (label_0) (rate(data_total[w])))Both need per-series increments of the cumulative
data_totalbefore grouping. A CMS or increase keyed by group over cumulative values is wrong, as explained in #496. These 8 queries are dropped from the input used in #824. The planner keeps failing loudly on unsupported leaves.Moved elsewhere:
Updated done-when:
asap-planner --planner savedconverts the 46 remaining classic queries. There is one aggregation per plan deployment those queries reference, and each query routes to its plan's deployment. The test asserts this against the saved plan, not a hard-coded count.