[zzylol]

Answers Milind's question on Slack: how to run the flow end to end, from inferring parameters on real traces for a query workload to giving them to the rqe-optimizer.

What

docs/e2e_planning_flow.md, linked from the README. It covers:

1. the workload as RAQEs;
2. fitting the trace: ASAPQuery fit_skew.py → skew_summary.csv columns → MetricFacts/data_shape;
3. measuring the sketches with study_saturation.py: the accuracy grid with merge curves, the 1e9 subset, --phase optimizer-cost, --cost-shape, and the directory layout SaturationCurves::load expects;
4. running small_problem --milp and ASAPQuery's asap-optimizer-cli --milp (ASAPQuery#804), including the load-time checks and how accuracy is read;
5. reading the plan.

It ends with what isn't automated yet. Commands, flags and paths were checked against main (through #178/#179) and ASAPQuery#804.

Gaps it documents

• Expanding PromQL into RAQEs is manual.
• skew_summary.csv → facts is manual: no script writes data_shape, cardinality or value_range.
• The 1e9 subset's points and the --merge-shards-list values are picked by hand from the workload's N and m = S/x.
• Spatial filters are rejected.
• The ASAPQuery path depends on the open ASAPQuery#804.

🤖 Generated with Claude Code

[milindsrivastava1997]

What's kind, weight, range and how does it relate to data_shape?

[zzylol]

They're skew_summary.csv's key columns, from ASAPQuery asap-tools/dataset-analysis (see its README, "Definitions"):

• kind: keys or values.
  • A key query (group-by keys: sums, top-k, distinct counts) gets the Zipf fit → DataShape.zipf_s (θ) and distinct_keys (K).
  • A value query (quantiles) gets the power-law tail fit → DataShape.tail_index (a).
• weight (key queries only): what each key's rank-frequency counts. count is the row count per key; value is the value sum per key, with negatives clipped.
  • Use the one that matches how the sketch is fed: once per sample (count, e.g. topk over count_over_time) or weighted by the sample value (value, e.g. topk over sum/rate).
  • DataShape.zipf_s describes whichever insertion mode the engine uses.
• range: the query's lookback (instant, 5m, 1h, …), i.e. the RAQE's S. Each row is that query evaluated at every step with that window.

The relation to data_shape: WorkloadFacts.data_shape is keyed by (metric, grouping), not by range. If several RAQEs on one grouping have different ranges, take the worst case over them: smallest θ, largest K, smallest a. That's what the AutoSketch-vs-ASAP runner does per stream. I've added this to the doc.

🤖 Generated with Claude Code