Repository navigation
rqe-optimizer: fall back to each sketch's guarantee where the study measured nothing - #180
Merged
Merged
Conversation
…easured nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…better than measured From review on #180: - CMS top-k: the per-key 1 − e^−rows wasn't a guarantee for the whole top k. Now Markov per row, independent rows and a union bound over the K − k other keys give each top key's failure probability; keys count from the most frequent while the summed failure stays within 5%. - A fallback is never better than the last measurement it extends, and it applies only at or above the first N of the curves it would read, merge curves included. - HLL uses the one-sided 95% normal quantile (1.645), as CONFIDENCE says. - The Zipf normalizer is exact for the first 1000 terms and an Euler–Maclaurin tail beyond, so large K is O(1). - accuracy_with_source reports the source of the worst value only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ided rank and relative errors From the second review of #180: - A fallback's floor, and its first-N gate, read the measured shard counts either side of the merge count, the curves merged_error_at would have read, not only the smaller one or every merge curve. - KLL rank error and HLL relative error are absolute, so their 95% bounds are two-sided (KLL scaled by sqrt(ln 40 / ln 200), HLL z = 1.96). The top-k bound stays one-sided: CMS only overestimates. - Docs: the source is the returned value's; DDSketch's alpha isn't a hard bound on the measured metric, which the floor covers; the top-k bound assumes the heap ranks by final CMS estimates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 7, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 7, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 7, 2026
… in the planning flow From review on #181: - skew_summary.csv's kind, weight and range columns, and how ranges map onto data_shape (keyed by grouping: the worst case over its RAQEs' ranges). - A 'Fanning out over machines' section and scripts/merge_study_parts.py, which concatenates per-machine run directories, refusing a point in two parts. - Brought up to date: top-k needs no merge curves (heap m·k, #182), the guarantee fallback is merged (#180), main is through #183. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 7, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 7, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 8, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 8, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 9, 2026
… the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
added a commit
that referenced
this pull request
Oct 9, 2026
…ontier and SLA versions) (#138) * rqe-optimizer: evaluate AutoSketch vs ASAP on the synthetic and trace workloads ProjectASAP/ASAPQuery#777 (paper §6.3), consolidating #138, #139 and #141. - rqe-optimizer/examples/autosketch_vs_asap.rs (traces and synthetic): ASAP (milp::minimize), PerQuery-CostAware and AutoSketch-Adapted. Costs and accuracy come from --saturation-dir, as the planner reads them: SaturationCurves::accuracy for ASAP (merge curves for KLL and top-k, #179), autosketch_accuracy for AutoSketch, and the one-study cost table (#178). Each run is scored by analytical_cost_model::score at #777's two weight settings and the SLA grid. - scripts/run_autosketch_vs_asap.sh and plot_autosketch_vs_asap.py: the trace workloads alibaba_v2022 and google_2011. boom is left out for now. - scripts/export_autosketch_eval_table.py --synthetic: one workload-only table per grid point, plus plan.tsv. The data constants come from study_saturation's COST_*. The traces path is unchanged from main. - scripts/run_autosketch_vs_asap_synthetic.sh and plot_autosketch_vs_asap_synthetic.py: run the synthetic grid, then plot. - scripts/profiling_time.py and the results README: ASAP's one-time profiling time, and how to reproduce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: trace key queries keep one accumulator per key; report the accuracy source From review on #138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: one accuracy level, p95; plots join one RQE set and charge AutoSketch's benchmark - One accuracy level for every workload, synthetic and traces: 95% in each family's metric (rank, relative value and relative error at most 0.05; top-k precision at least 0.95). The strictness dimension is gone. - fig_objective_vs_latency joins only points over one RQE set (solid for the full set, dashed for the subset tight SLAs keep). - fig_planning_time adds AutoSketch's charged benchmark time (lower bound and 60 s per probe), as #777 section 7 counts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: summaries show Σ window/slide; synthetic summary to its own file AutoSketch-Adapted's ingest cost is window / slide overlapping sketches per deployment (window = the query's lookback), so the column explains its gap to ASAP. The synthetic and trace scripts both wrote summary.md into one results directory; the synthetic one is now summary_synthetic.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: RQEs carry their top-k k (#185) Raqe now has topk_k. The tables name it only in the query id (topk32_...), so the runner reads it there and stops on a top-k query that names none. On the p95 synthetic and trace tables every top-k query is k = 32, and the plans are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: p95 results for the synthetic grid and the traces Runs on clnode155 over the asap_sketchlib 0.3.0 study (#174, #178–#186): four synthetic workloads and alibaba_v2022 / google_2011, raw JSON, summaries and figures. The README gives the no-SLA headline, including Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is refilled from the study's raw records (0.77 h; the K = 1e4 column refill of #186 is its own run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: figures and tables give each method's absolute cost Cost figures label every bar and point with its value; the summaries and the results README list each method's objective instead of its ratio to ASAP. The by-dimension figure's legend moves above the panels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: metrics dimension, one dashboard per metric (21 · m RQEs) Shared replicas on one metric stop adding RQEs past r = 8 (52 at r = 8, 57 at r = 64), so the grid keeps shared r ∈ {1, 8} and adds metrics m ∈ {1, 8, 16}: copy i of the template set reads its own metric data_i, so nothing is shared across copies and RQEs grow as 21 · m (168, 336). The planning-time figure plots this dimension. The m = 8 and 16 runs are added; the r = 64 result is removed. Existing tables are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: AutoSketch benchmarks each config once per metric Synthetic tables name the metric each RQE reads (data, or data_i in the metrics dimension), and AutoSketch-Adapted's benchmark time counts a probed config once per metric, on that metric's data: 540, 4320 and 8640 s (60 s per probe) for m = 1, 8, 16. Trace tables name none and read one metric per dataset, as before. All five synthetic workloads are rerun with the current runner on four nodes; objectives are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: cost vs. total estimated latency for every workload One column per workload (five synthetic, two traces), one row per weight setting, one line per method across the SLA grid, each point labeled with its cost. Total latency is the sum of the RQEs' estimated latencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: finer SLA grid, {0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10} ms and none Checks whether AutoSketch-Adapted's memory objective makes it miss an SLA: it never does. Its plans never merge, so each RQE's latency is within 1.4% of the lowest any deployment reaches (equal for quantile, exact and trace RQEs), and RQEs no method can meet are excluded for all methods. All workloads rerun on four nodes; figures and summaries regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: AutoSketch probe list and measured benchmark time; traces with 6h/24h windows - The runner lists AutoSketch-Adapted's distinct probed (metric, config) pairs with the data shape each is benchmarked on, and drops the benchmark-time lower bound. scripts/autosketch_benchmark_time.py runs each probe's approxbench accuracy benchmark (1e8 items, exact baseline and scoring) and times it, as AutoSketch benchmarks probed configs. - Trace streams with more keys than samples per second are left out and listed (rounding their rate up to their key count inflated every method's cost). - The trace table is regenerated from ASAPQuery #812's skew summary (6h and 24h windows, a day of Alibaba). - Plots: one line per method over the full RQE set; AutoSketch's planning time with its measured benchmark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: cost by use (sketch-bench #188); Pareto frontier sweep; mixed set by default - The runner prices every plan by use (usage::usage_cost): w_cpu * AUC(CPU) + w_mem * AUC(memory), CPU elastic, latency the longest chain, reported. - ASAP is milp::minimize_usage_cost over every RQE; PerQuery the same over each RQE's own candidates (separate copies via `allowed`, no sharing); AutoSketch's plan is priced the same way. - No SLA grid. Each method's cost-latency frontier: ASAP and PerQuery are solved unbounded and for 12 bounds log-spaced from the tightest feasible one to the unbounded latency, plus AutoSketch's latency. - The synthetic grid is the mixed template set: shared r in {1, 8} and metrics m in {1, 8, 16} (50, 92, 400, 800 RQEs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: version 1 (no SLA) results on the mixed set, with AutoSketch's measured benchmark - Four workloads of the mixed template set (50, 92 with r = 8, 400 with m = 8, 800 with m = 16 RQEs), cost by use (#188), each method's cheapest plan and ASAP's and PerQuery's cost-latency frontiers (12 bounds plus AutoSketch's latency), CPU-only and Fargate weights, --runs 3, one node per workload. No RQE dropped, no sanity violation. - AutoSketch's benchmark time is measured: approxbench's accuracy run (1e8 items, exact baseline, scoring) once per distinct probed config and shape, serially on idle clnode155 (21-37 s each), charged once per probed (metric, config): 212, 1697 and 3394 s for m = 1, 8, 16. - plot_autosketch_vs_asap_synthetic.py: fig_frontier.png (cost vs. query latency, every point labeled), fig_planning_time.png (AutoSketch as search + measured benchmark; 60 s per probe only as a labeled reference), summary_synthetic.md. The SLA-era figures, the dashboard results and plot_autosketch_vs_asap_workloads.py are removed. - README: the cost model, version 1, how to reproduce, and the headline numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: version 2, a batch latency SLA, on the synthetic mixed set The runner adds version 2 of the evaluation beside version 1 (no latency constraint and the frontiers, unchanged in `results`): at each SLA of a grid (default 100, 300, 1000, 3000, 10000 ms; --slas-ms overrides), ASAP and PerQuery plan the cheapest plan whose batch latency (the longest chain) is at most the SLA, via minimize_usage_cost's latency bound, which is exact with elastic CPU. An SLA below a method's tightest feasible bound gives an infeasible record. AutoSketch's one plan is recorded at each SLA with meets_sla. Records go to `sla_results`, with sanity checks that every plan meets its SLA and that ASAP costs no more than PerQuery. Results on clnode109 (--runs 3) for the mixed set, r = 8 and m = 8, 16: no drops, no sanity violations; the tightest feasible SLA is 92 ms, and AutoSketch (latency 1011 ms) meets only 3000 and 10000 ms. AutoSketch's planning time is search plus its measured benchmark (scripts/autosketch_benchmark_time.py: approxbench accuracy runs at 1e8 items, 20-33 s each, serial and alone on the node): 197 s at m = 1, 3159 s at m = 16. scripts/plot_autosketch_vs_asap_sla.py draws fig_cost_vs_sla.png and summary_sla.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: remove the old trace results and figures They came from the earlier model (per-RQE SLA over a µs grid, mean-CPU cost); the traces are to be rerun on cost by use with both latency versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: trim the PR to what it needs - Result JSONs without per-RQE choices (`chosen`): the figures and summaries don't read them, and they were nearly all of the 2.9M lines; rerun the runner for them. - Machine files folded into the READMEs (all Intel Xeon E5-2683 v3, 56 cores); the synthetic run script no longer writes them. - The SLA-era trace scripts (plot_autosketch_vs_asap.py, run_autosketch_vs_asap.sh) are removed; the traces are to be rerun on cost by use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: keep only what the three figures need The PR's results are the frontier (version 1), cost vs. SLA (version 2) and planning time on the synthetic mixed set. The trace table goes back to the base version (the traces are to be rerun on cost by use, with their new table), and the ASAP profiling-time script and data, which no figure uses, are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: commit the inputs of the three figures rqe-optimizer/results/autosketch-vs-asap-inputs/ holds what the figures were computed from: the sketch-bench study on asap_sketchlib 0.3.0 (the cost table, sketches and exact aggregations, and the accuracy curves the runner reads as --saturation-dir) and the synthetic mixed-set tables with plan.tsv. The READMEs point the reproduce steps at them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: the raw approxbench records behind the figures' study rqe-optimizer/results/autosketch-vs-asap-inputs/raw/ holds the gzipped approxbench records (920 KB) that ../saturation/ was reduced from, with a README mapping each file to its machine, arguments, record count, run time and output. Rerunning them is 0.77 h of machine time over 5 machines in parallel (about 20-30 minutes of wall time). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Oct 11, 2026
…181) * docs: end-to-end planning flow from traces to an rqe-optimizer plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: kind/weight/range and data_shape, fanning out, and #180/#182/#183 in the planning flow From review on #181: - skew_summary.csv's kind, weight and range columns, and how ranges map onto data_shape (keyed by grouping: the worst case over its RAQEs' ranges). - A 'Fanning out over machines' section and scripts/merge_study_parts.py, which concatenates per-machine run directories, refusing a point in two parts. - Brought up to date: top-k needs no merge curves (heap m·k, #182), the guarantee fallback is merged (#180), main is through #183. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Since #171 and #179, the planner reads sketch accuracy from the saturation study's curves. Where the study measured nothing, the accuracy is unknown and the deployment ineligible:
Each algorithm has a published guarantee that holds there, independent of N.
What
rqe_optimizer::theory::bound: each sketch's guarantee at 95% confidence, in the metric the optimizer checks it by.2.296 / k^0.9723(DataSketches' 99% value), scaled to 95% bysqrt(ln 20 / ln 100), since KLL's error scales withsqrt(ln(1/δ)). It survives merges.α.1.96 · 1.04 / sqrt(2^lg_k).kkeys whose probability beats the (k+1)-th's by more thane/w. It needs1 − e^−rows ≥ 95%.SaturationCurves::accuracyfalls back to the bound for a bracketing point at or above its first N when the curves give nothing. Newaccuracy_with_sourcereportsAccuracySource::{Measured, Theory}.rqe_sketch_deployment_v1.mdeligibility.Verification
cargo test -p rqe-optimizer(65): bound values, the top-k rows/merge conditions, and sketches without a bound.-D warningsand fmt are clean.🤖 Generated with Claude Code