rqe-optimizer: check each family by its own accuracy metric by milindsrivastava1997 · Pull Request #173 · ProjectASAP/sketch-bench

milindsrivastava1997 · 2026-10-07T15:57:21Z

Closes #167

Summary

DDSketch rows report mean_relative_value_error (#164), but quantile RAQEs asked for mean_rank_err, so DD was never feasible. Each family now declares its own accuracy metric, and a RAQE's single accuracy_sla is checked against whichever metric the candidate's family reports.

Design decisions (please approve each)

The family decides the metric. FamilyProperties::accuracy: Option<(&str, AccuracyDirection)>, mirroring the comparator export_rqe_optimizer_costs.sh runs per variant:
- exact-*, hll, univmon-cardinality → relative_error ↓ (for exact accumulators this is the worst group, not relative_error_mean)
- kll-percall, hydra-kll → mean_rank_err ↓
- dd → mean_relative_value_error ↓
- top-k variants → precision_at_k ↑
- exact-delta-set (key tracker) → None; it is never a candidate.
Direction moves from the RAQE to the family. Raqe::accuracy_metric and Raqe::accuracy_direction are removed.
The RAQE keeps one scalar accuracy_sla for every family. The same number means different guarantees for rank error and relative value error; that caveat is tracked in rqe-optimizer: one quantile accuracy SLA means different guarantees for KLL and DD #172.
Exact accumulators are checked like any family. There is no "always exact" special case, so a broken accumulator is still caught.
A missing metric panics, on both accuracy sources. A cost-table row (or merge bracket) lacking its family's metric panics in table_accuracy_at, including DD's intentional omission (every true quantile is 0, or every probe is unanswered), which the message names. A saturation curve whose error_metric differs from its family's metric is refused by SaturationCurves::load as a stale study (it used to make the point infeasible). This includes today's DD curves, which record the max: regenerating them with the mean is Saturation study records DDSketch max relative value error; optimizer reads mean #175, and DD can't be planned until then. Points that simply weren't measured (config or shape outside the grid, unsaturated) stay None, i.e. infeasible.
AutoSketch keeps rqe-optimizer: read sketch accuracy from saturation curves #171's accuracy oracle. Raqe::meets_sla(sketch, accuracy) takes the direction from the family; the MILP's eligibility and AutoSketch both decide there.
dd is added to DEPLOYABLE_FAMILIES as ASAPQuery's single-population AggregationType::DDSketch.
validate_facts rejects an accuracy_sla no metric can meet: non-finite, negative, or above 1 when the capability has a higher-is-better (precision) family. An error ceiling above 1 stays valid, since relative_error can exceed 1. Without this, such a RAQE read as unservable instead of as bad input.

Test changes

New: one quantile RAQE checked against a KLL row and a DD row, each by its own metric; a #[should_panic] test for a DD row missing its metric.
Shared fixtures carry every metric key at a perfect value with an SLA of 0.5, so a row renamed to any family stays feasible. The AutoSketch top-k fixtures use precision floors (1 - error_budget).
load_refuses_a_curve_for_another_metric replaces the saturation test that expected None for a mismatched metric.
exact_accumulators_keep_the_cost_table_value: the exact-sum fixture reports relative_error instead of the top-k precision_at_k.
The undeployable-family test now uses countsketch-heap instead of dd.
small_problem.rs and the docs (rqe_sketch_deployment_v1.md, rqe_optimizer_TODO.md) no longer have a per-RAQE metric.

Validation

cargo test -p rqe-optimizer (60 passed), after rebasing onto rqe-optimizer: read sketch accuracy from saturation curves #171
cargo clippy -p rqe-optimizer --all-targets clean
cargo build --workspace --no-default-features

🤖 Generated with Claude Code

Families declare their accuracy metric and direction in FamilyProperties; a RAQE gives one accuracy_sla checked against each candidate's metric, so DDSketch (mean_relative_value_error) and KLL (mean_rank_err) can both serve one quantile RAQE. A cost-table row missing its family's metric, or a saturation curve for another metric, panics. dd is deployable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

milindsrivastava1997

Code review findings. #1 (inline on lib.rs:175) is blocking and verified; the rest were not independently verified.

Not on a diff line, rqe-optimizer/src/saturation.rs:290: check_cost_table checks the cost table against the curve's own point.error_metric, not the family's accuracy_key. So it reports nothing for a curve with the wrong metric, and small_problem panics later during candidate generation instead of failing early. Suggest checking point.error_metric == accuracy_key(sketch).0 there (or at load) and reporting it in mismatched.

🤖 Generated with Claude Code

milindsrivastava1997 · 2026-10-07T16:22:24Z

+        },
+        "hll" | "univmon-cardinality" => one_sketch_per_group("relative_error", LowerIsBetter),
+        "kll-percall" => one_sketch_per_group("mean_rank_err", LowerIsBetter),
+        "dd" => one_sketch_per_group("mean_relative_value_error", LowerIsBetter),


Blocking: wrong metric for dd. This uses mean_relative_value_error, but scripts/study_saturation.py:59 records DD curves as max_relative_value_error ("DDSketch's bound holds per quantile, so its curve is the worst one").

This isn't only a naming difference. RelativeValueErrorGT::score (aqpbm-core/src/accuracy/quantile.rs:137-138) scores the same 100 probe points once and records both the mean and the max error. The study keeps the max; this PR reads the mean. So the new assert_eq!(point.error_metric, ...) in SaturationCurves::accuracy fails for every DD point, and rerunning the study won't help because the script still writes max_*.

Fix: pick one quantity. Either change this to max_relative_value_error (matching DDSketch's per-quantile guarantee), or change the study to record the mean.

Decided: the optimizer keeps mean_relative_value_error, so KLL and DD are both judged on average error. Switching the study to the mean and regenerating DD's curves is tracked in #175. As of e24027e, SaturationCurves::load refuses a curve whose error_metric isn't its family's metric, so the current DD curves fail at load with "rerun the study" instead of panicking mid-plan.

milindsrivastava1997 · 2026-10-07T16:22:24Z

    "exact-max",                       // MultipleMinMax, sub_type "max"
    "exact-increase",                  // MultipleIncrease
    "kll-percall",                     // DatasketchesKLL
+    "dd",                              // DDSketch


Adding dd to DEPLOYABLE_FAMILIES puts the metric mismatch above on the default path. Every default MILP/AutoSketch run with a Quantile RAQE (e.g. the p99 RAQEs in small_problem) now evaluates DD and panics. Before, DD was only tried with --allow-undeployable-families.

Agreed that this is on the default path now. Until #175 regenerates DD's curves, small_problem stops at load with a clear message (e24027e) rather than panicking during candidate generation. Deploying DD needs #175 first.

milindsrivastava1997 · 2026-10-07T16:22:24Z

+    /// (sketch-bench#172).
    pub accuracy_sla: f64,
-    pub accuracy_direction: AccuracyDirection,
    /// Hard ceiling on modeled query latency; `None` for no limit. Only the


A single accuracy_sla is now compared against KLL's rank error and DD's relative value error, with DD enabled by default. A p99 RAQE with SLA 0.05 meant as 5% rank error would also accept DD at 5% value error, and the optimizer picks whichever is cheaper. So the plan's guarantee depends on which family wins. This is deferred to #172, but this PR turns it on by default.

Intended: one scalar SLA was an explicit scope decision for #167. The caveat that it means different guarantees for rank error and value error is #172.

milindsrivastava1997 · 2026-10-07T16:22:24Z

+    }
+
+    /// DD omits its metric when every true quantile is 0; a row without it
+    /// must not pass, nor be skipped silently.


table_accuracy_at now panics when a row or merge bracket lacks its family's metric. Before, that made one candidate infeasible; now it aborts the whole run. The comparator omits mean_relative_value_error on purpose when every exact quantile is zero, so a valid table can crash the optimizer.

Panicking here is an explicit decision: a missing metric should be loud, not a silent drop. In production table_accuracy_at only reads exact accumulators (sketches read saturation curves), and those always emit relative_error. So DD's all-zero omission can't reach this panic on the default path.

milindsrivastava1997 · 2026-10-07T16:22:24Z


    pub const METRIC: &str = "m";

+    /// Clears an SLA of 0.5 on every family's metric: errors 0, precision 1,


Test coverage: perfect_accuracy() puts every family's metric on every fixture row, so the shared tests (candidates, enumerate, milp, test_support::deployment) pass even if a family is mapped to the wrong metric. Only one_quantile_raqe_checks_each_family_by_its_own_metric would catch that, and it covers only kll and dd. The top-k and exact mappings have no negative test.

The load-time check (e24027e) now ties every studied family's metric to its curves, so a wrong mapping fails SaturationCurves::load; load_refuses_a_curve_for_another_metric covers it. That check would have caught the DD mismatch. Exact families are table-only; the comparator emits relative_error for them.

milindsrivastava1997 · 2026-10-07T16:22:24Z

-            }
-            let error = point.error_at(covered, raqe.accuracy_direction)?;
-            worst = Some(worst.map_or(error, |w| worse(w, error, raqe.accuracy_direction)));
+            assert_eq!(


The wrong-metric assert only runs on bracketing points the loop reaches. point.error_at(covered, direction)? returns early on the first unmeasured point, so a stale curve on the second point returns None (infeasible) instead of panicking. Whether a stale study is caught then depends on the RAQE's lookback.

Fixed in e24027e: the check moved to SaturationCurves::load, which sees every point, so it no longer depends on the lookback. The per-call assert is gone, and check_cost_table now compares against the family's metric by construction.

milindsrivastava1997 · 2026-10-07T16:22:24Z

@@ -357,7 +355,11 @@ mod tests {
            merge_cpu_secs: 1e-6,
            query_cpu_secs: 1e-6,
            // Error falls with width and depth, so feasibility is monotone.


Nit: stale comment. "Error falls with width and depth, so feasibility is monotone." sits right above the new "Precision rises with width and depth...". The fixture now stores precision_at_k, so the first comment should go.

Fixed in e24027e.

Address review on #173: SaturationCurves::load rejects a candidate family's curve whose error_metric isn't its accuracy_key, so a stale study (e.g. DD's max error before #175) fails before planning, whatever the lookback, and check_cost_table compares against the family's metric. Drops the per-call assert and a stale fixture comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

validate_facts rejected only a non-finite accuracy_sla, so a negative SLA or a precision floor above 1 made every candidate silently ineligible and the RAQE read as unservable. Every metric is >= 0 and the only higher-is-better one (precision_at_k) is a fraction, so the SLA must lie in [0, 1] when the capability has such a family, else [0, inf). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

milindsrivastava1997 linked an issue Oct 7, 2026 that may be closed by this pull request

Model per-family quantile accuracy metrics in optimizer RAQEs #167

Closed

milindsrivastava1997 force-pushed the 167-model-per-family-quantile-accuracy-metrics-in-optimizer-raqes branch from a3f4a65 to 9d923c6 Compare October 7, 2026 16:15

milindsrivastava1997 commented Oct 7, 2026

View reviewed changes

milindsrivastava1997 mentioned this pull request Oct 7, 2026

Saturation study records DDSketch max relative value error; optimizer reads mean #175

Closed

milindsrivastava1997 mentioned this pull request Oct 7, 2026

study_saturation: record DDSketch mean relative value error #176

Merged

milindsrivastava1997 merged commit 1041576 into main Oct 7, 2026
2 checks passed

milindsrivastava1997 deleted the 167-model-per-family-quantile-accuracy-metrics-in-optimizer-raqes branch October 7, 2026 16:44

This was referenced Oct 7, 2026

rqe-optimizer: one study for sketch cost and accuracy (#174) #178

Merged

eval: AutoSketch vs. ASAP on the synthetic mixed set (cost by use; frontier and SLA versions) #138

Merged

milindsrivastava1997 mentioned this pull request Oct 9, 2026

rqe-optimizer: workload_scenarios example (lookback/interval, grouping, type and SLA sweeps) #177

Merged


		pub const METRIC: &str = "m";

		/// Clears an SLA of 0.5 on every family's metric: errors 0, precision 1,

Conversation

milindsrivastava1997 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design decisions (please approve each)

Test changes

Validation

Uh oh!

milindsrivastava1997 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

milindsrivastava1997 commented Oct 7, 2026 •

edited

Loading