Skip to content

eval: accuracy target sweep (90%, 95%, 99%) for AutoSketch vs. ASAP - #206

Merged
zzylol merged 1 commit into
mainfrom
eval/accuracy-target-sweep
Oct 14, 2026
Merged

zzylol merged 1 commit into
mainfrom
eval/accuracy-target-sweep

Conversation

@zzylol

@zzylol zzylol commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Reruns the 12 synthetic §6.3 workloads at 90% and 99% accuracy. The 95% runs are the committed ones in autosketch-vs-asap-hydra.

Changes

  • autosketch_vs_asap: --target takes acc90, acc95 (alias p95) or acc99. Each means NN% accuracy in the family's own metric: error ≤ 0.10 / 0.05 / 0.01 for KLL rank, DDSketch relative value and HLL relative error, and top-k precision ≥ 0.90 / 0.95 / 0.99. The 95% behavior is unchanged.
  • scripts/plot_autosketch_vs_asap_targets.py: cost vs. target figures and a summary.
  • rqe-optimizer/results/autosketch-vs-asap-targets/: 24 result files, figures, and a README with the analysis.

Results (cheapest plan, CPU, vCPU). All runs have 0 RQEs dropped and 0 sanity violations.

Workload Target ASAP ASAP (no roll-ups) PerQuery AutoSketch
mixed 90% / 95% 3.54 3.54 10.5 3,716
mixed 99% 3.85 3.85 10.9 3,757
multi-grouping 90% / 95% 0.221 0.551 1.07 10.2
multi-grouping 99% 0.627 0.979 1.49 9.94
mixed + multi-grouping 90% / 95% 3.76 4.09 11.6 3,726
mixed + multi-grouping 99% 4.47 4.82 12.3 3,767
  • 90% = 95% everywhere. The 95% plans already use the cheapest measured config of each sketch, e.g. HLL lg_k=12, the smallest on the grid. A looser sweep needs smaller configs measured first.
  • 99%. ASAP's cost rises 9% on mixed and 2.8× on multi-grouping. ASAP stays 2.4–2.8× below PerQuery and 16–1,000× below AutoSketch.
  • Roll-ups. At 99% they save 36% on multi-grouping, against 60% at 95%.

Tests: the runner's example tests pass (10), including the new levels.

🤖 Generated with Claude Code

The runner's --target takes acc90, acc95 (p95, as in the committed runs) or
acc99: NN% accuracy in each family's own metric. The 12 synthetic workloads
rerun at 90% and 99%; scripts/plot_autosketch_vs_asap_targets.py plots cost
vs. target.

90% costs the same as 95%: the 95% plans already use the cheapest measured
configs. 99% raises ASAP's cost 9% on mixed and 2.8x on multi-grouping;
ASAP stays the cheapest at every target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol
zzylol merged commit 98ec50b into main Oct 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant