Skip to content

study_saturation: the cost shape as a full grid row and column - #186

Merged
zzylol merged 4 commits into
mainfrom
bracket-per-k
Oct 7, 2026
Merged

study_saturation: the cost shape as a full grid row and column#186
zzylol merged 4 commits into
mainfrom
bracket-per-k

Conversation

@zzylol

@zzylol zzylol commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Why

#178 added the cost table's shape (θ = 1.1, K = 1e4) to every config's curve grid as one isolated point, so the grid was no longer a full cross. SaturationCurves::bracketing_points brackets θ over all the grid's θ values and K over all its K values. A shape with θ = 1.1 at any other K therefore asked for a (1.1, K) point that was never measured, and got no accuracy.

Example: on the asap_sketchlib 0.3.0 rerun, small_problem's top_endpoints_1h (top-k, θ = 1.1, K = 1e3) was unservable.

What

  • Full row and column. study_saturation.py --cost-shape adds the cost shape as a full grid row and column:

    • θ = 1.1 joins --thetas and K = 1e4 joins --cardinalities, crossed;
    • a = 2 joins --alphas.

    The grid stays a full cross.

  • Lookup unchanged. bracketing_points keeps the rectangular bracket and its completeness check: a hole in the grid leaves the shape without accuracy.

    • The first version of this PR bracketed a sparse grid instead. Review pointed out that this bridges holes to farther Ks, which is only safe if accuracy is monotone in K.
    • One small fix kept: "quantile" is decided from every point, not the first.

  • Tests and docs.

Verification

  • Python tests: the grid holds θ = 1.1 at every K and K = 1e4 at every θ.
  • cargo test -p rqe-optimizer (75): a new test checks that a hole in the grid has no accuracy.
  • clippy -D warnings and fmt are clean.
  • The added row and column are being measured on the 0.3.0 curves. small_problem end to end follows.

🤖 Generated with Claude Code

zzylol and others added 3 commits October 7, 2026 21:42
@zzylol @claude
rqe-optimizer: bracket the shape parameter within each K
ff5b221 
The cost table's shape adds one (θ = 1.1, K = 1e4) point to the curve
grid (#178). bracketing_points bracketed θ over every point's θ and K over
every point's K, so a θ = 1.1 shape at another K asked for a (1.1, K) point
that was never measured and got no accuracy. small_problem's
top_endpoints_1h (θ = 1.1, K = 1e3) was unservable. Now θ is bracketed
within each K, and K among the K values that can bracket θ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol @claude
rqe-optimizer: one pass per K for bracketing; equal shapes all count
03109d6 
From review on #186:
- Gather each K's bracketing points in one pass; points with equal shapes
  (10000 vs 1e4 in two CSVs) all count instead of failing a length check.
- Quantile when every point is Pareto, not by the first point.
- Doc: the K bracket falls back to neighbouring Ks that have the
  parameter (conservative) rather than returning None.
- The test's extra point carries no merge curves of the point it copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol @claude
study_saturation: the cost shape as a full grid row and column; keep …
da9bf07 
…the rectangular bracket

From the second review of #186: rather than bracketing a sparse grid, keep
the grid a full cross. --cost-shape now adds θ = 1.1 and K = 1e4 to --thetas
and --cardinalities (and a = 2 to --alphas), so θ = 1.1 is measured at every
K and K = 1e4 at every θ. bracketing_points goes back to the rectangular
bracket with its completeness check: a hole in the grid leaves the shape
without accuracy instead of bridging to farther points (accuracy needn't be
monotone in K). Quantile is decided from every point, not the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol zzylol changed the title rqe-optimizer: bracket the shape parameter within each K study_saturation: the cost shape as a full grid row and column Oct 7, 2026
@zzylol @claude
rqe-optimizer: refuse a curve grid that isn't a full cross, naming th…
8142fa4 
…e holes

From the third review of #186:
- SaturationCurves::load refuses curves whose Zipf grid isn't a full cross
  of its θ and K values, naming the missing (θ, K). A hole used to leave
  every shape around it silently without accuracy.
- Quantile is decided by the first point again (the grids hold one
  distribution per config).
- The hole test removes the point by shape, not position; a study test
  checks a grid that already holds the cost shape adds nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol
zzylol merged commit 4475084 into main Oct 7, 2026
2 checks passed
zzylol added a commit that referenced this pull request Oct 7, 2026
@zzylol @claude
rqe-optimizer: the full-cross check is per top-k k
aebb053 
Rebased onto #186. Its load check groups grid points by their full
params, topk_k included, so each k's grid must be a full cross of θ and
K on its own and the k dimension doesn't trip it. The study crosses each
per-k top-k config with the cost-shape row and column like any config,
and the curve-at-k lookup keeps the rectangular bracket within the
chosen k. A test pins both: two k's on different grids load; a hole
within one k is named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol added a commit that referenced this pull request Oct 7, 2026
@zzylol @claude
top-k: k is a per-query variable (#185)
30e725a 
* top-k: k is a per-query variable, not the constant 32 (#184)

- sketch-bench: heap top-k variants take topk_k= (default 32); the heap
  defaults to k and a heap below k is refused; the answer is the heaviest
  k and the comparator grades precision@k at it (answered_k).
- study_saturation: top-k curves run at each k in --topk-ks (10, 32, 100);
  k != 32 is keyed in the config string (topk_k=k), so k = 32 keeps its
  old keys and older files read unchanged.
- rqe-optimizer: Raqe.topk_k (default TOPK_K); heap needed m·k and
  eligibility heap >= (L/x)·k; accuracy reads the curve at k, else the
  next larger measured k, and past every measured k the theory bound at
  k; output cost is k entries; AutoSketch searches the smallest measured
  heap >= k.
- Docs: rqe_sketch_deployment_v1.md, saturation_study.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* top-k k: floor heaps, carry topk_k, share heap pricing with AutoSketch (#185 review)

From the first review of #185:
- A top-k need below the smallest measured heap prices at that heap,
  never extrapolated below; candidates are keyed by (heap, k) and carry
  topk_k when k != 32, so the deployed sketch answers k. A RAQE needs a
  deployment answering at least its k.
- univmon-topk answers TOPK_K only: ineligible for any other k.
- AutoSketch resolves heaps with the MILP's heap_row (same floor and
  interpolation/extension), index for index, instead of picking a
  measured heap >= k.
- Accuracy picks the next larger measured k whose curves bracket the
  shape; past every such k, the guarantee at k, floored by the largest
  bracketing k's measurement at this N. Params compare without cloning.
- One topk_k decoding (params_topk_k / config_topk_k): heap_capacity's
  missing heap is the row's k, as in sketch-bench.
- --topk-ks is deduplicated and must include 32.
- Top-k query CPU scales by k/32 (measured at 32), a first approximation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: the full-cross check is per top-k k

Rebased onto #186. Its load check groups grid points by their full
params, topk_k included, so each k's grid must be a full cross of θ and
K on its own and the k dimension doesn't trip it. The study crosses each
per-k top-k config with the cost-shape row and column like any config,
and the curve-at-k lookup keeps the rectangular bracket within the
chosen k. A test pins both: two k's on different grids load; a hole
within one k is named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* Address second #185 review: answered-k query cost, fixed-k families

- Query CPU and output bytes scale by the deployment's answered k.
- answered_k/serves_topk_k in lib.rs: a fixed-k family (univmon-topk)
  answers TOPK_K only; used by candidates::is_eligible and AutoSketch.
- heap_row's floor sets an explicit heap; one topk_k encoder
  (topk_k_param) shared with with_topk_k.
- priced_parts tags sketch vs tracker instead of pointer identity.
- AutoSketch resolves heaps once per (k, top-k) in plan, indexes heap
  rows once, and keeps heap/topk_k off its search axes.
- study_saturation: old k=32 curve files need --topk-ks 32 for cost and
  crossover; say so in the docstring and exit message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 8, 2026
@zzylol @claude
eval: p95 results for the synthetic grid and the traces
ff091b4 
Runs on clnode155 over the asap_sketchlib 0.3.0 study (#174, #178–#186):
four synthetic workloads and alibaba_v2022 / google_2011, raw JSON,
summaries and figures. The README gives the no-SLA headline, including
Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is
refilled from the study's raw records (0.77 h; the K = 1e4 column refill
of #186 is its own run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol added a commit that referenced this pull request Oct 8, 2026
@zzylol @claude
eval: p95 results for the synthetic grid and the traces
eeec504 
Runs on clnode155 over the asap_sketchlib 0.3.0 study (#174, #178–#186):
four synthetic workloads and alibaba_v2022 / google_2011, raw JSON,
summaries and figures. The README gives the no-SLA headline, including
Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is
refilled from the study's raw records (0.77 h; the K = 1e4 column refill
of #186 is its own run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol added a commit that referenced this pull request Oct 8, 2026
@zzylol @claude
eval: p95 results for the synthetic grid and the traces
6e77975 
Runs on clnode155 over the asap_sketchlib 0.3.0 study (#174, #178–#186):
four synthetic workloads and alibaba_v2022 / google_2011, raw JSON,
summaries and figures. The README gives the no-SLA headline, including
Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is
refilled from the study's raw records (0.77 h; the K = 1e4 column refill
of #186 is its own run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol added a commit that referenced this pull request Oct 9, 2026
@zzylol @claude
eval: p95 results for the synthetic grid and the traces
1da0755 
Runs on clnode155 over the asap_sketchlib 0.3.0 study (#174, #178–#186):
four synthetic workloads and alibaba_v2022 / google_2011, raw JSON,
summaries and figures. The README gives the no-SLA headline, including
Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is
refilled from the study's raw records (0.77 h; the K = 1e4 column refill
of #186 is its own run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol added a commit that referenced this pull request Oct 9, 2026
@zzylol @claude
eval: AutoSketch vs. ASAP on the synthetic mixed set (cost by use; fr…
4214545 
…ontier and SLA versions) (#138)

* rqe-optimizer: evaluate AutoSketch vs ASAP on the synthetic and trace workloads

ProjectASAP/ASAPQuery#777 (paper §6.3), consolidating #138, #139 and #141.

- rqe-optimizer/examples/autosketch_vs_asap.rs (traces and synthetic): ASAP
  (milp::minimize), PerQuery-CostAware and AutoSketch-Adapted. Costs and
  accuracy come from --saturation-dir, as the planner reads them:
  SaturationCurves::accuracy for ASAP (merge curves for KLL and top-k,
  #179), autosketch_accuracy for AutoSketch, and the one-study cost table
  (#178). Each run is scored by analytical_cost_model::score at #777's two
  weight settings and the SLA grid.
- scripts/run_autosketch_vs_asap.sh and plot_autosketch_vs_asap.py: the
  trace workloads alibaba_v2022 and google_2011. boom is left out for now.
- scripts/export_autosketch_eval_table.py --synthetic: one workload-only
  table per grid point, plus plan.tsv. The data constants come from
  study_saturation's COST_*. The traces path is unchanged from main.
- scripts/run_autosketch_vs_asap_synthetic.sh and
  plot_autosketch_vs_asap_synthetic.py: run the synthetic grid, then plot.
- scripts/profiling_time.py and the results README: ASAP's one-time
  profiling time, and how to reproduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: trace key queries keep one accumulator per key; report the accuracy source

From review on #138:
- A traces key query (sum by key) keeps one exact accumulator per key: its
  groups are the window's keys, not the table's one sketch. A stream's
  card(g) is the largest over its RQEs.
- AutoSketch's benchmark lower bound uses one query of one instance, not
  the stream's q_r of them.
- --weights that names none of the weight settings is an error.
- The chosen report lists each RQE's accuracy source (#180's
  accuracy_with_source).
- Drop a duplicated BoomTest; capability_of's comment matches the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: one accuracy level, p95; plots join one RQE set and charge AutoSketch's benchmark

- One accuracy level for every workload, synthetic and traces: 95% in
  each family's metric (rank, relative value and relative error at most
  0.05; top-k precision at least 0.95). The strictness dimension is gone.
- fig_objective_vs_latency joins only points over one RQE set (solid for
  the full set, dashed for the subset tight SLAs keep).
- fig_planning_time adds AutoSketch's charged benchmark time (lower bound
  and 60 s per probe), as #777 section 7 counts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: summaries show Σ window/slide; synthetic summary to its own file

AutoSketch-Adapted's ingest cost is window / slide overlapping sketches per
deployment (window = the query's lookback), so the column explains its gap
to ASAP. The synthetic and trace scripts both wrote summary.md into one
results directory; the synthetic one is now summary_synthetic.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: RQEs carry their top-k k (#185)

Raqe now has topk_k. The tables name it only in the query id (topk32_...),
so the runner reads it there and stops on a top-k query that names none.
On the p95 synthetic and trace tables every top-k query is k = 32, and the
plans are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: p95 results for the synthetic grid and the traces

Runs on clnode155 over the asap_sketchlib 0.3.0 study (#174, #178–#186):
four synthetic workloads and alibaba_v2022 / google_2011, raw JSON,
summaries and figures. The README gives the no-SLA headline, including
Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is
refilled from the study's raw records (0.77 h; the K = 1e4 column refill
of #186 is its own run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: figures and tables give each method's absolute cost

Cost figures label every bar and point with its value; the summaries and
the results README list each method's objective instead of its ratio to
ASAP. The by-dimension figure's legend moves above the panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: metrics dimension, one dashboard per metric (21 · m RQEs)

Shared replicas on one metric stop adding RQEs past r = 8 (52 at r = 8,
57 at r = 64), so the grid keeps shared r ∈ {1, 8} and adds metrics
m ∈ {1, 8, 16}: copy i of the template set reads its own metric data_i,
so nothing is shared across copies and RQEs grow as 21 · m (168, 336).
The planning-time figure plots this dimension. The m = 8 and 16 runs are
added; the r = 64 result is removed. Existing tables are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: AutoSketch benchmarks each config once per metric

Synthetic tables name the metric each RQE reads (data, or data_i in the
metrics dimension), and AutoSketch-Adapted's benchmark time counts a
probed config once per metric, on that metric's data: 540, 4320 and
8640 s (60 s per probe) for m = 1, 8, 16. Trace tables name none and
read one metric per dataset, as before. All five synthetic workloads are
rerun with the current runner on four nodes; objectives are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: cost vs. total estimated latency for every workload

One column per workload (five synthetic, two traces), one row per weight
setting, one line per method across the SLA grid, each point labeled with
its cost. Total latency is the sum of the RQEs' estimated latencies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: finer SLA grid, {0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10} ms and none

Checks whether AutoSketch-Adapted's memory objective makes it miss an
SLA: it never does. Its plans never merge, so each RQE's latency is within
1.4% of the lowest any deployment reaches (equal for quantile, exact and
trace RQEs), and RQEs no method can meet are excluded for all methods.
All workloads rerun on four nodes; figures and summaries regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: AutoSketch probe list and measured benchmark time; traces with 6h/24h windows

- The runner lists AutoSketch-Adapted's distinct probed (metric, config)
  pairs with the data shape each is benchmarked on, and drops the
  benchmark-time lower bound. scripts/autosketch_benchmark_time.py runs
  each probe's approxbench accuracy benchmark (1e8 items, exact baseline
  and scoring) and times it, as AutoSketch benchmarks probed configs.
- Trace streams with more keys than samples per second are left out and
  listed (rounding their rate up to their key count inflated every
  method's cost).
- The trace table is regenerated from ASAPQuery #812's skew summary
  (6h and 24h windows, a day of Alibaba).
- Plots: one line per method over the full RQE set; AutoSketch's planning
  time with its measured benchmark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: cost by use (sketch-bench #188); Pareto frontier sweep; mixed set by default

- The runner prices every plan by use (usage::usage_cost): w_cpu * AUC(CPU)
  + w_mem * AUC(memory), CPU elastic, latency the longest chain, reported.
- ASAP is milp::minimize_usage_cost over every RQE; PerQuery the same over
  each RQE's own candidates (separate copies via `allowed`, no sharing);
  AutoSketch's plan is priced the same way.
- No SLA grid. Each method's cost-latency frontier: ASAP and PerQuery are
  solved unbounded and for 12 bounds log-spaced from the tightest feasible
  one to the unbounded latency, plus AutoSketch's latency.
- The synthetic grid is the mixed template set: shared r in {1, 8} and
  metrics m in {1, 8, 16} (50, 92, 400, 800 RQEs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: version 1 (no SLA) results on the mixed set, with AutoSketch's measured benchmark

- Four workloads of the mixed template set (50, 92 with r = 8, 400 with
  m = 8, 800 with m = 16 RQEs), cost by use (#188), each method's cheapest
  plan and ASAP's and PerQuery's cost-latency frontiers (12 bounds plus
  AutoSketch's latency), CPU-only and Fargate weights, --runs 3, one node
  per workload. No RQE dropped, no sanity violation.
- AutoSketch's benchmark time is measured: approxbench's accuracy run (1e8
  items, exact baseline, scoring) once per distinct probed config and
  shape, serially on idle clnode155 (21-37 s each), charged once per
  probed (metric, config): 212, 1697 and 3394 s for m = 1, 8, 16.
- plot_autosketch_vs_asap_synthetic.py: fig_frontier.png (cost vs. query
  latency, every point labeled), fig_planning_time.png (AutoSketch as
  search + measured benchmark; 60 s per probe only as a labeled
  reference), summary_synthetic.md. The SLA-era figures, the dashboard
  results and plot_autosketch_vs_asap_workloads.py are removed.
- README: the cost model, version 1, how to reproduce, and the headline
  numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: version 2, a batch latency SLA, on the synthetic mixed set

The runner adds version 2 of the evaluation beside version 1 (no latency
constraint and the frontiers, unchanged in `results`): at each SLA of a
grid (default 100, 300, 1000, 3000, 10000 ms; --slas-ms overrides), ASAP
and PerQuery plan the cheapest plan whose batch latency (the longest
chain) is at most the SLA, via minimize_usage_cost's latency bound, which
is exact with elastic CPU. An SLA below a method's tightest feasible
bound gives an infeasible record. AutoSketch's one plan is recorded at
each SLA with meets_sla. Records go to `sla_results`, with sanity checks
that every plan meets its SLA and that ASAP costs no more than PerQuery.

Results on clnode109 (--runs 3) for the mixed set, r = 8 and m = 8, 16:
no drops, no sanity violations; the tightest feasible SLA is 92 ms, and
AutoSketch (latency 1011 ms) meets only 3000 and 10000 ms. AutoSketch's
planning time is search plus its measured benchmark
(scripts/autosketch_benchmark_time.py: approxbench accuracy runs at 1e8
items, 20-33 s each, serial and alone on the node): 197 s at m = 1,
3159 s at m = 16. scripts/plot_autosketch_vs_asap_sla.py draws
fig_cost_vs_sla.png and summary_sla.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: remove the old trace results and figures

They came from the earlier model (per-RQE SLA over a µs grid, mean-CPU
cost); the traces are to be rerun on cost by use with both latency
versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: trim the PR to what it needs

- Result JSONs without per-RQE choices (`chosen`): the figures and
  summaries don't read them, and they were nearly all of the 2.9M lines;
  rerun the runner for them.
- Machine files folded into the READMEs (all Intel Xeon E5-2683 v3,
  56 cores); the synthetic run script no longer writes them.
- The SLA-era trace scripts (plot_autosketch_vs_asap.py,
  run_autosketch_vs_asap.sh) are removed; the traces are to be rerun on
  cost by use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: keep only what the three figures need

The PR's results are the frontier (version 1), cost vs. SLA (version 2)
and planning time on the synthetic mixed set. The trace table goes back
to the base version (the traces are to be rerun on cost by use, with
their new table), and the ASAP profiling-time script and data, which no
figure uses, are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: commit the inputs of the three figures

rqe-optimizer/results/autosketch-vs-asap-inputs/ holds what the figures
were computed from: the sketch-bench study on asap_sketchlib 0.3.0 (the
cost table, sketches and exact aggregations, and the accuracy curves the
runner reads as --saturation-dir) and the synthetic mixed-set tables with
plan.tsv. The READMEs point the reproduce steps at them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: the raw approxbench records behind the figures' study

rqe-optimizer/results/autosketch-vs-asap-inputs/raw/ holds the gzipped
approxbench records (920 KB) that ../saturation/ was reduced from, with a
README mapping each file to its machine, arguments, record count, run time
and output. Rerunning them is 0.77 h of machine time over 5 machines in
parallel (about 20-30 minutes of wall time).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol