[zzylol]

Stacked on #129 (dev-milind-turboprom); retarget to main after #129 merges. Does not change rqe-optimizer or AtomicCostEntry. Follow-up: #131 (accuracy after merging).

Conclusions (task 3: sketch accuracy, saturation and cost)

Full write-up with the math behind the conclusions: docs/saturation_conclusions.md

1. No sketch's theoretical error bound depends on N. Error that still moves with N comes from two small-N effects: a noisy ground truth when N is small, and a tail of keys still arriving while N < K. Past N_sat, error is set by the config alone, so a profile measured on a small window extrapolates to larger (merged) windows.

2. N_sat is mostly 1e3–1e6, with two exceptions:

   • With uniform keys and large K, CMS and CountSketch never saturate: at K = 1e7 the error is still rising at 1e9.
   • With large K, top-k precision keeps improving with N up to 1e9.

   So an N_sat measured only up to 1e7 is a lower bound when K is large.

3. Error scales with config size as theory predicts, so configs can be interpolated:

   Sketch	Error scales as
   CMS	1/w (steeper with skew)
   CountSketch	1/√w (≈ w^(−θ) with skew)
   HLL	1/√m
   KLL	1/k
   DDSketch	α_dd

4. The data distribution decides which sketch to use:

   • Low θ is the worst case for CMS and top-k, so size them from the lower θ bound.
   • At low skew and large K, CountSketch beats CMS at equal memory. For CallGraph, CountSketch needs 48 KB where CMS needs 192 KB.
   • Top-k is unreliable for θ ≤ 0.5.
   • KLL's error does not depend on the distribution.
   • DDSketch's rank error grows with α while its memory shrinks with α, so size rank error from the upper α bound and memory from the lower α bound.

5. Merging (Measure accuracy after merging m shards #131):

   • CMS, CountSketch, HLL and DDSketch merge exactly: merged error equals the single-sketch error.
   • KLL's error rises more for larger k: about 1.0–1.1× at k = 50/200 and 1.12–1.32× at k = 800 at N = 1e7, and up to 3–4× at N = 1e4–1e5.
   • Top-k is unchanged in the median but loses up to 40% precision at large K.

   So single-sketch curves are optimistic for top-k and KLL in merged windows.

6. Cost:

   • Per-item insert CPU is flat in N and in the distribution: CMS ≈ 22 ns, CountSketch ≈ 26 ns, HLL ≈ 3.4 ns, KLL ≈ 25 ns, DDSketch ≈ 16 ns. Top-k is the exception, rising from about 100 ns to 1,750 ns per item as θ goes from 0 to 2.
   • Memory is fixed by the config, except DDSketch, which needs more memory for heavier tails.

7. Sketch vs exact computation (exact = buffer the samples, compute at query time):

   • Memory: sketches save 10× after a few thousand items and 100× after 1e4–1e6 items, depending on sketch size.
   • CPU:
     • CMS and CountSketch are about 13× cheaper, HLL about 40×; neither reaches 100×.
     • Quantile sketches exceed 100× from 1e5–1e6 items, because exact computation has to sort.
     • At high skew, top-k is slower per item than exact; its saving is memory only.
   • Real windows hold millions of items, so sketches save at least one to two orders of magnitude of memory.

8. Worst-case parameters → configuration is automated (scripts/recommend_config.py). Examples:

   Query	Recommendation
   Google by (user)	CMS 3×1024 (12 KB)
   CallGraph by (service)	CountSketch 3×4096 (48 KB)
   MSRTMCR instant	192 KB
   p99 latency	DDSketch α = 0.01 (3–5 KB)

Limits:

• Some recommendations extrapolate beyond N = 1e7.
• Merged accuracy covers every top-k and KLL config, but only up to N = 1e7 and 64 shards.
• The exact baseline buffers the stream; with a streaming hash-map counter, the frequency sketches' memory advantage would be smaller.

Summary

Measures, for each sketch type, configuration and data distribution:

• Empirical error vs N, and N_saturation, the stream length after which error stops changing.
• Insert / merge / query CPU and memory, measured serially on an idle machine.
• Crossover N* against exact computation: the stream length from which exact computation stays at least 10× or 100× more expensive than the sketch.
• A recommended configuration per query, from the worst-case data parameters fitted in feat(asap-tools): label query sets and skew fits for cluster traces ASAPQuery#746.

Full method and results: docs/saturation_study.md. Figures, each folder with a README: docs/figures/saturation/.

What's added

Piece	What it does
--dataset pareto (--pareto-alpha, --pareto-scale, f64 or floored i64)	Heavy-tailed value streams for quantile sketches; i64 so the polars exact quantile baseline applies
scripts/study_saturation.py	Error-vs-N sweep over sketches × configs × distributions. Phases: --phase accuracy|cost|crossover|all. Supports --resume, --cost-rows 3, --points-from
scripts/recommend_config.py	Joins ASAPQuery's worst-case parameters (θ, K, N, α per query and range) with the grid curves. Picks the smallest-memory config per sketch family that meets the query's accuracy target, and marks extrapolated, not-yet-saturated and no-merge-curve results
scripts/plot_saturation_{curves,cost,crossover}.py	Error vs N with each sketch's theoretical bound; cost vs θ/α; exact vs sketch memory and CPU

N_sat is the first checkpoint after which every later seed-mean error stays within max(10% of the plateau, 2 standard errors) of the mean of the last 3 points.

N*(f) is the first N from which exact cost stays ≥ f × sketch cost at every larger N. CPU is counted as "ingest N items + answer one query" on both sides.

Grid

Sketch	Configs
CMS, CountSketch, CMS-heap top-k	cols 256–16384 × rows 3/5
HLL	lg_k 12/14/16
KLL	k 50/200/800
DDSketch	α 0.005–0.05

• Data: Zipf θ ∈ {0 … 2} × K ∈ {1e3, 1e5, 1e7}, or Pareto α ∈ {1.1, 1.5, 2, 3}.
• Error runs: N up to 1e7 with 3 seeds (595 points), plus 185 points run to 1e9.
• Cost runs: serial, on an idle machine, rows=3 only (rows=5 cost ≈ 5/3 of that).

Key results

Every theoretical bound is N-independent. Error that still changes with N comes from small-N effects: a noisy ground truth, or a tail of keys still filling in while N < K.

Plateau error vs sketch size (log-log slope against each sketch's size parameter: width w for CMS/CountSketch/top-k, register count m = 2^lg_k for HLL, k for KLL, α for DDSketch):

Sketch	Theory	Measured
CMS	−1	−1.0 at low skew; steeper with skew
CountSketch	−0.5	−0.43 to −0.51 at low skew; ≈ −θ with skew
HLL	−0.5	−0.46 to −0.66
KLL	≈ −1	−1.03 to −1.18
DDSketch	+1	0.98 to 1.08

Run to 1e9, 67 of the 108 points not saturated by 1e7 saturate:

• Uniform keys with K = 1e7 never saturate CMS or CountSketch.
• Top-k at K = 1e7 keeps improving with N.
• So an N_sat from curves that stop at 1e7 is a lower bound when K is large.

Cost per item at N = 1e7, flat in N and in the distribution unless noted:

Sketch	Insert CPU
CMS	≈ 22 ns
CountSketch	≈ 26 ns
HLL	≈ 3.4 ns
KLL	≈ 25 ns
DDSketch	≈ 16 ns
CMS-heap top-k	100 ns at θ = 0, rising to 1,750 ns at θ = 2

DDSketch memory shrinks from 6.3 KB to 2.3 KB as the Pareto α grows from 1.1 to 3.

Crossover vs exact (exact = buffer the samples, compute at query time):

Sketch	Memory 10× / 100× from N =	CPU 10× / 100× from N =
CMS / CountSketch 3×1024	5.6e3 / 1e5	1e3 / not reached (≈ 13×)
CMS / CountSketch 3×16384	1.8e5 / 3.2e6	1e3 / not reached
HLL lg_k = 12	5.6e3 / 5.6e4	3.2e3 / not reached (≈ 40×)
KLL k = 200	5.6e3 / 1e5	1e3 / 1e6
DDSketch α = 0.01	5.6e3 / 5.6e4	1e3 / 1.8e5
CMS-heap top-k	as CMS	not reached: slower per item than exact at θ ≥ 1

Recommendations, with the cost data:

Query	Recommendation
Google by (user)	CMS 3×1024 (12 KB)
CallGraph by (service)	CountSketch 3×4096 (48 KB), against CMS 3×16384 (192 KB)
MSRTMCR by (msname)	instant needs 192 KB; 5m/1h ranges need 48 KB
p99 latency	DDSketch α = 0.01 (3.4–4.6 KB)

Known limits

• The exact baseline buffers the stream. A streaming hash-map counter would move the frequency memory crossover later.
• Top-k and KLL error after merging differs from a single sketch (see Measure accuracy after merging m shards #131). Their recommendations are marked when there's no merge curve for that point.
• Pareto α above 3 (e.g. Google cpu_rate) is clamped to the edge of the grid.

Validation

• cargo test -p aqpbm-datagen passes, including a Pareto α recovery test.
• clippy -D warnings is clean on the touched crates.
• python3 -m unittest scripts/test_study_saturation.py scripts/test_recommend_config.py: 22 + 11 tests pass.

🤖 Generated with Claude Code

[milindsrivastava1997]

Proposed fixes for the two medium review findings

1. Crossover matches saved cost records by order (study_saturation.py L373, L425)

Fix: label each record with its key when it's saved, and look records up by that key when rebuilding.

• Cost phase: write each line as {"key": [variant, config, dist, param, k], **record}.
• Exact phase: label each line with [variant, dataset, n, "timed"|"prepared"].
• Rebuild (--phase crossover): load the file into a dict keyed by those labels. If a point is missing, exit with a clear message, e.g. "cost phase didn't measure X; rerun with the same --cost-rows". Extra records in the file are ignored.
• Effect: next(saved) and the hidden dependence on matching --cost-rows both go away. A mismatch gives a clear exit instead of wrong numbers or a StopIteration.

Catch: saturation_cost.jsonl / exact_cost.jsonl files written before this change have no labels, so crossover can't read them. If existing runs need to be kept:

• Cheaper option: keep matching by order and add a guard that counts the non-skipped points, compares that against the number of lines in the file, and exits if they differ. That catches the --cost-rows mismatch but not records stored in a different order.
• Fallback option: use the labelled keys, and fall back to matching by order for files without labels.

2. --resume deletes curves outside the current grid (L316)

Fix: on resume, read the existing CSV's raw rows, grouped by point key, and write back:

• every row for points not in the current grid, unchanged;
• for points in the grid, rows only if the curve is complete (its sizes equal ns), which is what happens today.

That's about 5 changed lines. The point of rewriting the file (dropping a point that was cut off mid-write) still holds. A partial curve outside the grid stays in the file until some resume that includes that point drops it and reruns it.

Alternative: make --resume refuse to run when the grid is narrower than what's already in the CSV.

Open questions

1. Is there existing cost/exact JSONL output that must stay usable? If so, use the count guard or the fallback option; if not, use the labelled keys.
2. For fix 2, should resume keep the curves outside the grid (the fix above) or refuse to run (the alternative)?

[milindsrivastava1997]

Two low-severity findings from code review.

[milindsrivastava1997]

This Pareto arm never runs. The cardinality lookup just before the match (column.cardinality.or_else(|| column.distribution.domain()...)?) returns None for Pareto, so the function exits first: domain() is None for Pareto, --dataset pareto sets cardinality: None, and ColumnSpec validation rejects any cardinality on an unbounded distribution. As a result, every Pareto record (including the KLL/DDSketch cost runs) is exported with no erp_shape, and nothing warns about it. Fix: give unbounded families a path that doesn't need a cardinality, or remove this arm.

[milindsrivastava1997]

One failed run is only reported after every queued run finishes. All accuracy futures are submitted at the start. If one approxbench run fails, f.result() raises CalledProcessError, but leaving the with ThreadPoolExecutor block calls shutdown(wait=True). That waits for every remaining queued run and then throws their results away, since none are written to the curve CSV. On the default grid this can waste hours before the error appears. Fix: cancel the pending futures in an except block, or call pool.shutdown(cancel_futures=True) before re-raising.

[milindsrivastava1997]

point_key compares param as a number but cardinality as a raw string. --points-from asks for "a filtered saturation.csv". If that file is filtered with pandas, the Pareto rows' empty cardinality turns the column into floats, so the Zipf values come back as 1000.0. "1000.0" never equals str(1000), so:

• --points-from exits with "names N points but only 0 are in the grid".
• recommend_config.py:256 raises ValueError on int(k[4]) when given a curve CSV saved the same way.

Fix: normalise cardinality with int(float(c)) when it isn't empty.