Repository navigation
eval: AutoSketch vs. ASAP with Hydra candidates and a roll-up ablation (§6.3) - #203
Merged
Merged
Conversation
The polars exact baselines keyed a label subset by its `;`-joined values,
asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its
column (`label0:a;label1:b`), so on label columns that share a value
domain the baseline merged a key1 group with the same key2 group and
scored nonzero against itself. subset_key now builds the 0.3.0 key over
hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty
labels keep their column, and every lookup goes through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
The generator gets a metric-level ordered label schema (labels with cardinality, child_of fan-out and skew; per-value data shape) and per-RQE `grouping` and `covers_share`. The 10 templates are now the "classic" set, byte-for-byte the committed tables; "all" adds templates 11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry several groupings so roll-ups (and later Hydra) can share. The plan runs both sets over the grid. The runner builds each schema stream's MetricFacts from the schema (labels = schema + x; cardinality per used grouping and the full set), takes Raqe.grouping_labels from the RQE's grouping, maps the `cardinality` capability to HLL, and reports covers_share and roll-up deployment groupings in each choice. Classic output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…mment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
`--merge-split contiguous|interleaved` (default contiguous, unchanged) picks how merge runs split the stream into shards. Interleaved reorders the stream so the existing contiguous `partition` deals it round-robin by record, at the contiguous split's shard sizes, so every row's folds take it unchanged. Tests: hydra-cms, hydra-cs and hydra-hll grids folded from 4 and 7 shards, under both splits, answer every probed subpopulation exactly as the single stream; and every scored group on a fixed 20k-record stream lands inside its hydra doc §2.2 bound (CMS also never under). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
A `child_of: <index>` column with `fan_out: f` nests under an earlier string column: its value is the parent row's value, a `.`, and a child index in [0, f) drawn from the column's own distribution. Bad indices, a fan_out that disagrees with the domain, and fields that would not mean what they say on a child are refused before drawing. Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint 25, status 4, user_id value) for the hydra rows, docs, and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Every hydra-* row (lib and polars, all five variants and hydra-univmon's four statistics) takes `--group-columns J,K,...` (default `0`, today's behavior) in place of the hardcoded SCORED_LABEL_COLUMN. A subpopulation probe is now positioned (`Vec<Option<String>>`, Some at the grouped columns), so `hydra_shared::query` asks any non-empty subset of the key columns and the polars baselines look up `subset_key` with that subset's mask (`group_key`, generalizing `prefix_key`). Each grouped comparator reports its per-group error in its own metric (relative error; mean rank error for hydra-kll) and summarises it as err_mean, err_p50, err_p90, err_max, groups_scored, plus schema_width, records and fanned_mass. `--per-group-out PATH` writes the rows as `group_key,n_q,error`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…cuts
A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew
child indices 0, 1 and 2; child_of now requires whole uniform bounds.
hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first
2 and 3 labels for the Hydra study's schema-width sweep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Review of #196. http's labels are now Zipf-skewed (region 0.5, service 1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS burst targets a tail subnet, so covers_share selects a few groups. The generator computes each group's share (product of its labels' shares) and, per RQE, the covered groups (share >= tau, plus always the largest) and the smallest covered group's share and items per window. The runner reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's items instead of the mean group's; covers_share = null is unchanged. The schema test reads an inline HLL curve fixture instead of the gitignored study curves. The synthetic plot keys runs by template set too, and both plot scripts title classic "mixed" and all "mixed + multi-grouping". README: the coverage rule, the N the runner uses, and which schema fields are descriptive. label_set.keys_per_window is commented as the universe K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
… {service, endpoint}
- covers_share = null covers every group, so the runner reads its accuracy
at the smallest group (min_covered_share), like a tau RQE's smallest
covered group. Classic RQEs (no covers_share) are unchanged.
- Template 17 (negative control) groups by {service, endpoint}: its
smallest group holds 4.5e4 items in 5m, on the curves.
- The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the
vacuous `assert covered`); a test checks the old full-label-set template
fails it.
- Covered facts go through validate_facts.
- README: null semantics, template 17, and the numbers' provenance (fresh
`all` table, committed saturation inputs).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…curacy - Group keys escape `\`, `:` and `;` as polars_shared::subset_key does. - Groups that cannot be scored (zero true statistic, e.g. zero entropy) are written with an empty `error`, so n_q sums to the stream's records. - --per-group-out with no accuracy measurement is refused instead of writing an empty file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
- hydra-kll-cdf (lib, polars): asks the KLL grid HydraQuery::Cdf(x) at each group's own values at phi = 0.01..0.99, scored by absolute CDF error (SubpopCdfErrorGT, comparator subpop-cdf-error). The polars baseline is the exact share at or below x over the same sorted groups. - hydra-univmon-sum (lib, polars): each record inserted with count = value, so the UnivMon grid's L1 is the group's sum, scored by relative error against exact per-group sums (SubpopSumGT, comparator subpop-sum). Integer value dtypes only; f64/str and values outside [0, i32::MAX] are refused by name. - memory_hydra_kll counts each cell as kll_lib_bytes does, merge buffer of cell_k items included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Conflicts: both sides add a Requirement field and its CLI flag; keep both. The merge tests' probes take #199's `Option` columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
The grid clones its cells from one template, and a cloned Vec keeps no capacity, so a cell's merge buffer is empty at build and grows to about cell_k items at its first compaction. Say the footprint counts it at that bound. The CDF test now asks a group of ties (at or below) and a mixed-tie group on a grid wide enough that it shares no cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
- `merge_split` (contiguous|interleaved) is a record field next to `merge_shards` (BenchSection and MergeMetrics; schema regenerated). Merge records stamp it; a record without it split contiguously, so existing results parse, and flatten refuses two different splits in one row as it does two shard counts. - `MergeSplit` moves to aqpbm-core (it is a record type) and is parsed by clap directly instead of matching strings. - The interleaved copy of the stream is built only when a merge runs. - README states the round-robin exactly (contiguous shard sizes, so a short last shard shifts the pattern); MergeMetrics docs name the split. - hydra-cs bound test: note its margin (passes from beta ~4.8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…dra/study Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis # Conflicts: # aqpbm-cli/src/main.rs
configs/datagen/hydra_http.yaml, hydra_http_latency.yaml and hydra_flows.yaml
follow the §6.3 eval's http and flows schemas (labels in schema order,
cardinalities, child_of, skews; user_id Zipf 0.8, latency Pareto a = 2,
src_ip uniform). The flows DDoS burst is not generated: datagen draws each
column independently.
--phase hydra runs HYDRA_SWEEPS: the eval's variants on those datasets (every
grouping, W 1024/4096/16384, N 1e5..1e7, 1/4/16 interleaved shards, 3 seeds)
and every statistic on hydra_hier{_d2,_d3,}.yaml, one run per grouping with
--per-group-out, into hydra_saturation.csv with the coverage columns
err_max_cov_0.01/0.05 (share >= tau plus the largest group). --jobs under a
memory budget, --resume, and --hydra-shard i/n to split it across nodes.
--phase optimizer-cost adds Hydra rows (eval variants x W x dataset);
MeasuredAt gains dataset and schema_width, the reducer reading the width off
the row's scores.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
The lib row inserts each value as a UnivMon count and refuses negatives; the truth (and so the polars baseline) accepted them, so a spec with negatives ran on polars and failed on lib. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…cript Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
… checks --phase hydra --resume now writes the kept rows to hydra_saturation.csv.tmp and os.replace()s it over the table, and keeps a row only when it ends in a line end and every column parses (integers, numbers or "" for the error columns, non-empty text). The CSV format is unchanged, so tables written by 003eb31 resume as before. The eval-spec test also checks each label's cardinality, skew and child_of against PR #196's SCHEMAS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…ows too A hydra-univmon-cardinality grid at W=16384 merged from 16 shards outgrows a 251 GB node, and the cost phase can't resume; these flags let the cost phase leave a variant or width out, as they do the hydra phase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the largest covered group can be harder than the smallest. The generator also emits max_covered_share / covered_max_N (the largest group, always covered); the runner reads each covered RQE's accuracy at both the smallest and the largest covered group and keeps the worse (direction-aware), unknown if either read is. Classic RQEs and Hydra's lookup are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…y per service A f64 column with `scale_by: <label column>` and `scale_range: [lo, hi]` multiplies each row by a factor fixed per label value, log-uniform in [lo, hi] from a hash of the value and the column's seed. Validation: earlier string label column, f64 only, 0 < lo <= hi. Absent fields keep every existing spec's output byte-identical. hydra_http_latency.yaml scales latency by service in [1, 10], so groups differ in latency scale and a Hydra-KLL mixture is no longer the queried group's own CDF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…_port) (dst_subnet, dst_port) has 1e6 groups; answering each every minute takes about 49 s on one core, which set every plan's batch latency above the largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet, proto) has 3e3 groups and keeps three groupings over one stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…tes, with measured accuracy by coverage - FamilyProperties for hydra-hll, hydra-univmon-cardinality and hydra-kll: one fixed-size grid, DeltaSet key tracker, no roll-up, and the new answers_any_subgrouping. Added to Capability::families, not to DEPLOYABLE_FAMILIES. - Candidates: on a metric with a hydra_dataset, one grid per schema Λ (each RAQE grouping and the union, where card(Λ) is known); eligible for any RAQE with ∅ ≠ G_r ⊆ Λ. The subpopulations ceiling is gone. - Accuracy: hydra_saturation.csv under the saturation dir (optional), keyed by variant, config, dataset and grouping; the column the RAQE's accuracy_covers_share selects; worst over records and seeds; lossy merges bracketed by merge_shards, unknown past the largest. - Runner: ASAP and PerQuery plan with undeployable families; RQEs carry covers_share; http/flows streams name their Hydra dataset; results report hydra_rqes for workloads Hydra may serve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Round-1 review of #201: - One Hydra grid per (stream, variant, config, window, slide), over the metric's full schema only (MetricFacts::hydra_dataset now carries the dataset and the schema), the width the study measures. - MeasuredAt gains optional `dataset` and `schema_width`; a Hydra row is a candidate on a metric only if measured on its dataset. - Runner: card(full schema) = product of label fan-outs, capped at the stream's series; AutoSketch also allows undeployable families (it skips Hydra itself, #159). - read_csv checks required columns and short rows; config params must be numbers. - Docs: Hydra held to the worst case, built only at the full schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…ect long CSV rows validate_facts reports a Hydra schema outside the metric's labels or without a nonzero cardinality, instead of a later panic. The Hydra accuracy fold skips empty cells (no covered group scored in that run) and is unknown only when every row left the column empty. read_csv rejects rows longer than the header too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
After rebasing onto #196's two-ended coverage read: #201's runner test RQEs need max_covered_share, and #196's coverage test schema needs the label cardinality the full-schema Hydra grid reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
The runner adds the method `asap-norollup` ("ASAP (no roll-ups)"): ASAP's
MILP, candidates, frontier sweep and SLA grid, with each RQE allowed only
deployments at its own grouping or Hydra grids. Every result counts
`rolled_up_rqes` (RQEs served by a finer non-Hydra deployment), and sanity
checks asap <= asap-norollup <= perquery unbounded and at each SLA.
The plot scripts draw it as a dashed violet line, add
fig_rollup_ablation.png (unbounded cost with and without roll-ups, absolute,
CPU and Fargate panels) and a roll-up section in both summaries; results
without the method still plot. The run script passes --no-chosen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
… for the §6.3 eval # Conflicts: # aqpbm-core/src/atomic_costs.rs
Results of the §6.3 synthetic eval (ProjectASAP/ASAPQuery#777) rerun with roll-ups (#190) and their ablation, Hydra candidates measured on the eval's own data (#195–#202), the multi-grouping template set (#196), #192's KLL footprint and a re-measured cost table. 8 workloads, 0 RQEs dropped, 0 sanity violations. Roll-ups serve 40% of the multi-grouping RQEs from a finer deployment and halve ASAP's deployments, but save only 3.3–3.6% (CPU) / 4.4–4.8% (Fargate). Hydra is eligible for many RQEs but never chosen: its insert fans out to every label subset (1.9–5.1 µs per record vs. 3.85 ns for a per-group HLL). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Template 16 is now p99 latency GROUP BY CUBE(region, service, endpoint),
all 7 groupings, replacing {service}, {region, service}, {service, status}.
A new `multigroup` template set runs the multi-grouping templates alone, so
the roll-up ablation is not diluted by the classic set.
Roll-ups now save 60% on multi-grouping (0.551 -> 0.221 vCPU, 26 -> 5
deployments) and 8% on mixed + multi-grouping. Classic is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Collaborator
Author
|
Added f8cecf9: template 16 is now p99 latency 🤖 Generated with Claude Code |
3 tasks done
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
# Conflicts: # .gitignore # docs/rqe_optimizer_candidates.md # docs/rqe_optimizer_cost_model.md # scripts/export_autosketch_eval_table.py # scripts/plot_autosketch_vs_asap_sla.py # scripts/plot_autosketch_vs_asap_synthetic.py # scripts/test_export_autosketch_eval_table.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Results of the §6.3 synthetic eval (ProjectASAP/ASAPQuery#777) with Hydra candidates and a roll-up ablation. Everything is under
rqe-optimizer/results/autosketch-vs-asap-hydra/: figures, result JSONs, summaries, inputs, and a README with the analysis.Base. The base is
hydra/eval-integration, which merges the open stack #195 → #199 → #200 → #202 (with #197 and #198) and #196 → #201. Retarget to main once those merge. The accuracy curves come from #138's study, committed in #194.Findings
Figures
fig_frontier.png: cost vs. query latency.fig_cost_vs_sla.png: cost vs. batch-latency SLA.fig_planning_time.png: planning time.fig_rollup_ablation.png: ASAP with vs. without roll-ups.The README has every table, the method definitions, the inputs, the reproduce commands and the caveats.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis