Repository navigation
polars: tag Hydra baseline subset keys by column (#74) - #195
Merged
Merged
Conversation
The polars exact baselines keyed a label subset by its `;`-joined values,
asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its
column (`label0:a;label1:b`), so on label columns that share a value
domain the baseline merged a key1 group with the same key2 group and
scored nonzero against itself. subset_key now builds the 0.3.0 key over
hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty
labels keep their column, and every lookup goes through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…mment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
zzylol
force-pushed
the
74-polars-tagged-subkeys
branch
from
October 11, 2026 16:35
912c3ab to
4149b7e
Compare
zzylol
added a commit
that referenced
this pull request
Oct 11, 2026
…n (§6.3) (#203) * polars: tag Hydra baseline subset keys by column (#74) The polars exact baselines keyed a label subset by its `;`-joined values, asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its column (`label0:a;label1:b`), so on label columns that share a value domain the baseline merged a key1 group with the same key2 group and scored nonzero against itself. subset_key now builds the 0.3.0 key over hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty labels keep their column, and every lookup goes through it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: multi-grouping anomaly-detection templates (http, flows) The generator gets a metric-level ordered label schema (labels with cardinality, child_of fan-out and skew; per-value data shape) and per-RQE `grouping` and `covers_share`. The 10 templates are now the "classic" set, byte-for-byte the committed tables; "all" adds templates 11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry several groupings so roll-ups (and later Hydra) can share. The plan runs both sets over the grid. The runner builds each schema stream's MetricFacts from the schema (labels = schema + x; cardinality per used grouping and the full set), takes Raqe.grouping_labels from the RQE's grouping, maps the `cardinality` capability to HLL, and reports covers_share and roll-up deployment groupings in each choice. Classic output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * polars: write subset key tags without a temporary String; fix test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: interleaved merge split; merge exactness and bound tests `--merge-split contiguous|interleaved` (default contiguous, unchanged) picks how merge runs split the stream into shards. Interleaved reorders the stream so the existing contiguous `partition` deals it round-robin by record, at the contiguous split's shard sizes, so every row's folds take it unchanged. Tests: hydra-cms, hydra-cs and hydra-hll grids folded from 4 and 7 shards, under both splits, answer every probed subpopulation exactly as the single stream; and every scored group on a fixed 20k-record stream lands inside its hydra doc §2.2 bound (CMS also never under). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: child_of columns for hierarchical labels A `child_of: <index>` column with `fan_out: f` nests under an earlier string column: its value is the parent row's value, a `.`, and a child index in [0, f) drawn from the column's own distribution. Bad indices, a fan_out that disagrees with the domain, and fields that would not mean what they say on a child are refused before drawing. Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint 25, status 4, user_id value) for the hydra rows, docs, and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: score any target grouping; per-group error output Every hydra-* row (lib and polars, all five variants and hydra-univmon's four statistics) takes `--group-columns J,K,...` (default `0`, today's behavior) in place of the hardcoded SCORED_LABEL_COLUMN. A subpopulation probe is now positioned (`Vec<Option<String>>`, Some at the grouped columns), so `hydra_shared::query` asks any non-empty subset of the key columns and the polars baselines look up `subset_key` with that subset's mask (`group_key`, generalizing `prefix_key`). Each grouped comparator reports its per-group error in its own metric (relative error; mean rank error for hydra-kll) and summarises it as err_mean, err_p50, err_p90, err_max, groups_scored, plus schema_width, records and fanned_mass. `--per-group-out PATH` writes the rows as `group_key,n_q,error`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: whole uniform bounds for child_of; 2- and 3-label hierarchy cuts A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew child indices 0, 1 and 2; child_of now requires whole uniform bounds. hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first 2 and 3 labels for the Hydra study's schema-width sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: skewed http labels, per-RQE group coverage, fresh-checkout test Review of #196. http's labels are now Zipf-skewed (region 0.5, service 1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS burst targets a tail subnet, so covers_share selects a few groups. The generator computes each group's share (product of its labels' shares) and, per RQE, the covered groups (share >= tau, plus always the largest) and the smallest covered group's share and items per window. The runner reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's items instead of the mean group's; covers_share = null is unchanged. The schema test reads an inline HLL curve fixture instead of the gitignored study curves. The synthetic plot keys runs by template set too, and both plot scripts title classic "mixed" and all "mixed + multi-grouping". README: the coverage rule, the N the runner uses, and which schema fields are descriptive. label_set.keys_per_window is commented as the universe K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint} - covers_share = null covers every group, so the runner reads its accuracy at the smallest group (min_covered_share), like a tau RQE's smallest covered group. Classic RQEs (no covers_share) are unchanged. - Template 17 (negative control) groups by {service, endpoint}: its smallest group holds 4.5e4 items in 5m, on the curves. - The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the vacuous `assert covered`); a test checks the old full-label-set template fails it. - Covered facts go through validate_facts. - README: null semantics, template 17, and the numbers' provenance (fresh `all` table, committed saturation inputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: the hydra_hier usage example passes --dtype Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: list every group in --per-group-out; escape keys; refuse no accuracy - Group keys escape `\`, `:` and `;` as polars_shared::subset_key does. - Groups that cannot be scored (zero true statistic, e.g. zero entropy) are written with an empty `error`, so n_q sums to the stream's records. - --per-group-out with no accuracy measurement is refused instead of writing an empty file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: KLL CDF and UnivMon sum rows; KLL cell merge buffer - hydra-kll-cdf (lib, polars): asks the KLL grid HydraQuery::Cdf(x) at each group's own values at phi = 0.01..0.99, scored by absolute CDF error (SubpopCdfErrorGT, comparator subpop-cdf-error). The polars baseline is the exact share at or below x over the same sorted groups. - hydra-univmon-sum (lib, polars): each record inserted with count = value, so the UnivMon grid's L1 is the group's sum, scored by relative error against exact per-group sums (SubpopSumGT, comparator subpop-sum). Integer value dtypes only; f64/str and values outside [0, i32::MAX] are refused by name. - memory_hydra_kll counts each cell as kll_lib_bytes does, merge buffer of cell_k items included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra-kll: the footprint's merge buffer is the steady state; tie tests The grid clones its cells from one template, and a cloned Vec keeps no capacity, so a cell's merge buffer is empty at build and grows to about cell_k items at its first compaction. Say the footprint counts it at that bound. The CDF test now asks a group of ties (at or below) and a mixed-tie group on a grid wide enough that it shares no cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: record merge_split; review fixes - `merge_split` (contiguous|interleaved) is a record field next to `merge_shards` (BenchSection and MergeMetrics; schema regenerated). Merge records stamp it; a record without it split contiguously, so existing results parse, and flatten refuses two different splits in one row as it does two shard counts. - `MergeSplit` moves to aqpbm-core (it is a record type) and is parsed by clap directly instead of matching strings. - The interleaved copy of the stream is built only when a merge runs. - README states the round-robin exactly (contiguous shard sizes, so a short last shard shifts the pattern); MergeMetrics docs name the split. - hydra-cs bound test: note its margin (passes from beta ~4.8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: --phase hydra on the eval's datasets; Hydra cost rows configs/datagen/hydra_http.yaml, hydra_http_latency.yaml and hydra_flows.yaml follow the §6.3 eval's http and flows schemas (labels in schema order, cardinalities, child_of, skews; user_id Zipf 0.8, latency Pareto a = 2, src_ip uniform). The flows DDoS burst is not generated: datagen draws each column independently. --phase hydra runs HYDRA_SWEEPS: the eval's variants on those datasets (every grouping, W 1024/4096/16384, N 1e5..1e7, 1/4/16 interleaved shards, 3 seeds) and every statistic on hydra_hier{_d2,_d3,}.yaml, one run per grouping with --per-group-out, into hydra_saturation.csv with the coverage columns err_max_cov_0.01/0.05 (share >= tau plus the largest group). --jobs under a memory budget, --resume, and --hydra-shard i/n to split it across nodes. --phase optimizer-cost adds Hydra rows (eval variants x W x dataset); MeasuredAt gains dataset and schema_width, the reducer reading the width off the row's scores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra-univmon-sum: the ground truth refuses negative values too The lib row inserts each value as a UnivMon count and refuses negatives; the truth (and so the polars baseline) accepted them, so a spec with negatives ran on polars and failed on lib. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: Hydra cost rows at 3 runs; their test; datagen path from the script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: Hydra cost rows' run count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: atomic hydra table rewrite, stricter resume, schema checks --phase hydra --resume now writes the kept rows to hydra_saturation.csv.tmp and os.replace()s it over the table, and keeps a row only when it ends in a line end and every column parses (integers, numbers or "" for the error columns, non-empty text). The CSV format is unchanged, so tables written by 003eb31 resume as before. The eval-spec test also checks each label's cardinality, skew and child_of against PR #196's SCHEMAS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: --hydra-variants/--hydra-ws narrow the Hydra cost rows too A hydra-univmon-cardinality grid at W=16384 merged from 16 shards outgrows a 251 GB node, and the cost phase can't resume; these flags let the cost phase leave a variant or width out, as they do the hydra phase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: hold covered RQEs to the worse of their smallest and largest group Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the largest covered group can be harder than the smallest. The generator also emits max_covered_share / covered_max_N (the largest group, always covered); the runner reads each covered RQE's accuracy at both the smallest and the largest covered group and keeps the worse (direction-aware), unknown if either read is. Classic RQEs and Hydra's lookup are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * aqpbm-datagen: scale_by/scale_range; hydra_http_latency scales latency per service A f64 column with `scale_by: <label column>` and `scale_range: [lo, hi]` multiplies each row by a factor fixed per label value, log-uniform in [lo, hi] from a hash of the value and the column's seed. Validation: earlier string label column, f64 only, 0 < lo <= hi. Absent fields keep every existing spec's output byte-identical. hydra_http_latency.yaml scales latency by service in [1, 10], so groups differ in latency scale and a Hydra-KLL mixture is no longer the queried group's own CDF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port) (dst_subnet, dst_port) has 1e6 groups; answering each every minute takes about 49 s on one core, which set every plan's batch latency above the largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet, proto) has 3e3 groups and keeps three groupings over one stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage - FamilyProperties for hydra-hll, hydra-univmon-cardinality and hydra-kll: one fixed-size grid, DeltaSet key tracker, no roll-up, and the new answers_any_subgrouping. Added to Capability::families, not to DEPLOYABLE_FAMILIES. - Candidates: on a metric with a hydra_dataset, one grid per schema Λ (each RAQE grouping and the union, where card(Λ) is known); eligible for any RAQE with ∅ ≠ G_r ⊆ Λ. The subpopulations ceiling is gone. - Accuracy: hydra_saturation.csv under the saturation dir (optional), keyed by variant, config, dataset and grouping; the column the RAQE's accuracy_covers_share selects; worst over records and seeds; lossy merges bracketed by merge_shards, unknown past the largest. - Runner: ASAP and PerQuery plan with undeployable families; RQEs carry covers_share; http/flows streams name their Hydra dataset; results report hydra_rqes for workloads Hydra may serve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: Hydra grids at the full schema from their dataset's rows Round-1 review of #201: - One Hydra grid per (stream, variant, config, window, slide), over the metric's full schema only (MetricFacts::hydra_dataset now carries the dataset and the schema), the width the study measures. - MeasuredAt gains optional `dataset` and `schema_width`; a Hydra row is a candidate on a metric only if measured on its dataset. - Runner: card(full schema) = product of label fan-outs, capped at the stream's series; AutoSketch also allows undeployable families (it skips Hydra itself, #159). - read_csv checks required columns and short rows; config params must be numbers. - Docs: Hydra held to the worst case, built only at the full schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: validate the Hydra schema, skip empty Hydra cells, reject long CSV rows validate_facts reports a Hydra schema outside the metric's labels or without a nonzero cardinality, instead of a later panic. The Hydra accuracy fold skips empty cells (no covered group scored in that run) and is unknown only when every row left the column empty. read_csv rejects rows longer than the header too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: test fixtures carry max_covered_share and label cardinalities After rebasing onto #196's two-ended coverage read: #201's runner test RQEs need max_covered_share, and #196's coverage test schema needs the label cardinality the full-schema Hydra grid reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: roll-up ablation, ASAP without roll-ups The runner adds the method `asap-norollup` ("ASAP (no roll-ups)"): ASAP's MILP, candidates, frontier sweep and SLA grid, with each RQE allowed only deployments at its own grouping or Hydra grids. Every result counts `rolled_up_rqes` (RQEs served by a finer non-Hydra deployment), and sanity checks asap <= asap-norollup <= perquery unbounded and at each SLA. The plot scripts draw it as a dashed violet line, add fig_rollup_ablation.png (unbounded cost with and without roll-ups, absolute, CPU and Fargate panels) and a roll-up section in both summaries; results without the method still plot. The run script passes --no-chosen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: AutoSketch vs. ASAP with Hydra candidates and a roll-up ablation Results of the §6.3 synthetic eval (ProjectASAP/ASAPQuery#777) rerun with roll-ups (#190) and their ablation, Hydra candidates measured on the eval's own data (#195–#202), the multi-grouping template set (#196), #192's KLL footprint and a re-measured cost table. 8 workloads, 0 RQEs dropped, 0 sanity violations. Roll-ups serve 40% of the multi-grouping RQEs from a finer deployment and halve ASAP's deployments, but save only 3.3–3.6% (CPU) / 4.4–4.8% (Fargate). Hydra is eligible for many RQEs but never chosen: its insert fans out to every label subset (1.9–5.1 µs per record vs. 3.85 ns for a per-group HLL). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: cube template and a multi-grouping-only workload Template 16 is now p99 latency GROUP BY CUBE(region, service, endpoint), all 7 groupings, replacing {service}, {region, service}, {service, status}. A new `multigroup` template set runs the multi-grouping templates alone, so the roll-up ablation is not diluted by the classic set. Roll-ups now save 60% on multi-grouping (0.551 -> 0.221 vCPU, 26 -> 5 deployments) and 8% on mixed + multi-grouping. Classic is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: each workload's chosen plans, gzipped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #74. This is PR 1 of the Hydra measurement plan in #193 (
docs/rqe_optimizer_hydra.md§5.2).Problem. asap_sketchlib 0.3.0's Hydra keys each label subset by column (
label0:abc;label1:x), so equal values in different label columns never alias in the sketch. Thepolarsexact baselines still built 0.2.2's untagged key (abc;x). On label columns that share a value domain, such asconfigs/datagen/hydra_columns.yaml's shared alphabet, they merged a key1 group with the key2 group of the same value. As a result the exact rows scored nonzero.Change
polars_shared::subset_keybuilds the 0.3.0 subkey overhydra_shared'slabel{i}schema, escaping\,:and;as the library does.prefix_keyis the lookup for a probe of the leading columns, and every polars lookup uses it: frequency, cardinality, univmon and hydra-kll.Tests
subset_keys_tag_each_value_with_its_column.a_value_shared_by_two_columns_is_counted_once: recordsa;aanda;b. Groupain column 0 holds 2 records; the old key counted 3.cargo test -p sketch-benchpasses (98), as do fmt and clippy.Evidence. These are the polars rows on the shared-alphabet data:
hydra_columns.yaml, and its_f64version for KLL. Grids: cms 3×128 with 3×512 cells, hll 3×128, kll 3×64 with k=200.--operations query --metrics accuracy.hydra-cmspolarsare_all0.596,aae_all3.74hydra-hllpolarsare_all0.249,aae_all56.8hydra-kllpolarsmean_rank_err0.00093,max_rank_err0.036🤖 Generated with Claude Code
https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis