study: --phase hydra on the eval's datasets; Hydra cost rows

[zzylol]

Implements PR 6 of the Hydra spec: the eval's datasets, --phase hydra, and Hydra rows in the optimizer cost table.

Base hydra/more-statistics (#200). This branch also contains #197 (datagen/child-of) and #198 (hydra/merge-split, at 6e649de, with merge_split in records) until they merge. The merge commits resolve the Requirement/CLI conflicts by keeping both sides, and the #198 merge tests take #199's Option probe columns.

What changed

• configs/datagen/hydra_http.yaml, hydra_http_latency.yaml, hydra_flows.yaml: eval: multi-grouping anomaly-detection templates (http, flows) for Hydra and roll-ups #196's SCHEMAS. Labels are in schema order. http: region Zipf 0.5 over 4, service Zipf 1.1 over 25, endpoint child_of service ×25 Zipf 1.1, status Zipf 2.0 over 4. The value is user_id Zipf 0.8 over 1e6, or latency f64 Pareto a=2 scale 1000 × a per-service factor log-uniform in [1, 10] (scale_by: 1). flows: dst_subnet Zipf 1.1 over 1e3, dst_port Zipf 1.2 over 1e3, proto 3 values, src_ip uniform over 1e6. The headers say what is not modelled (see Deviations).
• --phase hydra (HYDRA_SWEEPS):
  • eval sweep: hydra-hll and hydra-univmon-cardinality on http and flows, hydra-kll on http_latency. R=3, W∈{1024,4096,16384}, every grouping (15 for http, 7 for flows), N∈{1e5,1e6,1e7}, shards {1,4,16} interleaved, 3 seeds. That is 4779 runs.
  • generic sweep: all 10 statistics on hydra_hier{_d2,_d3,}, W∈{1024,4096}, N=1e6, shards {1,4}, 2 seeds. That is 2000 runs.
  • Each run uses --group-columns and --per-group-out and adds one row to hydra_saturation.csv (the CSV is described below). A study seed rewrites the spec's row_num and offsets every seed by 1000·(s−1).
  • Runs execute under --jobs, admitted only while their estimated memory fits --hydra-mem-gb. --resume keeps finished rows. A failed run is reported, the phase exits 1, and the next --resume reruns it.
  • --hydra-shard i/n takes every n-th run of a fixed list. The --hydra-{ns,ws,seeds,variants,datasets,groupings} flags narrow the sweep.
• --phase optimizer-cost: with hydra in --families (now the default), it adds the eval variants × W × dataset at full schema and COST_N, with 3 runs after 1 warm-up. Non-Hydra rows are unchanged.
  • insert_cpu_secs is per record and merge_cpu_secs per grid merge. query_cpu_secs is per probe, which is a group, or a group and quantile for KLL. Memory is per grid.
  • MeasuredAt gains dataset: Option<String> and schema_width: Option<u32> (skipped when None, as rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage #201 adds them). The reducer reads schema_width from the row's scores and the study sets dataset.
  • Hydra rows skip the seed-mean rerun: their accuracy comes from the CSV.

hydra_saturation.csv

variant, statistic, config, R, W, dataset, group_columns, schema_width, records, fanned_mass, merge_shards, merge_split, seed, err_mean, err_p50, err_p90, err_max, groups_scored, err_max_cov_0.01, err_max_cov_0.05

• group_columns holds label names in schema order, joined by , and quoted.
• merge_split comes from the record and is empty for m=1.
• err_max_cov_τ is the max error over groups with n_q/records ≥ τ, plus the largest group. Unscored groups can be the largest but contribute no error.

Tests

• scripts/test_study_saturation.py has 45 tests, 11 of them new. They cover coverage (threshold, largest group, unscored largest), spec rewriting, the eval specs' labels (names, plus cardinality, skew and child_of/fan_out against eval: multi-grouping anomaly-detection templates (http, flows) for Hydra and roll-ups #196's SCHEMAS), and the phase end to end against a fake approxbench with fixture per-group CSVs: columns, coverage values, merge_split, a failing run, shard disjointness, resume, and a resume over cut-off rows (a row with a field that does not parse, a last row with no line end) that rewrites the table through hydra_saturation.csv.tmp + os.replace. One more test checks the Hydra cost rows (runs, datasets, no seed reruns).
• Mutation checks: dropping the largest-group rule fails a test, and so does picking the largest among scored groups only.
• A Rust test checks MeasuredAt.schema_width.
• fmt, clippy -D warnings and cargo test --workspace pass.

Evidence

• Per-service latency scaling (bb46dfd; hydra-kll, rows=3 cols=1024 cell_k=200, --group-columns 1, N=1e6, release): err_mean / err_max 0.0024 / 0.0048 before, 0.063 / 0.18 after; polars 0 both. All other configs/datagen/*.yaml generate byte-identical tables to fed6f4d.
• Smoke (release binary, N=1e4, W=1024, one grouping, 1 seed, 15 runs, 6 s). Values are err_max / cov 0.01 / cov 0.05:
  • hll http {service}: 1.16 / 1.02 / 0.088
  • kll latency {service}: 0.041 / 0.041 / 0.0021 (before per-service scaling)
  • hll flows {dst_subnet}: 130 / 0.36 / 0.050
  • univmon flows: 160 / 0.43 / 0.32
• Cost smoke (--families hydra --one-config, W=1024, 38 min at 5 runs): every Hydra row was kept with its dataset and width. For example, hydra-hll on http: 51 MB per grid, 6.6 µs insert per record, 55 ms per merge, 0.35 ms per probe. hydra-univmon-cardinality on http: 228 MB, 38 µs per record, 0.59 s per merge.
• Calibration (here, 48 jobs, one grouping per dataset, seed 1, every W/N/m, 255 runs, plus the finest groupings): the heaviest run is univmon on http at N=1e7, about 550–960 s. univmon at W=16384 with 16 shards peaks at about 85 GB, and hll at W=16384 with 16 shards at 17 GB. Scaled to the full sweep, that is about 570k core-s and 9.7M GB·s.

Time estimate and sharding (4 nodes, plus one for the cost phase)

• CPU-bound: about 0.9 h per node at 48 jobs.
• Memory-bound: univmon W=16384 merge-16 runs (85 GB each) allow only 2 at a time under a 200 GB budget, which makes it about 3.4 h per node.
• Plan for about 3.5 h per node. This assumes the nodes have about 250 GB like this one; lower --hydra-mem-gb otherwise.

On node i of clnode109=0, clnode155=1, clnode178=2, clnode167=3 (the running study uses detached checkouts of 003eb31; later heads keep its CSV format, so --resume from them continues those runs):

cd /mydata/sketch-bench-node && source /mydata/sb-env.sh
git fetch origin && git checkout -B hydra/study origin/hydra/study
cargo build --release -p aqpbm-cli
nohup python3 scripts/study_saturation.py --phase hydra --jobs 48 --hydra-mem-gb 200 \
  --hydra-shard i/4 --out out_hydra_i \
  --binary "${CARGO_TARGET_DIR:-target}/release/approxbench" > out_hydra_i.log 2>&1 &
# after an interruption: same command plus --resume

Then gather the four hydra_saturation.csv files and run awk 'FNR>1||NR==1' out_hydra_*/hydra_saturation.csv > hydra_saturation.csv. The shards are disjoint by construction.

Separately, run --phase optimizer-cost serially on clnode138. Its Hydra rows take about 1.5–2 h; the univmon W=16384 merge holds about 80 GB.

Deviations

• flows DDoS burst not generated. datagen draws columns independently, so it cannot tie 1e4 sources to subnet 999. The cov 0.05 column on flows therefore reads the top Zipf subnets (shares about 18%, 8%, 5%), not the burst subnet. That subnet has few distinct sources and is probably harder.
• flows proto is uniform (eval: multi-grouping anomaly-detection templates (http, flows) for Hydra and roll-ups #196's skew: 0.0). The amendment's 80/18/2% split was not used.
• http latency is scaled per service: each service's Pareto latency × a fixed factor, log-uniform in [1, 10] (new datagen scale_by/scale_range, bb46dfd). No 10× endpoint anomaly is injected.
• UnivMon cell is smaller: 3×256, 8 layers, heap 64, instead of the canonical 5×2048. The canonical cell makes a 3×16384 grid 32 GB per shard.
• query_cpu_secs for hydra-kll is per (group, quantile) probe.

Known limitation

• hydra-univmon-cardinality reads about 100% error at N ≥ 1e6 on these datasets: the cell's 8 layers × heap saturate at about 1e6 distinct values per cell (16 layers gives 0.83 at cols=64). Its rows are correct measurements of this cell, and the optimizer rules it out by accuracy.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis