Skip to content

feat(univmon): check_shape refuses a pyramid shape without building it - #229

Open
zzylol wants to merge 1 commit into
mainfrom
fix/univmon-fallible-shape-check
Open

zzylol wants to merge 1 commit into
mainfrom
fix/univmon-fallible-shape-check

Conversation

@zzylol

@zzylol zzylol commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Why

UnivMon::init_univmon panics on a shape it cannot hash independently (#220), and both ASAPv1 wire doors refuse more than MATRIX_MAX_ROWS (20) rows. A caller that sizes a pyramid from configuration cannot ask whether a shape is valid without catching a panic, so ASAPPlanner restated the rules twice: its executor's check_dimensions (positive dimensions, 20 rows, 19 layers, rows * (ceil_log2(cols) + 1) <= 128) and its accuracy model's hash_independent, which missed the 20-row bound. And the constructor itself still built a 21-row pyramid that neither wire door would carry.

What

  • UnivMon::check_shape(heap_size, sketch_row, sketch_col, layer_size) -> Result<(), String>: accepts exactly the shapes init_univmon builds.
  • init_univmon panics exactly when check_shape fails. That adds one case: more than MATRIX_MAX_ROWS rows, which the wire already refused. Every shape that serialized before still builds and hashes as before.

Builds on #220 (merged).

How

check_shape composes the wire's own checks (check_univmon_dimensions, check_layer_size, l2hh_wire::check_dimensions, which carries MATRIX_MAX_ROWS and the 128-bit hash budget) plus a usize overflow check on the counter count. init_univmon's five assert!s become one call. So the constructor and the wire cannot drift.

Key code interfaces added / modified

src/sketch_framework/univmon.rs

impl UnivMon {
    /// Checks a pyramid shape without building it: every dimension positive,
    /// at most MAX_LAYER_SIZE layers, at most MATRIX_MAX_ROWS rows (what the
    /// ASAPv1 wire carries), rows whose column and sign bits fit one 128-bit
    /// key hash, and a counter count that fits usize. Err names the broken
    /// rule; it is bad input. Used before a caller accepts a configuration.
    pub fn check_shape(heap_size: usize, sketch_row: usize, sketch_col: usize, layer_size: usize) -> Result<(), String>;   // new

    /// Creates a pyramid. Panics when `check_shape` fails.
    pub fn init_univmon(heap_size: usize, sketch_row: usize, sketch_col: usize, layer_size: usize) -> Self;   // was: panicked on its own asserts and built > MATRIX_MAX_ROWS rows
}

Usage, end to end (checked by a scratch test at the head commit):

use asap_sketchlib::UnivMon;

// 1. A planner sized an L2 norm query as a UnivMon with a 64-key heap, 5 x 2048 rows and
//    12 layers. Before accepting the plan the executor asks the library. `ok` is Ok(()).
let ok = UnivMon::check_shape(64, 5, 2048, 12);
// `too_deep` is Err("CountL2HH rows 21 exceeds MATRIX_MAX_ROWS 20"): the plan is rejected, not
// built into a state the wire refuses.
let too_deep = UnivMon::check_shape(64, 21, 2, 1);
// `too_wide` is Err("CountL2HH rows=11, cols=2048 needs 132 hash bits (11 column bits + 1 sign
// bit per row), past the 128 the key hash carries").
let too_wide = UnivMon::check_shape(64, 11, 2048, 1);

// 2. An accepted shape builds; `sketch` is the population's state, fed at ingestion and read
//    by the query (12 layers).
let sketch = UnivMon::init_univmon(64, 5, 2048, 12);

Before this PR

Callers copied the rules or caught panics; init_univmon(4, 21, 16, 3) built a pyramid whose serialize_to_bytes then failed.

After this PR

One fallible call answers whether a shape is valid; init_univmon refuses the 21-row pyramid up front. ASAPPlanner's companion PR deletes its check_dimensions and points its accuracy model at check_shape.

Verification

  • New check_shape_accepts_exactly_what_init_univmon_builds: eleven shapes (the 10 x 2048 / 11 x 2048 hash-bit boundary, MAX_LAYER_SIZE and one past, each zero dimension, 20 rows and 21). It failed first on the 21-row shape, which init_univmon built.
  • univmon_rejects_too_many_layer_rows now checks that construction refuses 21 rows (it used to build one to check the encoder refused it); the decoder half is unchanged. univmon_refuses_constructing_past_the_shift_bound expects the wire check's message ("exceeds MAX_LAYER_SIZE").
  • cargo fmt --all -- --check; clippy with -D warnings for --all-features, default features, experimental and octo-runtime; cargo test --all-features --locked (every target) passes.

Limitations

The error is a String, like the wire checks it reuses. UnivMonPyramid (univmon_optimized) keeps its own asserts; it has no planner caller.

A caller that sizes a UnivMon from configuration had to restate the
shape rules (positive dimensions, MAX_LAYER_SIZE layers, rows whose
column and sign bits fit one 128-bit hash, a row count the ASAPv1 wire
carries) to avoid init_univmon's panics. check_shape exposes them as one
fallible call built from the wire's own checks, and init_univmon panics
exactly when it fails.

init_univmon now also refuses more than MATRIX_MAX_ROWS rows. Both wire
doors already refused such a pyramid, so the constructor built a state
it could not serialize. Every shape that serialized before still builds
and hashes as before.
GordonYuanyc added a commit that referenced this pull request Oct 20, 2026
…223, #228, #229, #233, #238) (#242)

* feat(univmon): check_shape refuses a pyramid shape without building it

`UnivMon::check_shape(heap_size, rows, cols, layers)` accepts exactly the
shapes `UnivMon::new` builds: positive dimensions, at most UNIVMON_MAX_LAYERS
layers, at most MATRIX_MAX_ROWS rows, rows within the 128-bit hash budget and
a counter count that fits usize. `new` panics exactly when it fails, so it no
longer builds a pyramid the ASAPv1 wire refuses.

Ports #229.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(univmon): count every cardinality candidate, direct L2, documented error bounds

`estimate_cardinality` counts every heap key, including zero-weight keys and
keys whose Count Sketch estimate is not positive. `estimate_l2` reads layer 0's
Count Sketch F2 estimate in standard mode and sums the strata's F2 estimates in
terminal mode; `estimate_g_sum` keeps the paper's recurrence, and its flag is
now `count_every_key` (`true` counts every heap key). New
`top_layer_complete()`. The L2 bound and the distinct-count bound, stated
jointly with `top_layer_complete()`, are derived in docs/api/api_univmon.md,
with Monte Carlo tests.

Ports #221.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(univmon_two_tier): count every cardinality candidate and direct L2

`UnivMonTwoTier` gets the estimators `UnivMon` has: `estimate_cardinality`
counts every heap key whatever its estimate, zero-weight keys included;
`estimate_l2` reads layer 0's Count Sketch F2 estimate in standard mode and
sums the strata's in terminal mode; new `top_layer_complete()`. The API doc
states the bounds in the two-tier layout's terms, with Monte Carlo tests.

Ports #222.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(hydra): group-total query from per-cell mass

`Hydra::query_group_total(key)` returns a subpopulation's estimated total
mass: the minimum over rows of the subkey's cell `HydraCounter::total_mass()`,
a Count-Min cell's inner row sum or a KLL cell's retained weight. Count Sketch,
HyperLogLog and UnivMon counters return an error. The total is read from
existing cell state, so the wire format is unchanged. `query_key` and the new
query share one subkey hash helper.

Ports #223.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(kll): allocate the item buffer as items arrive

Kll's item buffer starts empty, then holds 16 items and doubles up to
max_capacity; a full buffer compacts only at max_capacity, so retained
items, coin tosses, quantiles, merges and ASAPv1 bytes match the eagerly
allocated buffer for the same input and seed. The merge scratch buffer
starts empty; merge, from_portable_state and the ASAPv1 decoder size the
buffer to what they hold.

lazy_allocation_matches_eager compares every level and the coin with a
test-only eager copy (k = 8 to 1024, up to 50,000 items, with a merge).

Ports #238.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(kll): Kll::memory_bytes and KllDynamic::memory_bytes

Both report the bytes the sketch holds now: Kll its struct, item buffer,
level array, merge scratch space and memoized CDF; KllDynamic its struct,
item buffer and level bounds.

Ports #238.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kmv): estimate and merge by shared reference, memory_bytes, early reject past the k-th hash

Kmv::estimate takes &self and Kmv::merge takes &Kmv. Kmv::memory_bytes
is the struct plus the hash slots the heap has allocated
(CommonHeap::allocated_slots, crate-private). A full sketch turns away a
hash at or above its largest retained one before scanning the retained
hashes for a duplicate; the retained set and every estimate are unchanged.

Ports #228.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(heap)!: TopKHeap::estimated_memory_bytes; UnivMonQ::memory_bytes

TopKHeap::estimated_memory_bytes(heap_size, key) is the bytes a heap of
that capacity holds in steady state once keys shaped like key churn
through it: the struct, entry and digest arrays, the digest index (std
HashMap layout, grown once more when tombstones force a resize of an index
over half full) and the owned key bytes. A heap holds at most this.

UnivMonQ::estimated_memory_bytes(&self) becomes memory_bytes(&self) and
counts what the instance holds: its counters, candidate tables (bucket
count from the largest capacity() seen), candidate heaps and ordered-sample
heap.

tests/heap_memory_bytes.rs checks both against a counting global
allocator: the heap estimate as an upper bound after a fill and exactly
after churn, UnivMon-Q exactly.

Ports #233 (the heap half).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(cms,cs)!: check_shape, clear and memory accounting for the matrix sketches

CountMinSketch, CountSketch and their WithHeap and WithSpaceSaving forms
gain check_shape(rows, cols), which accepts the ASAPv1 geometry,
1 <= rows <= MATRIX_MAX_ROWS and 1 <= cols <= u32::MAX, on every hashing
path. with_dimensions, new and from_storage panic on any other shape. Past
20 rows the regular path and the fast path's per-row (Rows) layout hash
row r + 20 with row r's seed; the packed layouts reuse no seed. Tests that
built such a shape to watch the encoder refuse it check check_shape
instead, Hydra's included.

clear() zeroes the counters (and empties the heap for the WithHeap types)
and returns the Nitro sampler to its start (Nitro::reset, through the new
MatrixStorage::reset_sampler), keeping shape, allocation and admission
rate.

TopKHeap::memory_bytes reports the bytes a heap holds, taking the digest
index's bucket count from the largest capacity() it has reported, which
evictions' tombstones cannot lower. The WithHeap types add memory_bytes
and estimated_memory_bytes(rows, cols, heap_size, key).
tests/heap_memory_bytes.rs checks all of them against a counting
allocator, after a fill and after 50 times heap_size evicting keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(hash)!: a seed for every seed index; matrix sketches take any row count

Seed index i hashes with seed_for_index(i) = SEED_LIST[i % 20] *
(i / 20 + 1) + i / 20 in wrapping u64 arithmetic; indexes below 20 keep
their seed. The matrix hash passes seed indexes through unreduced, so rows
past 20 on the regular path and the fast path's per-row layout hash
independently, and UnivMon-Q's occurrence priorities hash at their own seed.

check_shape on Count-Min, Count Sketch and their WithHeap and
WithSpaceSaving forms refuses only a zero dimension and a counter count past
usize; Elastic, Hydra and UnivMon build any row count. to_asapv1 and
from_asapv1 keep the MATRIX_MAX_ROWS bound.

The UnivMon-Q residual-CDF accuracy test hashes each of its 32 sketches at
its own seed index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(cms,cs)!: memory_bytes, CountMinF64 at the root, check_shape without type parameters

CountMinSketch and CountSketch gain memory_bytes(&self): the struct plus
its rows * cols counters, exact for Vector2D storage, checked against a
counting allocator in tests/heap_memory_bytes.rs. CountMinF64 is
re-exported from sketches and the crate root.

check_shape on CountMinSketch, CountSketch, their WithHeap and
WithSpaceSaving forms is defined on the type with its default parameters,
so CountMinSketch::check_shape(3, 64) resolves without naming storage,
path or hasher. A call naming non-default type parameters no longer
compiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(asapv1)!: seed_derivation is "seed_list_index_scaled"

The ASAPv1 seed_derivation string names seed_for_index. Bytes carrying
"seed_list_index_wrap" no longer decode. The golden submodule moves to the
regenerated fixtures (sketchlib-golden-bytes#3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: zz_y <zeyingz@umd.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant