Skip to content

fix(univmon): reject layer seeds and row geometries that break hash independence - #220

Merged
zzylol merged 2 commits into
mainfrom
fix/univmon-hash-independence
Oct 15, 2026
Merged

zzylol merged 2 commits into
mainfrom
fix/univmon-hash-independence

Conversation

@zzylol

@zzylol zzylol commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UnivMon's error bounds (documented in #221) treat every hash a key meets as an independent random draw. Two construction-time gaps break that. This PR rejects both at construction. Valid shapes hash exactly as before, and no wire bytes change.

Fixed

  • Overlapping row fields in CountL2HH. Row r reads its column from bits [r·b, (r+1)·b) of one 128-bit hash (b = ⌈log2 cols⌉) and its sign from bit 127 − r.
    • Once rows·(b+1) > 128, a deep row's column bits overlap the shallow rows' sign bits.
    • Once rows·b ≥ 128, the shift runs past the hash.
    • The ASAPv1 encoder and decoder already refused these shapes, but the constructor built them silently.
    • CountL2HH::with_dimensions_and_seed (and so init_univmon and the UnivMonPyramid constructor) now panics on them. So does the OctoSketch L2hhWorkerSketch, which used to check only rows·b ≤ 128 and so accepted the sign overlap (for example 11 × 2048).
    • One helper, check_l2hh_hash_bits, now serves the constructor, the worker and both wire doors.
  • Layer seeds.
    • Layer i hashes at seed index i, and seed indices wrap modulo the 20-entry SEEDLIST.
    • Layer 19 therefore hashed with BOTTOM_LAYER_FINDER (19), the same seed whose bits decide which keys reach which layer. Layer 20 reused layer 0's seed.
    • MAX_LAYER_SIZE drops from 64 to BOTTOM_LAYER_FINDER (19). Layers 0..=18 use 19 distinct seeds, none of them the finder's. A const assertion pins BOTTOM_LAYER_FINDER < SEEDLIST.len().
    • UnivMonPyramid, and both UnivMon wire decoders through check_layer_size, share the cap.

Why reject instead of drawing more hash

  • Columns: drawing a second 128-bit word per key would change which cells existing shapes use. Every accepted shape already fits: the default 5 × 2048 needs 60 bits, and 10 × 2048 or 9 × 4096 still fit.
  • Seeds: the seed list is inlined in every ASAPv1 sketch's metadata (seed_list). Extending it would change the bytes of every sketch in the crate and every golden. Re-mapping layer seeds would break the wire contract that layer i hashes at seed index i.
  • Capping at 19 layers changes nothing for any pyramid that was correct before. With L ≥ ⌈log2(F0_max/k)⌉ + 2, 19 layers cover F0 up to k·2^17, about 4M distinct keys at k = 32.

Behaviour change

  • init_univmon and the CountL2HH constructor panic on the shapes above.
  • The UnivMon decoders reject layer_size > 19, where they used to accept up to 64. Such a pyramid's layer 19 hashed at the finder's seed.
  • MAX_LAYER_SIZE is public and its value changes.

Tests (all fail on main)

  • countl2hh_rejects_column_bits_that_reach_the_sign_bits (11 × 2048 = 132 bits).
  • init_univmon_rejects_column_bits_that_reach_the_sign_bits.
  • a_univmon_worker_refuses_column_bits_that_reach_the_sign_bits.
  • layer_seeds_are_distinct_and_never_the_layer_finder.
  • init_univmon_rejects_a_layer_on_the_finder_seed.

Also updated:

  • the_widest_accepted_univmon_geometry_still_addresses_distinct_rows goes from 11 × 2048 (which overlaps) to 10 × 2048.
  • countl2hh_accepts_the_widest_disjoint_geometry is new.
  • The two l2hh_wire encoder-rejection tests now build their out-of-budget sketch directly, because the constructor no longer can.

Golden files

None change. No golden uses more than 19 layers or a shape over the hash budget, and asapv1_golden passes unchanged.

Sequencing

This PR is independent of #221; they touch different functions and different sections of api_univmon.md and docs/tests.md, so they merge in either order.

#176 and #198

Verification

  • cargo fmt --check.
  • Clippy -D warnings and cargo test --locked for each of the four CI feature sets.
  • RUSTDOCFLAGS="-D warnings" cargo doc.

…ndependence

Layer i's Count Sketch hashes at seed index i, and seed indices wrap
modulo the 20-entry seed list. So layer 19 hashed at
BOTTOM_LAYER_FINDER, the seed whose bits choose which keys reach each
layer, and layer 20 reused layer 0's seed. MAX_LAYER_SIZE drops from 64
to BOTTOM_LAYER_FINDER (19): layers 0..=18 use 19 distinct seeds, none
of them the finder's. A const assertion pins
BOTTOM_LAYER_FINDER < SEEDLIST.len().

CountL2HH slices every row's column and sign from one 128-bit hash.
Once rows * (ceil_log2(cols) + 1) > 128, deep rows' column bits
overlap shallow rows' sign bits, and from rows * ceil_log2(cols) >= 128
the shift runs past the hash. The wire doors already refused these
shapes, but the constructor built them silently. Now the constructor,
the OctoSketch L2hhWorkerSketch (which checked column bits only) and
both wire doors share check_l2hh_hash_bits.

Shapes that were valid hash exactly as before, so no wire bytes or
goldens change.

@GordonYuanyc GordonYuanyc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes perfect sense. LGTM!

@zzylol
zzylol merged commit e05fbd4 into main Oct 15, 2026
6 checks passed
@zzylol
zzylol deleted the fix/univmon-hash-independence branch October 15, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants