diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..cec140ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,10 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- An ASAPv1 golden fixture for CountL2HH (`0x19 0x00`), `count_l2hh_2x4_seed7`: + a non-zero seed index, and cells and `l2` accumulators set apart from each + other that together reach every msgpack integer width, positive fixint + through uint64 and negative fixint through int64. ### Changed @@ -88,6 +92,13 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **CountL2HH's `MessagePackCodec` impl** and its module, + `message_pack_format::native::countsketch_topk`. Use + `CountL2HH::serialize_to_bytes` / `CountL2HH::deserialize_from_bytes`, which + emit the same ASAPv1 bytes. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..d6e5d63c 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit d6e5d63c39c637016fe069f27acc320983c7947d diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..acabb9ad 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +The [Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) of `asapv1_golden/README.md` lists the `kind_id`s the fixtures cover. For a kind without a fixture, this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..956a8180 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for; +[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists them. A kind +without a fixture has **no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..af3c10c7 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2123,6 +2123,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | --- | --- | --- | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | +| `count_l2hh_2x4_seed7_matches_golden` | A known CountL2HH state, its `l2` accumulators and seed index included, serializes to its checked-in golden bytes. | Builds `CountL2HH` from its named serde form with counts `[[127, 128, 65535, -32768], [-32, -33, -2147483648, i64::MIN]]`, `l2` `[65536, i64::MAX]` and `seed_idx` `7`, verifies it serializes to `asapv1_golden/count_l2hh_2x4_seed7.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, `seed_idx()` `7`, the eight cells, the two `l2` accumulators, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | diff --git a/src/message_pack_format/native/countsketch_topk.rs b/src/message_pack_format/native/countsketch_topk.rs deleted file mode 100644 index 9cc4e26c..00000000 --- a/src/message_pack_format/native/countsketch_topk.rs +++ /dev/null @@ -1,15 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::countsketch_topk::CountL2HH`]. - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::countsketch_topk::CountL2HH; -use crate::{HashProfile, SketchHasher}; - -impl MessagePackCodec for CountL2HH { - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..cfaaae1a 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -9,7 +9,6 @@ pub mod countminsketch; pub mod countsketch; -pub mod countsketch_topk; pub mod ddsketch; pub mod hll; pub mod kll; diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..5d2ff847 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountL2HH, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -30,6 +30,7 @@ const GOLDEN_CMS_F64: &str = include_str!("../asapv1_golden/cms_f64_fast_2x3.hex const GOLDEN_CS_REGULAR: &str = include_str!("../asapv1_golden/cs_i64_regular_2x4.hex"); const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex"); const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); +const GOLDEN_L2HH: &str = include_str!("../asapv1_golden/count_l2hh_2x4_seed7.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); @@ -287,3 +288,75 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// CountL2HH: counts, l2 accumulators and seed index set through the sketch's +// serde form -> serialize == golden, and golden round-trips. +// --------------------------------------------------------------------------- + +/// CountL2HH cells are signed: row 0 is positive fixint max / uint8 / uint16 / +/// int16, row 1 is negative fixint min / int8 / int32 / int64. +const L2HH_COUNTS: [[i64; 4]; 2] = [[127, 128, 65535, -32768], [-32, -33, -2147483648, i64::MIN]]; +/// One accumulator per row, set independently of the cells: uint32 / uint64. +const L2HH_L2: [i64; 2] = [65536, i64::MAX]; +const L2HH_SEED_INDEX: usize = 7; + +#[derive(serde::Serialize, serde::Deserialize)] +struct L2hhMatrixForm { + data: Vec, + rows: usize, + cols: usize, +} + +#[derive(serde::Serialize, serde::Deserialize)] +struct L2hhVectorForm { + data: Vec, +} + +/// The named serde form of `CountL2HH`. +#[derive(serde::Serialize, serde::Deserialize)] +struct L2hhForm { + counts: L2hhMatrixForm, + l2: L2hhVectorForm, + row: usize, + col: usize, + seed_idx: usize, +} + +fn l2hh_form(counts: &[[i64; 4]; 2], l2: &[i64; 2], seed_idx: usize) -> L2hhForm { + L2hhForm { + counts: L2hhMatrixForm { + data: counts.iter().flatten().copied().collect(), + rows: 2, + cols: 4, + }, + l2: L2hhVectorForm { data: l2.to_vec() }, + row: 2, + col: 4, + seed_idx, + } +} + +#[test] +fn count_l2hh_2x4_seed7_matches_golden() { + let want = decode_hex(GOLDEN_L2HH); + + let form = l2hh_form(&L2HH_COUNTS, &L2HH_L2, L2HH_SEED_INDEX); + let sketch: CountL2HH = + rmp_serde::from_slice(&rmp_serde::to_vec_named(&form).expect("serde form")) + .expect("CountL2HH from its serde form"); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "CountL2HH bytes diverge from golden"); + + let decoded: CountL2HH = CountL2HH::deserialize_from_bytes(&want).expect("decode"); + let flat: Vec = L2HH_COUNTS.iter().flatten().copied().collect(); + assert_eq!(decoded.as_storage().as_slice(), flat.as_slice()); + assert_eq!(decoded.rows(), 2); + assert_eq!(decoded.cols(), 4); + assert_eq!(decoded.seed_idx(), L2HH_SEED_INDEX); + let state: L2hhForm = + rmp_serde::from_slice(&rmp_serde::to_vec_named(&decoded).expect("serde form")) + .expect("decoded serde form"); + assert_eq!(state.l2.data, L2HH_L2); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +}