Repository navigation
refactor(hll)!: ASAPv1 only — drop proto, portable and native formats - #166
Draft
GordonYuanyc wants to merge 3 commits into
Draft
GordonYuanyc wants to merge 3 commits into
GordonYuanyc wants to merge 3 commits into
Conversation
HyperLogLog serializes only as ASAPv1. Remove the portable HllSketch, HllSketchDelta and HllVariant with the sparse-register helpers, the native MessagePackCodec shims, the HLLDelta proto message and the SketchEnvelope hll field (12, now reserved). HyperLogLogState stays for HydraCell. Add HyperLogLogImpl::from_storage and HyperLogLogHIPImpl::from_storage, and check the three HLL ASAPv1 goldens through the native types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Make the HLL wire DTOs, kind_id constants and metadata builder private to wire.rs, state only what the remaining HLL proto messages are, shorten the HIP registers_of doc, and drop a stray proptest regression file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump asapv1_golden to the commit adding hll_classic_p14, hll_ertl_mle_p14 and hll_hip_p14, built from fixed registers holding the maximum P14 rank, and check each in both directions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the ASAPv1-only series: one PR per sketch group, each off
main. This one removes HyperLogLog's protobuf, portable msgpack and native-shim formats; ASAPv1 (0x01 0x01Classic,0x01 0x02ErtlMLE,0x01 0x03HIP) is the only HLL format left.Draft because of the open questions below.
Removed
HLLDelta;SketchEnvelope.hll(12) is nowreserved 12; reserved "hll";.HyperLogLogState,HLLSparseRegistersandHLLVariantstay becausehydra.protoembedsHyperLogLogState; they go with the Hydra PR or the final cleanup.message_pack_format::portable::hll:HllSketch,HllSketchDelta,HllVariant(and their crate-root re-exports),registers_from_state,decode_sparse_registers,encode_sparse_registers, and the proto/msgpack delta codecs.message_pack_format::native::hll(theMessagePackCodecimpls for the native HLL types).src/sketches/testdata/hll_envelope_golden.hexand the tests that only exercised the removed formats.Added
HyperLogLogImpl::from_storage(registers)andHyperLogLogHIPImpl::from_storage(registers, kxq0, kxq1, est), so a native HLL can be built from known raw state (mirrorsCountMin::from_storage). The three existing HLL goldens are now checked through the native types in both directions.hll_classic_p14,hll_ertl_mle_p14,hll_hip_p14; registers include the P14 maximum rank 51), pushed to sketchlib-golden-bytes; submodule bumped. sketchlib-go's HLL is fixed at P14, so these are the fixtures it checks against.Open questions
SketchEnvelope{HyperLogLogState}plus bareHLLDelta. After this PR Rust decodes neither; ASAPv1 has no HLL delta.sketch_state == Noneand no error.Downstream
ASAPQuery-backend (via ASAPPlanner's codec crates) decodes the proto full state (dense and sparse) and deltas; ASAPPlanner/ASAPQuery name
HllSketch/HllVariant.Verification (rustc 1.99)
cargo fmt --check; clippy-D warningsfor all four CI feature sets;RUSTDOCFLAGS="-D warnings" cargo doc; gen-proto drift clean; filtered tests for the touched areas (--lib -- hll,--test pbt -- hll::,asapv1_golden,conformance_kit,msgpack_compat,sketches_go_parity_probe,xtest_consumer,e2e -- cardinality::,hll_custom_precision). Implemented and reviewed by separate agents; review findings are addressed in the second commit; the third adds the P14 goldens.🤖 Generated with Claude Code