Skip to content

feat(setagg)!: ASAPv1 payloads for SetAggregator and DeltaResult; drop the portable format - #177

Closed
GordonYuanyc wants to merge 2 commits into
mainfrom
asapv1-only/setagg
Closed

GordonYuanyc wants to merge 2 commits into
mainfrom
asapv1-only/setagg

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator

Part of the ASAPv1-only series: one PR per group, each off main. SetAggregator (0x08 0x00) and DeltaResult (0x09 0x00) only had the portable msgpack format and their ASAPv1 payloads were "TBD". This PR designs them (spec §3.21, §3.22), moves both types onto ASAPv1, and removes the portable format.

Draft for a decision on the payload design below.

Design (spec §3.21 / §3.22)

  1. Metadata is {metadata_version: 1} only: no hash-spec group (nothing hashed reaches the wire) and no key_type (the kind_id fixes the element type as str, as with Coco).
  2. Payloads: SetAggregator [values], DeltaResult [added, removed]. Empty: [[]] / [[], []].
  3. Canonical order: every array strictly ascending in UTF-8 byte order (code point order, not UTF-16). The encoder emits exactly one encoding per set; decode rejects out-of-order or duplicate members rather than re-sorting (as KMV and UniformSampling do).
  4. Members are str and valid UTF-8; bin is rejected.
  5. DeltaResult's added and removed must be disjoint, enforced on encode and decode.
  6. Two kind_ids (from Go's registry), rather than DeltaResult as a variant under SetAggregator's family.

Code

  • Types now live in sketches::set_aggregator with a wire.rs (crate-root asap_sketchlib::{SetAggregator, DeltaResult} still resolve). In-memory API unchanged; new serialize_to_bytes / deserialize_from_bytes.
  • Removed: message_pack_format::portable::{set_aggregator, delta_set_aggregator}, their MessagePackCodec impls, and DeltaResult's serde derives (the serde form was the removed wire format).
  • Goldens (pushed to sketchlib-golden-bytes; submodule bumped): set_aggregator_strings (empty string, proper-prefix chain, 31/32-byte fixstr/str8 boundary, 2/3/4-byte UTF-8), set_aggregator_empty, delta_result_strings, delta_result_empty. Hand-decoded by the reviewer; checked in both directions.

Decisions for you (reviewer recommended keeping each as is)

  • metadata without key_type; reject vs canonicalize out-of-order input; enforce disjointness; drop DeltaResult's serde derives (public API break); two kind_ids; module location under sketches/.

Downstream

ASAPQuery's delta_set_aggregator_accumulator.rs calls DeltaResult::to_msgpack(); it must switch to serialize_to_bytes(), its sink bytes change format, and a DeltaResult with a key in both sets now fails to encode. sketchlib-go has no implementation yet.

Verification (rustc 1.99)

cargo fmt --check; clippy -D warnings for all four CI feature sets; RUSTDOCFLAGS="-D warnings" cargo doc; gen-proto drift clean; --lib -- set_aggregator, asapv1_golden, pbt -- set_aggregator::, e2e -- set_aggregator, msgpack_compat. Wire code mutation-checked. Implemented and reviewed by separate agents; review findings are addressed in the second commit.

🤖 Generated with Claude Code

GordonYuanyc and others added 2 commits October 2, 2026 01:11
…ult; drop the portable format

SetAggregator (0x08 0x00) and DeltaResult (0x09 0x00) get ASAPv1 payloads
(spec §3.21, §3.22): metadata is {metadata_version: 1}; the payloads are
[values] and [added, removed], each an array of str in strictly ascending
UTF-8 byte order. DeltaResult refuses a key in both sets on both sides.

Both types move to sketches::set_aggregator with serialize_to_bytes /
deserialize_from_bytes; the crate-root re-exports are kept. The portable
modules, their MessagePackCodec impls and DeltaResult's serde derives are
removed. Adds four golden fixtures and bumps asapv1_golden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…8, widen the string fixture

Adds SetAggregator and DeltaResult to the Q-NOHASH and Q-ORDER decisions,
requires every member to be valid UTF-8, states the encoder's one encoding
per set, and numbers §3.22's rules in full. The DeltaResult doc states the
disjointness serializing enforces, and read_canonical checks neighbours
before collecting. set_aggregator_strings gains a 31-byte fixstr and a
proper-prefix chain; asapv1_golden is bumped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc

Copy link
Copy Markdown
Collaborator Author

Outdated. ASAPv1 encodings and goldens landed in #232 and #235 against the current spec; removing the legacy formats is redone on current main in #206 and #207.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant