Skip to content

docs(asapv1): peek_kind_id, SetAggregator/DeltaResult, sample_p and delta kinds - #213

Merged
GordonYuanyc merged 4 commits into
mainfrom
asapv1-p1/peek-kind-id
Oct 9, 2026
Merged

GordonYuanyc merged 4 commits into
mainfrom
asapv1-p1/peek-kind-id

Conversation

@GordonYuanyc

@GordonYuanyc GordonYuanyc commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #185, closes #184, closes #181, closes #180. Part of #179.

Spec only: docs/asapv1_wire_format.md. One commit per issue.

#185 peek_kind_id

peek_kind_id checks only rules 1-3 and that kind_id, both length fields, and the metadata_len + payload_len bytes after them lie inside the buffer, then returns the kind_id bytes and ignores bytes after the payload. This is what Rust split and sketchlib-go Split accept today.

#184 SetAggregator and DeltaResult (§3.21, §3.22)

  • Metadata {metadata_version: 1}; payloads [values] and [added, removed].
  • Members are msgpack str holding valid UTF-8, strictly ascending by UTF-8 bytes; added and removed are disjoint.
  • Registry rows 0x08 0x00 / 0x09 0x00 are designed: rows marked implemented or designed have a Section 3 payload.

#181 sample_p (Section 2)

  • Optional f64, last key of the map, written only when p < 1; decode accepts 0 < sample_p < 1 and echoes it back.
  • Carried by HLL, Count-Min, Count Sketch and DDSketch. Rescaled by 1/p: HLL cardinality; Count-Min key frequency and row total; Count Sketch key frequency; DDSketch count and sum.
  • HLL admits by key. A state whose counters already carry a 1/p weight carries no key.
  • States merge only when their sample_p are equal, absent counting as 1.

#180 delta kinds (§3.23)

  • Variant 0x80 | v is the delta of state variant v; Hydra's reserved range ends at 0x07 0x7f.
  • Deltas for HLL Classic/Ertl, Count-Min, CMSHeap, Count Sketch, DDSketch and CSHeap. A delta's metadata is its state kind's map.
  • Payloads list changed positions only: strictly ascending indices, non-zero values. Count-Min/Count Sketch cells use row * cols + col; HLL carries new register values (apply = max); heap kinds carry the producer's whole heap, which replaces the state's.
  • DDSketch: [indices, deltas, negative_indices, negative_deltas, zero_count]; apply order and the representative r = (1 + alpha)·gamma^k are fixed; the index range is given as a formula.
  • Apply is all-or-nothing: a kind or metadata mismatch, or an integer overflow, rejects the delta and leaves the state unchanged.

Open questions

  • sample_p merge rule: ASAPPlanner today merges a sampled state into a reset base at p = 1, and p can differ by edge and epoch; under this rule those merges are rejected, so downstream must carry p across reset.
  • HLL admission function: the Rust collector admits by FNV-1a, sketchlib-go by splitmix64 of the canonical hash; states admitted by different functions over-count when merged.
  • Whether a full decode rejects bytes after the payload is not stated.

🤖 Generated with Claude Code

GordonYuanyc and others added 4 commits October 8, 2026 23:17
peek_kind_id checks the envelope rules 1-3 and the framing lengths,
returns the kind_id bytes, and ignores bytes after the payload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
§3.21 and §3.22 define the two payloads: metadata {metadata_version: 1},
payloads [values] and [added, removed], members as UTF-8 str in strictly
ascending byte order, added and removed disjoint. The registry gains the
designed status for rows whose payload is specified but not yet encoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HLL, Count-Min, Count Sketch and DDSketch carry an optional f64 sample_p,
written only when p < 1. Consumers multiply the listed estimates by 1/p;
decode accepts 0 < p < 1; states merge only with equal sample_p.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The top bit of the variant byte marks a delta of the state variant.
§3.23 defines the HLL Classic/Ertl, Count-Min, CMSHeap, Count Sketch,
DDSketch and CSHeap deltas: sparse ascending indices with non-zero values,
the state kind's metadata, heap kinds carrying the whole heap, and
all-or-nothing apply rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc GordonYuanyc changed the title docs(asapv1): state what peek_kind_id checks and returns docs(asapv1): peek_kind_id, SetAggregator/DeltaResult, sample_p and delta kinds Oct 9, 2026
@GordonYuanyc
GordonYuanyc merged commit db027e5 into main Oct 9, 2026
1 check passed
@GordonYuanyc
GordonYuanyc deleted the asapv1-p1/peek-kind-id branch October 9, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ASAPv1: public peek_kind_id ASAPv1: SetAggregator and DeltaResult payloads ASAPv1: sampling probability (sample_p) ASAPv1: delta kinds

1 participant