feat: add kll sketch implementation and tests - #76
Open
PsiACE wants to merge 3 commits into
Open
Conversation
PsiACE
marked this pull request as ready for review
July 29, 2026 04:16
tisonkun
requested review from
ariesdevil,
notfilippo and
tisonkun
and
a lite review from Copilot
August 18, 2026 12:05
Contributor
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a new optional kll feature implementing a KLL quantiles sketch with Apache DataSketches-compatible compact serialization, plus unit/serde compatibility tests (including Java/C++ acceptance fixtures).
Changes:
- Introduce
datasketches::kllmodule withKllSketchAPI, sorting/query logic, and compact (de)serialization forf32,f64,i64, andString. - Wire the new sketch into crate feature flags and codec family IDs.
- Add KLL behavior tests and cross-language serialization compatibility tests (Java/C++ fixtures + negative serde tests).
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| datasketches/tests/serde_tests/kll.rs | Adds Java/C++ compact-serialization compatibility tests and rejects malformed inputs. |
| datasketches/tests/serde_tests.rs | Registers KLL serde tests behind the kll feature. |
| datasketches/tests/kll_test/sketch.rs | Adds behavior tests for ranks/quantiles/CDF/PMF/merge/reset and custom comparators. |
| datasketches/tests/kll_test/main.rs | Adds KLL test harness module entrypoint. |
| datasketches/src/lib.rs | Exposes kll module behind the kll feature. |
| datasketches/src/kll/sorted_view.rs | Implements sorted-view-based rank/quantile/CDF/PMF queries. |
| datasketches/src/kll/sketch.rs | Core KLL sketch implementation + compact serialization/deserialization. |
| datasketches/src/kll/serialization.rs | Defines KLL preamble/flag/layout constants used for compatibility. |
| datasketches/src/kll/mod.rs | Adds the public kll module surface (types, aliases, constants). |
| datasketches/src/kll/helper.rs | Adds capacity math and RNG bit helper for compaction/downsampling. |
| datasketches/src/error.rs | Adds invalid_preamble_ints helper for KLL deserialize validation. |
| datasketches/src/codec/mod.rs | Includes kll in codec gating. |
| datasketches/src/codec/family.rs | Adds Family::KLL ID metadata behind the kll feature. |
| datasketches/Cargo.toml | Adds kll feature and registers kll_test binary test target. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+749
to
+753
| fn compress_while_updating(&mut self) { | ||
| let level = self.find_level_to_compact(); | ||
| if level + 1 == self.levels.len() { | ||
| self.levels.push(Vec::new()); | ||
| } |
Comment on lines
+740
to
+747
| fn internal_update(&mut self, item: T) { | ||
| if self.num_retained() >= self.capacity() { | ||
| self.compress_while_updating(); | ||
| } | ||
| self.n += 1; | ||
| self.is_level_zero_sorted = false; | ||
| self.levels[0].insert(0, item); | ||
| } |
Comment on lines
+256
to
+263
| /// Returns the normalized rank of the given item. | ||
| pub fn rank(&self, item: &T, inclusive: bool) -> Option<f64> { | ||
| if self.is_empty() { | ||
| return None; | ||
| } | ||
| let view = build_sorted_view(&self.levels, self.comparator.clone()); | ||
| Some(view.rank(item, inclusive)) | ||
| } |
| } | ||
|
|
||
| fn serialize(value: &Self, bytes: &mut SketchBytes) { | ||
| bytes.write_u32_le(value.len() as u32); |
Member
|
I'll review this PR in the next 2 days :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Add KLL quantiles with current DataSketches semantics and interoperable compact serialization.
What changes are included in this PR?
f32,f64,i64, andString, compatible with current Java and C++ implementations.Are there any user-facing changes?
Yes. This adds the optional
kllfeature and publicKllSketchAPI. Existing users are unaffected unless the feature is enabled.AI Usage Statement
Developed with OpenAI Codex. The implementation and tests were checked against the Java and C++ references and validated with the repository test and lint workflows.