diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b705eb..08cd5b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,22 +7,29 @@ All significant changes to this project will be documented in this file. ### Breaking changes * Move the `hash_value` module to `hash::value`. -* Remove `ThetaSketch::builder`, `ThetaUnion::builder`, and `TupleSketch::builder`. Construct `ThetaSketchBuilder`, `ThetaUnionBuilder`, and `TupleSketchBuilder` with `Default::default` instead. -* Standardize Theta and Tuple set-operation constructors. Zero-configuration operators use - `Default::default()`, `TupleIntersection::new(policy)` uses the default seed, and `with_seed` - accepts a custom seed. This replaces the previous `new(seed)` and `new_with_default_seed` - methods. +* Rename `Coupon::from_hash` to `Coupon::from_value` to reflect that the method hashes the supplied value itself. +* Remove `ThetaSketch::builder`; construct `ThetaSketchBuilder` with `Default::default` instead. +* Replace the sealed `ThetaSketchView` trait with a concrete borrowed `ThetaSketchView<'a>`. Call `as_view()` when a view value is needed; Theta set operations continue to accept references to mutable and compact sketches directly. +* Change `ThetaSketch` and `CompactThetaSketch` iterators to yield `ThetaEntry` instead of raw `u64` hashes. Call `ThetaEntry::hash` to access a retained hash. +* Replace `ThetaIntersection::new(seed)` and `new_with_default_seed()` with `with_seed(seed)` and `Default::default()`, respectively. Replace `result()` and `result_with_ordered(ordered)` with `to_sketch(ordered)`. +* Make `CountMinValue` and `UnsignedCountMinValue` marker traits. Their previously exposed numeric constants and helper methods have been removed; use the corresponding primitive integer operations and conversions directly. ### New features +* Add Tuple sketches behind the `tuple` feature, including custom summary update and combination policies, serialization, compact sketches, union, intersection, A-not-B, Jaccard similarity, and exact sketch-state equality checks that ignore summary values. +* Add Theta union, A-not-B, Jaccard similarity, and exact sketch-state equality checks. * `FrequentItemsSketch` now supports borrowed-key updates via `update_ref` and `update_with_count_ref`, allowing sketches such as `FrequentItemsSketch` to update from `&str` without allocating on existing-key hits. Frequency queries also accept borrowed key forms matching `Borrow`. * `FrequentItemsSketch` no longer requires item types to implement `Clone` for core updates, queries, and serialization. Custom `FrequentItemValue` implementations can now be non-`Clone`; APIs that return or merge owned items still require `Clone`. -* `CountMinSketch` and `FrequentItemsSketch` now expose `estimated_size()`, reporting the in-memory footprint of the sketch in bytes, following the other sketches. -* The stateful set operations `HllUnion`, `CpcUnion`, `ThetaUnion`, `ThetaIntersection`, `TupleUnion`, and `TupleIntersection` now expose `estimated_size()`, reporting the in-memory footprint of the operator's internal state in bytes. +* Add `estimated_size()` to `BloomFilter`, `CountMinSketch`, `CpcSketch`, `FrequentItemsSketch`, `HllSketch`, `TDigestMut`, `TDigest`, mutable and compact Theta and Tuple sketches, and the stateful `HllUnion`, `CpcUnion`, `ThetaUnion`, `ThetaIntersection`, `TupleUnion`, and `TupleIntersection` operators. ### Bug fixes +* HLL serialization now emits the compact auxiliary-map flag required for Java to read HLL4 images and matches Java and C++ coupon ordering for compact Set images. * `FrequentItemsSketch::serialize` now writes the full 8-byte preamble for an empty sketch, matching the Java and C++ encoding. Empty sketches previously serialized to 6 bytes, which `FrequentItemsSketch::deserialize` rejected with an insufficient-data error. +* `FrequentItemsSketch` now preserves total weight and error state across serialization and merge when a purge removes every active item. +* T-Digest interpolation now keeps quantiles finite for extreme finite inputs. Deserialization rejects non-finite extrema, invalid centroid weights, and total-weight overflow as invalid data instead of allowing invalid state or panicking. +* Legacy Theta version 2 exact images with retained entries now deserialize as non-empty sketches and preserve their entries and exact estimates. +* Bloom filter deserialization now rejects inconsistent cached bit counts, preventing malformed images from hiding populated bits and violating the no-false-negative guarantee. * `CpcSketch` and `CpcWrapper` now classify out-of-range fields in serialized images as `InvalidData` rather than `InvalidArgument`. ## v0.3.0 (2026-05-18) diff --git a/README.md b/README.md index 18268c7..f8d1593 100644 --- a/README.md +++ b/README.md @@ -35,15 +35,74 @@ [actions-badge]: https://github.com/apache/datasketches-rust/actions/workflows/ci.yml/badge.svg [actions-url]: https://github.com/apache/datasketches-rust/actions/workflows/ci.yml -This is the core Rust component of the DataSketches library. It contains a subset of the sketching algorithms and can be accessed directly from user applications. +Apache DataSketches Rust provides stochastic streaming algorithms for answering queries over large data sets with compact, mergeable summaries. It is the core Rust component of Apache DataSketches and currently implements a subset of the algorithms available in the other language components. -Note that we have parallel core library components for Java, C++, Python, and Go implementations of many of the same sketch algorithms: +## Getting started -- [datasketches-java](https://github.com/apache/datasketches-java) -- [datasketches-cpp](https://github.com/apache/datasketches-cpp) -- [datasketches-python](https://github.com/apache/datasketches-python) -- [datasketches-go](https://github.com/apache/datasketches-go) +Sketch implementations are opt-in Cargo features; the crate enables none by default. For example, add the HyperLogLog implementation with: -Please visit the main [DataSketches website](https://datasketches.apache.org) for more information. +```shell +cargo add datasketches --features hll +``` -If you are interested in making contributions to this site, please see our [Community](https://datasketches.apache.org/docs/Community/) page for how to contact us. +Then build a sketch and query its distinct-count estimate: + +```rust +use datasketches::hll::HllSketch; +use datasketches::hll::HllType; + +let mut sketch = HllSketch::new(12, HllType::Hll8); +for user in ["alice", "bob", "alice", "carol"] { + sketch.update(user); +} + +assert!(sketch.estimate() >= 3.0); +``` + +Enable multiple algorithms by listing their features together, such as `features = ["hll", "theta"]` in `Cargo.toml`. + +## Available sketches + +| Feature | Main types | Use case | +| --- | --- | --- | +| `bloom` | `BloomFilter` | Space-efficient probabilistic set membership with a configurable false-positive rate. | +| `countmin` | `CountMinSketch` | Approximate point-frequency queries over a stream. | +| `cpc` | `CpcSketch`, `CpcUnion`, `CpcWrapper` | Highly compact distinct-count estimation and unions. | +| `frequencies` | `FrequentItemsSketch` | Heavy-hitter discovery with upper and lower frequency bounds. | +| `hll` | `HllSketch`, `HllUnion` | Fast distinct-count estimation and unions. | +| `tdigest` | `TDigestMut`, `TDigest` | Quantile and rank estimation, with high accuracy near distribution tails. | +| `theta` | `ThetaSketch` and set operations | Distinct counts, set expressions, and Jaccard similarity. | +| `tuple` | `TupleSketch` and set operations | Theta-style keys with user-defined summaries attached to retained entries. | + +See the [API documentation](https://docs.rs/datasketches) for configuration, accuracy guarantees, serialization, and examples for each algorithm. + +## Compatibility + +The minimum supported Rust version is 1.86.0. The crate currently supports little-endian targets only. + +Supported serialization formats are tested with fixtures produced by Apache DataSketches Java, C++, and Go through the [DataSketches TCK](https://github.com/apache/datasketches-tck). When values must hash identically across language implementations, use the compatibility wrappers in `hash::value`. + +See the [changelog](CHANGELOG.md) for release notes and migration guidance. + +## Other language implementations + +Apache DataSketches also provides core library components for other languages: + +- [Java](https://github.com/apache/datasketches-java) +- [C++](https://github.com/apache/datasketches-cpp) +- [Python](https://github.com/apache/datasketches-python) +- [Go](https://github.com/apache/datasketches-go) + +Visit the [Apache DataSketches website](https://datasketches.apache.org) for algorithm documentation, research background, and project-wide resources. + +## Community and contributing + +Questions, bug reports, and feature requests are welcome through [GitHub issues](https://github.com/apache/datasketches-rust/issues) and [GitHub discussions](https://github.com/apache/datasketches-rust/discussions). The [Apache DataSketches community page](https://datasketches.apache.org/docs/Community/) lists the public mailing lists and other ways to participate. + +See [CONTRIBUTING.md](CONTRIBUTING.md) to build, test, and contribute to the Rust component. All project participation is governed by the [Apache Software Foundation Code of Conduct](https://www.apache.org/foundation/policies/conduct.html). + +To report a security vulnerability, follow the [ASF security reporting process](https://www.apache.org/security/) instead of opening a public issue. + +## License + +Licensed under the [Apache License, Version 2.0](LICENSE).