feat: add accelerated IEEE CRC32 and CRC32C checksums - #27
Merged
Merged
Conversation
tisonkun
marked this pull request as ready for review
September 15, 2026 05:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in IEEE CRC32 and Castagnoli CRC32C for records, pages, and protocol payloads, with one-shot and incremental APIs, finalized-checksum resumption, segment combination, and
std::io::Writeintegration. The library remains dependency-free, allocation-free, andno_stdcompatible. Raise the workspace MSRV from Rust 1.85 to 1.89 so the library and every benchmark comparison share one compiler baseline.Accelerate both variants with AArch64 CRC/PMULL, fused CRC/SHA3 loops, and x86-64 PCLMULQDQ plus AVX2/AVX-512 VPCLMULQDQ. Keep a slicing-by-8 scalar fallback and differential tests for every available backend, unaligned inputs, streaming boundaries, and large shift lengths.
Design Notes
The API model groups general-purpose hashing, error-detecting checksums, compatible hashing, and legacy digests by their intended use.
Each CRC state stores one
u32.digestpreserves the state, andfrom_digestaccepts a finalized checksum. The two polynomials have distinct names and types; these APIs consume bytes withoutHasheradapters. Combine functions take the right segment's exactu64length.stdenables runtime CPU detection;no_stduses compile-time target features. Folding schedules draw on corsix/fast-crc32 and crc-fast; their MIT notices are collected inLICENSE, which is included in the published crate. Kernel loads accept unaligned input and remain inside the slice.The ecosystem benchmarks compare stock
crc3.4.0 (Table<1>andTable<16>),crc32fast1.5.1,crc32c0.6.8, andcrc-fast1.10.0. All comparisons are enabled by default, with no benchmark feature flags. Stable and Rust 1.89 CI both build every workspace target and run workspace tests. The suites cover short inputs, dispatch boundaries, one-byte offsets, and streaming chunks. CI compiles the benchmark targets without running performance measurements. Performance regressions are checked locally against the base revision with matching environments and sampling settings.1 MiB one-shot medians measured at
cd9e5babelow are Hashcrew / crc-fast, in microseconds, using Rust 1.98.0,std, default targets, and a reused buffer on one thread. Each value is the median of the four longer runs:Bulk throughput is competitive across these hosts. The full M1 matrix still has roughly 6–13% gaps at some 16 KiB + 1 / 64 KiB cases and sub-nanosecond gaps at a few tiny inputs; this draft does not establish pointwise dominance. The shared M1 runner also shows substantial run-to-run variation. The AVX-512 path is compiled and has conditional differential coverage; these measurements exercise AVX2 and AArch64 hardware.
Measurements and CPU/toolchain details at
cd9e5baare preserved as artifacts. CI validation: Linux/macOS/Windows on Rust 1.89 and stable, feature/target checks, lint, and Miri. Native AArch64 Linux container tests and local macOS measurements also passed. The default-enabled benchmark suites passed a localcargo x bench -- --testsmoke check, and package inspection verifies the consolidatedLICENSE.