Skip to content

feat: add accelerated IEEE CRC32 and CRC32C checksums - #27

Merged
tisonkun merged 9 commits into
mainfrom
codex/crc-api
Sep 15, 2026
Merged

tisonkun merged 9 commits into
mainfrom
codex/crc-api

Conversation

@tisonkun

@tisonkun tisonkun commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Add opt-in IEEE CRC32 and Castagnoli CRC32C for records, pages, and protocol payloads, with one-shot and incremental APIs, finalized-checksum resumption, segment combination, and std::io::Write integration. The library remains dependency-free, allocation-free, and no_std compatible. Raise the workspace MSRV from Rust 1.85 to 1.89 so the library and every benchmark comparison share one compiler baseline.

Accelerate both variants with AArch64 CRC/PMULL, fused CRC/SHA3 loops, and x86-64 PCLMULQDQ plus AVX2/AVX-512 VPCLMULQDQ. Keep a slicing-by-8 scalar fallback and differential tests for every available backend, unaligned inputs, streaming boundaries, and large shift lengths.

Design Notes

The API model groups general-purpose hashing, error-detecting checksums, compatible hashing, and legacy digests by their intended use.

Each CRC state stores one u32. digest preserves the state, and from_digest accepts a finalized checksum. The two polynomials have distinct names and types; these APIs consume bytes without Hasher adapters. Combine functions take the right segment's exact u64 length.

std enables runtime CPU detection; no_std uses compile-time target features. Folding schedules draw on corsix/fast-crc32 and crc-fast; their MIT notices are collected in LICENSE, which is included in the published crate. Kernel loads accept unaligned input and remain inside the slice.

The ecosystem benchmarks compare stock crc 3.4.0 (Table<1> and Table<16>), crc32fast 1.5.1, crc32c 0.6.8, and crc-fast 1.10.0. All comparisons are enabled by default, with no benchmark feature flags. Stable and Rust 1.89 CI both build every workspace target and run workspace tests. The suites cover short inputs, dispatch boundaries, one-byte offsets, and streaming chunks. CI compiles the benchmark targets without running performance measurements. Performance regressions are checked locally against the base revision with matching environments and sampling settings.

1 MiB one-shot medians measured at cd9e5ba below are Hashcrew / crc-fast, in microseconds, using Rust 1.98.0, std, default targets, and a reused buffer on one thread. Each value is the median of the four longer runs:

Host CRC32C IEEE CRC32
Linux, AMD EPYC 7763 41.19 / 55.99 41.17 / 82.35
macOS, Apple M1 runner 13.94 / 13.81 13.96 / 13.67
macOS, local Apple M4 Max 9.67 / 10.32 9.67 / 10.38

Bulk throughput is competitive across these hosts. The full M1 matrix still has roughly 6–13% gaps at some 16 KiB + 1 / 64 KiB cases and sub-nanosecond gaps at a few tiny inputs; this draft does not establish pointwise dominance. The shared M1 runner also shows substantial run-to-run variation. The AVX-512 path is compiled and has conditional differential coverage; these measurements exercise AVX2 and AArch64 hardware.

Measurements and CPU/toolchain details at cd9e5ba are preserved as artifacts. CI validation: Linux/macOS/Windows on Rust 1.89 and stable, feature/target checks, lint, and Miri. Native AArch64 Linux container tests and local macOS measurements also passed. The default-enabled benchmark suites passed a local cargo x bench -- --test smoke check, and package inspection verifies the consolidated LICENSE.

@tisonkun tisonkun closed this Sep 13, 2026
@tisonkun
tisonkun deleted the codex/crc-api branch September 13, 2026 13:09
@tisonkun tisonkun reopened this Sep 14, 2026
@tisonkun tisonkun changed the title feat: add IEEE CRC32 and CRC32C checksums feat: add accelerated IEEE CRC32 and CRC32C checksums Sep 14, 2026
@tisonkun
tisonkun marked this pull request as ready for review September 15, 2026 05:49
@tisonkun
tisonkun merged commit af83b37 into main Sep 15, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant