[mkitti]

Summary

Adds c-blosc2 as an opt-in build backend for the Blosc codec (-Dblosc_backend=blosc2). The default stays c-blosc 1.x, so nothing changes for existing builds.

Chunks written through this backend are in the Blosc1 format and decodable by C-Blosc 1.x. This is the point of the PR and it is not true of stock c-blosc2. Related: #864, #865, #854, #870, #463.

Forward compatibility

c-blosc2's README says its buffers are not readable by C-Blosc 1.x. I checked whether that still holds through the blosc1_* API (c-blosc 1.21.6 vs c-blosc2 3825863, 735 cases: blosclz/lz4/lz4hc/zlib/zstd × no/byte/bit shuffle × typesizes 1–16 × sizes 1 B–100 KB, including sizes that are not a multiple of the typesize):

writer → reader	decoded
stock c-blosc2 (blosc1_compress, with or without BLOSC_BLOSC1_COMPAT, or blosc2_compress_ctx) → C-Blosc 1.x	0
C-Blosc 1.x → stock c-blosc2	all except 56 bit-shuffled buffers with leftover bytes (a reader bug)
patched c-blosc2 → C-Blosc 1.x	all
C-Blosc 1.x → patched c-blosc2	all

The gap is in the writer: the format version byte (5 vs 2), the bitshuffle layout for blocks whose element count is not a multiple of 8, and the block-split threshold. packages/c-blosc2-blosc1-compat/patches/ is a four-patch series, one concern each:

Patch	What	Upstream
0001	reader: copy the leftover bytes of bit-shuffled version 2 blocks	bug, issue #812, PR #813
0002	opt-in flag to write the 16-byte Blosc1 header with format version 2	not submitted
0003	version 2 bitshuffle rule when writing	not submitted
0004	C-Blosc 1.x block-split rule when writing	not submitted

The flag is off by default and there is no ABI change. Output decodes with C-Blosc 1.x but is not byte-identical to what it writes. Only plain shuffle/bitshuffle with blosclz, lz4, lz4hc, zlib and zstd are supported in this mode. I do not know whether upstream will accept 0002–0004, so the package is self-contained (no numcodecs imports) and can be split out or dropped. See its README and TODO.md.

What changed

• packages/c-blosc2-blosc1-compat/: meson build for c-blosc2 (upstream is CMake only), the patch series, a compatibility check (tests/run.py), and tools to keep the wrap in sync and to test the series against a newer c-blosc2.
• subprojects/: wraps for c-blosc2 (pinned commit, patched through packagefiles) and for zstd, lz4 and zlib from WrapDB.
• blosc_backend option. zstd, lz4 and zlib are resolved separately from Blosc, so the zstd and lz4 codecs share them. -Dblosc_zstd=disabled / -Dblosc_zlib=disabled leave them out of Blosc only.
• src/numcodecs/_blosc_shim.h gives both libraries one C API, which blosc.pyx uses. numcodecs.blosc.LIBRARY reports which one is in use.
• blosc.pyx also picks up the frame validation and empty-buffer handling from Fix empty Blosc buffer round-trips #854, the zero-itemsize fix from Use a typesize of 1 in Blosc for zero-itemsize buffers #870, and clamps a requested blocksize to the buffer size and a multiple of the typesize as C-Blosc 1.x does (c-blosc2 writes unreadable chunks otherwise).
• Tests: Blosc1-format output, the shuffle round-trip from perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864, Fix empty Blosc buffer round-trips #854's frame tests, Use a typesize of 1 in Blosc for zero-itemsize buffers #870's zero-itemsize test, new tests for the decompress errors and blocksize handling, and tests that the wrap copies, diff_files and UPSTREAM pin stay consistent. Compressors a build leaves out are skipped.
• .github/workflows/blosc2-compat.yaml: checks the wrap copies are in sync, runs the forward-compat check against a real C-Blosc 1.21.6, runs the whole test suite on the c-blosc2 backend (Linux x86_64 and arm64, macOS arm64, Windows, and free-threaded 3.14t), and weekly tries the series on upstream c-blosc2 main.

Performance

#864's NEON copy is not needed for this backend: c-blosc2 builds its own NEON shuffle on aarch64 (the shuffle_neon symbols are in the extension). Single thread, Apple Silicon, 1 MiB arrays, SHUFFLE, decode, c-blosc 1.x → c-blosc2: lz4 uint16 1.5 → 4.6 GB/s, lz4 float32 1.6 → 8.9, blosclz uint16 1.8 → 20.4, blosclz float64 2.3 → 7.1. Encode is mixed (lz4 uint16 725 vs 1180 MB/s, others comparable or faster); one machine, not explained.

Notes for reviewers

• CI on the default backend is unchanged and passes; the new job covers the c-blosc2 backend.
• c-blosc2 has no snappy, and the wrapped zstd needs a C++ compiler (as in build: replace c-blosc git submodule with meson wrap subprojects #865).
• This does not remove the c-blosc submodule or change CI for the default backend; that belongs with build: replace c-blosc git submodule with meson wrap subprojects #865. Expect a rebase between this PR, perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 and build: replace c-blosc git submodule with meson wrap subprojects #865: all touch the blosc part of src/numcodecs/meson.build and meson.options. perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 is still relevant for the default c-blosc 1.x backend.
• pyproject.toml's license-files still lists the c-blosc submodule's licenses, so a c-blosc2 wheel does not yet ship c-blosc2's own. build: replace c-blosc git submodule with meson wrap subprojects #865 reworks this.
• .pre-commit-config.yaml now excludes *.patch, because the whitespace hook would rewrite the patches' context lines.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (54fc274) to head (c603e68).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #879   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          914       914           
=========================================
  Hits           914       914           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mkitti]

Performance: c-blosc 1.x vs c-blosc2 backend

Single thread (numcodecs.blosc.use_threads = False), compress(array, cname, 5, SHUFFLE), MB/s of uncompressed data, best of 40 runs after one warm-up. Apple Silicon (aarch64, macOS), Python 3.13, release build. Both backends built from this branch with default options (-Dblosc_backend=blosc1 is c-blosc 1.21.7.dev; blosc2 is c-blosc2 3825863 with the patch series). Ratio is uncompressed / compressed size.

cname	data	ratio 1.x → c-blosc2	encode 1.x → c-blosc2	decode 1.x → c-blosc2
lz4	uint16, 1 MiB	2.2 → 2.5	1180 → 725	1541 → 4641
lz4	float32, 1 MiB	1.4 → 1.4	1782 → 2981	1614 → 8883
lz4	float64, 1 MiB	1.2 → 1.2	1821 → 3721	2667 → 7406
lz4	uint16, 16 MiB	2.2 → 2.5	1073 → 672	1544 → 4352
blosclz	uint16, 1 MiB	2.0 → 2.0	1964 → 1158	1849 → 20410
blosclz	float32, 1 MiB	1.4 → 1.3	1154 → 1248	1305 → 7840
blosclz	float64, 1 MiB	1.2 → 1.2	1121 → 1691	2340 → 7059
blosclz	uint16, 16 MiB	2.0 → 2.0	1930 → 1120	1868 → 20243

What this shows:

• Decode is 3x to 11x faster on c-blosc2. c-blosc 1.x has no NEON shuffle, so its dispatcher falls back to scalar byte shuffle, which is the problem perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 describes (about 2.5 GB/s here). c-blosc2 ships its own NEON shuffle (shuffle_neon / unshuffle_neon are in the built extension), so no extra NEON code is needed for this backend.
• Encode is mixed: faster for float32/float64 with lz4 and for blosclz float64, slower for uint16 (lz4 725 vs 1180 MB/s, blosclz 1158 vs 1964). I have not found why. A guess I have not tested is the block-split rule that the Blosc1 output mode forces; lz4 uint16 also compresses better (2.5 vs 2.2), which fits a different split.
• This is one machine, one run of the script, and my own synthetic data (cumulative sums, so that shuffle matters), so it is not comparable to the table in perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864, which used different data. x86_64 (SSE2/AVX2) has not been measured.

Script

import sys, time
import numpy as np
sys.path.insert(0, sys.argv[1])
import numcodecs.blosc as b
b.use_threads = False
print(b.LIBRARY, b.VERSION_STRING)
def mbs(f, n, reps=40):
    f(); best = 1e9
    for _ in range(reps):
        t = time.perf_counter(); f(); best = min(best, time.perf_counter() - t)
    return n / best / 1e6
rng = np.random.default_rng(0)
cases = [("uint16 1MiB", np.cumsum(rng.integers(0, 3, 2**19)).astype('u2')),
         ("float32 1MiB", np.cumsum(rng.normal(scale=1e-3, size=2**18)).astype('f4')),
         ("float64 1MiB", np.cumsum(rng.normal(scale=1e-3, size=2**17))),
         ("uint16 16MiB", np.cumsum(rng.integers(0, 3, 2**23)).astype('u2'))]
for cname in ("lz4", "blosclz"):
    for name, a in cases:
        raw = a; n = a.nbytes
        enc = b.compress(raw, cname.encode(), 5, b.SHUFFLE)
        e = mbs(lambda: b.compress(raw, cname.encode(), 5, b.SHUFFLE), n)
        d = mbs(lambda: b.decompress(enc), n)
        _, _, memcpyed = b._cbuffer_metainfo(enc)
        print(f"{cname:8s} {name:13s} ratio {n/len(enc):5.1f} memcpy={memcpyed} encode {e:7.0f}  decode {d:7.0f} MB/s")

Run it as python bench.py <directory containing the built numcodecs> for each backend.

Two traps I hit while writing it, in case someone repeats the measurement: passing bytes gives typesize 1, so the shuffle does nothing, and data that does not compress is stored raw (check blosc._cbuffer_metainfo(enc)[2], the memcpy flag), which makes decode look like 30+ GB/s.

[mkitti]

Follow-up to the performance comment above: the encode slowdown it shows for 2-byte data, and where it comes from.

What is slower. On the c-blosc2 backend, encoding 2-byte types (uint16) is about 25% to 45% slower than with c-blosc 1.x at the same codec, level and shuffle (lz4 725 vs 1137 MB/s, blosclz 1158 vs 1964 in the table above). 4- and 8-byte types are comparable or faster, and decode is much faster everywhere I measured. Against c-blosc 1.x with #864's NEON applied, c-blosc2 is slower on encode in most rows (see my comment on #864).

Cause: a different automatic block size, not slower code. In a C-level test (1 MiB, typesize 2, clevel 5, one thread), c-blosc 1.x stores 256 KiB blocks and c-blosc2 stores 128 KiB blocks:

	block size	lz4 ratio	lz4 encode	blosclz encode
c-blosc 1.x	256 KiB	2.19	~925 MB/s	2024 MB/s
c-blosc2 default	128 KiB	2.52	~735 MB/s	1169 MB/s
c-blosc2 forced to 256 KiB	256 KiB	2.19	1777 MB/s	2110 MB/s

At the same block size c-blosc2 produces the same ratio and is as fast or faster. The two libraries compute the default differently for splittable codecs: c-blosc 1.x uses min(128 KiB, 256 KiB) * typesize at clevel 5, and c-blosc2 (blosc_stune_next_blocksize in blosc/stune.c) uses a fixed 64 KiB for clevels 4 to 6, times the typesize. The smaller block gives a better ratio and, for typesize 2, a slower encode. It looks deliberate on their side, so I filed it as a question: Blosc/c-blosc2#814 (with a reproduction using python-blosc and python-blosc2).

Correction: I first guessed on #864 that the Blosc1 output mode's block-split rule was responsible. That was wrong. Stock c-blosc2 with its default Blosc2 output is equally slow, and the patched Blosc1 mode is within a few percent of it.

What this means for the PR.

• It does not affect correctness or the default backend; the opt-in c-blosc2 backend just makes a different speed/ratio choice for small typesizes at the default block size.
• Blosc(blocksize=...) already lets a user pick the block size explicitly; with blocksize=262144 on 2-byte data the c-blosc2 backend matches c-blosc 1.x's ratio and is faster.
• An option I have not implemented: for blocksize=0 the c-blosc2 backend could choose c-blosc 1.x's block size itself, so output matches c-blosc 1.x chunks more closely. That would give up c-blosc2's better ratio for small types, and it would need checking across typesizes, levels and buffer sizes, so I would rather decide it after upstream answers Documentation cleanup: fix typos, grammar, and naming consistency #814.

Caveats: one machine (Apple Silicon), one buffer size for the block-size experiment, synthetic data, no x86_64.

Drafted with Claude Code.

[d-v-b]

so did they avoid supporting the old blosc1 format header to encourage people to switch to blosc2?

[mkitti]

It did not seem to encourage us very much here, did it?

[d-v-b]

this is a big change so I'd like some eyes from @zarr-developers/python-core-devs.

my pov: this effort already revealed one bug in blosc2. I wonder if there are more? for me, a perf improvement isn't worth a moderate chance of data corruption / bugs. I personally am not qualified to evaluate the c code or the build backend changes so I would like to see some very comprehensive property-based tests that fuzz the new codec and ensure that the range of bytes payloads, with important edge cases, that we see in zarr come out byte-identical after this swap.

[martindurant]

Thanks for the ping to maintainers. I am surprised that there would be a push to implement this in C with a massive PR rather than call the python or C API - is the overhead really that crushing? This has always puzzled me about numcodecs specifically for blosc; at least the cython-on-Clib codecs are reasonable.
I don't know about the bug mentioned above or potential others, but I think for simplicity and ease of packaging/distribution, it would be better to call public APIs of installable packages.

Generally, the comparison with what blosc does internally versus zarr's sharding filter is super interesting for me. (yes, I realise here we are in numcodecs/zarr1 land) I feel like the two libraries are sort of doing the same thing and it may be better to (hope to) converge. blosc has done some interesting things with in-memory compressed arrays and operation fusion that zarr has not been interested in or left to other frameworks.

cc @FrancescAlted , might find this interesting.

[mkitti]

@martindurant The C code patches included in this pull request add a function which c-blosc2 nor its Python wrapping actually does by itself, which is writing write conpatible c-blosc (v1) frames that c-blosc (v1) can decode.

Forward compatibility has long been a blocker here inhibiting the migration to c-blosc2.

Other pull requests started to pull code from c-blosc2 for ARM Neon support, as an example, which I think is more unsustainable than just using c-blosc2 which already maintains such support.

Regardless of depedency and integration mechanism, continued dependence on c-blosc (v1) has become unsustainable, untenable, and irresponsible in any form.

[d-v-b]

the blosc v1 codec is baked in to the zarr v3 spec. so we are on the hook for supporting it.

[mkitti]

this is a big change

As currently written this pull request actually changes nothing by default. The default is to use the c-blosc (v1) "backend". It must be enabled by a compiler flag whichis disabled by default.

This pull request allows for the very testing you are requesting to begin.

[d-v-b]

This pull request allows for the very testing you are requesting to begin.

I think what I was saying is that, given the scale of the intended change (swapping out our vendored blosc1 for a patched blosc2), I would need to see those tests before I can think about supporting this effort.

[mkitti]

the blosc v1 codec is baked in to the zarr v3 spec. so we are on the hook for supporting it.

The blosc v1 codec is distinct from the c-blosc (v1) code base, which is seeing less and less developer attention. I have submitted issues and pull requests to the v1 code base in the past but decided it was no longer worthwhile to do about a year ago.

The patches included here show a path to supporting the v1 codec from the c-blosc2 code base. That said the lack of upstream support for writing the c-blosc v1 format suggest that we should begin deprecating writing with that codec.

[d-v-b]

I agree with @martindurant that the best outcome would be for us to just import all this stuff from a published package. @FrancescAlted would applying the patches here to blosc2 be in scope for the project? then we could just import blosc2 from pypi, which would be far simpler than what we have here.

[FrancescAlted]

so did they avoid supporting the old blosc1 format header to encourage people to switch to blosc2?

Just let me make clear that we are not supporting old blosc1 very actively mainly for two reasons:

1. All action is happening in blosc2, so there is not much point for us in keep maintaining blosc1 for performance improvements (fixing important bugs is a different thing, and we tried to keep doing this with our limited resources).
2. Contrarily to blosc2, blosc1 has not received any funding for more than 10 years now, strongly affecting our maintenance work.

Having said that, it is completely true that we encourage people to switch to blosc2. We really think that this is the way to go forward, and for me, adding support for blosc2 in zarr makes a lot of sense. In blosc2 we are actively supporting backward compatibility with blosc1, and I think this could facilitate a lot the transition to blosc2.

[FrancescAlted]

I agree with @martindurant that the best outcome would be for us to just import all this stuff from a published package. @FrancescAlted would applying the patches here to blosc2 be in scope for the project? then we could just import blosc2 from pypi, which would be far simpler than what we have here.

Absolutely. But IMO, providing a way to generate blosc1-compatible format inside blosc2 has little sense, because for me, focusing on the new format should bring more value for users (it is way more flexible) than keep trying using just the old one. Having said that, if you find that this is paramount for you, I am fine with that (within our finite resources).

[mkitti]

In blosc2 we are actively supporting backward compatibility with blosc1, and I think this could facilitate a lot the transition to blosc2.

Thank you for replying, @FrancescAlted . What is holding adoption of c-blosc2 back here is that c-blosc2 does not support writing c-blosc v1 compatible chunks. What I have done here is provide patches for c-blosc2 which allow for that - write chunks that c-blosc1 can read to enable the transition to the c-blosc2 code base. This seems more practical than trying to backport ARM Neon patches from c-blosc2 to c-blosc v1.

The question Davis has is "Would you accept patches that enable writing c-blosc v1 formatted chunks in c-blosc2?"

[FrancescAlted]

In blosc2 we are actively supporting backward compatibility with blosc1, and I think this could facilitate a lot the transition to blosc2.

Thank you for replying, @FrancescAlted . What is holding adoption of c-blosc2 back here is that c-blosc2 does not support writing c-blosc v1 compatible chunks. What I have done here is provide patches for c-blosc2 which allow for that - write chunks that c-blosc1 can read to enable the transition to the c-blosc2 code base. This seems more practical than trying to backport ARM Neon patches from c-blosc2 to c-blosc v1.

The question Davis has is "Would you accept patches that enable writing c-blosc v1 formatted chunks in c-blosc2?"

Yep. Although this does not make a lot of sense to me. See my reply to him above.