Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@

(release_0.17.0)=

* Add fixed BitRound compatibility data and test decoding with a codec restored
from its JSON configuration through the registry, contributing to {issue}`348`.

* Allow building against a system Zlib. Note, this is only required if blosc is still
vendored, and a vendored Zlib is no longer built if blosc is not.
By {user}`Elliott Sales de Andrade <QuLogic>`, {issue}`860`
Expand Down
54 changes: 54 additions & 0 deletions fixture/bitround/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# BitRound compatibility data

These fixtures establish a compatibility baseline from Numcodecs commit
`1f83681cf310ac9aac8041ed6ff4a5dcc42a7aad`
(`0.17.1.dev1+g1f83681cf`), generated with Python 3.13.5 and NumPy 2.5.4 on
little-endian Windows x86_64. They are not fixtures from an earlier release.

The input is `[1.15625, -1.15625, 2.6875, -2.6875]`, shape `(4,)`.
`encoded.00.dat` stores the encoded `<f4` input as `<i4`; `encoded.01.dat`
stores the encoded `<f8` input as `<i8`. Each `config.json` is the corresponding
codec's JSON configuration. The raw files follow the other codec fixtures'
`codec.NN/config.json` and `encoded.NN.dat` naming convention.

| Directory | keepbits | Expected decoded values (both float widths) |
| --- | --- | --- |
| `codec.00` | 2 | `[1.25, -1.25, 2.5, -2.5]` |
| `codec.01` | 3 | `[1.125, -1.125, 2.75, -2.75]` |

The expectations can be checked without Numcodecs. The positive inputs are
`37/32` and `43/16`, both exactly representable in float32 and float64. Keeping
`k` fraction bits gives spacing `2**-k` in `[1, 2)` and `2**(1-k)` in `[2, 4)`.
Nearest rounding therefore gives `5/4, 5/2` for `k=2` and `9/8, 11/4` for `k=3`;
the negative inputs give their negatives. None of these cases is a tie.
All expected values are also exactly representable, so no tolerance is needed.

The files were generated once at the stated baseline with:

```python
import json
from pathlib import Path

import numpy as np
from numcodecs import BitRound

data = [1.15625, -1.15625, 2.6875, -2.6875]
for j, keepbits in enumerate([2, 3]):
codec = BitRound(keepbits=keepbits)
directory = Path('fixture/bitround') / f'codec.{j:02d}'
directory.mkdir(parents=True, exist_ok=True)
(directory / 'config.json').write_text(
json.dumps(codec.get_config(), sort_keys=True, indent=4) + '\n'
)
for i, dtype in enumerate(['<f4', '<f8']):
encoded = codec.encode(np.array(data, dtype=dtype))
(directory / f'encoded.{i:02d}.dat').write_bytes(encoded.tobytes())
```

Generation was independently checked against exact rational rounding and
`struct.pack('<4f', *expected)` / `struct.pack('<4d', *expected)` for the values
above. The test only reads these files, round-trips the saved configuration
through JSON, and decodes with the registry-returned instance. It supplies the
integer dtype explicitly because BitRound does not store dtype or shape metadata.
The test checks reading this fixed data; it does not require future encoders to
produce identical bytes for arbitrary input or freeze all rounding behavior.
4 changes: 4 additions & 0 deletions fixture/bitround/codec.00/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"id": "bitround",
"keepbits": 2
}
Binary file added fixture/bitround/codec.00/encoded.00.dat
Binary file not shown.
Binary file added fixture/bitround/codec.00/encoded.01.dat
Binary file not shown.
4 changes: 4 additions & 0 deletions fixture/bitround/codec.01/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"id": "bitround",
"keepbits": 3
}
Binary file added fixture/bitround/codec.01/encoded.00.dat
Binary file not shown.
Binary file added fixture/bitround/codec.01/encoded.01.dat
Binary file not shown.
37 changes: 37 additions & 0 deletions tests/test_bitround.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
import json
from pathlib import Path

import numpy as np
import pytest

from numcodecs.bitround import BitRound, max_bits
from numcodecs.registry import get_codec

# adapted from https://github.com/milankl/BitInformation.jl/blob/main/test/round_nearest.jl

Expand Down Expand Up @@ -79,3 +83,36 @@ def test_errors():
BitRound(keepbits=10).encode(np.array([0]))
with pytest.raises(ValueError):
BitRound(-1)


@pytest.mark.parametrize(('array_index', 'dtype'), [(0, '<f4'), (1, '<f8')])
@pytest.mark.parametrize(
('codec_index', 'keepbits', 'expected'),
[
(0, 2, [1.25, -1.25, 2.5, -2.5]),
(1, 3, [1.125, -1.125, 2.75, -2.75]),
],
)
def test_backwards_compatibility(array_index, dtype, codec_index, keepbits, expected):
# Read fixed data only: the common helper can generate missing fixtures and
# decodes with the original codec rather than the registry-restored instance.
codec_dir = (
Path(__file__).resolve().parents[1] / 'fixture' / 'bitround' / f'codec.{codec_index:02d}'
)
config = json.loads((codec_dir / 'config.json').read_text())
codec = get_codec(json.loads(json.dumps(config)))
assert isinstance(codec, BitRound)
# decode does not use keepbits, so check configuration separately from data.
assert codec.keepbits == keepbits

# Raw BitRound data has no dtype or shape header. Supply the stored signed
# integer dtype, including byte order, and check the resulting flat array.
encoded = np.frombuffer(
(codec_dir / f'encoded.{array_index:02d}.dat').read_bytes(),
dtype=dtype.replace('f', 'i'),
)
decoded = codec.decode(encoded)
expected = np.array(expected, dtype=dtype)
assert decoded.dtype == expected.dtype
assert decoded.shape == expected.shape
np.testing.assert_array_equal(decoded, expected)
Loading