A rectilinear chunk spec with a short trailing chunk is silently normalized to a regular grid, changing resize semantics

[thomas-maschler]

Zarr version

v3.3.0

Numcodecs version

v0.16.5

Python Version

3.12.13

Operating System

Mac

Installation

Using Pixi into virtual environment

Description

3.3.0 picks between RegularChunkGridMetadata and RectilinearChunkGridMetadata from the values of the chunk edges rather than the form of the request. Since is_regular_1d counts "all chunks equal, last one smaller" as regular, an explicitly rectilinear spec whose edges happen to be uniform-plus-a-short-tail is stored as a regular grid.

For the array as created the two are equivalent. They diverge under resize: a FixedDimension grows by extending the uniform pattern, a VaryingDimension by appending an edge. The caller gets a grid that behaves differently from the one they asked for, with no warning and nothing in the metadata recording the substitution.

Why this matters

The use case is an append-only time series written in two regimes: a coarse backfill that lays down large chunks, then an incremental forward-fill that appends small ones. Rectilinear chunks make each appended window its own chunk, so the forward-fill never rewrites an edge chunk — that invariant is the entire reason for using them.

The collapse makes the invariant contingent on whether the initial backfill plan happens to look uniform. Edges of (168,) * 13 + (24,) become regular and every later append straddles a boundary; genuinely varied edges stay rectilinear and work. Same code, same intent, different outcome. With concurrent writers the shared-chunk rewrite is also a commit conflict rather than a clean append.

Origin

Introduced by #3899. 3.2.1's resolve_chunks dispatched on input syntax (_is_rectilinear_chunks), so a nested sequence always yielded a rectilinear grid; create_chunk_grid_metadata now dispatches on is_regular_nd after normalization to ChunksTuple. This reads as a side effect of consolidating the two normalization paths rather than an intended change — it's absent from the PR's "User-visible behavior changes" table, from changes/3899.bugfix.md, and from the 3.3.0 release notes.

Possible resolutions

There's no way today to request a rectilinear grid independently of the edge values; init_array calls create_chunk_grid_metadata(outer_chunks) with no override.

1. Honor the requested kind — a nested-sequence chunks yields a rectilinear grid even when the edges are uniform. Restores 3.2.x behavior.
2. Accept explicit grid metadata (or a chunk_grid="rectilinear" hint) in create_array.
3. Add an explicit chunk-extension mode to resize/append, per Add append method to VaryingDimension for explicit chunk extension #3870, which already raises this exact question.

Filing as design feedback against #4025, before rectilinear chunks leave experimental.

Steps to reproduce

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues
import hashlib
import zarr
zarr.config.set({"array.rectilinear_chunks": True})
store: dict[str, object] = {}
a = zarr.create_array(
    store=store, shape=(24,), chunks=[[10, 10, 4]], dtype="float32", zarr_format=3
)
print("at create:   ", a.metadata.chunk_grid)
def touched(start: int, stop: int, val: int) -> list[str]:
    """Chunk keys created or modified by writing a[start:stop]."""
    def snap() -> dict[str, str]:
        return {
            k: hashlib.md5(bytes(v.to_bytes())).hexdigest()
            for k, v in store.items()
            if k.startswith("c/")
        }
    before = snap()
    a[start:stop] = val
    after = snap()
    return sorted(k for k in after if after[k] != before.get(k))
print("write [20:24]", touched(20, 24, 2))
a.resize((34,))  # append a fourth 10-wide chunk
print("after resize:", a.metadata.chunk_grid)
appended = touched(24, 34, 3)
print("write [24:34]", appended)
assert appended == ["c/3"], (
    f"expected the appended region to be exactly one new chunk, got {appended}"
)

On main:

at create: RegularChunkGridMetadata(chunk_shape=(10,))
write [20:24] ['c/2']
after resize: RegularChunkGridMetadata(chunk_shape=(10,))
write [24:34] ['c/2', 'c/3']
AssertionError: expected the appended region to be exactly one new chunk, got ['c/2', 'c/3']

Swapping the dependency to zarr==3.2.1 passes, and shows the grid the spec asked for:

at create: RectilinearChunkGridMetadata(chunk_shapes=((10, 10, 4),))
write [20:24] ['c/2']
after resize: RectilinearChunkGridMetadata(chunk_shapes=((10, 10, 4, 10),))
write [24:34] ['c/3']

Additional output

No response

[d-v-b]

Thanks for this report, and sorry for the breakage.

Honor the requested kind — a nested-sequence chunks yields a rectilinear grid even when the edges are uniform. Restores 3.2.x behavior.

this looks the best to me right now. longer term we will want something more explicit.

[d-v-b]

a simple idea, that would need buy-in from folks concerned about public API, would be top-level (i.e., importable from zarr) RegularChunks and RectilinearChunks classes that wrap / annotate tuples of ints as being part of one chunk grid or another. This is a bit in between asking users to remember that nested tuples are different from plain tuples on one hand and forcing users to declare the exact chunk grid class they want on the other.