feat: add table_shard_size_bytes to control the zarr shard size of tables

[Tomatokeftes]

Closes #1178.

Adds a keyword-only table_shard_size_bytes: int | None to SpatialData.write and
SpatialData.write_element, forwarded to write_table as shard_size_bytes. It is a target size in
bytes of uncompressed data for a single zarr shard of every array inside a table group.

sdata.write("data.zarr", table_shard_size_bytes=128 * 1024**2)
sdata.write_element("table", table_shard_size_bytes=8 * 1024**2)  # per table

Why a byte budget and not a chunks/shards tuple

This is deliberately not symmetric with #1106. A table is a heterogeneous tree of zarr arrays of
mixed rank, length and dtype that all receive one shared dataset_kwargs from anndata, so no single
tuple can be honoured by all of them. Measured on anndata 0.12.16 and zarr 3.2.1 with an ordinary
table:

• a 2-D chunks raises on obs/_index
• a 1-D chunks raises on 2-D obsm
• any shards tuple raises on the uns scalars
• shards without chunks raises on divisibility
• chunks=<int> broadcasts, but shards=<int> raises TypeError

A table_write_kwargs mirroring raster_write_kwargs would therefore ship an API whose documented
happy path cannot execute. A scalar budget avoids that: zarr derives shard = chunk * n per array, so
shard % chunk == 0 and shard <= array hold by construction at every rank, length and dtype.

I offered raster_shard_size_bytes as a symmetric form on the issue and have since withdrawn it. The
mechanism does not carry over: array.target_shard_size_bytes is read only when zarr is asked for an
automatic shard shape, and nothing injects shards="auto" on the raster side (shards does not appear
in _io/io_raster.py), so the same construction there would be a silently inert argument. A byte budget
is also the wrong shape for raster, where rank is uniform and storage_options already carries an
explicit per-level chunks. #1106's raster_write_kwargs looks like the right form for that side, so
this PR stays table only.

How it is delivered

Nothing is passed into dataset_kwargs. Two process globals are scoped around the existing anndata
call, for one table's write:

• zarr.config["array.target_shard_size_bytes"]
• anndata.settings.override(zarr_write_format=3, auto_shard_zarr_v3=True)

anndata then injects shards="auto" itself, only at the four writers where that is safe, and yields
to the caller-set budget instead of installing its own 1 GB default. The existing table.write_zarr(...)
and write_adata(group, name, table) calls are unchanged, and the #1183 re-fetch is outside the
scoped block, so the encoding attributes are untouched.

To be clear about what this does and does not do: it narrows the global, it does not remove it. Today
a downstream writer has to hold zarr.config open across a whole sdata.write; after this it is
scoped to one element and restored on exit, including on exceptions. It is still a process global
underneath.

Both write branches are wrapped, so the semantics are uniform across the supported anndata range with
no version-conditional code.

Two things worth flagging

shards must never reach dataset_kwargs. zarr's _guess_num_chunks_per_axis_shard does not
terminate on a rank-0 array while array.target_shard_size_bytes is set, and every SpatialData table
carries rank-0 string scalars in uns/spatialdata_attrs. It is an unbounded pure-Python loop, not an
error, so it would hang the write rather than fail it. Filed upstream as
zarr-developers/zarr-python#4304 and fixed by zarr-developers/zarr-python#4305, merged 2026-09-02
but not yet in a release: the loop is still present in 3.3.0 and in every version at or above the
3.1.6 floor this PR gates on. Nothing here can reach it, and there is a fast test asserting that. Today the rank-0 arrays are kept away from that code path by two independent anndata mechanisms: write_scalar_zarr and write_null_zarr never call zarr_v3_sharding at all, and @zero_dim_array_as_scalar re-dispatches 0-d ndarrays before write_basic's sharding is reached.

zarr_write_format has to be overridden alongside the sharding setting. AnnData.write_zarr reopens
the group with mode="w" and zarr_format=settings.zarr_write_format, destroying and recreating the
group spatialdata just made; with that setting left at 2, the argument would be silently inert
(measured: table group format 3 before, 2 after, X/data shards None, no error and no warning).

Validation

All errors are TableWriteOptionsError, a new ValueError subclass re-exported from the top level.
All four are raised up front in write and write_element, before anything reaches disk, because
_write_element creates the element group and write writes every preceding element before the
table is reached.

• not a positive value (<= 0); the type itself is not checked at runtime
• zarr < 3.1.6: 3.1.4 added array.target_shard_size_bytes, but 3.1.4 and 3.1.5 still size the inner
  chunk with max_bytes=1024 where 1 MiB was intended (fixed by fix: auto-chunking when auto-sharding 1MiB number zarr-developers/zarr-python#3603),
  which would put roughly 130k inner chunks in a 128 MiB shard
• an anndata without zarr v3 auto-sharding support
• a zarr v2 table format, where sharding does not exist

The zarr and anndata gates are runtime checks, so zarr>=3.0.0 and anndata>=0.9.1 are unchanged and
no CI leg gains a dependency.

Setting the argument forces auto_shard_zarr_v3=True for the duration of each table write, so it
overrides an explicit False; there is no value that turns sharding off. The budget is a target, not
a bound: below the automatically chosen inner chunk it degenerates to one chunk per shard. Both are
documented.

Tests

tests/io/test_readwrite.py, on a purpose-built 4000 x 2000 CSR table (the shipped _get_table is
8 kB and cannot differentiate any budget):

• on-disk geometry for two budgets: shards present, shards % chunks == 0, and the smaller budget
  produces a strictly smaller shard
• every rank-0 array in the written table group is unsharded, for single-string and list region
• shards never reaches anndata, on both write branches (a fast guard, since the failure mode is a
  hang)
• the default write is byte-identical to a write with the argument absent
• both process globals are restored after a normal write and after one that raises
• rejected on a zarr v2 table format, with the store left uncreated
• invalid values rejected on write and write_element, with nothing written
• the Tables written on main lose the anndata encoding-type/encoding-version attributes #1183 encoding-metadata guard re-run with a budget set

Release notes

Added table_shard_size_bytes to SpatialData.write and SpatialData.write_element, to set a target
uncompressed size in bytes for the zarr shards of table arrays.

[codecov]

Codecov Report

❌ Patch coverage is 92.50000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.89%. Comparing base (ccf1ea0) to head (bbffcac).

Files with missing lines	Patch %	Lines
src/spatialdata/_io/_utils.py	92.30%	2 Missing ⚠️
src/spatialdata/_io/io_table.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1199   +/-   ##
=======================================
  Coverage   91.89%   91.89%           
=======================================
  Files          53       53           
  Lines        7942     7974   +32     
=======================================
+ Hits         7298     7328   +30     
- Misses        644      646    +2     

Files with missing lines	Coverage Δ
src/spatialdata/__init__.py	95.65% <ø> (ø)
src/spatialdata/_core/spatialdata.py	93.88% <100.00%> (+0.01%)	⬆️
src/spatialdata/_io/exceptions.py	63.63% <100.00%> (+3.63%)	⬆️
src/spatialdata/_io/io_table.py	89.83% <88.88%> (+0.35%)	⬆️
src/spatialdata/_io/_utils.py	87.18% <92.30%> (+0.52%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Tomaz-Vieira]

isinstance(x, bool) and not isinstance(x, int) are redundant. Plus, I don't think we should be checking the types of non-union arguments anyway =)

[Tomatokeftes]

Alright, sounds good, I shall drop both. I will keep only the <=0 check.

[Tomatokeftes]

Dropped in c74ccee, and removing it turned up something worth recording.

The check was load-bearing in a way I had not noticed. anndata honours array.target_shard_size_bytes only when it reads back as an int, because of the isinstance check in zarr_v3_sharding. A float fails it, so anndata installs its own 1 GB default and the user's number is discarded. Measured on the test fixture:

int   2*1024**2   -> shards=(250000,)   the requested 2 MiB
float 2.0*1024**2 -> shards=(400000,)   whole array, budget dropped
1e8               -> shards=(400000,)   same

Same number, different behaviour depending on its type, with no error and no warning. 1e8 is a natural way to write a budget, so this is easy to hit.

bbffcac normalizes rather than checks: budget = int(shard_size_bytes) inside _table_shard_budget. That is not a type check, nothing is raised and the signature stays honest, so I believe it sits fine with your point. There is a test asserting a float budget produces the same on-disk geometry as the equivalent int, and I confirmed it fails without the coercion.

[Tomaz-Vieira]

I know we're trying to be nice here by catering to many versions of anndata (and zarr!) but I really dislike that we essentially lie to our users on the function signature, only to immediately disappoint by throwing an exception if the versions of zarr and/or anndata aren't what we need. We also completely defeat the type checker's ability to tell if the arguments are good or not.

One way around it would be to name those parameters as something like table_shard_size_bytes_hint (note the "hint" at the end); This makes it clear that they may or may not apply and we can just do nothing if that feature isn't supported.

Alternatively, we could create the type TableShardBudget, with a method like TableShardBudget.try_create(...), which is clearly visibly fallible, and would go through the validation logic in this function. This way if a client fails to get a TableShardBudget, then they can react accordingly (and locally to their code!), and all functions that use the budget don't have to re-validate. And you could also make the TableShardBudget be itself the context manager.

Maybe there is a way to have different signatures depending on what dependencies we have, but that would have strange impacts in our versioning scheme, so I'm skeptical that this could work.

Curious to see what other people think

[Tomatokeftes]

I see the problem, and I think there might be a third option to bump the floors. This should remove rather than rename around it.

The two version checks only because pyproject.toml still says anndata>=0.9.1 and zarr.=3.0.0. The pieces this argument needs are older than they look:

• anndata.settings.auto_shard_zarr_v3 exists since anndata 0.12.5 (anndata#2167)
• the byte budget via zarr.config exists since anndata 0.12.14 (anndata#2427)
• zarr 3.1.6 is the first release that sizes the inner chunk correctly (zarr-python#3603)

The test matrix here already only runs anndata 0.12 and 0.13, and dask is pinned at >=2026.3.0, so anndata>=0.12.14 and zarr>=3.1.6 would not be unusual. With those floors both version checks are deleted and the signature is honest on every supported install.

What remains is for zarr v2 tabe format, and I suggest we keep raising there, since it is not a dependency proble, but a user asking for a format without sharding, and it already carries a deprecation warning. It also raises at the top of write() before a single element is written, so nobodoy ends up with a half-written store.

On the two alternatives: I prefer not to do _hint, as a silently ignored budgest is exactly the failure that motivate the issue, a table landing as a few hundred thousand files with nobody told. If you prefer not to touch the pins, I can do the TableShardBudget object instead. I will then validate in __init__ and raise rather than a try_create that returns None, but that is a detail. Tell me which and I will push it.

[Tomatokeftes]

One caveat against my own suggestion, since it changes how much the floor bump actually buys.

I checked the versions against the tags rather than trusting the PR numbers: auto_shard_zarr_v3 is absent in anndata 0.12.4 and present in 0.12.5; target_shard_size_bytes is absent in 0.12.13 and present in 0.12.14; and the has_auto_shard_size yield-to-caller branch is intact at 0.12.14. So the floors themselves are right.

But the matrix pins anndata>=0.12,<0.13, which resolves to the newest 0.12.x, 0.12.19 today. An anndata>=0.12.14 floor would therefore never actually be exercised by CI. It would be a supported-version claim we do not test. That is true of whatever floor we pick rather than an argument against this one, and it does not change my preference, but you should weigh it now rather than hear it from me later.

Still happy to do TableShardBudget instead if you would rather not move the pins. Either way I would keep the zarr v2 case raising, since that is a user asking for a format without sharding rather than a dependency problem.

[Tomatokeftes]

Small correction to my own caveat: the anndata>=0.12,<0.13 pin is in the hatch test-anndata-pandas matrix in pyproject.toml, and no workflow runs that matrix. The GitHub legs do uv sync --group=test, so every one of them resolves the newest anndata, 0.13.3.post0 today, and the prerelease leg installs git main. So anndata 0.12 is not exercised by CI at any version, and a floor bump neither adds nor removes coverage. Same conclusion, stated more precisely.

[Tomaz-Vieira]

I think this docstring could be a bit more succint; I find it a bit hard to understand in the context of this PR, and would find it even harder when browsing the code out of context.

[Tomatokeftes]

Fair. I will cut it to what the reader needs, and one line with the link to zarr-python#4304 explaining. Rest moves to PR description.

[Tomaz-Vieira]

I would just like to be absolutely sure that this is the only way to do this. Temporarily setting a global variable is a very dangerous design, even with the context managers (e.g.: how do we even know we're not already inside a context? what happens on multithreaded applications? etc), so if there is any way we could pass these arguments to the a function call, I'd much much much prefer that.

[Tomatokeftes]

I checked this before, and here is the short version: for a byte budget it is the only way I could find, and it is the way anndata itself already uses.

1. zarr has no per-call budget. create_array takes no such parameter. The only place the number is read is zarr.config.get("array.target_shard_size_bytes") in zarr/core/chunk_grids.py (line 916 on 3.3.0). This is global by zarr's design as far as I know.

2. anndata already sets it globally on every default write. zarr_v3_sharding in anndata/_io/specs/methods.py wraps every array creation in zarr.config.set({"array.target_shard_size_bytes": 1_000_000_000}) whenever nothing is set (lines 139-143 on 0.13.3). So the context manager here does not add a new mechanism. It replaces anndata's hard-coded 1 GB with the user's number for the duration of one table and then hands back whatever was there before.

3. Nesting is safe. donfig's ConfigSet records the previous values and restores them on exit, and anndata.settings.override does the same. If a caller is already inside a zarr.config.set, the innermost wins for one table and the outer value comes back afterwards. I think an explicit argument beats ambient config, so that is the behaviour I would want anyway.

4. Threads might be a problem. zarr.config is one process-wide dict with a lock around mutation, not thread-local storage, so a table written on another thread during this scope would see this budget. That problem also exists today on anndata's default path in the same shape. This PR does not widen it.

The per-call alternative is an explicit shards tuple in dataset_kwargs, and that does not work for a table. anndata broadcasts one dataset_kwargs to every array of the AnnData, the ranks differ, and a shards entry reaching the 0-d scalars in uns makes zarr hang (zarr-python#4304, fix in zarr-python#4305).

The real fix for your concern is an argument on anndata's side, something like a shard budget on write_zarr / write_elem. It does not exist yet, but I can open that issue on anndata. If it lands, the signature here stays as it is and only the body of _table_shard_budget would change.