Skip to content

Add --use-work-zarr option for intermediate OME-Zarr rechunking before segmentation/vessel processing#181

Draft
Copilot wants to merge 3 commits into
mainfrom
copilot/add-intermediate-ome-zarr-workflow-rule
Draft

Add --use-work-zarr option for intermediate OME-Zarr rechunking before segmentation/vessel processing#181
Copilot wants to merge 3 commits into
mainfrom
copilot/add-intermediate-ome-zarr-workflow-rule

Conversation

Copilot AI commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

When input SPIM data has suboptimal chunking or lives on slow network/cloud storage, bias-field correction and vessel segmentation rules suffer from inefficient random reads. This adds an optional pre-processing step that rechunks the full multi-channel, multi-scale zarr to a fast local disk before those rules run.

New option

--use-work-zarr: requires --work_dir to point to a fast local disk with sufficient capacity. Raises ValueError at workflow init if --work_dir is unset.

spimquant /bids /output participant \
  --work_dir /fast/local/nvme \
  --use-work-zarr

Changes

snakebids.yml

  • New --use-work-zarr flag with help text noting the --work_dir requirement

Snakefile

  • Validates --work_dir is set when --use-work-zarr is active
  • Defines work_zarr_path (always .ome.zarr directory, never .ozx, rooted at work_dir)
  • Defines spim_path_for_seg_vessels — routes downstream rules to either the rechunked work zarr or the original SPIM path
  • Pre-populates spim_json_overrides for rechunked paths so JSON sidecar channel/orientation overrides remain effective
  • Includes new rules/rechunk.smk

rules/rechunk.smk (new)

  • rechunk_spim_to_work rule: one output per subject, contains all channels and all scale levels; output is directory(work_zarr_path)

scripts/rechunk_spim.py (new)

  • Traverses the zarr group tree recursively with dask rechunking; preserves all .zattrs metadata verbatim
  • Supports .ome.zarr directories, .ome.zarr.zip, and .ozx inputs; raises a clear error for .ims
  • Writes zarr format 2 (OME-NGFF v0.4/v0.5 compatible with ZarrNii)
  • Closes zip stores explicitly in a finally block

rules/segmentation.smk, rules/vessels.smk

  • gaussian_biasfield, n4_biasfield, and run_vesselfm input spim now references spim_path_for_seg_vessels instead of inputs["spim"].path directly

Copilot AI changed the title Add --use-work-zarr option for intermediate OME-Zarr rechunking Add --use-work-zarr option for intermediate OME-Zarr rechunking before segmentation/vessel processing Jun 23, 2026
Copilot AI requested a review from akhanf June 23, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants