Skip to content

fix: reject pickling writable ZipStore - #4168

Open
AMBRA7592 wants to merge 8 commits into
zarr-developers:mainfrom
AMBRA7592:codex/zipstore-pickle-guard
Open

fix: reject pickling writable ZipStore#4168
AMBRA7592 wants to merge 8 commits into
zarr-developers:mainfrom
AMBRA7592:codex/zipstore-pickle-guard

Conversation

@AMBRA7592

@AMBRA7592 AMBRA7592 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Pickling a ZipStore doesn't serialize the archive; it serializes the parameters to reopen it. So pickling is safe exactly when reopening is safe.

For readers (mode="r"), it is: many independent processes can open the same path and just read. Round-trips fine.

For writers, it isn't. A ZIP has one central directory written at close. Fan out several unpickled writers to one path and each rewrites that directory on close; last close wins, the rest is corruption. Since cross-process fan-out is why you pickle at all, the writer pickle path leads straight into the one thing ZIP can't survive. Rejecting at __getstate__ fails fast instead.

Closes #3516.

For reviewers

Keying on _zmode != "r" (not read_only) is right: an append-mode store can be read_only=True yet still rewrite its directory on reopen. Only "r" reopens harmlessly.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@github-actions github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jul 20, 2026
@github-actions github-actions Bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jul 20, 2026
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.12%. Comparing base (5a4767b) to head (2b2ebe6).
⚠️ Report is 28 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4168      +/-   ##
==========================================
+ Coverage   93.99%   94.12%   +0.12%     
==========================================
  Files          91       92       +1     
  Lines       12795    12832      +37     
==========================================
+ Hits        12027    12078      +51     
+ Misses        768      754      -14     
Files with missing lines Coverage Δ
src/zarr/storage/_zip.py 98.19% <100.00%> (+0.01%) ⬆️
src/zarr/testing/store.py 98.33% <100.00%> (ø)

... and 16 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@d-v-b
d-v-b requested a review from mkitti July 22, 2026 21:42
@AMBRA7592

Copy link
Copy Markdown
Contributor Author

Hi @mkitti, following up once on this PR. After @d-v-b synchronized the branch on August 5, the full check suite passed. When you have time, could you please review whether the _zmode != "r" pickling guard is the right safety boundary? Thanks.

@mkitti

mkitti commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

I'm not quite sure I understand what gets pickled here. What if the file changes between when it is pickled and unpickled?

@AMBRA7592

Copy link
Copy Markdown
Contributor Author

Thanks, that is the important distinction. The ZIP data itself is not pickled. __getstate__ stores the path and reopening state after removing the live ZipFile and lock; __setstate__ then opens a new ZipFile at that path.

Therefore, if the archive changes between pickling and unpickling, the restored read-only store sees the archive as it exists at unpickle time. If the path was removed or no longer contains a valid ZIP, opening or reading will fail. This is path-reference semantics, not snapshot isolation. I verified that replacing a valid archive between those operations makes the restored store read the replacement contents.

That behavior is non-destructive for mode="r", although callers requiring a stable view must keep the archive unchanged while it is being read. For mode="w" or mode="a", reopening can itself mutate the archive, and independent unpickled writers can overwrite its central directory. That destructive reopening is the boundary this PR rejects.

I can add an explicit documentation sentence or regression test for the current-file semantics if you think it should be part of the contract.

@mkitti

mkitti commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Documentation and a regression test would be appreciated.

@AMBRA7592

Copy link
Copy Markdown
Contributor Author

Added in 2b2ebe67.

  • The ZipStore API notes and pickle user guide now state that a path-backed read-only pickle is a path reference, not an archive snapshot, and that callers requiring a stable view must keep the archive unchanged.
  • The regression test replaces a valid archive after pickling and verifies that the unpickled reader observes the replacement contents.

Local verification:

  • complete tests/test_store/test_zip.py: 83 passed, 18 skipped
  • Ruff check and format check passed
  • documentation lint passed
  • git diff --check passed

The updated PR CI is now running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Writing dask arrays to a ZipStore causes a corrupt zip file

3 participants