fix: reject pickling writable ZipStore - #4168
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4168 +/- ##
==========================================
+ Coverage 93.99% 94.12% +0.12%
==========================================
Files 91 92 +1
Lines 12795 12832 +37
==========================================
+ Hits 12027 12078 +51
+ Misses 768 754 -14
🚀 New features to boost your workflow:
|
…le-guard # Conflicts: # src/zarr/storage/_zip.py
…o codex/zipstore-pickle-guard
|
I'm not quite sure I understand what gets pickled here. What if the file changes between when it is pickled and unpickled? |
|
Thanks, that is the important distinction. The ZIP data itself is not pickled. Therefore, if the archive changes between pickling and unpickling, the restored read-only store sees the archive as it exists at unpickle time. If the path was removed or no longer contains a valid ZIP, opening or reading will fail. This is path-reference semantics, not snapshot isolation. I verified that replacing a valid archive between those operations makes the restored store read the replacement contents. That behavior is non-destructive for I can add an explicit documentation sentence or regression test for the current-file semantics if you think it should be part of the contract. |
|
Documentation and a regression test would be appreciated. |
|
Added in
Local verification:
The updated PR CI is now running. |
Summary
Pickling a
ZipStoredoesn't serialize the archive; it serializes the parameters to reopen it. So pickling is safe exactly when reopening is safe.For readers (
mode="r"), it is: many independent processes can open the same path and just read. Round-trips fine.For writers, it isn't. A ZIP has one central directory written at close. Fan out several unpickled writers to one path and each rewrites that directory on close; last close wins, the rest is corruption. Since cross-process fan-out is why you pickle at all, the writer pickle path leads straight into the one thing ZIP can't survive. Rejecting at
__getstate__fails fast instead.Closes #3516.
For reviewers
Keying on
_zmode != "r"(notread_only) is right: an append-mode store can beread_only=Trueyet still rewrite its directory on reopen. Only"r"reopens harmlessly.Author attestation
TODO
docs/user-guide/*.mdchanges/