Repository navigation
Conversation
- Decode empty frames instead of reporting invalid input data. - Stream decompression: fix a double free of the DStream and a free() of an uninitialized output pointer when ZSTD_initDStream fails, check malloc and ZSTD_createDStream results, raise errors outside the nogil block, detect truncated input, and grow the output buffer geometrically. - compress: check ZSTD_createCCtx, and always free the context when setting a parameter fails. - Report zstd error names as text rather than as the repr of bytes, and keep zstd's message when frame sizes cannot be determined. - Guard against overflow when summing frame content sizes. - Include checksum in Zstd.__repr__ when enabled. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Frames whose header claims a content size above PY_SSIZE_T_MAX raised a SystemError from PyBytes_FromStringAndSize. They now raise RuntimeError. Add tests for empty frames, truncated and garbage input, text error messages, content size overflow, destination buffer handling, buffer types, level clamping and repeated error paths. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes inconsistent error handling and a few memory-safety problems in
src/numcodecs/zstd.pyx. This follows up on the review comments in #848, which suggested these belong in a separate PR againstmain.Bugs fixed
Zstd().decode(Zstd().encode(b""))raisedinvalid input databecause a content size of 0 was treated as an error. An empty input is still rejected.stream_decompress. IfZSTD_initDStreamfailed, the DStream was freed inside thenogilblock and again infinally.free()of an uninitialized pointer.output.dstwas freed infinallybefore it was ever assigned if init failed.malloc,ZSTD_createDStreamandZSTD_createCCtxresults were never checked; these now raiseMemoryError.compress.ZSTD_createCCtxandZSTD_CCtx_setParameterran outside thetry, so a parameter error leaked the context.bytes. Messages looked likeZstd decompression error: b"Restored data doesn't match checksum". They are now plain text.invalid input data; the zstd reason is now appended (the existinginvalid input dataprefix is kept so current matches still work).Improvements
nogilblock and raised after it, rather than raising from within it.findTotalContentSizenow declares its exception behaviour.stream_decompresshas abytesreturn type; deadtry/finally: passremoved.Zstd.__repr__now includeschecksumwhen it is enabled (default output is unchanged).Testing
New tests cover empty frames, empty input, truncated known/unknown-size frames, text error messages, decoding unknown-size frames into
out, repr withchecksum, and multi-frame streaming growth. Full suite locally: 645 passed, 37 skipped.Backport
Once this is reviewed, it should be backported to 0.15.x alongside #848.
TODO:
🤖 Generated with Claude Code