Skip to content

Fix error handling in the Zstandard codec - #880

Draft
mkitti wants to merge 2 commits into
zarr-developers:mainfrom
mkitti:mkitti-zstd-error-handling
Draft

mkitti wants to merge 2 commits into
zarr-developers:mainfrom
mkitti:mkitti-zstd-error-handling

Conversation

@mkitti

@mkitti mkitti commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Fixes inconsistent error handling and a few memory-safety problems in src/numcodecs/zstd.pyx. This follows up on the review comments in #848, which suggested these belong in a separate PR against main.

Bugs fixed

  • Empty frames could not be decoded. Zstd().decode(Zstd().encode(b"")) raised invalid input data because a content size of 0 was treated as an error. An empty input is still rejected.
  • Double free in stream_decompress. If ZSTD_initDStream failed, the DStream was freed inside the nogil block and again in finally.
  • free() of an uninitialized pointer. output.dst was freed in finally before it was ever assigned if init failed.
  • Unchecked allocations. malloc, ZSTD_createDStream and ZSTD_createCCtx results were never checked; these now raise MemoryError.
  • Context leak in compress. ZSTD_createCCtx and ZSTD_CCtx_setParameter ran outside the try, so a parameter error leaked the context.
  • Error names formatted from bytes. Messages looked like Zstd decompression error: b"Restored data doesn't match checksum". They are now plain text.
  • Lost diagnostics. Failures while determining frame sizes were collapsed into a bare invalid input data; the zstd reason is now appended (the existing invalid input data prefix is kept so current matches still work).

Improvements

  • Errors in the streaming loop are recorded inside the nogil block and raised after it, rather than raising from within it.
  • Truncated streaming input is detected explicitly instead of relying on an earlier check to avoid looping.
  • The streaming output buffer grows geometrically (it grew by a fixed 128 KiB per step, which is quadratic for large outputs), with overflow checks.
  • Overflow check when summing frame content sizes across concatenated frames; findTotalContentSize now declares its exception behaviour.
  • stream_decompress has a bytes return type; dead try/finally: pass removed.
  • Zstd.__repr__ now includes checksum when it is enabled (default output is unchanged).

Testing

New tests cover empty frames, empty input, truncated known/unknown-size frames, text error messages, decoding unknown-size frames into out, repr with checksum, and multi-frame streaming growth. Full suite locally: 645 passed, 37 skipped.

Backport

Once this is reviewed, it should be backported to 0.15.x alongside #848.

TODO:

  • Unit tests and/or doctests in docstrings
  • Tests pass locally
  • Docstrings and API docs for any new/modified user-facing classes and functions
  • Changes documented in docs/release.md
  • Docs build locally
  • GitHub Actions CI passes
  • Test coverage to 100% (Codecov passes)

🤖 Generated with Claude Code

- Decode empty frames instead of reporting invalid input data.
- Stream decompression: fix a double free of the DStream and a free() of an
  uninitialized output pointer when ZSTD_initDStream fails, check malloc and
  ZSTD_createDStream results, raise errors outside the nogil block, detect
  truncated input, and grow the output buffer geometrically.
- compress: check ZSTD_createCCtx, and always free the context when setting
  a parameter fails.
- Report zstd error names as text rather than as the repr of bytes, and keep
  zstd's message when frame sizes cannot be determined.
- Guard against overflow when summing frame content sizes.
- Include checksum in Zstd.__repr__ when enabled.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@mkitti mkitti mentioned this pull request Oct 20, 2026
7 tasks done
Frames whose header claims a content size above PY_SSIZE_T_MAX raised a
SystemError from PyBytes_FromStringAndSize. They now raise RuntimeError.

Add tests for empty frames, truncated and garbage input, text error messages,
content size overflow, destination buffer handling, buffer types, level
clamping and repeated error paths.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant