avro: support snappy and zstandard codecs - #16
Merged
Conversation
Avro.Document previously errored on any codec but null/deflate. Route the two popular ones the engine can decompress — snappy and zstandard — through the codec oracle (a minimal in-memory Parquet file handed to Parquet.Document), inlining only the snappy/zstandard slice of codec-oracle/Codec.Decompress.pq rather than the whole thing. An Avro snappy block carries a 4-byte CRC-32 trailer that is not part of the snappy stream; it is stripped before decoding (not validated — sync markers already catch corruption). bzip2 and xz remain a clear error (no engine codec exists). Adds scalars-snappy/scalars-zstandard fixtures and query tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Avro.Documentpreviously errored on any codec butnull/deflate. This adds the two popular codecs the mashup engine can actually decompress: snappy and zstandard.Why
Binary.Decompressimplements onlydeflateandgzip, so snappy/zstandard Avro files — the common output of Spark, Hadoop, and Kafka pipelines — were unreadable. The engine ships those codecs anyway insideParquet.Document, which the repo'scodec-oraclealready exploits.How
REQUIRED FIXED_LEN_BYTE_ARRAYcolumn, one row, PLAIN encoding) and handed toParquet.Documentto decompress.codec-oracle/Codec.Decompress.pqis inlined — the twocodecIds, their size derivations (snappy preamble varint, zstdFrame_Content_Size), and the shared Parquet-wrapper builder. LZ4, Brotli, GZip, the framing/override options, etc. are left out since Avro never reaches them.bzip2andxzremain a clear error: the engine has no codec for them, so the oracle can't help.Reviewer notes
deflate/nullpath — the newscalars-snappy.avro/scalars-zstandard.avrofixtures decode to the same rows asscalars.avro, and their.pqoutbaselines are copies of it.avro/test/make_fixtures.py(now needspython-snappy+backports.zstd, the exact package names fastavro looks up).avro/README.md,avro/test/expected.md.