[BP-2.2][FLINK-40227][core] Use readFully to read Variant value/metadata in VariantSerializer.deserialize - #28852
Merged
Conversation
…ariantSerializer.deserialize VariantSerializer.deserialize filled the value and metadata byte arrays with DataInputView#read(byte[]) and ignored the returned count. That read is permitted to return a partial (short) read when the array straddles a buffer or compression-frame boundary in the source stream. A short read leaves the tail of the array unfilled and desyncs the stream, and when restoring large Variant keys from a checkpoint key-group stream on rescale it surfaces as MALFORMED_VARIANT (a misaligned read lands on a bad metadata header) or an OutOfMemoryError (a misaligned read lands on a length prefix). It is intermittent because whether an array straddles a boundary varies between restore attempts. Use readFully instead: it loops until each buffer is filled and throws EOFException only on a genuine end-of-stream. The on-wire format is unchanged, so this is fully snapshot-compatible and needs no state migration.
1 task
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #28810
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Anthropic Claude)
🤖 Generated with Claude Code