MINOR: Reject frame version, type and version varints that decode negative - #23273
Open
MahathirMohammadShuvo wants to merge 1 commit into
Open
MINOR: Reject frame version, type and version varints that decode negative#23273MahathirMohammadShuvo wants to merge 1 commit into
MahathirMohammadShuvo wants to merge 1 commit into
Conversation
…ative unsignedIntToShort checked only the upper bound. readUnsignedVarint decodes an unsigned 32-bit value into a signed int, so an encoded value in [2^31, 2^32) comes back negative, passes val > Short.MAX_VALUE, and is then truncated to its low 16 bits by the cast. The bytes 81 80 80 80 08 decode to -2147483647 and are accepted as frame version 1, which is DEFAULT_FRAME_VERSION, so parsing continues against a record that never declared it. The same bytes in the type field make a record deserialize as a different apiKey. The existing testParsingVersionTooLarge uses a four-byte varint, which stays positive and so has always taken the upper-bound branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
unsignedIntToShortinAbstractApiMessageSerdechecked only the upper bound:ByteUtils.readUnsignedVarintdecodes an unsigned 32-bit value into a signedint(result |= (tmp = buffer.get()) << 28), so an encoded value in[2^31, 2^32)comes back negative. A negative value is not> Short.MAX_VALUE,so it passes the guard and is then truncated to its low 16 bits by the cast.
Measured against a transcription of
readUnsignedVarint:81 80 80 80 08185 80 80 80 085ff ff ff ff 0f-1ff ff ff 7f0111DEFAULT_FRAME_VERSIONis1, so the first row is accepted as the defaultframe version and parsing continues against a record that never declared one.
The same bytes in the type field make the record deserialize as a different
apiKey.Scope
The guard now rejects a varint that decodes negative. It does not attempt to
reject every encoding above
Integer.MAX_VALUE—81 80 80 80 10encodes4294967297, and
16 << 28wraps to0, so it decodes to a positive1and isstill accepted. That is a property of the primitive rather than of this check.
The fix is deliberately in
unsignedIntToShortand not inreadUnsignedVarint.readVarintis built on the unsigned reader and zigzag-decodes its result, sothe primitive must be able to return negative —
81 80 80 80 08is thelegitimate encoding of
readVarint(-1073741825). Rejecting it there would breakrecord-batch lengths, offset deltas and every compact string, tagged field and
generated
read(). The bytes are a valid varint; only their interpretation as anunsigned short-range field is wrong.
No compatibility concern:
write()only ever emitsDEFAULT_FRAME_VERSION,apiKey()andversion(), all small non-negative shorts, so no compliant writercan produce an encoding the new guard rejects.
Tests
testParsingVersionEncodedAboveIntMaxadded next to the existingtestParsingVersionTooLarge, which uses a four-byte varint (ff ff ff 7f) thatstays positive — which is why the negative path was never covered.
./gradlew :metadata:test --tests "org.apache.kafka.metadata.MetadataRecordSerdeTest"—10 passed, 0 failed.
AbstractApiMessageSerde.javareverted totrunk, the new test isthe only failure; the other nine still pass.
val < 0alone makestestParsingVersionTooLargefail,so both halves of the condition are independently pinned.
./gradlew :server-common:checkstyleMain :metadata:checkstyleTest— clean.