Add Tablet.serializedSize() and comprehensive size validation tests.#824
Conversation
Pre-allocate serialization buffer using exact size estimation, support OBJECT type in tablet serialize/deserialize path, and consolidate serializedSize tests.
|
I found a functional issue.
So when the device/table name, measurement name, or schema properties contain non-ASCII characters, This is probably not an issue when TsFile is used through IoTDB, because IoTDB startup sets the default charset. But Suggested fix: make There is also a CodeQL alert for integer narrowing/overflow in |
| size = Math.addExact(size, Integer.BYTES); | ||
| size = | ||
| Math.addExact( | ||
| size, | ||
| ReadWriteIOUtils.sizeToWrite( | ||
| new Binary(bitMaps[i].getTruncatedByteArray(rowSize)))); | ||
| size = Math.addExact(size, Integer.BYTES); | ||
| size = Math.addExact(size, BitMap.getSizeOfBytes(rowSize)); |
There was a problem hiding this comment.
The two Integer.BYTES entries represent two different fields in the serialized format.
In writeBitMaps(), a non-empty bitmap is serialized as:
hasBitMapflag: 1 byterowSize: 4 bytesBinarylength prefix: 4 bytes- bitmap bytes:
BitMap.getSizeOfBytes(rowSize)
The previous code used:
ReadWriteIOUtils.sizeToWrite(new Binary(bitMaps[i].getTruncatedByteArray(rowSize)))That includes the Binary length prefix plus the actual bitmap bytes.
So the new code:
size = Math.addExact(size, Integer.BYTES); // rowSize
size = Math.addExact(size, Integer.BYTES); // Binary length prefix
size = Math.addExact(size, BitMap.getSizeOfBytes(rowSize)); // bitmap bytesis equivalent to the old calculation. The two integers are not duplicates: one is the bitmap logical size (rowSize), and the other is the length prefix
written by ReadWriteIOUtils.write(Binary, stream).
Pre-allocate serialization buffer using exact size estimation, support OBJECT type in tablet serialize/deserialize path, and consolidate serializedSize tests.