Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

### Bug Fixes

- **[client-v2]** Fixed `RowBinaryFormatWriter` throwing `UnsupportedOperationException: Unsupported data type: Nested` when inserting into a table with an un-flattened `Nested(...)` column (created with `flatten_nested = 0`). The `RowBinary` writer now serializes a `Nested(f1 T1, ..., fN TN)` column the same way it is read — identically to `Array(Tuple(T1, ..., TN))`. (https://github.com/ClickHouse/clickhouse-java/issues/2477)

- **[client-v2]** Fixed binary array decoding for nullable element types so `Array(Nullable(Float64))` and similar columns now return boxed arrays such as `Double[]` instead of `Object[]`. This keeps null-supporting arrays aligned with their element type while preserving the existing `Object[]` fallback for Variant/Dynamic/Geometry arrays. (https://github.com/ClickHouse/clickhouse-java/issues/2846)

- **[client-v2]** Fixed `Float32`/`Float64` columns throwing `ClassCastException` when a value of a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ public static void serializeData(OutputStream stream, Object value, ClickHouseCo
case Map:
serializeMapData(stream, value, column);
break;
case Nested:
serializeNestedTypeData(stream, value, column);
break;
case AggregateFunction:
serializeAggregateFunction(stream, value, column);
break;
Expand Down Expand Up @@ -130,6 +133,25 @@ private static void serializeNestedData(OutputStream stream, Object value, Click
serializeData(stream, value, column);
}

/**
* Serializes a {@code Nested} column. In {@code RowBinary} a {@code Nested(f1 T1, ..., fN TN)}
* column has the same layout as {@code Array(Tuple(T1, ..., TN))}: a var-uint element count
* followed by that many tuples, each carrying the N field values in declaration order. The
* value is therefore a list (or array) of tuples, matching what
* {@link BinaryStreamReader#readNested(ClickHouseColumn)} produces on read.
*/
private static void serializeNestedTypeData(OutputStream stream, Object value, ClickHouseColumn column) throws IOException {
if (value == null) {
writeVarInt(stream, 0);
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Null Nested rejected unlike Array

Medium Severity

RowBinaryFormatWriter still rejects a Java null for a non-nullable Nested column in writeValuePreamble, while non-nullable Array columns are allowed through and serialize as an empty array. The new serializeNestedTypeData writes a zero element count for null, but that path is never reached on insert.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e0f4674. Configure here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — I checked this empirically against a live server (ClickHouse 26.5) and I don't think we should let a Java null through for a non-nullable Nested column, because the Array path it would mirror is itself byte-incorrect.

Here is exactly what writeValuePreamble + the serializer emit for a non-nullable column given a Java null:

value bytes the client emits
Array(Int32) + null 00 00
Nested(a Int32, b String) + null throws IllegalArgumentException: An attempt to write null into not nullable column
Nested(a Int32, b String) + Arrays.asList() (empty) 00

The Array case emits two bytes: writeValuePreamble writes writeNonNull (0x00) and then serializeArrayData(null) writes the var-uint length 0 (0x00). Sending those bytes to the server for a single Array(Int32) column does not insert one empty array — it inserts a phantom second row:

# one 0x00  -> 1 row
printf '\x00'      | INSERT ... FORMAT RowBinary ;  SELECT count(), groupArray(arr)  ->  1  [[]]
# two 0x00  -> 2 rows  (the extra preamble byte is read as a second empty-array row)
printf '\x00\x00'  | INSERT ... FORMAT RowBinary ;  SELECT count(), groupArray(arr)  ->  2  [[],[]]

So Array doesn't correctly "serialize null as an empty array" — it silently corrupts the stream (and in a multi-column row that extra byte shifts every following column). Making Nested match Array would copy that bug.

The current Nested behavior — throwing for null into a non-nullable column — is the safe and consistent choice (it's what every non-Array/Dynamic type does) and it fails loudly instead of corrupting data. The correct way to write an empty Nested is an empty list, which round-trips correctly (00, a single empty row) and is covered by the unit test. The if (value == null) writeVarInt(0) guard in serializeNestedTypeData is a deliberate mirror of the identical guard in serializeArrayData and emits correct bytes if reached via direct/recursive serialization, so I've kept it.

I'm leaving this thread open. The pre-existing Array + null phantom-row issue is a separate behavior change unrelated to the Nested support in this PR, so I'm flagging it to be fixed on its own rather than folding it in here. If you'd prefer top-level null to mean "empty Nested", the correct fix would be a do-nothing preamble branch (emitting a single 00), not the Array branch — happy to add that, but it's a separate decision from the crash fix here.

}
List<?> tuples = convertArrayValueToList(value);
writeVarInt(stream, tuples.size());
for (Object tuple : tuples) {
serializeTupleData(stream, tuple, column);
}
Comment thread
cursor[bot] marked this conversation as resolved.
}

private static final Map<Class<?>, ClickHouseColumn> PREDEFINED_TYPE_COLUMNS = getPredefinedTypeColumnsMap();

private static Map<Class<?>, ClickHouseColumn> getPredefinedTypeColumnsMap() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,44 @@
};
}

@Test(dataProvider = "rowBinaryTypeData")
public void testRowBinaryTypeRoundTrip(String typeName, Object value) throws Exception {

Check warning on line 221 in client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/SerializerUtilsTest.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Update this method so that its implementation is not identical to "testNestedNullableRoundTrip" on line 147.

See more on https://sonarcloud.io/project/issues?id=ClickHouse_clickhouse-java&issues=AZ9yUVxyFoG4PDw31zWH&open=AZ9yUVxyFoG4PDw31zWH&pullRequest=2936
ClickHouseColumn column = ClickHouseColumn.of("v", typeName);

ByteArrayOutputStream out = new ByteArrayOutputStream();
SerializerUtils.serializeData(out, value, column);

Object actual = newReader(out.toByteArray()).readValue(column);
Assert.assertEquals(normalize(actual), normalize(value));
}

@DataProvider(name = "rowBinaryTypeData")
private Object[][] rowBinaryTypeData() {
return new Object[][] {
// A Nested(...) column has the same RowBinary layout as Array(Tuple(...)): a
// var-uint row count followed by that many tuples. Two rows detect a wrong count
// or a dropped field byte, which would shift every following tuple.
{"Nested(a Int32, b String)",
Arrays.asList(Arrays.asList(1, "x"), Arrays.asList(2, "y"))},

// Same value as arrays instead of Lists: an Object[][] of Object[] rows (a
// "matrix" array). convertArrayValueToList takes the array branch and each row is
// serialized by serializeTupleData's array branch, producing the same bytes as the
// List-shaped case above.
{"Nested(a Int32, b String)",
new Object[][] {{1, "x"}, {2, "y"}}},

// A Nullable field in the MIDDLE of the nested tuple, with a trailing fixed-width
// Float64: a dropped null-marker byte misaligns the Float64 and is caught. The
// second row exercises the null branch of that field.
{"Nested(a Int32, b Nullable(String), c Float64)",
Arrays.asList(Arrays.asList(7, "opt", 9.5d), Arrays.asList(7, null, 8.5d))},

// An empty Nested serializes as a zero-length array.
{"Nested(a Int32, b String)", Arrays.asList()},
};
}

// Normalizes Tuple (Object[]) and Array (ArrayValue / List) results to nested Lists so
// round-tripped values compare structurally regardless of the container representation the
// reader returns.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,27 @@ public void writeNestedTests() throws Exception {
writeTest(tableName, tableCreate, rows);
}

@Test (groups = { "integration" })
public void writeNestedTypeTests() throws Exception {
String tableName = "rowBinaryFormatWriterTest_writeNestedTypeTests_" + UUID.randomUUID().toString().replace('-', '_');
// flatten_nested = 0 keeps the column typed as Nested(...) in the schema instead of
// expanding it into parallel Array(...) sub-columns; the un-flattened Nested type is the
// one that has to be serialized as Array(Tuple(...)).
String tableCreate = "CREATE TABLE \"" + tableName + "\" " +
" (id Int32, " +
" n Nested(a UInt32, b Nullable(String)) " +
" ) Engine = MergeTree ORDER BY id SETTINGS flatten_nested = 0";

List<Object> nested = Arrays.asList(Arrays.asList(10L, "x"), Arrays.asList(20L, null));
Field[][] rows = new Field[][] {{
new Field("id", 1), //Row ID
new Field("n", nested).set(nested) //Nested
}
};

writeTest(tableName, tableCreate, rows);
}

@Test (groups = { "integration" })
public void writeNullableTests() throws Exception {
String tableName = "rowBinaryFormatWriterTest_writeNullableTests_" + UUID.randomUUID().toString().replace('-', '_');
Expand Down
Loading