When serialize FixedString and string is bigger than size defined in Table definition

[mzitnik]

Describe your feedback

Currently, the behavior is that if the string that is sent to serialize is bigger, we send only the length defined in the table schema.

A good approach is to have a flag that indicates if it's a bigger throw Exception or send only a string of a defined size.

Code example

How it's done today
writeFixedString only what is needed.

     BinaryStreamUtils.writeFixedString(stream, convertToString(value), column.getPrecision());

[chernser]

@mzitnik
While working on this issue I've got next thoughts:

• the flag will affect whole client. So it would be problematic to tolerate bigger strings from subset of columns
• what if just have a method that truncates string to defined length?
• if we truncate string at utf8 symbol position it will result string with completely different meaning.

Here is the test for the last point:

    @Test
    public void testUtfStringTruncate() {
        final String str = "a☺c";
        byte[] bytes = str.getBytes(StandardCharsets.UTF_8);
        System.out.println("bytes.length: " + bytes.length);
        for (int i = 0; i < bytes.length; i++) {
            System.out.println(String.format("bytes[%d]: %x", i, bytes[i]));
        }

        byte[] twoBytes= new byte[] {bytes[0], bytes[1]};
        String twoByteString = new String(twoBytes, StandardCharsets.UTF_8);
        System.out.println("twoByteString: " + twoByteString);
        System.out.println("twoByteString.length: " + twoByteString.length());
    }

Output:

bytes.length: 5
bytes[0]: 61
bytes[1]: e2
bytes[2]: 98
bytes[3]: ba
bytes[4]: 63
twoByteString: a�
twoByteString.length: 2

The correct string should be a string with only a because e2 is part of ☺. In this case 3 bytes used for a character.

[chernser]

/triage

[github-actions]

Triage Report

Effort to Fix: Small
Type: Feature Request
Affected Module: client-api-v2

Summary

When a Java string is serialized into a ClickHouse FixedString(N) column and its UTF-8 byte length exceeds N, the current code in BinaryStreamUtils.writeFixedString throws an exception via ClickHouseChecker.notLongerThan. The issue requests a configurable option (a flag or setting) to allow silent truncation to N bytes instead of throwing, with the important caveat that truncation must occur at a valid UTF-8 character boundary to avoid producing corrupted strings.

Recommended Labels & Priority

• Labels: client-api-v2, v2-feedback, area:data-type

Missing Information / Questions for User

1. Should the truncation option be a global client setting (e.g., ClientConfigProperties) or a per-column/per-insert setting? A global flag makes it hard to tolerate oversized strings only for a subset of columns.
2. What is the preferred truncation semantics: truncate to at most N bytes at the last valid UTF-8 character boundary (safe, may lose more than one byte), or is padding with replacement characters acceptable?
3. Which client version is being used (client-v2 is assumed from existing labels)?

Tests to Add

Test 1: FixedString insertion — string exactly at byte limit

// Insert a FixedString(4) column with a 4-byte ASCII string — should succeed
client.insert("test_table", List.of(Map.of("col", "abcd")));
// assert row reads back as "abcd"

Test 2: FixedString insertion — string exceeds limit, THROW mode (current default)

// Insert a FixedString(4) column with a 5-byte ASCII string — should throw
assertThrows(ClientException.class, () ->
    client.insert("test_table", List.of(Map.of("col", "abcde"))));

Test 3: FixedString insertion — UTF-8 multibyte string exceeds limit, TRUNCATE mode

// "a☺c" is 5 bytes in UTF-8; FixedString(2) should truncate safely to "a" (1 byte),
// not to "a\xe2" which is an invalid UTF-8 sequence.
// (Enable truncation mode on the client/settings before inserting.)
client.insert("test_table", List.of(Map.of("col", "a☺c")));
String result = // read back the value
assertTrue(result.equals("a") || result.isEmpty()); // no garbled bytes

Test 4: FixedString insertion — ASCII string exceeds limit, TRUNCATE mode

// FixedString(3) column with "abcde" — should truncate to "abc" without exception
// (Enable truncation mode on the client/settings before inserting.)
client.insert("test_table", List.of(Map.of("col", "abcde")));
String result = // read back the value
assertEquals("abc", result);