totals don't work with new Clickhouse server versions

[den-crane]

select sum(number), greater
    from (select number, number > 5 as greater from numbers(10))
group by greater with totals

ClickHouse 22.8.19.10 / ClickHouse JDBC Driver 0.4.6 (revision: dd91e17) -- totals OK
ClickHouse 23.3.5.9 / ClickHouse JDBC Driver 0.4.6 (revision: dd91e17) -- totals NOT OK

[zhicwu]

Thank you for reporting the issue. It appears that the problem is not limited to gRPC in the new version but also includes RowBinary or the method of generating the totals output.

22.8.15.23
00000000: 0201 7301 6706 5549 6e74 3634 0555 496e  ..s.g.UInt64.UIn
00000010: 7438 0f00 0000 0000 0000 001e 0000 0000  t8..............
00000020: 0000 0001 2d00 0000 0000 0000 00         ....-........

23.3.6.7
00000000: 0201 7301 6706 5549 6e74 3634 0555 496e  ..s.g.UInt64.UIn
00000010: 7438 0f00 0000 0000 0000 001e 0000 0000  t8..............
00000020: 0000 0001

[den-crane]

@zhicwu
Did you report this to Clickhouse?

[den-crane]

I would throw an error, that totals are not supported.

[zhicwu]

Did you report this to Clickhouse?

No I didn't, as typically there is no response from anyone.

I would throw an error, that totals are not supported.

It's a bit difficult because 1) it requires SQL parsing to understand if the result should have totals or not; and 2) it's hard to tell if the server missed totals or not in RowBinary format(because it does not have a separator like the blank line added into TSV format).

I think it's good idea to be more defensive and add more validations on client side in case the server missed anything or protocol changed. Perhaps it would be beneficial to transition to the native data format, considering it as a first-class citizen.

[den-crane]

It seems like totals is already detected https://github.com/search?q=repo%3AClickHouse%2Fclickhouse-java%20totals&type=code

I would throw an error always, no matter format.

to the native data format, considering it as a first-class citizen.

I don't like native, it requires much more compute power from the client. Some users will complain that JDBC driver needs 8GB RAM to process select results (to pivot very long native block). Go-driver has this issue, they just limited the max size of the native block in a result and reject to execute some queries at all.

[zhicwu]

It seems like totals is already detected

Yes, almost forgot that :p However, determining whether the last few rows in the RowBinary format represent totals or not can be challenging.

I don't like native, it requires much more compute power from the client.

Offloading serialization and deserialization to clients can be beneficial in certain scenarios, particularly for improving server scalability and handling higher request loads, particularly during ingestion processes. #1357 does not benefit RowBinary much but I believe it helps for Native.

[mzitnik]

@den-crane So looks like the problem solved on ClickHouse server side can we close it?