[Bug] Arrow Flight SQL: Iceberg BINARY column is declared as Arrow string but carries invalid UTF-8

[morningman]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Apache Doris 4.1.3-rc02, commit 31263df4dc1d4d3a27517d264802cd4d6b92c874

Client: Python + ADBC Flight SQL driver (adbc_driver_flightsql), FE arrow_flight_sql_port = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

What's Wrong?

When an Iceberg catalog is configured with "enable.mapping.varbinary" = "false", Doris Flight SQL declares an arbitrary BINARY column as Arrow string, but the payload actually contains non-UTF-8 bytes.

PyArrow fails before returning any row:

UnicodeDecodeError  (byte 0x84)

The Arrow schema for the query is id: int32, col1: string, col2: string. JDBC reads the same Iceberg rows without a problem, and with VARBINARY mapping enabled the ADBC binary read works.

What You Expected?

The Arrow type declared by the server and the encoding of the payload must agree. Arbitrary binary content should be sent as binary/large_binary, or, if it is sent as string, it must be a valid UTF-8 representation.

How to Reproduce?

1. Bring up the Iceberg regression fixture.
2. Create or use an Iceberg catalog with "enable.mapping.varbinary" = "false".
3. SWITCH to that catalog and USE test_varbinary.
4. Run the query over Python ADBC, inspect the schema and call to_pylist().

SWITCH test_iceberg_no_mapping;
USE test_varbinary;
SELECT * FROM test_ice_uuid_orc ORDER BY id;

Client side:

import adbc_driver_flightsql.dbapi as flight_sql

conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
                          db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT * FROM test_iceberg_no_mapping.test_varbinary.test_ice_uuid_orc ORDER BY id")
table = cur.fetch_arrow_table()
print(table.schema)       # id: int32, col1: string, col2: string
print(table.to_pylist())  # UnicodeDecodeError on byte 0x84

Anything Else?

Without VARBINARY mapping, the Iceberg binary/uuid column falls back to the Doris STRING type, and the Arrow conversion follows that declared type rather than the real content. Since Arrow string is defined as UTF-8, any consumer that validates the encoding (PyArrow does) fails on the whole batch.

Workaround: set "enable.mapping.varbinary" = "true" on the Iceberg catalog so the column is returned through the Arrow binary type.

Found with the external_table_p0/iceberg/test_iceberg_varbinary fixture; four no-mapping cases are affected, while the mapping-enabled binary reads keep passing over ADBC.

Tracking issue: #65615

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Initial assessment

Confirmed as a valid server-side Arrow schema/payload mismatch in the reported 4.1.3-rc02 commit (31263df4dc1d4d3a27517d264802cd4d6b92c874). This is not only a Python presentation issue: Arrow string carries UTF-8 semantics, while this path can place arbitrary Iceberg binary bytes in its value buffer. A validating client can therefore reject the entire batch before exposing rows.

The issue is actionable as written. The supplied fixture inserts non-UTF-8 sequences including byte 0x84, so no BE log or profile is needed to establish the type-contract problem.

Code evidence

• IcebergUtils.icebergPrimitiveTypeToDorisType() maps Iceberg UUID and BINARY to Doris STRING when enable.mapping.varbinary=false (fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:691-714).
• The file readers preserve that fallback: ORC BINARY becomes TYPE_STRING when mapping is disabled (be/src/format/orc/vorc_reader.cpp:1705-1738), and raw Parquet BYTE_ARRAY/UUID similarly become TYPE_STRING (be/src/format/parquet/schema_desc.cpp:251-258,323-329; the v2 reader has the same mapping).
• Arrow Flight constructs its result schema only from the final BE output expression types (be/src/exec/operator/result_sink_operator.cpp:58-67 and be/src/format/arrow/arrow_row_batch.cpp:220-238). At that point the original Iceberg binary identity is no longer present; Doris STRING is unconditionally mapped to Arrow utf8, while Doris VARBINARY maps to Arrow binary (arrow_row_batch.cpp:88-96,176-178).
• Batch conversion then uses DataTypeStringSerDe, which appends the stored bytes unchanged to StringBuilder (be/src/core/data_type_serde/data_type_string_serde.cpp:399-445). The Flight conversion path does not run Arrow full validation before returning the batch. This directly explains why schema inspection reports string and PyArrow later raises UnicodeDecodeError.
• With enable.mapping.varbinary=true, the FE/reader type remains VARBINARY, so the existing VARBINARY -> arrow::binary() path is internally consistent. The documented workaround is therefore sound.

The MySQL/JDBC control path succeeding does not contradict this diagnosis: that protocol can transport length-delimited byte sequences under Doris string metadata and does not impose Arrow's UTF-8 invariant.

Fix boundary and open design decision

Changing only the string byte-copy loop is insufficient. The Arrow type must be selected before the stream schema is published, so the result schema builder needs binary semantic information that is currently lost between Iceberg schema mapping and the result sink.

Two viable directions should be evaluated:

1. Carry an Arrow result-type hint/source binary provenance through the FE result sink/output expression path and emit binary (or fixed-size binary for UUID/FIXED where appropriate) for direct binary-origin results. The existing string SerDe already supports writing a Doris string column into binary/fixed-size-binary Arrow builders, so payload conversion support largely exists.
2. Deliberately expose these values as Doris VARBINARY for Flight SQL even when the catalog uses the legacy no-mapping mode.

The second option is simpler but may make SQL/result metadata protocol-dependent. The first option must define propagation through aliases, projections, expressions, nested fields, and casts so a genuine Doris STRING is never silently reclassified as binary. Silently hex/base64-encoding only on Flight would avoid invalid UTF-8 but would change values relative to the SQL result and should not be done without an explicit compatibility decision.

Missing information / triage notes

• No labels, assignee, or milestone were attached when this issue was observed. Please add the repository's bug plus Arrow Flight/external-catalog component labels and route it to both owners.
• Exact adbc_driver_flightsql and PyArrow versions would help document the client compatibility matrix, but they are not blocking for accepting the bug because the server-side schema invariant is independently visible in code.
• Maintainers should explicitly confirm the intended cross-protocol contract for enable.mapping.varbinary=false: byte-preserving binary output or a textual representation. That decision determines which fix direction is compatible.

Recommended next steps

1. Add a failing end-to-end Flight SQL regression using the existing test_iceberg_varbinary ORC and Parquet fixtures with mapping disabled. It should assert the Arrow field type, call a validating consumer such as Python ADBC/PyArrow to_pylist() (or validate(full=True)), and compare the exact returned bytes. Keep mapping-enabled cases as controls.
2. Add focused BE coverage proving ordinary valid Doris STRING still produces Arrow UTF-8 and binary-origin fallback data produces Arrow binary, including nullable values and both local/remote Flight readers.
3. Implement the chosen type-provenance/Flight typing approach at schema construction, then verify aliases/casts/nested columns and ensure MySQL/JDBC behavior is unchanged.

No code was modified during this analysis, and the reported failure was not independently rerun against a live cluster; the root-cause judgment above is based on the exact reported commit and the checked-in fixture bytes.

Breakwater-GitHub-Analysis-Slot: slot_c8ad7b1e212c