[Bug] Arrow Flight SQL: MAP with a NULL key cannot be returned as an Arrow MAP

[morningman]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Apache Doris 4.1.3-rc02, commit 31263df4dc1d4d3a27517d264802cd4d6b92c874

Client: Python + ADBC Flight SQL driver (adbc_driver_flightsql), FE arrow_flight_sql_port = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

What's Wrong?

Doris MAP semantics allow a NULL key, but the Arrow MAP specification requires the key field to be non-nullable. When a result contains a NULL map key, the Flight SQL DoGet fails and the whole result becomes unreadable:

Can not write null value of map key to arrow

Projecting the same value as map_entries(m), i.e. ARRAY<STRUCT<key, value>>, is returned over ADBC correctly and preserves the NULL key.

What You Expected?

Flight SQL should be able to return MAP data that Doris itself accepts, either through a compatible Arrow representation that preserves the NULL key, or with a clear, documented limitation; it should not fail the entire result at DoGet time.

How to Reproduce?

1. Create a Doris table with a MAP<STRING,INT> column.
2. Insert map(NULL,100).
3. Query the raw MAP column over Python ADBC; DoGet fails.
4. Query map_entries(m); the same data is returned through the compatible structure.

DROP TABLE IF EXISTS adbc_null_map;
CREATE TABLE adbc_null_map (
  k INT,
  m MAP<STRING,INT>
) DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES("replication_num"="1");

INSERT INTO adbc_null_map VALUES (1, map(NULL,100));

-- Fails through Arrow Flight SQL.
SELECT m FROM adbc_null_map;

-- Succeeds and preserves the NULL key.
SELECT map_entries(m) FROM adbc_null_map;

Client side:

import adbc_driver_flightsql.dbapi as flight_sql

conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
                          db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT m FROM adbc_null_map")
cur.fetch_arrow_table()

Anything Else?

This is a genuine model mismatch between Doris MAP and Arrow MAP, so it needs an explicit decision rather than only a test skip: either serialize such maps as list<struct<key, value>> (which round-trips the NULL key, as map_entries already shows), or reject them with a clear, documented error at plan/schema time instead of failing mid-stream in DoGet. Either way the limitation belongs in the Arrow Flight SQL documentation.

Related: #65182 skipped the Arrow-incompatible Map null-key regression cases; this issue tracks the underlying behavior.

Workaround: use map_entries(m) in SQL to project the MAP as ARRAY<STRUCT<key, value>>, or filter/replace NULL keys when the business logic allows it.

Tracking issue: #65615

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_ad34727ee091

Initial triage

This is a confirmed server-side type-model incompatibility, not an ADBC/PyArrow client issue. The supplied reproducer is sufficient for triage, and source inspection at 31263df4dc1d4d3a27517d264802cd4d6b92c874 matches the reported error exactly. One NULL key makes the affected DoGet stream unreadable, although it does not mutate or corrupt stored data. The documented map_entries(m) workaround is lossless for this case.

The issue currently has no labels. Suggested labels are kind/bug and area/rpc; it is also already indexed by #65615.

Verified mechanism

1. convert_to_arrow_type() maps every Doris TYPE_MAP to arrow::MapType based only on the expression type.
2. The Flight result sink builds and registers that schema during init() / prepare(), before result rows are available.
3. Both the local and remote Flight readers convert Doris blocks to Arrow record batches during DoGet; for example, the local reader calls convert_to_arrow_batch() only after fetching a result block.
4. DataTypeMapSerDe::write_column_to_arrow() inspects the actual key null map at that point and returns INVALID_ARGUMENT with the exact reported message when it finds a NULL key.

This is required by Arrow's MAP layout: the Arrow format states that neither the entries field nor the key field may be nullable (Arrow schema definition). Doris, by contrast, deliberately stores MAP keys in nullable child columns; map() also constructs a nullable key type. Static inspection of the current local master still finds the same rejection path, so there is no apparent master-side fix yet, although I did not run a live cluster reproduction.

map_entries(m) works for a concrete reason: it changes the Doris result type to ARRAY<STRUCT<key, value>>, and its key struct field is nullable. Arrow then receives a List/Struct schema rather than logical MAP, so the NULL key is representable.

Design constraints and recommendation

A per-row or per-batch fallback from Arrow MAP to List/Struct is not valid: the Flight stream schema is established before batches and must remain stable. Similarly, rejecting only MAPs that will contain a NULL key at plan/schema time is generally not possible from the current Doris type alone; determining that for a table result requires inspecting data. A type-time rejection would therefore have to reject all raw MAP outputs (or introduce a stronger non-null-key contract), including maps whose actual keys are valid for Arrow.

The maintainer decision should therefore be explicit and query-wide:

• For lossless Doris semantics, map every Doris MAP result (recursively, including nested MAPs) to list<struct<key nullable, value nullable>> for Flight SQL, or initially expose that mapping behind an explicit connection/session compatibility option. This changes the client-visible Arrow type for existing non-NULL maps, so it needs a compatibility and release-note decision.
• If preserving the existing Arrow MAP type is more important, keep the limitation but reject conservatively before streaming where feasible and return a protocol-specific error that names the result column and recommends map_entries(). Document that Doris MAP values containing NULL keys are not representable. Do not silently drop, replace, or filter NULL-key entries.

Whichever policy is chosen, the stale comment in DataTypeMapSerDe saying the NULL entry is ignored should be corrected; the implementation returns an error.

Missing information

No additional logs, profile, or client metadata are needed to establish this mechanism. The exact adbc_driver_flightsql and PyArrow versions would only be useful for pinning an end-to-end regression environment, not for determining the root cause. Runtime confirmation on current master remains useful before closing the issue, but it is not blocking design or implementation.

Suggested next steps

1. Decide the stable Flight SQL mapping policy for Doris MAP and document it in the Arrow Flight SQL type-compatibility section.
2. Add an end-to-end arrow_flight_sql regression using the provided table case; assert both the returned Arrow schema and preservation of the NULL key under the chosen policy.
3. Cover a mixed map with NULL and non-NULL keys, a NULL value with a non-NULL key, a NULL key appearing in a later result batch, and nested MAPs. Exercise both local and remote result-reader paths if the test framework supports them.
4. Remove the blanket skip added by [test](regression) skip Arrow-incompatible complex outputs #65182 only when the selected behavior is implemented and verified; otherwise keep the skip linked to the documented limitation.