Repository navigation
Conversation
map_entries declared its result as List(Struct(key, value)) with a nullable `value` field, but wrapped the input map's entries array unchanged. When the input's entry fields differ from that struct, ListArray::new panics. This happens for maps with non-nullable values, such as the `ints` column of parquet_map.parquet, for entry fields named something other than `key`/`value`, and for entry fields that carry metadata. Rebuild the entries StructArray with the declared result fields, reusing the input's child arrays, so the result always matches the declared type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
map_entries()panics on a valid map with non-nullable values #27012.Rationale for this change
map_entriespanics when the input map's entry fields are not exactlykey(non-null) andvalue(nullable) with no metadata. One such case is a map whose values are declared non-nullable. The repo's ownparquet_map.parquethas such a column, and on 55.2.0 it brings downdatafusion-cli:The same panic happens when the entry fields have other names. One example is
keys/values, the arrow-rsMapBuilderdefault:It also happens when the entry fields carry metadata, such as
PARQUET:field_id.map_keys,map_values,map_extract,cardinalityandm['k']all work on the same inputs.The cause is that
map_entriesdeclares its result asList(Struct(key, value))with a nullablevaluefield, asreturn_typedoes, but wraps the input's entries array unchanged.ListArray::newthen panics because the two struct types differ.What changes are included in this PR?
map_entries_innerrebuilds the entriesStructArraywith the declaredkey/valuefields. It reuses the input's child arrays, so no data is copied. The result always matches the declared return type, and that type does not change.One alternative would be to carry a non-nullable value through to the return type, as
map_valuesdoes since #15901. That changes the result type for those inputs, and it would still need the rebuild for other field names or metadata. So I kept the existing declared type. Happy to switch if reviewers prefer the other approach.What is the testing strategy for this PR?
New cases in
map.slt:map_entries(ints)on thedatatable (parquet_map.parquet, non-nullable values). It now returns[{key: bytes, value: 49960}]and[{key: bytes, value: 49689}]for the top two rows.map_entrieson a map cast to non-nullable values, and on a map cast tokeys/valuesfield names.arrow_typeofof the result, to pin that the declared type is unchanged:List(non-null Struct("key": non-null Utf8, "value": Int64)).Each new query panics without the change to
map_entries.rs, with the error from the issue. Locally I ranmap.sltand thedatafusion-functions-nestedunit tests. I left the full workspace and extended test runs to CI.Are there any user-facing changes?
map_entriesno longer panics on these maps and returns the entries. There is no API change and the result type is unchanged.