feat(core)!: support nested map expressions and import nested structs - #1062
feat(core)!: support nested map expressions and import nested structs#1062nielspardon wants to merge 1 commit into
Conversation
| * | ||
| * @return the key-value pairs | ||
| */ | ||
| public abstract Map<Expression, Expression> values(); |
There was a problem hiding this comment.
Could we avoid representing key_values as Map<Expression, Expression> here? Substrait maps allow repeated keys. During proto import, LinkedHashMap.put() overwrites an earlier entry when two key expressions are equal, so a valid plan is changed by a round trip. I reproduced this with two i32(1) keys and different values: the output had one key_values entry instead of two.
An ordered list of key/value pair objects would preserve repeated keys and serialization order. The existing literal-map POJO may have the same limitation, but I do not think we should add it to this new public API.
There was a problem hiding this comment.
Good catch, and reproduced — two i32(1) keys with different values came back as one entry, and it was the first pair that was dropped. NestedMap now holds List<NestedMap.KeyValue> rather than a Map, mirroring the proto's repeated KeyValue key_values, so repeated keys and pair order both survive.
#1063 needed the same treatment in the other direction: MAP['a', 1, 'a', 2] was collapsing to a MapLiteral and losing a pair, so that collapse now requires distinct keys too.
You're right about the literal map, and it's a bit worse than lossy — proto import there uses Collectors.toMap, which throws IllegalStateException on a duplicate key. Agreed it doesn't belong in this PR; filed as #1081.
Proto `Expression.Nested` has three arms — struct, list and map — but only `list` round-tripped. `Expression.NestedMap` did not exist at all, so the `map` arm was unreachable, and `ProtoExpressionConverter.from(Nested)` handled only `case LIST`: a plan carrying a nested struct failed on import with `UnsupportedOperationException: Unimplemented nested type: STRUCT`, even though the POJO to proto direction emitted one. (`VirtualTableScan` rows were unaffected — they use the separate `from(Nested.Struct)` overload.) Adds the `NestedMap` POJO with the usual visitor and converter wiring, and imports all three nested kinds on the way back from proto. A nested map holds its pairs as an ordered `List<NestedMap.KeyValue>`, mirroring the proto `repeated KeyValue key_values`: a map expression may repeat a key, and a `Map`-based representation would silently drop one of the pairs on import. `NestedMap.check()` rejects an empty map and heterogeneous key or value types with `IllegalArgumentException`. Isthmus still cannot convert nested structs or maps to and from Calcite; that follows separately. Part of #375 BREAKING CHANGE: `ExpressionVisitor` gains `visit(Expression.NestedMap)`. Direct implementors must add it; implementors extending `AbstractExpressionVisitor` inherit the `visitFallback` default and need no change.
807d49c to
91ff10d
Compare
Proto
Expression.Nestedhas three arms — struct, list and map — but onlylistround-tripped.Expression.NestedMapdid not exist at all, so themaparm was unreachable, andProtoExpressionConverter.from(Nested)handled onlycase LIST: a plan carrying a nested struct failed on import withUnsupportedOperationException: Unimplemented nested type: STRUCT, even though the POJO to proto direction emitted one. (VirtualTableScanrows were unaffected — they use the separatefrom(Nested.Struct)overload.)This adds the
NestedMapPOJO with the usual visitor and converter wiring, and imports all three nested kinds on the way back from proto.Two details worth calling out:
Nested.Mapis arepeated KeyValue key_valueswith no uniqueness constraint, so a map expression may repeat a key, and the order of the pairs is part of the value.NestedMapholdsList<NestedMap.KeyValue>accordingly. AMap-backed accessor turned a valid plan into a different one on import, since the second of two equal keys overwrote the first and a pair disappeared. The pre-existingMapLiteralhas the same limitation, tracked separately in fix(core)!: MapLiteral cannot represent a map with repeated keys #1081.@Value.Checkthrows rather than asserts.NestedList, written earlier, usesassertfor the same invariants, which does not run outside a-eaJVM.NestedMaprejects an empty map (pointing atExpressionCreator.emptyMap) and heterogeneous key or value types withIllegalArgumentException, matching the direction of Replaceassert-based validation with real exceptions in:coreand:isthmus#1047.Isthmus still cannot convert nested structs or maps to and from Calcite; that follows in a stacked PR, which closes #375.
Part of #375
BREAKING CHANGE:
ExpressionVisitorgainsvisit(Expression.NestedMap). Direct implementors must add it; implementors extendingAbstractExpressionVisitorinherit thevisitFallbackdefault and need no change.🤖 Generated with AI