Skip to content

feat(core)!: support nested map expressions and import nested structs - #1062

Open
nielspardon wants to merge 1 commit into
mainfrom
feat/core-nested-map-expressions
Open

feat(core)!: support nested map expressions and import nested structs#1062
nielspardon wants to merge 1 commit into
mainfrom
feat/core-nested-map-expressions

Conversation

@nielspardon

@nielspardon nielspardon commented Aug 5, 2026

Copy link
Copy Markdown
Member

Proto Expression.Nested has three arms — struct, list and map — but only list round-tripped. Expression.NestedMap did not exist at all, so the map arm was unreachable, and ProtoExpressionConverter.from(Nested) handled only case LIST: a plan carrying a nested struct failed on import with UnsupportedOperationException: Unimplemented nested type: STRUCT, even though the POJO to proto direction emitted one. (VirtualTableScan rows were unaffected — they use the separate from(Nested.Struct) overload.)

This adds the NestedMap POJO with the usual visitor and converter wiring, and imports all three nested kinds on the way back from proto.

Two details worth calling out:

  • The pairs are a list, not a map. Proto Nested.Map is a repeated KeyValue key_values with no uniqueness constraint, so a map expression may repeat a key, and the order of the pairs is part of the value. NestedMap holds List<NestedMap.KeyValue> accordingly. A Map-backed accessor turned a valid plan into a different one on import, since the second of two equal keys overwrote the first and a pair disappeared. The pre-existing MapLiteral has the same limitation, tracked separately in fix(core)!: MapLiteral cannot represent a map with repeated keys #1081.
  • @Value.Check throws rather than asserts. NestedList, written earlier, uses assert for the same invariants, which does not run outside a -ea JVM. NestedMap rejects an empty map (pointing at ExpressionCreator.emptyMap) and heterogeneous key or value types with IllegalArgumentException, matching the direction of Replace assert-based validation with real exceptions in :core and :isthmus #1047.

Isthmus still cannot convert nested structs or maps to and from Calcite; that follows in a stacked PR, which closes #375.

Part of #375

BREAKING CHANGE: ExpressionVisitor gains visit(Expression.NestedMap). Direct implementors must add it; implementors extending AbstractExpressionVisitor inherit the visitFallback default and need no change.

🤖 Generated with AI

*
* @return the key-value pairs
*/
public abstract Map<Expression, Expression> values();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid representing key_values as Map<Expression, Expression> here? Substrait maps allow repeated keys. During proto import, LinkedHashMap.put() overwrites an earlier entry when two key expressions are equal, so a valid plan is changed by a round trip. I reproduced this with two i32(1) keys and different values: the output had one key_values entry instead of two.

An ordered list of key/value pair objects would preserve repeated keys and serialization order. The existing literal-map POJO may have the same limitation, but I do not think we should add it to this new public API.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and reproduced — two i32(1) keys with different values came back as one entry, and it was the first pair that was dropped. NestedMap now holds List<NestedMap.KeyValue> rather than a Map, mirroring the proto's repeated KeyValue key_values, so repeated keys and pair order both survive.

#1063 needed the same treatment in the other direction: MAP['a', 1, 'a', 2] was collapsing to a MapLiteral and losing a pair, so that collapse now requires distinct keys too.

You're right about the literal map, and it's a bit worse than lossy — proto import there uses Collectors.toMap, which throws IllegalStateException on a duplicate key. Agreed it doesn't belong in this PR; filed as #1081.

Proto `Expression.Nested` has three arms — struct, list and map — but only
`list` round-tripped. `Expression.NestedMap` did not exist at all, so the
`map` arm was unreachable, and `ProtoExpressionConverter.from(Nested)`
handled only `case LIST`: a plan carrying a nested struct failed on import
with `UnsupportedOperationException: Unimplemented nested type: STRUCT`,
even though the POJO to proto direction emitted one. (`VirtualTableScan`
rows were unaffected — they use the separate `from(Nested.Struct)` overload.)

Adds the `NestedMap` POJO with the usual visitor and converter wiring, and
imports all three nested kinds on the way back from proto. A nested map
holds its pairs as an ordered `List<NestedMap.KeyValue>`, mirroring the
proto `repeated KeyValue key_values`: a map expression may repeat a key, and
a `Map`-based representation would silently drop one of the pairs on import.
`NestedMap.check()` rejects an empty map and heterogeneous key or value types
with `IllegalArgumentException`.

Isthmus still cannot convert nested structs or maps to and from Calcite;
that follows separately.

Part of #375

BREAKING CHANGE: `ExpressionVisitor` gains `visit(Expression.NestedMap)`.
Direct implementors must add it; implementors extending
`AbstractExpressionVisitor` inherit the `visitFallback` default and need no
change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support for Nested expressions

2 participants