feat(isthmus)!: preserve aggregate output types and semantics through Calcite - #1017
Conversation
db81ad0 to
4d7aa8f
Compare
|
@nielspardon This PR lets aggregate output types come either from the plan or from Calcite’s inference. I defaulted to the plan’s type because re-inference can change results, e.g. Keeping Which default do you prefer? |
nielspardon
left a comment
There was a problem hiding this comment.
Really nice piece of work — the problem statement is unusually clear and the design premise holds up.
I checked the two Calcite mechanics it rests on, because if either were wrong the wrapper would be
unnecessary complexity:
RelBuilder.AggCallImpl2.aggregateCall(...)re-creates every pre-builtAggregateCallwith
type = null(RelBuilder.java:4766-4773), so an explicit type genuinely cannot survive
RelBuilder.aggregate.- More fundamentally,
Aggregate's primary constructor asserts
typeMatchesInferred(aggCall, Litmus.THROW)(Aggregate.java:177), which Gradle test JVMs enable.
So a plainSqlAggFunctioncarrying a divergent stored type fails an assertion — the type must
travel on the operator's inference rule, and building aLogicalAggregatedirectly wouldn't help
either. Worth putting that inBoundSqlAggFunction's Javadoc; it's a stronger justification than
theRelBuilderone currently given.
I also validated the behaviour against the Substrait spec (local v0.98.0 checkout + upstream issue
history). Most of it is conformant, and two of the things that look like bugs here are actually
spec bugs — I've filed those upstream so they don't block you:
- substrait-io/substrait#1151 — the spec contradicts itself on
arguments/output_typefor
non-INITIAL_TO_RESULTphases.algebra.proto:1870-1872says "exactly the number of arguments
specified in the function definition" while:1855-1859says the inputs of an
INTERMEDIATE_TO_*phase are the intermediate values. Both were added by the same commit
(d4cfbe0, #231), and the author had described the intended[X] -> Y/[Y] -> Y/[Y] -> Z
model on #257 eight hours earlier. Your corevalidateIntermediateSignatureis the reading
upstream intended; it'salignArgumentsthat should move (see inline). - substrait-io/substrait#1150 —
quantile'sreturn: LIST?<any>uses a plainany, which carries
no identity, so its output type is underivable. It's the only standard-catalog aggregate your
deriveOutputTypefails on — worth listing as a known limitation and linking that issue.
Spec-conformant and verified, so no change needed: plain-any-binds-independently /
any1-identity; equalsIgnoringNullability (scalar_functions.md:128 licenses stripping only the
outermost nullability); the MIRROR override in both directions (scalar_functions.md:105 +
simple_extensions_schema.yaml:187-189 — "they will be ignored"; and tests/cases/boolean/or.test:5
asserts or(bool, bool) = bool independently); default MIRROR; DISCRETE exact matching; option
name/value case-insensitivity (algebra.proto:1876-1881 mandates it — pre-existing EnumArg.of is
the non-conformant one); unknown-option rejection; empty-preference rejection; variadic min
semantics; the INCONSISTENT branch; zero-groupings = one empty grouping set; and rejecting a
partial phase on a non-decomposable declaration.
On your open question — yes, keep PLAN_OUTPUT as the default. Silently narrowing
DECIMAL(38,2) to DECIMAL(10,2) changes results, and asking producers to know about a flag to get
their own types back is the sharper edge. Two things also make it less risky than the description
suggests: the "unrepresentable declared type throws" behaviour is unchanged from before (the old code
converted the declared type unconditionally too), and the wrapper is only added when the type
actually diverges or the invocation carries options/a phase.
About the length of this review
Up front: there are a lot of comments below, and I don't want that to read as a verdict on the
change. It isn't. This PR touches the function-binding, type-derivation and aggregate-conversion
surfaces all at once, and adds ~1000 non-test lines of new :core API — the comment count tracks
that surface area, not the quality of the work. Most items are a few lines; five come with a
ready-to-apply suggestion block and two more carry the replacement code inline.
Here's what I'd actually gate on, so the rest can be triaged or deferred without guessing:
Blocks merge — silent wrong answers (4)
| # | What |
|---|---|
| 2 | alignArguments drops the binding for any decomposable declaration whose arity isn't 1, so a partial count(*) silently comes back as a full count(i64). |
| 3 | UNSPECIFIED phase is treated as a full aggregation, contradicting algebra.proto:1834. |
| 5 | The splitter opt-out misses SqlSingletonAggFunction, so AggregateRemoveRule silently drops the binding's phase and options. |
| 8 | typeMatches's return true fall-through means EXTENSION_DECLARATION accepts plans the spec forbids. At minimum make the fallback throw. |
Worth doing in this PR (6) — 1 (wildcard predicates, spec conformance on new API), 4 (terminal
else), 6 (unwrapBound vs Calcite's assertion), 7 (unconditional inference), 10 (derive vs
resolveType, plus retargeting the five evaluator tests at the production path), 11 (the two
coverage Javadocs, which currently mis-describe what fails).
Fine as follow-ups (3 + the batched lists) — 9 (the intra-PR duplication), 12 (dead
required() guard), 13 (Immutables nit), the small-things list, and the ConverterProvider
architectural point, which is really a conversation about how it lands alongside #1035/#1036 rather
than a change to make here.
Of the missing tests, only the zero-arg count round trip (comment 2) is load-bearing — it's the
case that proves the fix. The rest can follow.
The recurring theme across the blocking four is the same: a few predicates and matchers are
permissive in ways that silently accept a plan rather than fail loudly. That matters more here than
it normally would, because EXTENSION_DECLARATION is sold as a validation mode — a validator that
returns true on a shape it doesn't understand is worse than one that refuses.
Happy to split this into two reviews (correctness now, polish later) if that's easier to work
through, or to pair on any of them.
Small things
alignArguments: hoist the loop-invariantif (!declaration.variadic().isPresent())out of the
while.validateOptionslowercases declared option names into a map per call; two declared names
differing only in case collide silently.- Strict validation derives the intermediate type twice per measure
(validateIntermediateSignatureplusoutputType()). ParameterBindings.isIntegerandReturnTypeEvaluator.resolveIntegerboth do
try { Integer.parseInt } catchon the same token — one helper returningOptionalIntwould do.
The.trim()is dead:SubstraitLexer.g4:10sends whitespace tochannel(HIDDEN).bindIntegersilently skips a numeric token rather than checking it against the actual value, so
a declaredDECIMAL<P,0>(factorial,functions_arithmetic_decimal.yaml:166-169) never
verifies that the actual scale is 0.- Tests:
assertEquals(false, ...)/assertEquals(true, ...)in
appliesMirrorNullabilityForScalars->assertFalse/assertTrue; the three "must not throw"
tests read better asassertDoesNotThrow; helper methods are interleaved with@Testmethods in
both new test classes. AggregateFunctionsimportsorg.checkerframework...Nullable, which nothing else in
isthmus/src/mainuses (core uses jspecify, and this PR uses jspecify inResolvedArgument).
Overridingequals(Object)/unwrap/getRollupdoesn't need it.- Architectural, worth a conversation rather than a change here:
AggregateConversionis threaded as
aSubstraitToCalciteconstructor parameter plus a secondConverterProviderfactory overload
plus anisDefault()dispatch branch.ConverterProvideris already the config carrier and is
gaining aBuilderin the stacked #1035/#1036 series — putting it there removes the extra
overload, theisDefault()branch, the "override both" warning, and the
new SubstraitToCalcite(provider, null, conversion)shape the tests are forced into.
Missing tests (batched comment)
- zero-arg
count@INTERMEDIATE_TO_RESULTround trip (comment 2) — the interesting phase case. - a declared
list<any1>against a non-list actual, andh(list<i32>, list<i32?>)(comment 8). UNSPECIFIEDphase (comment 3).CALCITE_INFERENCE+EXTENSION_DECLARATIONtogether.- a grouping-sets plan containing an explicit empty grouping — exercises the
hasEmptyGroupmirror
on the non-global path, currently only covered viaglobalAggregation. - the documented "
AGGREGATE_REDUCE_FUNCTIONSmay still drop the type" caveat isn't pinned, so a
future Calcite bump won't reveal if it degrades. - boundary test for the wildcard predicates (comment 1).
| private final @Nullable Type type; | ||
| private final @Nullable String enumValue; | ||
|
|
||
| private ResolvedArgument(Kind kind, @Nullable Type type, @Nullable String enumValue) { |
There was a problem hiding this comment.
Hand-written value class where its two same-PR siblings (ResolvedFunctionBinding,
ResolvedAggregateBinding) are @Value.Immutable with @Value.Check. 28 of ~69 lines are
equals/hashCode/toString/factories that Immutables would generate; FileOrFiles is the
precedent for the nested-discriminator-plus-Optional shape. Not blocking.
To be clear, this is not duplicating FunctionArg: Type extends FunctionArg, so a Type inside
a List<FunctionArg> already means "type argument" and you couldn't express "value argument
identified only by its type" there. That erasure is load-bearing for the staleness check, which
compares rebuilt ResolvedArguments rather than whole Expressions.
There was a problem hiding this comment.
Deferring as triaged (follow-up).
There was a problem hiding this comment.
AggregateConversion has the same shape — hand-written equals/hashCode/toString over two fields (:85, :98, :103) — so worth converting both in the same follow-up.
There was a problem hiding this comment.
Agreed — AggregateConversion goes into the same Immutables follow-up.
|
Two notes from the #379 side:
@rkondakov, on the open default question: I'd ship |
4d7aa8f to
82ce7dc
Compare
|
@nielspardon Thanks — the Calcite mechanics verification and the two upstream spec issues are hugely Blockers — all fixed, each with the test that proves it: #2 (intermediate-state phases align Worth-doing six — done: #1 (both predicates transcribe the lexer; Architectural point — done as well: #1035/#1036 are on main now, so Deferred as follow-ups, as triaged: #9 (the binder lives in Small things otherwise applied, including promoting two of them to real fixes with tests: the Default stays |
|
@alexandrefimov On #379: agreed, and thanks for flagging it before both halves got built — the aggregate observer On the default: staying with |
82ce7dc to
b4cd31f
Compare
|
Pushed a hardening round on top of the review fixes, from a deeper pass over the new surfaces
|
b4cd31f to
628039a
Compare
|
One more hardening pass over the wrapper/uniqueness surfaces (each item has a test; full build
|
| * @param converterProvider the converter provider containing configuration and converters | ||
| * @param aggregateConversion controls how aggregate output types are chosen and validated | ||
| */ | ||
| public SubstraitRelNodeConverter( |
There was a problem hiding this comment.
Its only caller passes what the two-arg form already derives — ConverterProvider:417 against :146. So this is a public constructor plus a second configuration channel that silently bypasses a subclass's getAggregateConversion(), for no call site that needs it. Cheaper to drop now than after release.
There was a problem hiding this comment.
Done — the three-argument constructor is gone: the two-argument one reads the provider's getAggregateConversion() directly, and ConverterProvider's factory calls it, so the provider is the single configuration channel and a subclass override always takes effect. Thanks also for the two calcite-core verifications (splittable-via-singleton subtyping and the type-blind AggregateCall hash) — good to have them pinned against 1.42.
There was a problem hiding this comment.
Thanks, this addresses the concern.
|
Checked two of the open threads against calcite-core 1.42 itself rather than the diff; both are answered by the current code.
|
70f32a8 to
919d761
Compare
|
Follow-up tightening on the enum-argument surface (tests included, full build green):
|
919d761 to
e7c14e3
Compare
nielspardon
left a comment
There was a problem hiding this comment.
Three documentation-only suggestions inline — comment and Javadoc text, no behaviour change. Nothing blocking from me.
nielspardon
left a comment
There was a problem hiding this comment.
LGTM with the small comment fixes
e7c14e3 to
44bde53
Compare
|
Pushed the round addressing the five comments above, plus a set of correctness fixes on the
One adjacent finding, out of scope here: core's proto→POJO conversion ( |
…ut types
Adds a resolved-binding model for extension function invocations, and makes
TypeExpressionEvaluator actually evaluate parameterized return types instead of
throwing.
ResolvedArgument keeps an argument's kind (value, type or enum) and its selected
enum option, so two invocations differing only by an enum argument — e.g.
std_dev(POPULATION, fp32) vs std_dev(SAMPLE, fp32) — are not conflated, and an
enum the plan left unspecified stays distinct from any specified option.
ResolvedFunctionBinding captures the semantic identity of an invocation: anchor,
ordered arguments and options. ResolvedAggregateBinding adds the aggregate phase
and invocation semantics, and selects the declaration's intermediate type for a
phase that stops at the intermediate state rather than its return type. A phase
that consumes intermediate state — including an unspecified phase, which the
AggregationPhase proto defines as implying INTERMEDIATE_TO_RESULT — takes
exactly that state as its single value argument, whatever the declaration's
arity.
FunctionBindingResolver keeps the two concerns apart. resolve() only captures
identity and performs no validation, so an invocation that merely differs from
its declaration still resolves. validate() opts into checking arity, argument
kinds and types, enum options, function options and the plan-declared output
type against the declaration. Validation fails closed rather than guessing: a
declared argument shape it cannot check structurally (lists, maps, structs,
function types) is rejected instead of silently accepted, a numeric literal
type parameter is a constraint the actual type must satisfy (DECIMAL<P,0> only
accepts a scale of exactly 0), and a declaration whose option names differ only
in case is rejected as ambiguous, since option names match case-insensitively.
A type argument is constrained by its declared pattern just as a value argument
is, and that pattern participates in parameter binding, so a declaration like
(<type: DECIMAL<P,S>>) -> DECIMAL<P,S> derives its return from the supplied
type. An INCONSISTENT variadic repetition binds no named parameters — each
repetition is independent — but a literal constraint still holds for every
repetition, not only the first.
TypeExpressionEvaluator previously threw UnsupportedOperationException("NYI")
for anything but an already concrete return type, so
SimpleExtension.Function.resolveType could not evaluate a parameterized
declaration at all. It now binds numbered wildcards (the any1 of
min(any1) -> any1) and integer type parameters (the P and S of DECIMAL<P,S>)
from the actual argument types and substitutes them, taking variadic parameter
consistency into account. Occurrences of one numbered wildcard must agree on a
single type, while each plain any binds independently; the wildcard family is
exactly the type grammar's any/any0-any9, so an ordinary parameter name that
merely starts with "any" is no longer treated as a wildcard, nor collapsed to
the "any" signature key. resolveType applies the declaration's nullability
policy (MIRROR over the value arguments), and the binding derivation delegates
to it, so there is a single production derivation path.
Derivations that are not supported — the parameterized type classes other than
decimal, and multi-line return programs — stay fail-closed: they raise rather
than fall back to a caller-supplied type, which is what makes a derived type
trustworthy enough to validate a plan against.
… Calcite A Substrait aggregate used to lose part of itself on the way to Calcite. The output type the plan declared was handed to AggregateCall.create, but RelBuilder re-infers a call's type from its operator, so the plan's type was replaced by Calcite's inference — sum(DECIMAL(10,2)), which the standard sum:dec declaration types as DECIMAL(38,2), came back as the argument's own width. Function options and the aggregation phase have no place in a Calcite aggregate call at all, so they were dropped, and converting back re-matched the operator against the extension catalog: among equally-shaped declarations it could only guess, it always produced a full INITIAL_TO_RESULT aggregation without options, and for a phase consuming an intermediate state it could not match at all, because the call's operands are then accumulator state rather than the declaration's arguments — a partial zero-argument count() silently came back as a full count(i64). Each converted measure now resolves a ResolvedAggregateBinding. Where Calcite cannot hold what the binding carries — the chosen output type differs from its inference; the invocation has options, a phase other than a full INITIAL_TO_RESULT aggregation, a type argument (only value arguments become Calcite operands), or an enum argument no operator kind encodes (only a single leading std_dev/variance distribution flag is operator-encoded, and only in the exact value and spelling the kind synthesizes and the declaration lists); or the declaration is not uniquely reconstructable — decided by consulting the same signature-key, signature-match and least-restrictive matchers the reverse direction runs, so a declaration another variant can shadow for these argument types and output type would come back decided by registration order — the operator is wrapped so that the binding and the type travel with it. The type has to travel on the operator's return-type inference: RelBuilder re-creates every pre-built call with no stored type, and Aggregate's constructor asserts typeMatchesInferred, so a stock operator carrying a divergent stored type is rejected outright. AggregateFunctionConverter rebuilds the invocation from the carried binding instead of re-matching, restoring enum and type arguments — which are not Calcite operands — from the binding. The binding records the plan as it was converted, so it is dropped again when a planner rule has since changed the call's arguments, and DISTINCT is always read off the Calcite call because a rule may legitimately have removed a redundant one. Calcite's return-type inference is consulted only to decide whether the plan's type diverges; when the binding forces the wrapper anyway it is skipped, and an inference failure under PLAN_OUTPUT counts as divergence rather than failing a conversion that never needed the inferred type. The aggregate-conversion policy is configured on the provider: ConverterProvider.builder().aggregateConversion(...) selects where the output type comes from (PLAN_OUTPUT preserves the plan's declared type, CALCITE_INFERENCE restores the previous behavior) and whether the plan is validated against the extension declaration (EXTENSION_DECLARATION), the two being independent. Two further conversion fixes come with it. Measures that are equal to Calcite but carry different types no longer collapse into a single column: AggregateCall equality ignores the stored type and RelBuilder deduplicates, so deduplication is now switched off for exactly those aggregates. And an aggregate with no groupings is built as one empty grouping set rather than none at all, so its measures are no longer re-inferred as if there were no empty group; converting such a plan back therefore spells the same global aggregation the way Calcite spells it, as a single empty grouping. BREAKING CHANGE: converting a Substrait aggregate to Calcite now preserves the plan's declared output type instead of Calcite's inferred one, and the resulting AggregateCall may carry a wrapper operator rather than the plain SqlAggFunction. Consumers that compare the operator by identity should inspect AggregateFunctions.boundBinding; before executing a converted plan, AggregateFunctions.unwrapBound(Aggregate) replaces bound calls with their delegates and re-infers their types. Rollup, splitting, singleton flattening and static flattening are disabled for a wrapped call, since all of them re-infer the transformed call's type from the underlying function or discard the call, losing the preserved type or binding; the wrapper delegates the volatility traits (isDeterministic, isDynamicFunction) so a volatile aggregate stays volatile. Configure ConverterProvider.builder().aggregateConversion(...) with OutputTypeSource.CALCITE_INFERENCE to restore the previous type behavior.
44bde53 to
2267eb9
Compare
Fixes #1016. Relates to #379.
What
A Substrait aggregate used to lose part of itself on the way to Calcite, and could not be
recovered on the way back. This PR gives an aggregate a resolved binding that travels with it, so
the plan's declared output type, its function options, its aggregation phase and the exact
extension declaration all survive a
Substrait → Calcite → Substraitround trip.Why — what was lost
AggregateCall.create, butRelBuilderre-infers a call's type from its operator, so Calcite's inference won.sum(DECIMAL(10,2)), whichsum:decdeclares asDECIMAL(38,2), came back with the argument's own width.AggregateCall.equalsignores the stored type andRelBuilderdeduplicates, so they collapsed into one column — the output arity changed.count(a)withoverflow: ERRORcame back without the option, and two counts differing only by an option were indistinguishable.INITIAL_TO_RESULT. A distributed plan with partial aggregates was silently turned into a plan of full aggregations.AGGREGATION_PHASE_UNSPECIFIEDis treated asINTERMEDIATE_TO_RESULT, as the proto specifies).count()silently came back as a fullcount(i64).How
Each converted measure resolves a
ResolvedAggregateBinding, which captures the semantic identityof the invocation: anchor, kind-aware arguments (value / type / enum, including the selected enum
option), options, phase and invocation semantics. Where Calcite cannot hold what the binding
carries — the chosen output type differs from its inference; the invocation has options, a phase
other than a full
INITIAL_TO_RESULTaggregation, a type argument (only value arguments becomeCalcite operands), or an enum argument no operator kind encodes (only a single leading std_dev/variance
distribution flag is operator-encoded); or the declaration is not uniquely reconstructable — reverse
re-matching resolves by signature key, falling back to wildcard/coercion matching, and picks by
registration order, so a declaration another variant can shadow for these argument types (e.g.
count(any)invoked withi32in a catalog that also declarescount(i32)) would come backdecided by load order — the aggregate operator is wrapped so that both the binding and the type
travel with it. The type has to travel on the operator's
return-type inference, not just on the
AggregateCall:RelBuilderre-creates every pre-builtcall with no stored type, and
Aggregate's constructor assertstypeMatchesInferred, so a plainoperator carrying a divergent stored type is rejected outright.
AggregateFunctionConverterthenrebuilds the invocation from that binding instead of re-matching the operator.
The binding records the plan as it was converted, so it is dropped again when a planner rule has
since changed the call's arguments, and
DISTINCTis always read off the Calcite call, because arule may legitimately have removed a redundant one. The wrapper opts out of the type-rederiving
operator hooks until they are binding-aware: rollup (
getRollup()), splitting(
SqlSplittableAggFunction), singleton flattening (SqlSingletonAggFunction, whichAggregateRemoveRuleunwraps) and static flattening (SqlStaticAggFunction, through which thatrule and
RelBuilder's already-unique optimization rewrite a call into a constant) — and itdelegates the volatility traits (
isDeterministic/isDynamicFunction) so a volatile aggregatestays volatile.
Supporting this needed
TypeExpressionEvaluatorto actually work: it used to throwUnsupportedOperationException("NYI")for anything but an already concrete return type. It nowbinds numbered wildcards (the
any1ofmin(any1) -> any1) and integer type parameters (thePand
SofDECIMAL<P,S>) from the actual argument types and substitutes them, honouring variadicparameter consistency; a numeric token like the
0ofDECIMAL<P,0>is a literal constraint theactual type must satisfy, not a parameter.
SimpleExtension.Function.resolveTypeapplies thedeclaration's nullability policy (
MIRRORover the value arguments), and the binding derivationdelegates to it, so there is one production path. Unsupported derivations stay fail-closed and
never fall back to a caller-supplied type — on the standard catalog that means the parameterized
type classes other than decimal (
varchar<L1>,fixedchar<L1>,precision_*<P>,interval_day<P>,list<anyN>, parameterized structs) and multi-line return programs;concat,assume_timezoneandstrptime_*are rejected today.quantileis the one standard aggregatewhose output type cannot be derived at all: its
LIST?<any>return uses a plainany, whichcarries no identity to bind (substrait-io/substrait#1150 tracks the spec fix). For phases that
consume intermediate state,
:corefollows the argument model upstream intended(substrait-io/substrait#1151): such an invocation carries exactly the accumulator state, not the
declaration's arguments.
Commits
feat(core)— the resolved-binding model (ResolvedArgument,ResolvedFunctionBinding,ResolvedAggregateBinding),FunctionBindingResolver(resolve vs. opt-in validate), and aworking
TypeExpressionEvaluator. Self-contained;:core:buildand:isthmus:buildare greenat this commit alone.
feat(isthmus)!— conversion changes: theAggregateConversionconfiguration onConverterProvider, the transport wrapper inAggregateFunctions, and the two conversionfixes (deduplication, global aggregate).
New configuration
AggregateConversionis configured on the provider —ConverterProvider.builder().aggregateConversion(...)— and has two independent settings:OutputTypeSource—PLAN_OUTPUT(new default) preserves the plan's declared type;CALCITE_INFERENCErestores the previous type behavior.FunctionBindingValidation—NONE(default) does not check the plan against the extensiondeclaration;
EXTENSION_DECLARATIONrequires the declared output type to match the derived oneand rejects the plan otherwise. Validation fails closed: a shape it cannot check structurally
(e.g. a declared
list<any1>) is rejected, not silently accepted.The default is
PLAN_OUTPUT+NONE: conversion never silently changes a type, but it also doesnot assert that the plan is spec-compliant.
Note that
CALCITE_INFERENCEdoes not make the wrapper fully opt-in: an invocation carryingoptions, a non-full aggregation phase, a type argument or a non-operator-encoded enum argument —
or naming a declaration another variant can shadow on the reverse path — is wrapped in every mode, since no stock Calcite
operator can express those. The output-type setting governs only the wrappers caused by a type
divergence.
Default:
PLAN_OUTPUT(resolved per review discussion)A converter silently changing a plan's declared type is a correctness problem —
DECIMAL(38,2)becomingDECIMAL(10,2)changes results — and it is what #1016 asks to fix bydefault; the PR is marked breaking accordingly. The compatibility costs are real and remain:
operator identity changes for wrapped calls (
call.getAggregation() == SUM-style comparisonsfail; use
AggregateFunctions.boundBinding/unwrapBound), and rollup / splitting / singletonflattening are disabled for them.
CALCITE_INFERENCErestores the previous type behavior forcallers that prefer it.
Breaking changes / migration
one. Configure
ConverterProvider.builder().aggregateConversion(new AggregateConversion( OutputTypeSource.CALCITE_INFERENCE, FunctionBindingValidation.NONE))to restore the previousbehavior.
AggregateCallmay carry a wrapper operator rather than the plainSqlAggFunction.Consumers comparing the operator by identity should inspect
AggregateFunctions.boundBinding;before executing a converted plan,
AggregateFunctions.unwrapBound(Aggregate)replaces boundcalls with their delegates and honestly re-infers their types (the carried type cannot survive
unwrapping —
Aggregate's constructor asserts the stored type against the operator's inference).all of them re-infer the transformed call's type from the underlying function or discard the
call, losing the carried type or binding. Rules that match on
SqlKindinstead (e.g.AGGREGATE_REDUCE_FUNCTIONS) haveno operator hook to veto and may still rewrite the call and drop the binding; this is pinned by
a test so a Calcite upgrade that changes it is noticed.
grouping instead of none — the same global aggregation, spelled the way Calcite spells it.
ParameterizedType.StringLiteral.isWildcard()now follows the type grammar exactly(
any/any0–any9, case-insensitive). A third-party declaration using an ordinary parametername that merely starts with
any(e.g.f(anything)) previously matched every argument typeand computed the same signature key as
f(any); it now behaves like any other named parameter.No shipped extension is affected — the standard catalog only uses
any,any1,any2.Spark
spark'sToAggregateFunction/ToLogicalPlanimplement a third argument model for phases(declared arguments in every phase, phase as orthogonal metadata, with a
count()→count(1)rewrite hack). This PR does not touch it; aligning Spark with theintermediate-state model that
:corenow enforces is follow-up work.Relation to #379
Aggregate function conversion is one of the four Substrait-vs-Calcite type-mismatch points listed
in #379 (scalars and windows were addressed by #1015 and #1059 via
TypeObserver). This PRresolves the consequence of the mismatch for aggregates — the plan's declared type is preserved
instead of silently replaced — but does not wire the aggregate observation point itself:
TypeObservationcarries anExpression, which anAggregateFunctionInvocationis not, sowidening it is deferred to a #379 follow-up. The declared-vs-inferred comparison lives in one
place in
SubstraitRelNodeConverterso that follow-up can attach the observer without recomputingthe inference (note it is skipped when opaque semantics force the wrapper regardless, and an
inference failure under
PLAN_OUTPUTcounts as "diverges" instead of failing the conversion).Testing
:core—FunctionBindingResolverTest,ResolvedAggregateBindingTest,TypeExpressionEvaluatorTest,ToTypeStringTestwildcard boundaries, plus a smallbinding_extensions.yamlfor wildcard / literal / option-collision / nested-shape cases.:isthmus— cases inSubstraitRelNodeConverterTest.Aggregatecovering declared decimal width,global aggregation and explicit empty grouping sets, operator identity, enum arguments, options,
phases (including a parameterized intermediate state, a zero-argument
count()atINTERMEDIATE_TO_RESULT, andUNSPECIFIED), duplicate measures, the rollup / split / remove /reduce rules, both validation modes and their combination, and unwrapping under assertions; a
CustomFunctionTestcase covers an operator whose return-type inference fails.