handle missing fields in SQL as null

[mccanne]

This commit improves SQL semantics in the face of missing fields by turning them into nulls and adding a nullish coalescing variant of the . operator to propagate nulls in a path expression. This operator is inserted by the semantic pass and is not user visible. It is displayed in DAG output as "??.".

To make this work, we fixed a problem in the optimizer where the flags in field.Chain weren't being propagated.

This also fixed a problem in agg functions like count which are supposed to ignore nulls. The change to change-star.yaml reflects this, where this test is now has a SQL-compatible result.

Closes #5984

[philrz]

A run of this branch at 7c021ad on the current sqllogic-ztests triggered 124 new similar failures in the "groupby" set, such as groupby/slt_good_0/q2177.yaml. Here's a simplified, self-contained repro:

$ super -version &&
  super -c "SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y"

Version: v0.3.0-379-g7c021ad3e

column "x" must appear in GROUP BY clause at line 1, column 8:
SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y
       ~
column "y" must appear in GROUP BY clause at line 1, column 8:
SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y
       ~

Whereas that works ok on current tip of main.

$ super -version &&
  super -c "SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y"

Version: v0.3.0-378-g2338849bb

{x:1,y:2}

I had Claude come up with the simplified repro, so it also gave its take on root cause and possible fixes, and that's available in a Gist if it helps at all.

[philrz]

Another finding along the way that's mentioned in that Gist is that on this branch the null type is disappearing from unions when I do SELECT *.

$ echo '
    {col0:22::(int64|null),col1:6::(int64|null),col2:8::(int64|null)}
    {col0:28::(int64|null),col1:57::(int64|null),col2:45::(int64|null)}
    {col0:82::(int64|null),col1:44::(int64|null),col2:71::(int64|null)}' > tab1.sup &&
  super -version &&
  super -c "SELECT * FROM tab1.sup"

Version: v0.3.0-379-g7c021ad3e

{col0:22,col1:6,col2:8}
{col0:28,col1:57,col2:45}
{col0:82,col1:44,col2:71}

Whereas they're maintained on tip of main.

$ super -version &&
  super -c "SELECT * FROM tab1.sup"

Version: v0.3.0-378-g2338849bb

{col0:22::(int64|null),col1:6::(int64|null),col2:8::(int64|null)}
{col0:28::(int64|null),col1:57::(int64|null),col2:45::(int64|null)}
{col0:82::(int64|null),col1:44::(int64|null),col2:71::(int64|null)}

[mccanne]

I just pushed a fix for this...

$ super -version &&
  super -c "SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y"

Version: v0.3.0-379-g7c021ad3e

column "x" must appear in GROUP BY clause at line 1, column 8:
SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y
       ~
column "y" must appear in GROUP BY clause at line 1, column 8:
SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y
       ~

[mccanne]

Another finding along the way that's mentioned in that Gist is that on this branch the null type is disappearing from unions when I do SELECT *.

Let's fix this in a subsequent PR. I think we can solve the more general problem of retaining typed NULLs throughout the runtime, which likely involves a small change to vector.Apply.

[mccanne]

rebased to new main

[nwt]

Nit: Don't delete the newline here.

[nwt]

Nits: Rename this to addChainToExpr. And update the doc comment (s/path/chain/g more or less).

[nwt]

Nit: The issue described in #5984 isn't tied to Parquet specifically so there's no need to use it here.

Suggested change

	super -f parquet -o integers.parquet integers.sup
	super -f parquet -o integers2.parquet integers2.sup
	super -s -c "SELECT * FROM integers.parquet LEFT OUTER JOIN integers2.parquet ON integers.i=integers2.k ORDER BY i;"
	super -s -c "SELECT * FROM integers.sup LEFT OUTER JOIN integers2.sup ON integers.i=integers2.k ORDER BY i;"

[nwt]

Nit: I think op would be a better name for this.