Skip to content

Generate and test the typed Spark SQL surface - #65

Merged
estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/typed-sql-surface
Oct 3, 2026
Merged

estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/typed-sql-surface

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

The build runs JMEOS's codegen_jvm.py a second time, --engine spark-sql,
into target/generated-sources/spark-sql, a source root beside the UDF
surface. MobilitySparkSql.registerAll registers that surface: each MEOS SQL
type a Spark type of its own over the form its catalog codec writes, and
each MobilityDB SQL name one registration whose builder chooses the
overload and its result type from the argument types while Spark plans the
call, generated from the overloads MobilityFlink's SQL surface uses.
GeneratedSqlSurfaceTest asserts it on a session registering that surface
alone, and the README states how it is built and called.

Witness. The UDF surface registers one udf() per name with one result type
over untyped bytes, so maxValue answers a tfloat alone and a tint fails
where PostgreSQL answers an int.

Why. Spark and MobilityFlink's Flink are generated from one model, so a
query and its answers move between the two engines unchanged; the test
asserts on Spark the queries MobilityFlink's GeneratedSqlSurfaceTest
asserts on Flink.

Measured. Built under Maven 3.9.16 with the generator of JMEOS
codegen/spark-sql-typed-surface, against MobilityDB 9b55b9737f, the
catalog of MEOS-API 5ba0cd8492 and the jar built from them: 31 tests pass,
the 8 of GeneratedSqlSurfaceTest among them, and the build prints no
warning. maxValue answers an int over a tint and a double over a tfloat,
lower('ABC') and round(2.5) stay Spark's, and the set-returning
signatures unfold through explode and inline.

The build runs JMEOS's codegen_jvm.py a second time, --engine spark-sql,
into target/generated-sources/spark-sql, a source root beside the UDF
surface. MobilitySparkSql.registerAll registers that surface: each MEOS SQL
type a Spark type of its own over the form its catalog codec writes, and
each MobilityDB SQL name one registration whose builder chooses the
overload and its result type from the argument types while Spark plans the
call, generated from the overloads MobilityFlink's SQL surface uses.
GeneratedSqlSurfaceTest asserts it on a session registering that surface
alone, and the README states how it is built and called.

Witness. The UDF surface registers one udf() per name with one result type
over untyped bytes, so maxValue answers a tfloat alone and a tint fails
where PostgreSQL answers an int.

Why. Spark and MobilityFlink's Flink are generated from one model, so a
query and its answers move between the two engines unchanged; the test
asserts on Spark the queries MobilityFlink's GeneratedSqlSurfaceTest
asserts on Flink.

Measured. Built under Maven 3.9.16 with the generator of JMEOS
codegen/spark-sql-typed-surface, against MobilityDB 9b55b9737f, the
catalog of MEOS-API 5ba0cd8492 and the jar built from them: 31 tests pass,
the 8 of GeneratedSqlSurfaceTest among them, and the build prints no
warning. maxValue answers an int over a tint and a double over a tfloat,
lower('ABC') and round(2.5) stay Spark's, and the set-returning
signatures unfold through explode and inline.
@estebanzimanyi
estebanzimanyi merged commit 5b4b7b1 into MobilityDB:main Oct 3, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the feat/typed-sql-surface branch October 3, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant