Repository navigation
Generate and test the typed Spark SQL surface - #65
Merged
estebanzimanyi merged 1 commit intoOct 3, 2026
Merged
Conversation
The build runs JMEOS's codegen_jvm.py a second time, --engine spark-sql,
into target/generated-sources/spark-sql, a source root beside the UDF
surface. MobilitySparkSql.registerAll registers that surface: each MEOS SQL
type a Spark type of its own over the form its catalog codec writes, and
each MobilityDB SQL name one registration whose builder chooses the
overload and its result type from the argument types while Spark plans the
call, generated from the overloads MobilityFlink's SQL surface uses.
GeneratedSqlSurfaceTest asserts it on a session registering that surface
alone, and the README states how it is built and called.
Witness. The UDF surface registers one udf() per name with one result type
over untyped bytes, so maxValue answers a tfloat alone and a tint fails
where PostgreSQL answers an int.
Why. Spark and MobilityFlink's Flink are generated from one model, so a
query and its answers move between the two engines unchanged; the test
asserts on Spark the queries MobilityFlink's GeneratedSqlSurfaceTest
asserts on Flink.
Measured. Built under Maven 3.9.16 with the generator of JMEOS
codegen/spark-sql-typed-surface, against MobilityDB 9b55b9737f, the
catalog of MEOS-API 5ba0cd8492 and the jar built from them: 31 tests pass,
the 8 of GeneratedSqlSurfaceTest among them, and the build prints no
warning. maxValue answers an int over a tint and a double over a tfloat,
lower('ABC') and round(2.5) stay Spark's, and the set-returning
signatures unfold through explode and inline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The build runs JMEOS's codegen_jvm.py a second time, --engine spark-sql,
into target/generated-sources/spark-sql, a source root beside the UDF
surface. MobilitySparkSql.registerAll registers that surface: each MEOS SQL
type a Spark type of its own over the form its catalog codec writes, and
each MobilityDB SQL name one registration whose builder chooses the
overload and its result type from the argument types while Spark plans the
call, generated from the overloads MobilityFlink's SQL surface uses.
GeneratedSqlSurfaceTest asserts it on a session registering that surface
alone, and the README states how it is built and called.
Witness. The UDF surface registers one udf() per name with one result type
over untyped bytes, so maxValue answers a tfloat alone and a tint fails
where PostgreSQL answers an int.
Why. Spark and MobilityFlink's Flink are generated from one model, so a
query and its answers move between the two engines unchanged; the test
asserts on Spark the queries MobilityFlink's GeneratedSqlSurfaceTest
asserts on Flink.
Measured. Built under Maven 3.9.16 with the generator of JMEOS
codegen/spark-sql-typed-surface, against MobilityDB 9b55b9737f, the
catalog of MEOS-API 5ba0cd8492 and the jar built from them: 31 tests pass,
the 8 of GeneratedSqlSurfaceTest among them, and the build prints no
warning. maxValue answers an int over a tint and a double over a tfloat,
lower('ABC') and round(2.5) stay Spark's, and the set-returning
signatures unfold through explode and inline.