[FLINK-39904][table] Add GeoParquet-compatible GEOGRAPHY Parquet support - #28832
Draft
davidchaava wants to merge 23 commits into
Draft
[FLINK-39904][table] Add GeoParquet-compatible GEOGRAPHY Parquet support#28832davidchaava wants to merge 23 commits into
davidchaava wants to merge 23 commits into
Conversation
Collaborator
gkalashyan-akv
force-pushed
the
task/geography-parquet-mapping
branch
from
July 28, 2026 15:56
6e9e0ab to
afe0a99
Compare
Author
|
@flinkbot run azure |
davidchaava
force-pushed
the
task/geography-parquet-mapping
branch
10 times, most recently
from
July 29, 2026 17:35
d226f9d to
69e9770
Compare
davidchaava
force-pushed
the
task/geography-parquet-mapping
branch
from
July 30, 2026 13:48
69e9770 to
f5b3e13
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This PR adds Parquet read/write support for the GEOGRAPHY logical type using GeoParquet-compatible metadata on top of Flink's existing Parquet version.
GEOGRAPHY values are stored as Parquet
BINARYvalues with WKB encoding. The writer emits GeoParquetgeokey-value metadata withencoding=WKBandedges=spherical, allowing downstream systems such as BigQuery to recognize the column as GEOGRAPHY.This keeps the implementation compatible with Flink's current Parquet dependency (
parquet-mr 1.15.2) and does not require upgrading to the native Parquet GEOGRAPHY logical type.Brief change log
BINARY.Related specifications
This PR follows the GeoParquet 1.1.0 specification for file-level
geometadata.GEOGRAPHY values are stored as Parquet
BINARYwith GeoParquet metadata usingencoding=WKBandedges=spherical. This keeps the implementation compatible with Flink's current Parquet dependency (parquet-mr 1.15.2) instead of relying on the native Parquet GEOGRAPHY logical type.Dependencies
This PR is stacked on top of:
Only the Parquet-specific changes should be reviewed here. Earlier GEOGRAPHY logical type, SQL function, and PyFlink changes are included only because this is a stacked branch.
Related changes
The SQL ST functions are tracked separately in apache/flink#28788.
This PR focuses on Parquet format interoperability for GEOGRAPHY. The ST functions PR provides the SQL constructor/accessor functions used by user-facing SQL workflows.
Verifying this change
This change is covered by unit tests in
flink-parquet.I also validated a generated GeoParquet file manually with a local Flink pipeline:
Wrote GEOGRAPHY values through the Flink filesystem connector with
format='parquet'.Read the same Parquet data back through Flink SQL.
Verified the generated file contains GeoParquet 1.1.0 metadata and was written with Flink's current Parquet version:
I additionally loaded the generated Parquet file into BigQuery and verified that BigQuery recognized the column as GEOGRAPHY.
BigQuery test table:
BigQuery schema:
Spatial query validation:
Result:
Does this pull request potentially affect one of the following parts
Documentation
This PR does not add end-user documentation. GEOGRAPHY SQL functions and broader user-facing SQL behavior are handled in the related ST functions PR.