[836] xtable-spark-runtime: XTableSyncListener for in-job metadata sync (listener)#839
Draft
vinishjail97 wants to merge 4 commits into
Draft
[836] xtable-spark-runtime: XTableSyncListener for in-job metadata sync (listener)#839vinishjail97 wants to merge 4 commits into
vinishjail97 wants to merge 4 commits into
Conversation
ConversionTargetFactory loads targets via ServiceLoader and eagerly instantiates every registered target when resolving any one of them. With a thin runtime where engines are provided, that hard-fails (e.g. DeltaKernelConversionTarget needs io.delta.kernel) even for a sync that never touches Delta. Skip providers whose engine library is not on the classpath so a subset of engines can be used (e.g. Hudi + Iceberg, no Delta). Also make the Delta-kernel check name-based to avoid loading DeltaKernelConversionTarget (and its io.delta.kernel dependencies) when Delta is absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New xtable-spark-runtime module producing a thin, drop-in bundle for in-job XTable metadata sync, plus RFC-3. Packaging (the Hudi/Iceberg model): - Spark, Hadoop, and the engine libs (Hudi/Iceberg/Delta) are provided; the user brings their own engine versions and the bundle stays compatible across them. - Curated shade allowlist: only XTable's own modules + guava/protobuf (relocated under org.apache.xtable.shaded.*). Libraries exchanged with the engines (avro/parquet/jackson/commons) are NOT relocated/bundled - relocating avro breaks the boundary (Hudi returns a real avro Schema). - Resulting bundle: ~4 MB. Includes XTableSparkSync, a standalone spark-submit entry point (the RunSync-equivalent for this bundle), and TableSyncSpec/XTableSyncService shared with the listener (stacked follow-up PR). Validation: ITXTableSparkRuntimeBundle spark-submits the shaded jar via $SPARK_HOME with Hudi + Iceberg supplied through --packages (versions from the root pom) and asserts the Iceberg metadata is produced. Skipped when SPARK_HOME is unset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stacked on the packaging PR. Adds the driver-side listener that runs an incremental ConversionController.sync after a successful write to a configured source table, activated via spark.xtable.* config only. - XTableSparkConfig: parse spark.xtable.* into TableSyncSpec (path- and name-based table selection). - PlanTargetResolver: best-effort write-path extraction from the plan. - XTableSyncListener: stateless, synchronous QueryExecutionListener that syncs on a recognized write matching a configured table; single-flight is inherent (single-threaded LiveListenerBus), per-table failures are caught as Throwable so a listener never destabilizes the bus. - Unit tests + ITXTableSyncListener (embedded local[*], Hudi -> Delta/Iceberg). Reuses TableSyncSpec/XTableSyncService from the packaging PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vinishjail97
force-pushed
the
836-spark-runtime-listener
branch
from
1d9c7db to
8f4e3a8
Compare
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
Second of two PRs splitting #837 into packaging and listener. This is the listener PR, stacked on the packaging PR #838 — review/merge #838 first. (Until #838 merges, the diff here shows both; it'll reduce to just the listener once #838 lands and this rebases.)
Adds the driver-side listener that runs an incremental
ConversionController.sync(...)in-job after a successful write, activated viaspark.xtable.*config only.Brief change log
XTableSparkConfig— parsespark.xtable.*intoTableSyncSpec(path- and name-based table selection).PlanTargetResolver— best-effort write-path extraction from the analyzed plan.XTableSyncListener— stateless, synchronousQueryExecutionListener: syncs inline on a recognized write matching a configured table; single-flight is inherent (single-threadedLiveListenerBus); per-table failures caught asThrowableso a listener never destabilizes the bus.TableSyncSpec/XTableSyncServicefrom [836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery #838.Usage
Verify this pull request
Unit tests (
TestXTableSparkConfig,TestPlanTargetResolver,TestXTableSyncListener) +ITXTableSyncListener(embeddedlocal[*]: a Hudi datasource write triggers the listener and syncs to Delta + Iceberg in-JVM, asserting theidsets match across all three formats).Part of #836.
🤖 Generated with Claude Code