[836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery by vinishjail97 · Pull Request #838 · apache/incubator-xtable

vinishjail97 · 2026-07-07T20:22:33Z

What is the purpose of the pull request

First of two PRs splitting #837 (per review feedback) into packaging and listener. This is the packaging PR (the listener stacks on top): it introduces the xtable-spark-runtime module producing a thin, drop-in Spark bundle, plus RFC-3, and validates the jar end-to-end via spark-submit.

Brief change log

RFC-3 (rfc/rfc-3/rfc-3.md).
New module xtable-spark-runtime_${scala.binary.version} (thin shaded bundle, ~3.6 MB).
XTableSparkSync — standalone spark-submit entry point (RunSync-equivalent for this bundle).
TableSyncSpec / XTableSyncService — shared sync building blocks (reused by the listener PR).
ConversionTargetFactory: make ServiceLoader target discovery resilient — skip targets whose engine lib isn't on the classpath (separable core change; see commit 1).
ITXTableSparkRuntimeBundle — spark-submit jar-validation test.

Packaging model (the important part)

Per review feedback, engines are not bundled. Spark, Hadoop, and Hudi/Iceberg/Delta are all provided — the user brings their own engine versions (Hudi 1.1 / 1.2 / …) and the thin XTable jar stays compatible across them.

Curated shade allowlist: only XTable modules + guava/protobuf (relocated under org.apache.xtable.shaded.*).
Libraries XTable exchanges with the engines are deliberately not relocated/bundled — relocating avro breaks the boundary (Hudi returns a real org.apache.avro.Schema → NoSuchMethodError). avro/parquet/jackson/commons come from the provided runtime.
Because target discovery uses ServiceLoader, it's made resilient so a user can run with only the engines they use (e.g. Hudi + Iceberg, no Delta).

Verify this pull request

ITXTableSparkRuntimeBundle (skipped unless SPARK_HOME is set) builds the shaded jar and runs a real sync:

SPARK_LOCAL_IP=127.0.0.1 mvn -pl xtable-spark-runtime -am verify

It creates a local Hudi table, spark-submits the bundle with Hudi + Iceberg supplied via --packages (versions from the root pom: Hudi 1.2.0, Iceberg 1.9.2), and asserts the Iceberg metadata is produced. Validated locally on Spark 3.5.2 (engines built for 3.4 → also a cross-version check).

Part of #836. Listener PR follows, stacked on this branch.

🤖 Generated with Claude Code

ConversionTargetFactory loads targets via ServiceLoader and eagerly instantiates every registered target when resolving any one of them. With a thin runtime where engines are provided, that hard-fails (e.g. DeltaKernelConversionTarget needs io.delta.kernel) even for a sync that never touches Delta. Skip providers whose engine library is not on the classpath so a subset of engines can be used (e.g. Hudi + Iceberg, no Delta). Also make the Delta-kernel check name-based to avoid loading DeltaKernelConversionTarget (and its io.delta.kernel dependencies) when Delta is absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

New xtable-spark-runtime module producing a thin, drop-in bundle for in-job XTable metadata sync, plus RFC-3. Packaging (the Hudi/Iceberg model): - Spark, Hadoop, and the engine libs (Hudi/Iceberg/Delta) are provided; the user brings their own engine versions and the bundle stays compatible across them. - Curated shade allowlist: only XTable's own modules + guava/protobuf (relocated under org.apache.xtable.shaded.*). Libraries exchanged with the engines (avro/parquet/jackson/commons) are NOT relocated/bundled - relocating avro breaks the boundary (Hudi returns a real avro Schema). - Resulting bundle: ~4 MB. Includes XTableSparkSync, a standalone spark-submit entry point (the RunSync-equivalent for this bundle), and TableSyncSpec/XTableSyncService shared with the listener (stacked follow-up PR). Validation: ITXTableSparkRuntimeBundle spark-submits the shaded jar via $SPARK_HOME with Hudi + Iceberg supplied through --packages (versions from the root pom) and asserts the Iceberg metadata is produced. Skipped when SPARK_HOME is unset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vinishjail97 · 2026-07-07T21:40:16Z

+  private static final String DELTA_KERNEL_TARGET_CLASS =
+      "org.apache.xtable.kernel.DeltaKernelConversionTarget";
+
+  // Name-based to avoid loading DeltaKernelConversionTarget (and its io.delta.kernel dependencies)


Remove this comment..the code is self explanatory.

Removed. Done in 96da5d6.

vinishjail97 · 2026-07-07T21:43:09Z

+        // A registered target whose engine library is not on the classpath (e.g. Delta when only
+        // Hudi/Iceberg are provided). Skip it so a subset of engines can still be used; a missing
+        // engine for the requested format surfaces below as NotSupportedException.
+        log.debug(


Make this a warn log so that it's visible for the user if some provided package is missing?

Changed to log.warn with a message pointing the user to provide the missing engine, so an absent provided package is visible. Done in 96da5d6.

vinishjail97 · 2026-07-07T21:45:35Z

+    }
+  }
+
+  private static Map<String, String> parseArgs(String[] args) {


Is this the right to parse? Any existing libraries that can be used to avoid hand-written logic?
https://github.com/apache/incubator-xtable/blob/main/xtable-utilities/src/main/java/org/apache/xtable/utilities/RunSync.java#L240

Good call. Switched to Apache Commons CLI (Options + DefaultParser + HelpFormatter), mirroring RunSync. commons-cli is bundled and relocated under org.apache.xtable.shaded.*. Done in 96da5d6.

…al bundle IT Review feedback on PR apache#838: - XTableSparkSync now parses args with Apache Commons CLI (mirrors RunSync) instead of hand-rolled parsing; the relocated commons-cli is bundled. - ConversionTargetFactory logs a warn (was debug) when it skips a registered target whose engine library is absent, so a missing provided engine is visible; dropped the self-explanatory comment on isDeltaKernelTarget. Bundle integration test (ITXTableSparkRuntimeBundle): - Supplies engines on a flat classpath via spark.driver.extraClassPath, and adds Avro 1.12 + parquet-avro to the engine classpath: Iceberg 1.9.2 (bumped in apache#784) compiles against Avro 1.12 APIs, which Spark 3.5's Avro 1.11.2 lacks. A flat classpath keeps one Avro on one loader, unlike --packages layering. - Adds jol-core (provided) that Hudi needs at runtime. - XTableSyncService writes target metadata at the source data path (required by Hudi; Iceberg data lives under <basePath>/data); IT passes --dataPath. - Readback mirrors ITConversionController: target read at the data path, Hudi metadata read options, per-format timestamp normalization, scalar-column data-equivalence, and non-vectorized Iceberg reads. - Relocates guava's com.google.thirdparty with the rest of guava. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vinishjail97 and others added 2 commits July 7, 2026 13:21

This was referenced Jul 7, 2026

[836] xtable-spark-runtime: XTableSyncListener for in-job metadata sync (listener) #839

Draft

[836] Introduce xtable-spark-runtime for in-job metadata sync #837

Closed

[836] RFC-3: correct bundle size to ~3.6 MB

9fcb4f8

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vinishjail97 commented Jul 7, 2026

View reviewed changes

vinishjail97 mentioned this pull request Jul 7, 2026

[736] Make the bundled xtable-utilities jar runnable again #840

Merged

This was referenced Jul 9, 2026

[736] Make the bundled xtable-utilities jar runnable again (Hudi 0.x) + run CI on branch-0.4 #841

Open

[836] xtable-spark-runtime: thin drop-in Spark bundle (Hudi 0.x) #843

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery#838

[836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery#838
vinishjail97 wants to merge 4 commits into
apache:mainfrom
vinishjail97:836-spark-runtime-packaging

vinishjail97 commented Jul 7, 2026 •

edited

Loading

Uh oh!

vinishjail97 Jul 7, 2026

Uh oh!

vinishjail97 Jul 7, 2026

Uh oh!

vinishjail97 Jul 7, 2026

Uh oh!

vinishjail97 Jul 7, 2026

Uh oh!

vinishjail97 Jul 7, 2026

Uh oh!

vinishjail97 Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

vinishjail97 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Brief change log

Packaging model (the important part)

Verify this pull request

Uh oh!

vinishjail97 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

vinishjail97 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

vinishjail97 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

vinishjail97 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

vinishjail97 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

vinishjail97 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vinishjail97 commented Jul 7, 2026 •

edited

Loading