[836] Introduce xtable-spark-runtime for in-job metadata sync

[vinishjail97]

What is the purpose of the pull request

Implements the first slice of #836 (RFC-3): a new xtable-spark-runtime module that runs incremental XTable metadata sync inside an existing Spark job, activated by config only — no separate RunSync process, no user code change.

A driver-side QueryExecutionListener runs ConversionController.sync(...) after a successful write to a configured source table:

spark-submit --packages org.apache.xtable:xtable-spark-runtime_2.12:<ver> \
  --conf spark.sql.queryExecutionListeners=org.apache.xtable.spark.XTableSyncListener \
  --conf spark.xtable.tables=orders \
  --conf spark.xtable.orders.basePath=/warehouse/db/orders \
  --conf spark.xtable.orders.sourceFormat=HUDI \
  --conf spark.xtable.orders.targets=ICEBERG,DELTA

Design details are in rfc/rfc-3/rfc-3.md.

Brief change log

• New module xtable-spark-runtime_${scala.binary.version} (Spark/Hadoop provided), registered in the parent pom.xml.
• XTableSparkConfig — parse spark.xtable.* into TableSyncSpec (path- and name-based table selection).
• XTableSyncService — build ConversionConfig(INCREMENTAL) and run ConversionController.sync(...).
• PlanTargetResolver — best-effort extraction of the written path from the analyzed plan (isolated Spark-internal dependence).
• XTableSyncListener — stateless, synchronous: on a recognized write matching a configured table, run the sync inline on the callback. Single-flight falls out for free (single-threaded LiveListenerBus); per-table failures are caught as Throwable so a listener never destabilizes the bus.
• RFC-3.

Design notes

• Stateless: the sync watermark already lives in the target's TableSyncMetadata, and INCREMENTAL auto-falls back to a full snapshot when unsafe, so there is nothing to track client-side.
• Synchronous: QueryExecutionListener callbacks are delivered asynchronously by Spark off SparkListenerSQLExecutionEnd, so a listener can't block write(); running inline is the simplest correct choice and completes before JVM exit via the bus drain on SparkContext.stop(). Async / "when to trigger" is intentionally left to the user's DAG as an opt-in follow-up.

Non-goals (follow-ups)

• Thin shaded/relocated bundle (curated allowlist, org.apache.xtable.shaded.*).
• StreamingQueryListener variant, CALL xtable.sync(...) procedure, async execution opt-in.

Verify this pull request

This change added tests and can be verified as follows:

• Unit: TestXTableSparkConfig, TestPlanTargetResolver, TestXTableSyncListener (path matching).
• Integration: ITXTableSyncListener — embedded local[*]; a real Hudi datasource write triggers the listener and syncs to Delta + Iceberg in the same JVM, asserting the id sets match across all three formats.

SPARK_LOCAL_IP=127.0.0.1 mvn -pl xtable-spark-runtime -am verify

Closes #836 (first slice).

🤖 Generated with Claude Code

[vinishjail97]

Splitting the PR into packaging and listener for easier validation.

[vinishjail97]

Split into two stacked PRs for easier review, per feedback:

• [836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery #838 — packaging: the thin xtable-spark-runtime bundle (engines provided, ~4 MB), RFC-3, resilient target discovery, and a spark-submit jar-validation test.
• [836] xtable-spark-runtime: XTableSyncListener for in-job metadata sync (listener) #839 — listener: the XTableSyncListener for in-job sync (stacked on [836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery #838).

Closing this combined PR in favor of those.