[736] Make the bundled xtable-utilities jar runnable again

[vinishjail97]

What is the purpose of the pull request

Make the bundled xtable-utilities jar runnable again via the documented flow
(the Hudi docs):

java -jar xtable-utilities/target/xtable-utilities_2.12-<ver>-bundled.jar \
  --datasetConfig my_config.yaml

Fixes #736. Building main and running that command today fails with a chain of
NoClassDefFoundErrors — #736 (the missing org.openjdk.jol) is only the last
link in that chain — and syncing to Hudi is silently skipped.

Root cause

Two upstream changes dropped several runtime dependencies from the shaded jar:

1. Automate license file generation and validation #822 ("Automate license file generation and validation") replaced the
   maven-shade bundle-everything default with an explicit <artifactSet>
   allowlist. Anything not on the list is no longer bundled.
2. The Hudi 1.x bump split the former hudi-common into
   hudi-common + hudi-io + hudi-hadoop-common, and xtable-core compiles
   against these with the Hudi Spark bundle in provided scope (a real Spark
   cluster supplies them). The standalone utilities jar has no such runtime.

Net effect — RunSync dies before doing any work:

Missing from bundle	Symptom
org.slf4j:slf4j-api	NoClassDefFoundError: org/slf4j/LoggerFactory (at first Hadoop Configuration init)
org.apache.hudi:hudi-hadoop-common	NoClassDefFoundError: org/apache/hudi/hadoop/fs/HadoopFSUtils
org.apache.hudi:hudi-io	(next runtime transitive, not allowlisted)
org.openjdk.jol:jol-core	NoClassDefFoundError: org/openjdk/jol/info/GraphLayout (#736, at the Hudi metadata-table read)

Separately, syncing to Hudi was silently skipped: hudi-java-client (the
Hudi write client, incl. HoodieJavaEngineContext) was declared twice —
once at compile, and again with <scope>test</scope> in the test block.
Maven's last-declaration-wins downgraded the effective scope to test, so the
shade plugin excluded it (even though it's on the allowlist — the allowlist only
filters what's eligible, it can't promote a test-scoped artifact back in).
ConversionTargetFactory then logged WARN … Skipping a registered ConversionTarget whose engine library is not on the classpath and dropped
HudiConversionTarget.

Brief change log

xtable-utilities/pom.xml only:

• Add org.slf4j:slf4j-api, org.apache.hudi:hudi-io, and
  org.openjdk.jol:jol-core to the shade <includes> allowlist.
• jol-core is pinned to test by the parent dependencyManagement, so
  override it to runtime (Hudi's ObjectSizeCalculator loads it at runtime,
  not only in tests).
• hudi-hadoop-common is only a provided/test transitive, so declare it as
  an explicit runtime dependency of the bundle and allowlist it.
• Remove the redundant test-scoped hudi-java-client declaration so the
  compile one takes effect and the Hudi write client is bundled.

Why utilities isn't published to Maven (context for reviewers)

xtable-utilities is not published to Maven and is excluded from the ASF
release process — see the dev@ thread "[DISCUSS] Excluding xtable-utilities
from the release process" and #536. It is an unshaded, un-relocated ~1 GB fat
jar that bundles the full transitive closure of Spark/Hadoop/Hive/etc.; that
form can't be released under ASF licensing/release guidelines. Because there's
no published artifact, users build the bundle locally — which is exactly the
path where these test/provided-scoped-but-runtime-needed gaps surface. This
PR keeps that local build working.

The maintained, long-term replacement is xtable-spark-runtime (#836): a
thin, relocated, config-only drop-in Spark bundle that runs incremental
ConversionController.sync(...) in-job (implementation in #838 / #839). This PR
is a targeted fix for the existing utilities jar in the meantime, not a
substitute for that effort.

Verify this pull request

Built the bundle locally and ran the documented command against real tables:

SPARK_LOCAL_IP=127.0.0.1 mvn clean package -pl xtable-utilities -am -DskipTests
SPARK_LOCAL_IP=127.0.0.1 java -jar xtable-utilities/target/xtable-utilities_2.12-*-bundled.jar \
  --datasetConfig <config>.yaml

Hudi → Delta + Iceberg (Hudi 1.2.0 source, table version 9, MDT enabled,
150 rows) — previously a NoClassDefFoundError:

INFO org.apache.hudi.metadata.HoodieTableMetadataUtil - Loading latest merged file slices for metadata table partition files
INFO org.apache.iceberg.metrics.LoggingMetricsReporter - ... addedRecords=150, totalRecords=150 ...
INFO org.apache.spark.sql.delta.OptimisticTransaction - Committed delta #0 ...
INFO org.apache.xtable.conversion.ConversionController - Sync is successful for the following formats ICEBERG,DELTA

_delta_log/00000000000000000000.json and Iceberg v{1,2,3}.metadata.json written; all 150 records committed.

Iceberg → Hudi (200-row path-based Iceberg source) — previously skipped:

INFO org.apache.xtable.conversion.ConversionController - Sync is successful for the following formats HUDI

A real Hudi table is written (.hoodie/…​.replacecommit COMPLETED + hoodie.properties + metadata table); no "Skipping … ConversionTarget" warning.

🤖 Generated with Claude Code