Skip to content

[iceberg] Warn when the Iceberg metadata committer factory is missing from the classpath - #9158

Open
bube wants to merge 1 commit into
apache:masterfrom
bube:warn-on-non-iceberg-catalog-impl-not-found
Open

[iceberg] Warn when the Iceberg metadata committer factory is missing from the classpath#9158
bube wants to merge 1 commit into
apache:masterfrom
bube:warn-on-non-iceberg-catalog-impl-not-found

Conversation

@bube

@bube bube commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Purpose

Closes #9157.

When metadata.iceberg.storage requests a sync to an external catalog but the committer factory cannot be discovered, Paimon skipped the sync silently. IcebergCommitCallback caught the FactoryException and discarded it, leaving metadataCommitter null so commitToExternalCatalog returned early on every commit. Commits kept succeeding and metadata files kept being written, so the job looked healthy while nothing reached the catalog. In our case a shaded jar had dropped the META-INF/services entry for IcebergRESTMetadataCommitterFactory.

This logs a WARN instead:

No IcebergMetadataCommitterFactory for 'metadata.iceberg.storage=rest-catalog' found on
the classpath, so table mydb.t will not be synced to the external catalog (commits and
metadata files are unaffected). Check that the module providing it is deployed and that
its META-INF/services/org.apache.paimon.factories.Factory entry survived shading.
Cause: Could not find any factories that implement
'org.apache.paimon.iceberg.IcebergMetadataCommitterFactory' in the classpath.

disabled, table-location and hadoop-catalog have no committer factory by design and must stay quiet. IcebergOptions.StorageType now answers that question instead of a switch in the caller:

HIVE_CATALOG("hive-catalog", "...", true),
REST_CATALOG("rest-catalog", "...", true);

It is a required constructor argument, so a new storage type cannot be added without stating its own answer. A committerFactoryIdentifier() accessor replaces the incidental use of toString() as the SPI lookup key. Both factories already derived their identifier from this enum, so this only names the contract.

Still a warning and not a failure, so jobs in this state will not start erroring on upgrade.

Tests

No new test. Reproducing a missing SPI registration inside a module that has one needs classloader manipulation that would test the harness more than the code. Verified by running it across every StorageType:

metadata.iceberg.storage warns?
table-location no
hadoop-catalog no
hive-catalog yes
rest-catalog yes

Existing suites pass on JDK 11:

  • paimon-core: IcebergCommitCallbackTest 28/28, IcebergCompatibilityTest 40/40
  • paimon-iceberg: 30 unit, 4 IT including IcebergRestMetadataCommitterITCase
  • paimon-hive-catalog: 62 unit, 14 IT

spotless, checkstyle, enforcer and rat are clean.

API and Format

No format change. StorageType gains two public accessors. Constants, values and descriptions are unchanged, so no config or generated-docs change.

Documentation

None needed.

… from the classpath

When metadata.iceberg.storage requests a sync to an external catalog but
the committer factory cannot be discovered, Paimon skipped the sync
silently. IcebergCommitCallback caught the FactoryException and discarded
it, leaving metadataCommitter null so commitToExternalCatalog returned
early on every commit. Commits kept succeeding and metadata files kept
being written, so the job looked healthy while nothing reached the
catalog.

Log a WARN in that case instead. The disabled, table-location and
hadoop-catalog storage types have no committer factory by design and must
stay quiet, so StorageType now carries whether it needs one as a required
constructor argument rather than a switch in the caller. A new
committerFactoryIdentifier() accessor replaces the incidental use of
toString() as the SPI lookup key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][iceberg] Metadata sync silently does nothing when the committer factory is missing from the classpath

1 participant