[iceberg] Warn when the Iceberg metadata committer factory is missing from the classpath - #9158
Open
bube wants to merge 1 commit into
Open
[iceberg] Warn when the Iceberg metadata committer factory is missing from the classpath#9158bube wants to merge 1 commit into
bube wants to merge 1 commit into
Conversation
… from the classpath When metadata.iceberg.storage requests a sync to an external catalog but the committer factory cannot be discovered, Paimon skipped the sync silently. IcebergCommitCallback caught the FactoryException and discarded it, leaving metadataCommitter null so commitToExternalCatalog returned early on every commit. Commits kept succeeding and metadata files kept being written, so the job looked healthy while nothing reached the catalog. Log a WARN in that case instead. The disabled, table-location and hadoop-catalog storage types have no committer factory by design and must stay quiet, so StorageType now carries whether it needs one as a required constructor argument rather than a switch in the caller. A new committerFactoryIdentifier() accessor replaces the incidental use of toString() as the SPI lookup key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #9157.
When
metadata.iceberg.storagerequests a sync to an external catalog but the committer factory cannot be discovered, Paimon skipped the sync silently.IcebergCommitCallbackcaught theFactoryExceptionand discarded it, leavingmetadataCommitternull socommitToExternalCatalogreturned early on every commit. Commits kept succeeding and metadata files kept being written, so the job looked healthy while nothing reached the catalog. In our case a shaded jar had dropped theMETA-INF/servicesentry forIcebergRESTMetadataCommitterFactory.This logs a
WARNinstead:disabled,table-locationandhadoop-cataloghave no committer factory by design and must stay quiet.IcebergOptions.StorageTypenow answers that question instead of a switch in the caller:It is a required constructor argument, so a new storage type cannot be added without stating its own answer. A
committerFactoryIdentifier()accessor replaces the incidental use oftoString()as the SPI lookup key. Both factories already derived their identifier from this enum, so this only names the contract.Still a warning and not a failure, so jobs in this state will not start erroring on upgrade.
Tests
No new test. Reproducing a missing SPI registration inside a module that has one needs classloader manipulation that would test the harness more than the code. Verified by running it across every
StorageType:metadata.iceberg.storagetable-locationhadoop-cataloghive-catalogrest-catalogExisting suites pass on JDK 11:
paimon-core:IcebergCommitCallbackTest28/28,IcebergCompatibilityTest40/40paimon-iceberg: 30 unit, 4 IT includingIcebergRestMetadataCommitterITCasepaimon-hive-catalog: 62 unit, 14 ITspotless, checkstyle, enforcer and rat are clean.
API and Format
No format change.
StorageTypegains two public accessors. Constants, values and descriptions are unchanged, so no config or generated-docs change.Documentation
None needed.