Skip to content

feat(datafusion): carry catalog identity on StaticTableProvider - #3016

Open
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:feat/static-provider-catalog-identity
Open

feat(datafusion): carry catalog identity on StaticTableProvider#3016
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:feat/static-provider-catalog-identity

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

related to discussion in github.com//pull/2862

  • Adds the struct: IcebergCatalogConfig { type, name, props }, which mirrors inputs to CatalogBuilder::load, as an optional field to StaticTableProvider and IcebergTableScan.
  • IcebergStaticTableProvider::with_catalog_config records it; config(), table_ident(), and snapshot_id() read it back (to recreate the Table struct with new live FileIO).
  • IcebergTableScan carries it into the physical plan, so a scan shipped to a remote worker can rebuild its own storage.

Nothing connects using these values. The field is an Option defaulting to None,
so existing constructors are unaffected and the public API change is purely additive.

It lives in iceberg-datafusion rather than iceberg-catalog-loader to avoid
pulling the glue/hms/rest/s3tables/sql catalogs in for a three-field struct.

Are these changes tested?

  • test_static_provider_propagates_catalog_config_into_scan
  • test_static_provider_exposes_catalog_identity

AI Disclosure

  • Helped identify the shape of IcebergCatalogConfig based off CatalogBuilder::load
  • PR validation.

@NoahKusaba NoahKusaba changed the title init feat(datafusion): carry catalog identity on StaticTableProvider Aug 17, 2026
@NoahKusaba

NoahKusaba commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Hey @mbutrovich, following up on the snapshot_id PR you reviewed.

As discussed, this lets IcebergStaticTableProvider record which catalog its table came from, so it can be rebuilt in another process.

Do you have time to take a look?

I have 3 more follow-up PR's planned that will extend same to IcebergTableProvider, the write path, and
metadata tables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StaticTableProvider can't be reconstructed in another process

1 participant