Skip to content

[FLINK-40453][postgres] Not load table schema when initializing record emitter - #4513

Open
naivedogger wants to merge 1 commit into
apache:masterfrom
naivedogger:cp-postgres-lazy-schema-to-community
Open

[FLINK-40453][postgres] Not load table schema when initializing record emitter#4513
naivedogger wants to merge 1 commit into
apache:masterfrom
naivedogger:cp-postgres-lazy-schema-to-community

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Fixes FLINK-40453.

PostgresPipelineRecordEmitter currently builds all CreateTableEvents eagerly in its constructor (and again in the applySplit() override), which opens a JDBC connection and fetches every captured table's schema up front — before any split is processed. This adds startup overhead that scales with the number of captured tables. A table's schema is only needed when its first record / low-watermark / schema-change event is emitted, so loading everything at initialization is wasteful.

Brief change log

  • Remove the eager generateCreateTableEvent() call from the constructor and drop the applySplit() override that pre-populates the cache.
  • Resolve and cache each CreateTableEvent lazily on demand, falling back to a JDBC lookup only when the split state does not already carry the table schema.
  • Behavior for snapshot / incremental / schema-change paths is unchanged.

Verifying this change

This change added tests and can be verified as follows:

  • Added unit tests in PostgresPipelineRecordEmitterTest, asserting that constructing the emitter opens no JDBC connection and that assigning a snapshot split does not trigger loading of all table schemas.

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant