Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions docs/trigger-cdc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Trigger CDC PDK contract

Trigger CDC adds three connector functions: `ensure`, `acknowledge`, and `cleanup`.
Reading uses the existing `StreamReadFunction` and ordinary `TapEvent` types.
`ConnectorFunctions` advertises `trigger_cdc` only when all four are registered.
Database/version-specific guarantees come back in `TriggerCdcInstallResult.capabilities`.
Registration advertises implementation support; it does not establish database privileges,
DDL support or a safe snapshot boundary. Those must be verified by ensure.

## Lifecycle and checkpoint integration

1. Call ensure with the effective storage config, a stable consumer id and the original
requested table list. It installs/verifies owned objects and registers the durable
subscription before a snapshot or read can start. Repeat on resume. Keep subscriptions
across stop/crash and do not expire them by TTL.
2. Check `validate(originalTables)` and require every table to succeed before starting the
task. Persist every capture id and generation. A partial result describes retryable
failures, not permission to silently omit those tables. `validate` checks report structure;
`getCaptureIds(originalTables)` additionally rejects any failed table. A usable result needs
an explicit readable status (CAPTURING, SNAPSHOTTING, STREAMING or DEGRADED), nonblank
database type/version and a STRICT error-policy declaration. DEGRADED still requires
operator attention. Apply the snapshot handover requirements below before starting a snapshot.
3. Invoke streamRead on the same node context. The connector uses the consumer/config/capture
mapping retained by ensure and polls committed journal records. The offset is a complete
`Map<String, TriggerCdcOffset>` keyed by capture id, even for a single table. Validate keys,
identities, generations and token ownership before resume. Emit ordinary TapEvents and a
fresh map/value snapshot on each callback. Stop on schema drift, disabled/missing triggers
or a subscription marked integrity broken.
4. Each token describes the cumulative delivered set for its capture, including earlier
unacknowledged batches. An out-of-order committed event must still be delivered: sequence
allocation order is not commit order. Track row membership rather than an event-id high
watermark. In-memory delivery tracking can avoid continuous duplicates while a checkpoint
is pending; loss of that tracking after restart safely replays unacknowledged events.
5. Persist the complete checkpoint, including captures that have not delivered events.
Only after every destination's writes and that recoverable checkpoint are durable, pass
entries with a valid delivery token to acknowledge; entries without a token stay
in the checkpoint and must not be acknowledged. Tokens are bound to consumer, capture
and generation. Duplicate or older acknowledgements are idempotent and cannot regress
progress. A crash before acknowledgement may cause replay; targets need idempotent writes.
6. Run bounded cleanup. Delete only rows acknowledged by every registered consumer and old
enough under retention. A paused or deleted task still holds its subscription in this
first version. Forced deletion requires explicit opt-in, usable affected consumer ids,
and integrity markings persisted before or atomically with the delete. Validate each result.

The current engine callback in `HazelcastSourcePdkDataNode.generateStreamReadConsumer`
enqueues events and updates an in-memory offset. Returning from it is not a durable checkpoint
notification. The engine integration must connect step 5 to durable progress covering every
sink; until then, it must not acknowledge on callback return or enable journal reclamation
based on delivery. This PR provides the PDK contracts, not that engine implementation.

## Snapshot and journal handover

`captureBoundarySupported` proves only that trigger installation leaves no capture gap.
It is necessary but insufficient for automatic full-to-incremental handover. It does not
identify which journal changes a snapshot already includes, nor authorize a replay position.

Before enabling automatic handover, the engine and connector must implement and verify a
snapshot/replay protocol for the specific database/version and destination write semantics.
That protocol must either identify and suppress journal changes covered by the snapshot,
or prove overlap replay safe, including updates, deletes, key changes and non-idempotent
writes. For example, a snapshot containing v3 followed by journal v2 then v3 can temporarily
regress the target; idempotent writes alone do not prevent old versions overwriting new ones.
A replay protocol that allows such intermediate states must explicitly define when the target
becomes usable and how it recovers safely after a crash at any intermediate point.

The handover state must durably bind the snapshot progress/completion and journal replay
position to the consumer, capture ids and generations. Recovery must restore a consistent
pair, retain all journal records still needed for the snapshot/replay protocol, and never
acknowledge overlap merely because the snapshot has finished. Normal acknowledgement still
requires durable destination writes and a recoverable checkpoint as described above.
Validate concurrent changes and crashes during snapshot, overlap replay and transition to
streaming before enabling the protocol. Until this integration is implemented and verified,
automatic full-to-incremental handover must remain disabled, even if captureBoundarySupported
is true. This PDK contract does not itself implement that protocol or add a handover API.

## Scope

There are no separate capability-query, SQL-plan, read, periodic-inspect or release APIs.
Ensure returns verified capabilities and performs startup checks; the stream reader performs
ongoing integrity checks. SQL preview/manual installation, automatic subscription retirement
and automatic trigger/table removal are deferred. Operators must explicitly retire abandoned
subscriptions outside this API; automatic expiry would lose paused consumers' data.

Tapdata API artifacts use `2.0.12-SNAPSHOT`; the parent, runner, modules and error-code
artifacts use `2.12-SNAPSHOT`. The companion changes in the engine/manager and connectors
repositories must be applied with this release. Third-party libraries and Scala's binary
version are independent of this release.
6 changes: 3 additions & 3 deletions error-code-root/contract-tests/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@
</properties>
<!-- Standalone tests: never added to the production reactor or BOM. -->
<dependencies>
<dependency><groupId>io.tapdata</groupId><artifactId>error-code-scanner</artifactId><version>2.11-SNAPSHOT</version><scope>test</scope></dependency>
<dependency><groupId>io.tapdata</groupId><artifactId>pdk-error-code</artifactId><version>2.11-SNAPSHOT</version><scope>test</scope></dependency>
<dependency><groupId>io.tapdata</groupId><artifactId>tapdata-api</artifactId><version>2.0.11-SNAPSHOT</version><scope>test</scope></dependency>
<dependency><groupId>io.tapdata</groupId><artifactId>error-code-scanner</artifactId><version>2.12-SNAPSHOT</version><scope>test</scope></dependency>
<dependency><groupId>io.tapdata</groupId><artifactId>pdk-error-code</artifactId><version>2.12-SNAPSHOT</version><scope>test</scope></dependency>
<dependency><groupId>io.tapdata</groupId><artifactId>tapdata-api</artifactId><version>2.0.12-SNAPSHOT</version><scope>test</scope></dependency>
<dependency><groupId>org.junit.jupiter</groupId><artifactId>junit-jupiter</artifactId><version>5.8.1</version><scope>test</scope></dependency>
</dependencies>
<build><plugins>
Expand Down
2 changes: 1 addition & 1 deletion error-code-root/error-code-bom/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>error-code-root</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>

<artifactId>error-code-bom</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion error-code-root/error-code-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>error-code-parent</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
<relativePath>../error-code-parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion error-code-root/error-code-parent/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>error-code-root</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>

<artifactId>error-code-parent</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion error-code-root/error-code-scanner/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>error-code-parent</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
<relativePath>../error-code-parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion error-code-root/pdk-error-code/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>error-code-parent</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
<relativePath>../error-code-parent/pom.xml</relativePath>
</parent>

Expand Down
6 changes: 3 additions & 3 deletions error-code-root/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-common-lib</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>error-code-root</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
<name>Error Code Root</name>
<packaging>pom</packaging>

Expand All @@ -22,7 +22,7 @@
</modules>

<properties>
<error.code.version>2.11-SNAPSHOT</error.code.version>
<error.code.version>2.12-SNAPSHOT</error.code.version>
</properties>

<dependencyManagement>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/connector-archetypes/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>plugin-kit</artifactId>
<groupId>io.tapdata</groupId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
20 changes: 10 additions & 10 deletions plugin-kit/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-common-lib</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
<artifactId>plugin-kit</artifactId>
<name>plugin-kit</name>
<packaging>pom</packaging>
Expand All @@ -25,19 +25,19 @@
<properties>
<proxy.file.name>${project.artifactId}-v${project.version}</proxy.file.name>
<connector.file.name>${project.artifactId}-v${project.version}</connector.file.name>
<tapdata.pdk.runner.verison>2.11-SNAPSHOT</tapdata.pdk.runner.verison>
<tapdata.api.verison>2.0.11-SNAPSHOT</tapdata.api.verison>
<tapdata.pdk.api.verison>2.0.11-SNAPSHOT</tapdata.pdk.api.verison>
<tapdata.pdk.runner.verison>2.12-SNAPSHOT</tapdata.pdk.runner.verison>
<tapdata.api.verison>2.0.12-SNAPSHOT</tapdata.api.verison>
<tapdata.pdk.api.verison>2.0.12-SNAPSHOT</tapdata.pdk.api.verison>
<tapdata.pdk.connector.core.version>1.0-SNAPSHOT</tapdata.pdk.connector.core.version>
<junit.jupiter.version>5.8.1</junit.jupiter.version>
<junit.platform.version>1.8.1</junit.platform.version>
<mockito.version>4.11.0</mockito.version>
<mockito.junit.jupiter.version>4.11.0</mockito.junit.jupiter.version>
<error-code-core.version>2.11-SNAPSHOT</error-code-core.version>
<error-code-scanner.version>2.11-SNAPSHOT</error-code-scanner.version>
<plugin-parent>2.11-SNAPSHOT</plugin-parent>
<tapdata.modules.version>2.11-SNAPSHOT</tapdata.modules.version>
<tapdata.proxy.version>2.11-SNAPSHOT</tapdata.proxy.version>
<error-code-core.version>2.12-SNAPSHOT</error-code-core.version>
<error-code-scanner.version>2.12-SNAPSHOT</error-code-scanner.version>
<plugin-parent>2.12-SNAPSHOT</plugin-parent>
<tapdata.modules.version>2.12-SNAPSHOT</tapdata.modules.version>
<tapdata.proxy.version>2.12-SNAPSHOT</tapdata.proxy.version>
<commons-io.version>2.18.0</commons-io.version>
<commons.lang3.version>3.12.0</commons.lang3.version>
<fastjson.version>1.2.83</fastjson.version>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<artifactId>tapdata-api</artifactId>
<groupId>io.tapdata</groupId>
<name>tapdata-api</name>
<version>2.0.11-SNAPSHOT</version>
<version>2.0.12-SNAPSHOT</version>

<properties>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>plugin-kit</artifactId>
<groupId>io.tapdata</groupId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-modules/async-tools-module/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>async-tools-module</artifactId>
<name>async-tool-module</name>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-modules/class-supervisor/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>

<artifactId>class-supervisor</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-modules/modules-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>modules-api</artifactId>
<name>modules-api</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>mongodb-storage-module</artifactId>
<name>mongodb-storage-module</name>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-modules/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>plugin-kit</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>tapdata-modules</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion plugin-kit/tapdata-modules/script-engines-module/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>script-engine-module</artifactId>
<name>script-engine-module</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>service-skeleton-module</artifactId>
<name>service-skeleton-module</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>tapdata-storage-module</artifactId>
<name>tapdata-storage-module</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>websocket-client-module</artifactId>
<name>websocket-client-module</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-modules</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<artifactId>websocket-server-module</artifactId>
<name>websocket-server-module</name>
Expand Down
11 changes: 8 additions & 3 deletions plugin-kit/tapdata-pdk-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@
<parent>
<groupId>io.tapdata</groupId>
<artifactId>plugin-kit</artifactId>
<version>2.11-SNAPSHOT</version>
<version>2.12-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>tapdata-pdk-api</artifactId>
<name>tapdata-pdk-api</name>
<version>2.0.11-SNAPSHOT</version>
<version>2.0.12-SNAPSHOT</version>
<modules>
</modules>

Expand All @@ -20,7 +20,7 @@
<dependency>
<groupId>io.tapdata</groupId>
<artifactId>tapdata-api</artifactId>
<version>2.0.11-SNAPSHOT</version>
<version>2.0.12-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>io.tapdata</groupId>
Expand All @@ -39,6 +39,11 @@
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.2.5</version>
</plugin>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
Expand Down
Loading
Loading