Skip to content

[feature](paimon) Forward-port Paimon table writes to master - #67395

Open
suxiaogang223 wants to merge 12 commits into
apache:masterfrom
suxiaogang223:codex/forward-pick-paimon-write-master
Open

[feature](paimon) Forward-port Paimon table writes to master#67395
suxiaogang223 wants to merge 12 commits into
apache:masterfrom
suxiaogang223:codex/forward-pick-paimon-write-master

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #65086

Related PR: #65868, #65955, #66321, #66325, #66334, #66346, #66498, #66547, #66612, #66626, #66685, #66810, #66990, #67023, #67085

Problem Summary:

Forward-port the complete Paimon table-write path from branch-4.1 to master in one integrated change, avoiding repeated rewrites of shared FE, BE, JNI, and connector interfaces.

This PR:

  • Upgrades the Paimon dependency baseline from 1.3.1 to 1.4.2. A Paimon 2.0 upgrade is intentionally deferred.
  • Adds the BE Paimon sink, JNI/Java writer, commit payload transport, memory lifecycle, and spill handling.
  • Adds native Paimon bucket routing and writer assignment.
  • Adds connector transactions, write planning, and commit handling.
  • Adds INSERT plus row-level DELETE, UPDATE, and MERGE planning.
  • Adds Variant V2 write serialization and JNI read support.
  • Keeps the Doris and Paimon table caches coherent for writes.
  • Forward-ports the final branch-4.1 Paimon write regression matrix and generated expected outputs.
  • Adapts branch-4.1 code to master APIs and compile checks.

PR #66790 is intentionally excluded from this forward-port.

Release note

Support writing Apache Paimon tables, including INSERT, DELETE, UPDATE, MERGE, native bucket routing, schema evolution, and Variant V2 values. The bundled Paimon runtime is upgraded to 1.4.2.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
      • build-support/clang-format.sh
      • build-support/check-format.sh
      • build-support/check-build-hygiene.sh
      • Maven checkstyle validation: 0 violations
      • BE compilation completed successfully through ./build.sh --be --fe
      • Final FE and Java extensions compilation completed successfully through ./build.sh --fe
    • No need to test or manual test. Explain why:

  • Behavior changed:

    • No.
    • Yes. Paimon tables can be written through the Doris execution pipeline and support row-level DML.

  • Does this need documentation?

    • No.
    • Yes. Documentation can follow in a separate PR.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@suxiaogang223
[chore](paimon) Upgrade Paimon to 1.4.2
89250b1 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66346

Problem Summary: Upgrade the master Paimon runtime from 1.3.1 to 1.4.2 and adapt the JNI scanner and Connector SPI read path to the updated fallback-table, catalog, partition, scan-statistics, and split-count APIs. This establishes the dependency baseline required by the branch-4.1 table-write forward-port without importing unrelated write implementations.

### Release note

Upgrade the Paimon connector runtime to 1.4.2.

### Check List (For Author)

- Test: Unit Test
    - PaimonJniScannerTest: 33 tests passed
    - Connector Paimon targeted tests: 182 tests passed
    - Maven validate passed
- Behavior changed: Yes (Paimon runtime is upgraded from 1.3.1 to 1.4.2)
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Add Paimon table write runtime
188d7c9 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#65868, apache#66612, apache#66810

Problem Summary: Add the BE Paimon sink, JNI writer backend, Java writer, commit payload transport, and worktree-local spill and memory lifecycle support required to write Paimon tables from master.

### Release note

Support writing Apache Paimon tables through the native Doris execution pipeline.

### Check List (For Author)

- Test: Static validation
    - BE clang-format/check-format and FE checkstyle; full build and tests are deferred until all forward-port picks are complete
- Behavior changed: Yes, adds Paimon table writes
- Does this need documentation: Yes, documentation can follow separately
@suxiaogang223
[feature](paimon) Route Paimon writes by native bucket semantics
c21bcda 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66685

Problem Summary: Forward-port external sink hash partitioning, Paimon fixed-bucket routing, native row hashing, and writer assignment so distributed Doris writers match Paimon bucket ownership.

### Release note

Paimon writes now use native Paimon-compatible routing for fixed bucket tables.

### Check List (For Author)

- Test: Static validation
    - BE clang-format/check-format and FE checkstyle; full build and tests are deferred until all forward-port picks are complete
- Behavior changed: Yes, changes Paimon writer distribution
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Add connector transaction and write planning
e4bed7b 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#65868, apache#66498, apache#66685

Problem Summary: Forward-port the connector transaction, commit-message bridge, connector-neutral write distribution contract, and Paimon write-plan selection needed to execute table writes on master. The plan preserves native Paimon fixed-bucket routing and exposes row-change capabilities for subsequent DML planning.

### Release note

Support planning and committing Paimon table writes through the connector framework.

### Check List (For Author)

- Test: Manual test
    - FE checkstyle passed; final compilation is deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon writes now use connector transactions and native write distribution planning.
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Add row-level Paimon DML
78771c3 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66990

Problem Summary: Forward-port changelog planning for Paimon DELETE, UPDATE, and MERGE statements. The planner emits row-kind plus full-row records, validates deterministic primary-key behavior for MERGE, and preserves branch evaluation with an explicit short-circuit expression.

### Release note

Support DELETE, UPDATE, and MERGE writes for Paimon tables.

### Check List (For Author)

- Test: Manual test
    - FE checkstyle passed; final compilation is deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon tables support connector row-level DML planning.
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Support Variant V2 Paimon writes
baed6fc 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66321

Problem Summary: Forward-port Variant V2 serialization for Paimon Arrow batches, including encoded value/type byte pairs, outer-null preservation, supported nested constructors, and array-to-Variant conversion behavior.

### Release note

Support writing Variant V2 values to Paimon tables.

### Check List (For Author)

- Test: Unit Test
    - Added serialization and cast coverage; BE format/check and FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Variant V2 values can be serialized for Paimon writes.
- Does this need documentation: No
@suxiaogang223
[improvement](paimon) Read Variant through Paimon JNI
7db3043 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66334, apache#66547, apache#66626

Problem Summary: Forward-port Variant V2 decoding across the Paimon Java scanner and native JNI bridge. The reader propagates nested access paths, materializes encoded Variant bytes into ColumnVariantV2, and accepts Paimon required-value leaves for fallback-only Variant arrays.

### Release note

Improve Paimon Variant reads with nested projection and native Variant V2 materialization.

### Check List (For Author)

- Test: Unit Test
    - Added Java vector/projection and BE Parquet schema/reader coverage; BE format/check, header hygiene, and FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon Variant reads support nested projection and fallback-array schemas.
- Does this need documentation: No
@suxiaogang223
[fix](paimon) Keep write table cache coherent
98eba3f 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#67023

Problem Summary: Doris and Paimon maintain separate table caches. After a Doris metadata-cache miss, Paimon CachingCatalog could return the same stale table handle, leaving writes planned with outdated bucket, path, or table options. Invalidate the inner Paimon entry before every outer-cache reload and verify the invalidate-before-get ordering.

### Release note

Ensure refreshed Paimon writes use current table metadata.

### Check List (For Author)

- Test: Unit Test
    - Added cache reload ordering coverage; FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon table reloads no longer reuse stale inner-cache entries.
- Does this need documentation: No
@suxiaogang223
[test](paimon) Forward-port table write coverage
ab064eb 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#65868, apache#66325, apache#67023, apache#67085

Problem Summary: Forward-port the final branch-4.1 Paimon write regression matrix to master, including append and primary-key modes, bucket routing, transactions, schema evolution, row-level DML, Variant V2, snapshot references, writer lifecycle, cache coherence, and stabilized external paths. Generated golden outputs are preserved byte-for-byte from branch-4.1.

### Release note

None

### Check List (For Author)

- Test: Regression test
    - Test cases and generated expected outputs were forward-ported; execution is deferred until the final integrated validation.
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[fix](be) Align Paimon partitioners with master compile checks
b364334 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66685

Problem Summary: branch-4.1 wrapped new partitioner code with compile_check_begin/end headers that no longer exist on master because conversion warnings are enforced globally. Remove the obsolete wrappers so the forward-ported Paimon routing code builds under the master warning policy.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - BE full build exposed the missing headers; clang-format/check and build hygiene passed. The incremental full build continues after this commit.
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[fix](paimon) Fix table options test compilation
9a555e1 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#67023

Problem Summary: The forward-ported Paimon table cache coherence test uses List to record catalog calls but did not import java.util.List, causing FE test compilation to fail. Add the missing import.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Maven checkstyle validation for the Paimon connector
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[fix](fe) Use expression arguments in array legality check
21ae767 
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#66321

Problem Summary: The Variant V2 forward-port referenced a branch-specific arguments field while validating ARRAY children. Master exposes legality-check inputs through getArguments(), so FE compilation failed. Use the master expression API while preserving validation of every array argument.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - FE compilation identified the issue; final incremental build follows this commit
- Behavior changed: No
- Does this need documentation: No
@hello-stephen

hello-stephen commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 Awaiting requested review from 924060929 924060929 is a code owner
@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner
@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@Gabriel39 Gabriel39 Awaiting requested review from Gabriel39 Gabriel39 is a code owner
@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner
@morningman morningman Awaiting requested review from morningman morningman is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Tracking: Paimon table write support

2 participants

@suxiaogang223 @hello-stephen