Skip to content

[AURON #2451] Add Paimon COW multi-commit correctness coverage #2452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
weimingdiit wants to merge 1 commit into
base: master
Choose a base branch
from
+18 −0
Open

[AURON #2451] Add Paimon COW multi-commit correctness coverage #2452

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions ...y/auron-paimon/src/test/scala/org/apache/auron/paimon/AuronPaimonV2IntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,24 @@ class AuronPaimonV2IntegrationSuite
}
}

test("paimon v2 COW primary-key table preserves latest value across commits") {
withTable("paimon.db.t_cow_multi_commit") {
sql("""
|create table paimon.db.t_cow_multi_commit (id int, v string)
|using paimon
|tblproperties (
| 'primary-key' = 'id',
| 'bucket' = '2',
| 'full-compaction.delta-commits' = '1'
|)
|""".stripMargin)
sql("insert into paimon.db.t_cow_multi_commit values (1, 'a'), (2, 'b')")
sql("insert into paimon.db.t_cow_multi_commit values (1, 'updated')")
val df = sql("select * from paimon.db.t_cow_multi_commit")
checkAnswer(df, Seq(Row(1, "updated"), Row(2, "b")))

@weiqingy weiqingy Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says this test checks that the query still uses NativePaimonV2TableScan, but checkAnswer is the only assertion here.

The catch is that the answer comes out the same either way. If the native scan bails out, Spark falls back to its own Paimon reader, which merges by primary key and returns those same two rows, so the test stays green even if it never touches the native path. t_mor at line 234 is exactly that combination: no native scan, correct answer.

And the bail-out is a real possibility here. PaimonScanSupport.scala:162-173 drops the native plan when a split isn't raw-readable, and PaimonConvertProvider.scala:52 and :95 then quietly leave Spark's own scan in place.

Every other test in this suite that expects a native scan says so, directly (lines 49, 59, 73, 91, 105, 116, 142, 169, 408) or through checkSparkAnswerAndNativePaimonScan and executedNativeScan. Would it be worth doing the same here? It would also make this the first test pinning the native path for a COW primary-key table past its first commit.

One line, in case it helps:

Suggested change
checkAnswer(df, Seq(Row(1, "updated"), Row(2, "b")))
checkAnswer(df, Seq(Row(1, "updated"), Row(2, "b")))
assertNativePaimonScanApplied(df)

And if the scan turns out not to stay native after a second commit, that feels worth knowing too.

}
}

test("paimon v2 native scan handles empty table") {
withTable("paimon.db.t_empty") {
sql("create table paimon.db.t_empty (id int, v string) using paimon")
Expand Down
Loading