[AURON #2451] Add Paimon COW multi-commit correctness coverage - #2452
[AURON #2451] Add Paimon COW multi-commit correctness coverage#2452weimingdiit wants to merge 1 commit into
Conversation
Signed-off-by: weimingdiit <weimingdiit@gmail.com>
1b655c2 to
61fcc87
Compare
| sql("insert into paimon.db.t_cow_multi_commit values (1, 'a'), (2, 'b')") | ||
| sql("insert into paimon.db.t_cow_multi_commit values (1, 'updated')") | ||
| val df = sql("select * from paimon.db.t_cow_multi_commit") | ||
| checkAnswer(df, Seq(Row(1, "updated"), Row(2, "b"))) |
There was a problem hiding this comment.
The PR description says this test checks that the query still uses NativePaimonV2TableScan, but checkAnswer is the only assertion here.
The catch is that the answer comes out the same either way. If the native scan bails out, Spark falls back to its own Paimon reader, which merges by primary key and returns those same two rows, so the test stays green even if it never touches the native path. t_mor at line 234 is exactly that combination: no native scan, correct answer.
And the bail-out is a real possibility here. PaimonScanSupport.scala:162-173 drops the native plan when a split isn't raw-readable, and PaimonConvertProvider.scala:52 and :95 then quietly leave Spark's own scan in place.
Every other test in this suite that expects a native scan says so, directly (lines 49, 59, 73, 91, 105, 116, 142, 169, 408) or through checkSparkAnswerAndNativePaimonScan and executedNativeScan. Would it be worth doing the same here? It would also make this the first test pinning the native path for a COW primary-key table past its first commit.
One line, in case it helps:
| checkAnswer(df, Seq(Row(1, "updated"), Row(2, "b"))) | |
| checkAnswer(df, Seq(Row(1, "updated"), Row(2, "b"))) | |
| assertNativePaimonScanApplied(df) |
And if the scan turns out not to stay native after a second commit, that feels worth knowing too.
2 participants
Which issue does this PR close?
Closes #2451
Rationale for this change
The existing Paimon COW test verifies only a single commit. It does not protect against stale rows being returned after a later commit updates an existing primary key.
This change adds multi-commit correctness coverage while keeping the native scan implementation unchanged.
What changes are included in this PR?
Updates the existing Paimon COW primary-key integration test to:
NativePaimonV2TableScan.Are there any user-facing changes?
No user-facing changes.
How was this patch tested?
UT.