[spark] Fix MERGE INTO silently skipping WHEN NOT MATCHED BY SOURCE actions

[kerwin-zk]

Purpose

MergeIntoPaimonTable extracts the target-only conjuncts of the merge condition and uses them to prune the target table before the full outer join (targetOnlyCondition / filteredTargetPlan, MergeIntoPaimonTable.scala:63-69).

That pruning is sound for WHEN MATCHED and WHEN NOT MATCHED: a target row that fails a target-only conjunct can never satisfy the whole merge condition, so it can never be matched, and dropping it cannot change the outcome of those actions.

It is not sound for WHEN NOT MATCHED BY SOURCE. The pruned-away rows are exactly the population that clause is defined over, so their actions are silently skipped — no error, no warning, just fewer rows changed.

Repro on a partitioned table holding pt = 'p1' and pt = 'p2' rows, with a source that only contains a = 1:

MERGE INTO target t
USING source s
ON t.a = s.a AND t.pt = 'p1'
WHEN MATCHED THEN UPDATE SET t.b = s.b
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET t.c = 'stale'

Every pt = 'p2' row should be updated to stale (no source row can ever match it), but none of them is:

Expected                Actual
 [1,100,c1,p1]          [1,100,c1,p1]
 [2,20,stale,p1]        [2,20,stale,p1]
![3,30,stale,p2]        [3,30,c3,p2]
![4,40,stale,p2]        [4,40,c4,p2]

The pruning was introduced together with MERGE INTO itself in #2331 (2023-11-17), one month before WHEN NOT MATCHED BY SOURCE was added in #2517 (2023-12-22), and its safety argument was never revisited.

Note the V2 row-level paths (ReplaceData / WriteDelta) are rewritten by Spark and are not affected, so today the same statement produces different results depending on whether the table qualifies for SparkTable.supportsV2RowLevelOps. Primary key tables never qualify, so they always take the affected V1 path.

Fix: disable the pruning when the merge has any WHEN NOT MATCHED BY SOURCE action. Setting targetOnlyCondition to None covers all three places it feeds — filteredTargetPlan, findCandidateDataSplits and targetDSWithFilePathCol — so there is no path left that can drop those rows.

Tests

CI

[YannByron]

@kerwin-zk Before settling on the fix, I think we should first agree on the semantics of the ONcondition in MERGE INTO, since the whole patch hinges on it: does ON define the scope of the target rows the statement may touch, or is it purely the predicate that decides which target rows count as matched?

Two readings, using your repro:

1. ON scopes the merge. t.pt = 'p1' restricts the statement to the p1 rows; anything outside that scope is simply out of the statement's reach, so WHEN NOT MATCHED BY SOURCE should only consider p1 rows. Under this reading the current pruning is correct and the p2 rows should stay untouched.
2. ON is only the matching predicate. WHEN NOT MATCHED BY SOURCE is defined over every target row for which no source row satisfies the merge condition. The p2 rows trivially qualify, so they must be updated — which is what this PR asserts.

The two readings give opposite expected results for the same statement, so which one we consider correct determines whether this is a bug fix or a behavior change.

[YannByron]

I checked the Delta Lake docs — it is reading 2: whenNotMatchedBySource is defined over every target row that does not match any source row based on the merge condition, and the sanctioned way to narrow that set is a clause-level condition, not ON.

+1 on the fix.

https://docs.databricks.com/aws/en/delta/merge