Skip to content

[core] Preserve partial-update sequence-group deletion state - #10296

Closed
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:fix/partial-update-delete-state-20260928
Closed

leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:fix/partial-update-delete-state-20260928

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Fix whole-row deletion state being lost while merging partial updates.

PartialUpdateMergeFunction.add() currently clears currentDeleteRow for every input. After a configured sequence group deletes a row, a subsequent retraction or an update to another group can clear that state. If an earlier insert was seen, the merged result becomes INSERT again, and lookup compaction can emit an update changelog instead of a delete.

Preserve the deletion state across inputs and initialize it in reset(). Clear it only for a normal non-sequence-group update or an accepted INSERT/UPDATE_AFTER to a group configured in partial-update.remove-record-on-sequence-group. The existing newer-or-equal group-version comparison still applies. Retractions, unrelated groups, and older or all-null ordering fields must not revive a deleted row.

Also preserve partial-update.remove-record-on-delete across a following UPDATE_BEFORE, while allowing a subsequent insert. Document the sequence-group deletion and restoration semantics. No on-disk format changes are introduced.

Tests

Followed a red/green TDD workflow with synthetic records:

  • Before the production fix: 70 tests ran; all 15 new regression cases failed with an unexpected INSERT, while the 55 existing cases passed.
  • After the fix: the same 70 tests passed.
  • Expanded final verification: 90 tests passed with zero failures, errors, or skips. This run did not enable fast-build, so the normal lifecycle quality checks were enabled.

Regression coverage includes:

  • All 120 orderings of five sequence-group deletes, for both equal and increasing internal sequence numbers.
  • Subsequent DELETE/UPDATE_BEFORE, unrelated updates, stale controlling-group updates, and valid reinsertion.
  • Carrying a tombstone across a merge boundary and resetting between keys.
  • Composite ordering fields and five projections, including an empty and a reordered projection.
  • Lookup changelog generation with the old row supplied either by input or by lookup: the result remains deleted and emits a single DELETE, not an update pair.

Focused red/green command:

mvn -B -ntp -pl paimon-core -am -Pfast-build \
  -DfailIfNoTests=false -DwildcardSuites=none \
  '-Dtest=PartialUpdateMergeFunctionTest,LookupChangelogMergeFunctionWrapperTest' test

Final verification command:

mvn -B -ntp -pl paimon-core -am \
  -DfailIfNoTests=false -DwildcardSuites=none \
  '-Dtest=PartialUpdateMergeFunctionTest,LookupChangelogMergeFunctionWrapperTest,LookupMergeFunctionTest,LookupMergeFunctionUnitTest,LookupMergeTreeCompactRewriterTest,PartialUpdateTableTest,PrimaryKeyPartialUpdateBlobTest' test

The full repository and Flink version matrix were not run locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant