Skip to content

[core] Preserve partial-update sequence-group deletion state - #10300

Open
qingwei727 wants to merge 1 commit into
apache:masterfrom
qingwei727:fix/partial-update-delete-state
Open

qingwei727 wants to merge 1 commit into
apache:masterfrom
qingwei727:fix/partial-update-delete-state

Conversation

@qingwei727

Copy link
Copy Markdown
Contributor

Purpose

Fix whole-row deletion state being lost while merging partial updates.

PartialUpdateMergeFunction.add() currently clears currentDeleteRow for every input. After a configured sequence group deletes a row, a subsequent retraction
or an update to another group can clear that state. If an earlier insert was seen, the merged result becomes INSERT again, and lookup compaction can emit an
update changelog instead of a delete.

Preserve the deletion state across inputs and initialize it in reset(). Clear it only for a normal non-sequence-group update or an accepted INSERT/
UPDATE_AFTER to a group configured in partial-update.remove-record-on-sequence-group. The existing newer-or-equal group-version comparison still applies.
Retractions, unrelated groups, and older or all-null ordering fields must not revive a deleted row.

Also preserve partial-update.remove-record-on-delete across a following UPDATE_BEFORE, while allowing a subsequent insert. Document the sequence-group
deletion and restoration semantics. No on-disk format changes are introduced.

Tests

Followed a red/green TDD workflow with synthetic records:

  • Before the production fix: 65 tests ran; all 10 new regression cases failed with an unexpected INSERT, while the 55 existing cases passed.
  • After the fix: the same 65 tests passed.
  • Expanded final verification: 85 tests passed with zero failures, errors, or skips. This run did not enable fast-build, so the normal lifecycle quality
    checks were enabled.

Regression coverage includes:

  • Subsequent DELETE/UPDATE_BEFORE, unrelated updates, stale controlling-group updates, and valid reinsertion.
  • Carrying a tombstone across a merge boundary and resetting between keys.
  • Composite ordering fields with an empty projection and a reordered projection.
  • Lookup changelog generation with the old row supplied either by input or by lookup: the result remains deleted and emits a single DELETE, not an update
    pair.

Focused red/green command:

mvn -B -ntp -pl paimon-core -am -Pfast-build \
  -DfailIfNoTests=false -DwildcardSuites=none \
  '-Dtest=PartialUpdateMergeFunctionTest,LookupChangelogMergeFunctionWrapperTest' test

Port apache#10296 at b43e36137f2b5b3f51aaf7290c6f96d67744675a.
Keep the production fix and deletion/restoration documentation unchanged.
Reduce regression coverage to six methods and ten parameterized invocations:
remove five-group permutations and retain empty/reordered projections.

Co-Authored-By: 仟弋 <yejunhao.yjh@alibaba-inc.com>
Co-Authored-By: Codex <noreply@openai.com>
AI-Model: unknown
AI-Contributed/Feature: 11/11
AI-Contributed/UT: 223/223
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant