Skip to content

fix: filter stale physical row ids in take - #8401

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-7508-1
Open

fix: filter stale physical row ids in take#8401
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-7508-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • validate non-stable physical row IDs against the dataset deletion mask before TakeExec reads data
  • drop index entries for deleted rows and removed fragments from both the address array and carried input columns
  • add regression coverage for stale physical row IDs that previously caused a RecordBatch length mismatch

Root cause

TakeExec::get_row_addrs only built a validity mask when stable row IDs were enabled. With default physical row IDs, stale index entries were passed through unchanged. Fragment reads omit deleted rows, so the taken columns could contain fewer rows than the upstream FTS batch and merge_with_schema failed with unequal RecordBatch lengths.

Validation

  • cargo test -p lance io::exec::take::tests (13 passed)
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #7508

@github-actions github-actions Bot added the bug Something isn't working label Aug 7, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

This closes the legacy TakeExec parity gap: physical row IDs are checked against the current deletion mask before carried columns and fetched data are merged, matching the existing stable-ID and modern live-view behavior. Deleted and removed-fragment hits now drop while surviving IDs, scores, and fetched values remain aligned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FTS query fails with "all columns in a record batch must have the same length" after frequent index optimize.

0 participants