Skip to content

Add equality-delete apply on PyArrow reads - #3805

Open
daviddallakyan2005 wants to merge 1 commit into
apache:mainfrom
daviddallakyan2005:equality-delete-reads
Open

Add equality-delete apply on PyArrow reads#3805
daviddallakyan2005 wants to merge 1 commit into
apache:mainfrom
daviddallakyan2005:equality-delete-reads

Conversation

@daviddallakyan2005

Copy link
Copy Markdown
Contributor

Closes #3270

Rationale for this change

Index equality delete files and apply them on PyArrow scans so CDC and merge-on-read tables can be read. Equality deletes still cannot be written.

This ports the DeleteFileIndex work from #3285 onto current main, skips bounds prune when an equality field has been dropped, and left anti joins matching equality delete rows after positional deletes. It supersedes #3285, which is open and conflicting. Credit to Alex Stephen for the index. Please close #3285 if this path is preferred.

When a data file has equality deletes, the scan concatenates that file’s record batches, aligns each matching equality-delete table by field id then concats, adds is_null / fill_null (and is_nan for floats) helper columns per key, left anti joins once, then yields the survivors; files with only position deletes still yield each scanner batch.

Are these changes tested?

Yes.

make test
3962 passed, 3 skipped, 1570 deselected

make lint
All prek hooks Passed

make test PYTEST_ARGS="-v -k equality"
168 passed, 5367 deselected

uv run python -m pytest tests/table/test_delete_file_index.py tests/io/test_pyarrow.py tests/catalog/test_scan_planning_models.py tests/table/test_validate.py tests/table/test_init.py -m "(unmarked or parametrize) and not integration" -v
371 passed, 57 deselected

make test-integration PYTEST_ARGS="-v --tb=short"
1350 passed, 71 skipped, 3 failed, 4111 deselected

The three integration failures match main without this diff. test_read_spark_written_puffin_dv raises DeprecationWarning for puffin.to_vector(). Two DuckDB tests failed to download the iceberg extension.

Are there any user-facing changes?

Yes. Tables that already store equality deletes can be scanned.

Index equality delete files and apply them on scan so CDC and
merge-on-read tables can be read. Equality deletes still cannot
be written.

Closes apache#3270.

Co-authored-by: Alex Stephen <alexstephen@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Equality Delete support

1 participant