Skip to content

fix: handle mixed dictionary/plain ArrowScan batches - #3667

Open
GayathriSrividya wants to merge 2 commits into
apache:mainfrom
GayathriSrividya:fix/arrowscan-mixed-dictionary-string
Open

fix: handle mixed dictionary/plain ArrowScan batches#3667
GayathriSrividya wants to merge 2 commits into
apache:mainfrom
GayathriSrividya:fix/arrowscan-mixed-dictionary-string

Conversation

@GayathriSrividya

Copy link
Copy Markdown
Contributor

closes #3260

Summary

  • decode dictionary-encoded columns in ArrowScan.to_table before concatenation
  • keep permissive concat behavior for other schema variations
  • add a regression test covering mixed plain-string and dictionary-encoded batches

Testing

  • /opt/homebrew/bin/ruff format pyiceberg/io/pyarrow.py tests/io/test_pyarrow.py
  • /opt/homebrew/bin/ruff check --fix pyiceberg/io/pyarrow.py tests/io/test_pyarrow.py
  • PYTHONPATH=. uv run --project . pytest tests/io/test_pyarrow.py -k mixed_dictionary_and_plain_string_batches -xvs

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ArrowScan to_table fails if the data is mixed between dict-encoded strings and plain strings.

1 participant