Skip to content

[IcebergIO] Fix partition values written by hash distribution - #41012

Open
ahmedabu98 wants to merge 3 commits into
apache:masterfrom
ahmedabu98:iceberg-hash-partition-values
Open

ahmedabu98 wants to merge 3 commits into
apache:masterfrom
ahmedabu98:iceberg-hash-partition-values

Conversation

@ahmedabu98

Copy link
Copy Markdown
Contributor

Use RecordWriterManager to write through the HASH distribution mode. RecordWriterManager determines a destination file's partition from its rows instead of an incoming partition path. Using the parsed partition path directly resulted in some errors:

  • year partitions stored the calendar year instead of years since 1970 (2024 instead of 54).
  • Strings with spaces or special characters stayed URL-encoded (a b/c became a+b%2Fc).
  • A null string became the string "null", so it shared a partition with real "null" values.
  • A null boolean became false.
  • Binary values became the bytes of their base64 text.
  • Nulls in numeric and date partitions, including year, month, day and hour partitions, crashed the write.
  • Identity partitions on timestamp or time columns crashed the write.

Added tests to make sure these are covered against future changes.

RecordWriterManager now numbers files per partition path, so a null and the string “null” no longer write the same file.
Also fixed Timestamp.MICROS, which used to crash.

@github-actions github-actions Bot added java io iceberg Iceberg IO connector size/m M: 50-249 lines changed labels Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @chamikaramj for label iceberg.
Backup reviewer: Abacn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.25%. Comparing base (acca614) to head (4e4fd8c).
⚠️ Report is 120 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #41012   +/-   ##
=========================================
  Coverage     58.25%   58.25%           
  Complexity     2277     2277           
=========================================
  Files          1136     1136           
  Lines        169341   169341           
  Branches       1482     1482           
=========================================
  Hits          98647    98647           
  Misses        68146    68146           
  Partials       2548     2548           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Reminder, please take a look at this pr: @chamikaramj

…ition-values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant