[core] Align null stats columns of partial writes by write schema - #10255
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
IcebergDataFileMeta.create interpreted null stats columns as covering the whole table schema, but a null value of valueStatsCols means the stats cover the whole WRITE schema. For a partial write the stats row follows the write-column order — which can differ from the table schema order, for example for a MERGE INTO whose SET clause lists columns differently — and nested writes record leaf paths, so bounds and null counts drifted onto the wrong fields in the Iceberg manifest. Derive the explicit stats column list from the write columns in their recorded order, mapping leaf paths to their top-level field. Assisted-by: GLM-5.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
IcebergDataFileMeta.createtreated a nullstatsColumnsas covering the whole Iceberg (table) schema in table order. A nullvalueStatsColsinstead means the stats cover the whole write schema, whose column order and set can differ from the table for a partial write. As a result the min/max bounds and null counts drifted onto the wrong Iceberg fields in the manifest, and a strict-subset write could read past the end of the stats row.This passes the file's
writeColsintocreate. WhenstatsColumnsis null andwriteColsis present, it builds the explicit stats-column list from the write columns in their recorded order, mapping a nested leaf path to its top-level field so each stats slot is attributed to the correct column. Full writes keepwriteColsnull and are unaffected.This closes #10253.
Tests
IcebergDataFileMetaTest.testNullStatsColumnsWithWriteColsAlignByWriteSchemapins that a partial write of(b, k)in write-column order maps the bounds and null counts tobandkand does not drift onto the unwritten columna.IcebergDataFileMetaTest.testNullStatsColumnsWithNestedWriteColsMapToTopLevelpins that a nested leaf path attributes its stats to the top-level field rather than to a neighboring column.API and Format
No.
Documentation
No.