Skip to content

[core] Align null stats columns of partial writes by write schema - #10255

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:m/core-077-iceberg-stats
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:m/core-077-iceberg-stats

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

IcebergDataFileMeta.create treated a null statsColumns as covering the whole Iceberg (table) schema in table order. A null valueStatsCols instead means the stats cover the whole write schema, whose column order and set can differ from the table for a partial write. As a result the min/max bounds and null counts drifted onto the wrong Iceberg fields in the manifest, and a strict-subset write could read past the end of the stats row.

This passes the file's writeCols into create. When statsColumns is null and writeCols is present, it builds the explicit stats-column list from the write columns in their recorded order, mapping a nested leaf path to its top-level field so each stats slot is attributed to the correct column. Full writes keep writeCols null and are unaffected.

This closes #10253.

Tests

IcebergDataFileMetaTest.testNullStatsColumnsWithWriteColsAlignByWriteSchema pins that a partial write of (b, k) in write-column order maps the bounds and null counts to b and k and does not drift onto the unwritten column a.

IcebergDataFileMetaTest.testNullStatsColumnsWithNestedWriteColsMapToTopLevel pins that a nested leaf path attributes its stats to the top-level field rather than to a neighboring column.

API and Format

No.

Documentation

No.

IcebergDataFileMeta.create interpreted null stats columns as covering
the whole table schema, but a null value of valueStatsCols means the
stats cover the whole WRITE schema. For a partial write the stats row
follows the write-column order — which can differ from the table
schema order, for example for a MERGE INTO whose SET clause lists
columns differently — and nested writes record leaf paths, so bounds
and null counts drifted onto the wrong fields in the Iceberg
manifest.

Derive the explicit stats column list from the write columns in their
recorded order, mapping leaf paths to their top-level field.

Assisted-by: GLM-5.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Iceberg manifest stats are misaligned for partial-column writes with null value-stats columns

1 participant