Skip to content

feat: add support for _spec_id metadata column#2695

Open
hsiang-c wants to merge 1 commit into
apache:mainfrom
hsiang-c:meta_col_spec_id
Open

feat: add support for _spec_id metadata column#2695
hsiang-c wants to merge 1 commit into
apache:mainfrom
hsiang-c:meta_col_spec_id

Conversation

@hsiang-c

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

  • If a projection includes _spec_id, which is a constant like the _file metadata column for all rows, add it to RecordBatchTransformerBuilder
  • Fix the partition value of manifest entry. The reason is we have the following partitions spec in crates/iceberg/testdata/example_table_metadata_v2.json and the x column is always 1 for all data files, so the identity partition transform value should be 1.
  "default-spec-id": 0,
  "partition-specs": [
    {
      "spec-id": 0,
      "fields": [
        {"name": "x", "transform": "identity", "source-id": 1, "field-id": 1000}
      ]
    }
  ],

Are these changes tested?

  • Yes, unit tests.

@hsiang-c

Copy link
Copy Markdown
Author

FYI @advancedxy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for projecting metadata columns _pos, _spec_id, and _partition in table scan

1 participant