Skip to content

test: add mixed-format schema evolution coverage#354

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:mixed-schema-tests
Open

test: add mixed-format schema evolution coverage#354
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:mixed-schema-tests

Conversation

@QuakeWang
Copy link
Copy Markdown
Contributor

Purpose

Linked issue: #258

Rust reader already had coverage for Java/Spark-generated full type boundary values, but schema evolution compatibility across mixed data file formats was not explicitly covered. In particular, existing fixtures did not verify that Parquet/ORC/Avro files written under different schema versions are read consistently by field id, with null-fill for added columns and casts for promoted types.

Brief change log

  • Add format_schema_evolution_add_column Spark fixture with Parquet old-schema files and ORC/Avro new-schema files.
  • Add format_schema_evolution_type_promotion Spark fixture with Parquet INT files and ORC/Avro BIGINT files.
  • Add Rust integration tests that assert:
    • planned files include Parquet, ORC, and Avro
    • planned files include multiple schema ids
    • reader output preserves null-fill and INT to BIGINT cast semantics

Tests

  • cargo test -p paimon-integration-tests test_read_format_schema_evolution -- --nocapture

API and Format

Documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant