Skip to content

fix: support bigquery array of struct type for dbt#1374

Open
sahmed1991 wants to merge 3 commits into
datacontract:mainfrom
sahmed1991:support-bigquery-array-struct-type
Open

fix: support bigquery array of struct type for dbt#1374
sahmed1991 wants to merge 3 commits into
datacontract:mainfrom
sahmed1991:support-bigquery-array-struct-type

Conversation

@sahmed1991

Copy link
Copy Markdown

Summary

Problem
datacontract import dbt (BigQuery adapter) failed with Unsupported type ARRAY<STRUCT<>> in bigquery json definition because map_type_from_bigquery only understood bare BigQuery type names, not compound types like ARRAY<STRUCT<...>>, STRUCT, STRING(100), or NUMERIC(10, 2).

Changes

datacontract/imports/bigquery_importer.py

  • Added ParsedBigQueryType dataclass modelling base_type, optional params (for STRING(n) / NUMERIC(p, s)), optional element_type (for ARRAY / RANGE), and ordered fields (for STRUCT<...>).
    Added parse_bigquery_type(str) -> ParsedBigQueryType — a proper recursive descent parser that handles arbitrary nesting, empty struct bodies (STRUCT<>), whitespace, anonymous struct fields, and parameterised inner types.
  • Added build_schema_property_from_bigquery_type(name, type_str, description, required) — builds a full SchemaProperty tree with nested items (arrays/ranges) and properties (structs), while lifting STRING(n) length to logicalTypeOptions.maxLength and NUMERIC(p, s) precision/scale to customProperties (matching ODCS v3.1.0 rules used elsewhere in the codebase).
  • Rewrote map_type_from_bigquery to delegate to the parser and return the logical mapping of the outermost type. Added ARRAY, STRUCT, RECORD, RANGE to the base-type mapping. Public signature is unchanged; existing callers continue to work.

datacontract/imports/dbt_importer.py

  • create_field now branches on adapter_type == "bigquery" and, for BigQuery columns, uses the new builder so nested items/properties land in the output data contract. Non-BigQuery adapters keep the previous flat behaviour.
  • Extracted the column-level metadata attach step (unique, primary_key, custom_props) into _apply_column_metadata so it works for either build path.

Tests

New file tests/test_import_bigquery_type_parser.py (54 tests) covers:

  • parse_bigquery_type: scalars, params, arrays, empty structs, ARRAY<STRUCT<>>, nested structs, deep nesting, whitespace, anonymous fields, and error cases (empty input, unbalanced </(, trailing garbage).
  • map_type_from_bigquery: simple types, parameterised types, arrays/structs mapping to array/object, and error handling.
  • build_schema_property_from_bigquery_type: scalars, STRING(n) maxLength, NUMERIC(p, s) precision/scale, arrays with items, structs with named/anonymous properties, empty struct, ARRAY<STRUCT<>> (the originally reported case), deeply nested types, and required/description propagation.

*tests/test_import_dbt.py: added 4 end-to-end tests that build an in-memory dbt BigQuery manifest and assert the importer produces the correct nested ODCS structure for ARRAY<STRUCT<>>, ARRAY<STRUCT<field ...>>, STRUCT<...> columns, and a regression guard for scalar columns.

Verification

  • 69 tests pass across tests/test_import_bigquery.py, tests/test_import_bigquery_type_parser.py, and tests/test_import_dbt.py (was 15 baseline).
  • 242 tests pass across the broader -k "dbt or bigquery" selection (3 unrelated skips), confirming no regressions.
  • ruff check and ruff format --check both pass on all touched files.
  • Tests pass (uv run pytest)
  • Code formatted (uv run ruff check --fix && uv run ruff format)
  • Docs updated (if relevant)
  • CHANGELOG.md entry added

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant