impl(bigquery): use arrow format with jobs.query - #6472
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds support for Arrow serialization in BigQuery query results, allowing the SDK to request and process Arrow record batches directly. Key changes include updating the query execution to request Arrow format with Zstd compression, implementing Arrow-to-Value conversion for various data types, and parsing Arrow schemas into BigQuery table schemas. Feedback on the changes suggests using unsigned_abs() to prevent overflow panics when formatting intervals, avoiding unnecessary clones when parsing range objects, and ensuring DataType::Interval is correctly mapped to "INTERVAL" in the schema conversion logic.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6472 +/- ##
==========================================
- Coverage 96.52% 96.35% -0.17%
==========================================
Files 304 304
Lines 87969 88401 +432
==========================================
+ Hits 84908 85180 +272
- Misses 3061 3221 +160 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Trying out arrow support on jobs.query. This is gonna break on CI because support for it is behind an allowlist.