Skip to content

fix(google-cloud): a BigQuery table created by a concurrent run is not an error - #330

Merged
aaaaahaaaaa merged 1 commit into
mainfrom
fix/bigquery-table-creation-race
Sep 9, 2026
Merged

fix(google-cloud): a BigQuery table created by a concurrent run is not an error#330
aaaaahaaaaa merged 1 commit into
mainfrom
fix/bigquery-table-creation-race

Conversation

@aaaaahaaaaa

Copy link
Copy Markdown
Contributor

Summary

BigQueryDestination._create_table raised on a 409 Conflict from BigQuery. Two runs writing
different partitions of the same asset whose table does not exist yet (a backfill, or a queue burst
after downtime, or two runs queued back to back on a fresh destination) both see no table, both create
it, and the loser fails with Already Exists, which fail-fast turns into a failed run with every
other asset canceled. _ensure_dataset already tolerated the same race for datasets; _create_table
now does the same and loads into the table that just appeared, which has the schema it wanted.

Found while materializing a fresh destination locally: two job runs for consecutive days executed in
parallel and the second failed on facebook_ads.campaigns__<account>. Production schedules one run
per job per day, which is why it has not shown there.

Verification

  • New test: get_table reports no table, create_table raises Conflict, the rows still load.
  • uv run --frozen pytest packages/interloper-google-cloud: 124 passed. ruff and ty clean.

By Digitl

…t an error

Two runs writing different partitions of the same new table race on its creation, and the
loser got a 409 Conflict that failed its whole run under fail-fast. Dataset creation already
tolerated that; table creation now does too and loads into the table that just appeared.

By Digitl
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@aaaaahaaaaa
aaaaahaaaaa merged commit e517712 into main Sep 9, 2026
11 checks passed
@aaaaahaaaaa
aaaaahaaaaa deleted the fix/bigquery-table-creation-race branch September 9, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant