Skip to content

[Feature Request]: Add possibility of configuring per-job reservation slots in BigQueryIO connector #41287

Open

Description

@mgorsk1

What would you like to happen?

BigQuery load jobs can use either the shared slot pool or reserved slots. Currently, when using BigQueryIO, using reserved slots relies on a reservation assignment being configured for the project.

This makes it difficult to control reservation usage for individual Beam workloads. Assigning a PIPELINE reservation at the project level affects other applicable jobs in the same project, which might not be desirable when only specific Beam workloads should consume a particular reservation.

The BigQuery API supports specifying a reservation directly at the job level through JobConfiguration.reservation. The field accepts a reservation in the following format: projects/{project}/locations/{location}/reservations/{reservation}. If it is not specified, BigQuery falls back to its reservation-assignment rules.

Proposed change

Add an optional withReservation(...) configuration to BigQueryIO, for example:

BigQueryIO.writeTableRows()
.to(table)
.withMethod(Method.FILE_LOADS)
.withReservation("projects/my-project/locations/us/reservations/my-reservation");

When configured, BigQueryIO should propagate this value to JobConfiguration.reservation for the BigQuery jobs it creates. When it is not configured, the existing behavior should remain unchanged.

This would allow Beam users to route specific BigQuery jobs to a reservation without requiring a project-wide reservation assignment.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions