fix(bq_driver): transparently handle transient stream disconnections and resume HTAPI reads by KanchanShu · Pull Request #1692 · googleapis/cpp-bigquery-odbc

KanchanShu · 2026-10-22T07:49:38Z

Description

Fixes an issue where long-running queries using the High-Throughput API (HTAPI / BigQuery Storage Read API) fail prematurely when intermediate network devices, proxies, or load balancers drop the connection (e.g., HTTP/2 RST_STREAM, TCP connection resets, proxy timeouts, or UNAVAILABLE errors).

Previously, any stream error in ReadNextResultsFromStream() immediately failed the query and returned a fatal error to the user application.

This PR implements transparent stream re-attachment and resumption:

Transient Error Detection: Added IsTransientStreamError() to identify retryable status codes (UNAVAILABLE, DEADLINE_EXCEEDED, ABORTED, RESOURCE_EXHAUSTED) as well as INTERNAL/UNKNOWN errors resulting from transport-level drops (rst_stream, connection reset, broken pipe, proxy, timeout, transport: lost connection).
Offset Resumption: Tracks the cumulative row offset (stream_row_offset_) as Arrow record batches are consumed. On a transient failure, the driver re-attaches to the existing ReadSession stream using ReadRowsRequest.set_offset(N) after an exponential backoff delay.
State Management: Persists read_stream_name_, stream_row_offset_, and was_htapi_enabled_ on StatementHandle, and resets them in CloseCursor() to avoid state leakage across reused statement handles.

Key Changes

google/cloud/odbc/bq_driver/internal/odbc_sql_execute_utils.h:
- Declared IsTransientStreamError().
google/cloud/odbc/bq_driver/internal/odbc_sql_execute_utils.cc:
- Implemented IsTransientStreamError() for status code and error message inspection.
- Updated ReadNextResultsFromStream() to execute an exponential backoff retry loop up to max_retries with read_rows_request.set_offset(stream_row_offset_).
- Initialized stream name and offset in FetchBQDataReadArrow().
google/cloud/odbc/bq_driver/internal/odbc_stmt_handle.h:
- Added fields and accessors for read_stream_name_, stream_row_offset_, and was_htapi_enabled_.
google/cloud/odbc/bq_driver/internal/odbc_stmt_handle.cc:
- Added cleanup of stream name, stream row offset, and HTAPI flags in CloseCursor().

Verification

Recompiled and verified that both libgoogle_cloud_odbc_bq_driver_internal.a and libgoogle_cloud_odbc_bq_driver.so build cleanly with zero errors.

shivamd-gpartner · 2026-10-23T06:32:23Z

+      std::string message = status.message();
+      std::transform(message.begin(), message.end(), message.begin(),
+                     [](unsigned char c) { return std::tolower(c); });
+      if (message.find("rst_stream") != std::string::npos ||


how do we know we need to retry for these messages? some probably can be permanent errors and we are unnecessarily retrying for them

Updated, To avoid retrying permanent errors, I removed broad terms (timeout, proxy) and kept only explicit TCP/HTTP-2 transport disconnects (rst_stream, stream removed, connection reset, broken pipe).

shivamd-gpartner · 2026-10-23T06:32:27Z

+  }
+
+  int max_retries = conn_handle->GetDsn().max_retries;
+  if (max_retries <= 0) {


don't change for max_retries = 0, as the user might not want retries

fixed. We now use conn_handle->GetDsn().max_retries directly without overriding 0, so setting MaxRetries=0 disables retries as expected.

shivamd-gpartner · 2026-10-27T05:46:55Z

+#ifdef BQ_DRIVER_INTEGRATION_TESTS
+TEST(StatementTest, SQLExecDirect_HTAPI_StreamRetryAndResume) {
+#ifdef _WIN32
+  _putenv_s("BIGQUERY_ODBC_TEST_INJECT_STREAM_ERROR", "1");


what is the utility of this flag?

Because transient network disconnects (RST_STREAM) cannot be deterministically reproduced against live BigQuery servers in CI, this test-only flag injects a simulated stream error mid-query.

It allows the integration test to reliably verify that the retry loop catches the error, re-attaches to the stream from the last offset (set_offset(N)), and resumes reading all rows without failing the query.

sachinpro · 2026-10-30T19:35:34Z

/gcbrun

sachinpro · 2026-10-30T20:12:36Z

/gcbrun

sachinpro · 2026-10-30T20:28:03Z

/gcbrun

sachinpro · 2026-10-30T22:18:04Z

/gcbrun

…and resume HTAPI reads

…silience_test.cc

…etry

sachinpro · 2026-10-31T01:55:02Z

/gcbrun

sachinpro · 2026-10-31T02:59:26Z

Here is summary of changes I made on top of the original PR to simplify the logic:

1. Implementation Summary: `HtapiStreamRetryPolicy`

Instead of manually tracking row offsets and re-opening ReadRows streams inside ReadNextResultsFromStream, we delegated stream retry and resumption to google-cloud-cpp's built-in ResumableStreamingReadRpc (BigQueryReadReadRowsStreamingUpdater automatically updates request.set_offset(request.offset() + response.row_count()) on every ReadRowsResponse):

odbc_sql_execute_utils.h:
- Defined HtapiStreamRetryPolicy, subclassing ::google::cloud::bigquery_storage_v1::BigQueryReadRetryPolicy and delegating failure classification to IsTransientStreamError.
odbc_sql_execute_utils.cc:
- Configured BigQueryReadRetryPolicyOption (std::make_shared<HtapiStreamRetryPolicy>(max_retries)) and BigQueryReadBackoffPolicyOption (ExponentialBackoffPolicy(100ms, 2000ms, 2.0)) on the Options passed to CreateReadSession and GetReadRowsStream in FetchBQDataReadArrow.
- Restored ReadNextResultsFromStream, odbc_stmt_handle.h, odbc_stmt_handle.cc, and odbc_sql_requests.cc to match origin/main with zero extra state fields.
odbc_sql_execute_utils_test.cc:
- Added unit tests for both IsTransientStreamError and HtapiStreamRetryPolicy (both passing).

2. Why `ReadAPI_IdleProxy_ConnectionDroppedBetweenQueries` Is Not Enough

It never triggers a proxy kill when keepalive is active: In resilience-tests.sh, ReadAPI_IdleProxy_ConnectionDroppedBetweenQueries runs with --idle-timeout 40 and a 45s pause. Because gRPC keepalive sends HTTP/2 PINGs every 30s (GRPC_ARG_KEEPALIVE_TIME_MS = 30000), the connection is never idle for 40s and asserts kills == 0 and reconnects == 0.
It pauses between separate queries, not mid-stream: The pause happens after SQLFreeStmt(SQL_CLOSE) when no ReadRows stream is active, so it does not test mid-stream offset resumption.
Why raw TCP RST (--mode reset) and a single drop are also not enough to distinguish main from this branch:
- In gRPC's POSIX TCP layer (tcp_annotate_error), a clean socket ECONNRESET/EOF (read_bytes <= 0) is explicitly tagged with GRPC_STATUS_UNAVAILABLE, which main's BigQueryReadRetryTraits already treats as retryable.
- However, when a corporate proxy or gateway injects an HTTP error response (502 Bad Gateway) or truncates/corrupts a TLS record mid-stream (--mode corrupt), secure_endpoint.cc fails with Unwrap failed (TSI_DATA_CORRUPTED) and chttp2_transport.cc closes the stream with StatusCode::kUnknown ("Stream removed (Unwrap failed (TSI_DATA_CORRUPTED))").
- Furthermore, inside google-cloud-cpp's ResumableStreamingReadRpc::Read, the loop condition is while (!retry_policy->IsExhausted() && (has_received_data_ || retry_policy->OnFailure(last_status))). If a tunnel has already delivered Batch 1 (has_received_data_ == true), the first disconnect short-circuits retry_policy->OnFailure(last_status) and resets has_received_data_ = false. A second disconnect on the reconnected tunnel before a full batch arrives (--reconnect-kill-after-bytes 100000 --max-kills 2) forces ResumableStreamingReadRpc to consult retry_policy->OnFailure(last_status).

3. Verification Results (`origin/main` vs. `htapi_stream_retry_fix`)

We updated idle_connect_proxy.py, resilience-tests.sh, and SQLExecDirect_HTAPI_StreamRetryAndResume to run with --kill-after-bytes 4000000 --reconnect-kill-after-bytes 100000 --max-kills 2 --mode corrupt:

Target	Result	Details
`origin/main`	FAILED	Tunnel #3 killed at `4,013,127 B`; Tunnel #4 reconnected at `offset = 668` and killed at `105,765 B`. Failed immediately with: `SQLFetch = [Google][ODBC BigQuery Driver] [BigQuery] Stream removed (Unwrap failed (TSI_DATA_CORRUPTED)) (500) SQLSTATE=HY000` (`kills: 2`, `reconnects: 1`, `storage_tunnels: 2`).
`htapi_stream_retry_fix`	PASSED	Tunnel #3 killed at `4,013,127 B`; Tunnel #4 reconnected and killed at `105,765 B`; `HtapiStreamRetryPolicy` classified `kUnknown ("Stream removed...")` as transient, opened Tunnel #5 (`kills: 2`, `reconnects: 2`, `storage_tunnels: 3`), resumed at `offset = 668`, and verified all 3,000 rows × 300 columns.

sachinpro · 2026-10-31T03:02:07Z

Windows benchmarks and linux benchmarks show no performance degradation.

sachinpro

Thanks a lot for fixing this.

I simplified the logic and added the integration tests to google/cloud/odbc/integration_tests/odbc_driver_tests/resilience_test.cc.

shivamd-gpartner reviewed Oct 23, 2026

View reviewed changes

shivamd-gpartner reviewed Oct 27, 2026

View reviewed changes

KanchanShu force-pushed the htapi_stream_retry_fix branch from 0f7d73a to ff3c244 Compare October 27, 2026 12:58

KanchanShu marked this pull request as ready for review October 27, 2026 13:05

KanchanShu requested a review from a team as a code owner October 27, 2026 13:05

sachinpro force-pushed the htapi_stream_retry_fix branch 2 times, most recently from 0ce78da to 57f7077 Compare October 30, 2026 19:33

sachinpro force-pushed the htapi_stream_retry_fix branch 2 times, most recently from 62caf1a to cc3cff8 Compare October 30, 2026 20:11

sachinpro force-pushed the htapi_stream_retry_fix branch from cc3cff8 to 1cbd0d0 Compare October 30, 2026 20:26

sachinpro force-pushed the htapi_stream_retry_fix branch from 1cbd0d0 to 8550ef6 Compare October 30, 2026 22:17

KanchanShu and others added 4 commits October 31, 2026 01:43

fix(bq_driver): transparently handle transient stream disconnections …

e1c1bb8

…and resume HTAPI reads

addded test case and fixed review comments

43b914b

test(resilience): move SQLExecDirect_HTAPI_StreamRetryAndResume to re…

83ff965

…silience_test.cc

fix(bq_driver): use custom BigQueryReadRetryPolicy for HTAPI stream r…

d0f0f81

…etry

sachinpro force-pushed the htapi_stream_retry_fix branch from 8550ef6 to d0f0f81 Compare October 31, 2026 01:45

sachinpro deployed to internal October 31, 2026 01:52 — with GitHub Actions Active

sachinpro deployed to internal October 31, 2026 02:19 — with GitHub Actions Active

sachinpro deployed to internal October 31, 2026 02:43 — with GitHub Actions Active

sachinpro approved these changes Oct 31, 2026

View reviewed changes

sachinpro merged commit dda551f into googleapis:main Oct 31, 2026
54 of 66 checks passed

Conversation

KanchanShu commented Oct 22, 2026

Description

Key Changes

Verification

Uh oh!

shivamd-gpartner Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

KanchanShu Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

shivamd-gpartner Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

KanchanShu Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

shivamd-gpartner Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

KanchanShu Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

sachinpro commented Oct 30, 2026

Uh oh!

sachinpro commented Oct 30, 2026

Uh oh!

sachinpro commented Oct 30, 2026

Uh oh!

sachinpro commented Oct 30, 2026

Uh oh!

sachinpro commented Oct 31, 2026

Uh oh!

sachinpro commented Oct 31, 2026

1. Implementation Summary: HtapiStreamRetryPolicy

2. Why ReadAPI_IdleProxy_ConnectionDroppedBetweenQueries Is Not Enough

3. Verification Results (origin/main vs. htapi_stream_retry_fix)

Uh oh!

sachinpro commented Oct 31, 2026

Uh oh!

sachinpro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

1. Implementation Summary: `HtapiStreamRetryPolicy`

2. Why `ReadAPI_IdleProxy_ConnectionDroppedBetweenQueries` Is Not Enough

3. Verification Results (`origin/main` vs. `htapi_stream_retry_fix`)