Skip to content

feat(pubsub): integrate publish hedging into MessageQueue - #9466

Open
feywind wants to merge 1 commit into
pubsub/hedging-telemetry-and-loggingfrom
pubsub/hedging-queue-integration-and-tests
Open

feywind wants to merge 1 commit into
pubsub/hedging-telemetry-and-loggingfrom
pubsub/hedging-queue-integration-and-tests

Conversation

@feywind

@feywind feywind commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Overview

This PR is Layer 5 of 7 in Stack #9467, which implements Publish Hedging for @google-cloud/pubsub to mitigate high-percentile (p99/p99.9) publish tail latencies by sending backup (hedged) publish RPCs when an in-flight batch takes longer than a configurable hedgeDelay threshold.

Stack Structure

  1. feat(gax): support cancelling API calls via CallOptions.signal #9462 - feat(gax): support AbortSignal cancellation in CallOptions and createApiCall
  2. feat(pubsub): add HedgingOptions validation and token bucket #9463 - feat(pubsub): add HedgingOptions validation and HedgingTokenBucket
  3. feat(pubsub): add CancellationSharer and HedgingScheduler #9464 - feat(pubsub): add CancellationSharer and HedgingScheduler
  4. feat(pubsub): add OpenTelemetry events and logging for hedged publish #9465 - feat(pubsub): add OpenTelemetry events and logging for hedged publish
  5. feat(pubsub): integrate publish hedging into MessageQueue #9466 - feat(pubsub): integrate publish hedging into MessageQueue (this PR)
  6. test(pubsub): add unit tests for publish hedging queue integration #9635 - test(pubsub): add unit tests for publish hedging queue integration
  7. test(pubsub): add publish hedging system tests #9636 - test(pubsub): add publish hedging system tests

Summary of Changes in This PR

This layer wires CancellationSharer, HedgingScheduler, AbortSignal cancellation, and OpenTelemetry/logging into Publisher, MessageQueue._publish, and PubSub.prototype.request:

  • PubSub.prototype.request (handwritten/pubsub/src/pubsub.ts):
    • Checks config.gaxOpts?.signal?.aborted before and after this.getClient_, immediately invoking the callback with a CANCELLED ServiceError if aborted before the GAPIC method is called.
  • Publisher Lifecycle (handwritten/pubsub/src/publisher/index.ts):
    • Instantiates HedgingScheduler in setOptions when hedging is configured (logging 'Hedging rate limited due to lack of tokens.' at debug level when rate-limited) and clears any pending timer on flush().
  • MessageQueue._publish & OrderedQueue (handwritten/pubsub/src/publisher/message-queues.ts):
    • Ensures OrderedQueue never hedges (allowsHedging() returns false).
    • When hedging is enabled on an unordered Queue:
      • Dispatches attempt 0 with its own AbortController and full GAX retry settings.
      • Schedules hedged attempts via hedgingScheduler.scheduleFirstHedge: each hedged attempt logs 'Publishing hedged attempt %i', records PubsubEvents.publishStartHedged, and dispatches a non-retrying (retry: null) publish RPC with timeout: attemptTimeout.milliseconds and its own AbortController.
      • Records PubsubEvents.publishEndHedged when a hedged attempt wins (or PubsubEvents.publishEnd otherwise) and resolves message callbacks with the winning attempt's messageIds.

@feywind
feywind added this pull request to stack #9467 September 29, 2026 02:09
@product-auto-label product-auto-label Bot added the api: pubsub Issues related to the Pub/Sub API. label Sep 29, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements publish hedging support in the Pub/Sub publisher, integrating a hedging scheduler, token bucket, and cancellation coordinator into the message queues, while ensuring ordered queues do not use hedging. It also adds support for abort signals in API requests to handle cancellations immediately. The review feedback suggests replacing shallow copies of gaxOpts with deep copies using extend to prevent accidental mutation of nested configuration options by underlying libraries.

Comment thread handwritten/pubsub/src/publisher/message-queues.ts
Comment thread handwritten/pubsub/src/publisher/message-queues.ts
Comment thread handwritten/pubsub/src/publisher/message-queues.ts
Comment thread handwritten/pubsub/src/publisher/hedging.ts Outdated
Comment thread handwritten/pubsub/src/publisher/message-queues.ts Outdated
Comment thread handwritten/pubsub/src/publisher/message-queues.ts Outdated
Comment thread handwritten/pubsub/src/publisher/message-queues.ts Outdated
Comment thread handwritten/pubsub/src/publisher/message-queues.ts Outdated
@feywind
feywind force-pushed the pubsub/hedging-queue-integration-and-tests branch 2 times, most recently from e89fc18 to 02582a7 Compare October 15, 2026 19:01
@feywind feywind changed the title feat(pubsub): integrate publish hedging into MessageQueue and add tests feat(pubsub): integrate publish hedging into MessageQueue Oct 15, 2026
@feywind
feywind marked this pull request as ready for review October 15, 2026 21:33
@feywind
feywind requested a review from a team as a code owner October 15, 2026 21:33
@github-actions
github-actions Bot requested a review from danieljbruce October 15, 2026 21:35
@feywind
feywind force-pushed the pubsub/hedging-queue-integration-and-tests branch from 02582a7 to 1231c0c Compare October 19, 2026 21:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: pubsub Issues related to the Pub/Sub API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant