Skip to content

chore(spanner): core implementation for a dynamic channel pool - #6522

Open
olavloite wants to merge 3 commits into
googleapis:mainfrom
olavloite:spanner-dynamic-channel-pool
Open

chore(spanner): core implementation for a dynamic channel pool#6522
olavloite wants to merge 3 commits into
googleapis:mainfrom
olavloite:spanner-dynamic-channel-pool

Conversation

@olavloite

Copy link
Copy Markdown
Contributor

Introduce modular channel pooling supporting static fixed-size pools and dynamic load-based scaling with Power of Two Least Busy (P2C) selection, parallel channel priming, and transaction affinity routing.

Introduce modular channel pooling supporting static fixed-size pools
and dynamic load-based scaling with Power of Two Least Busy (P2C)
selection, parallel channel priming, and transaction affinity routing.
@olavloite
olavloite requested review from a team as code owners August 24, 2026 13:28
@product-auto-label product-auto-label Bot added the api: spanner Issues related to the Spanner API. label Aug 24, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive dynamic channel pooling implementation for the Spanner client. It includes transaction channel affinity management, static and dynamic pool configurations, channel entry lifecycle tracking with RAII guards, a core channel pool engine utilizing Power of Two Least Busy selection, and background scaling workers for autonomous scale-up, scale-down, and priming. I have no feedback to provide as there are no review comments to assess.

@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.58287% with 70 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.54%. Comparing base (76a72fc) to head (f092480).

Files with missing lines Patch % Lines
src/spanner/src/channel_pool/scaler.rs 97.75% 38 Missing ⚠️
src/spanner/src/channel_pool/pool.rs 95.15% 24 Missing ⚠️
src/spanner/src/channel_pool/entry.rs 97.66% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6522      +/-   ##
==========================================
+ Coverage   96.51%   96.54%   +0.03%     
==========================================
  Files         304      308       +4     
  Lines       87943    90839    +2896     
==========================================
+ Hits        84878    87702    +2824     
- Misses       3065     3137      +72     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from f715d57 to 47e9ad4 Compare August 25, 2026 14:07
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic channel pooling implementation for the Spanner client, adding modules for affinity routing, pool configuration, channel entry lifecycles, and background scaling/priming. The review feedback suggests several robustness and performance improvements: using sort_unstable_by_key to avoid temporary allocations during sorting, replacing direct as u64 casts with u64::try_from to prevent truncation of large durations, and using is_finite() instead of is_nan() to properly validate float configuration parameters.

Comment thread src/spanner/src/channel_pool/scaler.rs Outdated
Comment thread src/spanner/src/channel_pool/entry.rs Outdated
Comment thread src/spanner/src/channel_pool/config.rs Outdated
@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from 47e9ad4 to 7a91a08 Compare August 25, 2026 15:57
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic channel pooling mechanism for Spanner, implementing capacity management, load-balanced channel selection (Power of Two Least Busy), error penalization, transaction affinity pinning, and background scaling. Feedback on the implementation highlights two critical issues in the background scaling loops: a potential CPU thrashing risk in the scale-up worker loop during its cooldown period, and a background task leak in the scale-down monitor loop that can persist for up to three minutes after the client is dropped. Suggestions are provided to sleep during the remaining cooldown and to use a shutdown notification signal to terminate the monitor loop immediately.

Comment thread src/spanner/src/channel_pool/scaler.rs Outdated
Comment thread src/spanner/src/channel_pool/scaler.rs
@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from 7a91a08 to 1db03f0 Compare August 26, 2026 06:53
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic channel pooling engine for the Spanner client, implementing capacity management, Power of Two Least Busy (P2C) channel selection, transaction affinity routing, and background scaling/priming workers. The review feedback highlights three critical concurrency and logic issues: a bug in the scale-up worker where cooldown sleep discards the triggering notification, a dangerous unstable sort on active entries using concurrent atomic loads that violates sorting invariants, and a check-then-act race condition when updating error penalty state across separate atomic variables.

Comment thread src/spanner/src/channel_pool/scaler.rs
Comment thread src/spanner/src/channel_pool/scaler.rs Outdated
Comment thread src/spanner/src/channel_pool/entry.rs Outdated
@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from 1db03f0 to c702a10 Compare August 26, 2026 07:50
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic channel pooling system for the Spanner client, featuring configurable static and dynamic pool strategies, atomic channel entry management, and background scaling workers. The reviewer provided a valuable suggestion to use Weak references in the scale-up worker loop to avoid holding strong Arc references across await points, which ensures cleaner resource cleanup during shutdown.

Comment thread src/spanner/src/channel_pool/scaler.rs Outdated
@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from c702a10 to 2beae05 Compare August 26, 2026 11:33
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic channel pooling implementation for the Spanner client, including transaction affinity routing, Power of Two Least Busy selection, and background scaling/priming. The feedback suggests wrapping physical channel creation in a timeout to prevent indefinite blocking during network hangs, and updating the scale-up cooldown timestamp when returning early to avoid busy-wakeup storms under sustained load.

Comment thread src/spanner/src/channel_pool/scaler.rs Outdated
Comment thread src/spanner/src/channel_pool/scaler.rs
@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from 2beae05 to 89d5832 Compare August 26, 2026 13:16
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic channel pooling mechanism for the Spanner client, including support for static and dynamic pool configurations, Power of Two Least Busy (P2C) load balancing, and transaction affinity management. The implementation includes background scaling workers for dynamic adjustments and RAII guards for resource management. Feedback suggests refining the cooldown throttling logic to distinguish between capacity-based and wakeup-based delays, and considering an active eviction mechanism for permanently unhealthy channels.

Comment thread src/spanner/src/channel_pool/scaler.rs
Comment thread src/spanner/src/channel_pool/scaler.rs
@olavloite
olavloite force-pushed the spanner-dynamic-channel-pool branch from 89d5832 to dbcfca9 Compare August 26, 2026 14:16
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new channel_pool module for the Spanner client, implementing support for both static and dynamic channel pooling. The changes include logic for transaction affinity, channel entry management, P2C (Power of Two Least Busy) load balancing, and background scaling workers to handle dynamic capacity management. I have no feedback to provide on these changes.

@olavloite
olavloite requested a review from rahul2393 August 26, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the Spanner API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant