fix(pg-pool): honor connectionTimeoutMillis and prevent connection leaks/hangs - #3721
Open
DijieDeng wants to merge 1 commit into
Open
fix(pg-pool): honor connectionTimeoutMillis and prevent connection leaks/hangs#3721DijieDeng wants to merge 1 commit into
DijieDeng wants to merge 1 commit into
Conversation
…aks/hangs The connection timeout in newClient() had two flaws that caused pool.connect() to hang indefinitely and leak connections under high concurrency: 1. The timeout callback only tore down the connection when client.connection existed OR !client.isConnected(). When neither condition held (e.g. during a slow DNS lookup), the timer fired but did nothing, so pool.connect() hung forever (brianc#3197). 2. The client.connect() success path never checked timeoutHit, so a connection that finished establishing after the deadline was handed back to the caller anyway and kept in the pool, leaking a connection on the Postgres side and eventually exhausting the pool (brianc#3543). 3. client.connection.stream.destroy() left zombie connections on the Postgres side; connection.end() closes them cleanly. The timeout now always rejects the checkout at the deadline (except for the native-client exemption), removes the client from the pool, pulses the queue, and ends the connection cleanly. The success path discards any client that connects after the timeout. Refs: brianc#3197, brianc#3543, brianc#2338, brianc#2243
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
pool.connect()can hang indefinitely under high concurrency and leak connections whenconnectionTimeoutMillisis set. This has been reported across several issues:connectionTimeoutMillisis not respected;pool.connect()hangs indefinitely (e.g. slow/misconfigured DNS, GKE DNS stalls)pool.connect()waits foreverRoot cause
In
packages/pg-pool/index.js,newClient()sets aconnectionTimeoutMillistimer, but:The timer only tore down the connection when
client.connectionexisted or!client.isConnected(). When neither condition held — for instance while a DNS lookup was still pending — the timer fired but did nothing, so the checkout never resolved. This is the indefinite hang (ConnectionTimeoutMillis is not respected and pool.connect hangs indefinitely #3197).The
client.connect()success path never checkedtimeoutHit. A connection that finished establishing after the deadline was handed to the caller anyway and kept in the pool. The connection leaked on the Postgres side (visible inpg_stat_activityas idle withquery_start IS NULL) and, with enough timeouts, exhausted the pool ([pg-pool] Connection timeouts cause connection leaks #3543, client in pool timed out, causing pool to fail for all future queries #2243).client.connection.stream.destroy()destroyed the socket without cleanly closing the connection, leaving zombie connections on the Postgres side.Fix
!client.connection && client.isConnected()), removes the client from_clients, pulses the queue, and ends the connection cleanly.timeoutHit: a client that connects after the timeout is discarded (client.end()) rather than handed back.client.connection.end()instead ofclient.connection.stream.destroy()for clean teardown.Tests
Added
does not retain a connection that establishes after the timeout (#3543)topackages/pg-pool/test/connection-timeout.js. It forces a connection to complete after the timeout has fired and asserts the checkout fails with a timeout error and thattotalCount/idleCountreturn to0(no leaked connection). Existing timeout tests continue to pass.This approach follows the community fix proposed in #3711 (by @malbolged), refined here with a focused description and the issue references consolidated.
Refs #3197 #3543 #2338 #2243