Skip to content

fix(pg-pool): honor connectionTimeoutMillis and prevent connection leaks/hangs - #3721

Open
DijieDeng wants to merge 1 commit into
brianc:masterfrom
DijieDeng:fix/pg-pool-connection-timeout-hang-3197
Open

fix(pg-pool): honor connectionTimeoutMillis and prevent connection leaks/hangs#3721
DijieDeng wants to merge 1 commit into
brianc:masterfrom
DijieDeng:fix/pg-pool-connection-timeout-hang-3197

Conversation

@DijieDeng

Copy link
Copy Markdown

Problem

pool.connect() can hang indefinitely under high concurrency and leak connections when connectionTimeoutMillis is set. This has been reported across several issues:

Root cause

In packages/pg-pool/index.js, newClient() sets a connectionTimeoutMillis timer, but:

  1. The timer only tore down the connection when client.connection existed or !client.isConnected(). When neither condition held — for instance while a DNS lookup was still pending — the timer fired but did nothing, so the checkout never resolved. This is the indefinite hang (ConnectionTimeoutMillis is not respected and pool.connect hangs indefinitely  #3197).

  2. The client.connect() success path never checked timeoutHit. A connection that finished establishing after the deadline was handed to the caller anyway and kept in the pool. The connection leaked on the Postgres side (visible in pg_stat_activity as idle with query_start IS NULL) and, with enough timeouts, exhausted the pool ([pg-pool] Connection timeouts cause connection leaks #3543, client in pool timed out, causing pool to fail for all future queries #2243).

  3. client.connection.stream.destroy() destroyed the socket without cleanly closing the connection, leaving zombie connections on the Postgres side.

Fix

  • The timeout callback now always rejects the checkout at the deadline (except for the existing native-client exemption where !client.connection && client.isConnected()), removes the client from _clients, pulses the queue, and ends the connection cleanly.
  • The success path now checks timeoutHit: a client that connects after the timeout is discarded (client.end()) rather than handed back.
  • Uses client.connection.end() instead of client.connection.stream.destroy() for clean teardown.

Tests

Added does not retain a connection that establishes after the timeout (#3543) to packages/pg-pool/test/connection-timeout.js. It forces a connection to complete after the timeout has fired and asserts the checkout fails with a timeout error and that totalCount/idleCount return to 0 (no leaked connection). Existing timeout tests continue to pass.

This approach follows the community fix proposed in #3711 (by @malbolged), refined here with a focused description and the issue references consolidated.

Refs #3197 #3543 #2338 #2243

…aks/hangs

The connection timeout in newClient() had two flaws that caused
pool.connect() to hang indefinitely and leak connections under high
concurrency:

1. The timeout callback only tore down the connection when
   client.connection existed OR !client.isConnected(). When neither
   condition held (e.g. during a slow DNS lookup), the timer fired but
   did nothing, so pool.connect() hung forever (brianc#3197).

2. The client.connect() success path never checked timeoutHit, so a
   connection that finished establishing after the deadline was handed
   back to the caller anyway and kept in the pool, leaking a connection
   on the Postgres side and eventually exhausting the pool
   (brianc#3543).

3. client.connection.stream.destroy() left zombie connections on the
   Postgres side; connection.end() closes them cleanly.

The timeout now always rejects the checkout at the deadline (except for
the native-client exemption), removes the client from the pool, pulses
the queue, and ends the connection cleanly. The success path discards
any client that connects after the timeout.

Refs: brianc#3197, brianc#3543, brianc#2338, brianc#2243
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant