Skip to content

fix: count connect timeouts, not statement timeouts, in connection_timeouts_total - #63

Merged
AlexeyShalaev merged 1 commit into
masterfrom
fix/55-timeout-counter
Oct 16, 2026
Merged

AlexeyShalaev merged 1 commit into
masterfrom
fix/55-timeout-counter

Conversation

@AlexeyShalaev

Copy link
Copy Markdown
Member

Closes #55.

Problem

postgres_db_connection_timeouts_total was counting the wrong things:

  • Counted, though it isn't a connection timeout: asyncpg's command_timeout firing on a running statement. on_error marked any TimeoutError that reached handle_error as a timeout.
  • Counted nowhere: a connect timeout while the pool opens a connection. asyncpg raises a builtin TimeoutError from connect(), which never reaches handle_error, and InstrumentedPool.connect only caught sqlalchemy.exc.TimeoutError.

While reproducing it I checked the other connect failures. With asyncpg, none of ConnectionRefusedError, InvalidPasswordError or the connect TimeoutError reaches handle_error, because the dialect doesn't translate connect errors. So a database that refused connections moved no series at all, not even the checkout wait.

Change

  • on_error records record_error(error_type=...) and no longer sets is_timeout. A statement timeout stays in connection_errors_total{error_type="TimeoutError"}, where it already was.
  • InstrumentedPool.connect records the wait however the checkout ends. A builtin TimeoutError (the connect timeout) counts as timed_out=True, like the pool timeout. Any other exception records the wait and, when the metrics object implements ErrorRecorder (now runtime_checkable), record_error(error_type=type(exc).__name__). An exception that is the driver's own DBAPI Error is skipped there, because the engine reports those through handle_error itself (psycopg does, asyncpg doesn't), so nothing is counted twice.
  • ErrorRecorder.record_error keeps its is_timeout parameter, and PostgresMetrics still honours it, so custom callers don't break. The library just no longer sets it.
  • HELP text, docstrings, the metrics table in the advanced guide and docs/agents.md describe the counter as "timeouts acquiring a connection: pool checkout timeouts and connect timeouts". The "Fixed since" table on the agents page gets a 0.4.0 row.

Considered and rejected

  • A separate postgres_db_statement_timeouts_total. Statement timeouts are already one label away in connection_errors_total, and a new series is a naming decision of its own. It's easy to add later if someone wants it.

Who notices

Anyone alerting on postgres_db_connection_timeouts_total: it stops moving on slow statements and starts moving on connect timeouts, which matches its HELP text and the "Checkouts that gave up waiting" panel. connection_errors_total gains connect failures (ConnectionRefusedError, InvalidPasswordError, …), and connection_checkout_wait_seconds gains the waits of failed checkouts.

Evidence

New tests against the old manager.py/metrics/postgres.py/protocols (swapped in from origin/master): 9 failed. Three of them are integration tests that read the registry by name: a command_timeout on PostgreSQL 17, a connect timeout against a local server that accepts TCP and never answers (deterministic, unlike an unroutable address on a CI runner), and a refused port.

The reporter's script after the fix:

1. asyncpg command_timeout=0.1 hits a running statement (the connection was established fine)
   raised: builtins.TimeoutError
   connection_timeouts_total            = 0.0
   connection_errors_total (all labels) = 1.0
   connection_checkout_wait_seconds_count = 1.0
2. asyncpg connect timeout=0.5 while the pool opens a connection (unroutable address)
   raised: builtins.TimeoutError
   connection_timeouts_total            = 1.0
   connection_errors_total (all labels) = 0
   connection_checkout_wait_seconds_count = 1.0
HELP connection_timeouts_total: Number of times acquiring a connection timed out: pool checkout timeouts and connect timeouts

Gate: make check clean; make test 632 passed, coverage 93.80%.

Touches session/manager.py and docs/agents.md like #62, so whichever merges second needs a rebase. I'll do it.

@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@AlexeyShalaev
AlexeyShalaev force-pushed the fix/55-timeout-counter branch from 92a386b to 35e7738 Compare October 16, 2026 15:07
@AlexeyShalaev
AlexeyShalaev merged commit 961e0e8 into master Oct 16, 2026
8 checks passed
@AlexeyShalaev
AlexeyShalaev deleted the fix/55-timeout-counter branch October 16, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

connection_timeouts_total counts asyncpg statement timeouts and misses connect timeouts

1 participant