Repository navigation
fix: count connect timeouts, not statement timeouts, in connection_timeouts_total - #63
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced Oct 16, 2026
AlexeyShalaev
force-pushed
the
fix/55-timeout-counter
branch
from
October 16, 2026 15:07
92a386b to
35e7738
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #55.
Problem
postgres_db_connection_timeouts_totalwas counting the wrong things:command_timeoutfiring on a running statement.on_errormarked anyTimeoutErrorthat reachedhandle_erroras a timeout.TimeoutErrorfromconnect(), which never reacheshandle_error, andInstrumentedPool.connectonly caughtsqlalchemy.exc.TimeoutError.While reproducing it I checked the other connect failures. With asyncpg, none of
ConnectionRefusedError,InvalidPasswordErroror the connectTimeoutErrorreacheshandle_error, because the dialect doesn't translate connect errors. So a database that refused connections moved no series at all, not even the checkout wait.Change
on_errorrecordsrecord_error(error_type=...)and no longer setsis_timeout. A statement timeout stays inconnection_errors_total{error_type="TimeoutError"}, where it already was.InstrumentedPool.connectrecords the wait however the checkout ends. A builtinTimeoutError(the connect timeout) counts astimed_out=True, like the pool timeout. Any other exception records the wait and, when the metrics object implementsErrorRecorder(nowruntime_checkable),record_error(error_type=type(exc).__name__). An exception that is the driver's own DBAPIErroris skipped there, because the engine reports those throughhandle_erroritself (psycopg does, asyncpg doesn't), so nothing is counted twice.ErrorRecorder.record_errorkeeps itsis_timeoutparameter, andPostgresMetricsstill honours it, so custom callers don't break. The library just no longer sets it.docs/agents.mddescribe the counter as "timeouts acquiring a connection: pool checkout timeouts and connect timeouts". The "Fixed since" table on the agents page gets a 0.4.0 row.Considered and rejected
postgres_db_statement_timeouts_total. Statement timeouts are already one label away inconnection_errors_total, and a new series is a naming decision of its own. It's easy to add later if someone wants it.Who notices
Anyone alerting on
postgres_db_connection_timeouts_total: it stops moving on slow statements and starts moving on connect timeouts, which matches its HELP text and the "Checkouts that gave up waiting" panel.connection_errors_totalgains connect failures (ConnectionRefusedError,InvalidPasswordError, …), andconnection_checkout_wait_secondsgains the waits of failed checkouts.Evidence
New tests against the old
manager.py/metrics/postgres.py/protocols (swapped in fromorigin/master): 9 failed. Three of them are integration tests that read the registry by name: acommand_timeouton PostgreSQL 17, a connect timeout against a local server that accepts TCP and never answers (deterministic, unlike an unroutable address on a CI runner), and a refused port.The reporter's script after the fix:
Gate:
make checkclean;make test632 passed, coverage 93.80%.Touches
session/manager.pyanddocs/agents.mdlike #62, so whichever merges second needs a rebase. I'll do it.