Skip to content

fix(gateway): keep idle connections open for an hour so pooled clients aren't reset - #802

Merged
earakely-scale merged 2 commits into
mainfrom
edgararakelyan/gateway-idle-keep-alive
Nov 19, 2026
Merged

earakely-scale merged 2 commits into
mainfrom
edgararakelyan/gateway-idle-keep-alive

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

The gateway ran uvicorn with its default keep-alive timeout, so it closed any connection idle for 5 seconds. Agents pool their connections: Claude Code's fetch, for one, keeps them longer than 5 s. When an agent paused about 5 seconds between two tool calls, it sent the next call on a pooled connection just as the gateway closed it. That call was reset before the gateway read it:

ECONNRESET: The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()

A write was lost the same way: the reset call never ran, so nothing on the server side knew to report it.

The gateway now keeps an idle connection open for an hour (IDLE_CONNECTION_TIMEOUT_S = 3600). That is longer than any pause between an agent's tool calls, so the gateway no longer closes a connection a client may reuse. Clients still close their own idle connections whenever they like.

Evidence

  • Trajectories:
    • Where the resets fall: across 209 Claude Code solver trajectories from early October to now, every one of the 12 resets came 4.8–5.2 s after the previous tool result. No reset came after any other pause.
    • How often: 1–2 calls in 1,000. The rate was steady across Claude Code versions 2.1.177, 2.1.280 and 2.1.322 and across environment versions, so this isn't a recent regression.
    • Not tool latency: each reset call failed 24–109 ms after it was sent.
  • Live, through the same HTTPS tunnel agents use:
    • Current gateway: it closed an idle connection at 5.00 s. A second request on the same connection after 4.90–6.00 s idle was refused before a response in 39 of 42 tries; after 4.50 s it was served in 6 of 6.
    • This branch: a second request after 4.90, 5.00, 5.10, 6, 10 or 30 s idle was served in all 36 tries. Idle connections stayed open for the full 40 minutes watched, while the environment was kept busy by other traffic, so the tunnel adds no shorter limit.

Changes

  • constants.py: IDLE_CONNECTION_TIMEOUT_S = 3600, with why.
  • gateway.py:
    • run() builds its server from a new _server_config(), which passes timeout_keep_alive=IDLE_CONNECTION_TIMEOUT_S;
    • nothing else about how the server is built or bound changes.

Not changed: the gateway's own connections to its child MCP servers. They expire idle connections at httpx's 5 s against the children's uvicorn 5 s, so the same race exists in principle. On the in-VM network its window is a fraction of a millisecond, and no trajectory showed it.

How it was tested

  • New test tst/unit/env/gateway/idle_connection_socket_test.py. It runs the gateway's own server config on a loopback socket, with the timeout patched to 2 s:

    • a connection reused after 0 s and 1 s idle is served;
    • the server then closes it between 1.5 s and 3.5 s.

    A second test pins the real value at 600 s or more and checks it reaches the uvicorn config. With timeout_keep_alive removed, both tests fail: the server closes at uvicorn's 5 s instead.

  • Suite: pytest tst/unit/env/gateway -n 8 passed, 1,172 tests. ruff check is clean.

  • Live: the measurements above came from two deployments of the same environment. One ran the current default gateway, the other a gateway image built from this branch.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

The gateway keeps idle HTTP connections open for one hour to reduce resets when agents reuse pooled connections.

  • The gateway keeps idle client connections open for an hour.

Reviews (2) · Last reviewed commit: "test(gateway): name the idle-connection ..." · Reviewed by Greptile

…s aren't reset

uvicorn's default keep-alive closed a connection after 5 s idle. An agent that paused about 5 s between tool calls sent the next one on its pooled connection as the gateway closed it, and the call was reset unread (ECONNRESET from Claude Code's fetch). IDLE_CONNECTION_TIMEOUT_S = 3600 outlasts any pause between an agent's calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 03:59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit ad0ef2a into main Nov 19, 2026
35 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/gateway-idle-keep-alive branch November 19, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant