Skip to content

mcp: detect dead transports, add Reconnecting state, background reconnect - #15540

Draft
Legoben wants to merge 1 commit into
bens/mcp_proactively_re-mint_managed_proxy_tokens_before_expiry_during_agent_runsfrom
bens/mcp_detect_dead_transports_add_reconnecting_state_background_reconnect
Draft

mcp: detect dead transports, add Reconnecting state, background reconnect#15540
Legoben wants to merge 1 commit into
bens/mcp_proactively_re-mint_managed_proxy_tokens_before_expiry_during_agent_runsfrom
bens/mcp_detect_dead_transports_add_reconnecting_state_background_reconnect

Conversation

@Legoben

@Legoben Legoben commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Problem

Nothing observed an MCP server dying: rmcp's cancellation token does not fire on natural death, and RunningService::waiting() consumes self. Detection was pull-only (Peer::is_transport_closed at the next tool call), so a dead server kept showing Running and advertising stale tools indefinitely. Separately, the built-in Factory MCP recorded its bearer optimistically before the spawn resolved, so a failed builtin spawn was never retried until the token value changed.

Changes

  • Push-based death signal: TransportLoggingWrapper (which wraps every transport — stdio, streamable HTTP, SSE) flips a tokio::sync::watch channel when receive() observes end-of-input or the transport is closed; exposed as TemplatableMCPServerInfo::transport_closed().
  • Background monitor: on spawn success the manager watches that signal; on death it flips the server to the new MCPServerState::Reconnecting and starts a background reconnect. Guards: superseded instances no-op (new per-uuid spawn generation counter, also applied to spawn completion callbacks to prevent stale-state clobbers), explicitly stopped servers are left alone, the circuit breaker surfaces FailedToStart instead of storming, and servers that would require interactive OAuth (authenticated transport, no cached credentials, no managed refresher) surface a re-auth error rather than popping a login flow. Background healing is deliberately single-attempt per death — tool calls remain the retry driver.
  • MCPServerState::Reconnecting: compile-time-only addition (the enum has no serde anywhere); presented as Starting in the settings card and TUI; non-terminal for the driver's startup wait. Reconnect lookup failures now surface as FailedToStart with the error message instead of leaving stale state.
  • Builtin fix: a failed spawn clears builtin_server_token, so the next auth event retries even with an unchanged token.
  • SimpleLogger::new_discarding_for_test (behind a new test-util feature) so transport tests don't need a filesystem/executor.

Flag-gated by McpSelfHeal (flag off restores lazy pull-only detection).

Tests

Transport-signal tests (end-of-input and explicit close flip the watch), monitor guards (stale generation, stopped server, breaker → FailedToStart, flag-off inert), and the reconnect-lookup-failure surfacing path. Full suites: cargo test -p mcp (42), -p warp --lib ai::mcp (86), driver (388); clippy clean; TUI feature build checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant