Correlate Toolbox logs with SSH sessions - #359
Open
fioan89 wants to merge 7 commits into
Open
Conversation
This drops support for Toolbox versions older than 3.7.2 but instead provides new APIs that can give better control and insight to the Coder plugin.
Keep one generated session ID for each workspace and agent pair so SSH reconnects share the same correlation value. Remove the entry only when Toolbox disposes the environment, allowing a later environment to begin a new session.
Add one logger wrapper that preserves existing logging calls and lets callers attach a connection session ID when a message belongs to a workspace session. Keep the existing log-and-show behavior in the same wrapper so messages are logged before they are displayed to the user.
Expose the Coder logger from the shared plugin context and use it for existing log-and-show calls. Keep popup creation and error handling inside the logger so callers use one place for logging and user notifications.
Connection diagnostics require one 16-byte, 32-character lowercase hexadecimal session ID for each logical Toolbox SSH connection. Logs that affect the connection and the coder ssh subprocess must carry the same ID. In Toolbox, the session belongs to the workspace and agent SSH transport: it may start without an IDE, outlive an IDE, and serve multiple JetBrains IDE launches. Unlike VS Code's one-to-one IDE and connection lifecycle, Toolbox has a shared polling loop that tracks additions, removals, and status changes across multiple workspaces. A single session ID cannot represent every active connection in that loop. We agreed on a hybrid direction: monitor status per workspace through WebSockets so each change can use the correct session (will be done in a separate PR), retain shared polling for additions and removals with logs multiplexed for every affected session. It remains an open question how shared polling requests should carry multiple IDs through baggage or another header. A future mode where the plugin runs inside the IDE will be closer to VS Code, so both lifecycle models will eventually need support. An additional problem is that the current Toolbox callbacks also do not fully describe the ssh connection state. `beforeConnection` reports an attempt to establish the ssh connection but not its reason or its outcome, while `afterDisconnect` only tells us whether the user explicitly disconnected. We therefore create an ID on the first connection attempt, retain and reuse it across automatic retries and every non-manual disconnect, remove it after a manual disconnect or environment disposal, and create a new ID when the user connects again. Since Toolbox cannot distinguish a temporary transport loss from a terminal non-manual failure, both remain part of the existing session. In terms of implementation, this PR implements a process-local workspace and agent registry, session-aware logging, propagation through CODER_TRACE_SESSION_ID, dynamic session lookup in connection and IDE paths, and multi-session fan-out for provider and SSH configuration work. It also records status and disconnect context and covers creation, reuse, manual removal, automatic retries, and environment disposal. In parallel with this PR work we asked JetBrains to add callbacks for successful connections and failed attempts, together with reasons that distinguish explicit connect, disconnect, and reconnect actions from automatic startup and retry, plugin-requested disconnects, transport loss, remote exit, and environment removal. Those signals would let the plugin track connection state and correlation IDs directly instead of inferring lifecycle from the limited callbacks and workspace status. - resolves https://linear.app/codercom/issue/DEVEX-667
Remove unused session-aware overloads and default arguments, then update the focused tests to match the smaller logging API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connection diagnostics require one 16-byte, 32-character lowercase hexadecimal session ID for each logical Toolbox SSH connection. Logs that affect the connection and the coder ssh subprocess must carry the same ID. In Toolbox, the session belongs to the workspace and agent SSH transport: it may start without an IDE, outlive an IDE, and serve multiple JetBrains IDE launches.
Unlike VS Code's one-to-one IDE and connection lifecycle, Toolbox has a shared polling loop that tracks additions, removals, and status changes across multiple workspaces. A single session ID cannot represent every active connection in that loop. We agreed on a hybrid direction: monitor status per workspace through WebSockets so each change can use the correct session (will be done in a separate PR), retain shared polling for additions and removals with logs multiplexed for every affected session. It remains an open question how shared polling requests should carry multiple IDs through baggage or another header. A future mode where the plugin runs inside the IDE will be closer to VS Code, so both lifecycle models will eventually need support.
An additional problem is that the current Toolbox callbacks also do not fully describe the ssh connection state.
beforeConnectionreports an attempt to establish the ssh connection but not its reason or its outcome, whileafterDisconnectonly tells us whether the user explicitly disconnected. We therefore create an ID on the first connection attempt, retain and reuse it across automatic retries and every non-manual disconnect, remove it after a manual disconnect or environment disposal, and create a new ID when the user connects again. Since Toolbox cannot distinguish a temporary transport loss from a terminal non-manual failure, both remain part of the existing session.In terms of implementation, this PR implements a process-local workspace and agent registry, session-aware logging, propagation through CODER_TRACE_SESSION_ID, dynamic session lookup in connection and IDE paths, and multi-session fan-out for provider and SSH configuration work. It also records status and disconnect context and covers creation, reuse, manual removal, automatic retries, and environment disposal.
In parallel with this PR work we asked JetBrains to add callbacks for successful connections and failed attempts, together with reasons that distinguish explicit connect, disconnect, and reconnect actions from automatic startup and retry, plugin-requested disconnects, transport loss, remote exit, and environment removal. Those signals would let the plugin track connection state and correlation IDs directly instead of inferring lifecycle from the limited callbacks and workspace status.