🤖 perf: xum server takes 13 min to bind its port on large deployments; TaskService.initialize runs O(workspaces) sequential recovery before listen()

[ibetitsmike]

Summary

On a large deployment (1,535 workspaces in config.json, 343 in taskStatus: "reported", 18 GB sessions dir), xum server takes 13 minutes from process start to binding its port. src/cli/server.ts awaits serviceContainer.initialize() (L135) before serverService.startServer() (L155), so the whole task-recovery pass sits on the critical path to accepting connections. During that window every client gets connection refused and Coder marks the app unhealthy; the user experience is "Mux cannot start".

Investigated live on dev.coder.com workspace cmux, @coder/xum 0.28.3-next.4.g0b52386f1 (= 0b52386f1 on main). Backend counterpart to #3960 (same deployment class, frontend side).

Measured timeline

Event	Time
node /tmp/mux/mux server --port 4000 starts	08:02:40
serviceContainer.initialize() returns	08:15:50
Listener bound (server.lock startedAt)	08:15:50.432Z

From ~/.xum/logs/mux.log (log.info, so invisible at the CLI default error level; only the file sink had it):

[startup] ServiceContainer.initialize completed
  totalMs: 786819
    workspaceService.initialize:  10529
    taskService.initialize:      775957   <- 98.6%

[startup] TaskService.initialize completed
  completedReportTaskCount:      343
  terminalAttentionDrainMs:      305839   (5m06s) -> pendingTerminalAttentionOwnerWorkspaceCount: 0
  patchGenerationRecoveryMs:     205078   (3m25s)
  cleanupReportedTasksMs:        131159   (2m11s)
  maybeStartQueuedTasksMs:          226

State on the box

• config.json: 2.47 MB, 1,535 workspaces (1,119 archived), taskStatus counts {reported: 343, interrupted: 172, running: 2}
• ~/.xum/sessions: 18 GB, 1,536 dirs, 4.9 GB of chat.jsonl
• sessions/*/terminal-attention/*.json: 1,795 files across 199 dirs, 0 pending (1,647 delivered, 148 superseded)

Where the time goes (all sequential, all before listen())

1. terminalAttentionDrainMs (306 s): TerminalAttentionStore.listPendingOwnerWorkspaceIds() (src/node/services/terminalAttentionStore.ts:229) readdirs the sessions dir and then await this.listPending(entry.name) sequentially for every one of the 1,535 dirs, reading all 1,795 JSON files, to discover that nothing is pending. A shell walk of the same tree took 2.75 s; in-process it took 306 s because each await yielded to a saturated event loop.

2. patchGenerationRecoveryMs (205 s): taskService.ts:2547 loops over all 343 completed tasks. Each maybeStartGenerationUnlocked call (gitPatchArtifactService.ts:374) does this.config.loadConfigOrDefault(), which is a synchronous readFileSync + JSON.parse of the whole 2.47 MB config with no caching (src/node/config/index.ts:1249), then readAgentDefinition / resolveAgentInheritanceChain FS lookups against up to two workspace checkouts.

3. cleanupReportedTasksMs (131 s): taskService.ts:2590 loops over the same 343 tasks; canCleanupReportedTask (taskService.ts:13102) calls loadConfigOrDefault() again per task.

Measured on the box: one loadConfigOrDefault()-equivalent parse = 36.9 ms. 343 x 2 = ~25 s of pure main-thread synchronous JSON parsing before any FS work or contention. At steady state the process still re-reads config.json ~3x/s.

Amplifier

Load average was 175 to 244 on 96 cores (eslint at 13.4 GB RSS, pixel-storybook, pnpm install), and Mux resumed 2 running agent tasks during TaskService.initialize (spawning tool bash and 7 duplicate mcp-grafana + coder exp mcp server child pairs) while the recovery loops were still running. Roughly 100x slowdown per awaited FS op versus idle. The design problem stands without the load: even at idle the sequential loops are O(workspaces) + O(completed tasks) on the connect path.

Red herrings in /tmp/mux.log

• MaxListenersExceededWarning for InitStateManager (11 > 10) and AIService (51 > 50): one listener per workspace/stream; cosmetic on large deployments.
• Client.listPrompts() called but server does not advertise prompts capability: normal MCP client chatter per server connect.

Proposed fixes (impact order)

1. Bind the listener before recovery. Start the HTTP server first, or run TaskService/WorkspaceService recovery in the background after listen(). This alone makes "cannot connect" structurally impossible regardless of state size. The startup keepalive interval in server.ts already exists to cover the gap the other way round.
2. Stop re-reading config.json per task. Pass one config snapshot into maybeStartGeneration and canCleanupReportedTask, or memoize loadConfigOrDefault() on file mtime/size. 686 synchronous 2.47 MB parses in a loop is the bulk of the CPU.
3. Do not scan 1,535 session dirs to find pending attention. Bound concurrency in listPendingOwnerWorkspaceIds() or maintain a pending-owner index so startup does not pay O(sessions) for an empty result.
4. Emit the [startup] step durations at warn (or to stdout) when totalMs exceeds a threshold so this is diagnosable from /tmp/mux.log without turning on XUM_LOG_LEVEL=info.

Minor

The server bound [::1]:4000 only (Node resolves localhost to the first hosts entry order). curl 127.0.0.1:4000 is refused while curl '[::1]:4000' returns 200. Coder's app proxy dials localhost so it works, but IPv4-only clients would not.

---

Generated with xum • Model: anthropic:claude-fable-5-1 • Thinking: xhigh • Cost: $6.67

[linear-code]

MUX-126