From c72655bc38e4de4c370f3131748708826e0808c9 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 12 Nov 2026 05:18:27 -0300 Subject: [PATCH] runbooks: a workspace that ran out of memory The owner notice spin#1159 adds (OUT_OF_MEMORY, for spin#1116): its three causes - the guest's kernel killed processes, short at its ceiling, its host killed the machine - and what each asks. Co-Authored-By: Claude Opus 5.5 --- runbooks/workspace-out-of-memory.md | 66 +++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 runbooks/workspace-out-of-memory.md diff --git a/runbooks/workspace-out-of-memory.md b/runbooks/workspace-out-of-memory.md new file mode 100644 index 0000000..da49f97 --- /dev/null +++ b/runbooks/workspace-out-of-memory.md @@ -0,0 +1,66 @@ +--- +id: workspace-out-of-memory +title: A workspace ran out of memory +applies_to: + alerts: [] + jobs: [] + events: [OUT_OF_MEMORY] + dashboards: [fleet] +severity: info +owner: operator +last_reviewed: 2026-11-12 +--- + +## Symptoms + +The owner is emailed (on by default) and sees on [Settings → Notifications](/settings/notifications) +one of: + +> Workspace `` ran out of memory + +> Workspace `` is out of memory at its ceiling + +> Workspace `` ran out of memory and its host stopped it + +with what it held, its own memory and how far it may be lent, and the command +`spin workspace grant --memory `. `spin workspace history ` has the +event, `OUT_OF_MEMORY`, with the processes the guest's kernel killed. One is recorded for each of +the three causes at most every ten minutes, however often it runs out in between. + +## Impact + +- **Its kernel killed processes** (cause `guest`): a build, a language server or an editor's + server ended. The workspace runs on; what was killed is gone until it is started again. +- **Short at its ceiling** (cause `ceiling`): its host has lent it all it may have for a minute + and it is still short. A kill is likely next. +- **Its host killed the machine** (cause `host`): the whole workspace stopped, as after a power + cut. Its disk is kept. + +Nothing is resized on its own: the event is the owner's to act on. + +## Diagnose + +1. Which cause? `spin workspace history ` shows it, with what was killed. +2. What did it use? The workspace's page in the dashboard shows its last hour of memory against + what it held, while its host still has the hour. +3. Many workspaces on one runner killed by their host at once is the runner's memory, not the + workspaces': *Containment and DNS* on [Admin → Metrics](/admin/metrics?dashboard=fleet) counts + the host's kills. + +## Fix + +- **One workspace that needs more**: its owner runs the command in the notice, or picks a larger + preset where the notice says it is granted all its preset allows. +- **A runner killing machines**: follow [host-not-reporting](host-not-reporting.md) for the runner, + and see what it holds on [Admin → Runners](/admin/runners). + +## Verify + +After a grant, `spin workspace get ` shows the memory it holds, and no new +`OUT_OF_MEMORY` appears in its history under the same work. + +## Escalate + +If a workspace with memory to spare is reported out of memory, open an issue at +https://github.com/spin-stack/spin/issues with the workspace id, its history, the output of +`spin workspace status ` and `spin admin versions`.