diff --git a/runbooks/workspace-out-of-memory.md b/runbooks/workspace-out-of-memory.md new file mode 100644 index 0000000..da49f97 --- /dev/null +++ b/runbooks/workspace-out-of-memory.md @@ -0,0 +1,66 @@ +--- +id: workspace-out-of-memory +title: A workspace ran out of memory +applies_to: + alerts: [] + jobs: [] + events: [OUT_OF_MEMORY] + dashboards: [fleet] +severity: info +owner: operator +last_reviewed: 2026-11-12 +--- + +## Symptoms + +The owner is emailed (on by default) and sees on [Settings → Notifications](/settings/notifications) +one of: + +> Workspace `` ran out of memory + +> Workspace `` is out of memory at its ceiling + +> Workspace `` ran out of memory and its host stopped it + +with what it held, its own memory and how far it may be lent, and the command +`spin workspace grant --memory `. `spin workspace history ` has the +event, `OUT_OF_MEMORY`, with the processes the guest's kernel killed. One is recorded for each of +the three causes at most every ten minutes, however often it runs out in between. + +## Impact + +- **Its kernel killed processes** (cause `guest`): a build, a language server or an editor's + server ended. The workspace runs on; what was killed is gone until it is started again. +- **Short at its ceiling** (cause `ceiling`): its host has lent it all it may have for a minute + and it is still short. A kill is likely next. +- **Its host killed the machine** (cause `host`): the whole workspace stopped, as after a power + cut. Its disk is kept. + +Nothing is resized on its own: the event is the owner's to act on. + +## Diagnose + +1. Which cause? `spin workspace history ` shows it, with what was killed. +2. What did it use? The workspace's page in the dashboard shows its last hour of memory against + what it held, while its host still has the hour. +3. Many workspaces on one runner killed by their host at once is the runner's memory, not the + workspaces': *Containment and DNS* on [Admin → Metrics](/admin/metrics?dashboard=fleet) counts + the host's kills. + +## Fix + +- **One workspace that needs more**: its owner runs the command in the notice, or picks a larger + preset where the notice says it is granted all its preset allows. +- **A runner killing machines**: follow [host-not-reporting](host-not-reporting.md) for the runner, + and see what it holds on [Admin → Runners](/admin/runners). + +## Verify + +After a grant, `spin workspace get ` shows the memory it holds, and no new +`OUT_OF_MEMORY` appears in its history under the same work. + +## Escalate + +If a workspace with memory to spare is reported out of memory, open an issue at +https://github.com/spin-stack/spin/issues with the workspace id, its history, the output of +`spin workspace status ` and `spin admin versions`.