diff --git a/runbooks/host-wrote-foreign-volume.md b/runbooks/host-wrote-foreign-volume.md index 172ee79..65082f8 100644 --- a/runbooks/host-wrote-foreign-volume.md +++ b/runbooks/host-wrote-foreign-volume.md @@ -8,7 +8,7 @@ applies_to: dashboards: [storage] severity: critical owner: security -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -76,12 +76,23 @@ Treat the host as taken until proven otherwise. Re-key a disk that is not marked (its workspace stopped or quarantined first): ``` - spin admin volume rekey + spin admin volume rekey ``` A re-key deletes the old disk and every snapshot of it. More in [key-rotation](key-rotation.md). - -TODO: removing an autoscaling group's runner believed taken does not mark the disks it held for re-keying, and nothing lists them ([spin#1047](https://github.com/spin-stack/spin/issues/1047)) +- **The host is a runner of an autoscaling group**: remove it as taken. Its credentials are + revoked and it is cut off first, every team disk whose key it was given is marked for + re-keying, and then its machine is given back to its group: + + ``` + spin admin runner remove --taken + ``` + + It prints `Runner revoked; disks marked for re-keying (spin admin volume rekeys)`. It is + refused while a workspace's only disk is on the runner; add `--leave-workspaces` to remove it + anyway, and each such workspace runs again only once recovered onto another runner. In the + dashboard: the runner's menu, *Remove as compromised*. The marked disks are listed by + `spin admin volume rekeys`. ## Verify diff --git a/runbooks/identity-audit-failing.md b/runbooks/identity-audit-failing.md index e3a530f..c1733df 100644 --- a/runbooks/identity-audit-failing.md +++ b/runbooks/identity-audit-failing.md @@ -8,7 +8,7 @@ applies_to: dashboards: [] severity: critical owner: security -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -68,11 +68,15 @@ and every grant is recorded in the bucket where it cannot be taken back. spin admin jobs run identity-audit ``` - Each run checks only its own hour; hours missed while it failed are not checked again by the job. - Check them by hand: the `Decrypt` calls in the key service's log against the records under - `identity/granted/` for the same hours. + Each run audits every settled hour since the last one it audited, oldest first, at most 24 a + run. An hour it cannot read stops the run and is the next run's first. On + [Admin → Jobs](/admin/jobs?job=identity-audit), `hours` is the hours audited this run, `behind` + the settled hours still owed, and `unaudited` the hours the key service's log no longer keeps. - TODO: the audit does not go back over hours it missed while failing ([spin#1051](https://github.com/spin-stack/spin/issues/1051)) + If the run fails with `audit: the key service no longer keeps its log of hours that were never + audited`, those hours (the error names how many and between when) are the only ones to check + by hand: the `Decrypt` calls in whatever record of them you still have against the records + under `identity/granted/` for the same hours. ## Verify diff --git a/runbooks/key-rotation.md b/runbooks/key-rotation.md index acaecd6..e136acb 100644 --- a/runbooks/key-rotation.md +++ b/runbooks/key-rotation.md @@ -8,7 +8,7 @@ applies_to: dashboards: [] severity: info owner: security -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -54,12 +54,11 @@ workspace is stopped or quarantined. Watch it with `spin admin volume rekeys`. B stopped or quarantined workspace: ``` -spin admin volume rekey +spin admin volume rekey ``` -TODO: the CLI looks the workspace up as the caller's own before it calls the admin API, so it -reaches only an administrator's own workspaces; there is no command for another person's -([spin#1046](https://github.com/spin-stack/spin/issues/1046)). +It takes the workspace's id, not its name, and works for anybody's workspace. The id is in +`spin admin volume rekeys --json` (`workspace_id`) for a marked disk, or on the workspace's page. **Runner registration tokens**: `spin admin token revoke `. A pool's token is re-minted on the control plane's machine with `sudo spin-controlplane registration-token ...`. diff --git a/runbooks/volume-not-served.md b/runbooks/volume-not-served.md index bb61648..e8602c2 100644 --- a/runbooks/volume-not-served.md +++ b/runbooks/volume-not-served.md @@ -8,7 +8,7 @@ applies_to: dashboards: [fleet, storage] severity: critical owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -83,7 +83,16 @@ it writes from now on is lost with its host. Each failed restore reads the objec - **The host is the cause for many volumes**: stop it taking new work with `spin admin runner drain `, and look at it on a [host-shell](host-shell.md). -TODO: an administrator cannot list which workspaces have a refused volume, or recover a workspace they do not own ([spin#1045](https://github.com/spin-stack/spin/issues/1045)) +- **Which workspaces it is**: list every refused volume, with the refusal, the host, the + workspace and its owner (the listing is audited): + + ``` + spin admin volume refused + ``` + + If the owner cannot recover one, recover it for them with + `spin admin workspace recover --reason ""`, as in + [workspace-held](workspace-held.md). ## Verify diff --git a/runbooks/workspace-egress-high.md b/runbooks/workspace-egress-high.md index 124adb7..264ba4d 100644 --- a/runbooks/workspace-egress-high.md +++ b/runbooks/workspace-egress-high.md @@ -8,7 +8,7 @@ applies_to: dashboards: [fleet] severity: warning owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -34,7 +34,12 @@ send traffic somewhere it should not). spin workspace egress ``` - TODO: an administrator cannot read where a workspace they do not own sends ([spin#1045](https://github.com/spin-stack/spin/issues/1045)) + An administrator reads it for a workspace that is not theirs. It changes nothing, and the + reading is audited with the reason; if the audit entry cannot be written, it answers nothing: + + ``` + spin admin workspace egress --reason "" + ``` 3. See what it has cost this month: ``` diff --git a/runbooks/workspace-held.md b/runbooks/workspace-held.md index 61431fc..9216e1a 100644 --- a/runbooks/workspace-held.md +++ b/runbooks/workspace-held.md @@ -8,7 +8,7 @@ applies_to: dashboards: [fleet, storage] severity: critical owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -63,18 +63,26 @@ is ever moved on its own, and the owner must recover by hand. - **The host can be brought back** (a reboot, a network fault): bring it back. The workspace runs on it again with nothing lost. Nobody needs to recover anything. -- **The host is gone for good**: tell the owner. Only the owner can give up their work: the - command is theirs (`spin workspace recover --accept-data-loss`). It takes the disk - from the dead host and places it on a live one; then they start it. Afterwards retire the - host ([lost-runner](lost-runner.md)). +- **The host is gone for good**: tell the owner. The owner recovers it with + `spin workspace recover --accept-data-loss`. It takes the disk from the dead host + and places it on a live one; then they start it. Afterwards retire the host + ([lost-runner](lost-runner.md)). +- **The owner cannot recover it, and you recover it for them**: run it first without + `--accept-data-loss` to see the work that is given up; nothing changes until you pass it. + `--reason` is required: + + ``` + spin admin workspace recover --reason "" + spin admin workspace recover --reason "" --accept-data-loss + ``` + + It is audited, and written to the workspace's history with your reason, which its owner sees + on Activity. - **The installation should move such workspaces on its own** next time: set `rehome_after_seconds` and `rehome_max_unpublished_bytes` in the installation's configuration file, and apply the change on [Admin → Settings](/admin/settings). Giving up unpublished work without asking is a policy decision; make it on purpose. -TODO: an administrator cannot recover a workspace for its owner; `RecoverWorkspace` answers -only the owner ([spin#1045](https://github.com/spin-stack/spin/issues/1045)). - ## Verify `spin workspace get ` shows a live host, and the workspace starts. A recovery is in diff --git a/runbooks/workspace-stop-backup-failed.md b/runbooks/workspace-stop-backup-failed.md index 5d9b4c6..820f797 100644 --- a/runbooks/workspace-stop-backup-failed.md +++ b/runbooks/workspace-stop-backup-failed.md @@ -8,7 +8,7 @@ applies_to: dashboards: [storage, fleet] severity: warning owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms @@ -62,8 +62,13 @@ lost. spin workspace stop --force ``` -TODO: an administrator cannot stop another person's workspace; `StopWorkspace` answers only the -owner ([spin#1045](https://github.com/spin-stack/spin/issues/1045)). +- **The owner cannot stop it, and you stop it for them**: the same stop, audited with your reason; + the workspace's history names you. Add `--force` only to give up what only its runner has: + + ``` + spin admin workspace stop --reason "" + spin admin workspace stop --reason "" --force + ``` ## Verify