runbooks: the administrator's commands spin#1045-1047 and #1051 added, in place of their TODOs

runbooks/host-wrote-foreign-volume.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: [storage]
 severity: critical
 owner: security
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -76,12 +76,23 @@ Treat the host as taken until proven otherwise.
    Re-key a disk that is not marked (its workspace stopped or quarantined first):
 
    ```
-   spin admin volume rekey <workspace>
+   spin admin volume rekey <workspace-id>
    ```
 
    A re-key deletes the old disk and every snapshot of it. More in [key-rotation](key-rotation.md).
-
-TODO: removing an autoscaling group's runner believed taken does not mark the disks it held for re-keying, and nothing lists them ([spin#1047](https://github.com/spin-stack/spin/issues/1047))
+- **The host is a runner of an autoscaling group**: remove it as taken. Its credentials are
+  revoked and it is cut off first, every team disk whose key it was given is marked for
+  re-keying, and then its machine is given back to its group:
+
+  ```
+  spin admin runner remove --taken <runner-id>
+  ```
+
+  It prints `Runner revoked; <n> disks marked for re-keying (spin admin volume rekeys)`. It is
+  refused while a workspace's only disk is on the runner; add `--leave-workspaces` to remove it
+  anyway, and each such workspace runs again only once recovered onto another runner. In the
+  dashboard: the runner's menu, *Remove as compromised*. The marked disks are listed by
+  `spin admin volume rekeys`.
 
 ## Verify
 

runbooks/identity-audit-failing.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: []
 severity: critical
 owner: security
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -68,11 +68,15 @@ and every grant is recorded in the bucket where it cannot be taken back.
   spin admin jobs run identity-audit
   ```
 
-  Each run checks only its own hour; hours missed while it failed are not checked again by the job.
-  Check them by hand: the `Decrypt` calls in the key service's log against the records under
-  `identity/granted/` for the same hours.
+  Each run audits every settled hour since the last one it audited, oldest first, at most 24 a
+  run. An hour it cannot read stops the run and is the next run's first. On
+  [Admin → Jobs](/admin/jobs?job=identity-audit), `hours` is the hours audited this run, `behind`
+  the settled hours still owed, and `unaudited` the hours the key service's log no longer keeps.
 
-  TODO: the audit does not go back over hours it missed while failing ([spin#1051](https://github.com/spin-stack/spin/issues/1051))
+  If the run fails with `audit: the key service no longer keeps its log of hours that were never
+  audited`, those hours (the error names how many and between when) are the only ones to check
+  by hand: the `Decrypt` calls in whatever record of them you still have against the records
+  under `identity/granted/` for the same hours.
 
 ## Verify
 

runbooks/key-rotation.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: []
 severity: info
 owner: security
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -54,12 +54,11 @@ workspace is stopped or quarantined. Watch it with `spin admin volume rekeys`. B
 stopped or quarantined workspace:
 
 ```
-spin admin volume rekey <workspace>
+spin admin volume rekey <workspace-id>
 ```
 
-TODO: the CLI looks the workspace up as the caller's own before it calls the admin API, so it
-reaches only an administrator's own workspaces; there is no command for another person's
-([spin#1046](https://github.com/spin-stack/spin/issues/1046)).
+It takes the workspace's id, not its name, and works for anybody's workspace. The id is in
+`spin admin volume rekeys --json` (`workspace_id`) for a marked disk, or on the workspace's page.
 
 **Runner registration tokens**: `spin admin token revoke <token-id>`. A pool's token is
 re-minted on the control plane's machine with `sudo spin-controlplane registration-token ...`.

runbooks/volume-not-served.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: [fleet, storage]
 severity: critical
 owner: operator
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -83,7 +83,16 @@ it writes from now on is lost with its host. Each failed restore reads the objec
 - **The host is the cause for many volumes**: stop it taking new work with
   `spin admin runner drain <runner-id>`, and look at it on a [host-shell](host-shell.md).
 
-TODO: an administrator cannot list which workspaces have a refused volume, or recover a workspace they do not own ([spin#1045](https://github.com/spin-stack/spin/issues/1045))
+- **Which workspaces it is**: list every refused volume, with the refusal, the host, the
+  workspace and its owner (the listing is audited):
+
+  ```
+  spin admin volume refused
+  ```
+
+  If the owner cannot recover one, recover it for them with
+  `spin admin workspace recover <workspace-id> --reason "<why>"`, as in
+  [workspace-held](workspace-held.md).
 
 ## Verify
 

runbooks/workspace-egress-high.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: [fleet]
 severity: warning
 owner: operator
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -34,7 +34,12 @@ send traffic somewhere it should not).
    spin workspace egress <workspace>
    ```
 
-   TODO: an administrator cannot read where a workspace they do not own sends ([spin#1045](https://github.com/spin-stack/spin/issues/1045))
+   An administrator reads it for a workspace that is not theirs. It changes nothing, and the
+   reading is audited with the reason; if the audit entry cannot be written, it answers nothing:
+
+   ```
+   spin admin workspace egress <workspace-id> --reason "<why>"
+   ```
 3. See what it has cost this month:
 
    ```

runbooks/workspace-held.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: [fleet, storage]
 severity: critical
 owner: operator
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -63,18 +63,26 @@ is ever moved on its own, and the owner must recover by hand.
 
 - **The host can be brought back** (a reboot, a network fault): bring it back. The workspace runs
   on it again with nothing lost. Nobody needs to recover anything.
-- **The host is gone for good**: tell the owner. Only the owner can give up their work: the
-  command is theirs (`spin workspace recover <workspace> --accept-data-loss`). It takes the disk
-  from the dead host and places it on a live one; then they start it. Afterwards retire the
-  host ([lost-runner](lost-runner.md)).
+- **The host is gone for good**: tell the owner. The owner recovers it with
+  `spin workspace recover <workspace> --accept-data-loss`. It takes the disk from the dead host
+  and places it on a live one; then they start it. Afterwards retire the host
+  ([lost-runner](lost-runner.md)).
+- **The owner cannot recover it, and you recover it for them**: run it first without
+  `--accept-data-loss` to see the work that is given up; nothing changes until you pass it.
+  `--reason` is required:
+
+  ```
+  spin admin workspace recover <workspace-id> --reason "<why>"
+  spin admin workspace recover <workspace-id> --reason "<why>" --accept-data-loss
+  ```
+
+  It is audited, and written to the workspace's history with your reason, which its owner sees
+  on Activity.
 - **The installation should move such workspaces on its own** next time: set
   `rehome_after_seconds` and `rehome_max_unpublished_bytes` in the installation's configuration
   file, and apply the change on [Admin → Settings](/admin/settings). Giving up unpublished work
   without asking is a policy decision; make it on purpose.
 
-TODO: an administrator cannot recover a workspace for its owner; `RecoverWorkspace` answers
-only the owner ([spin#1045](https://github.com/spin-stack/spin/issues/1045)).
-
 ## Verify
 
 `spin workspace get <workspace>` shows a live host, and the workspace starts. A recovery is in

runbooks/workspace-stop-backup-failed.md

@@ -8,7 +8,7 @@ applies_to:
   dashboards: [storage, fleet]
 severity: warning
 owner: operator
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
@@ -62,8 +62,13 @@ lost.
   spin workspace stop <workspace> --force
   ```
 
-TODO: an administrator cannot stop another person's workspace; `StopWorkspace` answers only the
-owner ([spin#1045](https://github.com/spin-stack/spin/issues/1045)).
+- **The owner cannot stop it, and you stop it for them**: the same stop, audited with your reason;
+  the workspace's history names you. Add `--force` only to give up what only its runner has:
+
+  ```
+  spin admin workspace stop <workspace-id> --reason "<why>"
+  spin admin workspace stop <workspace-id> --reason "<why>" --force
+  ```
 
 ## Verify