runbooks: a database restored to a time or a chosen base backup

runbooks/database-restore.md

@@ -8,13 +8,14 @@ applies_to:
   dashboards: [spin-db-performance]
 severity: info
 owner: operator
-last_reviewed: 2026-11-09
+last_reviewed: 2026-11-12
 ---
 
 ## Symptoms
 
 Read this when a release replaces the control plane's machine, when a new control plane machine
-waits a long time before it serves, or when the database will not start.
+waits a long time before it serves, when the database will not start, or when a bad write (a
+lockout, a destructive change) has to be undone by restoring the database to before it.
 
 How it works, so the logs make sense:
 
@@ -29,13 +30,18 @@ How it works, so the logs make sense:
   The old machine's unit exits and stays down.
 - **Restore.** A machine with no database of its own, whose holder has not said it holds it for
   a minute, restores the newest base backup and replays every WAL segment after it.
+- **Restore to a point.** An operator on the control plane's machine can restore the database
+  to a time, a place in the WAL or a chosen base backup (`spin-database restore`, below). It
+  replays to that point and starts a new history there: the writes after it are left out, and
+  every later restore follows the new history.
 
 ## Impact
 
 While the new machine hands over or restores, the control plane does not start: the API and the
 dashboard are down. Running workspaces keep running. A restore after a machine died loses what
 that machine wrote and had not archived: at most about five minutes of writes, the archive's
-timeout.
+timeout. A restore to a point loses every write after that point, the bad one and every good one
+since: pick the latest point before the bad write.
 
 ## Diagnose
 
@@ -64,6 +70,15 @@ timeout.
    - `handed over without WAL the other machine could not archive`: some writes are only on the
      old machine's disk.
    - `the archive holds WAL and no complete base backup`: the database refuses to start empty.
+3. What the archive holds, to choose a point to restore to (on the control plane's machine):
+
+   ```
+   sudo spin-database bases
+   ```
+
+   One line per complete base backup: its id, when it was taken (UTC), its timeline and the WAL
+   segment it starts in. Each restores to any time from when it was taken to the end of the WAL,
+   whose last segment and the time the bucket says it was stored are printed below the list.
 
 ## Fix
 
@@ -83,13 +98,53 @@ timeout.
 - **WAL left on the old machine** or **no complete base backup**: stop and escalate. Do not delete
   anything in the archive or handover buckets, and do not start a database by hand.
 
-TODO: there is no command to restore the database to an earlier point in time or to a chosen
-base backup; a restore always takes the newest ([spin#1052](https://github.com/spin-stack/spin/issues/1052)).
+- **Undo a bad write** by restoring to a time before it. On the control plane's machine:
+
+  1. Find when the bad write happened (the audit log, the control plane's journal) and pick a
+     time just before it, in UTC: `2026-11-12T09:30:00Z`.
+  2. Stop the control plane and the database. Stopping the database archives its last segment;
+     its log says `stopped with everything it wrote archived`.
+
+     ```
+     sudo systemctl stop spin-controlplane.service spin-postgres.service
+     ```
+
+  3. Restore. The newest base backup taken before the time is laid and replayed to it:
+
+     ```
+     sudo spin-database restore --to 2026-11-12T09:30:00Z
+     ```
+
+     `--base <id>` (from `spin-database bases`) lays that base backup instead, and with no `--to`
+     replays it to the end. `--to-lsn <lsn>` (like `0/16B3748`) ends at a place in the WAL
+     instead of a time. The restore is laid beside the machine's database, which is replaced
+     only once the restore has come up: a refused or failed restore leaves it as it was. It
+     ends with `restored; start spin-postgres.service to serve it`.
+  4. Start them again; the database serves the restored copy:
+
+     ```
+     sudo systemctl start spin-postgres.service spin-controlplane.service
+     ```
+
+  What a restore refuses, and what to do instead:
+  - `the database is serving: on this machine`: step 2 was skipped; stop the database first.
+  - `the database is serving: on <machine>`: another machine holds the database. Restore there,
+    or stop its database first.
+  - `has not happened yet`: the time is in the future. For the newest state there is nothing to
+    restore; leave out `--to` to replay to the end.
+  - `its oldest base backup ... was taken at ..., after ...` or `base backup ... was taken at
+    ..., after ...`: the archive holds nothing that far back (seven base backups are kept), or
+    the base named is newer than the time. Choose a later time or an older base backup.
+  - `its WAL ends before ...`: the time is after the last write the archive holds. Leave out
+    `--to` to replay to the end.
+  - `holds no complete base backup <id>`: copy the id from `spin-database bases`.
 
 ## Verify
 
 `spin admin health` shows the database and its archive ok and exits 0, the dashboard loads, and
-*Since the last archived WAL* on the database dashboard is under five minutes.
+*Since the last archived WAL* on the database dashboard is under five minutes. After a restore to
+a point, what the bad write changed is back as it was, and `spin-database bases` lists a new base
+backup, on a timeline one higher, taken as the restore finished.
 
 ## Escalate