From 0f4fe42af0e3df411f325fbacc9d292084748f69 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 12 Nov 2026 16:26:59 -0300 Subject: [PATCH] runbooks: a database restored to a time or a chosen base backup, in place of its TODO spin#1175 adds `spin-database bases` and `spin-database restore --to/--to-lsn/--base` on the control plane's machine; the runbook says how an operator undoes a bad write with them and what each refusal means. Co-Authored-By: Claude Opus 5.5 --- runbooks/database-restore.md | 67 ++++++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 6 deletions(-) diff --git a/runbooks/database-restore.md b/runbooks/database-restore.md index 89540a7..e0216ea 100644 --- a/runbooks/database-restore.md +++ b/runbooks/database-restore.md @@ -8,13 +8,14 @@ applies_to: dashboards: [spin-db-performance] severity: info owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-12 --- ## Symptoms Read this when a release replaces the control plane's machine, when a new control plane machine -waits a long time before it serves, or when the database will not start. +waits a long time before it serves, when the database will not start, or when a bad write (a +lockout, a destructive change) has to be undone by restoring the database to before it. How it works, so the logs make sense: @@ -29,13 +30,18 @@ How it works, so the logs make sense: The old machine's unit exits and stays down. - **Restore.** A machine with no database of its own, whose holder has not said it holds it for a minute, restores the newest base backup and replays every WAL segment after it. +- **Restore to a point.** An operator on the control plane's machine can restore the database + to a time, a place in the WAL or a chosen base backup (`spin-database restore`, below). It + replays to that point and starts a new history there: the writes after it are left out, and + every later restore follows the new history. ## Impact While the new machine hands over or restores, the control plane does not start: the API and the dashboard are down. Running workspaces keep running. A restore after a machine died loses what that machine wrote and had not archived: at most about five minutes of writes, the archive's -timeout. +timeout. A restore to a point loses every write after that point, the bad one and every good one +since: pick the latest point before the bad write. ## Diagnose @@ -64,6 +70,15 @@ timeout. - `handed over without WAL the other machine could not archive`: some writes are only on the old machine's disk. - `the archive holds WAL and no complete base backup`: the database refuses to start empty. +3. What the archive holds, to choose a point to restore to (on the control plane's machine): + + ``` + sudo spin-database bases + ``` + + One line per complete base backup: its id, when it was taken (UTC), its timeline and the WAL + segment it starts in. Each restores to any time from when it was taken to the end of the WAL, + whose last segment and the time the bucket says it was stored are printed below the list. ## Fix @@ -83,13 +98,53 @@ timeout. - **WAL left on the old machine** or **no complete base backup**: stop and escalate. Do not delete anything in the archive or handover buckets, and do not start a database by hand. -TODO: there is no command to restore the database to an earlier point in time or to a chosen -base backup; a restore always takes the newest ([spin#1052](https://github.com/spin-stack/spin/issues/1052)). +- **Undo a bad write** by restoring to a time before it. On the control plane's machine: + + 1. Find when the bad write happened (the audit log, the control plane's journal) and pick a + time just before it, in UTC: `2026-11-12T09:30:00Z`. + 2. Stop the control plane and the database. Stopping the database archives its last segment; + its log says `stopped with everything it wrote archived`. + + ``` + sudo systemctl stop spin-controlplane.service spin-postgres.service + ``` + + 3. Restore. The newest base backup taken before the time is laid and replayed to it: + + ``` + sudo spin-database restore --to 2026-11-12T09:30:00Z + ``` + + `--base ` (from `spin-database bases`) lays that base backup instead, and with no `--to` + replays it to the end. `--to-lsn ` (like `0/16B3748`) ends at a place in the WAL + instead of a time. The restore is laid beside the machine's database, which is replaced + only once the restore has come up: a refused or failed restore leaves it as it was. It + ends with `restored; start spin-postgres.service to serve it`. + 4. Start them again; the database serves the restored copy: + + ``` + sudo systemctl start spin-postgres.service spin-controlplane.service + ``` + + What a restore refuses, and what to do instead: + - `the database is serving: on this machine`: step 2 was skipped; stop the database first. + - `the database is serving: on `: another machine holds the database. Restore there, + or stop its database first. + - `has not happened yet`: the time is in the future. For the newest state there is nothing to + restore; leave out `--to` to replay to the end. + - `its oldest base backup ... was taken at ..., after ...` or `base backup ... was taken at + ..., after ...`: the archive holds nothing that far back (seven base backups are kept), or + the base named is newer than the time. Choose a later time or an older base backup. + - `its WAL ends before ...`: the time is after the last write the archive holds. Leave out + `--to` to replay to the end. + - `holds no complete base backup `: copy the id from `spin-database bases`. ## Verify `spin admin health` shows the database and its archive ok and exits 0, the dashboard loads, and -*Since the last archived WAL* on the database dashboard is under five minutes. +*Since the last archived WAL* on the database dashboard is under five minutes. After a restore to +a point, what the bad write changed is back as it was, and `spin-database bases` lists a new base +backup, on a timeline one higher, taken as the restore finished. ## Escalate