From a01a4b91f5ac43998e4c289f73b0ffa2525ac11d Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Mon, 9 Nov 2026 17:56:11 -0300 Subject: [PATCH 1/2] check: the shape spin's app reads, held on every push (#1) Co-Authored-By: Claude Opus 5.5 --- .github/workflows/gate.yml | 34 ++++ .gitignore | 2 + CONTRIBUTING.md | 90 +++++++++++ README.md | 21 +++ Taskfile.yml | 20 +++ cmd/check/main.go | 267 ++++++++++++++++++++++++++++++ cmd/check/main_test.go | 117 ++++++++++++++ go.mod | 147 +++++++++++++++++ go.sum | 324 +++++++++++++++++++++++++++++++++++++ 9 files changed, 1022 insertions(+) create mode 100644 .github/workflows/gate.yml create mode 100644 .gitignore create mode 100644 CONTRIBUTING.md create mode 100644 README.md create mode 100644 Taskfile.yml create mode 100644 cmd/check/main.go create mode 100644 cmd/check/main_test.go create mode 100644 go.mod create mode 100644 go.sum diff --git a/.github/workflows/gate.yml b/.github/workflows/gate.yml new file mode 100644 index 0000000..2005719 --- /dev/null +++ b/.github/workflows/gate.yml @@ -0,0 +1,34 @@ +name: Gate + +# Holds every runbook to the shape spin's app reads, on every push and pull request. + +on: + push: + branches: [main] + pull_request: + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: gate-${{ github.ref }} + cancel-in-progress: true + +# go-tools' downloads (sfw, versions) ask GitHub with this: anonymous ones are the first it sheds +# under load, and they answered 504 for minutes on 2026-11-09. +env: + GITHUB_TOKEN: ${{ github.token }} + +jobs: + check: + # The check main's ruleset requires, named Gate in every spin-stack repository. + name: Gate + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: go.mod + # What a developer runs, with the Task go.mod pins. + - run: go tool task check diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..baab83c --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +.task/ +_output/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..3825836 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,90 @@ +# Contributing + +A runbook is what an operator of a Spinbox installation reads when something is wrong: an alert +fired, a job keeps failing, a person was told their workspace is waiting for them, or someone +asks for a procedure (a host shell, a restore). It says what is seen, who it hurts, how to find +out, what to do, how to know it worked and whom to call. Short, plain English, the commands as +they are typed. + +This repository is public. **Never** write a secret, a hostname or account id of an installation, +a person's name or email, or anything a customer did. Write ``, ``, +`` in code, for any installation. + +## A runbook + +`runbooks/.md`, named for its id: + +```markdown +--- +id: host-not-reporting +title: A host stopped reporting +applies_to: + alerts: [hosts-disconnected, host-heartbeat-stale] + jobs: [] + events: [] + dashboards: [fleet] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +What fires, what a person sees. + +## Impact + +Who and what it hurts, and what is lost if nothing is done. + +## Diagnose + +The commands (`spin ...`, `controlplane ...`) and the pages (Admin → Runners) that say why. + +## Fix + +What to do, in order, for each cause the diagnosis finds. + +## Verify + +How to know it worked: the alert resolves, the page reads healthy. + +## Escalate + +When to stop and whom to call, with what to bring. +``` + +- **`id`** is stable: spin's alert rules and pages link it (`/admin/runbooks/`), and spin's + tests fail on a link to one that does not exist. Renaming one is a change in spin too. +- **`applies_to`** is what sends an operator here, and what the app links from: `alerts` are + Grafana alert rule uids, `jobs` control plane job names (Admin → Jobs), `events` the + notices a workspace's owner gets (`HELD`, `STOP_BACKUP_FAILED`, `UNRESPONSIVE`, `REHOMED`), + `dashboards` Grafana dashboard uids. A procedure nobody is paged for applies to nothing and is + `severity: info`. +- **`severity`**: `critical` (someone is losing work or access now), `warning`, `info`. +- **`owner`**: the role that keeps it true: `operator`, `security`. +- **`last_reviewed`**: the date someone last held it against the release it describes. + +## What the app renders + +Markdown, and nothing else: raw HTML is refused here and dropped by the app. Links are to +another runbook (`[held](workspace-held.md)`, which stays in the app), to a path in the app +(`/admin/jobs`), or `https://`. No images. + +Ground every command and page in what spin has: a flag that does not exist is worse than no +runbook. Where spin has no way yet, write `TODO` and link the issue that will add one. + +## Check + +``` +go tool task check +``` + +Fails on a field missing, an id that is not its file's name or is used twice, a section missing +or out of order, raw HTML, a link to a runbook that does not exist, an AWS account id, a private +key or an installation's hostname. CI runs the same (Gate). + +## How a change reaches an installation + +spin pins this repository at a commit (`versions.yaml`, `runbooks`). A merged change here reaches +the app when spin is bumped (`task bump NAME=runbooks`, `task runbooks`) and released. The app's +**Open in repo** link is the file at the commit the release pins: propose the change on `main`. diff --git a/README.md b/README.md new file mode 100644 index 0000000..c4854ad --- /dev/null +++ b/README.md @@ -0,0 +1,21 @@ +# spin-stack / runbooks + +What an operator of a [Spinbox](https://github.com/spin-stack/spin) installation does when an +alert fires, a job keeps failing, a workspace's owner is told something went wrong, or a +procedure is asked for: one markdown file per runbook under `runbooks/`. + +An installation's administrators read them inside the app, at **Admin → Runbooks**, without +leaving it: spin pins this repository at a commit and serves the runbooks of the release it +runs. Every Grafana alert rule links its runbook, and so do Admin → Jobs and the notices a +workspace's owner gets. **Open in repo**, on each runbook in the app, is the file here at that +commit, to propose a change. + +``` +go tool task check # what CI runs: front matter, sections, links, nothing private +``` + +[CONTRIBUTING.md](CONTRIBUTING.md) is how a runbook is written and what the check refuses. + +## License + +Apache License 2.0, as spin. diff --git a/Taskfile.yml b/Taskfile.yml new file mode 100644 index 0000000..ce50967 --- /dev/null +++ b/Taskfile.yml @@ -0,0 +1,20 @@ +version: '3' + +# Operational runbooks for an installation of Spinbox. runbooks/*.md are the source; spin pins this +# repository and serves them to an installation's administrators, so `task check` holds each to +# the shape the app reads. + +tasks: + default: + cmds: + - task: check + + check: + desc: What CI runs - fail if a runbook's front matter, sections or links are not what the app reads, it carries raw HTML or what a public repository must not, a workflow step is not one task, or an install skips Socket Firewall. + cmds: + # The Task running this one: go.mod's is not on PATH, and taskgate asks it for the task names. + - go tool taskgate -task {{.TASK_EXE}} + - go tool sfw check + - go vet ./... + - go test ./... + - go run ./cmd/check diff --git a/cmd/check/main.go b/cmd/check/main.go new file mode 100644 index 0000000..4242b5a --- /dev/null +++ b/cmd/check/main.go @@ -0,0 +1,267 @@ +// Command check holds every runbook under runbooks/ to the shape the app reads: its front matter, +// its sections in order, no raw HTML, links to runbooks that exist, and nothing a public +// repository must not carry. It prints every problem and exits 1 when there is one. +package main + +import ( + "bytes" + "errors" + "fmt" + "io/fs" + "os" + "path" + "regexp" + "slices" + "sort" + "strings" + "time" + + "github.com/yuin/goldmark" + "github.com/yuin/goldmark/ast" + "github.com/yuin/goldmark/extension" + "github.com/yuin/goldmark/text" + "gopkg.in/yaml.v3" +) + +func main() { + problems, err := check(os.DirFS("."), "runbooks") + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + for _, p := range problems { + fmt.Println(p) + } + if len(problems) > 0 { + os.Exit(1) + } +} + +// Sections is every section a runbook has, in the order it has them: what is seen, who it hurts, +// how to find out, what to do, how to know it worked, and whom to call when it did not. +var Sections = []string{"Symptoms", "Impact", "Diagnose", "Fix", "Verify", "Escalate"} + +// Severities is what a runbook's severity may be: the alert rules' two, and info for a procedure +// nothing pages for. +var Severities = []string{"critical", "warning", "info"} + +// AppliesTo is what sends an operator to a runbook. The app links each from where these are shown: +// an alert rule by its uid, a job by its name, an owner's notice by its event (HELD), a Grafana +// dashboard by its uid. +type AppliesTo struct { + Alerts []string `yaml:"alerts"` + Jobs []string `yaml:"jobs"` + Events []string `yaml:"events"` + Dashboards []string `yaml:"dashboards"` +} + +// Front is a runbook's front matter. +type Front struct { + ID string `yaml:"id"` + Title string `yaml:"title"` + AppliesTo AppliesTo `yaml:"applies_to"` + Severity string `yaml:"severity"` + Owner string `yaml:"owner"` + LastReviewed string `yaml:"last_reviewed"` +} + +var ( + kebab = regexp.MustCompile(`^[a-z0-9]+(-[a-z0-9]+)*$`) + // What a public repository must not carry. An AWS account id is twelve digits on their own; a + // private key announces itself; an installation's hostname is its own, and a runbook says + // . + forbidden = []struct { + re *regexp.Regexp + what string + }{ + {regexp.MustCompile(`(^|[^0-9])[0-9]{12}([^0-9]|$)`), "an AWS account id"}, + {regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----`), "a private key"}, + {regexp.MustCompile(`(?i)\bspinbox\.dev\b`), "an installation's own hostname (write )"}, + {regexp.MustCompile(`\b(AKIA|ASIA)[A-Z0-9]{16}\b`), "an AWS access key id"}, + } +) + +// check reads every runbook under dir and returns what is wrong with them, file by file. +func check(fsys fs.FS, dir string) ([]string, error) { + names, err := fs.Glob(fsys, dir+"/*.md") + if err != nil { + return nil, err + } + if len(names) == 0 { + return nil, fmt.Errorf("no runbooks under %s/", dir) + } + sort.Strings(names) + var problems []string + ids := map[string]string{} + type link struct{ from, to string } + var links []link + for _, name := range names { + raw, err := fs.ReadFile(fsys, name) + if err != nil { + return nil, err + } + file := strings.TrimSuffix(path.Base(name), ".md") + front, out, targets := one(file, raw) + for _, p := range out { + problems = append(problems, name+": "+p) + } + if other, ok := ids[front.ID]; ok && front.ID != "" { + problems = append(problems, fmt.Sprintf("%s: id %q is used by %s too", name, front.ID, other)) + } else if front.ID != "" { + ids[front.ID] = name + } + for _, t := range targets { + links = append(links, link{name, t}) + } + } + for _, l := range links { + if _, ok := ids[l.to]; !ok { + problems = append(problems, fmt.Sprintf("%s: links runbook %q, which does not exist", l.from, l.to)) + } + } + return problems, nil +} + +// one checks a runbook on its own and returns its front matter, its problems and the ids of the +// runbooks it links. +func one(file string, raw []byte) (Front, []string, []string) { + var problems []string + front, src, err := split(raw) + if err != nil { + return Front{}, []string{err.Error()}, nil + } + var f Front + dec := yaml.NewDecoder(bytes.NewReader(front)) + dec.KnownFields(true) + if err := dec.Decode(&f); err != nil { + return Front{}, []string{"front matter: " + err.Error()}, nil + } + problems = append(problems, f.problems(file)...) + + for _, rule := range forbidden { + if rule.re.Match(raw) { + problems = append(problems, "carries "+rule.what) + } + } + + doc := goldmark.New(goldmark.WithExtensions(extension.GFM)).Parser().Parse(text.NewReader(src)) + var headings []string + var targets []string + err = ast.Walk(doc, func(n ast.Node, entering bool) (ast.WalkStatus, error) { + if !entering { + return ast.WalkContinue, nil + } + switch n := n.(type) { + case *ast.Heading: + title := string(n.Lines().Value(src)) + switch { + case n.Level == 1: + problems = append(problems, "has a level-one heading: the title is the front matter's") + case n.Level == 2: + headings = append(headings, strings.TrimSpace(title)) + } + case *ast.HTMLBlock, *ast.RawHTML: + problems = append(problems, "carries raw HTML, which the app does not render") + case *ast.Link: + if t, ok := runbookLink(string(n.Destination)); ok { + targets = append(targets, t) + } else if p := badLink(string(n.Destination)); p != "" { + problems = append(problems, p) + } + case *ast.AutoLink: + if p := badLink(string(n.URL(src))); p != "" { + problems = append(problems, p) + } + case *ast.Image: + problems = append(problems, "has an image: the app renders text") + } + return ast.WalkContinue, nil + }) + if err != nil { + problems = append(problems, err.Error()) + } + if !slices.Equal(headings, Sections) { + problems = append(problems, fmt.Sprintf("sections are %q, not %q", headings, Sections)) + } + return f, problems, targets +} + +func (f Front) problems(file string) []string { + var p []string + switch { + case f.ID == "": + p = append(p, "has no id") + case !kebab.MatchString(f.ID): + p = append(p, fmt.Sprintf("id %q is not kebab case", f.ID)) + case f.ID != file: + p = append(p, fmt.Sprintf("id %q is not the file's name %q", f.ID, file)) + } + if strings.TrimSpace(f.Title) == "" { + p = append(p, "has no title") + } + if !slices.Contains(Severities, f.Severity) { + p = append(p, fmt.Sprintf("severity %q is none of %q", f.Severity, Severities)) + } + if !kebab.MatchString(f.Owner) { + p = append(p, fmt.Sprintf("owner %q is not a kebab-case role", f.Owner)) + } + if _, err := time.Parse(time.DateOnly, f.LastReviewed); err != nil { + p = append(p, fmt.Sprintf("last_reviewed %q is not a date (YYYY-MM-DD)", f.LastReviewed)) + } + a := f.AppliesTo + if len(a.Alerts)+len(a.Jobs)+len(a.Events)+len(a.Dashboards) == 0 && f.Severity != "info" { + p = append(p, "applies to nothing: a runbook for an alert names it, and a procedure is info") + } + for _, list := range [][]string{a.Alerts, a.Jobs, a.Dashboards} { + for _, s := range list { + if !kebab.MatchString(s) { + p = append(p, fmt.Sprintf("applies_to %q is not kebab case", s)) + } + } + } + for _, e := range a.Events { + if e != strings.ToUpper(e) || strings.HasPrefix(e, "WORKSPACE_EVENT_TYPE_") { + p = append(p, fmt.Sprintf("event %q is not written as the app names it (HELD)", e)) + } + } + return p +} + +// split cuts the front matter from the body: the file opens with a line of three dashes, and the +// next such line closes it. +func split(raw []byte) ([]byte, []byte, error) { + const fence = "---\n" + if !bytes.HasPrefix(raw, []byte(fence)) { + return nil, nil, errors.New("does not open with front matter (---)") + } + rest := raw[len(fence):] + end := bytes.Index(rest, []byte("\n"+fence)) + if end < 0 { + return nil, nil, errors.New("front matter is not closed (---)") + } + return rest[:end+1], rest[end+1+len(fence):], nil +} + +var runbookFile = regexp.MustCompile(`^(?:\./)?([a-z0-9]+(?:-[a-z0-9]+)*)\.md(#[a-z0-9-]+)?$`) + +// runbookLink is the id a link to another runbook names: its file, beside this one. +func runbookLink(dest string) (string, bool) { + m := runbookFile.FindStringSubmatch(dest) + if m == nil { + return "", false + } + return m[1], true +} + +// badLink says what is wrong with a link that is not to another runbook: the app opens https and +// mailto, an anchor on the same page, and nothing else. +func badLink(dest string) string { + switch { + case strings.HasPrefix(dest, "https://"), strings.HasPrefix(dest, "mailto:"), strings.HasPrefix(dest, "#"): + return "" + case strings.HasPrefix(dest, "/"): + // A path in the app (/admin/jobs): the same installation that serves the runbook. + return "" + } + return fmt.Sprintf("link %q is neither another runbook, a path in the app, https nor mailto", dest) +} diff --git a/cmd/check/main_test.go b/cmd/check/main_test.go new file mode 100644 index 0000000..9d96b81 --- /dev/null +++ b/cmd/check/main_test.go @@ -0,0 +1,117 @@ +package main + +import ( + "strings" + "testing" + "testing/fstest" +) + +const good = `--- +id: %s +title: A host stopped reporting +applies_to: + alerts: [host-heartbeat-stale] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +The alert fired. See [the other](other.md) and [jobs](/admin/jobs). + +## Impact + +Workspaces on it. + +## Diagnose + +` + "`spin admin hosts`" + ` + +## Fix + +Drain it. + +## Verify + +It reports. + +## Escalate + +The release's maintainers: https://github.com/spin-stack/spin/issues +` + +func runbook(id string, edit func(string) string) string { + s := strings.Replace(good, "%s", id, 1) + if edit != nil { + s = edit(s) + } + return s +} + +func replace(old, new string) func(string) string { + return func(s string) string { return strings.Replace(s, old, new, 1) } +} + +func TestCheckRefusesWhatTheAppCannotRead(t *testing.T) { + cases := []struct { + name string + file string + edit func(string) string + want string // empty: no problem + }{ + {name: "a good runbook", file: "host-down", want: ""}, + {name: "id is not the file's name", file: "host-gone", want: `is not the file's name`}, + {name: "a section missing", file: "host-down", edit: replace("## Verify\n", ""), want: "sections are"}, + {name: "sections out of order", file: "host-down", edit: replace("## Impact", "## Fix2"), want: "sections are"}, + {name: "raw HTML", file: "host-down", edit: replace("Drain it.", "Drain it."), want: "raw HTML"}, + {name: "an HTML block", file: "host-down", edit: replace("Drain it.", ""), want: "raw HTML"}, + {name: "a runbook that does not exist", file: "host-down", edit: replace("other.md", "nowhere.md"), want: `links runbook "nowhere"`}, + {name: "a javascript link", file: "host-down", edit: replace("(/admin/jobs)", "(javascript:alert(1))"), want: "neither another runbook"}, + {name: "a severity of its own", file: "host-down", edit: replace("severity: critical", "severity: high"), want: "severity"}, + {name: "a date that is not one", file: "host-down", edit: replace("2026-11-09", "last week"), want: "last_reviewed"}, + {name: "a field nobody reads", file: "host-down", edit: replace("owner: operator", "owner: operator\nteam: x"), want: "front matter"}, + {name: "an event spelled as the proto does", file: "host-down", edit: replace("alerts: [host-heartbeat-stale]", "events: [WORKSPACE_EVENT_TYPE_HELD]"), want: "event"}, + {name: "applies to nothing and pages", file: "host-down", edit: replace(" alerts: [host-heartbeat-stale]\n", ""), want: "applies to nothing"}, + {name: "an account id", file: "host-down", edit: replace("Drain it.", "Account 123456789012."), want: "AWS account id"}, + {name: "an installation's hostname", file: "host-down", edit: replace("Drain it.", "Open app.spinbox.dev."), want: "hostname"}, + {name: "a level-one heading", file: "host-down", edit: replace("## Symptoms", "# Title\n\n## Symptoms"), want: "level-one"}, + {name: "no front matter", file: "host-down", edit: replace("---\nid:", "id:"), want: "front matter"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + fsys := fstest.MapFS{ + "runbooks/" + c.file + ".md": {Data: []byte(runbook("host-down", c.edit))}, + "runbooks/other.md": {Data: []byte(runbook("other", replace("(other.md)", "(host-down.md)")))}, + } + problems, err := check(fsys, "runbooks") + if err != nil { + t.Fatal(err) + } + got := strings.Join(problems, "\n") + if c.want == "" { + if got != "" { + t.Fatalf("want no problem, got:\n%s", got) + } + return + } + if !strings.Contains(got, c.want) { + t.Fatalf("want a problem saying %q, got:\n%s", c.want, got) + } + }) + } +} + +func TestCheckRefusesAnIDUsedTwice(t *testing.T) { + fsys := fstest.MapFS{ + "runbooks/other.md": {Data: []byte(runbook("other", nil))}, + "runbooks/copy.md": {Data: []byte(runbook("other", nil))}, + } + problems, err := check(fsys, "runbooks") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(strings.Join(problems, "\n"), "used by") { + t.Fatalf("want an id used twice refused, got %q", problems) + } +} diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..511f0c6 --- /dev/null +++ b/go.mod @@ -0,0 +1,147 @@ +module github.com/spin-stack/runbooks + +go 1.27.3 + +require ( + github.com/yuin/goldmark v1.8.6 + gopkg.in/yaml.v3 v3.0.1 +) + +require ( + cel.dev/expr v0.25.2 // indirect + charm.land/bubbles/v2 v2.1.1 // indirect + charm.land/bubbletea/v2 v2.0.8 // indirect + charm.land/lipgloss/v2 v2.0.6 // indirect + cloud.google.com/go v0.123.0 // indirect + cloud.google.com/go/auth v0.23.1 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect + cloud.google.com/go/compute/metadata v0.9.0 // indirect + cloud.google.com/go/iam v1.13.0 // indirect + cloud.google.com/go/monitoring v1.30.0 // indirect + cloud.google.com/go/storage v1.64.0 // indirect + github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 // indirect + github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.59.0 // indirect + github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.59.0 // indirect + github.com/Ladicle/tabwriter v1.0.0 // indirect + github.com/Masterminds/semver/v3 v3.5.0 // indirect + github.com/alecthomas/chroma/v2 v2.27.0 // indirect + github.com/atotto/clipboard v0.1.4 // indirect + github.com/aws/aws-sdk-go-v2 v1.43.4 // indirect + github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.16 // indirect + github.com/aws/aws-sdk-go-v2/config v1.32.35 // indirect + github.com/aws/aws-sdk-go-v2/credentials v1.19.34 // indirect + github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.35 // indirect + github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.35 // indirect + github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.35 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.36 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.15 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.28 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.35 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.36 // indirect + github.com/aws/aws-sdk-go-v2/service/s3 v1.106.5 // indirect + github.com/aws/aws-sdk-go-v2/service/signin v1.5.4 // indirect + github.com/aws/aws-sdk-go-v2/service/sso v1.33.4 // indirect + github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4 // indirect + github.com/aws/aws-sdk-go-v2/service/sts v1.45.4 // indirect + github.com/aws/smithy-go v1.27.6 // indirect + github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/chainguard-dev/git-urls v1.0.2 // indirect + github.com/charmbracelet/colorprofile v0.4.3 // indirect + github.com/charmbracelet/ultraviolet v0.0.0-20260811164956-006e29f97886 // indirect + github.com/charmbracelet/x/ansi v0.11.8 // indirect + github.com/charmbracelet/x/term v0.2.2 // indirect + github.com/charmbracelet/x/termios v0.1.1 // indirect + github.com/charmbracelet/x/windows v0.2.2 // indirect + github.com/clipperhouse/displaywidth v0.11.0 // indirect + github.com/clipperhouse/uax29/v2 v2.7.0 // indirect + github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect + github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect + github.com/dlclark/regexp2/v2 v2.7.1 // indirect + github.com/docker/cli v29.7.2+incompatible // indirect + github.com/docker/docker-credential-helpers v0.9.3 // indirect + github.com/dominikbraun/graph v0.23.0 // indirect + github.com/dustin/go-humanize v1.0.1 // indirect + github.com/elliotchance/orderedmap/v3 v3.1.1 // indirect + github.com/envoyproxy/go-control-plane/envoy v1.39.0 // indirect + github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect + github.com/fatih/color v1.19.0 // indirect + github.com/felixge/httpsnoop v1.1.0 // indirect + github.com/fsnotify/fsnotify v1.10.1 // indirect + github.com/go-jose/go-jose/v4 v4.1.4 // indirect + github.com/go-logr/logr v1.4.4 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/go-task/slim-sprig/v3 v3.0.0 // indirect + github.com/go-task/task/v3 v3.53.1 // indirect + github.com/go-task/template v0.2.0 // indirect + github.com/google/go-containerregistry v0.22.1 // indirect + github.com/google/s2a-go v0.1.9 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.20 // indirect + github.com/googleapis/gax-go/v2 v2.23.0 // indirect + github.com/hashicorp/aws-sdk-go-base/v2 v2.0.0-beta.74 // indirect + github.com/hashicorp/go-cleanhttp v0.5.2 // indirect + github.com/hashicorp/go-getter v1.8.8 // indirect + github.com/hashicorp/go-version v1.9.0 // indirect + github.com/joho/godotenv v1.5.1 // indirect + github.com/klauspost/compress v1.20.2 // indirect + github.com/klauspost/cpuid/v2 v2.4.0 // indirect + github.com/klauspost/pgzip v1.2.6 // indirect + github.com/lucasb-eyer/go-colorful v1.4.1 // indirect + github.com/mattn/go-colorable v0.1.15 // indirect + github.com/mattn/go-isatty v0.0.24 // indirect + github.com/mattn/go-runewidth v0.0.27 // indirect + github.com/mitchellh/go-homedir v1.1.0 // indirect + github.com/mitchellh/hashstructure/v2 v2.0.2 // indirect + github.com/muesli/cancelreader v0.2.2 // indirect + github.com/opencontainers/go-digest v1.0.0 // indirect + github.com/opencontainers/image-spec v1.1.1 // indirect + github.com/pierrec/lz4/v4 v4.1.27 // indirect + github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect + github.com/puzpuzpuz/xsync/v4 v4.5.0 // indirect + github.com/rivo/uniseg v0.4.7 // indirect + github.com/sajari/fuzzy v1.0.0 // indirect + github.com/sirupsen/logrus v1.9.4 // indirect + github.com/spf13/pflag v1.0.10 // indirect + github.com/spiffe/go-spiffe/v2 v2.8.1 // indirect + github.com/spin-stack/go-tools v0.0.0-20261109064146-2b473b1bc102 // indirect + github.com/stretchr/objx v0.5.3 // indirect + github.com/stretchr/testify v1.12.1 // indirect + github.com/u-root/u-root v0.16.0 // indirect + github.com/u-root/uio v0.0.0-20240224005618-d2acac8f3701 // indirect + github.com/ulikunitz/xz v0.5.16 // indirect + github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect + github.com/zeebo/xxh3 v1.1.0 // indirect + go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/contrib/detectors/gcp v1.45.0 // indirect + go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 // indirect + go.opentelemetry.io/otel v1.45.0 // indirect + go.opentelemetry.io/otel/metric v1.45.0 // indirect + go.opentelemetry.io/otel/sdk v1.45.0 // indirect + go.opentelemetry.io/otel/sdk/metric v1.45.0 // indirect + go.opentelemetry.io/otel/trace v1.45.0 // indirect + go.yaml.in/yaml/v3 v3.0.6 // indirect + golang.org/x/crypto v0.58.0 // indirect + golang.org/x/net v0.61.0 // indirect + golang.org/x/oauth2 v0.36.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/term v0.47.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/time v0.15.0 // indirect + google.golang.org/api v0.293.0 // indirect + google.golang.org/genproto v0.0.0-20260724162435-b2f20204f0df // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260724162435-b2f20204f0df // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260807164820-c8921c73eeea // indirect + google.golang.org/grpc v1.83.0 // indirect + google.golang.org/protobuf v1.36.12 // indirect + mvdan.cc/sh/moreinterp v0.0.0-20260817215856-d6550df7ed8d // indirect + mvdan.cc/sh/v3 v3.13.2-0.20260817215856-d6550df7ed8d // indirect +) + +tool ( + github.com/go-task/task/v3/cmd/task + github.com/spin-stack/go-tools/cmd/sfw + github.com/spin-stack/go-tools/cmd/taskgate +) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..3c457a2 --- /dev/null +++ b/go.sum @@ -0,0 +1,324 @@ +cel.dev/expr v0.25.2 h1:K6j46C81hXtZQfuX60cVWQFBJahKSE2gfRbNuvr5bFs= +cel.dev/expr v0.25.2/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4= +charm.land/bubbles/v2 v2.1.1 h1:7r55WzBxpo/R3z98hGmY7KKPd3ET6vsf0Fb9sDHOV60= +charm.land/bubbles/v2 v2.1.1/go.mod h1:GE6M31gaWZVXzGw73OeuTTgy4lX+OtkH0E5ymnNsHxo= +charm.land/bubbletea/v2 v2.0.8 h1:SxTJMhCAI3lbPmy4SgX5LWZ24AdINr4I6UEqzZvYJuY= +charm.land/bubbletea/v2 v2.0.8/go.mod h1:2SkdgoTXluXJHOUwAoRlRXF/28vklb1rFl6GcgV1/ss= +charm.land/lipgloss/v2 v2.0.6 h1:EaGKeuA8FvF+v2BT5VmZd2LoYLaMZJXA5n34th8nCIQ= +charm.land/lipgloss/v2 v2.0.6/go.mod h1:ipDDJNSGa1hlwDtSfW1s2/xR8Vdhbut4PXh2zEKZd0Q= +cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE= +cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU= +cloud.google.com/go/auth v0.23.1 h1:1tPpBPG02lQHmoiAvs9egyCASqXP0xgobptjZzov/Jg= +cloud.google.com/go/auth v0.23.1/go.mod h1:4DhBRcqvtljQN3dJ57qtqbib5ZGCYE5f2crfiiC2EM0= +cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc= +cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c= +cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs= +cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10= +cloud.google.com/go/iam v1.13.0 h1:ufT3FPT5rFFXu6UtLkNoxaOaV5EuA1dsSkmemCSTo6U= +cloud.google.com/go/iam v1.13.0/go.mod h1:gHXdDEiPDvqd1q1KwBDGQlgZY/BwY760zU2LhOZS5w0= +cloud.google.com/go/logging v1.19.0 h1:NCqhdVUg3wQ8Cobdf16FDSuTGi3+6+hdSBHrY5TsR6Q= +cloud.google.com/go/logging v1.19.0/go.mod h1:i40NZCHC9Gqvod4yE+yQfDWwlgwW/SrshkkGibCHxcA= +cloud.google.com/go/longrunning v1.2.0 h1:WjYH3YHBGCxGJP9M4dWGHBfXr/cFIjMkNgWcJj7/iMM= +cloud.google.com/go/longrunning v1.2.0/go.mod h1:5KMQALFGOCtFoi2xSOA1u3H7WKlhmckgiyFw7+LGQp0= +cloud.google.com/go/monitoring v1.30.0 h1:r/d+JUbyKmJ8b07iznuKfzVzrIXTWxHQ3lBRm3x2LlY= +cloud.google.com/go/monitoring v1.30.0/go.mod h1:htlUR0QWVMrjFzZmN4LGnMAve9xB/eduwjmINxVZ8RM= +cloud.google.com/go/storage v1.64.0 h1:KLpxI/oX9LxeRsNqn877d2WyeT3ryiEwnGt8pwcSPZg= +cloud.google.com/go/storage v1.64.0/go.mod h1:lWyAtwvDZHdL3k68WVKbESP6bmWaV23ZJJ/JEVw/ZaQ= +cloud.google.com/go/trace v1.16.0 h1:GmQovzFc5F0CNfl0VLgL64aoTtu7xsM0YajW2GlG9+E= +cloud.google.com/go/trace v1.16.0/go.mod h1:r+bdAn16dKLSV1G2D5v3e58IlQlizfxWrUfjx7kM7X0= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 h1:bN1gA3of5bXtbnLsRPrwfmbbe7A5UWFlcTHseujLnpc= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0/go.mod h1:Yj5vHEz/aAepZGliRJsA6uvHAVAQyEwajq9ORCHPxzM= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.59.0 h1:c/Ivw7FuawPLfrr+zB0LZKeCchO2cAHQpF2qZ6OV7rQ= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.59.0/go.mod h1:Zba7lknY/d78oxbKqFTmCsaGwfpzeJ3ktrrLXtnTV6g= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.59.0 h1:xTXsqDOj5k9mK3VVWHYUryryJCIdYfXxdjKFwpzINUw= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.59.0/go.mod h1:V9g30lTKzfUsEW+gpWssck6u9IhARajmipodImLLcwI= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.59.0 h1:18FRm6ZcN/x9+ZmhMr96hLcTtlLn2/gHPuDLVeg7XcY= +github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.59.0/go.mod h1:YqwkQPrWSC7+byyc1VlKbWLBF5JsW5IoL6xUkemYSXk= +github.com/Ladicle/tabwriter v1.0.0 h1:DZQqPvMumBDwVNElso13afjYLNp0Z7pHqHnu0r4t9Dg= +github.com/Ladicle/tabwriter v1.0.0/go.mod h1:c4MdCjxQyTbGuQO/gvqJ+IA/89UEwrsD6hUCW98dyp4= +github.com/Masterminds/semver/v3 v3.5.0 h1:kQceYJfbupGfZOKZQg0kou0DgAKhzDg2NZPAwZ/2OOE= +github.com/Masterminds/semver/v3 v3.5.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= +github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0= +github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k= +github.com/alecthomas/chroma/v2 v2.27.0 h1:FodwmyOBgJULFYmDqibcp9pvfDLWdtPRh9v/r5BXYZs= +github.com/alecthomas/chroma/v2 v2.27.0/go.mod h1:NjJ3ciIgrqBNeIkWZ4e46nseoLDslxU1LmfCoL+wcY8= +github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs= +github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4= +github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4= +github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI= +github.com/aws/aws-sdk-go-v2 v1.43.4 h1:b9FTvbRwy+JCsfp2Wp6wV/KbOx3Aj7nkoFb2cRX0IhE= +github.com/aws/aws-sdk-go-v2 v1.43.4/go.mod h1:70vwSy16txshwG+g55WkpgPKDIByzHI8ccBsOteo3bQ= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.16 h1:aiuaKlDweRC5qExJondpWjOgyzMHpofpwspGXUtwn4c= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.16/go.mod h1:nG/LOlmox9BDe9HvQnXWzgcK8uKbgBMZ/Hp5pVt/21I= +github.com/aws/aws-sdk-go-v2/config v1.32.35 h1:UEzXuET8E42lxBPijuACu/tEK7v5lFPlk0Q+GT5WD9E= +github.com/aws/aws-sdk-go-v2/config v1.32.35/go.mod h1:KaMtJpFa2JlL2BStjjHQVwQpzZEmw+ND/EgVrfFoo2g= +github.com/aws/aws-sdk-go-v2/credentials v1.19.34 h1:y6GkSmcv5myd1ngrYbGmiLlwQqB6TQhOuN/tbSSuWDY= +github.com/aws/aws-sdk-go-v2/credentials v1.19.34/go.mod h1:w3dTcnDVoQIewjo7JG45hduAToikiIFLC4FIO7fndvw= +github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.35 h1:+S7kbJoLDDQ5tE+lHrUBgMkzC8NLgsaioS2F3dVoFAE= +github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.35/go.mod h1:Ak7xXviIARfFdNUJ9Etb0bdVDt/KAvKjMGJVLWXDzik= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.35 h1:kzVuGlatQtYinwBJEEyLAbggepCoavosiaHHX9+fD+c= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.35/go.mod h1:0yLx0yEI+SfqeJMPvOtIEFoZbiQYXMGszBueiutQyaI= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.35 h1:WK6CjihTuLisCjSKKbildJ79sGZZgbBz3iNa7VsKIhU= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.35/go.mod h1:KYleN57luLoe97R7vTnx8PMcVrr9gAcRECtOjl91DNg= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.36 h1:jbGY4CXLzZElOXgGsexlC3Hi+3YM0rSmk4opFXKqg/k= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.36/go.mod h1:uBu/9aKsS/UQGc72RAt3y54kjgYQxmhut8ZD2dXCDNE= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.15 h1:JJLBQxwY+AFwuPAi5ivGc1ChnTdUt4cXMv7e76m2c/Y= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.15/go.mod h1:lQknBIe78MVL0cQOQDlag8KGflMbMEVFx9mB6O8ENvk= +github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.28 h1:Q1TF1J9jVD+vFo0LzNnmNdQ9EAt52TS+MQlq9Ir+Yxo= +github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.28/go.mod h1:4KqXXC/p1hrotmouDFbrRoWaLy962b9PMUReCG6+uWo= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.35 h1:BBEElKh4a+rKshvjrfpajTe9CbpZvrbb4Jkg2PB7RzA= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.35/go.mod h1:zaZk983w//8beSruBVec/mr4CmDwgZitW/qzGhAAX0g= +github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.36 h1:EUIwBoN+q7UmhAejxgD27APiRjh1vwCFo53gSqdT0BM= +github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.36/go.mod h1:6u00gmlTGR6W0b2k9NBrld7MnOEmf1Spqx0VVt6AqyE= +github.com/aws/aws-sdk-go-v2/service/s3 v1.106.5 h1:HpN6GgZ3T8pSvRp81ZsgumNjlvRsa+9M0ZL2o6W4uLY= +github.com/aws/aws-sdk-go-v2/service/s3 v1.106.5/go.mod h1:5FTZoQxhmLEiCAtYVk6V+t0iS/B5yGZVLZ3Wq5FDJZI= +github.com/aws/aws-sdk-go-v2/service/signin v1.5.4 h1:cOJELVNrq5Q3Udry2GLuHUM7MhwpeaQRdYaoa6GI/yI= +github.com/aws/aws-sdk-go-v2/service/signin v1.5.4/go.mod h1:f4LxzKBtaTxD7xh3PiVg3CE1tchQemfmghaJr+NbK2c= +github.com/aws/aws-sdk-go-v2/service/sso v1.33.4 h1:AMW7a7S8iQaHjBYZdU3PCq4GKRPijTPRAc7e6XtEThY= +github.com/aws/aws-sdk-go-v2/service/sso v1.33.4/go.mod h1:QQNsFV1DVXoXcZt18FS8lI8rtUrlDyAuWZLQ5shunv4= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4 h1:AsbZcJAQPRmHDJG8K1N0pof/1zPWjVT8TFlTWuGLSvo= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4/go.mod h1:6imqztH0//t0mKbl6yWl7swSEl7F/w32oAmqB3vP1ag= +github.com/aws/aws-sdk-go-v2/service/sts v1.45.4 h1:w/AryDYMjSUANSQ2uoZxJovUsMTwWJNTv3IMex30Y+4= +github.com/aws/aws-sdk-go-v2/service/sts v1.45.4/go.mod h1:WeBiAa67azG7Su9Vf+ChGDBLiAozJCXzdjXiPBUwtbc= +github.com/aws/smithy-go v1.27.6 h1:0zjT8jgK3jbrTT7JJ3EE6JsMhX8JTrZ+f1sEndYDXrA= +github.com/aws/smithy-go v1.27.6/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= +github.com/aymanbagabas/go-udiff v0.4.1 h1:OEIrQ8maEeDBXQDoGCbbTTXYJMYRCRO1fnodZ12Gv5o= +github.com/aymanbagabas/go-udiff v0.4.1/go.mod h1:0L9PGwj20lrtmEMeyw4WKJ/TMyDtvAoK9bf2u/mNo3w= +github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d h1:xDfNPAt8lFiC1UJrqV3uuy861HCTo708pDMbjHHdCas= +github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d/go.mod h1:6QX/PXZ00z/TKoufEY6K/a0k6AhaJrQKdFe6OfVXsa4= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/chainguard-dev/git-urls v1.0.2 h1:pSpT7ifrpc5X55n4aTTm7FFUE+ZQHKiqpiwNkJrVcKQ= +github.com/chainguard-dev/git-urls v1.0.2/go.mod h1:rbGgj10OS7UgZlbzdUQIQpT0k/D4+An04HJY7Ol+Y/o= +github.com/charmbracelet/colorprofile v0.4.3 h1:QPa1IWkYI+AOB+fE+mg/5/4HRMZcaXex9t5KX76i20Q= +github.com/charmbracelet/colorprofile v0.4.3/go.mod h1:/zT4BhpD5aGFpqQQqw7a+VtHCzu+zrQtt1zhMt9mR4Q= +github.com/charmbracelet/ultraviolet v0.0.0-20260811164956-006e29f97886 h1:rdnVWKgJpTVXKuKuJyxDJ+NFJdUaUqGvyGy61OcvlbA= +github.com/charmbracelet/ultraviolet v0.0.0-20260811164956-006e29f97886/go.mod h1:nAw0d9PhFp1qdzi2xhQU5YOu5sVpDIHWlaW2Uz/bCro= +github.com/charmbracelet/x/ansi v0.11.8 h1:JMFwp0CgDC2+jcOB162HH5k7I3FVbgFSMMYg7dSPBQQ= +github.com/charmbracelet/x/ansi v0.11.8/go.mod h1:ZNN+3mXny/516oTQPLMPIBeSINvNJJQ8uQXDgbeJxY0= +github.com/charmbracelet/x/exp/golden v0.0.0-20250806222409-83e3a29d542f h1:pk6gmGpCE7F3FcjaOEKYriCvpmIN4+6OS/RD0vm4uIA= +github.com/charmbracelet/x/exp/golden v0.0.0-20250806222409-83e3a29d542f/go.mod h1:IfZAMTHB6XkZSeXUqriemErjAWCCzT0LwjKFYCZyw0I= +github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk= +github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI= +github.com/charmbracelet/x/termios v0.1.1 h1:o3Q2bT8eqzGnGPOYheoYS8eEleT5ZVNYNy8JawjaNZY= +github.com/charmbracelet/x/termios v0.1.1/go.mod h1:rB7fnv1TgOPOyyKRJ9o+AsTU/vK5WHJ2ivHeut/Pcwo= +github.com/charmbracelet/x/windows v0.2.2 h1:IofanmuvaxnKHuV04sC0eBy/smG6kIKrWG2/jYn2GuM= +github.com/charmbracelet/x/windows v0.2.2/go.mod h1:/8XtdKZzedat74NQFn0NGlGL4soHB0YQZrETF96h75k= +github.com/clipperhouse/displaywidth v0.11.0 h1:lBc6kY44VFw+TDx4I8opi/EtL9m20WSEFgwIwO+UVM8= +github.com/clipperhouse/displaywidth v0.11.0/go.mod h1:bkrFNkf81G8HyVqmKGxsPufD3JhNl3dSqnGhOoSD/o0= +github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= +github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= +github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik= +github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= +github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s= +github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dlclark/regexp2/v2 v2.7.1 h1:yqDtwI1ptXXvEUNpYTk2lad4jLtAcKqkzepn4savSk4= +github.com/dlclark/regexp2/v2 v2.7.1/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU= +github.com/docker/cli v29.7.2+incompatible h1:dlkwallR8XqfeVnA2ELEhdwvb4lsSwuB4IgsG8Q9cLY= +github.com/docker/cli v29.7.2+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8= +github.com/docker/docker-credential-helpers v0.9.3 h1:gAm/VtF9wgqJMoxzT3Gj5p4AqIjCBS4wrsOh9yRqcz8= +github.com/docker/docker-credential-helpers v0.9.3/go.mod h1:x+4Gbw9aGmChi3qTLZj8Dfn0TD20M/fuWy0E5+WDeCo= +github.com/dominikbraun/graph v0.23.0 h1:TdZB4pPqCLFxYhdyMFb1TBdFxp8XLcJfTTBQucVPgCo= +github.com/dominikbraun/graph v0.23.0/go.mod h1:yOjYyogZLY1LSG9E33JWZJiq5k83Qy2C6POAuiViluc= +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/elliotchance/orderedmap/v3 v3.1.1 h1:eV7lfZ5fVL8d36b8Wogqi/eqm7R/kZcftA9Yiyj+63M= +github.com/elliotchance/orderedmap/v3 v3.1.1/go.mod h1:G+Hc2RwaZvJMcS4JpGCOyViCnGeKf0bTYCGTO4uhjSo= +github.com/envoyproxy/go-control-plane v0.14.0 h1:hbG2kr4RuFj222B6+7T83thSPqLjwBIfQawTkC++2HA= +github.com/envoyproxy/go-control-plane v0.14.0/go.mod h1:NcS5X47pLl/hfqxU70yPwL9ZMkUlwlKxtAohpi2wBEU= +github.com/envoyproxy/go-control-plane/envoy v1.39.0 h1:1uwRDYPYG8BIBU9Mj1sUAebNmlM6beu/ZKKweSLDxk8= +github.com/envoyproxy/go-control-plane/envoy v1.39.0/go.mod h1:5e4ylfTZO723MEEFsCpSW4ZEBWR8mwkEyXfwJBTCZ9c= +github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 h1:/G9QYbddjL25KvtKTv3an9lx6VBE2cnb8wp1vEGNYGI= +github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4= +github.com/envoyproxy/protoc-gen-validate v1.3.3 h1:MVQghNeW+LZcmXe7SY1V36Z+WFMDjpqGAGacLe2T0ds= +github.com/envoyproxy/protoc-gen-validate v1.3.3/go.mod h1:TsndJ/ngyIdQRhMcVVGDDHINPLWB7C82oDArY51KfB0= +github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w= +github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= +github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc= +github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE= +github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= +github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= +github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA= +github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= +github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-quicktest/qt v1.102.0 h1:HSQxCeh5YZH3EL3W39ixjtyaEhcWSXQHtHnMBzSs474= +github.com/go-quicktest/qt v1.102.0/go.mod h1:p4lGIVX+8Wa6ZPNDvqcxq36XpUDLh42FLetFU7odllI= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/go-task/task/v3 v3.53.1 h1:4myl5NHvq30uSkkoWH4Fz0OapYVGRci3AiN11yJti08= +github.com/go-task/task/v3 v3.53.1/go.mod h1:Rb6QWLxLK710eldhNbispsUirhdhZwT37USDE5iyk7k= +github.com/go-task/template v0.2.0 h1:xW7ek0o65FUSTbKcSNeg2Vyf/I7wYXFgLUznptvviBE= +github.com/go-task/template v0.2.0/go.mod h1:dbdoUb6qKnHQi1y6o+IdIrs0J4o/SEhSTA6bbzZmdtc= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/go-containerregistry v0.22.1 h1:RZuuSYhTvlDvtsK+NkutoCZ//C0X2ebLK8X8l3ULs84= +github.com/google/go-containerregistry v0.22.1/go.mod h1:bJR35SK8XgisYmhg/FMQ/5RK0S/XrOAqLBV5/LR2XE0= +github.com/google/martian/v3 v3.3.3 h1:DIhPTQrbPkgs2yJYdXU/eNACCG5DVQjySNRNlflZ9Fc= +github.com/google/martian/v3 v3.3.3/go.mod h1:iEPrYcgCF7jA9OtScMFQyAlZZ4YXTKEtJ1E6RWzmBA0= +github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0= +github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.3.20 h1:t/xL64VUoN69MuMRQuJETqYGOw4Z9mSRJK9epIEtwFk= +github.com/googleapis/enterprise-certificate-proxy v0.3.20/go.mod h1:L3D/IQExI6LqEjBdXcZQ1WluSgigQmSwBboFstVPM4w= +github.com/googleapis/gax-go/v2 v2.23.0 h1:Tchl7qkvE7Ip3y+ztvNufYFvkfqTe7NfLTYGIdJRLuE= +github.com/googleapis/gax-go/v2 v2.23.0/go.mod h1:rBQKOVJCdb8IFEzg+FCwlt1LP/xMDGuqUXhUG+XMXEg= +github.com/hashicorp/aws-sdk-go-base/v2 v2.0.0-beta.74 h1:mymLUKThnV9wFvogOK8NnsMP9/vlhnjXY98gr2QIGW8= +github.com/hashicorp/aws-sdk-go-base/v2 v2.0.0-beta.74/go.mod h1:Bh9qYL8ehmDxSg14Tk8oxFCP90XHfs6NxV1D84884xA= +github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= +github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= +github.com/hashicorp/go-getter v1.8.8 h1:sRakhf+EH6s0LZLO2IgBwZ6hAFp+fZOZMNREwVELV80= +github.com/hashicorp/go-getter v1.8.8/go.mod h1:fqFlibKpwfns/s4oljLB3upJspyfFLQhS7031PlfUDc= +github.com/hashicorp/go-version v1.9.0 h1:CeOIz6k+LoN3qX9Z0tyQrPtiB1DFYRPfCIBtaXPSCnA= +github.com/hashicorp/go-version v1.9.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= +github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM= +github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg= +github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0= +github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= +github.com/klauspost/compress v1.20.2 h1:v5tiZvm1KGLyg5d5oXdXe3lsSPrbEUiKoXGia//I7Qo= +github.com/klauspost/compress v1.20.2/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI= +github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw= +github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU= +github.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU= +github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/lucasb-eyer/go-colorful v1.4.1 h1:1EO+WB73+EH8EVbzlrG3KLAfEypQWVHIBqlTf+2hNss= +github.com/lucasb-eyer/go-colorful v1.4.1/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0= +github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY= +github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= +github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI= +github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= +github.com/mattn/go-runewidth v0.0.27 h1:Feg/Oou5zI/wnpgDF6omIU0OokC9GxLC/WRknhVlIR0= +github.com/mattn/go-runewidth v0.0.27/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8= +github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= +github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4= +github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE= +github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA= +github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo= +github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= +github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= +github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk= +github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= +github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo= +github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/puzpuzpuz/xsync/v4 v4.5.0 h1:vOSWu6b57/emh+L/Cw0BeQfvxa/cogFywXHeGUxQxAg= +github.com/puzpuzpuz/xsync/v4 v4.5.0/go.mod h1:VJDmTCJMBt8igNxnkQd86r+8KUeN1quSfNKu5bLYFQo= +github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ= +github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88= +github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc= +github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs= +github.com/sajari/fuzzy v1.0.0 h1:+FmwVvJErsd0d0hAPlj4CxqxUtQY/fOoY0DwX4ykpRY= +github.com/sajari/fuzzy v1.0.0/go.mod h1:OjYR6KxoWOe9+dOlXeiCJd4dIbED4Oo8wpS89o0pwOo= +github.com/sebdah/goldie/v2 v2.8.0 h1:dZb9wR8q5++oplmEiJT+U/5KyotVD+HNGCAc5gNr8rc= +github.com/sebdah/goldie/v2 v2.8.0/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI= +github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw= +github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4= +github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w= +github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g= +github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= +github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spiffe/go-spiffe/v2 v2.8.1 h1:eXZMLsu+3MLEPJyGJkolqtVrteZfQdUpOWj6LTiDl/E= +github.com/spiffe/go-spiffe/v2 v2.8.1/go.mod h1:47Q0Q9/AqGha8QLHp+kxpH4Wca7X7EnOtlIJy3mxZ3U= +github.com/spin-stack/go-tools v0.0.0-20261109064146-2b473b1bc102 h1:xoQmNO0ya9P6U1JVBPoI9Q3ltyBurMqWAORBQl6/OH0= +github.com/spin-stack/go-tools v0.0.0-20261109064146-2b473b1bc102/go.mod h1:z8XsGI1fEXvJf5J5BOMkv9veB2rDkmRFs2FtSfluzVM= +github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4= +github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0= +github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= +github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= +github.com/u-root/u-root v0.16.0 h1:wY40O83MBVks97+Is0WlFlOPSwKQMIrWP9R1IsrExg8= +github.com/u-root/u-root v0.16.0/go.mod h1:yL/XdSSW27PdGLgUh4MNRBy54mKM+TBLzpwiB4nwj90= +github.com/u-root/uio v0.0.0-20240224005618-d2acac8f3701 h1:pyC9PaHYZFgEKFdlp3G8RaCKgVpHZnecvArXvPXcFkM= +github.com/u-root/uio v0.0.0-20240224005618-d2acac8f3701/go.mod h1:P3a5rG4X7tI17Nn3aOIAYr5HbIMukwXG0urG0WuL8OA= +github.com/ulikunitz/xz v0.5.16 h1:ld6NyySjx5lowVKwJvMRLnW5nxKX/xnpSiFYZ/Lxur0= +github.com/ulikunitz/xz v0.5.16/go.mod h1:H9Rt/W6/Qj27PGauhQc6nfCDy7vHpzsOThBSaYDoEhw= +github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no= +github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM= +github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4= +github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg= +github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ= +github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0= +github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs= +github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/contrib/detectors/gcp v1.45.0 h1:9jR0ZPRok9ryaOQ2Wx8rg5F7Aon59mxrqbVI60/vlBk= +go.opentelemetry.io/contrib/detectors/gcp v1.45.0/go.mod h1:VSme3o2fvSg5bVg0dRzyHaj4Z5EVhG+g2Fde6LKzmQA= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0 h1:oECp5f+hN7nkwjU/8BxQ/q23bGPb8FIrD839owX222E= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0/go.mod h1:DqEFwLumhzMBDQv9PcWbyoDxHI/4lAk6CM4nJBH39sc= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04= +go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU= +go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ= +go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.44.0 h1:hqxVTu/GtBF+vJ8d1fzW7fRxZFvgoDjWcxwwCaFDYpU= +go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.44.0/go.mod h1:z5fVEF4X5v0ESvlJqBrrFlBVoj5EQuefZpzsu7R+x5Q= +go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M= +go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s= +go.opentelemetry.io/otel/metric/x v0.67.0 h1:PcicCNZFkZ4bXfSooXdo3WN7RBOVOtjVdo1wD358Uns= +go.opentelemetry.io/otel/metric/x v0.67.0/go.mod h1:FBjCWZe6wgcqxcMtjdGiClDKXb2YxxXii0CXftE4QtI= +go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw= +go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA= +go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o= +go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA= +go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag= +go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc= +go.yaml.in/yaml/v3 v3.0.6 h1:zcDfvjXEqTQmpjniUBLc91xYJMB2c4QOFxEF/LCEckM= +go.yaml.in/yaml/v3 v3.0.6/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= +golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54= +golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w= +golang.org/x/exp v0.0.0-20260718201538-764159d718ef h1:LkZ48HFgy/TvhTI0bcWkjgFkgLyKUwcTbDjS0DUjw+A= +golang.org/x/exp v0.0.0-20260718201538-764159d718ef/go.mod h1:EdfpwwqSu+0Li0mzskwHU6FWDV3t9Q+RZDo3QMUtL3Q= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= +golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= +golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM= +golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= +golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/api v0.293.0 h1:p9XIWOf63U4OgYx120ZwVU8+vl4XTPmWfgVPnmOAS9w= +google.golang.org/api v0.293.0/go.mod h1:6n5tjEB1gzwniZTepZ0g5u+wM7Bof5GeULCx/zh8ZE0= +google.golang.org/genproto v0.0.0-20260724162435-b2f20204f0df h1:A7u56ilVtWR3CySHErsWAsimNwD4FoX9KWd8SUCn6vc= +google.golang.org/genproto v0.0.0-20260724162435-b2f20204f0df/go.mod h1:4RPHZcXMGptb7aRZb+M43xI7ixmjEtASEf2VtOOBZbU= +google.golang.org/genproto/googleapis/api v0.0.0-20260724162435-b2f20204f0df h1:NsJx+hCSwIBI6+C4BuJIkb8xOG1M+nfQDsqIrQHT92k= +google.golang.org/genproto/googleapis/api v0.0.0-20260724162435-b2f20204f0df/go.mod h1:1brfde68Npq6+WA75c1EHWPijZEG1kMus61ygPZfn4A= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260807164820-c8921c73eeea h1:kVhQEPTpKQahD5+JSBTfBB19wcgQTTjAIn45MBqnyHk= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260807164820-c8921c73eeea/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ= +google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ= +google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= +google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q= +gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA= +mvdan.cc/sh/moreinterp v0.0.0-20260817215856-d6550df7ed8d h1:VxKlXmio7XEU6iZ8AgMjDfm071ceDmbjpRnEcgxAqn0= +mvdan.cc/sh/moreinterp v0.0.0-20260817215856-d6550df7ed8d/go.mod h1:SR7UOjTQQhhrUu6d6lNtjAmdqmlMqr0lR/UWTxY8wJM= +mvdan.cc/sh/v3 v3.13.2-0.20260817215856-d6550df7ed8d h1:LlqUgrIy872/RmROjVtmRg3oK4RvvqZZr5SfqHSg7Ko= +mvdan.cc/sh/v3 v3.13.2-0.20260817215856-d6550df7ed8d/go.mod h1:3tr4oDy8eX1SeblVXiVLVGk0PbLzqwQqOAEdu9yKWBM= From b75b7adb67dcc866d301f7f96b6d7c597258b110 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Mon, 9 Nov 2026 18:09:42 -0300 Subject: [PATCH 2/2] runbooks: every alert rule, job and owner notice of spin, and the procedures an operator is asked for (#1) Thirty-three runbooks grounded in spin's code: one per Grafana alert rule (some covering a pair), every control plane job named in exactly one, the four owner notices, and the procedures - host shell, quarantine, recover, the database's restore and handover, key rotation, a lost runner, a lockout, egress denials, third-party terms. What spin cannot do yet is a TODO naming its issue. Co-Authored-By: Claude Opus 5.5 --- runbooks/admin-lockout.md | 110 +++++++++++++++++++ runbooks/api-errors.md | 78 +++++++++++++ runbooks/api-latency.md | 62 +++++++++++ runbooks/auth-failures-looping.md | 77 +++++++++++++ runbooks/control-plane-resources.md | 73 +++++++++++++ runbooks/database-archive-behind.md | 83 ++++++++++++++ runbooks/database-restore.md | 100 +++++++++++++++++ runbooks/database-slow.md | 75 +++++++++++++ runbooks/egress-denied.md | 103 ++++++++++++++++++ runbooks/host-not-reporting.md | 86 +++++++++++++++ runbooks/host-shell.md | 103 ++++++++++++++++++ runbooks/host-wrote-foreign-volume.md | 95 ++++++++++++++++ runbooks/identity-audit-failing.md | 87 +++++++++++++++ runbooks/job-failing.md | 127 ++++++++++++++++++++++ runbooks/kept-volumes-growing.md | 74 +++++++++++++ runbooks/key-rotation.md | 119 ++++++++++++++++++++ runbooks/lost-runner.md | 88 +++++++++++++++ runbooks/nothing-published.md | 65 +++++++++++ runbooks/object-store-requests-runaway.md | 65 +++++++++++ runbooks/quarantine-workspace.md | 79 ++++++++++++++ runbooks/rate-limited.md | 69 ++++++++++++ runbooks/runners-behind.md | 73 +++++++++++++ runbooks/start-failures-unclassified.md | 67 ++++++++++++ runbooks/supply-vulnerabilities.md | 75 +++++++++++++ runbooks/third-party-terms.md | 85 +++++++++++++++ runbooks/tls-ask-unreachable.md | 70 ++++++++++++ runbooks/volume-not-served.md | 98 +++++++++++++++++ runbooks/workspace-egress-high.md | 77 +++++++++++++ runbooks/workspace-held.md | 90 +++++++++++++++ runbooks/workspace-rehomed.md | 62 +++++++++++ runbooks/workspace-stop-backup-failed.md | 78 +++++++++++++ runbooks/workspace-suspends-failing.md | 80 ++++++++++++++ runbooks/workspace-unresponsive.md | 80 ++++++++++++++ 33 files changed, 2753 insertions(+) create mode 100644 runbooks/admin-lockout.md create mode 100644 runbooks/api-errors.md create mode 100644 runbooks/api-latency.md create mode 100644 runbooks/auth-failures-looping.md create mode 100644 runbooks/control-plane-resources.md create mode 100644 runbooks/database-archive-behind.md create mode 100644 runbooks/database-restore.md create mode 100644 runbooks/database-slow.md create mode 100644 runbooks/egress-denied.md create mode 100644 runbooks/host-not-reporting.md create mode 100644 runbooks/host-shell.md create mode 100644 runbooks/host-wrote-foreign-volume.md create mode 100644 runbooks/identity-audit-failing.md create mode 100644 runbooks/job-failing.md create mode 100644 runbooks/kept-volumes-growing.md create mode 100644 runbooks/key-rotation.md create mode 100644 runbooks/lost-runner.md create mode 100644 runbooks/nothing-published.md create mode 100644 runbooks/object-store-requests-runaway.md create mode 100644 runbooks/quarantine-workspace.md create mode 100644 runbooks/rate-limited.md create mode 100644 runbooks/runners-behind.md create mode 100644 runbooks/start-failures-unclassified.md create mode 100644 runbooks/supply-vulnerabilities.md create mode 100644 runbooks/third-party-terms.md create mode 100644 runbooks/tls-ask-unreachable.md create mode 100644 runbooks/volume-not-served.md create mode 100644 runbooks/workspace-egress-high.md create mode 100644 runbooks/workspace-held.md create mode 100644 runbooks/workspace-rehomed.md create mode 100644 runbooks/workspace-stop-backup-failed.md create mode 100644 runbooks/workspace-suspends-failing.md create mode 100644 runbooks/workspace-unresponsive.md diff --git a/runbooks/admin-lockout.md b/runbooks/admin-lockout.md new file mode 100644 index 0000000..3b7f50c --- /dev/null +++ b/runbooks/admin-lockout.md @@ -0,0 +1,110 @@ +--- +id: admin-lockout +title: No administrator can sign in +applies_to: + alerts: [] + jobs: [] + events: [] + dashboards: [] +severity: info +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +Nobody can reach the Admin pages: every administrator is disabled, demoted, deleted, or signs in +through an identity provider that is disabled or gone. Or the one administrator left lost every +passkey and no other can clear them. + +The control plane refuses any change that would cause this: the database keeps an administrator +who can sign in (an instance admin, not disabled, with a password or whose last sign-in was +through an enabled installation identity provider), and the dashboard greys out such actions on +[Admin → Users](/admin/users) and [Admin → Security](/admin/security) with the reason. A lockout +still happens from outside the API: a database restored from before, or an edit nobody reviewed. + +## Impact + +Workspaces keep running and people keep working, but nobody can administer the installation: +no runners, settings, users, jobs or approvals. There is no break-glass in the API by design: the +way back is on the control plane's machine. + +## Diagnose + +1. If somebody is still an administrator but lost their passkeys, another administrator clears + them on [Admin → Users](/admin/users) (*Clear passkeys*); they register a new one on their next + session. Stop here if that works. +2. If the installation's first administrator never chose their own password, that account still + signs in. On the control plane's machine: + + ``` + sudo spin-controlplane bootstrap-password --user + sudo spin-controlplane bootstrap-password + ``` + +3. Otherwise open the database. `controlplane database exec` is what the database's unit runs, + not a shell; the shell is the machine's psql over the database's own socket, as root: + + ``` + sudo /usr/libexec/postgresql18/psql -h /run/spin-postgres -U spin spin + ``` + + Who is an administrator, and how they sign in: + + ``` + SELECT id, email, disabled, identity_provider_id, last_login_at + FROM users WHERE is_instance_admin; + SELECT id, name, provider_type, enabled + FROM identity_providers WHERE team_id IS NULL; + ``` + +## Fix + +Change as little as gets one person back in, in one of these ways, then do the rest from the +dashboard: + +- **The installation's identity provider was disabled**: enable it again. + + ``` + UPDATE identity_providers SET enabled = true WHERE id = ''; + ``` + +- **Make an administrator of somebody who signs in through an enabled provider**: + + ``` + UPDATE users SET is_instance_admin = true WHERE email = ''; + ``` + +- **An administrator was disabled by mistake**: + + ``` + UPDATE users SET disabled = false, disabled_at = NULL, disabled_reason = NULL + WHERE email = ''; + ``` + +- **The last administrator lost every passkey**: what *Clear passkeys* does. + + ``` + DELETE FROM user_passkeys WHERE user_id = ''; + ``` + +These writes are not in the audit log. Write down what you changed, when and why, and give it to +whoever owns security for the installation. + +## Verify + +The person signs in and opens [Admin → Users](/admin/users). The database's own rule holds again: + +``` +SELECT installation_has_an_admin_way_in(NULL, NULL); +``` + +answers `t`. Then make a second administrator, so one person's slip cannot do this again. + +## Escalate + +If none of these gets anyone in, or the database refuses the update, stop. Do not delete users +or providers. Bring the two `SELECT` results (with emails removed if you post them), the +installation's release (`spin version` from any CLI that still reaches it), and open an issue at +https://github.com/spin-stack/spin/issues. If the lockout came from somebody acting on purpose, +treat it as a security incident and follow https://github.com/spin-stack/spin/security/policy. diff --git a/runbooks/api-errors.md b/runbooks/api-errors.md new file mode 100644 index 0000000..cd5b594 --- /dev/null +++ b/runbooks/api-errors.md @@ -0,0 +1,78 @@ +--- +id: api-errors +title: The API fails more than one call in twenty +applies_to: + alerts: [api-error-rate] + jobs: [] + events: [] + dashboards: [overview, spin-api-performance, spin-db-performance] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**High API Error Rate** fires when, over the last 5 minutes, more than 5% of the control plane's +API calls ended with any status other than `OK`, and it stays so for 5 minutes. The rule divides +`rpc_server_call_duration_seconds_count{rpc_response_status_code!="OK"}` by all calls; the +description gives the share. + +People see pages that fail to load, `spin` commands that print an error, editors that cannot +connect. + +## Impact + +Everyone who uses the installation: the dashboard, the `spin` CLI, editor extensions and agents +all go through this API. Workspaces already running keep running, but nobody can start, stop or +open one while the calls that do it fail. + +## Diagnose + +1. Open [Admin → Metrics](/admin/metrics?dashboard=spin-api-performance) (API Performance). + *Failed requests by status* says what kind of failure; *Methods that fail most* says where. + - `UNAUTHENTICATED`, `PERMISSION_DENIED`: callers turned away. Many at once is a client with a + dead credential, or a brute force: see [auth-failures-looping](auth-failures-looping.md). + - `RESOURCE_EXHAUSTED`: the rate limit refusing: see [rate-limited](rate-limited.md). + - `INTERNAL`, `UNAVAILABLE`, `DEADLINE_EXCEEDED`: the control plane's own faults. +2. Take a failing method to [Admin → Traces](/admin/traces): each failed call is kept whole, with + the database statements and calls to runners under it. +3. Read what the control plane logged: + + ``` + spin admin logs 'service_name:spin-controlplane AND severity_text:error' --since 1h + ``` + +4. Ask what the installation knows is broken: + + ``` + spin admin health + ``` + + A failed database, log store or collector shows here first. A whole service failing at once is + usually the database: [Admin → Metrics](/admin/metrics?dashboard=spin-db-performance). + +## Fix + +- **Database down or saturated**: follow [database-slow](database-slow.md). On the control + plane's machine, `systemctl status spin-postgres.service` says whether it runs. +- **The control plane itself unwell** (memory, goroutines climbing): follow + [control-plane-resources](control-plane-resources.md). +- **Runners not answering** (errors on start, stop, connect): follow + [host-not-reporting](host-not-reporting.md). +- **One client failing in a loop** (one method, `UNAUTHENTICATED`): it is not the installation; + find and stop the client ([auth-failures-looping](auth-failures-looping.md)). +- **A release just went out**: a CLI or runner of another release is refused by the control + plane. `spin admin versions` shows what runs where; people update their CLI with `spin update`. + +## Verify + +The *Error rate* stat on [Admin → Metrics](/admin/metrics?dashboard=overview) falls under 5% and +the alert resolves after its next evaluation. `spin admin health` exits 0. + +## Escalate + +If the errors are `INTERNAL` and nothing above explains them, stop and open an issue at +https://github.com/spin-stack/spin/issues with: the alert's value, the failing methods, two or +three trace ids from Admin → Traces, the matching log lines, and `spin version` and +`spin admin versions`. diff --git a/runbooks/api-latency.md b/runbooks/api-latency.md new file mode 100644 index 0000000..f7f93bd --- /dev/null +++ b/runbooks/api-latency.md @@ -0,0 +1,62 @@ +--- +id: api-latency +title: API calls are slow +applies_to: + alerts: [api-latency] + jobs: [] + events: [] + dashboards: [overview, spin-api-performance, spin-db-performance] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**High API Latency (P99)** fires when the 99th percentile of every API call's duration over the +last 5 minutes (`rpc_server_call_duration_seconds`, all methods together) is above 1 second, for 5 +minutes. + +People see pages that hang before they load, `spin` commands that take seconds, a workspace that +takes long to open. + +## Impact + +Everyone using the installation, a little; and the work that waits on the API, such as starting +and stopping workspaces, more. Nothing is lost, but slow calls that time out become errors +([api-errors](api-errors.md)). + +## Diagnose + +1. Open [Admin → Metrics](/admin/metrics?dashboard=spin-api-performance). *p95 latency of the 10 + slowest methods* names the slow ones. One method slow is that method's work; every method + slow at once is something they share, usually the database. +2. For a slow method, open its calls in [Admin → Traces](/admin/traces). The spans under a call say + where the time went: a database statement, a call to a runner, the object store. +3. Check the database: [Admin → Metrics](/admin/metrics?dashboard=spin-db-performance). A pool near + its maximum, or *Waiting for a connection* climbing, is [database-slow](database-slow.md). +4. Check the control plane's own process: *Memory each service's Go runtime uses* on + [Admin → Metrics](/admin/metrics?dashboard=overview). A process near its limits is + [control-plane-resources](control-plane-resources.md). + +## Fix + +- **The database**: follow [database-slow](database-slow.md). +- **Calls to runners** (a start, a stop, a connect) waiting on a host: follow + [host-not-reporting](host-not-reporting.md) for the host the trace names. +- **The object store** answering slowly: the storage dashboard's *How long the store takes to + answer, by request* ([Admin → Metrics](/admin/metrics?dashboard=storage)); the store's own + status page is the cloud's. +- **One method slow by design** (a long listing, a search over a month of logs): nothing to fix + in the installation; the alert resolves when those calls stop. + +## Verify + +*API p99 latency* on [Admin → Metrics](/admin/metrics?dashboard=overview) is back under one second, +and the alert resolves. + +## Escalate + +If one method is slow every time and its trace shows the time inside the control plane (not the +database, a runner or the store), open an issue at https://github.com/spin-stack/spin/issues with +the method, two or three trace ids from Admin → Traces, and `spin version`. diff --git a/runbooks/auth-failures-looping.md b/runbooks/auth-failures-looping.md new file mode 100644 index 0000000..1b78ba5 --- /dev/null +++ b/runbooks/auth-failures-looping.md @@ -0,0 +1,77 @@ +--- +id: auth-failures-looping +title: A credential is refused again and again +applies_to: + alerts: [auth-failures-looping] + jobs: [] + events: [] + dashboards: [fleet, spin-api-performance] +severity: warning +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**Credentials Refused In A Loop** fires when the control plane refuses presented credentials more +than once every ten seconds, averaged over 5 minutes, for one kind (`auth_failures_total`, by +`kind`), and keeps doing so for 15 minutes. The `kind` label says why: + +| `kind` | Means | +|---|---| +| `invalid_credential` | No session, API key, runner token or proxy token has this value: expired, revoked, or guessed. | +| `user_disabled` | The credential is real and its user is disabled. | +| `scope_denied` | An API key used for a call its scopes do not cover. | +| `lookup_failed` | The database could not say either way. | + +## Impact + +A person gives up after a few tries; something that keeps trying for 15 minutes is a program: a +runner revoked or replaced whose machine still runs, a script with a revoked key, a CLI of a +disabled user, or someone guessing. The first ones only waste requests. Guessing is an attack. + +## Diagnose + +1. Which kind, and since when: *Refused credentials and rate-limited requests* on + [Admin → Metrics](/admin/metrics?dashboard=fleet). +2. `lookup_failed`: not a credential problem; the database is failing. Go to + [database-slow](database-slow.md). +3. Find the caller. Refused calls are `UNAUTHENTICATED` or `PERMISSION_DENIED` on + [Admin → Metrics](/admin/metrics?dashboard=spin-api-performance), *Methods that fail most*; the + method says what is calling: + - `spin.agent.v1.*` methods (a heartbeat, a volume report): a runner. Its log, or the control + plane's, names the host: + + ``` + spin admin logs 'body.message:"runner token"' --since 1h + ``` + + Compare with `spin admin runner list`: a revoked host still trying is this. + - Calls a CLI or a script makes: an API key or a session. [Admin → Audit](/admin/audit) shows + recent key revocations and users disabled. +4. Rate limiting at the same time on `auth` ([rate-limited](rate-limited.md)) with addresses you + do not know is guessing. + +## Fix + +- **A revoked runner still running**: terminate its machine; it can never connect again. See + [lost-runner](lost-runner.md). +- **A script or tool with a revoked or expired key**: its owner gives it a new key + (`spin api-key create `) or stops it. +- **A disabled user's client**: nothing to fix in the installation; tell the person, or leave it. +- **`scope_denied`**: the key's owner makes a key with the scopes the tool needs. +- **Guessing**: block the source at the network edge. Keep the `auth` rate limit on + ([Admin → Security](/admin/security)). If any guess may have succeeded, end sessions and revoke + keys of the accounts it targeted ([key-rotation](key-rotation.md)). + +## Verify + +The refusals for that kind fall under one every ten seconds on the fleet dashboard and the alert +resolves. + +## Escalate + +Guessing that succeeded, or that you cannot stop at the edge, is a security incident: report it +privately at https://github.com/spin-stack/spin/security/advisories/new with the kind, the methods, +the times and `spin version`. A refusal loop with no client to explain it goes to +https://github.com/spin-stack/spin/issues. diff --git a/runbooks/control-plane-resources.md b/runbooks/control-plane-resources.md new file mode 100644 index 0000000..278dc1f --- /dev/null +++ b/runbooks/control-plane-resources.md @@ -0,0 +1,73 @@ +--- +id: control-plane-resources +title: The control plane holds too many goroutines or too much memory +applies_to: + alerts: [high-goroutines, high-memory] + jobs: [] + events: [] + dashboards: [overview] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- **High Goroutine Count** fires when the control plane's process + (`go_goroutine_count{job="spin-controlplane"}`) runs more than 10,000 goroutines for 5 minutes. +- **High Memory Usage** fires when its Go runtime uses more than 1 GiB + (`go_memory_used_bytes{job="spin-controlplane"}`) for 10 minutes. + +Usually nobody sees anything yet. Later the API slows ([api-latency](api-latency.md)) and the +process can be killed for memory. + +## Impact + +Both are early signs. A count that keeps climbing is something started and never ended: streams +from runners or browsers that are not closed, or work that waits forever. If the machine runs out +of memory, the control plane is killed and every API call fails until it is back. + +## Diagnose + +1. Look at the shape on [Admin → Metrics](/admin/metrics?dashboard=overview), *Memory each + service's Go runtime uses*, over a day or more: + - A step that stays flat after a burst (many workspaces, many people) is load: the process + holds what the load needs. + - A line that climbs steadily without load to match is a leak. +2. Compare with what it serves: hosts connected (`spin admin runner list`), workspaces running and + people signed in. Each connected runner and each open terminal or editor holds goroutines. +3. Read what the control plane logged around when the climb started: + + ``` + spin admin logs 'service_name:spin-controlplane AND (severity_text:error OR severity_text:warn)' --since 6h + ``` + +4. For a leak, take the goroutines' stacks before anything else, on the control plane's machine. + `SIGQUIT` makes the Go runtime print every goroutine's stack to the journal and exit; systemd + starts the unit again (`Restart=on-failure`), so this is also a restart: + + ``` + systemctl kill -s QUIT spin-controlplane.service + journalctl -u spin-controlplane.service --since "5 minutes ago" > goroutines.txt + ``` + +## Fix + +- **Load**: nothing is wrong; if the machine has room, the thresholds are low for this + installation. If it does not, give the control plane's machine more memory (a larger instance + type); read [database-restore](database-restore.md) before replacing the machine. +- **A leak**: take the stacks (step 4), which restarts the process and gives the memory back. + If you cannot take them, `systemctl restart spin-controlplane.service` alone does the same. + The API is down for the seconds the process takes to start; running workspaces are not + touched. + +## Verify + +The goroutine count and memory fall on [Admin → Metrics](/admin/metrics?dashboard=overview) and the +alerts resolve. Watch for a day: a leak climbs again at the same rate. + +## Escalate + +A leak is a defect in Spinbox. Open an issue at https://github.com/spin-stack/spin/issues with the +memory panel over the climb, the goroutine stacks from step 4 (read them before you post them: the +tracker is public), and `spin version`. diff --git a/runbooks/database-archive-behind.md b/runbooks/database-archive-behind.md new file mode 100644 index 0000000..248e557 --- /dev/null +++ b/runbooks/database-archive-behind.md @@ -0,0 +1,83 @@ +--- +id: database-archive-behind +title: The control plane's database is not reaching its archive +applies_to: + alerts: [database-archive-behind] + jobs: [] + events: [] + dashboards: [spin-db-performance] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**The Database's Archive Is Behind** fires when PostgreSQL has finished WAL files it has not sent +to the archive (`database_wal_archive_pending > 0`), or any send to the archive failed in the last +15 minutes (`database_wal_archive_failures_total`), and this lasts 12 minutes: twice the archive +timeout (5 minutes) and two more, since one file is always on its way. + +A quiet database archives nothing for hours and is not behind: the age of the last archived file +alone does not fire this. + +Nobody using the installation sees anything. + +## Impact + +The archive (base backups and WAL in the bucket the installation document names under +`database.archive`) is what a replaced control plane machine restores the database from. What has +not reached it is lost if the machine dies now, and an archive that stopped is found out only at +that restore. A release that replaces the machine needs it too. + +## Diagnose + +1. Ask the installation: + + ``` + spin admin health + ``` + + The `database archive` part says whether it is behind and why. +2. *WAL waiting for the archive*, *Since the last archived WAL* and *Archive failures in the last + hour* on [Admin → Metrics](/admin/metrics?dashboard=spin-db-performance). +3. Read why the sends fail. PostgreSQL runs the archive command under the database's unit, on the + control plane's machine: + + ``` + journalctl -u spin-postgres.service --since "1 hour ago" + ``` + + The usual causes: the machine's role lost permission on the archive bucket or its key, the + bucket's endpoint is unreachable, or the disk under the WAL is full. +4. Check the daily base backup: `systemctl list-timers spin-postgres-backup.timer` and + `journalctl -u spin-postgres-backup.service`. + +## Fix + +- **Permission or key**: restore the control plane machine's access to the archive bucket and its + key (on AWS, the role's policy and the KMS key policy, both from the installation's Terraform). + PostgreSQL retries the waiting files on its own. +- **Network**: restore the route to the bucket's endpoint. +- **Disk full**: free space on the machine's data disk. PostgreSQL keeps every WAL file it has not + archived, so an archive that stays down fills the disk; that is the next outage. +- Once it catches up, take a fresh base backup so a restore does not need the long WAL tail: + + ``` + systemctl start spin-postgres-backup.service + ``` + +For restoring from the archive, or moving the database to a new machine, see +[database-restore](database-restore.md). + +## Verify + +`spin admin health` shows the archive ok, *WAL waiting for the archive* is 0 or 1 on the database +dashboard, and the alert resolves. + +## Escalate + +If the archive keeps failing with permissions and network correct, stop before replacing the +control plane's machine: a replacement now loses data. Open an issue at +https://github.com/spin-stack/spin/issues with the `spin-postgres.service` journal lines (remove +bucket names first: the tracker is public), `spin admin health`, and `spin version`. diff --git a/runbooks/database-restore.md b/runbooks/database-restore.md new file mode 100644 index 0000000..89540a7 --- /dev/null +++ b/runbooks/database-restore.md @@ -0,0 +1,100 @@ +--- +id: database-restore +title: The control plane's database - archive, restore, and handover at a release +applies_to: + alerts: [] + jobs: [] + events: [] + dashboards: [spin-db-performance] +severity: info +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +Read this when a release replaces the control plane's machine, when a new control plane machine +waits a long time before it serves, or when the database will not start. + +How it works, so the logs make sense: + +- The database is PostgreSQL on the control plane's machine (`spin-postgres.service`, which runs + `controlplane database exec`). Every WAL segment, and a base backup a day + (`spin-postgres-backup.timer`), go sealed to the installation's archive bucket + (`database.archive` in the installation's document). Seven base backups are kept: daily ones, + one before every handover and one after every restore. +- **Handover, at a release.** The new machine asks, in the handover bucket + (`database.handover`), for the database. The machine that holds it stops, takes a base backup, + archives what it wrote and lets go; the new machine lays that backup and replays the archive. + The old machine's unit exits and stays down. +- **Restore.** A machine with no database of its own, whose holder has not said it holds it for + a minute, restores the newest base backup and replays every WAL segment after it. + +## Impact + +While the new machine hands over or restores, the control plane does not start: the API and the +dashboard are down. Running workspaces keep running. A restore after a machine died loses what +that machine wrote and had not archived: at most about five minutes of writes, the archive's +timeout. + +## Diagnose + +1. From anywhere: + + ``` + spin admin health + ``` + + The *database* and *database archive* parts say whether it answers and whether WAL is waiting. + [Admin → Metrics](/admin/metrics?dashboard=spin-db-performance), section *The archive*, shows + WAL waiting, time since the last archived WAL and archive failures. +2. On the control plane's machine (its Session Manager shell, or however the installation reaches + it): + + ``` + sudo systemctl status spin-postgres.service + sudo journalctl -u spin-postgres.service --since -1h + ``` + + The log lines say what it is doing: + - `asked the machine that holds it to hand it over`, then `handed over`: a normal release. + - `did not take up the request; restoring what the archive has` or `stopped saying so before + it handed it over`: the old holder is gone; a restore follows. + - `laid the archive's newest base backup`: replaying; this can take minutes. + - `handed over without WAL the other machine could not archive`: some writes are only on the + old machine's disk. + - `the archive holds WAL and no complete base backup`: the database refuses to start empty. + +## Fix + +- **Before a release**, make sure the archive is current: `spin admin health` shows the archive + healthy, and [database-archive-behind](database-archive-behind.md) is not firing. To take a + fresh base backup now, on the control plane's machine: + + ``` + sudo systemctl start spin-postgres-backup.service + ``` + +- **A handover is slow**: the new machine waits as long as the old one keeps saying it holds the + database. Let it finish; do not stop the old machine mid-handover. +- **The old machine is gone**: nothing to do; the new machine restores from the archive on its + own. Keep both buckets and the installation's encryption key: the archive is sealed under keys + derived from it, and a machine without every version of the key cannot read it. +- **WAL left on the old machine** or **no complete base backup**: stop and escalate. Do not delete + anything in the archive or handover buckets, and do not start a database by hand. + +TODO: there is no command to restore the database to an earlier point in time or to a chosen +base backup; a restore always takes the newest ([spin#1052](https://github.com/spin-stack/spin/issues/1052)). + +## Verify + +`spin admin health` shows the database and its archive ok and exits 0, the dashboard loads, and +*Since the last archived WAL* on the database dashboard is under five minutes. + +## Escalate + +Stop at any error line above, at a restore that has not finished replaying after an hour, or +at anything that would mean deleting from the archive. Bring the `spin-postgres.service` +journal from both machines, `spin admin health` output, the release you moved from and to +(`spin admin versions` before and after), and open an issue at +https://github.com/spin-stack/spin/issues. diff --git a/runbooks/database-slow.md b/runbooks/database-slow.md new file mode 100644 index 0000000..4662bec --- /dev/null +++ b/runbooks/database-slow.md @@ -0,0 +1,75 @@ +--- +id: database-slow +title: The control plane's database is slow or its pool is full +applies_to: + alerts: [db-pool-saturation, db-query-latency] + jobs: [] + events: [] + dashboards: [spin-db-performance, overview] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- **Database Connection Pool Saturation** fires when more than 80% of the control plane's + connection pool is in use (`db_client_connections_usage{state="used"}` over + `db_client_connection_max`) for 5 minutes. +- **High Database Query Latency** fires when the 99th percentile of statement durations over 5 + minutes (`db_client_operation_duration_seconds`) is above 0.5 seconds, for 5 minutes. + +People see a slow dashboard and slow commands; [api-latency](api-latency.md) often fires with it. + +## Impact + +PostgreSQL holds the whole installation and every API call waits on it, as does the work that +starts and stops workspaces. A full pool makes every caller queue for a connection; long enough, +calls time out and fail ([api-errors](api-errors.md)). + +## Diagnose + +1. Open [Admin → Metrics](/admin/metrics?dashboard=spin-db-performance). + - *Statement latency by operation* and *Statements a second by operation*: is one kind of + statement slow, or are there simply many more? + - *The pool over time* and *Waiting for a connection*: how full, and how long callers wait. The + pool holds at most 10 connections in this release (*Pool maximum*). +2. Find the slow callers in [Admin → Traces](/admin/traces): their spans carry each statement and + how long it took. +3. On the control plane's machine, check the database itself: + + ``` + systemctl status spin-postgres.service + journalctl -u spin-postgres.service --since "1 hour ago" + ``` + +4. Check the machine: CPU, memory and disk I/O of the control plane's machine (the cloud's console, + or `top` and `iostat` on the machine). The database shares it with the control plane and its + stores. + +## Fix + +- **One statement slow** (the same operation in every slow trace): that is a defect in Spinbox, + not something to tune; escalate with the trace. +- **A burst of callers** (statements a second climbing with requests a second): find who on + [Admin → Metrics](/admin/metrics?dashboard=spin-api-performance), *The 10 busiest methods*. A + client looping on one method is stopped at the client; a limit set too high is lowered on + [Admin → Security](/admin/security) (*Rate limits and sessions*). +- **The machine is out of CPU, memory or I/O**: give the control plane's machine more, which is a + change of its instance type and a replacement of the machine. Read + [database-restore](database-restore.md) before replacing it: the database moves by handover. +- **Long-held connections** with few statements: restart the control plane, which closes its pool: + `systemctl restart spin-controlplane.service`. Brief: the API is down while it starts. + +TODO: the pool's size (10) is fixed in the binary; there is no setting to change it ([spin#1053](https://github.com/spin-stack/spin/issues/1053)) + +## Verify + +On [Admin → Metrics](/admin/metrics?dashboard=spin-db-performance), connections in use drop below +8 of 10 and statement p99 below 0.5 s. Both alerts resolve. + +## Escalate + +Open an issue at https://github.com/spin-stack/spin/issues when one statement is slow on its own, +or the pool fills with no burst of callers to explain it. Bring the dashboard's panels for the +window, two or three trace ids from Admin → Traces, and `spin version`. diff --git a/runbooks/egress-denied.md b/runbooks/egress-denied.md new file mode 100644 index 0000000..7b9ad8d --- /dev/null +++ b/runbooks/egress-denied.md @@ -0,0 +1,103 @@ +--- +id: egress-denied +title: A workspace cannot reach something +applies_to: + alerts: [] + jobs: [] + events: [] + dashboards: [fleet] +severity: info +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +Someone says their workspace cannot reach a host: a connection hangs or is reset, a name does not +resolve (`REFUSED`), `git` or a package manager fails, or an API through the egress proxy +answers with an error. Or the workspace was stopped and held in quarantine after an agent in it +tried something its guard refused. + +## Impact + +One workspace, or every workspace under one network policy. Nothing is lost; work that needs +the network waits. + +How the network decides, so the diagnosis makes sense: + +- A workspace runs under its **network policies** (tags). A workspace with any tag reaches only + what its tags allow, as a union. A create that names none runs under the installation's + `default_network_tags`, or else `offline`, which allows nothing. +- The runner enforces the policy: a name a policy does not allow is answered `REFUSED`, and an + address it does not allow is dropped. +- A **brokered** name is reached only through the runner's egress proxy, which may add a team's + secret to the request. +- Inside a sealed workspace, the **guard** refuses an agent's attempts past its bounds; the first + denial stops the workspace and holds it in quarantine. + +## Diagnose + +The owner runs these on their workspace: + +1. Which policies it runs under, and which exist: + + ``` + spin workspace get + spin workspace policies + ``` + +2. What it has reached, with the runner's own counters, the names it asked for and the + brokered requests of the last hour: + + ``` + spin workspace egress + ``` + + A destination missing here, with its lookup refused, is the policy. +3. To see each lookup and each first connection let through or refused as it happens, turn on + the watch and read it from the workspace's logs: + + ``` + spin workspace watch + spin workspace logs 'service_name:network' --since 1h + spin workspace watch --off + ``` + +An administrator also checks: + +- [Admin → Network](/admin/network): the policies the installation declares, what each allows, + and how many runners have installed each. A policy declared but not installed on the runner + is not enforced there. A team's own policies: `spin team policy list `. +- *Containment and DNS* and *Egress proxy* on [Admin → Metrics](/admin/metrics?dashboard=fleet): + egress drops, resolver outcomes, and requests the proxy refused. +- A workspace in quarantine: `spin workspace denials ` ([quarantine-workspace](quarantine-workspace.md)). + +## Fix + +- **The destination is not in any of its policies**: add a rule to the right policy on + [Admin → Network](/admin/network), or to a team's (`spin team policy set --rule ...`), + then the owner moves the workspace onto it while it runs (the tags given are the whole set): + + ``` + spin workspace retag --network-tag --network-tag + ``` + +- **It runs under `offline`** because it was created with no tag and the installation has no + `default_network_tags`: retag it, and set the default on Admin → Network. +- **A brokered request fails**: the team's secret is missing or not allowed for that host + (`spin team secret list `; `spin team secret set` names the hosts it may be sent to). +- **It was held after a denial**: follow [quarantine-workspace](quarantine-workspace.md). +- **It reaches far more than expected**: see [workspace-egress-high](workspace-egress-high.md). + +## Verify + +`spin workspace egress ` lists the destination with packets both ways, and the +owner's command works. Turn the watch off when done. + +## Escalate + +If the destination is in an installed policy and is still dropped, or the proxy refuses a request +its rule allows, stop and open an issue at https://github.com/spin-stack/spin/issues with the +policy (rules only, no secrets), the `spin workspace egress` output, the watch's log lines, the +runner id from `spin workspace get` and `spin admin versions`. A workspace reaching what its +policy refuses is a security problem: follow https://github.com/spin-stack/spin/security/policy. diff --git a/runbooks/host-not-reporting.md b/runbooks/host-not-reporting.md new file mode 100644 index 0000000..faf5883 --- /dev/null +++ b/runbooks/host-not-reporting.md @@ -0,0 +1,86 @@ +--- +id: host-not-reporting +title: A host stopped reporting +applies_to: + alerts: [hosts-disconnected, host-heartbeat-stale] + jobs: [] + events: [] + dashboards: [fleet] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- **Active Hosts Are Not Connected** fires when the database counts more hosts `active` than are + holding a connection to the control plane (`hosts{state="active"}` minus `hosts_connected`), + for 5 minutes: a partition, or a host that died without being marked. +- **A Host Stopped Reporting** fires when a connected host (`host`) has sent no heartbeat for over + a minute (`host_heartbeat_age_seconds > 60`), for 2 minutes. + +Owners of workspaces on that host see them stop answering, or fail to start or stop. Their owners +may be told their workspace is unresponsive ([workspace-unresponsive](workspace-unresponsive.md)). + +## Impact + +Placement skips a host that is not connected, so the fleet has less room. Workspaces on it cannot +be started, stopped or reached through the control plane. What they wrote and had not published is +on that host alone. Believe nothing else the host reports while its heartbeat is old: its capacity +and its volumes' ages are what it last said. + +## Diagnose + +1. See the host as the control plane does: + + ``` + spin admin runner list + spin admin health + ``` + + A host that is not connected shows `(not connected: , ago)`; the reason is + why its stream ended. The same on [Admin → Runners](/admin/runners), with the host's own page. +2. *Per host: heartbeat age* and *Hosts: in the database, and connected* on + [Admin → Metrics](/admin/metrics?dashboard=fleet). +3. Read what it logged before it went quiet: + + ``` + spin admin logs 'service_name:spin-runner AND severity_text:error' --since 1h + ``` + +4. Look at the machine itself in the cloud's console: running, stopped, terminated, failing its + status checks. On AWS, an autoscaling group may have replaced it. +5. If the machine runs, on it: + + ``` + systemctl status spin-runner.service + journalctl -u spin-runner.service --since "1 hour ago" + ``` + + Runners have no SSH; on AWS they are reached through Session Manager. + +## Fix + +- **The runner process stopped**: `systemctl restart spin-runner.service` on the machine. A runner + that restarts comes back as the same host and its workspaces are waited for, nothing lost. +- **The network between them** (the machine runs, the runner logs it cannot reach the control + plane): fix the route, the security group or the DNS name. The host reconnects on its own. +- **The runner was revoked** (its log says its token is invalid): it will never connect again. + Replace the machine ([lost-runner](lost-runner.md)). +- **The machine is gone for good**: follow [lost-runner](lost-runner.md). Its workspaces move to + another host either by their owners (`spin workspace recover `) or by the control + plane on its own, past the installation's rehome threshold + ([workspace-rehomed](workspace-rehomed.md)). +- **Cannot tell yet**: stop placing on it until you know: `spin admin runner drain `. + +## Verify + +`spin admin runner list` shows the host connected, or the replacement connected and the old one +gone. *Per host: heartbeat age* drops under a minute and both alerts resolve. + +## Escalate + +If the machine and its network are healthy and the runner still does not connect, or disconnects +again and again, open an issue at https://github.com/spin-stack/spin/issues with the host id, the +stream-end reason from `spin admin runner list`, the runner's journal, the control plane's log +lines that name the host, and `spin version` and `spin admin versions`. diff --git a/runbooks/host-shell.md b/runbooks/host-shell.md new file mode 100644 index 0000000..83b45db --- /dev/null +++ b/runbooks/host-shell.md @@ -0,0 +1,103 @@ +--- +id: host-shell +title: Get a root shell on a runner +applies_to: + alerts: [] + jobs: [] + events: [] + dashboards: [] +severity: info +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +A runner misbehaves in a way nothing from outside explains (the metrics, `spin admin logs`, the +runner's own checks on its page under [Admin → Runners](/admin/runners)), and someone needs root +on it. Ask for a root shell; never get onto a runner another way. + +## Impact + +A root shell is for a while, with a reason, and costs the runner: + +- The runner is emptied first: each workspace is suspended and resumed on another runner. With + `--incident` it is not: the shell opens on the runner as it is, its workspaces still running. +- Every session is recorded by the control plane and kept in the installation's logs bucket, + under Object Lock. +- Once the shell ends (its minutes are up, or it is closed) the runner is **retired**: revoked, + every disk it held the key of marked to be re-keyed, and its machine given back to its group, + which replaces it. A runner someone had root on runs no workspace again. + +## Diagnose + +Before asking, know which runner and why: + +``` +spin admin runner list +spin admin shell list +``` + +`spin admin shell list ` shows that runner's shells: one may be asked for already. + +## Fix + +1. Ask, and open a session when it opens. The reason is shown to whoever approves and is kept: + + ``` + spin admin shell open --reason "" --minutes 20 + ``` + + For a runner whose running workspaces are part of the problem: + + ``` + spin admin shell open --reason "" --incident + ``` + + `--minutes` is at most the installation's `host_shell_max_minutes` (30 by default). The same + is offered on the runner's page, under *Root shells*. +2. Where the installation needs a second administrator (`host_shell_needs_approval`), another + administrator approves it, by its id from `spin admin shell list`: + + ``` + spin admin shell approve + ``` + + Only then is the runner touched. To refuse or withdraw one not yet approved: + + ``` + spin admin shell close + ``` + +3. Running `spin admin shell open ` again opens another session on the same open + shell. End the shell as soon as you are done; the runner is retired either way: + + ``` + spin admin shell close + ``` + +4. Read what was done, each session in turn, as asciicast v2: + + ``` + spin admin shell recording > shell.cast + asciinema play shell.cast + ``` + +TODO: `host_shell_needs_approval` and `host_shell_max_minutes` cannot be set from Admin → +Settings, the CLI or the configuration file; only the admin API takes them ([spin#1048](https://github.com/spin-stack/spin/issues/1048)). + +## Verify + +`spin admin shell list` shows the shell `ended`, with RETIRED `true`. The runner is gone from +[Admin → Runners](/admin/runners) or revoked, and its group has started a replacement. +`spin admin volume rekeys` lists the disks it held while they are re-keyed +([key-rotation](key-rotation.md)). [Admin → Audit](/admin/audit) has `host_shell_requested`, +`host_shell_approved`, `host_shell_session_opened` and `host_shell_session_ended`. + +## Escalate + +If a shell will not open after approval (stuck emptying), or the runner is not retired after it +ends, stop and open an issue at https://github.com/spin-stack/spin/issues with the shell id, the +runner id, `spin admin shell list` output and `spin admin versions`. If the shell found signs of +compromise, follow https://github.com/spin-stack/spin/security/policy and do not post it in a +public issue. diff --git a/runbooks/host-wrote-foreign-volume.md b/runbooks/host-wrote-foreign-volume.md new file mode 100644 index 0000000..172ee79 --- /dev/null +++ b/runbooks/host-wrote-foreign-volume.md @@ -0,0 +1,95 @@ +--- +id: host-wrote-foreign-volume +title: A host wrote a volume that is not its +applies_to: + alerts: [host-writing-volume-not-its] + jobs: [] + events: [] + dashboards: [storage] +severity: critical +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**A Host Wrote A Volume That Is Not Its** fires as soon as a restore passed over a signed HEAD +whose writer did not hold the volume (`unadmitted_heads_total`, by `writer_host`, any in 15 +minutes). The `writer_host` label names the host that wrote it. + +Only a host writing a volume it no longer holds makes one: a host that lost the volume, or was +replaced, and writes on with the credential it kept. + +## Impact + +Two hosts writing one disk is what the storage design is arranged against. The restore ignored +the foreign HEAD, so the workspace's data is what its rightful writer published. But the named +host still has a credential for the bucket and the key of every disk it held, and is using them. +Treat the host as taken until proven otherwise. + +## Diagnose + +1. Find the host in `spin admin runner list`: its id is the alert's `writer_host`. Note its state; + a host that should be gone (revoked, removed, its machine terminated) and still writing is the + worst case. +2. See what it wrote: *Refused writers: compare-and-set failures* on + [Admin → Metrics](/admin/metrics?dashboard=storage), and the runners' refusals: + + ``` + spin admin logs 'service_name:spin-runner AND body.message:"will not serve a volume"' --since 24h + ``` + + A `VOLUME_REFUSAL_PUBLISH_FENCED` refusal is the rightful writer finding another host published + over it. +3. In the cloud, find the machine that answers to that host: on AWS, the instance whose role made + the requests, in CloudTrail's record of the bucket. + +## Fix + +1. Take the host out for good. + - Registered with a token: revoke it. Its credentials are deleted, its stream cut, and every + team's disk whose key it was given is marked to be re-keyed: + + ``` + spin admin runner revoke + ``` + + If it still holds workspaces' disks the revoke is refused and names them. Move them first, + or pass `--leave-workspaces`; their owners then recover them + ([workspace-rehomed](workspace-rehomed.md)). + - A machine of an autoscaling group: revoke is refused. Give it back to its group, which + terminates it: + + ``` + spin admin runner remove + ``` + + This does not mark its disks for re-keying; re-key them by hand (step 3). +2. Make sure the machine is terminated in the cloud, and do not reuse its disk. +3. Re-key what it held. The `workspace-rekeys` job copies each marked disk under a new key once + its workspace is stopped or quarantined. See what is marked and how far along: + + ``` + spin admin volume rekeys + ``` + + Re-key a disk that is not marked (its workspace stopped or quarantined first): + + ``` + spin admin volume rekey + ``` + + A re-key deletes the old disk and every snapshot of it. More in [key-rotation](key-rotation.md). + +TODO: removing an autoscaling group's runner believed taken does not mark the disks it held for re-keying, and nothing lists them ([spin#1047](https://github.com/spin-stack/spin/issues/1047)) + +## Verify + +`spin admin runner list` shows the host revoked or gone. No new `unadmitted_heads_total` in 15 +minutes, and the alert resolves. `spin admin volume rekeys` ends with nothing left marked. + +## Escalate + +Always. Report it privately at https://github.com/spin-stack/spin/security/advisories/new with +the host id, when it was revoked or removed if it was, the runners' log lines, the cloud's record +of the requests, and `spin version`. Do not open a public issue. diff --git a/runbooks/identity-audit-failing.md b/runbooks/identity-audit-failing.md new file mode 100644 index 0000000..e3a530f --- /dev/null +++ b/runbooks/identity-audit-failing.md @@ -0,0 +1,87 @@ +--- +id: identity-audit-failing +title: The installation's identity key was opened more than granted, or its audit fails +applies_to: + alerts: [] + jobs: [identity-audit] + events: [] + dashboards: [] +severity: critical +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +The `identity-audit` job failed on [Admin → Jobs](/admin/jobs?job=identity-audit), and the generic +job alerts fire for it ([job-failing](job-failing.md)). Its error says which of two things: + +- **`the identity key was opened where no grant answers for it`**: in some hour, the cloud's key + service opened the identity key more times than the control plane granted it to hosts. The + error gives how many, of how many, and the hour. The control plane also logs `The identity key + was opened more often than the installation granted it`. +- **Anything else**: the audit could not read the key service's log or the grant records, and + nothing was checked for that hour. + +The job runs every hour and checks the hour that ended 30 minutes before, since the key service's +log arrives late. It exists only where the identity key is kept by a cloud's key service (on AWS, +KMS, with CloudTrail as its log). + +## Impact + +The identity key signs the tokens a workspace presents to clouds, vaults and clusters outside +Spinbox, which let it in by what it is. Each host that needs it is granted it by the control plane, +and every grant is recorded in the bucket where it cannot be taken back. + +- **Opened more than granted**: someone opened the key outside Spinbox's grants, or used a grant + twice. Whoever holds it can sign tokens as any workspace, and every relying party that trusts + this installation trusts them. Treat it as a compromise. +- **The audit fails**: a use of the key nobody granted goes unnoticed for as long as it fails. + +## Diagnose + +1. Read the error and the counts of its last runs (`openings`, `grants`, `unrecorded`) on + [Admin → Jobs](/admin/jobs?job=identity-audit). +2. **Opened more than granted**: in the cloud's key service log (CloudTrail on AWS), list the + `Decrypt` calls on the identity key in that hour, and who made each: which role, from which + machine. Compare with the hosts that joined or restarted then (`spin admin runner list`). + The grant records are under `identity/granted/` in the installation's bucket, by day. +3. **The audit fails**: the error names what it could not read: the key service's log (the control + plane's permission to read CloudTrail) or the bucket (`identity/granted/`). + +## Fix + +- **Opened more than granted**: + 1. Find and stop who opened it: remove the permission that let them (the key policy, the role + they assumed), and terminate a machine you do not know. + 2. If a runner did it, revoke it (`spin admin runner revoke `); on an autoscaling + group, `spin admin runner remove `. See + [host-wrote-foreign-volume](host-wrote-foreign-volume.md) for what else a taken host held. + 3. Tell the relying parties that trust this installation's identity tokens to stop trusting them + until the key is replaced. + + TODO: the identity key cannot be rotated; it is made once and every relying party trusts its one public key ([spin#1050](https://github.com/spin-stack/spin/issues/1050)) +- **The audit fails**: restore the control plane's permission to read the key service's log and + the bucket, then run it again: + + ``` + spin admin jobs run identity-audit + ``` + + Each run checks only its own hour; hours missed while it failed are not checked again by the job. + Check them by hand: the `Decrypt` calls in the key service's log against the records under + `identity/granted/` for the same hours. + + TODO: the audit does not go back over hours it missed while failing ([spin#1051](https://github.com/spin-stack/spin/issues/1051)) + +## Verify + +The next run on [Admin → Jobs](/admin/jobs?job=identity-audit) succeeds, with `openings` no more +than `grants`, and the job alerts resolve. + +## Escalate + +An opening nobody granted is always reported, privately, at +https://github.com/spin-stack/spin/security/advisories/new with the job's error, the key service's +log of the hour, and `spin version`. Do not open a public issue. An audit that fails with +permissions correct goes to https://github.com/spin-stack/spin/issues. diff --git a/runbooks/job-failing.md b/runbooks/job-failing.md new file mode 100644 index 0000000..bcce0dd --- /dev/null +++ b/runbooks/job-failing.md @@ -0,0 +1,127 @@ +--- +id: job-failing +title: A control plane job keeps failing, or has not succeeded in two days +applies_to: + alerts: [job-runs-failing, job-not-succeeding] + jobs: [sessions, expired-data, workspace-trash, workspace-grants, prebuilds, forge-revocations, forwards, stale-workspaces, notifications, shares, usage, collect-releases, storage-gc, bill] + events: [] + dashboards: [fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- **A Job Keeps Failing** fires when one job (`job`) failed more than 3 runs in the last hour + (`job_run_duration_seconds_count{outcome="failed"}`). +- **A Job Has Not Succeeded In Two Days** fires when a job's last success is more than 48 hours + old (`job_last_success_timestamp_seconds`), for 15 minutes. That is a job failing every run, or + one that stopped running at all, which a failure count cannot show. + +Some jobs have their own runbook: `supply-vulnerabilities` +([supply-vulnerabilities](supply-vulnerabilities.md)), `identity-audit` +([identity-audit-failing](identity-audit-failing.md)), `silent-supervisors` +([workspace-unresponsive](workspace-unresponsive.md)), `forget-gone-hosts` +([lost-runner](lost-runner.md)), `workspace-rekeys` ([key-rotation](key-rotation.md)), +`vscode-servers` and `agent-clis` ([third-party-terms](third-party-terms.md)). The rest are below. + +## Impact + +A job repairs or cleans something on a schedule. While it fails, that work stays undone; each job +below says what. Nothing breaks at once, but it piles up. + +## Diagnose + +1. Open the job on [Admin → Jobs](/admin/jobs): its last runs, each failure's error, what its last + good run did, and when it runs next. Or: + + ``` + spin admin jobs list + ``` + +2. Read the error. Most failures are one of three things the job reaches: the database, the + object store, or something outside (a forge, an email server, the cloud's bill). +3. A job not run at all (no recent runs, `NEXT` in the past): the control plane is not running its + jobs. `spin admin health` names every job whose last run failed, with its error, but not one + that stopped running. +4. The fleet dashboard has *What the jobs did, and the runs that failed* and *Since each job last + succeeded*: [Admin → Metrics](/admin/metrics?dashboard=fleet). + +### The jobs + +- **`sessions`**: deletes expired sign-in sessions. While it fails they stay in the database; an + expired one signs nobody in. Check the database. +- **`expired-data`**: deletes expired API keys, sign-in, device and registration codes, forge + states, and audit entries past their retention. While it fails they pile up; an expired one + admits nobody. Check the database. +- **`workspace-trash`**: erases workspaces whose days in the trash are over, with their disks, and + gives back what they held of their runners. While it fails, deleted workspaces keep their disks + and their runners' room ([kept-volumes-growing](kept-volumes-growing.md)). Check the runners + holding those disks and the object store. +- **`workspace-grants`**: gives back the cores and memory a workspace was granted for a time, once + the time is up. While it fails, workspaces keep them and their runners offer less. Check the + database and the runners. +- **`prebuilds`**: watches each project's prebuild refs on its forge, builds when one moves, and + gives back old prebuilds. While it fails, new workspaces start from an older prebuild. Check the + forge's connection on [Admin → Integrations](/admin/integrations) and the project's prebuilds + (`spin project prebuilds `). +- **`forge-revocations`**: revokes at the forge the tokens people and apps gave up, and finishes + deleting an app. While it fails, a token given up still works at the forge. Check that the forge + is reachable and the app's credentials on [Admin → Integrations](/admin/integrations). This one + matters for security: escalate if it fails for more than a day. +- **`forwards`**: asks again whether each open tunnel into a workspace is still allowed, and has its + runner close those that are not. While it fails, a tunnel outlives a sign-out, a revoked key or + a disabled account. Check the runners; this one matters for security too. +- **`stale-workspaces`**: fails workspaces stuck creating, starting or stopping past the deadline. + While it fails, such a workspace stays stuck and its owner cannot leave it. Check the database. +- **`notifications`**: sends each person what happened to their workspaces, through the channels + the installation has. While it fails nobody is told; what is due waits for the next run. Check + the email settings on [Admin → Integrations](/admin/integrations). +- **`shares`**: forgets a snapshot's share outside its team once nothing may clone from it, which + destroys the share's key. While it fails, keys of ended shares are kept. Check the object store + and the key service. +- **`usage`**: records what each workspace held of its runner since the last sample, which the + bill is shared out by. While it fails, that time is counted to nobody. Check the database. +- **`collect-releases`**: removes extension releases nothing runs and that are not their + extension's newest. While it fails they stay in the store and are paid for. Check the object + store; `spin admin storage list` shows what is kept. +- **`storage-gc`**: deletes objects in the bucket nothing reaches any more, and reads what the + bucket holds. While it fails they stay, the bucket costs what they take, and + [Admin → Storage](/admin/storage) shows an older reading. Check the object store's permissions. +- **`bill`**: reads what the cloud bills the installation, a few times a day. While it fails, + [Admin → Costs](/admin/costs) shows the last reading. Check the control plane's permission to + read the bill and the installation's `bill_scope`. + +## Fix + +1. Fix what the error names: the database ([database-slow](database-slow.md)), the object store, + the forge or the email server. +2. Run the job again without waiting for its schedule, from its page on + [Admin → Jobs](/admin/jobs) (**Run now**), or: + + ``` + spin admin jobs run + ``` + + It is not waited for; `spin admin jobs list` says how it went. A job already running is + refused. +3. A job that does not run at all: restart the control plane on its machine, + `systemctl restart spin-controlplane.service`. + +## Verify + +The job's next run on [Admin → Jobs](/admin/jobs) succeeds. **A Job Keeps Failing** resolves once +the last hour has no more than 3 failures; **A Job Has Not Succeeded In Two Days** resolves with +the first success. + +## Escalate + +If the error is not about something the job reaches (a panic, a database error that names a +column, a failure that repeats with everything else healthy), it is a defect. Open an issue at +https://github.com/spin-stack/spin/issues with the job's name, the failed run's error from Admin → +Jobs, the control plane's log lines around it, and `spin version`: + +``` +spin admin logs 'service_name:spin-controlplane AND severity_text:error' --since 24h +``` diff --git a/runbooks/kept-volumes-growing.md b/runbooks/kept-volumes-growing.md new file mode 100644 index 0000000..17eb6e3 --- /dev/null +++ b/runbooks/kept-volumes-growing.md @@ -0,0 +1,74 @@ +--- +id: kept-volumes-growing +title: A host keeps disk for more and more volumes +applies_to: + alerts: [kept-volumes-growing] + jobs: [] + events: [] + dashboards: [storage] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**A Host Keeps Disk For More And More Volumes** fires when a host (`instance`) keeps local disk +for more than 20 volumes more than it did 24 hours ago (`kept_volumes` now minus 24 hours ago), +for an hour. + +A kept volume is one the host no longer serves but still holds on its disk: a released volume +often comes back, so the host keeps it until the control plane says it was deleted, or that +another host holds it now. + +## Impact + +Each kept volume is a workspace's disk on that host. Growing without end, they fill the host's +disk and leave less room to place new workspaces. Nothing is lost. + +## Diagnose + +1. Open *Kept, not served: released volumes a host still holds, and the ones it freed* on + [Admin → Metrics](/admin/metrics?dashboard=storage). It shows the kept count per host and the + ones freed, by why (`deleted`, `moved`). + - **Nothing freed for `deleted`** while workspaces are being deleted: deletions are not + reaching the host. + - **Freed as usual, kept still climbing**: released volumes nobody takes back - stopped + workspaces whose disk stays on this host. +2. Check that deletions finish: the deleted workspaces wait out the installation's trash days, + then the `workspace-trash` job erases them. A host frees a deleted volume only once it is gone + both from the database and from the object store, and only when the store answers. On + [Admin → Jobs](/admin/jobs?job=workspace-trash), is the job succeeding? +3. Check that the host asks: the runner logs when it cannot free a volume. + + ``` + spin admin logs 'service_name:spin-runner AND body.message:"could not free the disk"' --since 24h + spin admin logs 'service_name:spin-runner AND body.message:"volumes this host keeps"' --since 24h + ``` + +4. Check how full the host's disk is: *Disk promised against disk taken, per host* on the same + dashboard. + +## Fix + +- **`workspace-trash` failing**: follow [job-failing](job-failing.md); the kept disk is freed at + the host's next report after the trash is erased. +- **The runner cannot free the disk** (log lines from step 3): the error says why; usually the + host's own disk or file system. Look at it on a [host-shell](host-shell.md). +- **Stopped workspaces piling up on this host**: that is what keeping is for, and they free + themselves when deleted or moved. If the host's disk is running out, stop placing on it with + `spin admin runner drain `, and ask the owners of workspaces they no longer use to + delete them. +- **Workspaces in the trash hold it**: the trash keeps their disks for the installation's + trash days ([Admin → Settings](/admin/settings)); fewer days frees them sooner. + +## Verify + +The kept count on the storage dashboard stops climbing, and freed counts move again. The alert +resolves when the host keeps no more than 20 volumes over the day before. + +## Escalate + +If deleted workspaces are erased (the job succeeds) and the host still frees none for `deleted`, +it is a defect: open an issue at https://github.com/spin-stack/spin/issues with the host id, the +panel over two days, the runner's log lines and `spin version`. diff --git a/runbooks/key-rotation.md b/runbooks/key-rotation.md new file mode 100644 index 0000000..acaecd6 --- /dev/null +++ b/runbooks/key-rotation.md @@ -0,0 +1,119 @@ +--- +id: key-rotation +title: Rotate or revoke an installation's keys and credentials +applies_to: + alerts: [] + jobs: [workspace-rekeys] + events: [] + dashboards: [] +severity: info +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +A key or credential is suspected taken (a leaked token, a runner believed compromised, a person +who left), or is due for rotation. Also read this when the **workspace-rekeys** job fails on +[Admin → Jobs](/admin/jobs). + +## Impact + +A taken credential acts as whoever it belongs to until it is revoked. A runner's keys open the +disks it held until those disks are re-keyed. While **workspace-rekeys** fails, the revoked +runner's keys still open those disks, and their workspaces wait to start. + +## Diagnose + +Find what the credential reaches, then pick its section under Fix: + +- [Admin → Audit](/admin/audit): what was done with it, from where. +- `spin admin runner list`, `spin admin token list --include-used`: runners and registration + tokens. +- `spin api-key list` (your own); [Admin → Users](/admin/users) lists each person's API keys. +- `spin admin volume rekeys`: the disks being re-keyed, and those a revoked runner marked. + +## Fix + +**A runner believed taken** (registered with a token): revoke it. Its credentials are deleted, +its stream is cut, and every disk it held the key of is marked to be re-keyed: + +``` +spin admin runner revoke +``` + +It refuses while workspaces' disks are on it; `--leave-workspaces` revokes anyway, and each of +those runs again only once its owner recovers it ([workspace-held](workspace-held.md)). An +autoscaling group's machine is given back instead ([lost-runner](lost-runner.md)); that does not +mark its disks for re-keying. A runner someone had a root shell on is revoked this way on its own +([host-shell](host-shell.md)). + +**A workspace's disk key** (re-keyed: copied under a new key, and the old disk and every snapshot +of it destroyed). The **workspace-rekeys** job does this for the disks a revoke marked, once each +workspace is stopped or quarantined. Watch it with `spin admin volume rekeys`. By hand, for a +stopped or quarantined workspace: + +``` +spin admin volume rekey +``` + +TODO: the CLI looks the workspace up as the caller's own before it calls the admin API, so it +reaches only an administrator's own workspaces; there is no command for another person's +([spin#1046](https://github.com/spin-stack/spin/issues/1046)). + +**Runner registration tokens**: `spin admin token revoke `. A pool's token is +re-minted on the control plane's machine with `sudo spin-controlplane registration-token ...`. + +**API keys**: the owner runs `spin api-key revoke `; an administrator revokes anyone's +on [Admin → Users](/admin/users). Keys a key delegated go with it. + +**Sessions and people**: disable the person's sign-in on [Admin → Users](/admin/users); they run +`spin logout` on a machine they still hold. A person who lost their passkeys: *Clear passkeys* on +the same page ([admin-lockout](admin-lockout.md) if no administrator can sign in). + +**SSH keys**: the owner runs `spin ssh-key delete `; a guest is taken out with +`spin workspace guests remove `. + +**The proxy's token** (an installation whose proxy is not on AWS), on the control plane's +machine: + +``` +sudo spin-controlplane proxy-token --name --revoke +sudo spin-controlplane proxy-token --name --expires 720h +``` + +**GitHub App**: its private key is never kept. To rotate its client secret, delete it on +[Admin → Integrations](/admin/integrations) and create another; every connection is revoked at +GitHub and people connect again (`spin git connect`). A person gives up their own token with +`spin git disconnect github`. + +**Team secrets**: replace the value at its source, then set it again from standard input: + +``` +spin team secret set --host +``` + +**The installation's encryption key** (seals secrets in the database, and the database's password +and archive keys are derived from it): check it on the control plane's machine: + +``` +sudo spin-controlplane crypto status +sudo spin-controlplane crypto verify +``` + +TODO: no procedure or command adds a new version of the installation's key to where an +installation keeps it, before `controlplane crypto rotate` ([spin#1049](https://github.com/spin-stack/spin/issues/1049)). A suspected leak of +this key is a security incident: escalate. + +## Verify + +[Admin → Audit](/admin/audit) shows each revoke. `spin admin volume rekeys` empties as the job +finishes; the job's last run on Admin → Jobs succeeded. `spin admin token list` and +`spin api-key list` no longer show what was revoked. + +## Escalate + +A leaked installation key, a runner that reconnects after a revoke, or **workspace-rekeys** +failing again and again are security problems: report them privately through +https://github.com/spin-stack/spin/security/policy with the audit entries, the job's error from +`spin admin jobs list`, and `spin admin versions`. diff --git a/runbooks/lost-runner.md b/runbooks/lost-runner.md new file mode 100644 index 0000000..4bb4fd3 --- /dev/null +++ b/runbooks/lost-runner.md @@ -0,0 +1,88 @@ +--- +id: lost-runner +title: A runner is gone for good, or should be forgotten +applies_to: + alerts: [] + jobs: [forget-gone-hosts] + events: [] + dashboards: [fleet] +severity: info +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +[Admin → Runners](/admin/runners) shows a runner *Not connected* that will not come back: its +machine was terminated, replaced or lost. Or the list keeps machines an autoscaling group started +long ago. Also read this when the **forget-gone-hosts** job fails on [Admin → Jobs](/admin/jobs). + +## Impact + +A runner that is gone still counts in the list and may still be the only home of some workspaces' +disks: those workspaces cannot start until they are recovered. While **forget-gone-hosts** fails, +the runners list keeps every machine an autoscaling group ever started. + +How a runner leaves depends on how it came: + +- **An autoscaling group's machine** is forgotten on its own by **forget-gone-hosts** once it has + been silent for `host_forget_after_hours` (24 by default), and no workspace's disk lives only + on it. Its other volumes are detached first. +- **A runner registered with a token** (a machine that stays) is never forgotten on its own: a + silent one is somebody's to look at, and it leaves only by revoke. + +## Diagnose + +``` +spin admin runner list +``` + +Open the runner on [Admin → Runners](/admin/runners): its row's menu offers the one way it can +leave (*Revoke runner*, *Give back to the* `` *group*, or *Forget runner*), or says why it is refused, +naming the workspaces whose disks are only there. Whether it is just slow to report, rather than +gone: [host-not-reporting](host-not-reporting.md). + +## Fix + +1. **Workspaces whose disk is only on it**: their owners recover them, giving up what the runner + had not published ([workspace-held](workspace-held.md)): + + ``` + spin workspace recover --accept-data-loss + ``` + +2. **An autoscaling group's machine**: give it back, or forget it now instead of waiting + `host_forget_after_hours`. A connected one is cordoned and given back (its group terminates + it and shrinks by one); one that is not connected is given back if the group still has it and + forgotten at once: + + ``` + spin admin runner remove + ``` + +3. **A runner registered with a token**: revoke it. That machine can never connect again, and + the disks it held are marked to be re-keyed ([key-rotation](key-rotation.md)): + + ``` + spin admin runner revoke + ``` + + If workspaces are still on it and their owners will recover them, `--leave-workspaces` + revokes anyway. +4. **The job fails**: `spin admin jobs list` shows its error; it needs the database. Run it + again with `spin admin jobs run forget-gone-hosts`. + +## Verify + +The runner is gone from `spin admin runner list` (or shown revoked). [Admin → Audit](/admin/audit) +has `host_given_back`, `host_forgotten` or `runner_revoked` with your user. The fleet dashboard's +*Hosts: in the database, and connected* on [Admin → Metrics](/admin/metrics?dashboard=fleet) +shows the two counts close together. + +## Escalate + +If `remove` or `revoke` is refused for a runner no workspace names, or the cloud refuses to give +the machine back, stop and open an issue at https://github.com/spin-stack/spin/issues with the +runner id, the refusal's text, the trace id from [Admin → Traces](/admin/traces) and +`spin admin versions`. A runner that reconnects after a revoke is a security problem: follow +https://github.com/spin-stack/spin/security/policy. diff --git a/runbooks/nothing-published.md b/runbooks/nothing-published.md new file mode 100644 index 0000000..3f75c8f --- /dev/null +++ b/runbooks/nothing-published.md @@ -0,0 +1,65 @@ +--- +id: nothing-published +title: A volume has published nothing for 30 minutes +applies_to: + alerts: [nothing-published] + jobs: [] + events: [] + dashboards: [storage, fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**A Volume Has Published Nothing For 30 Minutes** fires when, for some volume a host serves, the +host last published a commit to the object store more than 30 minutes ago +(`max(last_successful_commit_age_seconds) > 1800`), for 10 minutes. The description gives the +oldest age. + +Nobody sees anything: the workspace runs as usual. + +## Impact + +What the workspace wrote since its last commit lives only on its host. If the host is lost now, +that work is lost: this age is the recovery point (RPO). A stop or a move also waits on publishing. + +## Diagnose + +1. Find the volume and its host: *RPO: time since each volume last published* on + [Admin → Metrics](/admin/metrics?dashboard=storage), with *Unpublished bytes* beside it. Then + `spin admin storage list` says whose volume it is. +2. One volume or all of a host's? + - **All on one host**: the host cannot publish. Check *Is the store answering at all* and *A + host's reconciliation* on the storage dashboard, and the host on + [Admin → Runners](/admin/runners). + - **One volume**: check whether its host refuses it ([volume-not-served](volume-not-served.md)), + and *Rotations* and *A publish, end to end* on the storage dashboard. +3. Read the runner's warnings: + + ``` + spin admin logs 'service_name:spin-runner AND severity_text:warn' --since 1h + ``` + +## Fix + +- **The store is not answering** for that host: its network or credentials. Follow + [object-store-requests-runaway](object-store-requests-runaway.md) for the store's side and + [host-not-reporting](host-not-reporting.md) if the host is unwell. +- **The volume is refused** (`durability_lost`, `publish_fenced`, ...): follow + [volume-not-served](volume-not-served.md). +- **The host is stuck and the store is fine**: move its workspaces off before you lose it. + `spin admin runner drain --suspend` suspends each running workspace, which publishes + its disk, and resumes it on another runner. A workspace whose suspension fails stays where it + is ([workspace-suspends-failing](workspace-suspends-failing.md)). + +## Verify + +The volume's line on the RPO panel drops back to minutes, and the alert resolves. + +## Escalate + +A volume that does not publish while its host, the store and the volume all look healthy is a +defect. Open an issue at https://github.com/spin-stack/spin/issues with the volume id, the storage +panels over the window, the runner's log lines and `spin version`. diff --git a/runbooks/object-store-requests-runaway.md b/runbooks/object-store-requests-runaway.md new file mode 100644 index 0000000..d4d2e41 --- /dev/null +++ b/runbooks/object-store-requests-runaway.md @@ -0,0 +1,65 @@ +--- +id: object-store-requests-runaway +title: A host keeps asking the object store +applies_to: + alerts: [objectstore-requests-runaway] + jobs: [] + events: [] + dashboards: [storage] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**A Host Keeps Asking The Object Store** fires when one runner (`instance`, its host id) makes +more than 10 requests a second to the object store, averaged over 10 minutes +(`objectstore_request_duration_seconds_count{job="spin-runner"}`), for 15 minutes. + +A busy host makes about 1.4 a second at most and none at rest. The one storm seen was a restore +retried every cycle: over 25 a second from one host. + +## Impact + +Every request is on the cloud bill. Nothing is lost, but a storm left alone costs money every +hour, and is usually a symptom of a volume the host cannot serve. + +## Diagnose + +1. Open [Admin → Metrics](/admin/metrics?dashboard=storage): + - *Requests to the cloud, by service and operation* and *How long the store takes to answer, by + request* say which operation (`op`) the host repeats. + - *Is the store answering at all*: requests that fail and are retried. + - *A host's reconciliation: how long a cycle takes, and cycles that failed*: a host that fails + every cycle and tries again. +2. Read that host's warnings in the runners' log, and look for one volume named over and over: + + ``` + spin admin logs 'service_name:spin-runner AND severity_text:warn' --since 1h + ``` + +3. A volume restore that keeps failing is the known cause: check for + [volume-not-served](volume-not-served.md) firing too. + +## Fix + +- **One volume retried** (a restore, a publish): fix that volume + ([volume-not-served](volume-not-served.md)). Once it is served or moved, the requests stop. +- **The store refusing** (access denied, throttling): the host's credentials or the bucket's + policy changed. Check the host's role in the cloud and *Credentials hosts asked for, and what the + store's STS minted* on the storage dashboard. +- **Nothing explains it**: stop the host taking new work (`spin admin runner drain `) + and look at it on a [host-shell](host-shell.md). Draining with `--suspend` moves its running + workspaces elsewhere. + +## Verify + +The host's request rate on the storage dashboard drops back under one or two a second and the +alert resolves. Check the next day's cost on [Admin → Costs](/admin/costs). + +## Escalate + +A host that repeats a request with no failing volume or store error to explain it is a defect. +Open an issue at https://github.com/spin-stack/spin/issues with the host id, the operation, the +dashboard panels and the runner's log lines, and `spin version`. diff --git a/runbooks/quarantine-workspace.md b/runbooks/quarantine-workspace.md new file mode 100644 index 0000000..9e9a75f --- /dev/null +++ b/runbooks/quarantine-workspace.md @@ -0,0 +1,79 @@ +--- +id: quarantine-workspace +title: Quarantine a workspace, read its denials, release it +applies_to: + alerts: [] + jobs: [] + events: [] + dashboards: [] +severity: info +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +A workspace is held in quarantine, or should be: + +- **The control plane held it.** The workspace's guard refused something an agent in it tried + (a denial). The supervisor reports each denial, and the first one stops the workspace by force + and holds it. Its owner sees "this workspace is held in quarantine until an installation + administrator releases it" on every start, move or delete. +- **An administrator decides to hold it**: it is suspected of abuse or compromise, and its disk + must be kept as it is. + +## Impact + +While held, the workspace cannot be started, moved or deleted by its owner, and its disk is kept +as it was. Nothing else of the owner's is affected. Releasing it puts it back to stopped; the +record of what it attempted stays. + +## Diagnose + +All of these are an administrator's and take the workspace's id (not its name): it is on the +workspace's page address (`/workspaces/`), in the owner's `spin workspace list -o json`, and +on its entries in [Admin → Audit](/admin/audit). + +What the guard refused, newest first: target, which user and process, and why: + +``` +spin workspace denials --limit 100 +``` + +[Admin → Audit](/admin/audit) shows `workspace_quarantined` and `workspace_unquarantined`, with +who and the reason. + +## Fix + +- **Hold a workspace.** A running one is stopped first, forced, and held once its runner says it + stopped. The reason is kept on the audit entry: + + ``` + spin workspace quarantine --reason "" + ``` + +- **If its disk key may be taken** (the runner it ran on is suspect), re-key the disk while it is + held: see [key-rotation](key-rotation.md). +- **Release it** once the denials are understood (an agent tool that needed something the + policy refuses, not an escape): + + ``` + spin workspace unquarantine + ``` + + It goes back to stopped, with its disk as it was. A quarantine whose stop ended failed is + released as failed, and no longer held. +- **The denials show an attempt to escape**: keep it held, and escalate as a security problem. + +## Verify + +`spin workspace denials ` still answers (the record stays). After a release the owner can +start the workspace; the audit has the `workspace_unquarantined` entry. + +## Escalate + +Denials that show an attempt to get past the guard, or a workspace that keeps running after +`quarantine`, are security problems: report them privately through +https://github.com/spin-stack/spin/security/policy, with the workspace id, the denials, the +audit entries and `spin admin versions`. Never paste denials into a public issue: they can name +what the agent was after. diff --git a/runbooks/rate-limited.md b/runbooks/rate-limited.md new file mode 100644 index 0000000..695cac1 --- /dev/null +++ b/runbooks/rate-limited.md @@ -0,0 +1,69 @@ +--- +id: rate-limited +title: Requests are being rate limited +applies_to: + alerts: [rate-limited] + jobs: [] + events: [] + dashboards: [fleet, spin-api-performance] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**Requests Are Being Rate Limited** fires as soon as the control plane refused more than 50 +requests for the per-client rate limit in 10 minutes, for one kind of limit +(`rate_limit_rejected_total`, by `limit_type`): + +- `auth`: the sign-in calls, which have their own lower limit. +- `api`: every other call. + +The client refused gets `RESOURCE_EXHAUSTED`: "Rate limit exceeded. Please wait before retrying." + +## Impact + +Either the limit is too low for how the installation is used (a team behind one address, an agent +that calls the API in bursts), and real people are refused; or someone is guessing credentials or +hammering the API, and the limit is doing its job. Both want a look. + +## Diagnose + +1. *Refused credentials and rate-limited requests* on + [Admin → Metrics](/admin/metrics?dashboard=fleet): which limit, and whether credentials are + refused at the same time (a brute force: [auth-failures-looping](auth-failures-looping.md)). +2. Find who: the control plane logs each refusal with the client's address and the method. + + ``` + spin admin logs 'service_name:spin-controlplane AND body.message:"Rate limit exceeded"' --since 1h + ``` + +3. Is the address one you know: your office, a CI runner, the installation's own proxy? Is the + method one that a tool calls in a loop? +4. The limits in force are on [Admin → Security](/admin/security), *Rate limits and sessions*. + +## Fix + +- **A known client, a limit too low**: raise the limit (requests a minute and burst) on + [Admin → Security](/admin/security). It applies at once, without a restart. +- **A tool looping**: fix the tool. Its owner can see it in their own use; an API key it holds can + be revoked by its owner (`spin api-key revoke `). +- **Every client shows one address** (the proxy's, or a load balancer's): the control plane is not + seeing the client's real address, and everyone shares one budget. It trusts `X-Forwarded-For` + only from a local peer; check what sits in front of it. +- **Guessing or abuse from an unknown address**: leave the limit as it is; block the address at + the network edge (security group, firewall). Treat a sign-in brute force as a security event + ([auth-failures-looping](auth-failures-looping.md)). + +## Verify + +No more than 50 refusals in 10 minutes for that limit, and the alert resolves. The people who were +refused can work. + +## Escalate + +If the refusals come from addresses that should not be limited and raising the limit does not stop +them, open an issue at https://github.com/spin-stack/spin/issues with the log lines (remove the +addresses first: the tracker is public) and `spin version`. Abuse you cannot stop at the edge: +report it privately at https://github.com/spin-stack/spin/security/advisories/new. diff --git a/runbooks/runners-behind.md b/runbooks/runners-behind.md new file mode 100644 index 0000000..0e3d744 --- /dev/null +++ b/runbooks/runners-behind.md @@ -0,0 +1,73 @@ +--- +id: runners-behind +title: Runners run another release than the control plane +applies_to: + alerts: [runners-behind] + jobs: [] + events: [] + dashboards: [fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**Runners Are On Another Release Than The Control Plane** fires when connected runners report a +release (`hosts_by_version`, by `version`) that the control plane does not run +(`build_info{job="spin-controlplane"}`), for 30 minutes. The `version` label is the runners'. + +Usually seen after an update: a runner the update did not reach, or one that came back on an old +image or document. + +## Impact + +An installation runs one release in every half. A control plane serves a host of its own release +and refuses others, so a runner left behind takes no new work and its workspaces cannot be started +or stopped through it. The fleet has that much less room. + +## Diagnose + +1. See what runs where: + + ``` + spin admin versions + ``` + + It lists the control plane's release and, per host, the runner's release, its machine's and + whether this control plane serves it (`SERVED`, with why not). *Releases: what each process + runs, and the connected runners by release* on [Admin → Metrics](/admin/metrics?dashboard=fleet) + shows the same over time. +2. For each host behind: is it a machine of an autoscaling group, or one registered with a token? + `spin admin runner list` shows its class; the cloud's console shows its image. +3. A runner runs the release its machine's image carries. A machine started after the update from + an old image (an old launch template, an old document) comes back behind. + +## Fix + +- **An autoscaling group's runner**: make sure the group now starts machines of the control + plane's release (its launch template names the new image), then give the old one back to its + group, which drains it, terminates it and starts a new one: + + ``` + spin admin runner remove + ``` + + A runner that is the only home of a workspace's disk is refused, naming them; their owners stop + or move them first. +- **A runner registered with a token** (self-hosted): drain it with `--suspend` so its workspaces + move, install the control plane's release on the machine, and join it again with a new token + (`spin admin token create`). See [lost-runner](lost-runner.md) for taking the old host out. +- **The control plane is the one behind** (an update stopped half way): finish the update of the + control plane. + +## Verify + +`spin admin versions` shows every host on the control plane's release and served. The alert +resolves. + +## Escalate + +If a runner of the right release is still reported on another, or the control plane refuses a +runner of its own release, open an issue at https://github.com/spin-stack/spin/issues with +`spin admin versions` and `spin version`. diff --git a/runbooks/start-failures-unclassified.md b/runbooks/start-failures-unclassified.md new file mode 100644 index 0000000..0272464 --- /dev/null +++ b/runbooks/start-failures-unclassified.md @@ -0,0 +1,67 @@ +--- +id: start-failures-unclassified +title: Workspace starts failed where the runner named no part +applies_to: + alerts: [start-failure-unclassified] + jobs: [] + events: [] + dashboards: [fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**Unclassified Start Failures** fires as soon as any workspace became `FAILED` while creating or +starting, in the last hour, with no part named for the failure +(`workspace_failures_total{phase=~"creating|starting",failure="unclassified"}`). + +The owner sees a failed workspace with a summary of what went wrong, but without the advice a +known failure gets (which part failed, what to try). + +## Impact + +The owner's workspace failed to start. Every other failure names the part that failed (the +volume, the guest, resources, no host...), which is what tells the owner and you what to do; an +unclassified one is a place in Spinbox where that name is missing. That is a defect to report, +even when the workspace starts the second time. + +## Diagnose + +1. See how many, and whether they come with known failures: *Failures by phase and part* on + [Admin → Metrics](/admin/metrics?dashboard=fleet). +2. Find the failures in the logs, from the control plane and the runners: + + ``` + spin admin logs 'severity_text:error' --since 1h + ``` + +3. The workspace's owner sees what failed, the host's detail and the trace id: + + ``` + spin workspace get + ``` + + Open that trace in [Admin → Traces](/admin/traces): the start is one trace from the request to + the guest answering, and the failing span says where. + +## Fix + +- Find the cause in the trace and the logs and fix it as for any start failure: the host + ([host-not-reporting](host-not-reporting.md)), the volume + ([volume-not-served](volume-not-served.md)), the image or the network. +- The owner starts the workspace again (`spin workspace start `), or deletes it if it + never ran. +- Report the missing name either way (Escalate). + +## Verify + +New failures on *Failures by phase and part* name a part, and the alert resolves after an hour +with no unclassified failure. + +## Escalate + +Always open an issue at https://github.com/spin-stack/spin/issues, so the failure gets its name: +bring the failure summary and reason the workspace shows, the trace id from Admin → Traces, the +log lines, and `spin version`. diff --git a/runbooks/supply-vulnerabilities.md b/runbooks/supply-vulnerabilities.md new file mode 100644 index 0000000..4325f6d --- /dev/null +++ b/runbooks/supply-vulnerabilities.md @@ -0,0 +1,75 @@ +--- +id: supply-vulnerabilities +title: A vulnerability is known in what this installation runs +applies_to: + alerts: [supply-vulnerabilities] + jobs: [supply-vulnerabilities] + events: [] + dashboards: [fleet] +severity: warning +owner: security +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- **A Vulnerability Is Known In What This Installation Runs** fires when the last successful run of + the `supply-vulnerabilities` job counted one or more vulnerabilities + (`job_last_success_count{job="supply-vulnerabilities",count="vulnerabilities"} > 0`), for an + hour. +- The job itself failing shows on [Admin → Jobs](/admin/jobs?job=supply-vulnerabilities) and fires + the generic job alerts ([job-failing](job-failing.md)). + +Once a day the job reads the software bill of materials (SBOM) of every package the release's image +carries and asks OSV (`api.osv.dev`) which vulnerabilities are known in those versions. + +## Impact + +A vulnerability published after the release was built, which the release's own checks could not +have caught. Whether it matters depends on the package and on whether Spinbox uses the affected +part. While the job fails, a new vulnerability goes unreported. + +## Diagnose + +1. See which ones. The control plane logs each finding, with the vulnerability's id and the + package: + + ``` + spin admin logs 'service_name:spin-controlplane AND body.message:supplyvuln' --since 48h + ``` + +2. The job's page on [Admin → Jobs](/admin/jobs?job=supply-vulnerabilities) shows what its last run + counted: SBOMs, components and vulnerabilities. So does *What each job's last successful run + found* on [Admin → Metrics](/admin/metrics?dashboard=fleet). +3. Read each vulnerability at OSV (`https://osv.dev/vulnerability/` and its id): its severity, the + fixed version, and the affected functions. +4. Check whether a newer Spinbox release carries the fixed version: its release notes, or a + finding that is gone after updating. +5. If the job fails instead: the run's error on Admin → Jobs. Usually the control plane's machine + cannot reach `api.osv.dev`. A machine with no SBOM (one not installed from a release image) + logs `no SBOM on this machine` and counts nothing. + +## Fix + +- **A newer release carries the fix**: update the installation to it. +- **No release carries it yet**: report it (Escalate). Until a release has it, judge the + exposure from the advisory: what an attacker needs, and whether that part of the package is + reachable in your installation. +- **The job fails to reach OSV**: allow the control plane's machine HTTPS to `api.osv.dev`, then + run it again: + + ``` + spin admin jobs run supply-vulnerabilities + ``` + +## Verify + +After an update, the job's next run (it also runs when the control plane starts) counts 0 +vulnerabilities, and the alert resolves after an hour. + +## Escalate + +Tell the maintainers about a known vulnerability no release fixes yet. If it is exploitable in +Spinbox, report it privately at https://github.com/spin-stack/spin/security/advisories/new with the +id, the package, why you think it is reachable, and `spin version`. Otherwise open an issue at +https://github.com/spin-stack/spin/issues asking for the bump. diff --git a/runbooks/third-party-terms.md b/runbooks/third-party-terms.md new file mode 100644 index 0000000..56c8345 --- /dev/null +++ b/runbooks/third-party-terms.md @@ -0,0 +1,85 @@ +--- +id: third-party-terms +title: VS Code servers or agent CLIs are not published because their terms are not accepted +applies_to: + alerts: [] + jobs: [vscode-servers, agent-clis] + events: [] + dashboards: [] +severity: info +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- The **vscode-servers** or **agent-clis** job on [Admin → Jobs](/admin/jobs) succeeds every day + but counts only `off`, and publishes nothing. +- A publish is refused with "its third-party terms are not accepted", naming Admin → Extensions. +- `spin code` does not wait for the installation to prepare a VS Code server; VS Code downloads + its own into each workspace instead. +- Workspaces have no Claude Code, Codex or T3 Code on their `PATH`. +- The job fails (an error, not `off`) on Admin → Jobs. + +## Impact + +No Spinbox release carries the VS Code server or the agent CLIs: only their makers may +distribute them, so the installation downloads them itself, and only once an administrator +accepts their terms. Each item (the VS Code server; the agent CLIs) is accepted on its own. +Until then: + +- VS Code downloads its own server into every workspace it connects to: slower, and it needs + the workspace's network to reach Microsoft. +- Workspaces carry no agent CLIs from the installation. + +An acceptance of an older text counts as none: when a release changes the terms, they must be +accepted again. While a job fails with the terms accepted, workspaces keep the server and agents +last published, and a newer VS Code may not connect. + +## Diagnose + +``` +spin admin jobs list +``` + +The job's counts say why a day published nothing: `off` (terms not accepted), `released` (the +newest is published already), `busy` (another publish held the job), or `published`. + +[Admin → Extensions](/admin/extensions), *Third-party software*, shows each item, who accepted it +and when, and a warning when the accepted text is older than the current one. Its *VS Code +servers* and *Agent CLIs* tabs show what is published. + +## Fix + +- **Terms not accepted**: an administrator reads them on + [Admin → Extensions](/admin/extensions) (*Third-party software*) with each part's license, and + accepts the items the organization agrees to. The pinned releases start publishing at once, and + the daily jobs keep them current. Accepting is the organization's decision as licensee, not the + operator's alone. +- **Accepted as an older text**: accept again on the same page. +- **Accepted, and the job fails**: read its error with `spin admin jobs list`. It downloads from + the makers and uploads to the object store, so check that the control plane reaches both + (`spin admin health`, [object-store-requests-runaway](object-store-requests-runaway.md)). Then + run it again: + + ``` + spin admin jobs run vscode-servers + spin admin jobs run agent-clis + ``` + +- **To stop downloading**: *Revoke* on the same page. New downloads stop; what was published stays + until an administrator removes it. + +## Verify + +The next run counts `published` or `released`, not `off`. The *VS Code servers* and *Agent CLIs* +tabs list the new releases, and `spin code ` says it is preparing the installation's +server instead of letting VS Code download one. [Admin → Audit](/admin/audit) has +`third_party_terms_accepted`. + +## Escalate + +If the terms are accepted and the job still fails with the network and store healthy, stop and +open an issue at https://github.com/spin-stack/spin/issues with the job's error, the run's trace +id from [Admin → Traces](/admin/traces) and `spin admin versions`. Questions about the licenses +themselves go to the organization's own legal counsel, not to Spinbox. diff --git a/runbooks/tls-ask-unreachable.md b/runbooks/tls-ask-unreachable.md new file mode 100644 index 0000000..e5768fe --- /dev/null +++ b/runbooks/tls-ask-unreachable.md @@ -0,0 +1,70 @@ +--- +id: tls-ask-unreachable +title: The edge cannot ask the control plane about certificates +applies_to: + alerts: [tls-ask-unreachable] + jobs: [] + events: [] + dashboards: [fleet] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**The Edge Cannot Ask About Certificates** fires as soon as the proxy failed to ask the control +plane whether a name may have a certificate, in the last 10 minutes +(`tls_ask_total{answer="control_plane_unreachable"}`). + +The proxy asks before it gets a certificate for a workspace's name it has none for. When it cannot +ask, it hands out none: people opening a new workspace, or a newly exposed port, get a TLS error in +the browser. + +## Impact + +No new workspace name gets a certificate on that edge until the proxy can ask again. Names that +already have one keep working. + +## Diagnose + +1. *Certificates asked for* and *The edge's questions, as the control plane answered* on + [Admin → Metrics](/admin/metrics?dashboard=fleet). +2. Is the control plane up? If [api-errors](api-errors.md) is firing too, start there. +3. Read why the proxy could not ask. On the proxy's machine, its journal has a line for each + failure, with the name it was asked for: + + ``` + systemctl status spin-proxy.service + journalctl -u spin-proxy.service --since "1 hour ago" | grep "could not ask the control plane" + ``` + + The error says which: a connection refused or timed out (network), or an unauthenticated + answer (the proxy's token). + +## Fix + +- **The control plane is down**: bring it back (`systemctl status spin-controlplane.service` on its + machine); the proxy asks again on the next handshake. +- **The network between proxy and control plane**: fix the route, security group or DNS name. +- **The proxy's token is refused**: an AWS installation's proxy proves itself and renews its token + every half hour; check its journal for why it could not. A proxy given a token by hand needs a + new one, made on the control plane's machine: + + ``` + spin-controlplane proxy-token --name --expires 720h + ``` + + Install it where the proxy's document says it is kept (`install.join.token_at`), readable by root + alone; the proxy reads the file again when it changes. + +## Verify + +No new `control_plane_unreachable` answers in 10 minutes and the alert resolves. A new workspace +opens in the browser without a certificate error. + +## Escalate + +If the proxy cannot ask while the control plane answers everything else, open an issue at +https://github.com/spin-stack/spin/issues with the proxy's log lines, the control plane's log lines +for the same minutes, and `spin version`. diff --git a/runbooks/volume-not-served.md b/runbooks/volume-not-served.md new file mode 100644 index 0000000..bb61648 --- /dev/null +++ b/runbooks/volume-not-served.md @@ -0,0 +1,98 @@ +--- +id: volume-not-served +title: A host will not serve a workspace's disk +applies_to: + alerts: [volume-refused, volume-restores-failing] + jobs: [] + events: [] + dashboards: [fleet, storage] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +- **A Volume Is Not Served** fires when the database records any volume whose host reported it + will not serve it (`volumes_refused`, by `refusal`), for 5 minutes. The `refusal` label says why. +- **A Volume Keeps Failing To Restore** fires when more than 3 restores of a volume's published + chain onto a host failed in 30 minutes, across the fleet + (`volume_restores_total{outcome="failed"}`), for 10 minutes. Restores back off from five seconds + to five minutes, so this is usually one volume stuck, not a host in trouble. + +The workspace's owner sees a workspace that will not start, or one that stopped writing. + +## Impact + +The workspace on that volume cannot start, or runs with a disk that is not being published: what +it writes from now on is lost with its host. Each failed restore reads the object store again. + +## Diagnose + +1. Find the volume and its refusal in the runners' log: + + ``` + spin admin logs 'service_name:spin-runner AND body.message:"will not serve a volume"' --since 24h + ``` + + Each line names the `volume_id` and the `refusal` (as `VOLUME_REFUSAL_ISOLATED` for the + alert's `isolated`). For restores, search + `body.message:"restore finished without a chain"` the same way. +2. Find whose it is: `spin admin storage list` lists every volume with what it is. +3. The workspace's owner sees every party's view, the volume's included, with + `spin workspace status `. +4. Read the refusal: + + | `refusal` | What it means | + |---|---| + | `image_missing` | The volume's layers are not where the host must find them: a stray delete, a lifecycle expiry in the bucket. | + | `durability_lost` | QEMU flagged the disk image corrupt. The last commit is the newest point that can be restored. | + | `no_read_view` | The chain cannot be opened end to end: an unreadable ancestry, or a store that would not answer. | + | `no_key` | The volume is encrypted and the host holds no key, or the wrong one. | + | `lease_lost` | The host's lease lapsed and it gave the device up. | + | `isolated` | The host could reach neither the control plane nor the store, and paused the guest. | + | `publish_fenced` | Another host published to this volume: this one is not its writer any more. | + | `history_unanchored` | The bucket's history is not the one the database anchors: something other than the writer changed the bucket. | + | `attach_failed` | Everything else: QMP not answering, a `qemu-img` run that failed. The log line's error says which. | + +5. Look at the host: [Admin → Runners](/admin/runners) and *Volumes: refused, unplaced, at the depth + ceiling* on [Admin → Metrics](/admin/metrics?dashboard=fleet). + +## Fix + +- **`isolated`, `lease_lost`**: the host's network. When it reaches the control plane and the + store again it reports and serves. If it does not come back, follow + [host-not-reporting](host-not-reporting.md). +- **`publish_fenced`**: two hosts wrote one volume. Follow + [host-wrote-foreign-volume](host-wrote-foreign-volume.md) before anything else. +- **`history_unanchored`, `image_missing`**: the bucket was changed from outside Spinbox. Find + who in the cloud's audit log of the bucket; Object Lock kept the version the database anchors. + Do not delete anything in the bucket by hand. Escalate. +- **`no_key`**: the host's key does not open the volume. Check the host's access to the key + service (on AWS, its role's permission on the KMS key); the log line's detail says which. +- **`durability_lost`, `no_read_view`, `attach_failed`**: move the workspace off the host. The + owner runs, and reads the cost before accepting it: + + ``` + spin workspace recover + spin workspace recover --accept-data-loss + ``` + + A refused volume counts as stopped, so recover does not wait for the host. The workspace + restarts from the last commit the store has; see [workspace-rehomed](workspace-rehomed.md). +- **The host is the cause for many volumes**: stop it taking new work with + `spin admin runner drain `, and look at it on a [host-shell](host-shell.md). + +TODO: an administrator cannot list which workspaces have a refused volume, or recover a workspace they do not own ([spin#1045](https://github.com/spin-stack/spin/issues/1045)) + +## Verify + +`volumes_refused` is back to 0 on [Admin → Metrics](/admin/metrics?dashboard=fleet) and the +workspace starts. The restore alert resolves once 30 minutes pass without failed restores. + +## Escalate + +`history_unanchored` and `image_missing` are always escalated: open an issue at +https://github.com/spin-stack/spin/issues with the refusal, the runner's log line and +`spin version`. If you think someone changed the bucket on purpose, report it privately instead: +https://github.com/spin-stack/spin/security/advisories/new. diff --git a/runbooks/workspace-egress-high.md b/runbooks/workspace-egress-high.md new file mode 100644 index 0000000..124adb7 --- /dev/null +++ b/runbooks/workspace-egress-high.md @@ -0,0 +1,77 @@ +--- +id: workspace-egress-high +title: A workspace sent a lot today +applies_to: + alerts: [workspace-egress-high] + jobs: [] + events: [] + dashboards: [fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**A Workspace Sent A Lot Today** fires for each workspace whose machine has sent more than 100 GiB +since midnight UTC (`workspace_sent_today_bytes`, which exists only past that bound). The alert +names the workspace and its id. It resolves at midnight UTC, when the day's total starts again. + +## Impact + +Egress is billed by the byte: 100 GiB a day is a visible line on the bill. It can be legitimate +work (pushing images, syncing data), a mistake (a loop uploading), or abuse (the workspace used to +send traffic somewhere it should not). + +## Diagnose + +1. See it among others: *Workspaces that sent a lot today* on + [Admin → Metrics](/admin/metrics?dashboard=fleet). +2. See where it is sending. The workspace's owner runs this; it lists each destination with bytes + sent and received, while the workspace runs: + + ``` + spin workspace egress + ``` + + TODO: an administrator cannot read where a workspace they do not own sends ([spin#1045](https://github.com/spin-stack/spin/issues/1045)) +3. See what it has cost this month: + + ``` + spin admin costs by workspace + ``` + +4. Ask the owner what the workspace is doing. + +## Fix + +- **Legitimate**: nothing to do. If the cost matters, agree a limit with the owner. +- **A mistake, or more than it should**: limit what its network carries each way, in megabits a + second. A running machine takes it at once; open connections stay open: + + ``` + spin workspace bandwidth 50 + ``` + + `0` lifts the limit. Only an administrator can set it. +- **Abuse**: quarantine the workspace, which stops it and keeps its disk as it is for review + ([quarantine-workspace](quarantine-workspace.md)): + + ``` + spin workspace quarantine --reason "sent 100 GiB to an unknown destination" + ``` + +- **A network policy would have stopped it**: see [egress-denied](egress-denied.md) for how policies + work, and [Admin → Network](/admin/network). + +## Verify + +The workspace's rate on *Workspaces that sent a lot today* flattens. The alert itself stays until +midnight UTC, because the day's total does not go down. + +## Escalate + +If the traffic looks like an attack from or on the installation, report it privately at +https://github.com/spin-stack/spin/security/advisories/new. If the counted bytes do not match what +`spin workspace egress` shows, open an issue at https://github.com/spin-stack/spin/issues with +both and `spin version`. diff --git a/runbooks/workspace-held.md b/runbooks/workspace-held.md new file mode 100644 index 0000000..61431fc --- /dev/null +++ b/runbooks/workspace-held.md @@ -0,0 +1,90 @@ +--- +id: workspace-held +title: A workspace waits for its owner after its host stopped answering +applies_to: + alerts: [] + jobs: [] + events: [HELD] + dashboards: [fleet, storage] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +The owner is emailed (on by default) and sees on their notifications page +([Settings → Notifications](/settings/notifications)) and in the workspace's history: + +> Workspace `` is waiting for you: its host stopped answering + +with what moving it would give up and the command: + +``` +spin workspace recover --accept-data-loss +``` + +The workspace cannot start: a workspace starts only on its own host, and that host is silent. + +## Impact + +The owner has no workspace until either its host comes back or they give up the work the host +had not published to the object store. That work is on the silent host's disk and nowhere else. + +The control plane moves a workspace off a silent host on its own only when the installation +says so: `rehome_after_seconds` set, and the host's unpublished bytes at or under +`rehome_max_unpublished_bytes`. HELD means one of these: the cost is over that limit, or the +host never reported what it had not published. Both settings unset (the default) means nothing +is ever moved on its own, and the owner must recover by hand. + +## Diagnose + +1. Is the host coming back? Open [Admin → Runners](/admin/runners), or: + + ``` + spin admin runner list + ``` + + A runner that is not connected, with its last heartbeat old, is the cause. If many runners + went at once, follow [host-not-reporting](host-not-reporting.md) first. +2. What would moving cost? *What losing a host would cost* on + [Admin → Metrics](/admin/metrics?dashboard=storage) shows the RPO and unpublished bytes per + host. The owner sees the same for their workspace by running `recover` without the flag: + + ``` + spin workspace recover + ``` + + which changes nothing and prints the cost. +3. `spin workspace status ` (run by the owner) shows what the control plane, the + supervisor, the host and the volume each believe. + +## Fix + +- **The host can be brought back** (a reboot, a network fault): bring it back. The workspace runs + on it again with nothing lost. Nobody needs to recover anything. +- **The host is gone for good**: tell the owner. Only the owner can give up their work: the + command is theirs (`spin workspace recover --accept-data-loss`). It takes the disk + from the dead host and places it on a live one; then they start it. Afterwards retire the + host ([lost-runner](lost-runner.md)). +- **The installation should move such workspaces on its own** next time: set + `rehome_after_seconds` and `rehome_max_unpublished_bytes` in the installation's configuration + file, and apply the change on [Admin → Settings](/admin/settings). Giving up unpublished work + without asking is a policy decision; make it on purpose. + +TODO: an administrator cannot recover a workspace for its owner; `RecoverWorkspace` answers +only the owner ([spin#1045](https://github.com/spin-stack/spin/issues/1045)). + +## Verify + +`spin workspace get ` shows a live host, and the workspace starts. A recovery is in +[Admin → Audit](/admin/audit) as `workspace_recovered`, with the bytes given up. + +## Escalate + +Stop if `recover --accept-data-loss` fails with no host able to take the disk +(`RESOURCE_EXHAUSTED`) while runners have room, or if the host is connected and the workspace +still will not start. Bring the workspace id, the runner id, the output of +`spin workspace status `, the trace id of the failed call from +[Admin → Traces](/admin/traces) and `spin version`, and open an issue at +https://github.com/spin-stack/spin/issues. diff --git a/runbooks/workspace-rehomed.md b/runbooks/workspace-rehomed.md new file mode 100644 index 0000000..c1c72cd --- /dev/null +++ b/runbooks/workspace-rehomed.md @@ -0,0 +1,62 @@ +--- +id: workspace-rehomed +title: A workspace was moved off a host that stopped answering +applies_to: + alerts: [] + jobs: [] + events: [REHOMED] + dashboards: [fleet, storage] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +The owner sees on [Settings → Notifications](/settings/notifications) (email is off by default +for this one): + +> Workspace `` was moved off a host that stopped answering + +with how much work its host had not published, and `spin workspace start `. + +## Impact + +The host stopped answering for longer than `rehome_after_seconds`, and what it had not +published was at or under `rehome_max_unpublished_bytes`, so the control plane moved the +workspace to another host on its own, as the installation's settings allow. The owner lost the +writes that host had not published (the amount is in the notice). The workspace is stopped on +its new host until they start it. + +Many such notices at once mean a host, or several, went away. + +## Diagnose + +1. Which host went silent? [Admin → Audit](/admin/audit) has a `workspace_recovered` entry per + move, with the bytes given up; *Per host: heartbeat age* on + [Admin → Metrics](/admin/metrics?dashboard=fleet) shows which runner went quiet. +2. Why? Follow [host-not-reporting](host-not-reporting.md). +3. Was the loss larger than expected? *What losing a host would cost* on + [Admin → Metrics](/admin/metrics?dashboard=storage) shows how far behind each host's + publishing was. A host that publishes far behind its RPO loses more when it goes. + +## Fix + +- The owner starts the workspace: `spin workspace start `. +- Deal with the host that went: bring it back, or retire it ([lost-runner](lost-runner.md)). +- If the loss is more than the installation is willing to give up without asking, lower + `rehome_max_unpublished_bytes` (or unset `rehome_after_seconds` to move nothing on its own) + in the installation's configuration file and apply the change on + [Admin → Settings](/admin/settings). Workspaces over the limit then wait for their owner + instead ([workspace-held](workspace-held.md)). + +## Verify + +The workspace runs on its new host. The silent host is back, or gone from +[Admin → Runners](/admin/runners). + +## Escalate + +If workspaces are moved while their host is connected and heartbeating, stop: that is a defect. +Open an issue at https://github.com/spin-stack/spin/issues with the audit entries, the runner's +id and heartbeat age at the time, and `spin admin versions`. diff --git a/runbooks/workspace-stop-backup-failed.md b/runbooks/workspace-stop-backup-failed.md new file mode 100644 index 0000000..5d9b4c6 --- /dev/null +++ b/runbooks/workspace-stop-backup-failed.md @@ -0,0 +1,78 @@ +--- +id: workspace-stop-backup-failed +title: A workspace could not be stopped because its disk could not be published +applies_to: + alerts: [] + jobs: [] + events: [STOP_BACKUP_FAILED] + dashboards: [storage, fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +The owner is emailed (on by default) and sees on [Settings → Notifications](/settings/notifications): + +> Workspace `` could not be stopped + +with the reason the last attempt failed, and the command: + +``` +spin workspace stop +``` + +The workspace is still running. + +## Impact + +On a runner that can be taken away (a preemptible or autoscaling group's machine), a stop +publishes the disk to the object store first, so the workspace can start on any runner later. +That publish was tried three times and failed every time, so the stop was given up and the +workspace runs on. Nothing is lost yet: the disk is where it was. But the workspace keeps +costing, and if the machine is reclaimed before a stop succeeds, what it had not published is +lost. + +## Diagnose + +1. Read the reason in the notice, or in the owner's `spin workspace history `. +2. Is the runner connected? [Admin → Runners](/admin/runners), or `spin admin runner list`. A + stop on such a runner is refused while the runner is not connected. +3. Is the object store answering? *The object store itself* on + [Admin → Metrics](/admin/metrics?dashboard=storage), and *Publishing* for failed publishes. + `spin admin health` names a failed part. +4. Search what the runner logged around the attempts: + + ``` + spin admin logs 'service_name:runner AND severity_text:error' --since 1h + ``` + +## Fix + +- **The runner was disconnected for a while**: once it is connected again, the owner runs + `spin workspace stop ` again. +- **The object store refused or timed out**: fix the store first (see + [volume-not-served](volume-not-served.md) and + [object-store-requests-runaway](object-store-requests-runaway.md)), then stop again. +- **The runner is about to go, and the work on it does not matter**: the owner can stop without + publishing, giving up what only that runner has: + + ``` + spin workspace stop --force + ``` + +TODO: an administrator cannot stop another person's workspace; `StopWorkspace` answers only the +owner ([spin#1045](https://github.com/spin-stack/spin/issues/1045)). + +## Verify + +`spin workspace get ` shows it stopped (suspended at its published disk), and a +start places it on any runner. + +## Escalate + +If the store answers and the runner is connected but the stop still fails, stop and open an +issue at https://github.com/spin-stack/spin/issues with the reason from the notice, the +workspace and runner ids, the failing call's trace id from [Admin → Traces](/admin/traces), the +runner's error log lines and `spin admin versions`. diff --git a/runbooks/workspace-suspends-failing.md b/runbooks/workspace-suspends-failing.md new file mode 100644 index 0000000..6e4a895 --- /dev/null +++ b/runbooks/workspace-suspends-failing.md @@ -0,0 +1,80 @@ +--- +id: workspace-suspends-failing +title: A workspace could not be suspended +applies_to: + alerts: [workspace-suspends-failing] + jobs: [] + events: [] + dashboards: [fleet] +severity: critical +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +**A Workspace Could Not Be Suspended** fires as soon as any suspension failed in the last 30 +minutes (`workspace_suspend_duration_seconds_count{outcome="failed"}`, by `reason`). The `reason` +label says who asked: + +| `reason` | Who asked | +|---|---| +| `drain` | A host being drained with `--suspend`, or going away for good. | +| `owner` | The workspace's owner put it down. | +| `stop` | A stop on a host that can be taken away, which publishes the disk alone. | +| `move` | A move to a host with room for what it holds. | +| `idle` | Nobody used it for as long as the installation allows. | +| `running_cap` | It ran at a stretch for as long as the installation allows. | + +## Impact + +A suspension that fails leaves the workspace running where it was. For `drain`, that is a +workspace still running on a machine about to go: if the machine goes, its owner can recover it +only by giving up what it had not published. For `stop`, the owner is told the stop's backup +failed ([workspace-stop-backup-failed](workspace-stop-backup-failed.md)). + +## Diagnose + +1. See how many and which reason: *Workspaces suspended off their host, by who asked and how it + ended* on [Admin → Metrics](/admin/metrics?dashboard=fleet). +2. For a drain, the workspaces are named: + - in what `spin admin runner drain --suspend` printed: `Workspace ... was not + suspended and runs on: ...`; + - in [Admin → Audit](/admin/audit), the `host_evacuated` entry of the drain. +3. Read why it failed in the control plane's log: + + ``` + spin admin logs 'service_name:spin-controlplane AND severity_text:warn AND body.message:suspend' --since 1h + ``` + +4. A suspension writes the workspace's memory and disk to the object store: check the host + ([Admin → Runners](/admin/runners)) and the store (*Checkpoint memory* panels on + [Admin → Metrics](/admin/metrics?dashboard=storage)). + +## Fix + +- **A drain**: do not take the machine away yet. Run the drain again; it suspends what still runs: + + ``` + spin admin runner drain --suspend + ``` + + If a workspace fails again, its owner can stop it, which publishes its disk and gives up only + what was in memory (`spin workspace stop `). Only then retire the machine. +- **The host cannot write to the store**: follow + [object-store-requests-runaway](object-store-requests-runaway.md) or + [nothing-published](nothing-published.md). +- **The host is not answering**: follow [host-not-reporting](host-not-reporting.md). +- **`idle`, `running_cap`, `owner`, `move`**: the workspace runs on where it is, nothing lost. + Fix the cause above. + +## Verify + +`spin admin runner list` shows no workspaces left on a drained host. No new failures on the panel, +and the alert resolves after 30 minutes. + +## Escalate + +If suspensions fail with the host and the store healthy, open an issue at +https://github.com/spin-stack/spin/issues with the reason, the control plane's log lines, a trace +id from [Admin → Traces](/admin/traces) for the failed suspension, and `spin version`. diff --git a/runbooks/workspace-unresponsive.md b/runbooks/workspace-unresponsive.md new file mode 100644 index 0000000..56d9240 --- /dev/null +++ b/runbooks/workspace-unresponsive.md @@ -0,0 +1,80 @@ +--- +id: workspace-unresponsive +title: A workspace stopped responding and was stopped +applies_to: + alerts: [] + jobs: [silent-supervisors] + events: [UNRESPONSIVE] + dashboards: [fleet] +severity: warning +owner: operator +last_reviewed: 2026-11-09 +--- + +## Symptoms + +The owner sees on [Settings → Notifications](/settings/notifications) (email is off by default +for this one): + +> Workspace `` stopped responding and was stopped + +with the command `spin workspace start `. + +The **silent-supervisors** job on [Admin → Jobs](/admin/jobs) counts each such stop as +`supervisor_silence_stopped`; [Admin → Audit](/admin/audit) has a `workspace_supervisor_silence` +entry with the last heartbeat. After a control plane restart or a new leader, the job reports +`still_listening` for a few minutes and stops nothing. + +## Impact + +The supervisor inside the workspace said nothing for longer than the control plane waits, so the +workspace was stopped by force. Its disk is kept; a start boots it again, as after a power cut: +running processes and unsaved editor state are gone. + +If the **silent-supervisors** job itself fails, such a workspace runs on unwatched: its machine +may reach out while nothing reports what it does. + +## Diagnose + +1. One workspace, or many? Many at once on one runner is the runner, not the guests: follow + [host-not-reporting](host-not-reporting.md). +2. Why did the guest go quiet? The owner reads its console: + + ``` + spin workspace console --lines 300 + ``` + + or, where the installation collects consoles, every run of it: + + ``` + spin workspace logs 'panic OR "out of memory"' --since 24h + ``` + + An out-of-memory guest is the usual cause; *Containment and DNS* on + [Admin → Metrics](/admin/metrics?dashboard=fleet) counts OOM kills. +3. If the job fails: `spin admin jobs list` shows its last error. + +## Fix + +- **The guest ran out of memory**: the owner starts it again and gives it more + (`spin workspace grant --memory `), or uses a larger preset. +- **It keeps happening to one workspace with nothing in its console**: compare + `spin workspace status `, which shows what the supervisor last said, and escalate. +- **The job fails**: it fails when it cannot reach the database; see + [database-slow](database-slow.md). Then run it again: + + ``` + spin admin jobs run silent-supervisors + ``` + +## Verify + +The owner's `spin workspace start ` brings it to running and `spin workspace status` +shows the supervisor reporting. The job's last run on Admin → Jobs succeeded. + +## Escalate + +If guests with enough memory go silent again and again, or the job fails with the database +healthy, stop and open an issue at https://github.com/spin-stack/spin/issues with the workspace +id, the console tail, the audit entries, the job's error from `spin admin jobs list` and +`spin admin versions`.