Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 64 additions & 1 deletion docs/manual-test-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -1166,7 +1166,7 @@ go vet ./...
### Known limitations / follow-ups

- **Talos-mutated-field allowlist**: Phase 2B reports cert hashes / timestamps as divergence today; the verify is off by default until an allowlist lands.
- **`talm upgrade` has no pre-upgrade gates** (Phase 2C runs *after*, not before): the upgrade flow wraps `talosctl upgrade` and doesn't route through `buildApplyClosure` / `applyOneFileDirectPatchMode`, so Phase 1 / Phase 2A do not run. Phase 2C (post-upgrade version verify) was added precisely to catch the silent-rollback class without that refactor. Full pre-upgrade gates would require reproducing the gate calls in `upgrade_handler.go` or refactoring the apply flow.
- **`talm upgrade` does not run the apply-path gates**: the upgrade flow wraps `talosctl upgrade` and doesn't route through `buildApplyClosure` / `applyOneFileDirectPatchMode`, so Phase 1 / Phase 2A do not run. Two gates of its own cover the classes that matter most: the pre-upgrade path check refuses a move Talos will not make, and Phase 2C catches the silent rollback afterwards. Bringing the apply gates over would still require reproducing the gate calls in `upgrade_handler.go` or refactoring the apply flow.
- **Phase 1/2 on `--insecure`**: the safety gates can't run before the chart renders, and the chart's `lookup` calls need an authenticated COSI connection. Insecure path = effectively no gates today.

## D. Apply (insecure / maintenance path)
Expand Down Expand Up @@ -1320,6 +1320,69 @@ Expected: stderr emits the "Synced machine.install.image" line immediately after

Regression anchor: contract tests `TestContract_WriteBackInstallImage_*` pin the on-disk shape of the patch (scalar swap, idempotency, silent-skip on orphan files, structural errors, file-list fan-out). A regression that fires the write-back BEFORE verify (or instead of verify) would silently pin the body to an image the node never actually ran — the failure-path scenario above would catch it. Cross-reference: `pkg/commands/upgrade_image_writeback.go`.


### E5. Node body image divergence is reported

The upgrade target comes from `values.yaml`, so a node body naming something else is ignored and then rewritten. Both directions are reported, but only one of them is a problem.

```bash
# Point a node body at a NEWER Talos than values.yaml names.
sed -i.bak 's|image: ghcr.io/cozystack/cozystack/talos:.*|image: ghcr.io/cozystack/cozystack/talos:<A_NEWER_TAG>|' nodes/node0.yaml
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
```

Do not pipe this into `head`. With `2>&1 |` talm's stderr is the pipe, `head` closes it after five lines, and a Go process takes SIGPIPE on fd 2 rather than an error — talm dies after the RPC has already fired, skipping the post-upgrade verify and the `install.image` write-back that E4 covers. The pipeline still exits 0, so nothing says it happened.

Expected: `Using image from values.yaml: <ref>` followed by a `warning:` line naming `nodes/node0.yaml`, the body's image, and the resolved target, telling you to put that image in values.yaml if the file is the one you meant. The warning fires because the body names a newer Talos than the target: the upgrade is going somewhere the file does not.

`<A_NEWER_TAG>` only has to be newer than what `values.yaml` names — the comparison is ordered down to the patch, so `v1.13.1` against a `v1.13.0` target warns just like a newer minor does. A node moving backwards by a patch is the same incident one granularity down, and the pre-upgrade path check lets it through because Talos allows patch moves inside a minor.

Now the ordinary direction. Restore the body, then bump `values.yaml` forward a minor and leave the bodies as rendered, which is what an operator raising the cluster's version actually does:

```bash
mv nodes/node0.yaml.bak nodes/node0.yaml
cp values.yaml values.yaml.orig
sed -i '' 's|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:<A_NEWER_TAG>"|' values.yaml
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
```

Expected: the divergence reports as a plain line with no `warning:` label and no advice. That is the canonical flow, where every body trails the target until the write-back resyncs it, so advice to bump values.yaml would be advice to repeat what you just did.

Watch for:

- A `warning:` label on the trailing-body case, which is every node after a values.yaml bump.
- Silence on a body that names another registry at the same version: that one cannot be shown to trail, so it is a divergence.
- The report firing when the body already matches the target, or on a side-patch that declares no `install.image`.

### E6. Pre-upgrade path check

The refusal itself stops before the RPC, so that half needs no node that reboots. The escape-hatch flag is a different matter — see the warning below it.

Pick a target Talos actually refuses: two minors back, or several minors forward. One minor back is a supported rollback and must NOT be blocked.

```bash
sed -i '' 's|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:v1.10.5"|' values.yaml
talm upgrade -f nodes/node0.yaml
```

Expected: the upgrade is refused with `refusing this upgrade: host version <running> is too new to downgrade to Talos 1.10.5`, and a hint offering the two ways out. Nothing is sent to the node.

Do not run `--skip-upgrade-path-check` on this fixture to see the escape hatch work. It does work, and that is the problem: the RPC goes out and the node attempts a cross-minor downgrade, which is what the hint two lines above says ends in an A/B rollback or a node that will not boot. Exercise the flag only against a target the node can actually take, or on a stand you are willing to rebuild.

Watch for:

- A refusal on one minor back, which Talos supports: the guard must follow the matrix, not a version comparison.
- A node that is unreachable blocking the upgrade: that case prints `warning: upgrade-path check skipped for <node>` and carries on.
- A missing hint on the refusal: `errors.Join` drops the hints of the errors it joins, so the way out has to be attached to the joined error.

Restore `values.yaml` afterwards, from the copy E5 took before it edited anything:

```bash
mv values.yaml.orig values.yaml
```

E5 and E6 both edit `values.yaml`, so they must not share one backup: a second `sed -i.bak` overwrites the first backup with already-edited content, and the restore then puts back E5's value rather than the original. The sections that follow read `values.yaml::image`.

## F. CA rotation

### F1. Rotate CA dry-run
Expand Down
9 changes: 9 additions & 0 deletions docs/operations/safety-gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
| Pre-apply drift preview | `--skip-drift-preview` | on |
| Post-apply state verification | `--skip-post-apply-verify` | **off** |
| Post-upgrade version verify | `--skip-post-upgrade-verify` | on |
| Pre-upgrade path check | `--skip-upgrade-path-check` | on |

## 1. Declared-resource existence

Expand Down Expand Up @@ -49,6 +50,14 @@ After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig a

After `talm upgrade` reports success, waits the configured reconcile window (default 90s; tune via `--post-upgrade-reconcile-window` for slow hardware / large image pulls) for the node to finish booting, then reads `runtime.Version` COSI and compares the running version's `(Major, Minor)` contract against the contract parsed from the target image tag. Point releases share a minor contract; cross-minor mismatch surfaces as a hint-bearing blocker. Catches the silent A/B rollback case where the upgrade RPC acks success but Talos rolled back to the previous partition (cross-vendor image, missing extensions, failed boot readiness check, slow boot exceeding the configured window). Best-effort surrender on digest-pinned images and unparseable tags.

## 5. Pre-upgrade path check

Before `talm upgrade` fires the RPC, reads `runtime.Version` COSI from every target node and asks Talos's own compatibility matrix (`pkg/machinery/compatibility`) whether the node can be moved to the target. That is the same matrix the Talos installer runs as its own pre-flight, so this gate answers with the authority rather than a version comparison of its own.

Two shapes are refused. A target too far forward — Talos accepts upgrades only from a bounded range of older versions — would otherwise fail inside the installer after the image has been pulled. A downgrade past what Talos allows (it supports at most one minor back) is invisible to gate 4, because once the downgrade takes, running equals target and the verify passes; the post-upgrade write-back then pins the node body to the older image.

The usual way in is a `values.yaml` nobody bumped, so the refusal carries Talos's own verdict and offers bumping it or passing `--image`. To try anyway, `--skip-upgrade-path-check`. Best-effort surrender on digest-pinned images, unparseable tags, and a target newer than any minor this talm build's matrix knows — that last one means the binary is too old to have an opinion, not that the path is wrong. A node whose version cannot be read is reported and skipped rather than blocked: an unreachable node is a worse reason to stop an upgrade than the move this prevents.

## Skip flags and the maintenance path

The skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.
27 changes: 24 additions & 3 deletions docs/reference/upgrade.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
filippo.io/age v1.3.1
github.com/BurntSushi/toml v1.6.0
github.com/Masterminds/sprig/v3 v3.3.0
github.com/blang/semver/v4 v4.0.0
github.com/cockroachdb/errors v1.14.0
github.com/cosi-project/runtime v1.16.3
github.com/dustin/go-humanize v1.0.1
Expand Down Expand Up @@ -70,7 +71,6 @@ require (
github.com/aws/aws-sdk-go-v2/service/sts v1.45.4 // indirect
github.com/aws/smithy-go v1.27.7 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/brianvoe/gofakeit/v7 v7.14.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
Expand Down
Loading
Loading