fix(host-provisioning): apt retry, drop gpg, AWS CLI v2, atomic writes (26.04 live-verified)#178
Merged
Merged
Conversation
…or (v5 deploy)
The re-deploy still failed on two host-provisioning issues the earlier lock/keyring
work didn't fully cover:
- `E: Could not get lock /var/lib/apt/lists/lock`. `-o DPkg::Lock::Timeout` does NOT
cover `apt-get update`'s lists lock on Ubuntu 24.04's apt, so the ~4 concurrent
per-host installs (ingress + mesh nginx, Postgres, awscli) still raced it. New
`internal/aptlock.UpdateCmd` retries `apt-get update` until the lock frees; used
by the nginx, Postgres, and backup installs.
- `gpg: cannot open '/dev/tty'`. `sudo gpg --dearmor` fails on a host with no
controlling tty. Drop gpg entirely — install the ARMORED signing key (.asc) and
reference it via apt's signed-by (apt reads armored keys directly). Applied to the
nginx and Postgres keyrings.
- The nginx install used `sudo -E apt-get`, which the host sudoers ignores ("'-E' is
ignored"), so DEBIAN_FRONTEND=noninteractive never reached apt (debconf fell back
to a Teletype frontend). Pass it on the sudo command line instead, matching the
other installs (incl. the mesh openssl install).
Regression tests: aptlock retry + bash -n; nginx install (no gpg / no -E, .asc key,
retry update); Postgres install (.asc key, no gpg, retry update).
Claude-Session: https://claude.ai/code/session_015CmCU1fcbVKUR2k1dquEWV
…m Ubuntu 24.04) Verified on a live Ubuntu 24.04 host: `apt-get install awscli` fails with "Package 'awscli' has no installation candidate" — the package was dropped from noble. Install the official AWS CLI v2 bundle instead (self-contained, current, supported), extracted with unzip; `aws/install --update` is idempotent. Caught by an end-to-end run of the rendered install scripts against a throwaway Hetzner host — which also confirmed the retry-update survives 5-way apt contention, the armored .asc keyrings work with no gpg tty error, `sudo VAR=val` works under the deploy sudoers, and the Postgres cluster initdb+start comes up with 0700 PGDATA and accepts a psql connection.
pedromvgomes
changed the title
…stdin fails on 26.04) Verified on a live Ubuntu 26.04 (resolute, coreutils 9) host: `sudo install -m <mode> /dev/stdin <path>` fails "install: No such file or directory" when the target ALREADY EXISTS — which the Postgres config write always hits, because initdb pre-creates postgresql.conf/pg_hba.conf. That aborted the cluster apply, so the postmaster never started. Switch the postgres/otelcol/dbbackup writeFileScript helpers to the same stage-then-atomic-mv pattern already used in internal/remote: base64 -d | tee to a mktemp temp, chmod (+chown) the temp, then mv into place. A 0600 file is never briefly world-readable and the replace is atomic. (remote.go already did this.) Caught by re-running the end-to-end install-script suite on 26.04 — the ACTUAL deploy target (24.04 passed because its older install accepts /dev/stdin over an existing file). That run also confirmed on 26.04: all 5 apt installs pass concurrently, .asc keyrings, AWS CLI v2, and the full Postgres cluster now comes up (active, PGDATA 0700, psql connects, pgdg26.04 build).
pedromvgomes
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Host-provisioning fixes for the self-hosted-Postgres + observability deploy, validated end-to-end on a live Ubuntu 26.04 host (
resolute, coreutils 9) — the actual deploy target — by rendering the real install scripts, running the 5 apt installs concurrently, and driving the full Postgres cluster flow to a livepsqlconnection.Fixed
apt-get updatelists-lock race.-o DPkg::Lock::Timeoutdoesn't coverapt-get update's lists lock — newinternal/aptlock.UpdateCmdretries it. Live: survives 5 concurrent installs.gpg: cannot open '/dev/tty'. Droppedgpg --dearmor— install the armored.asckey and use aptsigned-bydirectly. Live: keyrings valid, no tty error.sudo -Eignored by the deploy sudoers → passDEBIAN_FRONTENDon the command line. Live:sudo VAR=valworks under the realNOPASSWD:ALLsudoers.awscliapt package gone from Ubuntu 24.04+ → install the official AWS CLI v2 bundle. Live:aws-cli/2.35.16.install /dev/stdinover an existing file fails on coreutils 9 (Ubuntu 26.04) —"install: No such file or directory"— which the Postgres config write always hits (initdb pre-createspostgresql.conf), aborting the cluster apply. The postgres/otelcol/dbbackupwriteFileScripthelpers now stage to a temp then atomicmv(the patterninternal/remotealready used). Live: cluster comes up.Validated live on 26.04 (the real target)
.asckeyrings; AWS CLI v2.install → initdb → apply → **active**, PGDATA 0700,psqlconnects —PostgreSQL 17.10 (pgdg26.04 build).resolute-pgdghaspostgresql-17; nginx.orgresolutemainline hasnginx+nginx-module-acme.(An initdb "invalid locale" error during testing was a test artifact — the local
sshclient forwarding an ungeneratedLANG; the real Go-SSH deploy doesn't forward it.)Tests
aptlock(retry +bash -n); nginx install (no gpg/-E,.asc, retry); Postgres install (.asc, retry) + writeFileScript (temp+mv); dbbackup (AWS CLI v2, temp+mv); otelcol (temp+mv).CGO_ENABLED=0 go build,go test -race ./...,golangci-lintall clean.Merge Commit Message
https://claude.ai/code/session_015CmCU1fcbVKUR2k1dquEWV