Skip to content

image: the SSH host keys are made from the boot's first dispatch - #248

Merged
aledbf merged 1 commit into
mainfrom
keys-early
Nov 13, 2026
Merged

aledbf merged 1 commit into
mainfrom
keys-early

Conversation

@aledbf

@aledbf aledbf commented Nov 13, 2026

Copy link
Copy Markdown
Member

Replaces #247, rebased onto main after #245 (the conflicts were both PRs adding to the same assertions and to ssh-first's rows; both kept).

Three image-side changes for the cold boot to the first SSH login, measured one by one and together against the image as built, judged on the first login over vsock.

Fleet (task boot:ssh-first, lab run 43483372488, release v20261113.02, 20 boots of each, interleaved, 2 vCPUs)

row sshd accepts p50/p95 first login p50/p95 accept to login p50/p95
as built 309/334 489/506 177/194
keys early 315/325 468/494 154/168
generators masked 306/314 489/501 180/192
inert units masked 306/316 484/501 178/191
batch (all three) 311/321 469/490 156/173

No row lost a login (0/20 each). Keys early alone had already measured 486 to 468 and 177 to 155 in lab run 43463472762. Only the keys moved the login, so this PR ships them and leaves the other two out.

Per item

  1. Keys early: shipped. spin-machine-sshd-keygen.service now has DefaultDependencies=no, Conflicts=/Before=shutdown.target and WantedBy=sysinit.target. optimize-systemd.sh enables it with a link in sysinit.target.wants. The sshd@.service.d/keygen.conf ordering stays. build.sh asserts the link and DefaultDependencies=no on the finished filesystem. In the built image, the keygen starts before sysinit.target and is done by then, and no unit fails. boot:ssh-first and boot:logind pass on it locally. One consequence for consumers: a guest checkpointed after boot now always carries host keys, so the restorer has to hand each VM its own. That was already true whenever anything had logged in before the checkpoint. image/README.md says this.
  2. Generators: left out. The image runs four system generators: sshd-socket-generator, systemd-getty-generator, systemd-run-generator and systemd-ssh-generator. The rest are already masked. sshd-socket-generator is the slowest: it loads libcrypto and writes nothing, because sshd_config is at its defaults. In a guest it took 3.5-23.5 ms, against about 1.5 ms for each systemd generator. The generators run in parallel, though. Masking it and systemd-run-generator took the generators phase from 5.3 to 3.7 ms p50 (laptop, 8 boots each), and on the fleet the login did not move (489). The fleet's generators phase is 6.5 ms. The ~44 ms before it is PID 1's own setup, not generators.
  3. The ~29-32 ms gap: explained, not removable. After UnitsLoadFinish, no unit runs for about 32 ms p50 on the fleet (lab run 43361486792). That time is PID 1 itself. systemd stamps UnitsLoadFinish after it enumerates mounts, devices and slices, and only then loads the boot's units: coldplug, then Activating default unit: default.target. That parses every unit file default.target reaches, plus a stub for every name an After=/RequiresMountsFor= gives. It then builds the initial transaction (64 jobs) and garbage-collects the stubs before it dispatches the first job (systemd-ask-password.socket). This comes from task boot:trace with systemd.log_level=debug, which ends in "Loaded units and determined initial transaction". The gap only shrinks along with the transaction. Masking the 11 jobs whose condition fails on every boot here (pcr*, kmod-static-nodes, apparmor, the distribution's sshd-keygen, getty-static, getty@tty1) did not move the login (484). The explanation is now a comment in boot/waits_probe_test.go.

boot/ssh_first_test.go keeps one row, "as this branch", which applies keys early to older releases too. The experiment rows are gone and their results are in its comment. The probe gains an accept-to-login column.

🤖 Generated with Claude Code

spin-machine-sshd-keygen.service is wanted by sysinit.target without default dependencies,
so the keys exist before sshd-vsock.socket listens and the first login's sshd@ no longer
waits for them before its banner. On the bench fleet, 20 boots of each interleaved (task
boot:ssh-first, lab run 43483372488): the first SSH login 489/506 -> 468/494 ms p50/p95,
accept to login 177 -> 154 ms. build.sh asserts the want and DefaultDependencies=no.

Measured beside it and left out, the conclusions kept by the probes: masking
sshd-socket-generator and systemd-run-generator (generators 5.3 -> 3.7 ms, login 489), and
masking the eleven initial-transaction units whose condition always fails here (login 484);
all three together 469. The ~32 ms with no unit running after the unit load is PID 1 loading
the default target's units and building the initial transaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf merged commit 3e8075f into main Nov 13, 2026
3 checks passed
@aledbf
aledbf deleted the keys-early branch November 13, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant