Skip to content

boot: where a boot's time goes, and the first SSH login 60 ms sooner - #240

Merged
aledbf merged 5 commits into
mainfrom
boot-waits-and-jobs
Nov 13, 2026
Merged

aledbf merged 5 commits into
mainfrom
boot-waits-and-jobs

Conversation

@aledbf

@aledbf aledbf commented Nov 13, 2026

Copy link
Copy Markdown
Member

Where a boot's time goes, measured on the bench fleet, and what that bought: the first SSH login 60 ms sooner and the kernel 6.2 ms faster. Every number below is the bench fleet (AWS c8i.4xlarge, KVM nested), boots interleaved, p50.

What changes in the release

Change Measured Run
kernel/patches/0005: a device's PCI capability list is walked once, kept in struct pci_dev kernel start to "Freeing unused kernel image" 95.1 → 88.9 ms; acpi_init 14 → 10; virtio_pci_driver_init 6.5 → 4.7; inet_init (reads no config space) 7.50 → 7.49; ECAM accesses per boot 1,032 → 680 lab 43327612594 (20 boots)
spin lingers, logind back in the boot first SSH login over vsock 570/626 → 510/525 ms p50/p95; sshd accepts no later (324 → 321) lab 43308825692 (10 boots)
No MOTD, pam_motd off in sshd's and login's stacks a login −7 ms, the first −12 ms lab 43308825692
sshd@.service waits for the host keys SSH over vsock and ssh .host worked not at all: the per-connection sshd found no host keys and dropped the connection boot:ssh-first

Together, the first login from launch should be about 505 ms against 570. That is an estimate, image and kernel measured apart, because the lab boots published releases; the release's own boot:ssh-first and boot:initcalls confirm it.

The console prompt comes about 80 ms later with linger, because spin's manager now starts in the boot. Nothing a workspace does waits for it, but it is what boot:bench's usable column reads.

No fingerprint moves: nothing under machine/ changes, and a resumed guest runs the kernel it was saved with.

New probes

  • boot:waits: what waited on what between PID 1 and the login, from systemd's timestamps after the boot: a critical path that accounts for every millisecond, the gaps with no unit activating, and what still starts behind the prompt.
  • boot:jobs: PID 1's job queue recorded live over /run/systemd/private by boot/jobwatch, a diagnostic binary in the boot's overlay, beside boots without it. It also records the jobs the first SSH login sets off.
  • boot:ssh-first: from the host, over vsock (boot/vsockproxy): when sshd accepts, when the first login is done, and what a second one costs.
  • boot:exits: KVM's counters per phase (exits by kind, EPT faults, how the guest's RAM is mapped), and a row without virtio-rng.
  • boot:mmio: whose exits they are, from QEMU's trace: every PCI config access by device, every MMIO/PIO access by region.

All five run in the lab (lab.yml).

Fixed tools

  • SPIN_GRAPH and SPIN_CPU_SHARE traced nothing since the kernel forced lockdown at confidentiality (1400f6a), which closes tracefs, and reported "the trace is empty". They now print the lockdown level and say how to trace a kernel B built without it (kernel/README.md).
  • parseGraph read a function that migrated CPUs (acpi_init opened on CPU 1, closed on CPU 0) as nothing.
  • boot:trace takes SPIN_TRACE_APPEND and FLAGS.

Measured and not shipped

  • The getty without its wait for dev-ttyS0.device: 342 against 345 ms (lab 43303190733).
  • Host keys generated in the boot: 518 against 514 (lab 43312419007).
  • virtio-rng removed: −13.6 ms to the prompt (lab 43321809471). That is a fingerprint change, so it is not here.
  • pci=noearly: the guest gets no ECAM and does not boot.
  • PID 1's ~25 ms before the generators is the BPF verifier checking systemd's socket-bind program (bpf_check). Measured, nothing done.

Checked

  • task lint, task test.
  • compat.yml on the branch (43329268857): a checkpoint of the release before resumes here; boot:roms; boot:lockdown.
  • kernel.yml (43330642231) and image.yml (43330637817) on the branch. image/build.sh asserts on the finished filesystem: no pam_motd, spin lingering, the logind want not overridden, and the sshd@ drop-in.
  • report's matrix boots on the patched kernel, every row (lab 43328179961).
  • Hotplug on the patched kernel, by hand: spin-machine attach/detach of a disk works, and every device's MSI-X vectors are identical to the release kernel's.
  • boot:logind with logind in the boot.

Notes for review

  • The commit "Remove motd" also carries linger and logind in the boot (configure-system.sh, optimize-systemd.sh, 10-seats.conf, build.sh, boot/logind_test.go).
  • boot:logind's SPIN_LOGIND_NO_SEATS=1 control was written to fail, and on this image (systemd 259) it passes. So the 10-seats.conf drop-in may no longer be needed. Not settled here.
  • go.mod gains github.com/godbus/dbus/v5 v5.1.0, used only by boot/jobwatch, which is never in a release.

🤖 Generated with Claude Code

aledbf and others added 5 commits November 12, 2026 21:00
Signed-off-by: Manuel de Brito Fontes <aledbf@gmail.com>
Signed-off-by: Manuel de Brito Fontes <aledbf@gmail.com>
Signed-off-by: Manuel de Brito Fontes <aledbf@gmail.com>
Signed-off-by: Manuel de Brito Fontes <aledbf@gmail.com>
…bwatch

The pull request's gate failed on 130 mutations no test refused. The
library code now has a test for each edge it decides on: a unit's name,
each way a unit sits in the dump, console noise between its blocks, what
a unit waited on and the ties between candidates, where a gap begins and
ends and which unit ended it, the order behind the prompt, every event's
fields, when a job ran, and Lag's cut-offs. startingAt and After answer
from a map, so their tests ask 64 times, through many orders.

jobwatch's main, watch and now and vsockproxy's main are held by the
lanes that run them in a guest (mutate-lane: ./boot.TestBootJobs,
./boot.TestFirstSSH). Exempted with the reason on the line: comparisons
of a duration with zero, where every duration is whole microseconds, and
the scanner buffer's size and ParseUint's bit size, past what any input
reaches. ParseJobs loses an assignment the loop after it already made.

task mutate against origin/main: 181 refused, 0 survived.

Signed-off-by: Manuel de Brito Fontes <aledbf@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf merged commit e5b8003 into main Nov 13, 2026
7 checks passed
@aledbf
aledbf deleted the boot-waits-and-jobs branch November 13, 2026 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant