Replacing QEMU with libkrun, Cloud Hypervisor or Firecracker: evaluated 2026-11-14, staying on QEMU #259

An evaluation, on 2026-11-14, of replacing QEMU as the VMM under this machine with libkrun, Cloud Hypervisor or Firecracker. The decision is to stay on QEMU. This issue records what was compared so the question does not have to be researched again from nothing.

What the machine asks of its VMM

Read from machine/, and from spin's internal/runner/qemu and internal/storage/qcow:

• A live checkpoint: migrate to a file while the guest runs, -incoming / migrate-incoming to resume. Measured pause 50-92 ms at 1.2 GB used, 198 ms at 2.7 GB; the stopped way paused 414-940 ms.
• A live block graph: blockdev-snapshot, blockdev-reopen, change-backing-file, blockdev-add/del. The disk is sealed at the instant the guest is paused, and chains are compacted while the machine runs.
• qcow2 layers made with extended_l2=on,cluster_size=128k (spin internal/storage/qcow/chain.go), which took a first write onto the base from 82 MB to 13 MB of overlay growth (docs/memory-and-disk.md).
• CPU hotplug over ACPI. microvm was turned down for lacking it (qemu/devices.mak).
• virtio-mem to grow past boot memory.
• Disk hotplug on one virtio-scsi controller: attach 1.5 ms, detach 0.7 ms, against 135 ms and 6.1 s for virtio-blk behind PCIe root ports, which also cost ~10 ms a port on every boot.
• vmgenid, so a resumed guest reseeds its random pool.
• Descriptor handover (add-fd, fdsets): QEMU opens nothing by name.
• A promise about checkpoints across upgrades: the versioned machine type (pc-q35-11.1), which the fingerprint and task boot:compat rest on.
• A named CPU model with enforce=on, so a checkpoint moves between hosts.
• vhost-vsock, the kernel's datapath.

The comparison

libkrun

qcow2 is not what is missing: KRUN_DISK_FORMAT_QCOW2 exists, through the imago crate, with read-only, direct I/O and sync modes. It also has an external kernel payload, a PCI device manager, virtio-blk, net over a TAP, rng, a userspace vsock, a balloon with free page reporting, and pause / resume / shutdown on a running VM.

Missing, each of them something to write: saving and restoring memory, device and vCPU state; any change to a disk after the VM is built; CPU hotplug; virtio-mem; virtio-scsi and disk hotplug; vmgenid; descriptor handover; a monitor (events, block statistics, balloon statistics); a compatibility contract between versions. It is a library whose krun_vmm_run takes the process, so a binary of our own would stand where qemu-system-x86_64 does.

Not checked: whether imago reads extended_l2, and whether discard reaches the qcow2.

Cloud Hypervisor

Has: /vm.snapshot and /vm.restore, restore on demand through userfaultfd and copy-on-write restore with reflinks; CPU hotplug over ACPI on x86; virtio-mem; vmgenid; a balloon with statistics; discard and write-zeroes; rate limiters; PCI hotplug of virtio-blk disks; an HTTP API with an event monitor; seccomp and Landlock.

What stands in the way:

• Today's disks do not open. Its qcow2 refuses extended_l2 and the external data file: block/src/formats/qcow/header.rs accepts only the dirty, corrupt and compression incompatible bits.
• No live overlay rotation. Nothing like blockdev-snapshot, blockdev-reopen or change-backing-file; the root disk cannot change under a running VM.
• The checkpoint pauses the VM while memory is written. send-migration to a receiver of our own may give a precopy to a file; not verified.
• Checkpoints age out sooner. Migration compatibility is promised from the two previous versions only (n-2, from v54).
• CPU profiles are experimental.
• vsock is emulated in userspace and reached through a Unix socket; disks are given by path, not by descriptor; there is no virtio-scsi.

Firecracker

Has: /snapshot/create and /snapshot/load with memory loaded on demand; virtio-mem (/hotplug/memory); vmgenid; a balloon with statistics and hinting; rate limiters; discard as a hole punch; the jailer and seccomp.

What stands in the way:

• Raw disks only: no image format and no block graph, so layers would move to the host (dm-thin, dm-snapshot or reflinks) and spin's storage/qcow would be replaced, not ported. Inferred from its block documentation, which describes raw devices only; its source was not read.
• No CPU hotplug.
• A snapshot needs the VM paused, and diff snapshots are a developer preview.
• Device hotplug is a developer preview, through PCIe root ports and pciehp, where the guest waits five seconds to let a device go: the mechanism already measured and turned down here.
• Snapshots do not move between CPU models without a CPU template, and the state format takes a new major version with nearly every change.
• The vsock device is reset when a snapshot is taken, so open connections to the guest end at every checkpoint.

What a move would cost whichever is chosen

• spin's internal/runner/qemu (the QMP client, handover, checkpoint, hotplug) and the QMP half of internal/storage/qcow.
• This repository's machine/ (Args, the fingerprint) and the boot and compat lanes.
• Every checkpoint saved by QEMU: none resumes on another VMM, so every workspace cold boots at the cut.

Decision

Stay on QEMU. The device set is already an allowlist (--without-default-devices): 363 to 225 instantiable types and a 23.7 to 8.7 MB binary when it was made (docs/log.md, 09-10). What the smaller VMMs offer in boot time was measured and taken here.

Not compared: memory a VM costs its host beyond the guest's RAM. The only number here is an estimate, ~50 MB, a subtraction made in passing in docs/memory-and-disk.md, and Firecracker's published figure is measured another way. A probe that measures ours is being added; its result goes in a comment here. What a move would buy is attack surface, and its price is the block graph and the long life of a checkpoint.

If attack surface becomes the reason, the candidate is Cloud Hypervisor, and one experiment on construct decides it: checkpoint pause, restore on demand, and disk attach and detach, against today's QEMU. If the pause comes out near today's, what is left is extended_l2 and a live snapshot, both as contributions upstream.

Sources

• libkrun: include/libkrun.h and src/devices/Cargo.toml on main.
• Cloud Hypervisor: docs/snapshot_restore.md, docs/live_migration.md, docs/hotplug.md, docs/memory.md, docs/vsock.md, docs/cpu.md, release-notes.md, the OpenAPI file, block/src/formats/qcow/header.rs.
• Firecracker: docs/snapshotting/snapshot-support.md, docs/snapshotting/versioning.md, docs/device-hotplug.md, docs/memory-hotplug.md, docs/block.md, the swagger file.