Layered/incremental snapshots by ludfjig · Pull Request #1926 · hyperlight-dev/hyperlight

ludfjig · 2026-10-12T20:00:48Z

Summary

Snapshots share immutable memory layers. A capture copies only the pages changed since its parent snapshot, and Sandbox::snapshot switches the sandbox to the capture so the next one is incremental too. This makes snapshot() much faster for larger sandboxes. It also improves memory density: each snapshot holds only its changed pages and shares the rest with its parent, where main holds a full copy of guest memory per snapshot. See HIP 0003.

Deviations from the merged HIP 0003 draft (the HIP is updated):

The VM maps each layer whole, one mapping per layer. A layer stays in a snapshot while it gives at least one page. Its pages that a newer layer replaced stay mapped, so a guest can read data it overwrote before a snapshot from the older layers of that snapshot.
Each blob holds guest data followed by the page tables of the snapshot that created it, as on main. A restore copies the newest page tables into scratch.
Sandbox::snapshot installs the capture, because otherwise every later capture copies all pages changed since the last restore again. Installing removes map_region and map_file_cow regions, because the snapshot holds their mapped pages and later blobs may reuse their addresses.
Snapshots from earlier versions are rejected instead of loaded as one layer. This is simpler, and there is little reason to keep backwards compatibility here.

Changes

Snapshot memory is a list of immutable shared blobs, one per layer, each mapped whole.
Capture keeps parent layers that still give a page and copies changed pages into one new blob.
Sandbox::snapshot installs the capture into the running sandbox. Scratch pages it frees are zeroed.
The OCI format stores ordered data layers and a transport layer (config v4, ABI 7). The config names the layer whose page tables a restore uses.
Snapshot performance: adjacent page-table walk mappings are merged, and page-table pages are cached during capture.
KVM: KVM_X86_QUIRK_SLOT_ZAP_ALL is disabled where supported, so deleting a memory slot flushes only that slot.
New benchmarks snapshots/call_and_create and snapshots/restore_alternate cover the cost moved into the next call and restores to a non-current snapshot.

Breaking changes

Sandbox::snapshot removes map_region and map_file_cow regions from the sandbox, which only restore did before. Region pages the guest has not mapped become unreachable at the snapshot instead of at the next restore. Snapshots hold the same pages as before.
PtRootFinder receives a SnapshotMemoryReader and returns Result<Vec<u64>>.
A snapshot holds at most 30 memory layers, and a capture fails when it would exceed this. This bounds restore cost by limiting the number of hypercalls to map memory.

Commits

Add call_and_create and restore_alternate snapshot benchmarks. Lands first so the benchmarks run before and after.
Implement layered and incremental snapshots.
Install captured snapshots into the sandbox.
vmem: merge adjacent mappings in walk_va_spaces.
Cache page-table pages in PageTableReader.
Disable KVM_X86_QUIRK_SLOT_ZAP_ALL.
PR feedback.

Benchmarks

Compared to main. restore, restore_alternate and call_and_create compare to the first commit, which adds the last two.

MSHV: Linux 6.6 mshv.
KVM: Linux 6.17.

Highlights

Benchmark	MSHV	KVM
`snapshots/create/default`	440 µs → 249 µs (−43%)	380 µs → 262 µs (−31%)
`snapshots/create/large`	171 ms → 1.68 ms (−99%)	162 ms → 1.7 ms (−99%)
`snapshots/call_and_create/medium`	38.1 ms → 726 µs (−98%)	35 ms → 778 µs (−98%)
`snapshots/restore_alternate/large`	4.41 ms → 106 µs (−98%)	1.66 ms → 131 µs (−92%)
`snapshots/restore/large`	1.2 ms → 91.6 µs (−92%)	403 µs → 52 µs (−87%)

Guest calls show no consistent change.

Regressions

Benchmark	MSHV	KVM	Cause
`snapshot_files/load_snapshot_unverified/default`	88.5 µs → 112 µs (+27%)	93.3 µs → 123 µs (+31%)	Opens and maps 2 data blobs instead of 1
`snapshot_files/load_snapshot/default`	617 µs → 701 µs (+14%)	613 µs → 701 µs (+14%)	Also hashes 88 KiB more: base layer pages that the newer layer replaced, and the base layer's page tables
`snapshot_files/save_snapshot/default`	853 µs → 941 µs (+10%)	35.5 ms → 43.4 ms (+22%)	Writes, hashes and fsyncs 2 data blobs and 88 KiB more. fsync is slow on the KVM host
`snapshot_files/cold_start_via_snapshot/default`	1.8 ms → 1.97 ms (+9%)	2.59 ms → 2.85 ms (+10%)	Includes `load_snapshot`
`sandboxes/sandbox_from_snapshot/small`	823 µs → 812 µs (~)	877 µs → 978 µs (+12%)	Maps 2 layers instead of 1. Each costs a memory slot on KVM and a map call on MSHV. MSHV shows it only at `default` (+11%)

Kernels before 6.12 cannot disable KVM_X86_QUIRK_SLOT_ZAP_ALL, so each memory slot change flushes all guest mappings. WHP and aarch64 were not benchmarked.

A saved snapshot stores each layer in full. Base layer pages that a newer layer replaced stay in the file, which adds 88 KiB for default.

Closes #1956
Closes #1551

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs agreement between a 0.5 s and a 3 s run, or a 3 s run alone.

snapshots

Benchmark	Size	MSHV	KVM
`call_and_create`	default	556 µs → 372 µs (−33%)	471 µs → 409 µs (−13%)
`call_and_create`	small	2.67 ms → 416 µs (−84%)	2.65 ms → 466 µs (−82%)
`call_and_create`	medium	38.1 ms → 726 µs (−98%)	35 ms → 778 µs (−98%)
`call_and_create`	large	172 ms → 1.78 ms (−99%)	161 ms → 1.85 ms (−99%)
`create`	default	440 µs → 249 µs (−43%)	380 µs → 262 µs (−31%)
`create`	small	2.57 ms → 291 µs (−89%)	2.47 ms → 324 µs (−87%)
`create`	medium	41.4 ms → 606 µs (−99%)	36.9 ms → 606 µs (−98%)
`create`	large	171 ms → 1.68 ms (−99%)	162 ms → 1.7 ms (−99%)
`restore`	default	64.9 µs → 65.7 µs (~)	20.6 µs → 21 µs (~)
`restore`	small	66.3 µs → 67 µs (~)	21.4 µs → 21.5 µs (~)
`restore`	medium	74.5 µs → 71.1 µs (−5%)	26.4 µs → 25.7 µs (~)
`restore`	large	1.2 ms → 91.6 µs (−92%)	403 µs → 52 µs (−87%)
`restore_alternate`	default	164 µs → 78.7 µs (−52%)	248 µs → 96.5 µs (−61%)
`restore_alternate`	small	270 µs → 79.7 µs (−71%)	268 µs → 93 µs (−65%)
`restore_alternate`	medium	901 µs → 83.4 µs (−91%)	255 µs → 103 µs (−60%)
`restore_alternate`	large	4.41 ms → 106 µs (−98%)	1.66 ms → 131 µs (−92%)

guest_calls

Benchmark	Size	MSHV	KVM
`call`	default	45.8 µs → 45.3 µs (~)	18.4 µs → 17.3 µs (~)
`call`	small	46.2 µs → 43.8 µs (−5%)	18.5 µs → 17.7 µs (~)
`call`	medium	47.3 µs → 43.8 µs (~)	19.8 µs → 17.3 µs (~)
`call`	large	45.4 µs → 44 µs (~)	18.5 µs → 18 µs (~)
`call_with_host_function`	default	77.5 µs → 75.2 µs (~)	32.9 µs → 34.5 µs (~)
`call_with_host_function`	small	75.4 µs → 74.1 µs (~)	32.8 µs → 34.7 µs (~)
`call_with_host_function`	medium	74 µs → 75.2 µs (~)	34.8 µs → 32.8 µs (~)
`call_with_host_function`	large	73.5 µs → 75.2 µs (~)	33.5 µs → 32.7 µs (~)
`call_with_restore`	default	140 µs → 133 µs (~)	64 µs → 62.9 µs (~)
`call_with_restore`	small	141 µs → 140 µs (~)	64.4 µs → 63.8 µs (~)
`call_with_restore`	medium	147 µs → 138 µs (~)	70.4 µs → 67.9 µs (~)
`call_with_restore`	large	201 µs → 166 µs (~)	102 µs → 89.1 µs (~)
`different_thread`	-	75.5 µs → 74 µs (~)	42.4 µs → 42.9 µs (~)
`interrupt_latency`	-	47.8 µs → 48.9 µs (~)	32.8 µs → 35.9 µs (~)

guest_functions_with_large_parameters

Benchmark	Size	MSHV	KVM
`guest_call_with_large_parameters`	-	164 ms → 163 ms (~)	872 ms → 916 ms (~)

sample_workloads

Benchmark	Size	MSHV	KVM
`24K_in_8K_out_c`	-	56.3 µs → 56.5 µs (~)	26.9 µs → 27.6 µs (~)
`24K_in_8K_out_rust`	-	54.6 µs → 56.1 µs (~)	26.6 µs → 27.9 µs (~)

sandboxes

Benchmark	Size	MSHV	KVM
`create_initialized`	default	1.59 ms → 1.72 ms (~)	2.62 ms → 2.8 ms (~)
`create_initialized`	small	4.44 ms → 4.35 ms (~)	4.72 ms → 4.58 ms (~)
`create_initialized`	medium	11.8 ms → 11.8 ms (~)	9.9 ms → 9.81 ms (~)
`create_initialized`	large	37.8 ms → 38.2 ms (~)	27.6 ms → 27.5 ms (~)
`create_initialized_and_drop`	default	22.4 ms → 21.4 ms (~)	10.9 ms → 14.1 ms (~)
`create_initialized_and_drop`	small	29.4 ms → 28.9 ms (~)	14.8 ms → 13.3 ms (~)
`create_initialized_and_drop`	medium	31.9 ms → 32.2 ms (~)	19.9 ms → 20.7 ms (~)
`create_initialized_and_drop`	large	58.8 ms → 57.9 ms (~)	41.7 ms → 39.6 ms (~)
`sandbox_from_snapshot`	default	599 µs → 665 µs (+11%)	865 µs → 971 µs (~)
`sandbox_from_snapshot`	small	823 µs → 812 µs (~)	877 µs → 978 µs (+12%)
`sandbox_from_snapshot`	medium	2.17 ms → 2.02 ms (~)	975 µs → 1.01 ms (~)
`sandbox_from_snapshot`	large	6.29 ms → 6.06 ms (~)	1.06 ms → 1.22 ms (+15%)

snapshot_files

Benchmark	Size	MSHV	KVM
`cold_start_via_evolve`	default	1.67 ms → 1.61 ms (~)	3.05 ms → 2.96 ms (~)
`cold_start_via_evolve`	small	4.74 ms → 4.55 ms (~)	5.12 ms → 5.76 ms (~)
`cold_start_via_evolve`	medium	12 ms → 12.3 ms (~)	10.3 ms → 11 ms (~)
`cold_start_via_evolve`	large	37.6 ms → 38.8 ms (~)	27.9 ms → 29.1 ms (~)
`cold_start_via_snapshot`	default	1.8 ms → 1.97 ms (+9%)	2.59 ms → 2.85 ms (+10%)
`cold_start_via_snapshot`	small	9.14 ms → 8.93 ms (~)	8.55 ms → 8.95 ms (~)
`cold_start_via_snapshot`	medium	54.7 ms → 53.8 ms (~)	49.8 ms → 50.9 ms (~)
`cold_start_via_snapshot`	large	204 ms → 205 ms (~)	193 ms → 194 ms (~)
`cold_start_via_snapshot_unverified`	default	1.31 ms → 1.4 ms (~)	2.06 ms → 2.2 ms (~)
`cold_start_via_snapshot_unverified`	small	2.18 ms → 2.08 ms (−4%)	2.03 ms → 2.17 ms (~)
`cold_start_via_snapshot_unverified`	medium	3.68 ms → 3.59 ms (~)	2.2 ms → 2.25 ms (~)
`cold_start_via_snapshot_unverified`	large	7.7 ms → 7.73 ms (~)	2.33 ms → 2.59 ms (~)
`load_snapshot`	default	617 µs → 701 µs (+14%)	613 µs → 701 µs (+14%)
`load_snapshot`	small	6.65 ms → 6.71 ms (~)	6.49 ms → 6.68 ms (~)
`load_snapshot`	medium	50.3 ms → 50.4 ms (~)	48.1 ms → 48.8 ms (~)
`load_snapshot`	large	197 ms → 198 ms (~)	190 ms → 192 ms (~)
`load_snapshot_unverified`	default	88.5 µs → 112 µs (+27%)	93.3 µs → 123 µs (+31%)
`load_snapshot_unverified`	small	89.6 µs → 113 µs (+26%)	95.1 µs → 119 µs (+25%)
`load_snapshot_unverified`	medium	90.4 µs → 114 µs (+26%)	96.3 µs → 120 µs (+24%)
`load_snapshot_unverified`	large	94 µs → 118 µs (+26%)	99 µs → 125 µs (+27%)
`save_snapshot`	default	853 µs → 941 µs (+10%)	35.5 ms → 43.4 ms (+22%)
`save_snapshot`	small	8.97 ms → 9.09 ms (~)	55.2 ms → 61.5 ms (~)
`save_snapshot`	medium	68.2 ms → 69.6 ms (~)	462 ms → 402 ms (~)
`save_snapshot`	large	270 ms → 271 ms (~)	2.03 s → 1.75 s (~)

Copilot

🔵 Needs a closer look

The VMM memory boundary changes substantially, and OCI loading maps oversized blobs before enforcing resource limits.

1 open finding

Preflight descriptor sizes before mapping to prevent excessive reservations · New

What changed in this PR

Adds layered incremental snapshots that share immutable memory, persist OCI data and page-table layers, and optimize restore mappings.

Changes:

Adds shared snapshot layers with incremental capture and installation.
Introduces snapshot ABI 6 and OCI config v4.
Optimizes page-table walks, KVM invalidation, tests, and benchmarks.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Adds zero-init mapping test support.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Bumps golden snapshots to v6.
`src/hyperlight_host/src/sandbox/uninitialized.rs`	Documents capture behavior and uses GPA spans.
`src/hyperlight_host/src/sandbox/trace/mem_profile.rs`	Reads layered memory by GPA.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pins ABI 6 media types.
`src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs`	Defines layered snapshot memory.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Maps and accesses snapshot layers.
`src/hyperlight_host/src/sandbox/snapshot/file/mod.rs`	Saves and loads layered OCI snapshots.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Adds config v4 and new layer media types.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Defines the layered snapshot schema.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Tests layered persistence and validation.
`src/hyperlight_host/src/sandbox/mod.rs`	Exports `SnapshotMemoryReader`.
`src/hyperlight_host/src/sandbox/initialized.rs`	Installs captures and updates snapshot APIs.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Updates scratch transport tests.
`src/hyperlight_host/src/mem/shared_mem.rs`	Adds immutable memory and range mappings.
`src/hyperlight_host/src/mem/layout.rs`	Removes flat snapshot layout state.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Disables the slot-wide KVM flush quirk.
`src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs`	Uses range-based snapshot mappings.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Adopts layered VM memory.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs`	Updates mapping-state fault tests.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Manages multiple snapshot mappings.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Adopts layered VM memory.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Resolves layered debug memory.
`src/hyperlight_host/build.rs`	Removes obsolete memory-read configuration.
`src/hyperlight_host/benches/benchmarks.rs`	Adds incremental snapshot benchmarks.
`src/hyperlight_common/src/vmem.rs`	Coalesces adjacent mappings.
`src/hyperlight_common/src/arch/amd64/vmem.rs`	Applies mapping coalescing on AMD64.
`src/hyperlight_common/src/arch/aarch64/vmem.rs`	Applies mapping coalescing on AArch64.
`proposals/0003-incremental-snapshots/README.md`	Updates the snapshot design.
`docs/snapshot-versioning.md`	Documents ABI 6 versioning.
`docs/snapshot-oci-format.md`	Documents layered OCI storage.
`docs/paging-development-notes.md`	Describes incremental capture behavior.
`CHANGELOG.md`	Records breaking API and format changes.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ludfjig · 2026-10-27T17:45:24Z

This mostly looks sensible (some minor nits/questions below); my main concern is with the way that the memory access code uses raw pointers to read/write from a HostSharedMemory, which does not seem like it upholds the usual shared memory invariants.

the only raw pointer read left is for map_region/map_file_cow memory, which isn't a HostSharedMemory, and the host has to keep it unmodified anyway. Main read it with from_raw_parts too. Scratch is now locked for the whole access, see this.

call_and_create times a guest call plus snapshot creation, so the call pays for memory remapping done by the previous snapshot. restore_alternate restores two snapshots in turn, so each restore targets memory that is not current. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Share immutable data blobs between related snapshots. Each blob holds guest data followed by the page tables of the snapshot that created it. Persist layered snapshots as OCI image layouts. Capture copies only pages that changed since the snapshot the sandbox was created from or last restored to. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Snapshot capture switches the running sandbox to the captured snapshot. The next capture reuses its layers and copies only pages changed after it. Scratch pages used before the capture become free again. They are zeroed so guest allocations return zeroed pages, as after restore. Capture removes map_region and map_file_cow regions. Pages of these regions that the guest has not mapped are lost. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

walk_va_spaces merges mappings that have contiguous physical and virtual addresses and the same kind. Snapshot capture classifies and rebuilds each merged mapping once. Criterion snapshots/create: * MSHV: medium 1.18 ms to 950 us (-20%), large 3.56 ms to 2.85 ms (-20%) * KVM: medium 1.34 ms to 1.04 ms (-22%), large 3.59 ms to 2.91 ms (-19%) Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

A page-table walk reads every entry through GuestPhysicalMemoryView, which resolves the backing and copies 8 bytes through a locked volatile copy. The reader keeps a copy of the last table page it read, so each table costs one copy. Snapshot capture and crash dumps use the reader. Criterion snapshots/create: * MSHV: default 344 us to 248 us (-28%), medium 950 us to 594 us (-37%) * KVM: default 499 us to 379 us (-24%), medium 1.04 ms to 746 us (-28%) Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

By default, deleting a memslot flushes the guest mappings of every memslot. KVM keeps that flush as a workaround for VMs with assigned GPUs. Hyperlight assigns no devices and disables the quirk when the kernel supports it (Linux 6.12 and later). Deleting a memslot then flushes only that memslot's mappings. Snapshot capture deletes a memslot when a layer drops out of the snapshot. Restore to another snapshot and region unmapping also delete memslots. Criterion on KVM (Linux 6.17): * snapshots/call_and_create: 24% to 58% faster * snapshots/restore_alternate: 32% to 52% faster Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

* Iterate snapshot mappings in reverse and test membership with contains. * Rename GuestVirtualMemoryReader::read and the public SnapshotMemoryReader::read to read_exact. * Move the snapshot memory test helpers into the tests module. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

* Hold the scratch lock for a whole guest memory access. GuestPhysicalMemoryView borrows scratch as a slice under the write lock, and callers get the view through a closure. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

* Rename the snapshot config field host_page_size to layer_alignment and store the alignment the layers use: the larger of the host and guest page sizes. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

* Correct the gdb write comment. Snapshot layers are writable copies in gdb builds, and only dynamic regions are refused. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

snapshot_with_options with another scratch size installs the capture, like a same-size capture. The sandbox switches to new scratch of that size, and the next capture builds on the new snapshot's layers. A new scratch size moves the scratch base and the page tables, so install replaces scratch through the restore path. The vCPU keeps its state except CR3. RNG reseed and log-level requests the guest has not applied yet carry over to the new scratch. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

The guest can map any page of a snapshot layer, including pages a later layer replaced. Sandbox::snapshot now says that data overwritten before a snapshot stays readable, and that zeroing it does not remove it. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-10-30T00:37:45Z

Benchmark Results

Measured commit: 5c1bec8c30c8
Baseline commit: a4563dbd585d

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`372.06 µs` (➖ 1.00x slower)
`byte_chunks`		`781.08 ns` (➖ 1.02x faster)
`vec_bytes`		`584.50 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`141.28 ns` (➖ 1.01x faster)
`262144`	`521.55 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`79.73 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.68 ns` (➖ 1.00x slower)	`7.68 ns` (➖ 1.00x faster)	`7.68 ns` (➖ 1.02x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`117.35 µs` (➖ 1.27x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.93 µs` (➖ 1.08x faster)	`2.05 µs` (➖ 1.07x slower)
`262144`	`7.06 µs` (➖ 1.01x slower)	`7.17 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.03 µs` (➖ 1.05x faster)	`1.10 µs` (➖ 1.08x slower)
`65536`	`6.15 µs` (➖ 1.05x faster)	`6.22 µs` (➖ 1.02x faster)
`262144`	`28.41 µs` (➖ 1.04x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`674.61 µs` (➖ 1.01x faster)
`byte_chunks`		`689.85 ns` (➖ 1.01x faster)
`vec_bytes`		`512.42 ns` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`172.10 ns` (➖ 1.01x slower)
`262144`	`503.33 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`72.79 ms` (➖ 1.01x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`6.76 ns` (➖ 1.00x faster)	`6.77 ns` (➖ 1.00x slower)	`6.77 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`54.69 µs` (➖ 1.21x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.12 µs` (➖ 1.01x faster)	`2.11 µs` (➖ 1.02x faster)
`262144`	`7.53 µs` (➖ 1.00x slower)	`7.53 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`703.89 ns` (➖ 1.02x faster)	`708.51 ns` (➖ 1.02x faster)
`65536`	`7.03 µs` (➖ 1.01x faster)	`7.05 µs` (➖ 1.01x faster)
`262144`	`29.85 µs` (➖ 1.03x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`300.63 µs` (➖ 1.06x slower)
`byte_chunks`		`963.42 ns` (➖ 1.02x slower)
`vec_bytes`		`725.99 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`192.96 ns` (➖ 1.10x faster)
`262144`	`730.59 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`63.96 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.61 ns` (➖ 1.01x slower)	`9.54 ns` (➖ 1.10x faster)	`10.28 ns` (➖ 1.06x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`109.90 µs` (➖ 1.25x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.27 µs` (➖ 1.01x slower)	`2.40 µs` (➖ 1.03x faster)
`262144`	`9.70 µs` (➖ 1.06x faster)	`9.83 µs` (➖ 1.06x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.32 µs` (➖ 1.07x slower)	`1.25 µs` (➖ 1.03x faster)
`65536`	`8.16 µs` (➖ 1.19x faster)	`9.41 µs` (➖ 1.03x faster)
`262144`	`35.73 µs` (➖ 1.06x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`676.65 µs` (➖ 1.02x slower)
`byte_chunks`		`900.50 ns` (➖ 1.01x slower)
`vec_bytes`		`651.82 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`254.08 ns` (➖ 1.20x faster)
`262144`	`646.69 ns` (➖ 1.14x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`66.92 ms` (➖ 1.04x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.79 ns` (➖ 1.01x slower)	`8.69 ns` (➖ 1.00x slower)	`8.72 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`52.19 µs` (➖ 1.19x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.30 µs` (➖ 1.00x slower)	`2.28 µs` (➖ 1.01x faster)
`262144`	`8.23 µs` (➖ 1.04x slower)	`7.92 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`893.77 ns` (➖ 1.03x faster)	`890.09 ns` (➖ 1.04x faster)
`65536`	`7.47 µs` (➖ 1.00x slower)	`7.51 µs` (➖ 1.00x slower)
`262144`	`39.78 µs` (➖ 1.05x slower)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`1.87 ms` (➖ 1.02x faster)
`byte_chunks`		`1.17 µs` (➖ 1.01x faster)
`vec_bytes`		`769.02 ns` (➖ 1.07x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`229.63 ns` (➖ 1.22x faster)
`262144`	`784.87 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`74.26 ms` (➖ 1.05x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.93 ns` (➖ 1.10x slower)	`9.99 ns` (➖ 1.01x faster)	`9.94 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`846.57 µs` (➖ 1.06x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.36 µs` (➖ 1.17x faster)	`2.36 µs` (➖ 1.01x slower)
`262144`	`9.15 µs` (➖ 1.04x faster)	`9.28 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.29 µs` (➖ 1.03x faster)	`1.27 µs` (➖ 1.05x faster)
`65536`	`9.08 µs` (➖ 1.02x faster)	`9.56 µs` (➖ 1.19x faster)
`262144`	`38.62 µs` (➖ 1.19x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.91 ms` (➖ 1.09x faster)
`byte_chunks`		`1.17 µs` (➖ 1.03x faster)
`vec_bytes`		`756.59 ns` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`213.84 ns` (➖ 1.00x faster)
`262144`	`753.42 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`99.19 ms` (➖ 1.07x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.27 ns` (➖ 1.02x slower)	`9.62 ns` (➖ 1.00x faster)	`9.66 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`658.84 µs` (➖ 1.18x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.44 µs` (➖ 1.05x slower)	`2.30 µs` (➖ 1.00x faster)
`262144`	`7.47 µs` (➖ 1.02x faster)	`7.65 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.08 µs` (➖ 1.00x slower)	`1.19 µs` (➖ 1.11x slower)
`65536`	`8.30 µs` (➖ 1.18x faster)	`8.22 µs` (➖ 1.00x faster)
`262144`	`44.14 µs` (➖ 1.17x faster)

_{Reported by cargo ci bench-report --candidate run:40976317940 --config-file bench_report.toml.}

addressed

ludfjig added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. regen-goldens Regenerate snapshot golden fixtures labels Oct 12, 2026

Copilot AI balanced review requested due to automatic review settings October 12, 2026 20:00

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 12, 2026

ludfjig requested review from danbugs, dblnz and devigned as code owners October 12, 2026 20:00

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026

ludfjig requested review from jprendes, jsturtevant and syntactically as code owners October 12, 2026 20:00

ludfjig added the regen-goldens Regenerate snapshot golden fixtures label Oct 12, 2026

ludfjig requested review from andreiltd and simongdavies as code owners October 12, 2026 20:00

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 12, 2026

ludfjig requested a review from squillace as a code owner October 12, 2026 20:00

ludfjig added this pull request to stack #1929 October 12, 2026 20:01

ludfjig mentioned this pull request Oct 12, 2026

Layered/incremental snapshots #1902

Closed

Copilot started reviewing on behalf of ludfjig October 12, 2026 20:05

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/snapshot/file/mod.rs Resolved

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/1-layered-snapshots branch 2 times, most recently from 67c5cc8 to 7ca7b5e Compare October 13, 2026 21:01

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/1-layered-snapshots branch from 7ca7b5e to 67561dc Compare October 13, 2026 23:00

This comment has been minimized.

Sign in to view

andreiltd reviewed Oct 14, 2026

View reviewed changes

Comment thread src/hyperlight_common/src/arch/aarch64/vmem.rs Resolved

andreiltd reviewed Oct 14, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs Resolved

andreiltd reviewed Oct 14, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/mem/mgr.rs Resolved

andreiltd reviewed Oct 14, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/mem/mgr.rs Resolved

This comment has been minimized.

Sign in to view

jsturtevant reviewed Oct 28, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs Resolved

jsturtevant previously approved these changes Oct 28, 2026

View reviewed changes

ludfjig dismissed jsturtevant’s stale review via a837c68 October 29, 2026 20:01

ludfjig force-pushed the ll/1-layered-snapshots branch from cfaa18c to a837c68 Compare October 29, 2026 20:01

This comment has been minimized.

Sign in to view

ludfjig added 12 commits October 29, 2026 16:50

PR feedback

f8a17f6

* Hold the scratch lock for a whole guest memory access. GuestPhysicalMemoryView borrows scratch as a slice under the write lock, and callers get the view through a closure. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

PR feedback

f9873fa

* Rename the snapshot config field host_page_size to layer_alignment and store the alignment the layers use: the larger of the host and guest page sizes. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

PR feedback

fc6ba7d

* Correct the gdb write comment. Snapshot layers are writable copies in gdb builds, and only dynamic regions are refused. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

ludfjig force-pushed the ll/1-layered-snapshots branch from 07fcbff to 5c1bec8 Compare October 29, 2026 23:55

jsturtevant reviewed Oct 30, 2026

View reviewed changes

Comment thread proposals/0003-incremental-snapshots/README.md Resolved

jsturtevant approved these changes Oct 30, 2026

View reviewed changes

ludfjig mentioned this pull request Oct 30, 2026

Document updated threat-model after #1926 #2002

Open

ludfjig merged commit b99eb5f into main Oct 30, 2026
71 checks passed

ludfjig deleted the ll/1-layered-snapshots branch October 30, 2026 18:45

github-actions Bot removed the ready-for-review PR is ready for (re-)review label Oct 30, 2026

Conversation

ludfjig commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Breaking changes

Commits

Benchmarks

Highlights

Regressions

Full results

snapshots

guest_calls

guest_functions_with_large_parameters

sample_workloads

sandboxes

snapshot_files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ludfjig commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Oct 30, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

ludfjig commented Oct 12, 2026 •

edited

Loading

ludfjig commented Oct 27, 2026 •

edited

Loading