Skip to content

[DO NOT REVIEW] openvmm_core: run a quiesced microVM again only through its rollback - #5010

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
feature/microvm/devfrom
copilot/nvx-issue-1170-snapshot-resume-bypass
Nov 10, 2026
Merged

[DO NOT REVIEW] openvmm_core: run a quiesced microVM again only through its rollback#5010
Pedro Henrique Penna (ppenna) merged 2 commits into
feature/microvm/devfrom
copilot/nvx-issue-1170-snapshot-resume-bypass

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Makes the VM worker run a quiesced microVM again only through the snapshot's rollback, for microsoft/nvx#1170. Once the worker had quiesced a microVM for a guest-requested snapshot and handed its state to the snapshot controller, an ordinary Resume could run the VM again without ResumeAfterFailedSnapshot: ReleaseSnapshotBoundary released the boundary of the quiesced VM, after which nothing refused the resume, and QuiesceForSnapshot stopped and saved a running VM that held no boundary, so that none refused it.

Two commits on 46bfe02ed, the head of feature/microvm/dev. microsoft/nvx#1430 pins this head.

Let a microVM test be its snapshot controller (b5d20bf84)

  • Need. Petri launched the VM worker without the snapshot boundary's channels, so the snapshot-request port let each guest request continue at once, as it does without a snapshot destination, and no VMM test could take the snapshot controller's part, let alone send the controller's requests in an order that the controller does not.
  • Change. PetriVmConfigOpenVmm::with_microvm_snapshot_boundary gives the port and the VM worker the boundary's channels, and keeps the worker's reports of its boundaries for the test. PetriVmOpenVmm then waits for a boundary with wait_for_microvm_snapshot_boundary, and sends the controller's requests with quiesce_microvm_for_snapshot, release_microvm_snapshot_boundary, and resume_microvm_after_failed_snapshot. A VM launched without the option is launched as before. The hooks in petri's upstream files are a parameter of Worker::launch and an accessor of the worker's RPC channel; the rest is in petri's microVM module.
  • Test. snapshot_boundary_release_and_rollback takes the controller's two ways out of a boundary: it releases the guest's first boundary without a snapshot, and quiesces the VM for the second one's snapshot and then rolls the snapshot back. Each time, an ordinary resume fails, and the guest continues after its request once the test lets it.

Run a quiesced microVM again only through its rollback (4c2697aef)

  • Cause. The snapshot boundary is what refuses an ordinary resume (snapshot_rpc::filter refuses Resume while the boundary's vCPU stop guard is held), and two requests could leave a quiesced VM without one. release_snapshot_boundary checked only that a boundary was active, so after QuiesceForSnapshot it resumed host input, completed the guest's snapshot transaction, and dropped the stop guard of the stopped VM, and the following Resume started it. quiesce_for_snapshot checked only that the VM ran, so it stopped and saved a VM that held no boundary, and the following Resume started it at once. The VM controller of openvmm_entry sends neither order: it quiesces only at a boundary that the worker reported, and after a quiesce it ends the source or rolls the snapshot back. But the capture's guarantee that a source whose snapshot may be published runs again only through the rollback rested on that order alone.
  • Change. release_snapshot_boundary now refuses the boundary of a quiesced VM, before it resumes input or changes the boundary, so only the rollback, which starts the state units before it releases the boundary, releases it, and the boundary keeps refusing an ordinary resume until then. quiesce_for_snapshot rejects a VM that holds no boundary with SnapshotQuiesceError::Rejected, before it stops anything. The snapshot guide's paragraph on the RPCs that a boundary refuses says that the worker refuses these orders too.
  • Scope. The controller's own orders are unchanged: a release without a snapshot, after a missing destination, a rejected preflight, or a rejected quiesce, finds the VM running, and so does the rollback's own release, which runs after the units start. A rollback request sent outside any boundary is left as it is: it does not bypass the rollback, and hardening it is a separate change.
  • Tests. snapshot_quiesce_resumes_only_on_rollback quiesces the VM at the guest's boundary and requires the boundary's release and an ordinary resume to fail, the partition's state unit to stay stopped, and the guest not to continue for 5 s, and then the rollback to let the guest continue and power off. snapshot_quiesce_requires_a_boundary quiesces a running VM whose guest requested no snapshot and requires a rejection that leaves the partition's state unit running. Without the fix, the release and the resume succeed, the partition runs, and the guest continues, and the quiesce outside a boundary returns the VM's saved state and leaves the partition stopped.

Validation

With Rust 1.97.1, at this head:

  • Windows: cargo clippy --all-targets for openvmm_core, petri, and vmm_tests, and cargo doc --no-deps for the three, with warnings denied, without warnings; cargo nextest run --profile agent for openvmm_core and petri, 152 passed; cargo xtask fmt, clean.

  • VMM tests, with cargo xflowey vmm-tests-run selecting the three new tests and phase_1_lifecycle, on NVX's test virtual machines and locally on WHP. "Base" is this head with the worker change reverted:

    Host Backend Base This head
    nvx-kvm-test KVM the two regression tests fail; the other two pass 4 passed
    nvx-mshv-test MSHV the two regression tests fail; the other two pass 4 passed
    nvx-whp-test WHP the two regression tests fail; the other two pass 4 passed
    local WHP the two regression tests fail; the other two pass 4 passed

  • NVX's snapshot-core, snapshot-rollback, and snapshot-tiers microVM scenarios, with 1 and 4 vCPUs, pass on the three hosts with OpenVMM built from this change's code.

This head's OpenVMM PR run 42785708766 ran the three new tests and phase_1_lifecycle in its run vmm-tests jobs on x64-linux-amd-kvm, x64-linux-intel-mshv, x64-windows-intel, and x64-windows-amd, where they all passed, and passed every job but x64-windows-amd-snp, which failed in 5 Hyper-V OpenHCL SEV-SNP TPM tests that this change does not touch.

On MSHV, a guest of petri's packaged kernel never wakes from sleep: its LAPIC timer is masked, and no PIT interrupt wakes the halted vCPU, with or without a snapshot boundary. So no test here waits on a guest timer.

Landing

microsoft/nvx#1430 pins this head. Land it by fast-forwarding feature/microvm/dev to 4c2697aef28d7fa69f788ce4e65b6554c1cfb935, since this repository's merge button squashes and would leave NVX's pin off the branch.

@ppenna
petri: let a microVM test be its snapshot controller
b5d20bf 
A microVM's guest requests a snapshot by writing to the
snapshot-request port, and the VM worker then holds it at a snapshot
boundary, with host input gated and the vCPUs stopped, until its
snapshot controller, the VM controller of openvmm_entry, releases the
boundary without a snapshot, or quiesces the VM for the snapshot and
then ends it or rolls the snapshot back. Petri launched the VM worker
without the boundary channels, so the port let each request continue
at once, as it does without a snapshot destination, and no VMM test
could take the controller's part, let alone send the controller's
requests in an order that the controller does not.

PetriVmConfigOpenVmm::with_microvm_snapshot_boundary gives the port and
the VM worker the boundary channels, and keeps the worker's reports of
its boundaries for the test. PetriVmOpenVmm then waits for a boundary
with wait_for_microvm_snapshot_boundary, and sends the controller's
requests with quiesce_microvm_for_snapshot,
release_microvm_snapshot_boundary, and
resume_microvm_after_failed_snapshot. A VM launched without the option
is launched as before.

Test: as a microVM's snapshot controller, the test releases the
guest's first snapshot boundary without a snapshot, and quiesces the
VM for the second one's snapshot and then rolls the snapshot back.
Each time, an ordinary resume does not pass the boundary, and the
guest continues after its request once the test lets it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
openvmm_core: run a quiesced microVM again only through its rollback
4c2697a 
Once the VM worker had quiesced a microVM for a snapshot and handed its
state to the snapshot controller, an ordinary Resume could run the VM
again without the snapshot's rollback, in two ways
(microsoft/nvx#1170):

- ReleaseSnapshotBoundary released the boundary of a quiesced VM: it
  resumed host input, completed the guest's snapshot transaction, and
  dropped the vCPU stop guard of the stopped VM. With the boundary
  gone, nothing refused a Resume, which started the VM again.
- QuiesceForSnapshot stopped and saved a running VM that held no
  snapshot boundary, so no boundary refused the Resume that followed.

The controller sends neither order: it quiesces only at a boundary that
the worker reported, and after a quiesce it ends the source or rolls
the snapshot back with ResumeAfterFailedSnapshot, which starts the
state units before it releases the boundary. But the worker relied on
that order for the capture's guarantee that a source whose snapshot
may be published runs again only through the rollback.

The worker now holds it itself. release_snapshot_boundary refuses the
boundary of a quiesced VM, before it resumes input or changes the
boundary, so only the rollback, which runs the VM first, releases it,
and the boundary keeps refusing an ordinary Resume until then.
quiesce_for_snapshot rejects a VM that holds no snapshot boundary with
SnapshotQuiesceError::Rejected, without stopping it. The controller's
own orders are unchanged: a release without a snapshot, and the
rollback's release, both find the VM running. The OpenVMM Guide says
that the worker refuses the other orders.

Tests, with the test as the microVM's snapshot controller: after a
quiesce at the guest's boundary, the boundary's release and an ordinary
resume both fail, the partition's state unit stays stopped, and the
guest does not continue for 5 s, and then the rollback lets it continue
and power off; and a quiesce of a running VM that requested no
snapshot is rejected and leaves the partition's state unit running.
Without the fix, the release and the resume succeed, the partition
runs, and the guest continues, and the quiesce outside a boundary
returns the VM's saved state and leaves the partition stopped.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. labels Nov 10, 2026
@ppenna Pedro Henrique Penna (ppenna) mentioned this pull request Nov 10, 2026
Merged
@ppenna
Pedro Henrique Penna (ppenna) merged commit 4c2697a into feature/microvm/dev Nov 10, 2026
73 of 74 checks passed
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown

6 Petri tests failed (1 unstable)

syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 11, 2026
@ppenna
openvmm: pin the rollback as the only way to run a quiesced microVM
1c08dcc 
An ordinary resume could run a microVM again after the VM worker had
quiesced it for a snapshot and handed its state to the snapshot
controller, without the snapshot's rollback (microsoft#1170). The worker
released the snapshot boundary of a quiesced VM, after which nothing
refused a resume, and it quiesced a running VM that held no snapshot
boundary, so that none refused one. NVX's controller sends neither
order, but the capture's guarantee that a source runs again only
through the rollback of a failed capture rested on that alone.

Pin 4c2697aef28d7fa69f788ce4e65b6554c1cfb935, the head of
microsoft/openvmm#5010, two commits on 46bfe02ed, the head of
feature/microvm/dev:

- b5d20bf84 petri: let a microVM test be its snapshot controller
- 4c2697aef openvmm_core: run a quiesced microVM again only through
  its rollback

The revision has the worker refuse to release the boundary of a
quiesced VM, before it resumes host input or changes the boundary, so
that only the rollback, which starts the VM's state units first,
releases it; and reject a quiesce outside a guest-requested boundary
without stopping the VM. Its VMM tests take the snapshot controller's
part, which petri can now do.

Evidence: with the worker change reverted on the revision, both new
VMM tests fail on nvx-kvm-test (KVM), nvx-mshv-test (MSHV), and
nvx-whp-test (WHP), all virtual machines, and locally on WHP: the
release of a quiesced VM's boundary and an ordinary resume succeed,
the partition runs, and the guest continues; and a quiesce outside a
boundary returns the saved state of all 14 state units and leaves the
partition stopped. At the revision they pass on all four, as do the
test of the controller's own orders and phase_1_lifecycle. NVX's
snapshot-core, snapshot-rollback, and snapshot-tiers scenarios, with 1
and 4 vCPUs, pass on the three hosts with OpenVMM built from the
revision's code.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 11, 2026
@ppenna
doc: map the rollback-only resume in the OpenVMM roadmap
d4878c2 
Maps NVX's new pin, 4c2697aef, the head of microsoft/openvmm#5010,
at position 240, after position 239, b5d20bf84, its other commit,
which follows position 238, 46bfe02ed, the head of feature/microvm/dev,
which microsoft/openvmm#5005 merged after microsoft/openvmm#5004 and
microsoft/openvmm#5003. The pin is 109 commits past the mapped tip, and
positions 132-240 stay unassigned.

Positions 239 and 240 change the snapshot boundary of PR17. Position
239 lets a VMM test be the snapshot controller: it gives the boundary's
channels to petri's handle of the snapshot-request port, which PR17's
position 076 adds without one, and to petri's launch of the VM worker,
whose boundary parameters position 072 adds, keeps them in the petri
microVM support of PR15's position 065, and adds a test beside position
065's cold-boot test. Position 240 has the worker of position 072
refuse to release the boundary of a quiesced VM and reject a quiesce
outside a boundary, extends the snapshot guide's paragraph that
position 076 adds, and adds tests beside position 239's. Replaying them
on upstream main conflicts in every file but petri's start of the VM,
since upstream main lacks the petri microVM support of position 065
and the boundary of positions 072 and 076, so they follow PR15 and PR17
upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 11, 2026
@ppenna
doc: map the terminal state-unit fixups in the OpenVMM roadmap
b547a2a 
Map positions 241 and 242, 78e752de1 and 6ddd6ca50, which
microsoft/openvmm#5012 proposes for feature/microvm/dev on its head,
position 240, 4c2697aef, which microsoft/openvmm#5010 merged, and move
NVX's pin to position 242, 111 commits past the mapped tip. Position
241 has a stop of the state units also stop a unit in the terminal
state in which PR09's position 015 leaves a unit whose task does not
answer its start, and has a reset, save, restore, or time advance fail
while a unit is in that state, which only a later start refused.
Position 242 has the VM worker tear the VM down when the start again of
a failed restart, which position 015 adds, fails, and has its teardown
stop the state units when they run.

Both are therefore PR09 fixups, recorded as the PR12 fixup at v5.1
position 221 was: the DAG gives them a fixup step, PR09's summary row,
graph node, and mapping count them, and the fold sentence after the
DAG sends them into position 015, and position 241's check of a time
advance, with its test, into PR10's position 019, which adds the time
advance. Replaying them on upstream main, ba105dd7f, conflicts in four
of their five files, all but the new test file, since upstream main
lacks the start rollback of position 015 and the time advance of
position 019, so they follow PR09 upstream; microsoft/openvmm#4872's
head makes the same changes in its form of position 015.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Guide release-nvx Targets a branch under feature/microvm/.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ppenna