[Pedro Henrique Penna (ppenna)]

Fixes #1138

What and why

A guest's snapshot request could crash a multi-vCPU microVM, or lose the device
accesses of its other vCPUs. A port or MMIO access that another vCPU made
during the request went unanswered, a read returning all ones and a write
lost; an interrupt from another device was dropped; and an EOI or a PCI
configuration access panicked the VM worker with "lock should not be closed".

Root cause: a snapshot boundary gates host input before it stops the vCPUs,
and resumes a gate that failed before any of them stops, so the vCPUs run
through both. OpenVMM's chipset device unit
(vmotherboard/src/chipset/backing/arc_mutex/state_unit.rs) nevertheless
changed each gate with the device's mutex closed, from the call to the end of
the change's wait, as it changes the device's other states, which run only
while the vCPUs are stopped, and a vCPU's access that met the closed mutex was
dropped. The window lasted the whole wait of a virtio transport's gate for its
device task, and of a remote device's gate for its worker, and even a gate
that completes at once, as most devices' gates do, closed the device for an
instant, which a vCPU that signals EOIs in a loop met within one to five
gates.

microsoft/openvmm#5005, vmotherboard: gate a device's input without closing it to the vCPUs, splits a gate change into two parts.
ChangeDeviceState::quiesce_input and resume_input make the synchronous
part and return what remains as an InputGateWait, an owned, boxed future
that does not borrow the device. The chipset device unit makes the call under
the device's ordinary lock, as a vCPU's access does, releases the lock, and
then awaits the wait, so a vCPU's access comes before the synchronous part or
after it and is served either way. The virtio transports and the remote device
proxy send their gate's request in the synchronous part, on the channel that
carries the commands of the guest's accesses, so that their device takes the
gate in the order of those accesses, and wait for its answer without the
device; the erased chipset devices forward both parts. Only the input gate
changed: the other state changes still close the device, since they run with
the vCPUs stopped, and the remote device worker is unchanged.

Commits:

• openvmm: pin input gating that keeps devices open to the vCPUs moves the
  gitlink, and nothing else, from cba21d109, the head of
  [DO NOT REVIEW] openvmm_helpers: refuse a FIFO artifact without waiting for a writer openvmm#5004, which openvmm: pin refusing a FIFO snapshot artifact without waiting #1424 pins, to
  46bfe02ed65697e858f66ce55eb60831b3ac3461, the head of
  [DO NOT REVIEW] vmotherboard: gate a device's input without closing it to the vCPUs openvmm#5005, one commit on cba21d109; its message carries the
  host evidence below. NVX needs no coordinated change: the control-session
  vectors, source excerpts, contract versions, and CPU profiles that nvx-rs
  copies from OpenVMM match the revision's.
• doc: map the input gating that keeps devices open in the OpenVMM roadmap
  maps position 238, unassigned, 107 commits past the mapped tip, with the
  mapped work that it changes: the input gate that PR09's position 018 adds
  and that PR13's position 040 forwards through the virtio transports. It
  conflicts with upstream main in eight of its nine files, all but the new
  file of the transports' tests, so it follows PR09 and PR13 upstream.
• doc: say which files of the input gating conflict upstream corrects that
  count in the roadmap, which said that position 238 conflicts in all eight of
  the files that it changes, after review.

Decisions:

• The device's lock is not held across the gate's wait, which would stall the
  vCPUs that access the device for the whole wait and risk a deadlock with a
  device task that needs a vCPU's access to finish; and the vCPUs' access
  paths do not tolerate a closed device, which would hide the dropped accesses
  rather than serve them.
• The split applies to the input gate only. openvmm: VPCI hot removal closes the chipset bus while VPs can still access it #1156, a VPCI hot removal that
  closes the chipset bus while the vCPUs can still access it, can use the same
  pattern later.
• A pre-existing, related defect that review found, a guest's virtio reset
  clearing a device's input gate while a boundary holds host input, is openvmm: a guest's virtio device reset lets host input past a held input gate #1416,
  outside this change.

46bfe02ed is the commit that the validation before this restack tested,
a91d4c7a4 on d6addf8c1, rebased onto cba21d109: git range-diff shows
the patch identical, and the commits between change none of its files.

Stack

This is layer 3 of a stack of four pull requests, one for each of four issues,
each based on the branch of the one below it; their OpenVMM pull requests
stacked the same way. Layers 1 and 2 have landed, so this pull request now
targets dev, and microsoft/openvmm#5005 targets feature/microvm/dev.

Layer	NVX pull request (base)	Issue	OpenVMM pull request (base)	Pinned OpenVMM revision
1, merged	#1423 (dev)	#1364	microsoft/openvmm#5003 (feature/microvm/dev), landed	ac3dc27a7, position 236
2, merged	#1424 (dev)	#1154	microsoft/openvmm#5004 (feature/microvm/dev), landed	cba21d109, position 237
3, this one	#1425 (dev)	#1138	microsoft/openvmm#5005 (feature/microvm/dev)	46bfe02ed, position 238
4	#1427 (agents/issue-1138-input-gate-lock)	#544, part 2	none	46bfe02ed, this layer's

Landing order, bottom first, one layer at a time; each OpenVMM pull request
lands before its NVX pull request merges:

1. Land [DO NOT REVIEW] openvmm_helpers: compute memory-expansion ranges without overflow openvmm#5003 by fast-forwarding feature/microvm/dev to
   ac3dc27a738808d38e1148d3caefc5e89111ee32, since OpenVMM's merge button
   squashes and would leave the pin off the branch. Then merge openvmm: pin overflow-free memory-expansion ranges #1423
   through the merge queue; GitHub deletes its branch and retargets openvmm: pin refusing a FIFO snapshot artifact without waiting #1424
   to dev.
2. Retarget [DO NOT REVIEW] openvmm_helpers: refuse a FIFO artifact without waiting for a writer openvmm#5004 to feature/microvm/dev
   (gh pr edit 5004 --repo microsoft/openvmm --base feature/microvm/dev),
   fast-forward feature/microvm/dev to
   cba21d109eeb99fbc573c79b2ed9f4754f734b25, and merge openvmm: pin refusing a FIFO snapshot artifact without waiting #1424; GitHub
   retargets this pull request to dev.
3. Retarget [DO NOT REVIEW] vmotherboard: gate a device's input without closing it to the vCPUs openvmm#5005 the same way, fast-forward
   feature/microvm/dev to 46bfe02ed65697e858f66ce55eb60831b3ac3461, and
   merge this pull request.
4. Merge nvx-rs: activate a clone from its restore packet's record #1427, which moves no pin.

Validation

Before this restack, with OpenVMM d6addf8c1 for "before" and a91d4c7a4
(the same patch on d6addf8c1) for "after". End to end, a microVM guest
requested 2000 snapshots, each of which gated host input and then failed its
preflight, since its destination existed, while its other vCPUs read the
magic value of the virtio-fs device's MMIO window, and with 4 vCPUs the
console's too, in a loop; 3 runs per cell. Reads that returned all ones:

Profile	Backend	Host type	2 vCPUs, before	4 vCPUs, before	2 and 4 vCPUs, after
nvx-kvm-test	kvm	virtual machine	586-786	2068-2358	0
nvx-mshv-test	mshv	virtual machine	271-360	898-1184	0
nvx-whp-test	whp	virtual machine	145-166	246-358	0

Before, OpenVMM logged that the device did not respond; after, it logged no
such warning. With a91d4c7a4, the snapshot and SMP scenarios and the full
default microVM suite, with 1, 2, and 4 vCPUs, passed on all three hosts.

OpenVMM's tests then: without the fix, the deterministic test failed, with the
port write dropped ("device didn't respond"), the EOI panicking ("lock should
not be closed"), and the interrupt lost, and the 1000-round stress test failed
in 10 of 10 runs; with it, 20 of 20 runs passed. Mutation checks of the virtio
and proxy gates ran too. The package checks passed: cargo clippy and
cargo doc on Windows for the five changed packages and the three virtio
devices, and cargo nextest, 545 tests on Windows and 550 on Linux.

Rerun for this restack, at OpenVMM 46bfe02ed and this head:

• OpenVMM, Rust 1.97.1: cargo clippy --all-targets and cargo doc --no-deps
  for vmcore, chipset_device_resources, virtio, vmotherboard,
  chipset_device_worker, virtiofs, virtio_net, and virtio_console,
  without warnings, and cargo nextest run --profile agent for those and
  state_unit, 545 passed and 2 skipped on Windows, and 550 passed and 2
  skipped on Linux (nvx-kvm-test, glibc), as before; cargo xtask fmt,
  clean.
• NVX: python scripts/nvx.py verify; python scripts/local_gates.py --base <layer 2's head> --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run, all passed, with no change to an
  existing test and the 4 vector and 1 CPU-profile comparisons passing against
  46bfe02ed; python scripts/openvmm_pin.py check-roadmap --worktree and
  check-branch --topic --target <layer 2's head>, passed. After the rebase,
  git range-diff shows this layer's two commits unchanged from those that
  the gates tested, on a base with the same tree, and this head's tree is the
  one that they tested; python scripts/rebase_check.py verify lists only the
  lower layers' commits, now in the base, as dropped.

At the exact revisions, for marking ready: the same end-to-end reproduction,
2000 requests per run with 2 and 4 vCPUs, 3 runs per cell, in a worktree of
da8157aec, this head but for the roadmap correction, with OpenVMM
cba21d109, #1424's pin and this change's base, for
"before" and this pin, 46bfe02ed, for "after". OpenVMM was built on each
host with build-openvmm (GNU for kvm, musl for mshv, MSVC for whp), but for
WHP's "before", which is the executable that #1424's CI built at cba21d109;
the guest artifacts are those of #1424's CI run, whose inputs this change
leaves alone. Reads that returned all ones:

Profile	Backend	Host type	2 vCPUs, before	4 vCPUs, before	2 and 4 vCPUs, after
nvx-kvm-test	kvm	virtual machine	613-962	1917-2196	0
nvx-mshv-test	mshv	virtual machine	283-378	980-1014	0
nvx-whp-test	whp	virtual machine	124-178	322-368	0

Every run passed its 2000 boundaries and exited with status 0. Before, each
run's OpenVMM log held "device didn't respond" warnings; after, none did, and
no run panicked.

CI run 42747993772
of this head, 3beaacf6f, passed, with OpenVMM built at 46bfe02ed: every
microVM scenario passed on KVM, MSHV, and WHP, 78, 70, and 70 of them, the
snapshot, SMP, and restore scenarios among them, and the debug kernel's 8 on
MSHV, as did NVX CLI on Linux and Windows, the performance regression gate,
Required status check, and OpenVMM CI of the pinned revision, after
OpenVMM's CI of the pin,
run 42743138458,
passed the 6 jobs that check-openvmm-ci requires. The merge group tests this
pull request again, merged into the latest dev.

[Copilot]

🟡 Changes recommended

The roadmap conflates eight conflicting files with nine total changed files in commit 46bfe02ed, leaving the upstream replay notes inaccurate.

1 open issue

• Distinguish eight conflicting files from nine changed files · New

What changed in this PR

This PR pins OpenVMM’s fix for #1138, keeping devices accessible to running vCPUs during snapshot input gating. The roadmap records the new revision and its upstream dependencies.

Changes:

• Updates the documented pin to 46bfe02ed, position 238, extending the chain to 107 commits beyond the mapped tip.
• Describes the split between the synchronous gate change and its asynchronous wait, with dependencies on PR09 and PR13.

File	Description
doc/​openvmm-upstream-roadmap.md	Updates the pin, chain counts, commit mapping, and upstream replay notes for the input-gating fix.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are reported for the roadmap documentation update.

No open issues 🎉

1 resolved since last review

• Distinguish eight conflicting files from nine changed files

_{Review effort: Balanced}