[Pedro Henrique Penna (ppenna)]

Fixes #1148

What and why

A state unit whose task does not answer its start request is left in
OpenVMM's terminal state StartUncertain, which only a later start refused.
At NVX's pin before this change, a later stop, reset, save, restore, or time
advance that reached such a unit panicked the VM worker, and a worker restart
whose recovery start failed kept serving a VM that it marked running over
stopped units, so that its next pause, or its teardown, panicked.

The issue's first trigger, an uncertain snapshot quiesce, no longer exists:
since #1288 the capture quiesces with StateUnits::stop, without a deadline,
so no unit is left QuiesceUncertain, as the issue's re-triage records. This
change fixes the other half, later requests that reach a unit in a terminal
state, in the form that microsoft/openvmm#4872's later revisions give the
chain's start rollback upstream, as that re-triage proposes.

Root cause, in OpenVMM:

• StateUnits::run_op (vmm_core/state_unit/src/lib.rs) asserts that a unit
  that is not in an operation's old state is already in its new state. A stop
  of a running set that holds a StartUncertain unit, as after a hot-add
  whose start is uncertain, panicked with "unit ... in StartUncertain state,
  should be Running or Stopped", and a reset, save, restore, or time advance
  after a failed start with "should be Stopped or Stopped".
• The VM worker's restart (openvmm/openvmm_core/src/worker/dispatch.rs)
  stops a running VM's state units before it saves the VM and starts them
  again when the save fails. When that start failed too, it reported the
  failure and continued, with the VM still marked running, so the next pause,
  reset, or pulse save and restore, and the teardown, which paused the VM,
  stopped units that did not run and panicked on StateUnits::stop's running
  assertion.

microsoft/openvmm#5012 fixes both, in two commits on 4c2697aef, the head of
feature/microvm/dev:

• 78e752de1, state_unit: handle a unit in a terminal state instead of panicking: a reset, save, restore, or time advance fails, naming each unit
  in a terminal state, before it sends any unit a request or checks that the
  units are stopped, and a stop also stops a unit whose start is uncertain,
  since it may be running, in reverse dependency order, and leaves it in that
  state, so that a later start still fails. OpenVMM unit tests cover both.
• 6ddd6ca50, openvmm_core: tear the VM down when a restart cannot resume it: the worker ends after such a restart, and its teardown stops the state
  units when they run, rather than when the VM is marked running.

No NVX microVM reaches these panics: a state unit's task cannot end without
aborting OpenVMM, whose builds abort on panic, and only the TPM, which a
microVM lacks, uses a remote device worker; the microVM profile refuses worker
restarts and has no hot-add. A capture whose rollback fails terminates its
source, as
Capture Transaction
specifies, and the worker's teardown now stops the state units only when they
run.

Commits:

• openvmm: pin the handling of state units in a terminal state moves the
  gitlink, and nothing else, from 4c2697aef, which openvmm: pin the rollback as the only way to run a quiesced microVM #1430 pins, to
  6ddd6ca505a7212d92e2a8e561a19635f7185d4a, the head of
  [DO NOT REVIEW] state_unit: handle a unit in a terminal state instead of panicking openvmm#5012; its message carries the evidence below. NVX needs
  no coordinated change: the control-session vectors, source excerpts,
  contract versions, and CPU profiles that nvx-rs copies from OpenVMM match
  the revision's.
• doc: map the terminal state-unit fixups in the OpenVMM roadmap maps
  positions 241 and 242 as PR09 fixups, to fold into position 015, the start
  rollback, and position 241's check of a time advance into PR10's position
  019, which adds the time advance. The pin is 111 commits past the mapped
  tip. The commits conflict with upstream main in all but the new test file,
  so they follow PR09 upstream.

Scope, with #1181's other children:

• openvmm: failed snapshot rollback leaves restarted devices running #1149 needs no change: position 015's rollback already stops the units that
  a partial start started.
• A unit whose task has ended still panics a stop that it does not answer
  (UnitRecvError), as any unit does; openvmm: stop, save, reset or restore panics if a state unit's task has exited #1363 makes that an error.
• Of state_unit: roll back a partial start instead of panicking openvmm#4872's later revisions, this leaves out keeping the
  units running after an uncertain start, the rollback's stop that a dependent
  that was not stopped blocks, stop_for_teardown, the removal of a failed
  hot-add's units, and OpenHCL's handling, which the chain takes when it
  adopts #4872's form of position 015.

Decisions:

• Fix form. Chosen: #4872's, where later requests fail and a stop also
  stops a terminal unit, while the worker keeps serving and reports the error.
  Alternative: the issue's patch, which ended the worker's request loop
  whenever a unit was terminal. Reasons: the issue's re-triage names #4872's
  form; it keeps the requester's error; and NVX's controller already
  terminates a source whose rollback fails.
• Order of the terminal check. Chosen: before the check that the units are
  stopped. Alternative: after it, as #4872 has it. Reason: this fork's resume
  is fallible, so once the chain keeps the units running after an uncertain
  start, as #4872 does, a reset after a failed resume would otherwise panic on
  that check instead of failing.
• Restart. Chosen: tear the VM down, as #4872 does. Alternative: mark the
  VM paused and keep serving it. Reason: the VM cannot resume, and it follows
  upstream.
• A terminal unit whose task has ended, at teardown. Not tolerated here, as
  #4872's stop_for_teardown does: it is a closed channel, which openvmm: stop, save, reset or restore panics if a state unit's task has exited #1363's
  fallible stop covers.
• Regression tests. OpenVMM unit tests in state_unit, which fail without
  the fix. No NVX scenario can reach the defect, and no harness of OpenVMM's VM
  worker can fail a unit's start after a failed save, so the restart change
  has no test.

Landing order: land microsoft/openvmm#5012 first, by fast-forwarding
feature/microvm/dev to the exact pinned revision,
6ddd6ca505a7212d92e2a8e561a19635f7185d4a, since OpenVMM's merge button
squashes and would leave the pin off the branch; then merge this pull request.

Validation

Locally, on Windows:

• OpenVMM at 6ddd6ca50: cargo clippy --all-targets -p state_unit -p openvmm_core -- -D warnings,
  cargo doc --no-deps -p state_unit -p openvmm_core with RUSTDOCFLAGS=-D warnings, and
  cargo xtask fmt --fix: passed, clean; cargo nextest run --profile agent -p state_unit -p openvmm_core:
  148 passed.
• Reproduction: with the three new state_unit tests alone on 46bfe02ed, 3 of
  20 failed, a stop and a reset panicking on the state assertion ("unit uncertain in
  StartUncertain state, should be Running or Stopped", "unit lost in StartUncertain
  state, should be Stopped or Stopped") and a reset of a running set on
  assertion failed: !self.running; with the fix, 20 of 20 passed.
• python scripts\local_gates.py --base origin/dev --run: test-changes (0 changes to
  existing tests), markdown-links, test-layout, openvmm-vectors, and openvmm-cpu-profiles
  passed. nvx-cli: passed, with 2188 tests (88 skipped), the adversarial suite's 84 (11 skipped), and the help commands. At this branch's first revision, on b6ad70074, 7 subtests of two guest-script tests in scripts/test_microvm_tests.py had timed out under Git for Windows' sh on this busy machine, as they did at origin/dev there; they pass now. openvmm-ci: waits for OpenVMM's CI of
  [DO NOT REVIEW] state_unit: handle a unit in a terminal state instead of panicking openvmm#5012.
• python scripts\openvmm_pin.py check-roadmap --worktree, check-branch --target origin/dev --topic,
  python scripts\check_markdown_links.py, and git diff --check: passed.

On the test hosts, all virtual machines, with this branch in a temporary worktree and
OpenVMM built from the pinned revision, whose binary differs from that of release
v0.1.0-dev.995230d0ebb4, which provided the guest artifacts:

Profile	Backend	Host type	Before: new tests alone on 4c2697aef	After: state_unit and openvmm_core	NVX scenarios after
nvx-kvm-test	kvm	virtual-machine	3 of 20 failed, as on Windows	148 passed; clippy and rustdoc clean	all five passed
nvx-mshv-test	mshv (musl)	virtual-machine	3 of 20 failed, as on Windows	148 passed; clippy and rustdoc clean	all five passed
nvx-whp-test	whp	virtual-machine	on Windows, locally, above	on Windows, locally, above	all five passed

The scenarios were lifecycle, managed-lifecycle, sandbox-lifecycle,
snapshot-core, and snapshot-rollback, which exercise the changed stop on a
capture's quiesce and on teardown, the rollback's start, and a restore's restore
and time advance; they also passed at the first revision of this change, on
46bfe02ed.

Left to CI: the full microVM suites on KVM, MSHV, and WHP, the runtime adapter, the
performance gates, and OpenVMM's own CI of the pin, its unit tests and VMM tests.

[Copilot]

🟢 Approval recommended

No active review issues were identified for the roadmap update documenting the terminal-state lifecycle and failed-restart teardown fixes.

No open issues 🎉

What changed in this PR

This PR updates NVX’s OpenVMM pin to address lifecycle requests on terminal state units and teardown after a failed restart, completing the remaining scope of #1148. The reviewed documentation maps those fixes into the upstream plan.

Changes:

• Records revision 6ddd6ca50, at position 242 and 111 commits beyond the mapped tip.
• Maps positions 241–242 as PR09 fixups, with the time-advance check and test assigned to PR10.
• Documents the fixes’ behavior, upstream dependencies, and differences from upstream proposals.

File	Description
doc/​openvmm-upstream-roadmap.md	Updates the recorded pin, commit mappings, graph, and upstream integration guidance for the terminal-state fixes.

_{Review effort: Balanced}