[Pedro Henrique Penna (ppenna)]

Fixes #1363

What and why

A registered OpenVMM state unit whose task has exited, by dropping its request
receiver or by panicking in a build that unwinds, made the VM worker panic with
UnitRecvError { .., source: Channel(Closed) } at the next stop, save, reset,
restore, or time advance that sent it a request, and in the rollback of a
failed start, instead of returning an error. OpenVMM's builds abort on panic,
so the process ended without reporting the failure or tearing the VM down. A
microVM stops its state units for every snapshot quiesce, pause, and teardown,
saves them for every capture, and restores them and advances their time for
every restore. Only start already failed instead, leaving the unit in a
terminal state. Only a unit whose task has already failed triggers this, so the
panic replaced an error that was fatal anyway, but the requester never saw it.

Root cause, in OpenVMM: StateUnits::run_op (vmm_core/state_unit/src/lib.rs),
which stop, reset, save, restore, advance_time, and the start rollback
of the chain's position 015 use, panicked on a UnitRecvError from a unit that
the set still held, by a stated policy that unit failure is not recoverable, and
StateUnits::stop returned (), so its callers could not see a failure.

microsoft/openvmm#5013 keeps the failure fatal without the panic, the option
that the issue recommends, in three commits on 6ddd6ca50, the head of
feature/microvm/dev:

• 2fcd1573d, state_unit: roll back a partial start without panicking on a lost unit: the rollback stops the units that started again in reverse
  dependency order, as state_unit: roll back a partial start instead of panicking openvmm#4872's later revisions do. A started
  unit that a unit that is not stopped depends on, such as a unit whose start is
  uncertain, is not stopped, a started unit whose stop is not answered is left
  in the terminal StartUncertain state, and the units then stay running, so
  that a later stop stops them.
• 5a750a862, state_unit: fail a stop instead of panicking when a unit is lost: StateUnits::stop returns an error, as state_unit: fail a stop instead of panicking when a unit is lost openvmm#4937 proposes
  upstream, and leaves a unit that does not answer in a new terminal state,
  StopUncertain, with its dependencies running, since it may still use them.
  The callers handle the error: a pause fails and the VM worker then tears the
  VM down, VmRpc::Pause becomes failable, a snapshot quiesce whose units do
  not all stop fails with the new SnapshotQuiesceError::Unrecoverable, on
  which the controller terminates the source with nothing published, and
  OpenHCL reports a failed stop to its requester.
• a8a0e3517, state_unit: fail a reset, save, restore, or time advance on a lost unit: run_op fails the operation, naming the unit, and leaves it in
  the operation's terminal state, so that a later start, reset, save, restore,
  or time advance fails too.

Commits:

• openvmm: pin state changes that fail instead of panicking on a lost unit
  moves the gitlink, and nothing else, from 6ddd6ca50, which openvmm: pin the handling of state units in a terminal state #1435 pins, to
  a8a0e35178f61c0e82cbe465b7ebf70e0f0bb543, the head of
  [DO NOT REVIEW] state_unit: fail a state change instead of panicking when a unit is lost openvmm#5013; its message carries the evidence below. NVX needs no
  coordinated change: the control-session vectors, source excerpts, contract
  versions, and CPU profiles that nvx-rs copies from OpenVMM match the
  revision's.
• doc: say that a device whose worker has ended fails its state change adds
  the failure to the design: such a device fails a capture's quiesce, and the
  source terminates with nothing published
  (5.3.3),
  and it fails the state change that it does not answer and never starts again
  (5.6.2).
• doc: map the lost-unit state changes in the OpenVMM roadmap maps position
  243 as a PR09 fixup, to fold into position 015, and its description of the
  snapshot rollback into PR17's position 072, and positions 244 and 245 as not
  yet assigned, and moves the pin to position 245, 114 commits past the mapped
  tip. All three conflict with upstream main, so they follow PR09 upstream.

Scope, with #1181's other children:

• openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148, which openvmm: pin the handling of state units in a terminal state #1435 fixed, has later requests fail on a unit in a terminal
  state; this extends its check to the new terminal states, and its stop of
  such a unit to a unit whose stop is uncertain.
• openvmm: failed snapshot rollback leaves restarted devices running #1149 needed no change. Its agent handed this change the rollback's ordered
  stop, item (d) of OpenVMM/microVM umbrella: snapshot quiesce, rollback, and device restoration in the VM worker #1181's notes, with its conditions: the units stay running
  whenever the rollback leaves a started unit running, a unit that failed to
  confirm its start is still stopped, the teardown stops a running set while
  the VM is paused (openvmm: pin the handling of state units in a terminal state #1435), and a reset checks for terminal units before it
  asserts that the units are stopped (openvmm: pin the handling of state units in a terminal state #1435). test_uncertain_start_blocks_dependency_rollback
  covers the test that it asked for: a started dependency of a unit whose start
  is uncertain keeps the units running, and a later stop stops it.
• Left out: keeping the units running after a start whose outcome is uncertain,
  and not only after an incomplete rollback, as #4872's later revisions do,
  which none of OpenVMM/microVM umbrella: snapshot quiesce, rollback, and device restoration in the VM worker #1181's children takes.

Decisions:

• Fail, without recovering. Chosen: a unit that does not answer fails the
  request and stays terminal, the issue's recommended option. Alternatives: make
  it recoverable, which needs a definition of the state of the units that
  already changed, or keep the panic as the worker's contract.
• Terminal states. Chosen: one per operation, StopUncertain,
  ResetUncertain, SaveUncertain, RestoreUncertain, and
  AdvanceTimeUncertain, beside StartUncertain. Alternative: one shared
  state. Reason: the state names the request that went unanswered; both refuse
  the same later requests.
• VmRpc::Pause. Chosen: failable, as the chain made VmRpc::Resume.
  Alternatives: drop the reply, or panic on the failed stop, as #4937's worker
  does. Reason: the requester sees the error.
• A failed quiesce. Chosen: the new SnapshotQuiesceError::Unrecoverable,
  on which the controller terminates the source. Alternatives: RollbackSafe,
  after which the source would resume, which it cannot, or Rejected, which
  refuses a request before anything is quiesced.
• The worker's reaction. Chosen: tear the VM down whenever state units run
  while the VM does not, which a failed stop, or a start whose rollback did not
  stop every unit, leaves. Alternative: keep serving and fail later requests, as
  openvmm: pin the handling of state units in a terminal state #1435 does for a terminal unit in a stopped set. Reason: such units can never
  start again, so the VM can never run again.
• OpenHCL. Chosen: report a failed stop to the requester of a pause, a
  diagnostic save, a restart, or servicing, and keep serving, as the chain does
  after a failed start. Alternative: tear down, as #4937 does. NVX does not use
  OpenHCL.
• Tests. OpenVMM unit tests in state_unit, which fail without the fix;
  no NVX scenario can make a unit's task end.

Changes to existing tests, all in microsoft/openvmm#5013 (NVX's
check_test_changes.py lists none): the state_unit tests that stop units,
among them #1435's in tests/terminal.rs, unwrap the stop's result; the
snapshot boundary filter's tests count Pause among the failable requests; and
the start rollback's tests share their recording unit with the new tests.

Landing order: land microsoft/openvmm#5013 first, by fast-forwarding
feature/microvm/dev to the exact pinned revision,
a8a0e35178f61c0e82cbe465b7ebf70e0f0bb543, since OpenVMM's merge button
squashes and would leave the pin off the branch; then merge this pull request.

Validation

Locally, on Windows, OpenVMM at the pin:

• cargo nextest run --profile agent -p state_unit -p openvmm_core -p openvmm_defs -p openvmm_entry -p petri:
  505 passed; cargo clippy --all-targets and cargo doc --no-deps with
  RUSTDOCFLAGS=-D warnings for those packages, without warnings;
  cargo check -p vmm_tests --tests; and cargo xtask fmt --fix, clean. Each
  of the three commits builds them and passes the state_unit tests, 24, 28,
  and 33.
• Each new test fails at its commit's parent: at 6ddd6ca50, two of the
  rollback's four panic in run_op with UnitRecvError and the rollback stops
  the dependency in the other two; at 2fcd1573d, three of the stop's four
  panic in run_op; at 5a750a862, four of the five reset, save, restore, and
  time advance tests panic in run_op with
  UnitRecvError { name: "lost", source: Channel(Closed) }.
• Reproduction, the issue's unit built with UnitBuilder::build whose receiver
  is dropped: at 6ddd6ca50, its stop, reset, save, restore, and time advance
  panic in run_op with UnitRecvError { name: "lost", source: Channel(Closed) },
  and its start fails; at the pin, all six fail with an error naming the unit.
• python scripts\local_gates.py --base origin/dev --run: test-changes (0
  changes to existing tests), markdown-links, test-layout, openvmm-vectors, and
  openvmm-cpu-profiles passed. nvx-cli, with NVX_RS naming a local release build: passed, with 2188 tests (88 skipped), the adversarial suite's 84 (11 skipped), and the help commands. openvmm-ci: waits for OpenVMM's
  CI of [DO NOT REVIEW] state_unit: fail a state change instead of panicking when a unit is lost openvmm#5013.
• python scripts\openvmm_pin.py check-roadmap --worktree,
  check-branch --topic, python scripts\check_markdown_links.py, and
  git diff --check: passed.

On the test hosts, all virtual machines, with this branch in a temporary
worktree and OpenVMM built from the pinned tree, whose binary differs from
that of release v0.1.0-dev.995230d0ebb4, which provided the guest artifacts.
The hosts built e9e626604, whose tree is the pin's: the pin rewords two of
its commit messages.

Profile	Backend	Host type	Before: reproduction at 6ddd6ca50	After: reproduction and OpenVMM checks	NVX scenarios after
nvx-kvm-test	kvm	virtual-machine	5 of 6 operations panic with UnitRecvError	6 of 6 fail with an error; the changed packages and underhill_core: 599 passed, clippy and rustdoc clean	all seven passed
nvx-mshv-test	mshv (musl)	virtual-machine	not run: same code as on nvx-kvm-test	not run: same code as on nvx-kvm-test	all seven passed
nvx-whp-test	whp	virtual-machine	5 of 6 operations panic with UnitRecvError	6 of 6 fail with an error; state_unit: 33 passed	all seven passed

The scenarios were guest-boot, lifecycle, managed-lifecycle,
sandbox-lifecycle, restore-downtime, snapshot-core, and
snapshot-rollback, with 1 and 2 vCPUs, which exercise the changed stop on a
pause, a capture's quiesce, and teardown, the rollback's start, and a restore's
restore and time advance.

Left to CI: the full microVM suites on KVM, MSHV, and WHP, the runtime adapter,
the performance gates, and OpenVMM's own CI of the pin, its unit tests and VMM
tests.

CI: run 42823643715
passed, with the full microVM suites on KVM, MSHV, and WHP, the runtime
adapter, the performance gate, and OpenVMM's CI of the pin. OpenVMM's
run 42822151781
of a8a0e3517 passed every job but run vmm-tests [x64-windows-amd-snp],
whose Hyper-V OpenHCL SEV-SNP memstat and TPM tests timed out after 600 s, as
a Hyper-V OpenHCL SEV-SNP TPM test did in that job of microsoft/openvmm#5012's
run, without this change.

[Copilot]

🔵 Needs a closer look

The pin changes dependency-ordered lifecycle transitions, rollback, and teardown, while exact-revision OpenVMM CI remains incomplete and prerequisite PR #5013 is still unmerged.

No open issues 🎉

What changed in this PR

Documents the OpenVMM lost-unit failure handling that addresses #1363: unanswered state changes return errors rather than panic, and an unrecoverable snapshot quiesce terminates the source without publishing a snapshot.

Changes:

• Updates the roadmap’s recorded pin and maps commits 243–245, including upstream dependencies and fixup placement.
• Clarifies snapshot behavior when a device worker has ended.
• Documents that a failed device cannot restart or allow the VM to run again.

File	Description
doc/​openvmm-upstream-roadmap.md	Records the new pin and maps the three lost-unit handling commits.
doc/​design/​5.6-shutdown-and-failure-semantics.md	Defines terminal failure behavior for an ended device worker.
doc/​design/​5.3-snapshot-protocol.md	Clarifies source termination and non-publication after a failed quiesce.

_{Review effort: Balanced}