Skip to content

host: drive a restore over NVXC for blocking callers (SL4.3) - #1476

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-sl4-3-restore-driver
Nov 11, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-sl4-3-restore-driver

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1113 (SL4.3, "A blocking caller can drive a restore"). Parent #1084, umbrella #1076.

What and why

NVX's blocking CLI callers, first SL3.8a's snapshot restore (#1114), need one bounded operation that drives a launched restore over NVXC to a usable control session. The pieces exist: SL2.16 (#1470) launches a restore with its bound restore-ready listener, SL2.18a's I/O-free RestoreMachine orders the activation to Ready, SL2.18b (#1354) reports the failure facts, and SL4.2 (#1331) added the activate message. This PR adds the driver that runs them:

  • Driver. The new private module host::restore_driver (feature cli) has restore(launched, record, workload, maps, cancel) -> Result<ControlSession, ControlRestoreError>. It drives RestoreMachine through OpenVMM's restore-ready event and the control sessions to the guest's Ready, then sends the mapping table and the workload's Start (APP_EXEC with the caller's request ID), and returns the live ControlSession, in which the workload's events arrive.
  • Peers. The restore-ready client and the control server must be the launched VM's held process (ReadyListener::receive and session::connect's peer checks); anything else gets no activation.
  • Sessions. Each session carries the same activate message, encoded once: the activation record and the workload's identity. A lost session is replaced, in at most three sessions in all; a malformed Ready, a third lost session or an unverified peer is final.
  • Budgets. 60 seconds from the call to Ready, then 30 seconds from Ready for the mapping table and Start.
  • Cancellation and exit. Every pending accept, read and write races the step's deadline, the caller's cancellation (an AtomicBool) and the VM's verified exit. The restore-ready endpoint and the control endpoint are waited for in slices of at most 50 ms, between which the driver checks both. While a session transfers, a scoped watcher thread checks them and ends the session's pending transfers through a duplicate of its connection, with the new crate-private platform::interrupt_transfers: shutdown(SHUT_RDWR) on Linux and CancelIoEx on Windows.
  • Errors. ControlRestoreError tells a failure of the machine (Machine(RestoreError), with its phase, evidence and source) from one of the mapping table (Maps) or of Start (Start). A failure stops nothing: the VM's process, outputs and restore-ready listener stay the caller's.
  • Session. ControlSession::activate (feature cli) sends the activate message and returns the activation that the guest's Ready reports.
  • Protocol. APP_ACTIVATE and encode_activate_payload are now used by the cli build, so their expect(dead_code) hooks no longer apply to it (see the test changes below), and protocol re-exports them for it.
  • Docs. The restore bullet of host's module docs and the nvx::host::restore section of the README describe the driver.

Layering

The driver is a child of host of its own, above vm, rather than part of host::session: the driver needs vm::Launched, and vm already depends on session through instance, so a driver in session would make the children of host cyclic, which tests/host_modules.rs refuses. host::session stays below vm.

The expect(dead_code) hook that SL3.8a ends

Nothing calls restore outside its tests yet, so it carries #[cfg_attr(not(test), expect(dead_code, reason = "SL3.8a's snapshot restore restores sandboxes through it"))], with its seam entry "nvx-rs/src/host/restore_driver.rs": 1 in scripts/test-layout-allowlist.json. SL3.8a (#1114) calls it from snapshot restore and then removes the hook and, with scripts/check_test_layout.py --tighten, the entry. The functions that only restore reaches need no hook of their own.

History: the first implementation, reworked into one commit

Enrique Saurez (@esaurez)'s first implementation is esaurez/slim-3.2.1-sl4.3-private-20261108 (b30be688, checkpoint on #1113), four commits: c2a199d8 (drive bounded blocking restores over NVXC), bc389dae (document blocking restore invariants), b7b17a68 (preserve completed pipe transfers during restore races) and b30be688 (break the restore session dependency cycle). It measured 565 core lines in 8 counted files, over the limit of 5. This PR reworks it into one commit, with Enrique Saurez (@esaurez) as its author, because the later commits fix and restructure the first, and the rework replaces two of their approaches rather than adding to them:

  • The driver moves out of host::session into host::restore_driver, which spares moving vm's capability generator below session into platform (b30be688's break of the cycle).
  • A watcher thread and platform::interrupt_transfers replace the restore checkpoints that c2a199d8 and b7b17a68 added to the Linux and Windows transports, with their platform tests.
  • It keeps the driver loop, the failure classification and the Python stand-in tests, which move to restore_driver/tests.rs. The stand-in is now spawned as a VM is (vm::spawn with its handle allowlist), so on Windows it inherits no other test's handles.

The commit message records the same history.

Size

Measured against feature/nvx-refactor 821de193: 432 core lines (stop 600) in 5 counted files: host/restore_driver.rs 394, host/session.rs 19, host/platform/windows/endpoint.rs 10, host/platform/linux/endpoint.rs 7 and host/mod.rs 2. protocol/app.rs, protocol/mod.rs, host/platform/linux/mod.rs and host/platform/windows/mod.rs change only attributes and imports (29 lines).

Stack

The branch was built on SL2.16's head 9bc9d403 before #1470 folded its fixups, and rebased with git rebase --onto origin/feature/nvx-refactor 9bc9d403 onto the merged 8b775211, whose SL2.16 differs from 9bc9d403 only in docs and the example. It was then rebased onto 821de193, after #1460 (SL3.5b) merged; that touches only the CLI's sandbox lifecycle and spawn helpers that the driver does not use, and the allowlist's neighboring lines. Neither rebase changed this commit's added or removed lines (git range-diff).

Not in this PR

The caller (snapshot restore, SL3.8a); any public API (everything is crate-private); stopping or cleaning up the VM after a failure, which stays the caller's; the per-snapshot lock, which the caller holds; retries beyond the three sessions.

Validation

On the head 8a761767 (on 821de193), on the local Windows workstation, with this branch's check-nvx-rs and check-quality commands (in nvx-rs/, NVX_BUNDLE=skip):

Check Result
cargo fmt --all --check passed
cargo clippy --locked with -D warnings: --all-targets --all-features, --lib, -p nvx --lib --no-default-features, --lib --bins --features cli, --lib --bins --no-default-features --features testing passed, all five
the same with --target x86_64-unknown-linux-musl: --all-targets --all-features, --lib --bins --features cli, -p nvx --lib --no-default-features passed (the Linux interrupt_transfers)
cargo test --locked --all-features --lib -- host::restore_driver host::session::tests::activate 11 passed (64 s: the budget test waits the real 60 s)
cargo test --locked --all-features --test host_modules 22 passed, no cycle
python scripts\check_test_layout.py passed
python scripts\check_test_changes.py --base origin/feature/nvx-refactor 3 changes, 39 added (below)
python scripts\public_api.py passed, no change on either platform

Earlier, on 40c86afa, the same patch on 9bc9d403:

  • Windows (local): MSRV 1.89 cargo check, cargo doc --no-deps --all-features, the doctests, the runtime adapter's Clippy, the Markdown links and cargo test --all-features passed, except three timing-sensitive tests outside this change: a_capture_ack_and_successful_output_without_timely_exit_are_stopped_but_never_finished and template_capture::vms_that_never_end_are_torn_down_at_the_deadline fail the same way at the base 5947e4e8 on this loaded workstation, and children_inherit_only_their_standard_handles is flaky there and passes alone.
  • Linux (nvx-kvm-test, an Azure Linux 3 VM, KVM): the Clippy configurations, scripts/public_api.py and the driver's tests passed. cargo test --all-features failed only the 28 guest:: tests that fail identically at the base on that host (a process-group kill returns EINVAL there).

No microVM scenario was run: nothing reaches the driver outside its tests until SL3.8a, and no runtime path changes. CI run 42983982090 of the previous head 8ba185aa (the same patch on 8b775211) passed in full. Left to CI: the full test suites on Linux and Windows, the microVM and runtime-adapter suites on KVM, MSHV and WHP, and the benchmarks.

Changes to existing tests (tests-modified)

python3 scripts/check_test_changes.py --base origin/feature/nvx-refactor lists 3 changes to existing tests, and 39 added items:

  1. APP_ACTIVATE's test hook (protocol/app.rs): its cfg_attr(not(any(test, ...)), expect(dead_code, ...)) adds feature = "cli", since the driver sends the message in the cli build; the reason becomes "only the CLI's restore driver sends it, and only nvx-init reads it".
  2. encode_activate_payload's test hook (protocol/app.rs): not(test) becomes not(any(test, feature = "cli")), since the driver encodes the payload in the cli build; the reason becomes "only the CLI's restore driver encodes activation requests".
  3. The test-layout allowlist gains the seam entry "nvx-rs/src/host/restore_driver.rs": 1 for the expect(dead_code) hook on restore, which SL3.8a removes.

No existing test is rewritten or removed: the tests of the session's capture, the activate payload's round trips, the restore machine, the platform endpoints and vm are unchanged. The added items are the driver's 9 tests and their Python stand-in, ControlSession::activate's 2 tests and the test helper Launched::stand_in in host::vm::tests.

@ppenna Pedro Henrique Penna (ppenna) added tests-modified Changes existing tests, test data, or test wiring feat-nvx-refactor labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 3 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified hook the test hooks of const APP_ACTIVATE in protocol::app nvx-rs/src/protocol/app.rs:74 nvx-rs/src/protocol/app.rs:75
modified hook the test hooks of fn encode_activate_payload in protocol::app nvx-rs/src/protocol/app.rs:194 nvx-rs/src/protocol/app.rs:198

Test policy

Change What Before After
loosened allowlist entry nvx-rs/src/host/restore_driver.rs (the seams allow 1 test hooks) scripts/test-layout-allowlist.json

Relocated unchanged: 0; added: 39. Compared the merge base 821de193f45f with 8a761767cbe2.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the private, CLI-gated blocking restore driver.

No open issues 🎉

What changed in this PR

Adds the private, CLI-gated blocking restore driver for SL4.3, connecting the existing restore state machine to NVX control sessions. This prepares the mechanism for later snapshot CLI integration without changing the public API or taking ownership of VM cleanup.

Changes:

  • Drives restore through verified peers, activation, mappings and workload Start, with bounded retries and deadlines.
  • Adds cancellation and VM-exit monitoring, backed by Linux socket shutdown and Windows pipe-transfer cancellation.
  • Adds driver and activation tests, documentation and a temporary test-layout allowance.
File Description
scripts/​test-layout-allowlist.json Allows the driver’s temporary unused-code hook.
nvx-rs/​src/​protocol/​mod.rs Re-exports activation helpers for CLI builds.
nvx-rs/​src/​protocol/​app.rs Updates unused-code attributes for CLI activation use.
nvx-rs/​src/​host/​vm/​tests.rs Adds a launched-VM stand-in helper.
nvx-rs/​src/​host/​session/​tests.rs Tests activation requests and Ready validation.
nvx-rs/​src/​host/​session.rs Adds the control-session activation exchange.
nvx-rs/​src/​host/​restore_driver/​tests.rs Tests retries, peer checks, budgets, interruptions and failures.
nvx-rs/​src/​host/​restore_driver.rs Implements bounded blocking restore orchestration.
nvx-rs/​src/​host/​platform/​windows/​mod.rs Exposes the Windows transfer-interruption helper internally.
nvx-rs/​src/​host/​platform/​windows/​endpoint.rs Cancels pending pipe transfers with CancelIoEx.
nvx-rs/​src/​host/​platform/​linux/​mod.rs Exposes the Linux transfer-interruption helper internally.
nvx-rs/​src/​host/​platform/​linux/​endpoint.rs Interrupts socket transfers through shutdown.
nvx-rs/​src/​host/​mod.rs Registers and documents the private CLI-gated driver.
nvx-rs/​README.md Documents blocking restore guarantees and budgets.

Review effort: Balanced

NVX's blocking CLI callers need one bounded operation that restores a
launched VM over NVXC and returns a usable control session. The private
restore_driver module, beside host::session and gated by the cli
feature, drives RestoreMachine to the guest's Ready, then sends the
mapping table and the workload's Start, and returns the live
ControlSession.

restore() checks that the restore-ready client and the control server
are the launched VM's held process, sends the same activate message,
the activation record and the workload's identity, in each session,
replaces a lost session in at most three sessions, and allows 60 s to
Ready and 30 s more for the mapping table and Start. Every pending
accept, read and write races the step's deadline, the caller's
cancellation and the VM's verified exit: the restore-ready endpoint
and the control endpoint are waited for in 50 ms slices, and a watcher
thread ends a session's pending transfers through a duplicate of its
connection, with the new crate-private
platform::interrupt_transfers, a shutdown on Linux and CancelIoEx on
Windows. ControlRestoreError tells a failure of the machine from one
of the mapping table or of Start, and a failure leaves the VM's
process to the caller. ControlSession::activate sends the activate
message and returns what the guest's Ready reports.

The driver is a child of host of its own, above vm, so that
host::session stays below vm and the children of host stay acyclic.

Reworked from the first implementation on
esaurez/slim-3.2.1-sl4.3-private-20261108 (b30be68), whose driver
loop, failure classification and Python peer tests it keeps: the
driver moved out of host::session, which spares moving vm's capability
generator below it, and a watcher thread replaces the restore
checkpoints that the first implementation added to the platform
transports.

Tests: a Python stand-in for OpenVMM and its guest, spawned as a VM
is, so that it inherits no other test's handles, checks the
reconnection with the same activate message and the live session that
returns, the third lost session and a malformed Ready, the refusal of
an unverified restore-ready client or control server, the 60 s and
30 s budgets, cancellation and the VM's exit at each pending accept
and read, a pending write ended by each, distinct machine, MAPS and
Start failures, and a cold record or a root workload refused before
any exchange.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) merged commit 35cedfd into feature/nvx-refactor Nov 11, 2026
48 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/slim-sl4-3-restore-driver branch November 11, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants