From a04149d38d67e8282d4192a3076a3d916357904c Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Fri, 16 Oct 2026 13:57:48 -0700 Subject: [PATCH 1/9] doc: describe the managed layered root and platform-clone activation State in the architecture guide the design that the image-backed runtime roadmap needs, so that its differences from the guide can be reviewed before implementation: - a managed layered root with every control feature, readiness after root assembly from boot tokens, and shared host and guest libraries; - a distro image that is one EROFS filesystem or a GPT disk of ordered layers; - control protocol version 2 with standard input, per-execution credits, and no total output limit; - platform-clone inputs: zero-filled shorter layers, a RAM-layout platform capture, an aggregate on a dormant filesystem slot, and an activation record in restore packet version 5; - a new network policy at restore for platform templates captured with a deny-all policy; - expected attachment identities, receipts, and pinned file children. Section numbers, tables, figures, and anchors are unchanged. Refs #535 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-system-components-and-responsibilities.md | 18 +++- doc/design/1.4-trust-boundaries.md | 9 +- .../1.5-compatibility-and-versioning-model.md | 20 ++-- .../2.1-machine-model-and-configuration.md | 43 +++++--- doc/design/3.1-platform-control-device.md | 12 ++- doc/design/3.4-block-storage.md | 21 +++- doc/design/3.5-network.md | 30 ++++-- doc/design/3.6-host-filesystem.md | 44 +++++--- .../4.1-guest-filesystem-and-image-model.md | 102 +++++++++++++----- .../4.2-guest-agents-and-workload-model.md | 13 ++- .../4.3-isolation-and-resource-model.md | 4 +- .../4.4-host-guest-control-interface.md | 77 +++++++++---- doc/design/5.1-boot-protocol.md | 19 ++-- .../5.2-readiness-and-runtime-control.md | 28 ++++- doc/design/5.3-snapshot-protocol.md | 4 +- ...-snapshot-artifact-and-attachment-model.md | 38 ++++--- doc/design/5.5-restore-protocol.md | 94 +++++++++++++--- .../5.6-shutdown-and-failure-semantics.md | 3 + 18 files changed, 442 insertions(+), 137 deletions(-) diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index bff0d4332..6aa468d32 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -17,6 +17,7 @@ flowchart TB direction LR Launcher["NVX launcher"] LifecycleApi["Sandbox lifecycle API"] + Embedder["Embedding host adapters"] ControlLibrary["Host control library"] Storage["Snapshot storage"] Attachments["External attachments"] @@ -42,6 +43,7 @@ flowchart TB Launcher --> Frontends LifecycleApi --> ControlLibrary + Embedder --> ControlLibrary ControlLibrary --> Frontends Launcher -. control session .-> Virtio ControlLibrary -. control session .-> Virtio @@ -73,7 +75,16 @@ flowchart TB - **Host control library** implements the host side of the control protocol, artifact discovery, blockless managed launch, VMM process supervision, host-path export with its access policy, delivery of the host mapping table, - and egress-rule translation. + and egress-rule translation. It also implements the host half of the + [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root): + its root plan, activation configuration, execution admission, and + execution streams. Adapters connect the library to an orchestrator: the + sandbox lifecycle API is one, and an embedding host with its own + orchestration, storage, and admission policy is another. An adapter + supplies policy, resources, and transport, and reuses the library's + validation and execution semantics instead of reimplementing them. +- **Embedding host adapters** connect orchestrators outside NVX to the host + control library. - **Snapshot storage** retains immutable machine state, memory, and optional paired scratch together with the identities needed to reconstruct external resources. @@ -124,7 +135,10 @@ flowchart TB processors, memory, entropy, and identity after restore. They also announce the guest's listeners when the host publishes them, check time ABI conformance, watch for clock violations, discipline the wall clock, and - perform the snapshot time steps. + perform the snapshot time steps. Their managed layered root comes from a + guest runtime library that another guest init can embed through its own + adapter, so root assembly, host mappings, and execution supervision behave + the same for every host. - **Workload** is the microVM's single tenant: it owns the microVM's lifetime, resource envelope, and network identity, and it may run any number of processes, all inside the isolation boundary that the guest agents diff --git a/doc/design/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md index 69afe4af6..78bf8f40b 100644 --- a/doc/design/1.4-trust-boundaries.md +++ b/doc/design/1.4-trust-boundaries.md @@ -88,7 +88,14 @@ Image layers are trusted inputs: the overlay filesystem interprets metadata stored in lower layers, and bytes read from any layer can appear in captured memory. A snapshot identifies its blocks by a storage generation, which transfers trust for layer and scratch contents to the storage owner; the VMM -trusts the control plane in the host to authenticate that generation. +trusts the control plane in the host to authenticate that generation. For the +attachments that a restore supplies, a host may name the identity that it +expects for each object, and the VMM compares it with the object that it +actually opened before any processor runs, so an object replaced between the +host's check and the VMM's open never reaches the guest. The VMM also reports +each opened object's identity and actual length in a receipt; see +[Host Attachments](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments). +A receipt identifies an object, not its contents. The VMM trusts the host control plane not to give the same writable file, such as a caller-supplied RAM backing or scratch file, to more than one VMM. It diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index 860545f14..d5ab14842 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -17,8 +17,9 @@ contract carries its own version or identity, independent of the others. | Device-private schemas | Saved queue and device state that must be understood before a device can resume. | | Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | | RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | -| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, and fresh entropy. | -| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, and lifecycle epochs. | +| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, fresh entropy, and, in a platform clone, an optional activation record. | +| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, execution streams, and lifecycle epochs. | +| Activation configuration | Schema and identity of the non-secret configuration that a managed layered root reads from boot tokens on a cold boot or from the restore packet's activation record in a platform clone. | | Management RPC machine profile | Identity of the microVM machine profile in the management RPC. | A change to a guest-visible address, interrupt, command-line token, feature @@ -61,17 +62,20 @@ snapshots stay restorable; a normative change creates a new revision. A reusable template represents a compatibility class, not an arbitrary fleet image. The class includes the backend, the CPU profile and generation, the effective CPUID, the guest kernel and guest agent build, the effective command -line, device roles and geometry, network identity and policy, and consumed -layer identities. Restoring guest memory restores the captured guest agent -build, not a newly deployed initramfs, so deployments rebuild or recertify -templates on relevant rollouts. +line, device roles and geometry, network identity, network policy unless a +`platform` template was captured with a deny-all policy, and consumed layer +identities. Restoring guest memory restores the captured guest agent build, +not a newly deployed initramfs, so deployments rebuild or recertify templates +on relevant rollouts. That build also fixes the activation-configuration +schema that the template's clones accept. Processor capacity and RAM geometry are immutable, but restore-time activation lets one template serve several online processor counts and RAM sizes. Templates are therefore keyed by capacities, base state, and supported activation ranges. Layer-role presence and device geometry stay fixed: a -platform template accepts a substitute layer only with the same recorded -logical length and block sizes. +platform template accepts a substitute layer only with the same block sizes +and a length no greater than its recorded logical length, which the device +continues to present, so one template serves every image up to that length. ## 1.5.3 Persisted State diff --git a/doc/design/2.1-machine-model-and-configuration.md b/doc/design/2.1-machine-model-and-configuration.md index d3701baa2..73f07f5eb 100644 --- a/doc/design/2.1-machine-model-and-configuration.md +++ b/doc/design/2.1-machine-model-and-configuration.md @@ -56,9 +56,11 @@ lifetime is one of: | Processor capacity, base RAM and optional RAM capacity, and CPU profile | Host, validated by the machine profile | VM | | Effective kernel command line | Machine profile, from host policy and caller arguments | VM | | Workload identity and lifecycle | Host | VM | -| Console, network, block, and host filesystem devices, with their identities and policies | Host | VM | -| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities | VM process | +| Console, network, block, and host filesystem devices, with their identities and policies, except a platform clone's network policy below | Host | VM | +| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities, or, for the unbound layers of a `platform` snapshot, within the recorded lengths | VM process | | Online processor and RAM activation targets | Host, within snapshot limits | Restore | +| Activation configuration of a platform clone | Host, validated by the guest agents | Restore | +| Network policy of a platform clone whose template was captured with a deny-all policy | Host | Restore | The host fixes the workload policy of a fresh boot. An optional nonzero numeric UID and GID become the fixed workload identity, and an optional @@ -68,22 +70,31 @@ Both are command-line tokens that the machine profile owns and that caller arguments cannot supply. A managed lifecycle requires a fixed workload identity and a live, authenticated control console. +The host also fixes the activation configuration of a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root). +A cold boot carries it as launcher tokens, and a platform clone receives it in +the restore packet's activation record. A launch that declares a `platform` +capture carries none, because that tier consumes only invariants. + ## 2.1.3 Storage Configuration Sandbox storage uses block devices with fixed roles rather than arbitrary disks. The roles are the layers of the sandbox root filesystem: a read-only -`distro` lower layer for the base operating system, beneath one writable -`scratch` layer that holds the sandbox's private changes. +`distro` image of the base operating system, either one filesystem or a GPT +disk of ordered lower layers, beneath one writable `scratch` layer that holds +the sandbox's private changes. [Block Storage](3.4-block-storage.md) describes each role. A disk without a role is rejected, and a disk-backed topology ends with writable scratch. The only exception is the RAM-backed overlay layout, which -requires exactly one read-only `distro` block and no scratch, keeps the -guest's overlay upper layer in RAM, and supports neither capture nor restore. -Snapshot capture with blocks requires a read-only `distro` layer followed by -writable scratch, both backed by cached, regular raw files with nonzero, -512-byte-aligned geometry. Blockless machines are also snapshot-capable and -carry no sandbox tier metadata. +requires exactly one read-only `distro` block and no scratch and keeps the +guest's overlay upper layer in RAM. It may be captured only at the `platform` +tier, before the guest agents mount the `distro` image or the overlay, so its +upper layer is never captured. Any other snapshot capture with blocks +requires a read-only `distro` layer followed by writable scratch. Every block +of a snapshot-capable machine is backed by a cached, regular raw file with +nonzero, 512-byte-aligned geometry. Blockless machines are also +snapshot-capable and carry no sandbox tier metadata. Host filesystem sharing uses the fixed filesystem slot rather than adding devices. The slot exports one host directory, or an aggregate of named @@ -96,10 +107,16 @@ Restore is stricter than cold boot. Guest-visible configuration is read from the snapshot's machine contract. Restore-time input may only: - Select the backend kind that the snapshot records. -- Supply required attachments. -- Bind a new directory export to a captured dormant filesystem slot, when the - snapshot records that capability. +- Supply required attachments. For a `platform` snapshot, these include + substitute read-only layers no longer than their recorded lengths; see + [Host Storage and Access Enforcement](3.4-block-storage.md#344-host-storage-and-access-enforcement). +- Bind a new directory export or aggregate to a captured dormant filesystem + slot, when the snapshot records that capability. - Select processor and RAM activation targets that the contract allows. +- For a `platform` snapshot, supply the activation record of the + [restore packet](5.5-restore-protocol.md#555-restore-packet) and, when the + snapshot was captured with a deny-all network policy, a new network policy; + see [Bounded State, Snapshots, and Restore](3.5-network.md#356-bounded-state-snapshots-and-restore). These process-local inputs never change processor capacity, RAM capacity, the command line, device placement, feature masks, or filesystem and network diff --git a/doc/design/3.1-platform-control-device.md b/doc/design/3.1-platform-control-device.md index 2f3584378..ba088efcd 100644 --- a/doc/design/3.1-platform-control-device.md +++ b/doc/design/3.1-platform-control-device.md @@ -32,8 +32,10 @@ machine omits, and carries: console. - The one-time [restore packet](5.5-restore-protocol.md#555-restore-packet), which gives a restored guest what it needs to repair its clocks, - processors, and memory, and the fresh random bytes from which it - [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh). + processors, and memory, the fresh random bytes from which it + [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh), and, + in a platform clone, the configuration with which it + [activates](5.5-restore-protocol.md#559-platform-clone-activation). - Host time samples, which the guest pairs with its own clock to set and discipline its wall clock. - The generation ID, a random identifier unique to each VM process. Every @@ -137,4 +139,8 @@ immediately after the output operation. After a gated restore, a write to the same port acknowledges that processor, memory, time, entropy, and workload state have been repaired. Host input is -released only after the VMM stops at that acknowledgement boundary. +released only after the VMM stops at that acknowledgement boundary. In a +platform clone, the guest agents validate the activation record before the +acknowledgement and +[activate](5.5-restore-protocol.md#559-platform-clone-activation) the workload +root after it. diff --git a/doc/design/3.4-block-storage.md b/doc/design/3.4-block-storage.md index f7f96e2c5..277aec906 100644 --- a/doc/design/3.4-block-storage.md +++ b/doc/design/3.4-block-storage.md @@ -10,7 +10,7 @@ arbitrary disks. | Role | Access | Purpose | | --- | --- | --- | -| `distro` | Read-only | Curated base operating-system layer | +| `distro` | Read-only | Curated base operating system: one EROFS filesystem, or a GPT disk of ordered layers | | `scratch` | Read-write | Private ext4 upper and work storage | Roles are unique and appear in fixed order; an omitted `distro` role leaves @@ -35,8 +35,10 @@ Each role offers a fixed capability set: virtio 1.0 split rings with indirect descriptors and event-index notification suppression, a per-request segment limit, block-size and topology reporting, and a write-back cache with explicit flush. The `distro` role is also advertised as read-only. Discard is never -offered. The device reports the backing store's capacity and its logical and -physical block sizes. +offered. The device reports the backing store's capacity, or the declared +length of an +[unbound platform-tier layer](#344-host-storage-and-access-enforcement), and +its logical and physical block sizes. ## 3.4.3 Request and Completion Path @@ -74,6 +76,16 @@ block must then be a nonempty, 512-byte-aligned raw regular file that uses cached I/O and is not reached through a final symbolic link. The VMM retains that opened file for flushing and paired-scratch publication. +An unbound `platform`-tier layer may be shorter than the length that its +device presents. A launch that declares a `platform` capture may give the +`distro` layer a logical length larger than its file, typically a small +placeholder, and the snapshot records that length. A restore may then supply +any layer file that is a whole number of logical blocks long and no longer +than the recorded length; a longer file is rejected. The device presents the +declared length and returns zeros for reads past the file's end, so one +template serves every image up to that length, and the padding consumes no +storage. Writes never reach the padding, because the role is read-only. + ## 3.4.5 Snapshot Contract and Quiesce The snapshot contract records each block's role, read-only flag, logical @@ -81,7 +93,8 @@ length, logical and physical block sizes, and identity. Every consumed block shares one identity, a caller-authenticated immutable storage generation; see [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). Platform-tier layers are recorded as unbound, because image binding has not -been consumed, so a restore may supply different layers of the same geometry. +been consumed, so a restore may supply different layers with the same block +sizes, no longer than the recorded logical length. Writable scratch follows one of two policies: - **Paired.** Capture publishes the exact scratch image with its length and diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index 28815f3dd..27a303571 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -170,7 +170,9 @@ to guest ports, at most 64 of them. Neither binds a LAN-facing listener. The host decision names no ports, and the endpoint owns no listener of its own, so publishing follows the ports on which the guest listens. The machine profile passes the ingress mode to the guest in a boot token that callers -cannot supply, and a guest agent then announces the guest's listeners. It +cannot supply, and a platform clone that installs a new policy at restore +receives it in its activation record instead. A guest agent then announces +the guest's listeners. It sends each announcement as one UDP datagram to a fixed port of the IPv4 gateway from a privileged source port, which the workload, holding no capabilities, cannot bind, so the workload cannot forge an announcement. An @@ -222,7 +224,9 @@ The gateway provides a DNS proxy over UDP and TCP port 53. The guest is pointed at the IPv4 gateway when the egress policy permits DNS to it, and otherwise at the IPv6 gateway when generic rules permit DNS there. Exact-endpoint policies never name a gateway, and without a permitted gateway -the guest receives no DNS server. +the guest receives no DNS server. A platform clone that installs a new policy +at restore receives its DNS server in the activation record rather than in a +boot token, because its command line was fixed under a deny-all policy. A guest query passes the normal egress checks before the gateway handles it. Upstream resolution uses the host resolver, whose upstream servers are not @@ -243,10 +247,24 @@ beyond it resets its connection. The snapshot records the network profile, identity, and policy digest. Restore reconstructs a fresh endpoint and requires the same network profile -and policy; native sockets, DNS requests, and flows are never serialized. -Published guest listeners and explicit host forwards are process-local -attachments, so capture and restore reject private-network ingress and -explicit host-loopback access, which create them. +and, with one exception, the same policy; native sockets, DNS requests, and +flows are never serialized. Published guest listeners and explicit host +forwards are process-local attachments, so capture and restore otherwise +reject private-network ingress and explicit host-loopback access, which +create them. + +The exception is a `platform` snapshot captured with a deny-all policy, which +denies egress by default without allow rules, denies host loopback without a +proxy endpoint, and grants no inbound access. Its guest has no DNS server and +announces no listeners, and its endpoint holds no flows, listeners, or +forwards, so a restore of it may supply a new policy, including +private-network ingress, explicit host-loopback access, and forwards. The +endpoint installs that policy before any restored processor runs, and the +guest agents apply the DNS server and ingress mode from the activation record +during [platform-clone activation](5.5-restore-protocol.md#559-platform-clone-activation). +The captured identity still applies, so explicit host-loopback access +requires that the template's IPv4 network be /29 or larger. Every other +snapshot keeps the same-policy rule. Capture quiesces the endpoint, drains completion ownership, and requires the saved queues to contain no unrepresented RX or TX packets. It saves the queue diff --git a/doc/design/3.6-host-filesystem.md b/doc/design/3.6-host-filesystem.md index 67a7dd8e7..0c37a93ea 100644 --- a/doc/design/3.6-host-filesystem.md +++ b/doc/design/3.6-host-filesystem.md @@ -46,7 +46,8 @@ descriptors and event-index notification suppression, but never packed rings or the DAX shared-memory window, so file data always travels through the queues. The machine profile rejects alternate tags, extra queues, shared-memory windows, PCI transport, and aggregates whose children change -after boot. +after the aggregate attaches, at cold boot or, on a dormant slot, at +restore. The guest opens the FUSE session with `INIT`, and every other request fails until it succeeds. The device speaks FUSE protocol version 7 and accepts guest @@ -108,7 +109,11 @@ name, which the guest never reaches, so the guest cannot list or change anything beside the file, a lookup below it fails with `ENOTDIR`, and every request fails with `EACCES` while anything other than a regular file holds the name. A file child takes no path policy, because its mode covers the whole -file. +file. By default, a file child follows its name, so a regular file that host +software renames over the name becomes the child's file. An attachment may +instead pin a file child to the object that its name reached when the export +opened: the child then serves only that object, through held nodes and new +lookups alike, and fails requests rather than serve a replacement. Child names are unique, use `[A-Za-z0-9._-]`, and are at most 64 bytes long. An aggregate holds at most 256 children with at most 128 KiB of policy paths, @@ -254,8 +259,12 @@ could not track or save. The host root is pinned by its platform object identity, device and inode number on Linux or volume and file ID on Windows, and revalidated when the -export opens; each aggregate child's root is pinned the same way. Every -operation stays confined to the export: host path resolution follows no +export opens; each aggregate child's root is pinned the same way. A host may +also supply the identity that it expects for a root or a pinned file child; +the VMM compares it with the object that it actually opened, refuses the +export on a mismatch, and reports the opened identities in its +[attachment receipts](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments). +Every operation stays confined to the export: host path resolution follows no symbolic link or reparse point, never leaves the root, and never follows the final component. On Windows hosts, guest-created links are reparse points that path resolution never follows, ancestors that are links or reparse points are @@ -322,13 +331,14 @@ the matching state: - The next node and handle IDs. - The fixed device shape, access mode, ownership mode, root identity, and access policy, or, for an aggregate, each child's name, kind, mode, root - identity, and policy. + identity, and policy, and whether a file child is pinned. Device state names objects relative to the export root and never records a host path. The snapshot's machine contract records the host side instead: the guest target, access mode, tag, queue counts, direct I/O, cache lifetimes, canonical host path, policy paths, ownership mode, and, for an aggregate, each -child's name, canonical host path, kind, mode, policy, and root identity. Its +child's name, canonical host path, kind, mode, policy, and root identity, and +the identity of each pinned file child's file. Its attachment inventory binds the export, with a live-revalidation reconnect policy, to the root's identity or, for an aggregate, to a digest of its children's names and root identities in order, so a restore can neither drop, @@ -369,8 +379,9 @@ it: 1. Before the guest runs, the launcher supplies the export again. It must reproduce the snapshot's target, mode, policy, ownership mode, canonical host paths, and root identities, and, for an aggregate, the same children - with the same names, kinds, and modes in the same order. Only a dormant - slot may instead gain a new export. + with the same names, kinds, modes, and pinning in the same order, each + pinned file child reaching its recorded file. Only a dormant slot may + instead gain a new export. 2. Before the transport changes any state, the device validates the saved state: a known schema version, the fixed device shape, a FUSE session that the device would negotiate, at minor version 31 or newer, tables within @@ -409,13 +420,16 @@ A dormant capture saves explicit unattached state, with no session, nodes, handles, root identity, or policy. It requires that the guest have no FUSE session on the slot, so the tag must not be mounted; unmounting ends the session. A dormant snapshot may restore with no export or bind one new -directory export, but not an aggregate. The resumed guest then mounts the tag -explicitly, because its boot-time mount step has already run. A snapshot -without this capability cannot add an export, and active state cannot restore -into a dormant slot. +directory export or aggregate, whose children are then fixed for the restored +process. The resumed guest then mounts the tag explicitly, because its +boot-time mount step has already run; the guest agents of a platform clone +learn the export's form from the +[activation record](5.5-restore-protocol.md#559-platform-clone-activation). +A snapshot without this capability cannot add an export, and active state +cannot restore into a dormant slot. Capture records the saved-state version that the export requires. Caller ownership, the access policy of a directory export with allowed or writable -paths, an aggregate, and a file child each require a newer version, so a -release that predates the feature rejects the snapshot rather than restore a -weaker export. +paths, an aggregate, a file child, and a pinned file child each require a +newer version, so a release that predates the feature rejects the snapshot +rather than restore a weaker export. diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index 7f53986f3..afcc89246 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -9,18 +9,23 @@ As Table 4.1 shows, the guest agents run workloads in one of two roots. | Root | Construction | Host content | | --- | --- | --- | -| Layered sandbox root | An overlay of one read-only image layer and private writable scratch | Live host shares bound at declared targets | +| Layered sandbox root | An overlay of a read-only image and a private writable upper layer on scratch or in guest RAM | Live host shares bound at declared targets, or, in a managed lifecycle, host directories bound by a host mapping table | | Blockless managed root | The guest's own initramfs root, without block devices | Host directories bound by a host mapping table | The guest kernel provides the facilities both roots need: virtio-blk, -virtio-fs, EROFS with compression and extended attributes, overlayfs, cgroup -v2 with the memory and process controllers, namespaces, BPF system calls, and -cgroup BPF programs. +virtio-fs, EROFS with compression and extended attributes, ext4, GPT partition +tables, overlayfs, cgroup v2 with the memory and process controllers, +namespaces, BPF system calls, and cgroup BPF programs. ## 4.1.1 Layered Sandbox Root -As Figure 4.1 shows, the layered root combines one immutable EROFS `distro` -layer with one private ext4 scratch device. +As Figure 4.1 shows for an EROFS image with its upper layer on scratch, the +layered root combines the immutable `distro` image with one private writable +upper layer. The image is either one EROFS filesystem or a GPT disk whose +first partition holds image metadata, such as the image's configuration, and +whose later partitions hold read-only ext4 lower layers, ordered from base to +top. The upper layer is on an ext4 scratch device or, in the RAM-backed +overlay layout, in guest RAM. ```text /run/nvx/layers/distro (EROFS) ----------> lower directory @@ -46,34 +51,42 @@ The guest agents assemble the root in this order: container cgroups, and move the guest agents into the guest-agent cgroup. 2. Resolve each role's MMIO address through the kernel's platform-device resources rather than assuming a device name or enumeration order. -3. Check any expected sector count, flush cached block-device buffers, mount - the `distro` layer read-only as EROFS, and check its expected filesystem - UUID. +3. Check any expected sector count and flush cached block-device buffers. + Mount an EROFS image read-only and check its expected filesystem UUID. For + a GPT image, reread the partition table, check its expected disk GUID, and + mount each lower partition read-only. A GPT reader takes the backup + header's location from the primary header's alternate-LBA field rather + than from the device's last block, because the device may present zeros + past a shorter image. 4. Resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and - create its upper and work directories. -5. Mount the overlay with the `distro` layer as its lower directory, - metadata-only copy-up, and extended inode numbers. + create its upper and work directories, or, in the RAM-backed overlay + layout, create them in a RAM filesystem mounted with `nosuid,nodev`. +5. Mount the overlay with the EROFS image, or the GPT image's layers from top + to base, as its lower directories, with metadata-only copy-up and extended + inode numbers. 6. Verify the workload identity in the assembled root, mount each live share at its target inside that root, and write the workload's machine ID to a runtime file. 7. Start the workload behind a barrier and place it in the workload cgroup before releasing it. 8. Wait for the workload, unmount the shares in reverse order and then the - overlay, the `distro` layer, and scratch, and report the workload's status. - -The UUID check detects attachment mistakes; it is not a content-integrity -proof, and neither the guest nor the VMM verifies layer contents. -Metadata-only copy-up interprets overlay metadata stored in the `distro` -layer, so that layer is a trusted input that must not carry untrusted overlay -attributes. A metadata-only change avoids copying file contents, but a data -write still copies the whole lower file into scratch, so scratch capacity must -account for copy-up. Scratch consumes host storage rather than guest RAM and -can run out of space independently of guest memory pressure. + overlay, the `distro` image's mounts, and scratch or the RAM filesystem, + and report the workload's status. + +The UUID and GUID checks detect attachment mistakes; they are not a +content-integrity proof, and neither the guest nor the VMM verifies layer +contents. Metadata-only copy-up interprets overlay metadata stored in the +`distro` layer, so that layer is a trusted input that must not carry untrusted +overlay attributes. A metadata-only change avoids copying file contents, but a +data write still copies the whole lower file into scratch, so scratch capacity +must account for copy-up. Scratch consumes host storage rather than guest RAM +and can run out of space independently of guest memory pressure. The initramfs remains the guest agents' root; it is not another workload -layer. The guest agents always keep the overlay upper layer on scratch, so the -machine's RAM-backed overlay layout, which has no scratch, serves only guests -that declare it. +layer. The guest agents keep the overlay upper layer on scratch or, when the +machine declares the RAM-backed overlay layout, which has no scratch, in guest +RAM, where the workload's writes consume guest memory instead of host +storage. ## 4.1.2 Live Host Shares @@ -113,7 +126,9 @@ The workload's mount namespace inherits these mounts, and teardown unmounts the binds in reverse order before the aggregate. Any refusal or mount failure aborts the sandbox instead of starting the workload without its shares. A managed sandbox records its shares when it is provisioned and reattaches them -on every start. +on every start. A [managed layered root](#414-managed-layered-root) may +instead receive host directories as a host mapping table, and a platform +clone must. ## 4.1.3 Blockless Managed Root @@ -138,3 +153,38 @@ host mapping table rather than an image: before mounting anything, and bind-mount each source at its target with `nosuid,nodev`, read-only when requested. They refuse to run workloads until every announced mapping is mounted, and a failed mount keeps them refusing. + +## 4.1.4 Managed Layered Root + +In a managed lifecycle, the layered sandbox root is a managed root with the +same control features as the blockless managed root, and it uses +[control protocol version 2](4.4-host-guest-control-interface.md#444-protocol-versions); +see also +[Feature Negotiation](4.4-host-guest-control-interface.md#442-feature-negotiation). +Its guest half is a guest runtime library and its host half is part of the +host control library, so every host that adopts this root, through the NVX +launcher or its own adapter, gets the same behavior. It differs from a +one-shot layered root in four ways: + +- **Readiness.** The guest agents assemble the root from boot tokens without + waiting for the host, and only then start the control service and report + readiness. Readiness therefore means that the root is assembled and the + control path is established, and the readiness report carries the identity + of the activation configuration that the guest consumed. +- **Host directories.** The command line may announce a number of host + mappings, which the host delivers over the control interface after + readiness, as for the [blockless managed root](#413-blockless-managed-root). + The guest agents mount the aggregate at a root-only location outside the + workload root and bind each mapping at its target inside the assembled + root, creating targets under the rules of + [Live Host Shares](#412-live-host-shares), with `nosuid,nodev`, and + read-only when requested. An announced count of zero mounts nothing. +- **Executions.** Each execution enters the assembled root with the + workload's identity and isolation and runs in its own child cgroup of the + workload cgroup; see + [Containment](4.2-guest-agents-and-workload-model.md#422-containment). +- **Platform clones.** A clone of a `platform` snapshot receives the same + activation configuration in the restore packet's activation record instead + of boot tokens, and its host directories only as a host mapping table, + because its command line, fixed at capture, carries no share tokens; see + [Platform-Clone Activation](5.5-restore-protocol.md#559-platform-clone-activation). diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index c233f71cc..e523c4555 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -57,11 +57,14 @@ elsewhere. ## 4.2.2 Containment -In the blockless managed root, each execution runs in its own cgroup. When the -workload's first process exits, the guest agent kills every process that -remains in that cgroup and, as the guest's init process, reaps the orphans it -inherits. It reports the outcome only after the cgroup's process count reaches -zero. +In a managed root, each execution runs in its own cgroup: a dedicated +execution cgroup in the blockless managed root, and a child of the workload +cgroup in the +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root). +When the workload's first process exits, the guest agent kills every process +that remains in that cgroup and, as the guest's init process, reaps the +orphans it inherits. It reports the outcome only after the cgroup's process +count reaches zero. The process controller charges a process until it is reaped, so no workload process, not even a zombie, outlives its execution. A failed kill, an unreadable count, or a settlement timeout is reported as a containment diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index a360fb779..6bc14b3dd 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -52,7 +52,9 @@ released. The guest agents' cgroup carries a default memory protection of 16 MiB, a soft reservation that shields the supervisor from reclaim under workload memory pressure but does not guarantee its survival. The workload cgroup carries optional memory and process-count limits and is the unit of -capture freeze, termination, and descendant cleanup. +capture freeze, termination, and descendant cleanup. In a managed lifecycle, +each execution's cgroup is a child of the workload cgroup, so those limits and +the capture freeze cover every execution. In the blockless managed root, each execution joins a dedicated cgroup before the workload runs, for termination and descendant accounting. This path does diff --git a/doc/design/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md index d5e8f4813..550e7d74a 100644 --- a/doc/design/4.4-host-guest-control-interface.md +++ b/doc/design/4.4-host-guest-control-interface.md @@ -23,7 +23,7 @@ Table 4.2 lists the operations of the control protocol. | `READY` | Reports readiness, and is the success reply to requests that return no other data | | `FEATURES` | Reports which control behaviors the guest image provides | | `MAPS` | Delivers a contiguous range of host mapping-table entries | -| `EXEC` | Runs one workload with a timeout, arguments, and an optional working directory and environment | +| `EXEC` | Runs one workload with a timeout, arguments, and an optional working directory and environment, and, in protocol version 2, standard input | | `CANCEL` | Kills the running execution that the request names | | Shutdown | Shuts the VM down gracefully between executions | @@ -33,9 +33,11 @@ and optionally an absolute working directory of at most 4,096 bytes and up to distinct, non-empty keys. The environment either replaces the default environment or is layered over it, each entry replacing the default variable of the same name. Output returns as separate standard output and standard -error streams within the credit window, at most 1 MiB of them combined for -each execution. As Table 4.3 shows, an execution that runs ends with an exit -report, which carries an outcome category and a status. +error streams within the credit window. Protocol version 1 allows at most +1 MiB of them combined for each execution; version 2 streams them without a +total limit, as [Protocol Versions](#444-protocol-versions) describes. As +Table 4.3 shows, an execution that runs ends with an exit report, which +carries an outcome category and a status. **Table 4.3.** Exit outcomes. @@ -45,7 +47,7 @@ report, which carries an outcome category and a status. | `signal` | A signal ended the workload; the status is 128 plus the signal number | | `timeout` | The workload exceeded its timeout and was killed; the status is 124 | | `cancelled` | A cancellation killed the workload; the status is 137 | -| `output-limit` | The workload's combined output exceeded 1 MiB, and the guest agents killed it; the status is 125 | +| `output-limit` | In protocol version 1 only, the workload's combined output exceeded 1 MiB, and the guest agents killed it; the status is 125 | | `failed` | The guest agents could not supervise the workload; the status is 125 | Otherwise, the guest agents reply with an error, which carries one of the @@ -85,12 +87,21 @@ therefore returns a four-byte little-endian bit mask of control features. | 5 | Working-directory refusal | | 6 | Host mapping table | -The blockless managed root provides all six features; the layered sandbox -root provides cancellation, workload account creation, and per-execution -environments. A feature bit is added together with the behavior it names. -Hosts ignore unknown bits and trailing bytes, read an `unsupported-operation` -refusal as no features, and terminate a guest that lacks a feature they need -instead of running workloads without the policy they requested. +The blockless managed root and the +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) +provide all six features. A feature bit is added together with the behavior +it names. Hosts ignore unknown bits and trailing bytes, read an +`unsupported-operation` refusal as no features, and terminate a guest that +lacks a feature they need instead of running workloads without the policy +they requested. + +A host and guest built together as a matched bundle need no `FEATURES` +request when they speak protocol version 2, because the bundle fixes the +features and the guest's readiness report carries its protocol version and +build identity. The host terminates a guest whose identity differs from the +bundle's instead of probing it per operation. A restored guest reports the +build captured in its snapshot, so the host compares it with the bundle that +built the template. ## 4.4.3 Session Semantics @@ -98,10 +109,11 @@ Requests are typed, length-bounded, ordered, and associated with one VM instance and epoch. Credit limits bound queued data in both directions. Malformed, unauthenticated, stale, duplicated, or out-of-order records fail closed. Only one execution runs at a time; any request other than a -cancellation that arrives during an execution is refused as `busy`. Shutdown -is therefore graceful only between executions: a host cancels a running -execution before it requests shutdown, or else terminates the VM, which fails -the execution. +cancellation that arrives during an execution is refused as `busy`. The +standard input and credit records of a version-2 execution belong to the +running execution and are not requests. Shutdown is therefore graceful only +between executions: a host cancels a running execution before it requests +shutdown, or else terminates the VM, which fails the execution. The control session is process-local even when guest parser state is snapshotted. Restore establishes a fresh host capability, peer identity, @@ -110,6 +122,35 @@ any host data reaches them. A request from before the capture therefore can never be replayed into a clone or a resumed process. Secrets and per-execution environments belong on the authenticated control -channel, not in the kernel command line, image layers, scratch, or snapshot -metadata. Boot configuration that the machine profile owns contains only the -non-secret invariants needed before the control channel exists. +channel, not in the kernel command line, image layers, scratch, snapshot +metadata, or the restore packet's activation record. Boot configuration that +the machine profile owns contains only the non-secret invariants needed before +the control channel exists. + +## 4.4.4 Protocol Versions + +The control protocol has two versions, which differ in their readiness report +and their execution streams. A session uses one version throughout. A +version-2 guest states its protocol version and build identity in its +readiness report and, in a managed layered root, the identity of its +activation configuration; a readiness report without them comes from a +version-1 guest. A host that requires another version terminates the guest +instead of downgrading a requested stream. + +- **Version 1** is the protocol that the preceding sections describe. An + execution has no standard input, its output flows within the session's + credit window, and its combined output is limited to 1 MiB. +- **Version 2** adds piped standard input with an explicit end of input, and + gives each execution its own credit windows in both directions. The host + grants credit for standard output and standard error as it consumes them, + and the guest grants credit for standard input as the workload drains it. + Output therefore has no total limit; only data in flight is bounded, and a + guest without credit stops reading the workload's output, which blocks the + workload instead of killing it. Execution credits are separate from the + control console's receive credit, and a cancellation, the exit report, and + session control still progress while an execution's window is exhausted. + +Both versions run one execution at a time and keep every other operation, +outcome, and error. Neither provides a terminal, window resizing, signals +other than cancellation, or concurrent executions; these remain outside the +control protocol. diff --git a/doc/design/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md index 290ef835b..c423a85bc 100644 --- a/doc/design/5.1-boot-protocol.md +++ b/doc/design/5.1-boot-protocol.md @@ -80,10 +80,15 @@ the [time ABI](3.2-clock-and-time.md#324-rates) MSRs, and `tsc_early_khz=` or `lapic_timer_hz=`, from the caller or a snapshot, is rejected. Launchers add runtime tokens of their own: one share token per host-share -child, the number of host mappings for a blockless managed root, and a request -to create the workload account. A caller may also declare the RAM-backed -overlay layout with exactly one `nvx_overlay_upper=ramfs` token; any other -form of that token is rejected. +child, the number of host mappings for a managed root, a request to create +the workload account, and, for a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root), +its activation configuration, whose identity the guest agents report in +readiness. Activation tokens stay small: counts, flags, and identities, never +tables. A launch that declares a `platform` capture adds none of these +runtime tokens, because that tier consumes only invariants. A caller may also +declare the RAM-backed overlay layout with exactly one +`nvx_overlay_upper=ramfs` token; any other form of that token is rejected. Embedded NULs are rejected, and the complete NUL-terminated command line must fit in 64 KiB. The effective string and its SHA-256 digest become part of the @@ -106,5 +111,7 @@ A cold boot: A cold boot writes no TSC: every backend starts the processors' TSCs together. The kernel then discovers the fixed virtio-mmio devices and starts the guest agents, which step the guest clock to host UTC before readiness, -assemble the workload root, apply isolation, and start either the one-shot -workload or the managed control service. +assemble the workload root from boot tokens without waiting for the host, +apply isolation, and start either the one-shot workload or the managed control +service. A host that manages the VM attaches to the control console in +parallel and observes readiness once the control path is established. diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index 1bbac96a9..d9cd91934 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -6,7 +6,12 @@ Readiness is an explicit lifecycle boundary, not the observation of arbitrary console text. A cold boot is usable when the guest agents have initialized the required devices and either prepared the one-shot workload or established the -managed control path. +managed control path. In a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root), +the guest agents assemble the workload root before they establish the control +path, so readiness also means that the root is assembled; a platform clone +reaches the same state through +[activation](5.5-restore-protocol.md#559-platform-clone-activation). ## 5.2.1 Control Frontends @@ -29,7 +34,10 @@ writes one fixed readiness record after every state unit has started and before it releases a restored processor. For a gated restore, host input is still gated at that point, and guest repair and its acknowledgement follow the event. A connection, write, or flush failure aborts startup, and the -endpoint is never saved in a snapshot. +endpoint is never saved in a snapshot. The event marks a started VM, not a +usable one: in a platform clone, activation also follows it, so a host runs +workloads only after the guest agents report readiness over the control +channel. ## 5.2.3 Management Exclusion @@ -61,11 +69,23 @@ of each sandbox, reconciles VMM processes with recorded state, and recovers interrupted launches. Teardown leaves no VMM process, control endpoint, capability, or runtime record, while supplied layers and scratch survive. +Executions also take effect one at a time per sandbox, as in the guest. A host +holds exec requests that arrive while an execution runs in a bounded queue +rather than forwarding them for the guest to refuse as `busy`. A full queue +refuses a request at once, and a waiter whose deadline passes or whose caller +leaves never starts. Stop takes priority over the queue: it fails the +waiters, cancels the running execution, and shuts the VM down, or terminates +the VM when the execution cannot be cancelled within a bound. + The sandbox lifecycle API exposes the same operations with request validation, backend capabilities, durable records, and stable error codes. Its default backend launches the VMM directly with the managed lifecycle and the -blockless managed root. An alternative backend delegates the lifecycle to a -separately supplied native library and an image-backed guest. +blockless managed root. Its image-backed backend runs the +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) +through the host control library and the guest runtime library. An embedding +host may drive the same libraries through its own adapter, for example in a +separately supplied native library, so that every adapter shares one root +assembly, activation, and execution behavior. ## 5.2.5 Authority diff --git a/doc/design/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md index d400ec0e6..de5b1a077 100644 --- a/doc/design/5.3-snapshot-protocol.md +++ b/doc/design/5.3-snapshot-protocol.md @@ -26,7 +26,9 @@ Before they write the snapshot port, the guest agents: 2. Save and suppress RCU stall detection, and the debug stall detectors. 3. For paired scratch, freeze the workload cgroup with a bounded wait, synchronize, and freeze the mounted scratch filesystem, or, for fresh - scratch, require scratch to be unmounted. + scratch, require scratch to be unmounted. A `platform` capture, including + one in the RAM-backed overlay layout, which has no scratch, also requires + that neither the `distro` image nor the overlay is mounted. 4. Write the capture request, with its scratch policy, to the snapshot port. A committed capture terminates the source, so the write returns in the source diff --git a/doc/design/5.4-snapshot-artifact-and-attachment-model.md b/doc/design/5.4-snapshot-artifact-and-attachment-model.md index 5b8c131f3..0075049f1 100644 --- a/doc/design/5.4-snapshot-artifact-and-attachment-model.md +++ b/doc/design/5.4-snapshot-artifact-and-attachment-model.md @@ -122,7 +122,7 @@ lifecycle tiers. | Tier | Restore policy | Scratch | Consumed configuration | Sharing scope | | --- | --- | --- | --- | --- | -| `platform` | Clone | Fresh | Invariants only | Trusted platform build | +| `platform` | Clone | Fresh, or none in the RAM-backed overlay layout | Invariants only | Trusted platform build | | `workload-start` | Clone | Paired | Invariants, image binding, and sandbox | Tenant-scoped warm workload or runtime | | `instance-checkpoint` | Resume | Paired | Invariants, image binding, and sandbox | One live instance with one claimed continuation | @@ -130,7 +130,9 @@ Tier metadata requires sandbox blocks; blockless snapshots do not declare a tier. The saved host-owned tier token must agree with the manifest. Platform lower layers have unbound identities, whereas later tiers bind every consumed layer. The consumed-section set records which configuration the guest had -already consumed at capture. +already consumed at capture. A platform clone receives the image binding and +sandbox sections that it has not consumed through +[platform-clone activation](5.5-restore-protocol.md#559-platform-clone-activation). The platform point removes kernel and guest agent initialization from later launches. A workload-start snapshot also removes runtime initialization, but @@ -183,23 +185,35 @@ declarative reconstruction policy. | Resource | Saved | Reconstructed or supplied on restore | | --- | --- | --- | -| Byte channel | Pending RX and TX bytes | Host endpoint, a fresh generation ID, and an optional restore packet | +| Byte channel | Pending RX and TX bytes | Host endpoint, a fresh generation ID, and a restore packet, which may carry a platform clone's activation record | | Boot console | Queue progress, staged RX, partial TX, endpoint identity, and reconnect policy | An eligible listener at its captured endpoint or an approved fresh endpoint of the same kind, a required client connection, or a supplied inherited terminal | | Control console | Attachment identity, broker policy, and bounded broker state without queued records, host output, or receive credit; never the capability or peer identity | An approved same-user local endpoint, possibly at a fresh location, with a fresh capability; the broker restarts with a fresh instance and epoch | -| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy; published guest listeners and host forwards are rejected | -| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export | -| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | +| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy, or with a new policy for a `platform` snapshot captured with a deny-all policy; otherwise, published guest listeners and host forwards are rejected | +| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export or aggregate | +| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, or substitutes no longer than their recorded lengths for a `platform` snapshot, plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | | Readiness endpoint | Nothing | An optional single-use endpoint for one restore | Attachment resolution happens before any processor starts. A supplied attachment must reproduce the saved identity, except that an eligible listener may replace only its endpoint while keeping its stable ID, attachment -and backend kinds, reconnect policy, required flag, length, and timeout. -Missing privileges, endpoint binding failures, a changed egress policy, -replaced filesystem objects, or a wrong attachment kind fail restore -explicitly. Orchestrators of reusable clones supply fresh private boot-console -and control endpoints, so that independent or concurrent restores never -collide with the terminated source. +and backend kinds, reconnect policy, required flag, length, and timeout, and +that a `platform` snapshot accepts substitute read-only layers and, when +captured with a deny-all policy, a new network policy. Missing privileges, +endpoint binding failures, any other changed egress policy, replaced +filesystem objects, or a wrong attachment kind fail restore explicitly. +Orchestrators of reusable clones supply fresh private boot-console and control +endpoints, so that independent or concurrent restores never collide with the +terminated source. + +For the attachments that a restore supplies, a host may name the identity +that it expects for each read-only layer, filesystem root, and pinned file +child. During attachment resolution, before any processor runs, the VMM +compares each with the object that it actually opened and fails the restore +on a mismatch, so an object replaced between the host's check and the VMM's +open never reaches the guest. The VMM also reports receipts to its launcher +that record, for each such attachment, its role and the identity and actual +length of the opened object. Receipts record identities, not contents, so +producing them reads no object's data. ## 5.4.8 Memory Sharing diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index bf04443f0..bc3aaa5dc 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -43,9 +43,10 @@ Before worker construction, restore: 7. Resolves console, network, filesystem, policy, and read-only layer attachments by stable ID or role. 8. Validates the state, captured memory, selected RAM target, block geometry - and identities, and any paired scratch, and claims a resume-policy - snapshot. -9. Generates the restore entropy and generation ID, and sets the generation + and identities, any attachment identities that the host expects, and any + paired scratch, and claims a resume-policy snapshot. +9. Generates the restore entropy and generation ID, checks the size of any + activation record that the launcher supplied, and sets the generation counter to the captured value plus one. 10. Maps captured memory privately with copy-on-write, adds fresh private backing for selected expansion ranges, and materializes scratch according @@ -201,7 +202,7 @@ every tier and for untiered snapshots. ```text offset size field 0 3 magic "OVR" - 3 1 version = 4 + 3 1 version = 4, or 5 with an activation record 4 1 flags 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 6 1 memory_range_count 0 unless MEMORY_TARGET @@ -212,6 +213,8 @@ offset size field 24 8 utc_ns u64, little-endian, host UTC at selection 32 16*n ranges n x { u64 gpa_start, u64 length }, LE 32+16n 64 entropy + 96+16n 4 activation_length version 5 only: u32, little-endian, 1..1024 +100+16n m activation version 5 only: m = activation_length bytes ``` **Figure 5.7.** Restore packet layout. @@ -227,13 +230,21 @@ offset size field | 4-7 | | Zero; the guest rejects a packet with any of them set | The UTC field is latched when the guest first selects the packet; every other -field is sealed before the first restored processor runs. A packet without -ranges is 96 bytes. The packet is authoritative, and the status bits mirror -it; the acknowledgement flag spares the guest from inferring gating from the -tier and targets. A VM process exposes at most one packet, and reading its -last byte consumes it, so a guest captured again in the same process never -sees an earlier packet. The guest also ignores a packet whose generation -equals its recorded generation. +field, including any activation record, is sealed before the first restored +processor runs. A version-4 packet without ranges is 96 bytes, and version 5 +adds 4 bytes plus the record. The packet is authoritative, and the status bits +mirror it; the acknowledgement flag spares the guest from inferring gating +from the tier and targets. A VM process exposes at most one packet, and +reading its last byte consumes it, so a guest captured again in the same +process never sees an earlier packet. The guest also ignores a packet whose +generation equals its recorded generation. + +The activation record is opaque to the VMM. Only a restore of a `platform` +snapshot accepts one, of 1 to 1,024 bytes, and the VMM never stores it in a +snapshot. The VMM writes version 5 only when a record is present, so a guest +captured before version 5 restores unchanged. The guest agents and the host +own the record's schema, as +[Platform-Clone Activation](#559-platform-clone-activation) describes. ## 5.5.6 Generations @@ -259,7 +270,7 @@ runs before the acknowledgement: a restore. 2. Bracket the packet selection with realtime clock reads, read the packet four bytes per access, and validate its magic, version, reserved bits, - counts, generation, and generation ID. + counts, any activation record's length, generation, and generation ID. 3. Set the wall clock: derive the offset and its uncertainty from the bracket, take up to two further time samples if the uncertainty exceeds 1 ms, step the clock by the offset, and set the frequency correction to the @@ -273,6 +284,11 @@ runs before the acknowledgement: restore the detector settings, and restart the wall-clock discipline at its fast cadence. +A platform clone whose packet carries an activation record also validates the +record before the acknowledgement and activates after it, on the readiness +path rather than at idle priority, as +[Platform-Clone Activation](#559-platform-clone-activation) describes. + A failure in the time ABI's part of the repair, which validates the packet, steps the wall clock, and restores the stall-detector settings, emits a violation event and powers off the guest with status 195, and a failed rerun @@ -337,3 +353,57 @@ run, and gating host input does not hold it back. The snapshot's caller must therefore provide the execution barrier, for example by capturing before any workload starts, and, where the guest does not reseed, repair the kernel's and the runtime's random state with the entropy handed to it. + +## 5.5.9 Platform-Clone Activation + +A `platform` snapshot consumes only invariants (Table 5.3), so each clone of a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) +still needs the image binding and sandbox configuration that a cold boot reads +from boot tokens. A restore appends no tokens to the captured command line, so +the host supplies this activation configuration as the activation record of +the [restore packet](#555-restore-packet). One configuration therefore has +two deliveries, boot tokens on a cold boot and the activation record in a +platform clone, and the guest agents validate both with the same rules and +report the same identity in readiness. A launch that declares a `platform` +capture carries no activation configuration: its guest agents capture before +they assemble the root, and every clone activates from its own record. + +The record is non-secret and small, because the guest reads it four bytes per +port exit. It carries: + +- The identity of the configuration, which readiness reports. +- The expected identity and sector count of the substituted `distro` image. +- The upper-layer settings that the template did not capture. +- The form of any export that the restore bound to the dormant filesystem + slot, and the number of host mappings. +- For a template captured with a deny-all network policy, the DNS server, the + ingress mode, and the guest's settings for any gateway proxy endpoint that + the new policy permits. + +Secrets, per-execution environments, and nonempty host mapping tables never +enter the record. They use the control channel after readiness, as they do +after a cold boot. + +Activation proceeds as follows: + +1. During [guest repair](#557-guest-repair), the guest agents read the record + with the packet and validate it before they acknowledge the restore. +2. After the acknowledgement, they flush the cached buffers of the + substituted `distro` image and assemble the root as steps 3 through 6 of + [Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) + describe, checking the image against the record. +3. They mount any export that the restore bound to the dormant filesystem slot + and apply the record's network settings. +4. They establish the control path with the restore's fresh capability, + instance, and epoch, and report readiness with the configuration's + identity. +5. The host checks that identity and then delivers any host mapping table; + the guest agents refuse executions until every announced mapping is + mounted. + +The guest agents of a managed layered root captured at the `platform` tier +require the record and fail repair without one. An activation failure writes +an error line and powers off the guest with status 1, as other repair failures +do, so a partially activated clone never becomes ready. A platform clone of a +guest that does not await activation needs no record and receives no +configuration beyond its captured command line. diff --git a/doc/design/5.6-shutdown-and-failure-semantics.md b/doc/design/5.6-shutdown-and-failure-semantics.md index 1873d6e63..93d2ed7a0 100644 --- a/doc/design/5.6-shutdown-and-failure-semantics.md +++ b/doc/design/5.6-shutdown-and-failure-semantics.md @@ -59,6 +59,9 @@ Failures are classified by whether execution can safely continue: - A failure after snapshot publication never resumes the source. - A post-restore repair failure before the acknowledgement, or a missing acknowledgement, leaves input gated and terminates the restored process. +- A platform-clone activation failure after the acknowledgement powers off + the guest before it reports readiness, so no workload runs in a partially + activated clone. - Malformed saved state or failed attachment reconstruction prevents any restored virtual processor from running. From 3e1c6005775c1a070089484fc1bc833f194088d5 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Fri, 16 Oct 2026 14:37:32 -0700 Subject: [PATCH 2/9] doc: resolve review findings in the platform-clone guide changes Make the guide changes consistent with the rest of the guide and tighten their wording: - run the boot conformance checks before a managed template's platform capture, which precedes readiness; - end platform-clone activation at readiness, so a host mapping failure keeps refusing executions as after a cold boot; - send the version-2 exit report after the last output of both streams; - carry the workload-account request and the image's UUID or GUID in the activation record; - let hosts offer a new network policy only to clones of a managed layered root, which learn their DNS server and ingress mode from the record; - make scratch optional in restore reconstruction for the RAM-backed layout; - complete Start once every announced mapping is mounted, and qualify the failure summary for guest-validated activation records. Refs #535 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-system-components-and-responsibilities.md | 10 +++--- doc/design/1.4-trust-boundaries.md | 8 ++--- .../1.5-compatibility-and-versioning-model.md | 10 +++--- .../2.1-machine-model-and-configuration.md | 2 +- doc/design/3.2-clock-and-time.md | 4 ++- doc/design/3.5-network.md | 25 ++++++------- .../4.1-guest-filesystem-and-image-model.md | 31 ++++++++-------- .../4.3-isolation-and-resource-model.md | 9 ++--- .../4.4-host-guest-control-interface.md | 6 ++-- .../5.2-readiness-and-runtime-control.md | 23 ++++++------ ...-snapshot-artifact-and-attachment-model.md | 13 ++++--- doc/design/5.5-restore-protocol.md | 35 +++++++++---------- .../5.6-shutdown-and-failure-semantics.md | 3 +- 13 files changed, 89 insertions(+), 90 deletions(-) diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index 6aa468d32..d3d0dfc39 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -78,13 +78,11 @@ flowchart TB and egress-rule translation. It also implements the host half of the [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root): its root plan, activation configuration, execution admission, and - execution streams. Adapters connect the library to an orchestrator: the - sandbox lifecycle API is one, and an embedding host with its own - orchestration, storage, and admission policy is another. An adapter - supplies policy, resources, and transport, and reuses the library's + execution streams. Its adapters, the sandbox lifecycle API and embedding + hosts, supply policy, resources, and transport, and reuse the library's validation and execution semantics instead of reimplementing them. -- **Embedding host adapters** connect orchestrators outside NVX to the host - control library. +- **Embedding host adapters** connect orchestrators outside NVX, with their + own storage and admission policy, to the host control library. - **Snapshot storage** retains immutable machine state, memory, and optional paired scratch together with the identities needed to reconstruct external resources. diff --git a/doc/design/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md index 78bf8f40b..5e51def76 100644 --- a/doc/design/1.4-trust-boundaries.md +++ b/doc/design/1.4-trust-boundaries.md @@ -89,11 +89,9 @@ stored in lower layers, and bytes read from any layer can appear in captured memory. A snapshot identifies its blocks by a storage generation, which transfers trust for layer and scratch contents to the storage owner; the VMM trusts the control plane in the host to authenticate that generation. For the -attachments that a restore supplies, a host may name the identity that it -expects for each object, and the VMM compares it with the object that it -actually opened before any processor runs, so an object replaced between the -host's check and the VMM's open never reaches the guest. The VMM also reports -each opened object's identity and actual length in a receipt; see +attachments that a restore supplies, the VMM checks any identities that the +host expects before any processor runs and reports the opened identities in +receipts; see [Host Attachments](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments). A receipt identifies an object, not its contents. diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index d5ab14842..7842257e5 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -63,11 +63,11 @@ A reusable template represents a compatibility class, not an arbitrary fleet image. The class includes the backend, the CPU profile and generation, the effective CPUID, the guest kernel and guest agent build, the effective command line, device roles and geometry, network identity, network policy unless a -`platform` template was captured with a deny-all policy, and consumed layer -identities. Restoring guest memory restores the captured guest agent build, -not a newly deployed initramfs, so deployments rebuild or recertify templates -on relevant rollouts. That build also fixes the activation-configuration -schema that the template's clones accept. +`platform` template of a managed layered root was captured with a deny-all +policy, and consumed layer identities. Restoring guest memory restores the +captured guest agent build, not a newly deployed initramfs, so deployments +rebuild or recertify templates on relevant rollouts. That build also fixes the +activation-configuration schema that the template's clones accept. Processor capacity and RAM geometry are immutable, but restore-time activation lets one template serve several online processor counts and RAM sizes. diff --git a/doc/design/2.1-machine-model-and-configuration.md b/doc/design/2.1-machine-model-and-configuration.md index 73f07f5eb..7b54621fe 100644 --- a/doc/design/2.1-machine-model-and-configuration.md +++ b/doc/design/2.1-machine-model-and-configuration.md @@ -60,7 +60,7 @@ lifetime is one of: | Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities, or, for the unbound layers of a `platform` snapshot, within the recorded lengths | VM process | | Online processor and RAM activation targets | Host, within snapshot limits | Restore | | Activation configuration of a platform clone | Host, validated by the guest agents | Restore | -| Network policy of a platform clone whose template was captured with a deny-all policy | Host | Restore | +| Network policy of a managed layered root's platform clone whose template was captured with a deny-all policy | Host | Restore | The host fixes the workload policy of a fresh boot. An optional nonzero numeric UID and GID become the fixed workload identity, and an optional diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index 5c0c3e808..6bddcd660 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -365,7 +365,9 @@ synchronization, offset, uncertainty, and frequency correction. the bound and steps its clock to host UTC, so the workload starts on host time. It runs every other boot check asynchronously after readiness, starting at idle priority, so the checks never delay readiness or take a processor from -the workload. The checks cover the identity leaves and explicit zero leaves, +the workload. A managed layered root launched for a `platform` capture reports +readiness only in its clones, so its guest agents run these checks before the +capture instead. The checks cover the identity leaves and explicit zero leaves, the CPU time bits on every CPU, the identity MSRs, the kernel's platform, clocksource, and LAPIC-timer selection, CPU flags, tick devices, the absence of forbidden clock parameters, stall-detector settings, a running violation diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index 27a303571..bbe6ea684 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -253,18 +253,19 @@ forwards are process-local attachments, so capture and restore otherwise reject private-network ingress and explicit host-loopback access, which create them. -The exception is a `platform` snapshot captured with a deny-all policy, which -denies egress by default without allow rules, denies host loopback without a -proxy endpoint, and grants no inbound access. Its guest has no DNS server and -announces no listeners, and its endpoint holds no flows, listeners, or -forwards, so a restore of it may supply a new policy, including -private-network ingress, explicit host-loopback access, and forwards. The -endpoint installs that policy before any restored processor runs, and the -guest agents apply the DNS server and ingress mode from the activation record -during [platform-clone activation](5.5-restore-protocol.md#559-platform-clone-activation). -The captured identity still applies, so explicit host-loopback access -requires that the template's IPv4 network be /29 or larger. Every other -snapshot keeps the same-policy rule. +The exception is a `platform` snapshot captured with a deny-all policy: a +default deny without allow rules, host loopback denied without a proxy +endpoint, and no inbound access. Its guest has no DNS server and announces no +listeners, and its endpoint holds no flows, listeners, or forwards, so a +restore may supply a new policy, including private-network ingress, explicit +host-loopback access, and forwards. The endpoint installs that policy before +any restored processor runs. Because the guest learns its DNS server and +ingress mode only from the +[activation record](5.5-restore-protocol.md#559-platform-clone-activation), +hosts supply a new policy only to clones of a managed layered root. The +captured identity still applies, so explicit host-loopback access requires an +IPv4 network of /29 or larger. Every other snapshot keeps the same-policy +rule. Capture quiesces the endpoint, drains completion ownership, and requires the saved queues to contain no unrepresented RX or TX packets. It saves the queue diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index afcc89246..fdcd68db6 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -78,9 +78,9 @@ content-integrity proof, and neither the guest nor the VMM verifies layer contents. Metadata-only copy-up interprets overlay metadata stored in the `distro` layer, so that layer is a trusted input that must not carry untrusted overlay attributes. A metadata-only change avoids copying file contents, but a -data write still copies the whole lower file into scratch, so scratch capacity -must account for copy-up. Scratch consumes host storage rather than guest RAM -and can run out of space independently of guest memory pressure. +data write still copies the whole lower file into the upper layer, whose +capacity must account for copy-up. Scratch consumes host storage rather than +guest RAM and can run out of space independently of guest memory pressure. The initramfs remains the guest agents' root; it is not another workload layer. The guest agents keep the overlay upper layer on scratch or, when the @@ -156,21 +156,20 @@ host mapping table rather than an image: ## 4.1.4 Managed Layered Root -In a managed lifecycle, the layered sandbox root is a managed root with the -same control features as the blockless managed root, and it uses -[control protocol version 2](4.4-host-guest-control-interface.md#444-protocol-versions); -see also -[Feature Negotiation](4.4-host-guest-control-interface.md#442-feature-negotiation). -Its guest half is a guest runtime library and its host half is part of the -host control library, so every host that adopts this root, through the NVX -launcher or its own adapter, gets the same behavior. It differs from a -one-shot layered root in four ways: +In a managed lifecycle, the layered sandbox root is a managed root: it +provides the same +[control features](4.4-host-guest-control-interface.md#442-feature-negotiation) +as the blockless managed root and uses +[control protocol version 2](4.4-host-guest-control-interface.md#444-protocol-versions). +A guest runtime library implements its guest half and the host control +library its host half, so every adapter that adopts this root behaves the +same. It differs from a one-shot layered root in four ways: - **Readiness.** The guest agents assemble the root from boot tokens without - waiting for the host, and only then start the control service and report - readiness. Readiness therefore means that the root is assembled and the - control path is established, and the readiness report carries the identity - of the activation configuration that the guest consumed. + waiting for the host, then start the control service. Readiness therefore + means that the root is assembled and the control path is established, and + the readiness report names the activation configuration that the guest + consumed. - **Host directories.** The command line may announce a number of host mappings, which the host delivers over the control interface after readiness, as for the [blockless managed root](#413-blockless-managed-root). diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 6bc14b3dd..8b707436e 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -8,10 +8,11 @@ The workload runs under a host-selected, non-root numeric UID and GID fixed at initial boot. The guest agents require an exact user and primary-group match in the workload root and reject an unavailable identity before starting the -workload. When the boot configuration requests it, they instead create an -account for a host-selected identity that the image lacks, and replace an -image account that uses the UID with another primary group or without a -usable home. This lets a host run workloads under its own user's IDs. +workload. When the boot configuration or a platform clone's activation record +requests it, they instead create an account for a host-selected identity that +the image lacks, and replace an image account that uses the UID with another +primary group or without a usable home. This lets a host run workloads under +its own user's IDs. Before the workload starts, the guest agents clear supplementary groups and every capability set and enable `no_new_privs`. With every capability cleared, diff --git a/doc/design/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md index 550e7d74a..5182002d1 100644 --- a/doc/design/4.4-host-guest-control-interface.md +++ b/doc/design/4.4-host-guest-control-interface.md @@ -147,8 +147,10 @@ instead of downgrading a requested stream. Output therefore has no total limit; only data in flight is bounded, and a guest without credit stops reading the workload's output, which blocks the workload instead of killing it. Execution credits are separate from the - control console's receive credit, and a cancellation, the exit report, and - session control still progress while an execution's window is exhausted. + control console's receive credit. The exit report follows the last output + of both streams, so a host that withholds credit delays it, but a + cancellation, credit grants, and session control still progress while an + execution's window is exhausted. Both versions run one execution at a time and keep every other operation, outcome, and error. Neither provides a terminal, window resizing, signals diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index d9cd91934..1ccf0197a 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -58,7 +58,7 @@ operations. | Operation | Effect | | --- | --- | | Provision | Records the sandbox's configuration, including its host shares | -| Start | Launches the VMM with a freshly generated control capability, reattaches the recorded shares, and waits for guest readiness | +| Start | Launches the VMM with a freshly generated control capability, reattaches the recorded shares, waits for guest readiness, and delivers any host mapping table, completing once every announced mapping is mounted | | Exec | Runs one workload over the control channel | | Stop | Shuts down the VM and releases its runtime resources | | Deprovision | Removes the sandbox's state | @@ -69,23 +69,22 @@ of each sandbox, reconciles VMM processes with recorded state, and recovers interrupted launches. Teardown leaves no VMM process, control endpoint, capability, or runtime record, while supplied layers and scratch survive. -Executions also take effect one at a time per sandbox, as in the guest. A host -holds exec requests that arrive while an execution runs in a bounded queue -rather than forwarding them for the guest to refuse as `busy`. A full queue -refuses a request at once, and a waiter whose deadline passes or whose caller -leaves never starts. Stop takes priority over the queue: it fails the -waiters, cancels the running execution, and shuts the VM down, or terminates -the VM when the execution cannot be cancelled within a bound. +The host runs one execution at a time per sandbox, as the guest does, and +holds requests that arrive meanwhile in a bounded queue rather than forwarding +them for the guest to refuse as `busy`. A full queue refuses a request at +once, and a waiter whose deadline passes or whose caller leaves never starts. +Stop takes priority over the queue: it fails the waiters, cancels the running +execution, and shuts the VM down, or terminates the VM when the execution +cannot be cancelled within a bound. The sandbox lifecycle API exposes the same operations with request validation, backend capabilities, durable records, and stable error codes. Its default backend launches the VMM directly with the managed lifecycle and the blockless managed root. Its image-backed backend runs the [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) -through the host control library and the guest runtime library. An embedding -host may drive the same libraries through its own adapter, for example in a -separately supplied native library, so that every adapter shares one root -assembly, activation, and execution behavior. +through the host control library and the guest runtime library, which an +embedding host may also drive through its own adapter, for example in a +separately supplied native library. ## 5.2.5 Authority diff --git a/doc/design/5.4-snapshot-artifact-and-attachment-model.md b/doc/design/5.4-snapshot-artifact-and-attachment-model.md index 0075049f1..692468361 100644 --- a/doc/design/5.4-snapshot-artifact-and-attachment-model.md +++ b/doc/design/5.4-snapshot-artifact-and-attachment-model.md @@ -190,7 +190,7 @@ declarative reconstruction policy. | Control console | Attachment identity, broker policy, and bounded broker state without queued records, host output, or receive credit; never the capability or peer identity | An approved same-user local endpoint, possibly at a fresh location, with a fresh capability; the broker restarts with a fresh instance and epoch | | Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy, or with a new policy for a `platform` snapshot captured with a deny-all policy; otherwise, published guest listeners and host forwards are rejected | | Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export or aggregate | -| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, or substitutes no longer than their recorded lengths for a `platform` snapshot, plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | +| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, or substitutes no longer than their recorded lengths for a `platform` snapshot, and, unless the snapshot has no scratch, private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | | Readiness endpoint | Nothing | An optional single-use endpoint for one restore | Attachment resolution happens before any processor starts. A supplied @@ -208,12 +208,11 @@ terminated source. For the attachments that a restore supplies, a host may name the identity that it expects for each read-only layer, filesystem root, and pinned file child. During attachment resolution, before any processor runs, the VMM -compares each with the object that it actually opened and fails the restore -on a mismatch, so an object replaced between the host's check and the VMM's -open never reaches the guest. The VMM also reports receipts to its launcher -that record, for each such attachment, its role and the identity and actual -length of the opened object. Receipts record identities, not contents, so -producing them reads no object's data. +compares each with the object that it opened and fails the restore on a +mismatch, so an object replaced after the host's check never reaches the +guest. The VMM also reports a receipt to its launcher for each such +attachment, with its role and the opened object's identity and actual length. +Receipts identify objects, not contents, so producing them reads no data. ## 5.4.8 Memory Sharing diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index bc3aaa5dc..c19cbc4f3 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -356,26 +356,25 @@ the runtime's random state with the entropy handed to it. ## 5.5.9 Platform-Clone Activation -A `platform` snapshot consumes only invariants (Table 5.3), so each clone of a +A [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) -still needs the image binding and sandbox configuration that a cold boot reads -from boot tokens. A restore appends no tokens to the captured command line, so -the host supplies this activation configuration as the activation record of -the [restore packet](#555-restore-packet). One configuration therefore has -two deliveries, boot tokens on a cold boot and the activation record in a -platform clone, and the guest agents validate both with the same rules and -report the same identity in readiness. A launch that declares a `platform` -capture carries no activation configuration: its guest agents capture before -they assemble the root, and every clone activates from its own record. +reads its activation configuration from boot tokens on a cold boot and from +the activation record of the [restore packet](#555-restore-packet) in a +platform clone, because a restore never extends the captured command line. +Both deliveries use the same validation rules and report the same identity in +readiness. A launch that declares a `platform` capture carries no activation +configuration, and its guest agents capture before they assemble the root. The record is non-secret and small, because the guest reads it four bytes per port exit. It carries: - The identity of the configuration, which readiness reports. -- The expected identity and sector count of the substituted `distro` image. +- The filesystem UUID or disk GUID that the substituted `distro` image must + have. - The upper-layer settings that the template did not capture. - The form of any export that the restore bound to the dormant filesystem slot, and the number of host mappings. +- Whether to create the workload account. - For a template captured with a deny-all network policy, the DNS server, the ingress mode, and the guest's settings for any gateway proxy endpoint that the new policy permits. @@ -397,13 +396,13 @@ Activation proceeds as follows: 4. They establish the control path with the restore's fresh capability, instance, and epoch, and report readiness with the configuration's identity. -5. The host checks that identity and then delivers any host mapping table; - the guest agents refuse executions until every announced mapping is - mounted. + +Readiness completes activation. The host then checks the reported identity +and delivers any host mapping table, which the guest agents mount as after a +cold boot, refusing executions until every announced mapping is mounted. The guest agents of a managed layered root captured at the `platform` tier require the record and fail repair without one. An activation failure writes -an error line and powers off the guest with status 1, as other repair failures -do, so a partially activated clone never becomes ready. A platform clone of a -guest that does not await activation needs no record and receives no -configuration beyond its captured command line. +an error line and powers off the guest with status 1, so a partially activated +clone never becomes ready. A platform clone of any other guest needs no record +and receives no configuration beyond its captured command line. diff --git a/doc/design/5.6-shutdown-and-failure-semantics.md b/doc/design/5.6-shutdown-and-failure-semantics.md index 93d2ed7a0..170e34378 100644 --- a/doc/design/5.6-shutdown-and-failure-semantics.md +++ b/doc/design/5.6-shutdown-and-failure-semantics.md @@ -49,7 +49,8 @@ describes. Failures are classified by whether execution can safely continue: -- Invalid configuration, artifacts, or attachments fail before guest entry. +- Invalid configuration, artifacts, or attachments fail before guest entry, + except an activation record, which the guest agents validate during repair. - Time-contract violations reject a cold boot, capture, or restore with a stable error code and never fall back. - A rejected capture preflight may return the still-consistent source to From 8459ea3c2f0bad8ef0a285e9a3dd9cdb0b78c0f8 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 17 Oct 2026 13:00:20 -0700 Subject: [PATCH 3/9] doc: name external orchestrators and external guest agents Replace "embedding host adapters" with external orchestrators, which drive the host control library through their own adapters, and describe an external guest agent that runs as the guest's init process and embeds the guest runtime library. The guide already uses "orchestrator" for the software that drives NVX and "guest agents" for its guest software. Refs #535 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-system-components-and-responsibilities.md | 23 +++++++++++-------- .../5.2-readiness-and-runtime-control.md | 7 +++--- 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index d3d0dfc39..d3a12da4d 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -17,7 +17,7 @@ flowchart TB direction LR Launcher["NVX launcher"] LifecycleApi["Sandbox lifecycle API"] - Embedder["Embedding host adapters"] + External["External orchestrators"] ControlLibrary["Host control library"] Storage["Snapshot storage"] Attachments["External attachments"] @@ -43,7 +43,7 @@ flowchart TB Launcher --> Frontends LifecycleApi --> ControlLibrary - Embedder --> ControlLibrary + External --> ControlLibrary ControlLibrary --> Frontends Launcher -. control session .-> Virtio ControlLibrary -. control session .-> Virtio @@ -78,11 +78,13 @@ flowchart TB and egress-rule translation. It also implements the host half of the [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root): its root plan, activation configuration, execution admission, and - execution streams. Its adapters, the sandbox lifecycle API and embedding - hosts, supply policy, resources, and transport, and reuse the library's - validation and execution semantics instead of reimplementing them. -- **Embedding host adapters** connect orchestrators outside NVX, with their - own storage and admission policy, to the host control library. + execution streams. Its adapters, in the sandbox lifecycle API and in + external orchestrators, supply policy, resources, and transport, and reuse + the library's validation and execution semantics instead of reimplementing + them. +- **External orchestrators** are orchestrators outside NVX, with their own + storage and admission policy, that drive the host control library through + their own adapters. - **Snapshot storage** retains immutable machine state, memory, and optional paired scratch together with the identities needed to reconstruct external resources. @@ -134,9 +136,10 @@ flowchart TB the guest's listeners when the host publishes them, check time ABI conformance, watch for clock violations, discipline the wall clock, and perform the snapshot time steps. Their managed layered root comes from a - guest runtime library that another guest init can embed through its own - adapter, so root assembly, host mappings, and execution supervision behave - the same for every host. + guest runtime library. An external guest agent may run as the guest's init + process instead, embedding the same library through its own adapter, so root + assembly, host mappings, and execution supervision behave the same for every + host. - **Workload** is the microVM's single tenant: it owns the microVM's lifetime, resource envelope, and network identity, and it may run any number of processes, all inside the isolation boundary that the guest agents diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index 1ccf0197a..845767ba9 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -82,9 +82,10 @@ validation, backend capabilities, durable records, and stable error codes. Its default backend launches the VMM directly with the managed lifecycle and the blockless managed root. Its image-backed backend runs the [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) -through the host control library and the guest runtime library, which an -embedding host may also drive through its own adapter, for example in a -separately supplied native library. +through the host control library and the guest runtime library. An external +orchestrator may drive the same host library through its own adapter, for +example in a separately supplied native library, and an external guest agent +may embed the same guest library. ## 5.2.5 Authority From 8668aed140ff8dbc79da1062e6ab00680389b32f Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 19 Oct 2026 21:34:14 -0700 Subject: [PATCH 4/9] doc: define image assembly and boot tokens Define bounded primary-GPT validation, explicit image formats, overlay option sets, nested mapping rules, and the layered-root account policy. Align the effective command-line limit and RAM-backed upper token with the guest kernel contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../4.1-guest-filesystem-and-image-model.md | 67 ++++++++++++------- doc/design/5.1-boot-protocol.md | 27 +++++--- 2 files changed, 59 insertions(+), 35 deletions(-) diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index fdcd68db6..ed99b13b5 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -52,35 +52,48 @@ The guest agents assemble the root in this order: 2. Resolve each role's MMIO address through the kernel's platform-device resources rather than assuming a device name or enumeration order. 3. Check any expected sector count and flush cached block-device buffers. - Mount an EROFS image read-only and check its expected filesystem UUID. For - a GPT image, reread the partition table, check its expected disk GUID, and - mount each lower partition read-only. A GPT reader takes the backup - header's location from the primary header's alternate-LBA field rather - than from the device's last block, because the device may present zeros - past a shorter image. + Mount an EROFS image read-only and check any expected filesystem UUID. For + a GPT image, validate only the primary header and entry array, reread the + partition table, and mount partitions 2 through N as read-only ext4 lower + layers with `ro,nosuid,nodev`. Never use the backup header and never add + `noload`. 4. Resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and create its upper and work directories, or, in the RAM-backed overlay - layout, create them in a RAM filesystem mounted with `nosuid,nodev`. + layout, create them in tmpfs mounted with `nosuid,nodev`. 5. Mount the overlay with the EROFS image, or the GPT image's layers from top - to base, as its lower directories, with metadata-only copy-up and extended - inode numbers. + to base, as its lower directories. Every image uses + `metacopy=on,xino=on` by default. An orchestrator whose lower layers may + carry untrusted overlay attributes selects + `metacopy=off,redirect_dir=nofollow`. The guest uses exactly the selected + option set without fallback and mounts the merged root `nosuid,nodev`. 6. Verify the workload identity in the assembled root, mount each live share - at its target inside that root, and write the workload's machine ID to a - runtime file. + at its target inside that root, and, when selected, write the generated + machine ID to a runtime file. 7. Start the workload behind a barrier and place it in the workload cgroup before releasing it. 8. Wait for the workload, unmount the shares in reverse order and then the - overlay, the `distro` image's mounts, and scratch or the RAM filesystem, + overlay, the `distro` image's mounts, and scratch or tmpfs, and report the workload's status. -The UUID and GUID checks detect attachment mistakes; they are not a -content-integrity proof, and neither the guest nor the VMM verifies layer -contents. Metadata-only copy-up interprets overlay metadata stored in the -`distro` layer, so that layer is a trusted input that must not carry untrusted -overlay attributes. A metadata-only change avoids copying file contents, but a -data write still copies the whole lower file into the upper layer, whose -capacity must account for copy-up. Scratch consumes host storage rather than -guest RAM and can run out of space independently of guest memory pressure. +GPT validation bounds the header and entry array, verifies their CRCs, +extents, indices, and GUIDs, and permits at most 255 used partitions, the +kernel's limit. Partition 1 holds opaque metadata bytes. The guest reads them +up to a caller-selected bound, trims trailing NUL and whitespace, and never +probes the partition as a filesystem; the adapter parses the result. After +the partition-table reread, every kernel partition must match its validated +entry. + +Expected identities are optional inputs: the EROFS filesystem UUID, or the +GPT disk GUID, partition GUIDs, and partition sizes. NVX's adapter passes the +EROFS UUID or GPT disk GUID from its boot token. These checks detect attachment +mistakes; they are not a content-integrity proof, and neither the guest nor +the VMM verifies layer contents. Metadata-only copy-up interprets overlay +metadata stored in the `distro` layer, so an orchestrator selects the +non-metacopy option set when lower layers may carry untrusted overlay +attributes. A metadata-only change avoids copying file contents, but a data +write still copies the whole lower file into the upper layer, whose capacity +must account for copy-up. Scratch consumes host storage rather than guest RAM +and can run out of space independently of guest memory pressure. The initramfs remains the guest agents' root; it is not another workload layer. The guest agents keep the overlay upper layer on scratch or, when the @@ -136,6 +149,8 @@ Without image layers, each workload runs directly in the guest's initramfs root, inside a dedicated execution cgroup. Host directories reach it through a host mapping table rather than an image: +- The host planner refuses a read-only path inside a read-write mapping on + every host. A read-write path inside a read-only mapping remains valid. - The host exports the outermost mapped directories, and each mapped file outside them on its own, as numbered children of one aggregate, so a file child exposes only its file. The guest agents mount the aggregate at a @@ -154,16 +169,18 @@ host mapping table rather than an image: `nosuid,nodev`, read-only when requested. They refuse to run workloads until every announced mapping is mounted, and a failed mount keeps them refusing. +The blockless root may create the workload account that its adapter requests. + ## 4.1.4 Managed Layered Root -In a managed lifecycle, the layered sandbox root is a managed root: it -provides the same -[control features](4.4-host-guest-control-interface.md#442-feature-negotiation) -as the blockless managed root and uses +In a managed lifecycle, the layered sandbox root uses [control protocol version 2](4.4-host-guest-control-interface.md#444-protocol-versions). A guest runtime library implements its guest half and the host control library its host half, so every adapter that adopts this root behaves the -same. It differs from a one-shot layered root in four ways: +same. Unlike the blockless root, it never creates workload accounts. The +identity must already exist in the image, or the adapter must select numeric +IDs without an account check. It differs from a one-shot layered root in four +ways: - **Readiness.** The guest agents assemble the root from boot tokens without waiting for the host, then start the control service. Readiness therefore diff --git a/doc/design/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md index c423a85bc..4e928e266 100644 --- a/doc/design/5.1-boot-protocol.md +++ b/doc/design/5.1-boot-protocol.md @@ -80,20 +80,27 @@ the [time ABI](3.2-clock-and-time.md#324-rates) MSRs, and `tsc_early_khz=` or `lapic_timer_hz=`, from the caller or a snapshot, is rejected. Launchers add runtime tokens of their own: one share token per host-share -child, the number of host mappings for a managed root, a request to create -the workload account, and, for a +child, the number of host mappings for a managed root, a blockless root's +request to create the workload account, and, for a [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root), its activation configuration, whose identity the guest agents report in readiness. Activation tokens stay small: counts, flags, and identities, never tables. A launch that declares a `platform` capture adds none of these -runtime tokens, because that tier consumes only invariants. A caller may also -declare the RAM-backed overlay layout with exactly one -`nvx_overlay_upper=ramfs` token; any other form of that token is rejected. - -Embedded NULs are rejected, and the complete NUL-terminated command line must -fit in 64 KiB. The effective string and its SHA-256 digest become part of the -snapshot machine contract. A restore reuses that string verbatim and appends -no tokens. +runtime tokens, because that tier consumes only invariants. The `distro` +token names the image format without probing fallback: +`nvx_layer=distro,MMIO,erofs[:UUID]` or +`nvx_layer=distro,MMIO,gpt[:DISK-GUID]`. A caller may declare the RAM-backed +overlay layout with exactly one `nvx_overlay_upper=ramfs` token, which selects +the tmpfs upper and permits no scratch block. Any other form or count of that +token is rejected. + +Launchers keep a 1,024-byte caller budget and refuse longer caller input +early. Embedded NULs are rejected. After appending its own tokens, the VMM +requires the effective command line to fit in 2,048 bytes including the +terminating NUL, at most 2,047 bytes of text, which is the guest kernel's +limit. It applies this check to cold boots and restores. The effective string +and its SHA-256 digest become part of the snapshot machine contract. A restore +reuses its saved command line verbatim and never rewrites or appends tokens. ## 5.1.3 Cold Boot Sequence From d95c1516a4326a0954f1965dd02d29f1ffaa0f63 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 19 Oct 2026 21:34:21 -0700 Subject: [PATCH 5/9] doc: define workload identity and execution isolation Make credentials, account mode, machine identity, namespaces, mounts, and device visibility adapter-selected within library rules. Explain why listener announcements remain authentic while publication is active and align the surrounding guest-runtime text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-system-components-and-responsibilities.md | 22 +++---- doc/design/3.5-network.md | 22 ++++--- doc/design/4-guest-runtime-architecture.md | 14 ++--- .../4.2-guest-agents-and-workload-model.md | 25 ++++---- .../4.3-isolation-and-resource-model.md | 63 ++++++++++--------- doc/design/5.5-restore-protocol.md | 10 +-- 6 files changed, 83 insertions(+), 73 deletions(-) diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index d3a12da4d..70bc68a99 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -129,17 +129,17 @@ flowchart TB cgroups, and process primitives used by the runtime. - **Guest agents** mount runtime filesystems, apply boot configuration, mount live host shares, and assemble the workload filesystem. They construct the - workload's root and credentials, and in the layered sandbox root its - namespaces and device view. They create workload accounts when requested, - supervise one-shot or managed workloads, prepare snapshots, and repair - processors, memory, entropy, and identity after restore. They also announce - the guest's listeners when the host publishes them, check time ABI - conformance, watch for clock violations, discipline the wall clock, and - perform the snapshot time steps. Their managed layered root comes from a - guest runtime library. An external guest agent may run as the guest's init - process instead, embedding the same library through its own adapter, so root - assembly, host mappings, and execution supervision behave the same for every - host. + workload's root, credentials, namespaces, and device view. In the blockless + root, they create workload accounts when requested; the layered root never + creates them. They supervise one-shot or managed workloads, prepare + snapshots, and repair processors, memory, entropy, and identity after + restore. They also announce the guest's listeners when the host publishes + them, check time ABI conformance, watch for clock violations, discipline the + wall clock, and perform the snapshot time steps. Their managed layered root + comes from a guest runtime library. An external guest agent may run as the + guest's init process instead, embedding the same library through its own + adapter, so root assembly, host mappings, and execution supervision behave + the same for every host. - **Workload** is the microVM's single tenant: it owns the microVM's lifetime, resource envelope, and network identity, and it may run any number of processes, all inside the isolation boundary that the guest agents diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index bbe6ea684..c86763822 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -174,15 +174,19 @@ cannot supply, and a platform clone that installs a new policy at restore receives it in its activation record instead. A guest agent then announces the guest's listeners. It sends each announcement as one UDP datagram to a fixed port of the IPv4 -gateway from a privileged source port, which the workload, holding no -capabilities, cannot bind, so the workload cannot forge an announcement. An -announcement carries a magic value, a version, a flag that marks omitted -listeners, and at most 64 entries in strictly increasing order of protocol and -port, each naming the protocol, TCP or UDP, the address families, and the -port. The endpoint consumes announcements before the egress policy, which -neither authorizes nor forwards them, and each announcement replaces the -published set. For each published port and address family, the endpoint keeps -one host listener at the guest's port number: +gateway from a privileged source port. While private-network ingress or +host-loopback publication is active, the workload is refused +`NET_BIND_SERVICE` and `NET_RAW`; `NET_ADMIN` is never granted. The runtime +owns `net.ipv4.ip_unprivileged_port_start` and keeps it at 1024 or above. A +UID 0 workload sees `/proc/sys` read-only, so it cannot lower that threshold. +The workload therefore cannot forge an announcement. An announcement carries +a magic value, a version, a flag that marks omitted listeners, and at most 64 +entries in strictly increasing order of protocol and port, each naming the +protocol, TCP or UDP, the address families, and the port. The endpoint +consumes announcements before the egress policy, which neither authorizes nor +forwards them, and each announcement replaces the published set. For each +published port and address family, the endpoint keeps one host listener at +the guest's port number: - **Private-network ingress** listens on every host address and relays only peers in ranges that are not globally reachable: the IPv4 private-use, diff --git a/doc/design/4-guest-runtime-architecture.md b/doc/design/4-guest-runtime-architecture.md index 2acbe7e88..e0149d565 100644 --- a/doc/design/4-guest-runtime-architecture.md +++ b/doc/design/4-guest-runtime-architecture.md @@ -3,13 +3,13 @@ [Architecture documentation](README.md) The NVX guest runtime turns one microVM into one workload sandbox. The guest -agents supervise the workload for the life of the VM. In the -[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), -they remain outside the workload's namespaces; the -[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) -creates none and contains the workload with its credentials and a cgroup. The -architecture does not introduce pod infrastructure, a pause container, or a -guest-side service for creating arbitrary additional containers. +agents supervise the workload for the life of the VM. They remain outside +each execution's private namespaces in both the +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) +and the +[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root). +The architecture does not introduce pod infrastructure, a pause container, +or a guest-side service for creating arbitrary additional containers. This chapter describes how the guest turns the virtual machine into an isolated workload sandbox. diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index e523c4555..bb113eef6 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -19,9 +19,9 @@ authority. Their responsibilities are: ```mermaid flowchart TB Kernel["Linux kernel"] - Agents["Guest agents
outside workload namespaces
in the layered root"] + Agents["Guest agents
outside workload namespaces"] Root["Assembled workload root"] - Isolation["Identity, capabilities,
and cgroups, plus namespaces
in the layered root"] + Isolation["Identity, capabilities,
cgroups, and namespaces"] Workload["Workload process tree"] Kernel --> Agents @@ -36,12 +36,12 @@ flowchart TB In a one-shot lifecycle, the guest agents start one workload, configured by non-secret kernel command-line tokens, and shut down the VM when it exits. In -a managed lifecycle, the guest agents stay resident and run a sequence of -workloads, one at a time, on request over the authenticated control channel. -Both modes preserve the same separation: the workload is always a child of a -guest agent. The guest agents never replace themselves with the workload, -because supervision, orphan reaping, and the control service all depend on -them remaining alive. +a managed lifecycle, the guest agents stay resident and run workloads on +request over the authenticated control channel, under the adapter's +concurrency policy. Both modes preserve the same separation: the workload is +always a child of a guest agent. The guest agents never replace themselves +with the workload, because supervision, orphan reaping, and the control +service all depend on them remaining alive. ## 4.2.1 Execution Launch @@ -72,9 +72,8 @@ failure, and every later execution first reaps what earlier workloads left and verifies that the cgroup is empty, so a session reset cannot bypass the check. -A cancellation that names the running execution kills the workload, and a +A cancellation that names an active execution kills its workload, and a cancellation for an execution that already finished is ignored. A session -reset means that the host session that could observe the workload is gone: -the guest agent kills the workload, sends nothing more for it, and -acknowledges the new epoch, so the next session never waits for an abandoned -workload. +reset means that the host session that could observe the workloads is gone: +the guest agent kills them, sends nothing more for them, and acknowledges the +new epoch, so the next session never waits for an abandoned workload. diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 8b707436e..2dcc303a9 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -5,45 +5,52 @@ ## 4.3.1 Identity and Credentials -The workload runs under a host-selected, non-root numeric UID and GID fixed at -initial boot. The guest agents require an exact user and primary-group match -in the workload root and reject an unavailable identity before starting the -workload. When the boot configuration or a platform clone's activation record -requests it, they instead create an account for a host-selected identity that -the image lacks, and replace an image account that uses the UID with another -primary group or without a usable home. This lets a host run workloads under -its own user's IDs. - -Before the workload starts, the guest agents clear supplementary groups and -every capability set and enable `no_new_privs`. With every capability cleared, -the workload cannot perform port I/O, so only the guest agents can request a -capture or acknowledge restore repair. +The adapter sets the workload's UID, GID, supplementary groups, and +capabilities within the guest runtime library's rules. Capabilities come only +from this allowlist: `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `FSETID`, `KILL`, +`SETGID`, `SETUID`, `SETPCAP`, `SETFCAP`, `NET_BIND_SERVICE`, `NET_RAW`, +`SYS_CHROOT`, and `AUDIT_WRITE`. The library refuses every other capability +and gives a non-root identity no capabilities. It always sets `no_new_privs`. + +The adapter also selects the account mode. Existing-account mode requires one +passwd entry for the UID, the requested primary GID, an existing group, and +an absolute home. NVX's adapter uses this mode by default. Numeric-ID mode +uses the requested IDs without an account check. The layered root never +creates accounts; the blockless root may create one when its adapter requests +it. NVX's adapter keeps a non-root identity with no capabilities. ## 4.3.2 Namespaces and Mounts -In the layered sandbox root, the workload receives private mount, PID, and UTS -namespaces. Its entry helper makes every mount private and provides: +Each execution receives private mount, PID, and UTS namespaces. The adapter +selects the hostname; NVX's default is `nvx-sandbox`. User namespaces remain +unavailable. The entry helper makes every mount private and provides: - A private `/proc` and a read-only `/sys`. -- A restricted `/dev`, devpts, and shared memory. -- The workload root, entered with `chroot`. +- A `/dev` that contains only allowlisted device nodes, devpts, and shared + memory. +- The workload root, moved over `/` in the execution's mount namespace, never + entered by `chroot` alone. - Only the live shares that the host declared. -- The workload's machine ID, bound read-only from a runtime file so that the - guest agents can refresh it while scratch is frozen. +- Either a generated per-boot machine ID bound read-only over + `/etc/machine-id`, which NVX selects by default, or the image's own file, as + the adapter chooses. - The guest time status, bound read-only. +Every other workload-visible mount is `nodev`. For UID 0, `/proc/sys`, +`/proc/sysrq-trigger`, `/proc/bus`, `/proc/fs`, and `/proc/irq` are read-only. +`/proc/kcore`, `/proc/keys`, `/proc/timer_list`, and similar sensitive files +are masked. + The guest agents never pivot away from the initramfs, so they keep their own root and stay outside the workload's namespaces, where they can supervise and tear down the workload. The control device is never exposed inside the -workload's namespaces. +workload's namespaces. Before entry, the helper closes old-root, control, +namespace, and process descriptors. The workload inherits no control +terminal. -The blockless managed root has no such isolation. Each execution runs in the -guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS -namespaces as the guest agents, so it sees the guest's own `/proc`, `/sys`, and -`/dev` and the host mappings bound into that root. Its -[credentials](#431-identity-and-credentials) and its -[execution cgroup](#433-cgroups-and-resources) contain it, but it has no -private mount, process, hostname, or device view. +The layered root moves its assembled root over `/`. The blockless root moves +its initramfs root over `/`, with the host mappings that the adapter declared, +and applies the same private namespace and device view. ## 4.3.3 Cgroups and Resources @@ -67,7 +74,7 @@ the layered root's optional memory and process-count limits. Isolation has two enforcement layers: - Guest policy constructs the workload's credentials, mounts, capabilities, - and cgroups and, in the layered sandbox root, its namespaces. + cgroups, and namespaces. - Host devices independently enforce block mutability, network policy, and host-filesystem access. diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index c19cbc4f3..b1131f6bb 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -319,8 +319,8 @@ To prevent this, every restore delivers fresh entropy, which a clone of a 3. For a clone, the guest agents credit all 64 bytes to the kernel's entropy pool and force the kernel's random-number generator to reseed, so all later kernel randomness depends on the fresh bytes. -4. They refresh the workload's machine ID, which they derive from the fresh - bytes, and its UTS namespace. +4. When the adapter selects a generated machine ID, they refresh it from the + fresh bytes. They also refresh the workload's UTS namespace. 5. The workload runtime's post-restore hook resets the runtime's own random-number state, which a kernel reseed cannot reach. 6. The guest agents acknowledge the restore and only then thaw scratch and @@ -334,8 +334,8 @@ As Table 5.7 shows, the refresh depends on the restore path. | Restore path | Guest random-number generator | Machine identity | | --- | --- | --- | -| `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it | -| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Refreshed, as for a clone | +| `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Generated machine ID and hostname refreshed; an image machine ID remains, and the runtime hook runs if present; `workload-start` requires the hook | +| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Generated machine ID and hostname refreshed, as for a clone | | `instance-checkpoint` tier (resume) | Not reseeded, by design; only the generation ID changes | Preserved | | Untiered, with nothing to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved | @@ -374,7 +374,7 @@ port exit. It carries: - The upper-layer settings that the template did not capture. - The form of any export that the restore bound to the dormant filesystem slot, and the number of host mappings. -- Whether to create the workload account. +- Whether to require an existing account or use numeric IDs. - For a template captured with a deny-all network policy, the DNS server, the ingress mode, and the guest's settings for any gateway proxy endpoint that the new policy permits. From 478aabbf873748e63635384938d9f6508d730a8d Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 19 Oct 2026 21:34:31 -0700 Subject: [PATCH 6/9] doc: define control and lifecycle compatibility Specify the nvx-rs adapter contract, runtime-store and runtime-identity rules, and NVXC v2 readiness validation. Define execution streaming, NVX queueing and Stop policy, and same-boot reattach behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../1.5-compatibility-and-versioning-model.md | 22 ++- .../4.4-host-guest-control-interface.md | 130 +++++++----------- .../5.2-readiness-and-runtime-control.md | 25 +++- 3 files changed, 85 insertions(+), 92 deletions(-) diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index 7842257e5..34fcd4f28 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -18,9 +18,12 @@ contract carries its own version or identity, independent of the others. | Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | | RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | | Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, fresh entropy, and, in a platform clone, an optional activation record. | -| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, execution streams, and lifecycle epochs. | +| Control protocol | NVXC v2 authenticated sessions between the host and the guest agents, request framing, operation semantics, execution streams, and lifecycle epochs. | | Activation configuration | Schema and identity of the non-secret configuration that a managed layered root reads from boot tokens on a cold boot or from the restore packet's activation record in a platform clone. | | Management RPC machine profile | Identity of the microVM machine profile in the management RPC. | +| Adapter API | The nvx-rs Rust API for adapter-facing `host`, `guest`, and protocol-only features. | +| Runtime store | The single versioned format for durable sandbox lifecycle state. | +| Runtime identity | The OpenVMM, kernel, and initramfs identities together with the boot and control contract versions. | A change to a guest-visible address, interrupt, command-line token, feature mask, queue shape, time rule, or device behavior requires a new contract @@ -38,6 +41,12 @@ extra devices, changed queue shapes, incompatible CPU surfaces, malformed state, or mismatched attachments are rejected before guest execution. A snapshot with an unsupported manifest version must be recaptured. +CI guards the nvx-rs adapter API with a public-API check. A change to the +adapter contract includes a migration note and conformance tests. NVXC has +only application version 2 and has no feature-probing request. The guest build +ID is derived at build time from its build inputs. The host checks that ID in +`READY` against the selected bundle. + ## 1.5.1 Restore Compatibility A snapshot restores only: @@ -83,3 +92,14 @@ The snapshot manifest is authoritative for a restored process. Native file descriptors, sockets, named-pipe handles, and hypervisor objects are never part of the portable state. They are recreated from stable identities and declared reconstruction policies. + +The runtime store has one versioned format and no readers for older formats. +Leaving an older format requires one stop of its sandboxes. Later host-side +upgrades are live and keep running sandboxes. + +Reattach is available only after an orchestrator restart within the same host +boot and only when the recorded runtime identity matches. A different runtime +identity stops the sandbox by identity-verified forced termination without +speaking a possibly different guest protocol. An upgrade of host-side code +only keeps existing sandboxes running; it does not reattach them across a +runtime-identity change. diff --git a/doc/design/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md index 5182002d1..acdcd6443 100644 --- a/doc/design/4.4-host-guest-control-interface.md +++ b/doc/design/4.4-host-guest-control-interface.md @@ -4,13 +4,13 @@ [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) Managed operation uses the dedicated control console rather than the boot -console. The VMM owns the outer transport: bounded framing, same-user local -endpoint authorization, capability authentication, reconnect epochs, and -receive-credit backpressure; see +console. The VMM owns the outer NVXS v1 transport: bounded framing, same-user +local endpoint authorization, capability authentication, reconnect epochs, +and receive-credit backpressure; see [Control Console](3.3-consoles.md#332-control-console). The guest agents own -the workload protocol carried inside that stream. Boot diagnostics and kernel -messages stay on the consoles, so framed control traffic never shares an -unstructured byte stream with them. +the NVXC v2 application protocol carried inside that stream. Boot diagnostics +and kernel messages stay on the consoles, so framed control traffic never +shares an unstructured byte stream with them. ## 4.4.1 Operations @@ -20,24 +20,22 @@ Table 4.2 lists the operations of the control protocol. | Operation | Purpose | | --- | --- | -| `READY` | Reports readiness, and is the success reply to requests that return no other data | -| `FEATURES` | Reports which control behaviors the guest image provides | +| `READY` | Reports the application version, guest build ID, and root profile, and is the success reply to requests that return no other data | | `MAPS` | Delivers a contiguous range of host mapping-table entries | -| `EXEC` | Runs one workload with a timeout, arguments, and an optional working directory and environment, and, in protocol version 2, standard input | +| `EXEC` | Runs one workload with a timeout, arguments, optional working directory and environment, and standard input | | `CANCEL` | Kills the running execution that the request names | -| Shutdown | Shuts the VM down gracefully between executions | +| Shutdown | Shuts the VM down gracefully | An execution request carries a 32-bit timeout and length-prefixed arguments, and optionally an absolute working directory of at most 4,096 bytes and up to 256 `KEY=VALUE` environment entries of at most 4,096 bytes each, with distinct, non-empty keys. The environment either replaces the default environment or is layered over it, each entry replacing the default variable -of the same name. Output returns as separate standard output and standard -error streams within the credit window. Protocol version 1 allows at most -1 MiB of them combined for each execution; version 2 streams them without a -total limit, as [Protocol Versions](#444-protocol-versions) describes. As -Table 4.3 shows, an execution that runs ends with an exit report, which -carries an outcome category and a status. +of the same name. Standard input, standard output, and standard error stream +within per-execution credit windows. Only data in flight is bounded; there is +no total output limit. As Table 4.3 shows, an execution that runs ends with an +exit report, which carries an outcome category, status, and output-truncation +flag. **Table 4.3.** Exit outcomes. @@ -47,7 +45,6 @@ carries an outcome category and a status. | `signal` | A signal ended the workload; the status is 128 plus the signal number | | `timeout` | The workload exceeded its timeout and was killed; the status is 124 | | `cancelled` | A cancellation killed the workload; the status is 137 | -| `output-limit` | In protocol version 1 only, the workload's combined output exceeded 1 MiB, and the guest agents killed it; the status is 125 | | `failed` | The guest agents could not supervise the workload; the status is 125 | Otherwise, the guest agents reply with an error, which carries one of the @@ -65,55 +62,36 @@ account for. | `containment-failed` | The guest agents could not prove that every workload process exited | | `mapping-failed` | A host mapping could not be mounted | | `mappings-incomplete` | Not every announced host mapping is mounted yet | -| `busy` | Another execution is running | | `invalid-request` | The request is malformed or invalid in the current configuration | | `unsupported-operation` | The guest does not implement the operation | -## 4.4.2 Feature Negotiation - -A successful readiness probe does not prove that an image enforces the -behaviors that a host depends on. As Table 4.5 shows, a `FEATURES` request -therefore returns a four-byte little-endian bit mask of control features. - -**Table 4.5.** Control feature bits. - -| Bit | Feature | -| ---: | --- | -| 0 | Cancellation | -| 1 | Reserved; never set | -| 2 | Workload account creation | -| 3 | Per-execution cgroup containment | -| 4 | Per-execution environment | -| 5 | Working-directory refusal | -| 6 | Host mapping table | - -The blockless managed root and the -[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) -provide all six features. A feature bit is added together with the behavior -it names. Hosts ignore unknown bits and trailing bytes, read an -`unsupported-operation` refusal as no features, and terminate a guest that -lacks a feature they need instead of running workloads without the policy -they requested. - -A host and guest built together as a matched bundle need no `FEATURES` -request when they speak protocol version 2, because the bundle fixes the -features and the guest's readiness report carries its protocol version and -build identity. The host terminates a guest whose identity differs from the -bundle's instead of probing it per operation. A restored guest reports the -build captured in its snapshot, so the host compares it with the bundle that -built the template. +## 4.4.2 Readiness and Bundle Conformance + +NVXC has one application version, v2. `READY` carries that version, the guest +build ID, and the root profile. A managed layered root also carries its +activation-configuration identity. The build derives its ID at build time +from its build inputs. + +On every fresh session, the host validates `READY` against the selected +bundle before it sends `MAPS` or `EXEC`. It terminates a guest whose +application version, build ID, root profile, or required activation identity +does not match. A restored guest reports the build captured in its snapshot, +so the host compares it with the bundle that built the template. ## 4.4.3 Session Semantics Requests are typed, length-bounded, ordered, and associated with one VM instance and epoch. Credit limits bound queued data in both directions. Malformed, unauthenticated, stale, duplicated, or out-of-order records fail -closed. Only one execution runs at a time; any request other than a -cancellation that arrives during an execution is refused as `busy`. The -standard input and credit records of a version-2 execution belong to the -running execution and are not requests. Shutdown is therefore graceful only -between executions: a host cancels a running execution before it requests -shutdown, or else terminates the VM, which fails the execution. +closed. Standard input and credit records name their execution and are not +requests. The protocol does not set an execution-concurrency policy; each +adapter does. + +Normal completion is lossless: the exit report follows the last output of +both streams. On cancellation, timeout, session drop, or Stop, pending output +drains for a bounded grace period. The guest then discards any remainder and +sets the output-truncation flag, so teardown cannot wait indefinitely for a +reader. The control session is process-local even when guest parser state is snapshotted. Restore establishes a fresh host capability, peer identity, @@ -129,30 +107,14 @@ the control channel exists. ## 4.4.4 Protocol Versions -The control protocol has two versions, which differ in their readiness report -and their execution streams. A session uses one version throughout. A -version-2 guest states its protocol version and build identity in its -readiness report and, in a managed layered root, the identity of its -activation configuration; a readiness report without them comes from a -version-1 guest. A host that requires another version terminates the guest -instead of downgrading a requested stream. - -- **Version 1** is the protocol that the preceding sections describe. An - execution has no standard input, its output flows within the session's - credit window, and its combined output is limited to 1 MiB. -- **Version 2** adds piped standard input with an explicit end of input, and - gives each execution its own credit windows in both directions. The host - grants credit for standard output and standard error as it consumes them, - and the guest grants credit for standard input as the workload drains it. - Output therefore has no total limit; only data in flight is bounded, and a - guest without credit stops reading the workload's output, which blocks the - workload instead of killing it. Execution credits are separate from the - control console's receive credit. The exit report follows the last output - of both streams, so a host that withholds credit delays it, but a - cancellation, credit grants, and session control still progress while an - execution's window is exhausted. - -Both versions run one execution at a time and keep every other operation, -outcome, and error. Neither provides a terminal, window resizing, signals -other than cancellation, or concurrent executions; these remain outside the -control protocol. +NVXC v2 uses piped standard input with an explicit end of input and gives each +execution its own credit windows in both directions. The host grants credit +for standard output and standard error as it consumes them, and the guest +grants credit for standard input as the workload drains it. A guest without +credit stops reading the workload's output, which blocks the workload instead +of killing it. Cancellation, credit grants, and session control still +progress while an execution's window is exhausted. + +NVXC execution credits are separate from the outer NVXS v1 transport's byte +credits. NVXC provides no terminal, window resizing, or signals other than +cancellation. diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index 845767ba9..4e2565da2 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -69,13 +69,24 @@ of each sandbox, reconciles VMM processes with recorded state, and recovers interrupted launches. Teardown leaves no VMM process, control endpoint, capability, or runtime record, while supplied layers and scratch survive. -The host runs one execution at a time per sandbox, as the guest does, and -holds requests that arrive meanwhile in a bounded queue rather than forwarding -them for the guest to refuse as `busy`. A full queue refuses a request at -once, and a waiter whose deadline passes or whose caller leaves never starts. -Stop takes priority over the queue: it fails the waiters, cancels the running -execution, and shuts the VM down, or terminates the VM when the execution -cannot be cancelled within a bound. +NVX's adapter admits one active execution per sandbox and keeps a FIFO of at +most eight waiters across CLI processes. The default queue wait is 60 seconds +and each invocation may override it. A full queue fails at once with +`queue-full`; an expired wait fails with `queue-timeout`. Other adapters choose +their own execution-concurrency policy. + +Stop bypasses the queue. It fences new admission, cancels the waiters, asks +the active execution's owner to cancel and release the slot, and then requests +graceful shutdown. If that sequence does not complete in time, NVX's CLI +reports the timeout and leaves the VM running; it never escalates. The +library's identity-verified forced termination remains available to other +callers. + +After an orchestrator restart, reattach is available only within the same host +boot and only for the same +[runtime identity](1.5-compatibility-and-versioning-model.md). A mismatch uses +the library's identity-verified forced termination without speaking the guest +protocol. The sandbox lifecycle API exposes the same operations with request validation, backend capabilities, durable records, and stable error codes. Its From 078ae660e2ccd776f0f24bead07f9aff4dca0444 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 19 Oct 2026 21:34:36 -0700 Subject: [PATCH 7/9] doc: scope mapping receipts to restore Keep cold mapping identity checks at provision and start. Reserve opened-object receipts for platform-clone restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/3.6-host-filesystem.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/design/3.6-host-filesystem.md b/doc/design/3.6-host-filesystem.md index 0c37a93ea..943ec642a 100644 --- a/doc/design/3.6-host-filesystem.md +++ b/doc/design/3.6-host-filesystem.md @@ -260,10 +260,11 @@ could not track or save. The host root is pinned by its platform object identity, device and inode number on Linux or volume and file ID on Windows, and revalidated when the export opens; each aggregate child's root is pinned the same way. A host may -also supply the identity that it expects for a root or a pinned file child; -the VMM compares it with the object that it actually opened, refuses the -export on a mismatch, and reports the opened identities in its -[attachment receipts](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments). +also supply the identity that it expects for a root or a pinned file child. +Cold mappings record these identities at provision and recheck them at start; +a mismatch refuses the export. Opened-object +[attachment receipts](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments) +belong to platform-clone restore and are not part of the cold mapping path. Every operation stays confined to the export: host path resolution follows no symbolic link or reparse point, never leaves the root, and never follows the final component. On Windows hosts, guest-created links are reparse points that From 31a834476fdd7ef5a6645ce0554912cc286572b8 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 19 Oct 2026 21:41:26 -0700 Subject: [PATCH 8/9] doc: keep namespaces a layered-root property and scope NVXC policy Keep private namespaces and the restricted device view specific to the layered root. Scope single-execution framing to NVXC, clarify adapter framing over NVXS, and tighten runtime identity, GPT, machine-ID, and execution-settlement details. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-system-components-and-responsibilities.md | 23 +++++------ .../1.5-compatibility-and-versioning-model.md | 18 ++++----- doc/design/4-guest-runtime-architecture.md | 14 +++---- .../4.1-guest-filesystem-and-image-model.md | 28 ++++++------- .../4.2-guest-agents-and-workload-model.md | 4 +- .../4.3-isolation-and-resource-model.md | 39 ++++++++++++------- .../4.4-host-guest-control-interface.md | 22 +++++++---- doc/design/5.5-restore-protocol.md | 5 ++- 8 files changed, 84 insertions(+), 69 deletions(-) diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index 70bc68a99..96cc8be67 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -129,17 +129,18 @@ flowchart TB cgroups, and process primitives used by the runtime. - **Guest agents** mount runtime filesystems, apply boot configuration, mount live host shares, and assemble the workload filesystem. They construct the - workload's root, credentials, namespaces, and device view. In the blockless - root, they create workload accounts when requested; the layered root never - creates them. They supervise one-shot or managed workloads, prepare - snapshots, and repair processors, memory, entropy, and identity after - restore. They also announce the guest's listeners when the host publishes - them, check time ABI conformance, watch for clock violations, discipline the - wall clock, and perform the snapshot time steps. Their managed layered root - comes from a guest runtime library. An external guest agent may run as the - guest's init process instead, embedding the same library through its own - adapter, so root assembly, host mappings, and execution supervision behave - the same for every host. + workload's root and credentials, and in the layered sandbox root its + namespaces and device view. In the blockless root, they create workload + accounts when requested; the layered root never creates them. They + supervise one-shot or managed workloads, prepare snapshots, and repair + processors, memory, entropy, and identity after restore. They also announce + the guest's listeners when the host publishes them, check time ABI + conformance, watch for clock violations, discipline the wall clock, and + perform the snapshot time steps. Their managed layered root comes from a + guest runtime library. An external guest agent may run as the guest's init + process instead, embedding the same library through its own adapter, so root + assembly, host mappings, and execution supervision behave the same for every + host. - **Workload** is the microVM's single tenant: it owns the microVM's lifetime, resource envelope, and network identity, and it may run any number of processes, all inside the isolation boundary that the guest agents diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index 34fcd4f28..03979155d 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -18,7 +18,7 @@ contract carries its own version or identity, independent of the others. | Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | | RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | | Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, fresh entropy, and, in a platform clone, an optional activation record. | -| Control protocol | NVXC v2 authenticated sessions between the host and the guest agents, request framing, operation semantics, execution streams, and lifecycle epochs. | +| Control protocol | NVXC v2, carried over the NVXS v1 outer session: authenticated sessions between the host and the guest agents, request framing, operation semantics, execution streams, and lifecycle epochs. | | Activation configuration | Schema and identity of the non-secret configuration that a managed layered root reads from boot tokens on a cold boot or from the restore packet's activation record in a platform clone. | | Management RPC machine profile | Identity of the microVM machine profile in the management RPC. | | Adapter API | The nvx-rs Rust API for adapter-facing `host`, `guest`, and protocol-only features. | @@ -93,13 +93,9 @@ descriptors, sockets, named-pipe handles, and hypervisor objects are never part of the portable state. They are recreated from stable identities and declared reconstruction policies. -The runtime store has one versioned format and no readers for older formats. -Leaving an older format requires one stop of its sandboxes. Later host-side -upgrades are live and keep running sandboxes. - -Reattach is available only after an orchestrator restart within the same host -boot and only when the recorded runtime identity matches. A different runtime -identity stops the sandbox by identity-verified forced termination without -speaking a possibly different guest protocol. An upgrade of host-side code -only keeps existing sandboxes running; it does not reattach them across a -runtime-identity change. +The runtime store has one versioned format and no readers for older formats; +leaving an older format requires one stop of its sandboxes. After that, +upgrades are live: after an orchestrator restart within the same host boot, a +host reattaches a running sandbox only when its recorded runtime identity +matches. A different runtime identity stops the sandbox by identity-verified +forced termination, without speaking a possibly different guest protocol. diff --git a/doc/design/4-guest-runtime-architecture.md b/doc/design/4-guest-runtime-architecture.md index e0149d565..2acbe7e88 100644 --- a/doc/design/4-guest-runtime-architecture.md +++ b/doc/design/4-guest-runtime-architecture.md @@ -3,13 +3,13 @@ [Architecture documentation](README.md) The NVX guest runtime turns one microVM into one workload sandbox. The guest -agents supervise the workload for the life of the VM. They remain outside -each execution's private namespaces in both the -[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) -and the -[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root). -The architecture does not introduce pod infrastructure, a pause container, -or a guest-side service for creating arbitrary additional containers. +agents supervise the workload for the life of the VM. In the +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), +they remain outside the workload's namespaces; the +[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) +creates none and contains the workload with its credentials and a cgroup. The +architecture does not introduce pod infrastructure, a pause container, or a +guest-side service for creating arbitrary additional containers. This chapter describes how the guest turns the virtual machine into an isolated workload sandbox. diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index ed99b13b5..304c7007f 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -77,11 +77,12 @@ The guest agents assemble the root in this order: GPT validation bounds the header and entry array, verifies their CRCs, extents, indices, and GUIDs, and permits at most 255 used partitions, the -kernel's limit. Partition 1 holds opaque metadata bytes. The guest reads them -up to a caller-selected bound, trims trailing NUL and whitespace, and never -probes the partition as a filesystem; the adapter parses the result. After -the partition-table reread, every kernel partition must match its validated -entry. +kernel's limit. The primary header's alternate LBA must lie inside the device, +which may present zeros past a shorter image. Partition 1 holds opaque +metadata bytes. The guest reads them up to a caller-selected bound, trims +trailing NUL and whitespace, and never probes the partition as a filesystem; +the adapter parses the result. After the partition-table reread, every kernel +partition must match its validated entry. Expected identities are optional inputs: the EROFS filesystem UUID, or the GPT disk GUID, partition GUIDs, and partition sizes. NVX's adapter passes the @@ -173,14 +174,15 @@ The blockless root may create the workload account that its adapter requests. ## 4.1.4 Managed Layered Root -In a managed lifecycle, the layered sandbox root uses -[control protocol version 2](4.4-host-guest-control-interface.md#444-protocol-versions). -A guest runtime library implements its guest half and the host control -library its host half, so every adapter that adopts this root behaves the -same. Unlike the blockless root, it never creates workload accounts. The -identity must already exist in the image, or the adapter must select numeric -IDs without an account check. It differs from a one-shot layered root in four -ways: +In a managed lifecycle, the layered sandbox root uses the shared guest +runtime and host libraries for root assembly, host mappings and execution. +NVX's adapter speaks +[NVXC v2](4.4-host-guest-control-interface.md#444-protocol-versions) over +NVXS; other adapters may carry their own application framing over NVXS and +keep their own admission policy. Unlike the blockless root, it never creates +workload accounts. The identity must already exist in the image, or the +adapter must select numeric IDs without an account check. It differs from a +one-shot layered root in four ways: - **Readiness.** The guest agents assemble the root from boot tokens without waiting for the host, then start the control service. Readiness therefore diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index bb113eef6..29ff51107 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -19,9 +19,9 @@ authority. Their responsibilities are: ```mermaid flowchart TB Kernel["Linux kernel"] - Agents["Guest agents
outside workload namespaces"] + Agents["Guest agents
outside workload namespaces
in the layered root"] Root["Assembled workload root"] - Isolation["Identity, capabilities,
cgroups, and namespaces"] + Isolation["Identity, capabilities,
and cgroups, plus namespaces
in the layered root"] Workload["Workload process tree"] Kernel --> Agents diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 2dcc303a9..1d3c8c7ad 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -6,11 +6,15 @@ ## 4.3.1 Identity and Credentials The adapter sets the workload's UID, GID, supplementary groups, and -capabilities within the guest runtime library's rules. Capabilities come only -from this allowlist: `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `FSETID`, `KILL`, -`SETGID`, `SETUID`, `SETPCAP`, `SETFCAP`, `NET_BIND_SERVICE`, `NET_RAW`, -`SYS_CHROOT`, and `AUDIT_WRITE`. The library refuses every other capability -and gives a non-root identity no capabilities. It always sets `no_new_privs`. +capabilities within the guest runtime library's rules. The identity is fixed +at initial boot. Capabilities come only from this allowlist: `CHOWN`, +`DAC_OVERRIDE`, `FOWNER`, `FSETID`, `KILL`, `SETGID`, `SETUID`, `SETPCAP`, +`SETFCAP`, `NET_BIND_SERVICE`, `NET_RAW`, `SYS_CHROOT`, and `AUDIT_WRITE`. +The library refuses every other capability and gives a non-root identity no +capabilities. It always sets `no_new_privs`. The allowlist excludes +`SYS_RAWIO`, and the guest agents revoke their port-I/O permission before +starting any workload stage, so only the guest agents can request a capture +or acknowledge restore repair. The adapter also selects the account mode. Existing-account mode requires one passwd entry for the UID, the requested primary GID, an existing group, and @@ -21,9 +25,10 @@ it. NVX's adapter keeps a non-root identity with no capabilities. ## 4.3.2 Namespaces and Mounts -Each execution receives private mount, PID, and UTS namespaces. The adapter -selects the hostname; NVX's default is `nvx-sandbox`. User namespaces remain -unavailable. The entry helper makes every mount private and provides: +In the layered sandbox root, each execution receives private mount, PID, and +UTS namespaces. The adapter selects the hostname; NVX's default is +`nvx-sandbox`. User namespaces remain unavailable. The entry helper makes +every mount private and provides: - A private `/proc` and a read-only `/sys`. - A `/dev` that contains only allowlisted device nodes, devpts, and shared @@ -31,9 +36,9 @@ unavailable. The entry helper makes every mount private and provides: - The workload root, moved over `/` in the execution's mount namespace, never entered by `chroot` alone. - Only the live shares that the host declared. -- Either a generated per-boot machine ID bound read-only over - `/etc/machine-id`, which NVX selects by default, or the image's own file, as - the adapter chooses. +- Either a generated per-boot machine ID, bound read-only from a runtime file + so that the guest agents can refresh it while scratch is frozen (NVX's + default), or the image's own `/etc/machine-id`, as the adapter chooses. - The guest time status, bound read-only. Every other workload-visible mount is `nodev`. For UID 0, `/proc/sys`, @@ -48,9 +53,13 @@ workload's namespaces. Before entry, the helper closes old-root, control, namespace, and process descriptors. The workload inherits no control terminal. -The layered root moves its assembled root over `/`. The blockless root moves -its initramfs root over `/`, with the host mappings that the adapter declared, -and applies the same private namespace and device view. +The blockless managed root has no such isolation. Each execution runs in the +guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS +namespaces as the guest agents, so it sees the guest's own `/proc`, `/sys`, and +`/dev` and the host mappings bound into that root. Its +[credentials](#431-identity-and-credentials) and its +[execution cgroup](#433-cgroups-and-resources) contain it, but it has no +private mount, process, hostname, or device view. ## 4.3.3 Cgroups and Resources @@ -74,7 +83,7 @@ the layered root's optional memory and process-count limits. Isolation has two enforcement layers: - Guest policy constructs the workload's credentials, mounts, capabilities, - cgroups, and namespaces. + and cgroups and, in the layered sandbox root, its namespaces. - Host devices independently enforce block mutability, network policy, and host-filesystem access. diff --git a/doc/design/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md index acdcd6443..d9e8a033b 100644 --- a/doc/design/4.4-host-guest-control-interface.md +++ b/doc/design/4.4-host-guest-control-interface.md @@ -24,7 +24,7 @@ Table 4.2 lists the operations of the control protocol. | `MAPS` | Delivers a contiguous range of host mapping-table entries | | `EXEC` | Runs one workload with a timeout, arguments, optional working directory and environment, and standard input | | `CANCEL` | Kills the running execution that the request names | -| Shutdown | Shuts the VM down gracefully | +| Shutdown | Shuts the VM down gracefully between executions | An execution request carries a 32-bit timeout and length-prefixed arguments, and optionally an absolute working directory of at most 4,096 bytes and up to @@ -62,6 +62,7 @@ account for. | `containment-failed` | The guest agents could not prove that every workload process exited | | `mapping-failed` | A host mapping could not be mounted | | `mappings-incomplete` | Not every announced host mapping is mounted yet | +| `busy` | Another execution is running; NVXC runs one execution at a time | | `invalid-request` | The request is malformed or invalid in the current configuration | | `unsupported-operation` | The guest does not implement the operation | @@ -83,15 +84,20 @@ so the host compares it with the bundle that built the template. Requests are typed, length-bounded, ordered, and associated with one VM instance and epoch. Credit limits bound queued data in both directions. Malformed, unauthenticated, stale, duplicated, or out-of-order records fail -closed. Standard input and credit records name their execution and are not -requests. The protocol does not set an execution-concurrency policy; each -adapter does. +closed. NVXC runs one execution at a time: any request other than a +cancellation that arrives during an execution is refused as `busy`; standard +input and credit records belong to the running execution and are not +requests. Shutdown is therefore graceful only between executions: a host +cancels a running execution before it requests shutdown. Other adapters carry +their own framing over NVXS and set their own concurrency. Normal completion is lossless: the exit report follows the last output of -both streams. On cancellation, timeout, session drop, or Stop, pending output -drains for a bounded grace period. The guest then discards any remainder and -sets the output-truncation flag, so teardown cannot wait indefinitely for a -reader. +both streams. On cancellation, timeout, handle drop, disconnect, reset, or +Stop, pending output drains for a bounded grace period. The guest then +discards any remainder and sets the output-truncation flag, so teardown cannot +wait indefinitely for a reader. Control and timers stay live throughout. The +logical outcome, delivery completion and process settlement are separate: an +execution completes only after its descendants settle. The control session is process-local even when guest parser state is snapshotted. Restore establishes a fresh host capability, peer identity, diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index b1131f6bb..5c898ad2b 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -319,8 +319,9 @@ To prevent this, every restore delivers fresh entropy, which a clone of a 3. For a clone, the guest agents credit all 64 bytes to the kernel's entropy pool and force the kernel's random-number generator to reseed, so all later kernel randomness depends on the fresh bytes. -4. When the adapter selects a generated machine ID, they refresh it from the - fresh bytes. They also refresh the workload's UTS namespace. +4. When the adapter selects a generated machine ID, they generate a new one + with `getrandom` after the reseed; an image's own machine ID is left + unchanged. They also refresh the workload's UTS namespace. 5. The workload runtime's post-restore hook resets the runtime's own random-number state, which a kernel reseed cannot reach. 6. The guest agents acknowledge the restore and only then thaw scratch and From 950c1fe07602ca88ff6d4f1129a9026d2e671dbb Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 19 Oct 2026 21:59:39 -0700 Subject: [PATCH 9/9] doc: state the tmpfs upper's default size The RAM-backed overlay layout mounts its tmpfs upper at the kernel's default size, half of guest memory, and adds no size token. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/4.1-guest-filesystem-and-image-model.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index 304c7007f..4e341879c 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -59,7 +59,8 @@ The guest agents assemble the root in this order: `noload`. 4. Resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and create its upper and work directories, or, in the RAM-backed overlay - layout, create them in tmpfs mounted with `nosuid,nodev`. + layout, create them in tmpfs mounted with `nosuid,nodev` at the kernel's + default size, half of guest memory. 5. Mount the overlay with the EROFS image, or the GPT image's layers from top to base, as its lower directories. Every image uses `metacopy=on,xino=on` by default. An orchestrator whose lower layers may