diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index bff0d4332..96cc8be67 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -17,6 +17,7 @@ flowchart TB direction LR Launcher["NVX launcher"] LifecycleApi["Sandbox lifecycle API"] + External["External orchestrators"] ControlLibrary["Host control library"] Storage["Snapshot storage"] Attachments["External attachments"] @@ -42,6 +43,7 @@ flowchart TB Launcher --> Frontends LifecycleApi --> ControlLibrary + External --> ControlLibrary ControlLibrary --> Frontends Launcher -. control session .-> Virtio ControlLibrary -. control session .-> Virtio @@ -73,7 +75,16 @@ flowchart TB - **Host control library** implements the host side of the control protocol, artifact discovery, blockless managed launch, VMM process supervision, host-path export with its access policy, delivery of the host mapping table, - and egress-rule translation. + and egress-rule translation. It also implements the host half of the + [managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root): + its root plan, activation configuration, execution admission, and + execution streams. Its adapters, in the sandbox lifecycle API and in + external orchestrators, supply policy, resources, and transport, and reuse + the library's validation and execution semantics instead of reimplementing + them. +- **External orchestrators** are orchestrators outside NVX, with their own + storage and admission policy, that drive the host control library through + their own adapters. - **Snapshot storage** retains immutable machine state, memory, and optional paired scratch together with the identities needed to reconstruct external resources. @@ -119,12 +130,17 @@ flowchart TB - **Guest agents** mount runtime filesystems, apply boot configuration, mount live host shares, and assemble the workload filesystem. They construct the workload's root and credentials, and in the layered sandbox root its - namespaces and device view. They create workload accounts when requested, + namespaces and device view. In the blockless root, they create workload + accounts when requested; the layered root never creates them. They supervise one-shot or managed workloads, prepare snapshots, and repair processors, memory, entropy, and identity after restore. They also announce the guest's listeners when the host publishes them, check time ABI conformance, watch for clock violations, discipline the wall clock, and - perform the snapshot time steps. + perform the snapshot time steps. Their managed layered root comes from a + guest runtime library. An external guest agent may run as the guest's init + process instead, embedding the same library through its own adapter, so root + assembly, host mappings, and execution supervision behave the same for every + host. - **Workload** is the microVM's single tenant: it owns the microVM's lifetime, resource envelope, and network identity, and it may run any number of processes, all inside the isolation boundary that the guest agents diff --git a/doc/design/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md index 69afe4af6..5e51def76 100644 --- a/doc/design/1.4-trust-boundaries.md +++ b/doc/design/1.4-trust-boundaries.md @@ -88,7 +88,12 @@ Image layers are trusted inputs: the overlay filesystem interprets metadata stored in lower layers, and bytes read from any layer can appear in captured memory. A snapshot identifies its blocks by a storage generation, which transfers trust for layer and scratch contents to the storage owner; the VMM -trusts the control plane in the host to authenticate that generation. +trusts the control plane in the host to authenticate that generation. For the +attachments that a restore supplies, the VMM checks any identities that the +host expects before any processor runs and reports the opened identities in +receipts; see +[Host Attachments](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments). +A receipt identifies an object, not its contents. The VMM trusts the host control plane not to give the same writable file, such as a caller-supplied RAM backing or scratch file, to more than one VMM. It diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index 860545f14..03979155d 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -17,9 +17,13 @@ contract carries its own version or identity, independent of the others. | Device-private schemas | Saved queue and device state that must be understood before a device can resume. | | Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | | RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | -| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, and fresh entropy. | -| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, and lifecycle epochs. | +| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, fresh entropy, and, in a platform clone, an optional activation record. | +| Control protocol | NVXC v2, carried over the NVXS v1 outer session: authenticated sessions between the host and the guest agents, request framing, operation semantics, execution streams, and lifecycle epochs. | +| Activation configuration | Schema and identity of the non-secret configuration that a managed layered root reads from boot tokens on a cold boot or from the restore packet's activation record in a platform clone. | | Management RPC machine profile | Identity of the microVM machine profile in the management RPC. | +| Adapter API | The nvx-rs Rust API for adapter-facing `host`, `guest`, and protocol-only features. | +| Runtime store | The single versioned format for durable sandbox lifecycle state. | +| Runtime identity | The OpenVMM, kernel, and initramfs identities together with the boot and control contract versions. | A change to a guest-visible address, interrupt, command-line token, feature mask, queue shape, time rule, or device behavior requires a new contract @@ -37,6 +41,12 @@ extra devices, changed queue shapes, incompatible CPU surfaces, malformed state, or mismatched attachments are rejected before guest execution. A snapshot with an unsupported manifest version must be recaptured. +CI guards the nvx-rs adapter API with a public-API check. A change to the +adapter contract includes a migration note and conformance tests. NVXC has +only application version 2 and has no feature-probing request. The guest build +ID is derived at build time from its build inputs. The host checks that ID in +`READY` against the selected bundle. + ## 1.5.1 Restore Compatibility A snapshot restores only: @@ -61,17 +71,20 @@ snapshots stay restorable; a normative change creates a new revision. A reusable template represents a compatibility class, not an arbitrary fleet image. The class includes the backend, the CPU profile and generation, the effective CPUID, the guest kernel and guest agent build, the effective command -line, device roles and geometry, network identity and policy, and consumed -layer identities. Restoring guest memory restores the captured guest agent -build, not a newly deployed initramfs, so deployments rebuild or recertify -templates on relevant rollouts. +line, device roles and geometry, network identity, network policy unless a +`platform` template of a managed layered root was captured with a deny-all +policy, and consumed layer identities. Restoring guest memory restores the +captured guest agent build, not a newly deployed initramfs, so deployments +rebuild or recertify templates on relevant rollouts. That build also fixes the +activation-configuration schema that the template's clones accept. Processor capacity and RAM geometry are immutable, but restore-time activation lets one template serve several online processor counts and RAM sizes. Templates are therefore keyed by capacities, base state, and supported activation ranges. Layer-role presence and device geometry stay fixed: a -platform template accepts a substitute layer only with the same recorded -logical length and block sizes. +platform template accepts a substitute layer only with the same block sizes +and a length no greater than its recorded logical length, which the device +continues to present, so one template serves every image up to that length. ## 1.5.3 Persisted State @@ -79,3 +92,10 @@ The snapshot manifest is authoritative for a restored process. Native file descriptors, sockets, named-pipe handles, and hypervisor objects are never part of the portable state. They are recreated from stable identities and declared reconstruction policies. + +The runtime store has one versioned format and no readers for older formats; +leaving an older format requires one stop of its sandboxes. After that, +upgrades are live: after an orchestrator restart within the same host boot, a +host reattaches a running sandbox only when its recorded runtime identity +matches. A different runtime identity stops the sandbox by identity-verified +forced termination, without speaking a possibly different guest protocol. diff --git a/doc/design/2.1-machine-model-and-configuration.md b/doc/design/2.1-machine-model-and-configuration.md index d3701baa2..7b54621fe 100644 --- a/doc/design/2.1-machine-model-and-configuration.md +++ b/doc/design/2.1-machine-model-and-configuration.md @@ -56,9 +56,11 @@ lifetime is one of: | Processor capacity, base RAM and optional RAM capacity, and CPU profile | Host, validated by the machine profile | VM | | Effective kernel command line | Machine profile, from host policy and caller arguments | VM | | Workload identity and lifecycle | Host | VM | -| Console, network, block, and host filesystem devices, with their identities and policies | Host | VM | -| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities | VM process | +| Console, network, block, and host filesystem devices, with their identities and policies, except a platform clone's network policy below | Host | VM | +| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities, or, for the unbound layers of a `platform` snapshot, within the recorded lengths | VM process | | Online processor and RAM activation targets | Host, within snapshot limits | Restore | +| Activation configuration of a platform clone | Host, validated by the guest agents | Restore | +| Network policy of a managed layered root's platform clone whose template was captured with a deny-all policy | Host | Restore | The host fixes the workload policy of a fresh boot. An optional nonzero numeric UID and GID become the fixed workload identity, and an optional @@ -68,22 +70,31 @@ Both are command-line tokens that the machine profile owns and that caller arguments cannot supply. A managed lifecycle requires a fixed workload identity and a live, authenticated control console. +The host also fixes the activation configuration of a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root). +A cold boot carries it as launcher tokens, and a platform clone receives it in +the restore packet's activation record. A launch that declares a `platform` +capture carries none, because that tier consumes only invariants. + ## 2.1.3 Storage Configuration Sandbox storage uses block devices with fixed roles rather than arbitrary disks. The roles are the layers of the sandbox root filesystem: a read-only -`distro` lower layer for the base operating system, beneath one writable -`scratch` layer that holds the sandbox's private changes. +`distro` image of the base operating system, either one filesystem or a GPT +disk of ordered lower layers, beneath one writable `scratch` layer that holds +the sandbox's private changes. [Block Storage](3.4-block-storage.md) describes each role. A disk without a role is rejected, and a disk-backed topology ends with writable scratch. The only exception is the RAM-backed overlay layout, which -requires exactly one read-only `distro` block and no scratch, keeps the -guest's overlay upper layer in RAM, and supports neither capture nor restore. -Snapshot capture with blocks requires a read-only `distro` layer followed by -writable scratch, both backed by cached, regular raw files with nonzero, -512-byte-aligned geometry. Blockless machines are also snapshot-capable and -carry no sandbox tier metadata. +requires exactly one read-only `distro` block and no scratch and keeps the +guest's overlay upper layer in RAM. It may be captured only at the `platform` +tier, before the guest agents mount the `distro` image or the overlay, so its +upper layer is never captured. Any other snapshot capture with blocks +requires a read-only `distro` layer followed by writable scratch. Every block +of a snapshot-capable machine is backed by a cached, regular raw file with +nonzero, 512-byte-aligned geometry. Blockless machines are also +snapshot-capable and carry no sandbox tier metadata. Host filesystem sharing uses the fixed filesystem slot rather than adding devices. The slot exports one host directory, or an aggregate of named @@ -96,10 +107,16 @@ Restore is stricter than cold boot. Guest-visible configuration is read from the snapshot's machine contract. Restore-time input may only: - Select the backend kind that the snapshot records. -- Supply required attachments. -- Bind a new directory export to a captured dormant filesystem slot, when the - snapshot records that capability. +- Supply required attachments. For a `platform` snapshot, these include + substitute read-only layers no longer than their recorded lengths; see + [Host Storage and Access Enforcement](3.4-block-storage.md#344-host-storage-and-access-enforcement). +- Bind a new directory export or aggregate to a captured dormant filesystem + slot, when the snapshot records that capability. - Select processor and RAM activation targets that the contract allows. +- For a `platform` snapshot, supply the activation record of the + [restore packet](5.5-restore-protocol.md#555-restore-packet) and, when the + snapshot was captured with a deny-all network policy, a new network policy; + see [Bounded State, Snapshots, and Restore](3.5-network.md#356-bounded-state-snapshots-and-restore). These process-local inputs never change processor capacity, RAM capacity, the command line, device placement, feature masks, or filesystem and network diff --git a/doc/design/3.1-platform-control-device.md b/doc/design/3.1-platform-control-device.md index 2f3584378..ba088efcd 100644 --- a/doc/design/3.1-platform-control-device.md +++ b/doc/design/3.1-platform-control-device.md @@ -32,8 +32,10 @@ machine omits, and carries: console. - The one-time [restore packet](5.5-restore-protocol.md#555-restore-packet), which gives a restored guest what it needs to repair its clocks, - processors, and memory, and the fresh random bytes from which it - [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh). + processors, and memory, the fresh random bytes from which it + [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh), and, + in a platform clone, the configuration with which it + [activates](5.5-restore-protocol.md#559-platform-clone-activation). - Host time samples, which the guest pairs with its own clock to set and discipline its wall clock. - The generation ID, a random identifier unique to each VM process. Every @@ -137,4 +139,8 @@ immediately after the output operation. After a gated restore, a write to the same port acknowledges that processor, memory, time, entropy, and workload state have been repaired. Host input is -released only after the VMM stops at that acknowledgement boundary. +released only after the VMM stops at that acknowledgement boundary. In a +platform clone, the guest agents validate the activation record before the +acknowledgement and +[activate](5.5-restore-protocol.md#559-platform-clone-activation) the workload +root after it. diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index 5c0c3e808..6bddcd660 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -365,7 +365,9 @@ synchronization, offset, uncertainty, and frequency correction. the bound and steps its clock to host UTC, so the workload starts on host time. It runs every other boot check asynchronously after readiness, starting at idle priority, so the checks never delay readiness or take a processor from -the workload. The checks cover the identity leaves and explicit zero leaves, +the workload. A managed layered root launched for a `platform` capture reports +readiness only in its clones, so its guest agents run these checks before the +capture instead. The checks cover the identity leaves and explicit zero leaves, the CPU time bits on every CPU, the identity MSRs, the kernel's platform, clocksource, and LAPIC-timer selection, CPU flags, tick devices, the absence of forbidden clock parameters, stall-detector settings, a running violation diff --git a/doc/design/3.4-block-storage.md b/doc/design/3.4-block-storage.md index f7f96e2c5..277aec906 100644 --- a/doc/design/3.4-block-storage.md +++ b/doc/design/3.4-block-storage.md @@ -10,7 +10,7 @@ arbitrary disks. | Role | Access | Purpose | | --- | --- | --- | -| `distro` | Read-only | Curated base operating-system layer | +| `distro` | Read-only | Curated base operating system: one EROFS filesystem, or a GPT disk of ordered layers | | `scratch` | Read-write | Private ext4 upper and work storage | Roles are unique and appear in fixed order; an omitted `distro` role leaves @@ -35,8 +35,10 @@ Each role offers a fixed capability set: virtio 1.0 split rings with indirect descriptors and event-index notification suppression, a per-request segment limit, block-size and topology reporting, and a write-back cache with explicit flush. The `distro` role is also advertised as read-only. Discard is never -offered. The device reports the backing store's capacity and its logical and -physical block sizes. +offered. The device reports the backing store's capacity, or the declared +length of an +[unbound platform-tier layer](#344-host-storage-and-access-enforcement), and +its logical and physical block sizes. ## 3.4.3 Request and Completion Path @@ -74,6 +76,16 @@ block must then be a nonempty, 512-byte-aligned raw regular file that uses cached I/O and is not reached through a final symbolic link. The VMM retains that opened file for flushing and paired-scratch publication. +An unbound `platform`-tier layer may be shorter than the length that its +device presents. A launch that declares a `platform` capture may give the +`distro` layer a logical length larger than its file, typically a small +placeholder, and the snapshot records that length. A restore may then supply +any layer file that is a whole number of logical blocks long and no longer +than the recorded length; a longer file is rejected. The device presents the +declared length and returns zeros for reads past the file's end, so one +template serves every image up to that length, and the padding consumes no +storage. Writes never reach the padding, because the role is read-only. + ## 3.4.5 Snapshot Contract and Quiesce The snapshot contract records each block's role, read-only flag, logical @@ -81,7 +93,8 @@ length, logical and physical block sizes, and identity. Every consumed block shares one identity, a caller-authenticated immutable storage generation; see [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). Platform-tier layers are recorded as unbound, because image binding has not -been consumed, so a restore may supply different layers of the same geometry. +been consumed, so a restore may supply different layers with the same block +sizes, no longer than the recorded logical length. Writable scratch follows one of two policies: - **Paired.** Capture publishes the exact scratch image with its length and diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index 28815f3dd..c86763822 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -170,17 +170,23 @@ to guest ports, at most 64 of them. Neither binds a LAN-facing listener. The host decision names no ports, and the endpoint owns no listener of its own, so publishing follows the ports on which the guest listens. The machine profile passes the ingress mode to the guest in a boot token that callers -cannot supply, and a guest agent then announces the guest's listeners. It +cannot supply, and a platform clone that installs a new policy at restore +receives it in its activation record instead. A guest agent then announces +the guest's listeners. It sends each announcement as one UDP datagram to a fixed port of the IPv4 -gateway from a privileged source port, which the workload, holding no -capabilities, cannot bind, so the workload cannot forge an announcement. An -announcement carries a magic value, a version, a flag that marks omitted -listeners, and at most 64 entries in strictly increasing order of protocol and -port, each naming the protocol, TCP or UDP, the address families, and the -port. The endpoint consumes announcements before the egress policy, which -neither authorizes nor forwards them, and each announcement replaces the -published set. For each published port and address family, the endpoint keeps -one host listener at the guest's port number: +gateway from a privileged source port. While private-network ingress or +host-loopback publication is active, the workload is refused +`NET_BIND_SERVICE` and `NET_RAW`; `NET_ADMIN` is never granted. The runtime +owns `net.ipv4.ip_unprivileged_port_start` and keeps it at 1024 or above. A +UID 0 workload sees `/proc/sys` read-only, so it cannot lower that threshold. +The workload therefore cannot forge an announcement. An announcement carries +a magic value, a version, a flag that marks omitted listeners, and at most 64 +entries in strictly increasing order of protocol and port, each naming the +protocol, TCP or UDP, the address families, and the port. The endpoint +consumes announcements before the egress policy, which neither authorizes nor +forwards them, and each announcement replaces the published set. For each +published port and address family, the endpoint keeps one host listener at +the guest's port number: - **Private-network ingress** listens on every host address and relays only peers in ranges that are not globally reachable: the IPv4 private-use, @@ -222,7 +228,9 @@ The gateway provides a DNS proxy over UDP and TCP port 53. The guest is pointed at the IPv4 gateway when the egress policy permits DNS to it, and otherwise at the IPv6 gateway when generic rules permit DNS there. Exact-endpoint policies never name a gateway, and without a permitted gateway -the guest receives no DNS server. +the guest receives no DNS server. A platform clone that installs a new policy +at restore receives its DNS server in the activation record rather than in a +boot token, because its command line was fixed under a deny-all policy. A guest query passes the normal egress checks before the gateway handles it. Upstream resolution uses the host resolver, whose upstream servers are not @@ -243,10 +251,25 @@ beyond it resets its connection. The snapshot records the network profile, identity, and policy digest. Restore reconstructs a fresh endpoint and requires the same network profile -and policy; native sockets, DNS requests, and flows are never serialized. -Published guest listeners and explicit host forwards are process-local -attachments, so capture and restore reject private-network ingress and -explicit host-loopback access, which create them. +and, with one exception, the same policy; native sockets, DNS requests, and +flows are never serialized. Published guest listeners and explicit host +forwards are process-local attachments, so capture and restore otherwise +reject private-network ingress and explicit host-loopback access, which +create them. + +The exception is a `platform` snapshot captured with a deny-all policy: a +default deny without allow rules, host loopback denied without a proxy +endpoint, and no inbound access. Its guest has no DNS server and announces no +listeners, and its endpoint holds no flows, listeners, or forwards, so a +restore may supply a new policy, including private-network ingress, explicit +host-loopback access, and forwards. The endpoint installs that policy before +any restored processor runs. Because the guest learns its DNS server and +ingress mode only from the +[activation record](5.5-restore-protocol.md#559-platform-clone-activation), +hosts supply a new policy only to clones of a managed layered root. The +captured identity still applies, so explicit host-loopback access requires an +IPv4 network of /29 or larger. Every other snapshot keeps the same-policy +rule. Capture quiesces the endpoint, drains completion ownership, and requires the saved queues to contain no unrepresented RX or TX packets. It saves the queue diff --git a/doc/design/3.6-host-filesystem.md b/doc/design/3.6-host-filesystem.md index 67a7dd8e7..943ec642a 100644 --- a/doc/design/3.6-host-filesystem.md +++ b/doc/design/3.6-host-filesystem.md @@ -46,7 +46,8 @@ descriptors and event-index notification suppression, but never packed rings or the DAX shared-memory window, so file data always travels through the queues. The machine profile rejects alternate tags, extra queues, shared-memory windows, PCI transport, and aggregates whose children change -after boot. +after the aggregate attaches, at cold boot or, on a dormant slot, at +restore. The guest opens the FUSE session with `INIT`, and every other request fails until it succeeds. The device speaks FUSE protocol version 7 and accepts guest @@ -108,7 +109,11 @@ name, which the guest never reaches, so the guest cannot list or change anything beside the file, a lookup below it fails with `ENOTDIR`, and every request fails with `EACCES` while anything other than a regular file holds the name. A file child takes no path policy, because its mode covers the whole -file. +file. By default, a file child follows its name, so a regular file that host +software renames over the name becomes the child's file. An attachment may +instead pin a file child to the object that its name reached when the export +opened: the child then serves only that object, through held nodes and new +lookups alike, and fails requests rather than serve a replacement. Child names are unique, use `[A-Za-z0-9._-]`, and are at most 64 bytes long. An aggregate holds at most 256 children with at most 128 KiB of policy paths, @@ -254,8 +259,13 @@ could not track or save. The host root is pinned by its platform object identity, device and inode number on Linux or volume and file ID on Windows, and revalidated when the -export opens; each aggregate child's root is pinned the same way. Every -operation stays confined to the export: host path resolution follows no +export opens; each aggregate child's root is pinned the same way. A host may +also supply the identity that it expects for a root or a pinned file child. +Cold mappings record these identities at provision and recheck them at start; +a mismatch refuses the export. Opened-object +[attachment receipts](5.4-snapshot-artifact-and-attachment-model.md#547-host-attachments) +belong to platform-clone restore and are not part of the cold mapping path. +Every operation stays confined to the export: host path resolution follows no symbolic link or reparse point, never leaves the root, and never follows the final component. On Windows hosts, guest-created links are reparse points that path resolution never follows, ancestors that are links or reparse points are @@ -322,13 +332,14 @@ the matching state: - The next node and handle IDs. - The fixed device shape, access mode, ownership mode, root identity, and access policy, or, for an aggregate, each child's name, kind, mode, root - identity, and policy. + identity, and policy, and whether a file child is pinned. Device state names objects relative to the export root and never records a host path. The snapshot's machine contract records the host side instead: the guest target, access mode, tag, queue counts, direct I/O, cache lifetimes, canonical host path, policy paths, ownership mode, and, for an aggregate, each -child's name, canonical host path, kind, mode, policy, and root identity. Its +child's name, canonical host path, kind, mode, policy, and root identity, and +the identity of each pinned file child's file. Its attachment inventory binds the export, with a live-revalidation reconnect policy, to the root's identity or, for an aggregate, to a digest of its children's names and root identities in order, so a restore can neither drop, @@ -369,8 +380,9 @@ it: 1. Before the guest runs, the launcher supplies the export again. It must reproduce the snapshot's target, mode, policy, ownership mode, canonical host paths, and root identities, and, for an aggregate, the same children - with the same names, kinds, and modes in the same order. Only a dormant - slot may instead gain a new export. + with the same names, kinds, modes, and pinning in the same order, each + pinned file child reaching its recorded file. Only a dormant slot may + instead gain a new export. 2. Before the transport changes any state, the device validates the saved state: a known schema version, the fixed device shape, a FUSE session that the device would negotiate, at minor version 31 or newer, tables within @@ -409,13 +421,16 @@ A dormant capture saves explicit unattached state, with no session, nodes, handles, root identity, or policy. It requires that the guest have no FUSE session on the slot, so the tag must not be mounted; unmounting ends the session. A dormant snapshot may restore with no export or bind one new -directory export, but not an aggregate. The resumed guest then mounts the tag -explicitly, because its boot-time mount step has already run. A snapshot -without this capability cannot add an export, and active state cannot restore -into a dormant slot. +directory export or aggregate, whose children are then fixed for the restored +process. The resumed guest then mounts the tag explicitly, because its +boot-time mount step has already run; the guest agents of a platform clone +learn the export's form from the +[activation record](5.5-restore-protocol.md#559-platform-clone-activation). +A snapshot without this capability cannot add an export, and active state +cannot restore into a dormant slot. Capture records the saved-state version that the export requires. Caller ownership, the access policy of a directory export with allowed or writable -paths, an aggregate, and a file child each require a newer version, so a -release that predates the feature rejects the snapshot rather than restore a -weaker export. +paths, an aggregate, a file child, and a pinned file child each require a +newer version, so a release that predates the feature rejects the snapshot +rather than restore a weaker export. diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index 7f53986f3..4e341879c 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -9,18 +9,23 @@ As Table 4.1 shows, the guest agents run workloads in one of two roots. | Root | Construction | Host content | | --- | --- | --- | -| Layered sandbox root | An overlay of one read-only image layer and private writable scratch | Live host shares bound at declared targets | +| Layered sandbox root | An overlay of a read-only image and a private writable upper layer on scratch or in guest RAM | Live host shares bound at declared targets, or, in a managed lifecycle, host directories bound by a host mapping table | | Blockless managed root | The guest's own initramfs root, without block devices | Host directories bound by a host mapping table | The guest kernel provides the facilities both roots need: virtio-blk, -virtio-fs, EROFS with compression and extended attributes, overlayfs, cgroup -v2 with the memory and process controllers, namespaces, BPF system calls, and -cgroup BPF programs. +virtio-fs, EROFS with compression and extended attributes, ext4, GPT partition +tables, overlayfs, cgroup v2 with the memory and process controllers, +namespaces, BPF system calls, and cgroup BPF programs. ## 4.1.1 Layered Sandbox Root -As Figure 4.1 shows, the layered root combines one immutable EROFS `distro` -layer with one private ext4 scratch device. +As Figure 4.1 shows for an EROFS image with its upper layer on scratch, the +layered root combines the immutable `distro` image with one private writable +upper layer. The image is either one EROFS filesystem or a GPT disk whose +first partition holds image metadata, such as the image's configuration, and +whose later partitions hold read-only ext4 lower layers, ordered from base to +top. The upper layer is on an ext4 scratch device or, in the RAM-backed +overlay layout, in guest RAM. ```text /run/nvx/layers/distro (EROFS) ----------> lower directory @@ -46,34 +51,57 @@ The guest agents assemble the root in this order: container cgroups, and move the guest agents into the guest-agent cgroup. 2. Resolve each role's MMIO address through the kernel's platform-device resources rather than assuming a device name or enumeration order. -3. Check any expected sector count, flush cached block-device buffers, mount - the `distro` layer read-only as EROFS, and check its expected filesystem - UUID. +3. Check any expected sector count and flush cached block-device buffers. + Mount an EROFS image read-only and check any expected filesystem UUID. For + a GPT image, validate only the primary header and entry array, reread the + partition table, and mount partitions 2 through N as read-only ext4 lower + layers with `ro,nosuid,nodev`. Never use the backup header and never add + `noload`. 4. Resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and - create its upper and work directories. -5. Mount the overlay with the `distro` layer as its lower directory, - metadata-only copy-up, and extended inode numbers. + create its upper and work directories, or, in the RAM-backed overlay + layout, create them in tmpfs mounted with `nosuid,nodev` at the kernel's + default size, half of guest memory. +5. Mount the overlay with the EROFS image, or the GPT image's layers from top + to base, as its lower directories. Every image uses + `metacopy=on,xino=on` by default. An orchestrator whose lower layers may + carry untrusted overlay attributes selects + `metacopy=off,redirect_dir=nofollow`. The guest uses exactly the selected + option set without fallback and mounts the merged root `nosuid,nodev`. 6. Verify the workload identity in the assembled root, mount each live share - at its target inside that root, and write the workload's machine ID to a - runtime file. + at its target inside that root, and, when selected, write the generated + machine ID to a runtime file. 7. Start the workload behind a barrier and place it in the workload cgroup before releasing it. 8. Wait for the workload, unmount the shares in reverse order and then the - overlay, the `distro` layer, and scratch, and report the workload's status. - -The UUID check detects attachment mistakes; it is not a content-integrity -proof, and neither the guest nor the VMM verifies layer contents. -Metadata-only copy-up interprets overlay metadata stored in the `distro` -layer, so that layer is a trusted input that must not carry untrusted overlay + overlay, the `distro` image's mounts, and scratch or tmpfs, + and report the workload's status. + +GPT validation bounds the header and entry array, verifies their CRCs, +extents, indices, and GUIDs, and permits at most 255 used partitions, the +kernel's limit. The primary header's alternate LBA must lie inside the device, +which may present zeros past a shorter image. Partition 1 holds opaque +metadata bytes. The guest reads them up to a caller-selected bound, trims +trailing NUL and whitespace, and never probes the partition as a filesystem; +the adapter parses the result. After the partition-table reread, every kernel +partition must match its validated entry. + +Expected identities are optional inputs: the EROFS filesystem UUID, or the +GPT disk GUID, partition GUIDs, and partition sizes. NVX's adapter passes the +EROFS UUID or GPT disk GUID from its boot token. These checks detect attachment +mistakes; they are not a content-integrity proof, and neither the guest nor +the VMM verifies layer contents. Metadata-only copy-up interprets overlay +metadata stored in the `distro` layer, so an orchestrator selects the +non-metacopy option set when lower layers may carry untrusted overlay attributes. A metadata-only change avoids copying file contents, but a data -write still copies the whole lower file into scratch, so scratch capacity must -account for copy-up. Scratch consumes host storage rather than guest RAM and -can run out of space independently of guest memory pressure. +write still copies the whole lower file into the upper layer, whose capacity +must account for copy-up. Scratch consumes host storage rather than guest RAM +and can run out of space independently of guest memory pressure. The initramfs remains the guest agents' root; it is not another workload -layer. The guest agents always keep the overlay upper layer on scratch, so the -machine's RAM-backed overlay layout, which has no scratch, serves only guests -that declare it. +layer. The guest agents keep the overlay upper layer on scratch or, when the +machine declares the RAM-backed overlay layout, which has no scratch, in guest +RAM, where the workload's writes consume guest memory instead of host +storage. ## 4.1.2 Live Host Shares @@ -113,7 +141,9 @@ The workload's mount namespace inherits these mounts, and teardown unmounts the binds in reverse order before the aggregate. Any refusal or mount failure aborts the sandbox instead of starting the workload without its shares. A managed sandbox records its shares when it is provisioned and reattaches them -on every start. +on every start. A [managed layered root](#414-managed-layered-root) may +instead receive host directories as a host mapping table, and a platform +clone must. ## 4.1.3 Blockless Managed Root @@ -121,6 +151,8 @@ Without image layers, each workload runs directly in the guest's initramfs root, inside a dedicated execution cgroup. Host directories reach it through a host mapping table rather than an image: +- The host planner refuses a read-only path inside a read-write mapping on + every host. A read-write path inside a read-only mapping remains valid. - The host exports the outermost mapped directories, and each mapped file outside them on its own, as numbered children of one aggregate, so a file child exposes only its file. The guest agents mount the aggregate at a @@ -138,3 +170,40 @@ host mapping table rather than an image: before mounting anything, and bind-mount each source at its target with `nosuid,nodev`, read-only when requested. They refuse to run workloads until every announced mapping is mounted, and a failed mount keeps them refusing. + +The blockless root may create the workload account that its adapter requests. + +## 4.1.4 Managed Layered Root + +In a managed lifecycle, the layered sandbox root uses the shared guest +runtime and host libraries for root assembly, host mappings and execution. +NVX's adapter speaks +[NVXC v2](4.4-host-guest-control-interface.md#444-protocol-versions) over +NVXS; other adapters may carry their own application framing over NVXS and +keep their own admission policy. Unlike the blockless root, it never creates +workload accounts. The identity must already exist in the image, or the +adapter must select numeric IDs without an account check. It differs from a +one-shot layered root in four ways: + +- **Readiness.** The guest agents assemble the root from boot tokens without + waiting for the host, then start the control service. Readiness therefore + means that the root is assembled and the control path is established, and + the readiness report names the activation configuration that the guest + consumed. +- **Host directories.** The command line may announce a number of host + mappings, which the host delivers over the control interface after + readiness, as for the [blockless managed root](#413-blockless-managed-root). + The guest agents mount the aggregate at a root-only location outside the + workload root and bind each mapping at its target inside the assembled + root, creating targets under the rules of + [Live Host Shares](#412-live-host-shares), with `nosuid,nodev`, and + read-only when requested. An announced count of zero mounts nothing. +- **Executions.** Each execution enters the assembled root with the + workload's identity and isolation and runs in its own child cgroup of the + workload cgroup; see + [Containment](4.2-guest-agents-and-workload-model.md#422-containment). +- **Platform clones.** A clone of a `platform` snapshot receives the same + activation configuration in the restore packet's activation record instead + of boot tokens, and its host directories only as a host mapping table, + because its command line, fixed at capture, carries no share tokens; see + [Platform-Clone Activation](5.5-restore-protocol.md#559-platform-clone-activation). diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index c233f71cc..29ff51107 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -36,12 +36,12 @@ flowchart TB In a one-shot lifecycle, the guest agents start one workload, configured by non-secret kernel command-line tokens, and shut down the VM when it exits. In -a managed lifecycle, the guest agents stay resident and run a sequence of -workloads, one at a time, on request over the authenticated control channel. -Both modes preserve the same separation: the workload is always a child of a -guest agent. The guest agents never replace themselves with the workload, -because supervision, orphan reaping, and the control service all depend on -them remaining alive. +a managed lifecycle, the guest agents stay resident and run workloads on +request over the authenticated control channel, under the adapter's +concurrency policy. Both modes preserve the same separation: the workload is +always a child of a guest agent. The guest agents never replace themselves +with the workload, because supervision, orphan reaping, and the control +service all depend on them remaining alive. ## 4.2.1 Execution Launch @@ -57,11 +57,14 @@ elsewhere. ## 4.2.2 Containment -In the blockless managed root, each execution runs in its own cgroup. When the -workload's first process exits, the guest agent kills every process that -remains in that cgroup and, as the guest's init process, reaps the orphans it -inherits. It reports the outcome only after the cgroup's process count reaches -zero. +In a managed root, each execution runs in its own cgroup: a dedicated +execution cgroup in the blockless managed root, and a child of the workload +cgroup in the +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root). +When the workload's first process exits, the guest agent kills every process +that remains in that cgroup and, as the guest's init process, reaps the +orphans it inherits. It reports the outcome only after the cgroup's process +count reaches zero. The process controller charges a process until it is reaped, so no workload process, not even a zombie, outlives its execution. A failed kill, an unreadable count, or a settlement timeout is reported as a containment @@ -69,9 +72,8 @@ failure, and every later execution first reaps what earlier workloads left and verifies that the cgroup is empty, so a session reset cannot bypass the check. -A cancellation that names the running execution kills the workload, and a +A cancellation that names an active execution kills its workload, and a cancellation for an execution that already finished is ignored. A session -reset means that the host session that could observe the workload is gone: -the guest agent kills the workload, sends nothing more for it, and -acknowledges the new epoch, so the next session never waits for an abandoned -workload. +reset means that the host session that could observe the workloads is gone: +the guest agent kills them, sends nothing more for them, and acknowledges the +new epoch, so the next session never waits for an abandoned workload. diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index a360fb779..1d3c8c7ad 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -5,36 +5,53 @@ ## 4.3.1 Identity and Credentials -The workload runs under a host-selected, non-root numeric UID and GID fixed at -initial boot. The guest agents require an exact user and primary-group match -in the workload root and reject an unavailable identity before starting the -workload. When the boot configuration requests it, they instead create an -account for a host-selected identity that the image lacks, and replace an -image account that uses the UID with another primary group or without a -usable home. This lets a host run workloads under its own user's IDs. - -Before the workload starts, the guest agents clear supplementary groups and -every capability set and enable `no_new_privs`. With every capability cleared, -the workload cannot perform port I/O, so only the guest agents can request a -capture or acknowledge restore repair. +The adapter sets the workload's UID, GID, supplementary groups, and +capabilities within the guest runtime library's rules. The identity is fixed +at initial boot. Capabilities come only from this allowlist: `CHOWN`, +`DAC_OVERRIDE`, `FOWNER`, `FSETID`, `KILL`, `SETGID`, `SETUID`, `SETPCAP`, +`SETFCAP`, `NET_BIND_SERVICE`, `NET_RAW`, `SYS_CHROOT`, and `AUDIT_WRITE`. +The library refuses every other capability and gives a non-root identity no +capabilities. It always sets `no_new_privs`. The allowlist excludes +`SYS_RAWIO`, and the guest agents revoke their port-I/O permission before +starting any workload stage, so only the guest agents can request a capture +or acknowledge restore repair. + +The adapter also selects the account mode. Existing-account mode requires one +passwd entry for the UID, the requested primary GID, an existing group, and +an absolute home. NVX's adapter uses this mode by default. Numeric-ID mode +uses the requested IDs without an account check. The layered root never +creates accounts; the blockless root may create one when its adapter requests +it. NVX's adapter keeps a non-root identity with no capabilities. ## 4.3.2 Namespaces and Mounts -In the layered sandbox root, the workload receives private mount, PID, and UTS -namespaces. Its entry helper makes every mount private and provides: +In the layered sandbox root, each execution receives private mount, PID, and +UTS namespaces. The adapter selects the hostname; NVX's default is +`nvx-sandbox`. User namespaces remain unavailable. The entry helper makes +every mount private and provides: - A private `/proc` and a read-only `/sys`. -- A restricted `/dev`, devpts, and shared memory. -- The workload root, entered with `chroot`. +- A `/dev` that contains only allowlisted device nodes, devpts, and shared + memory. +- The workload root, moved over `/` in the execution's mount namespace, never + entered by `chroot` alone. - Only the live shares that the host declared. -- The workload's machine ID, bound read-only from a runtime file so that the - guest agents can refresh it while scratch is frozen. +- Either a generated per-boot machine ID, bound read-only from a runtime file + so that the guest agents can refresh it while scratch is frozen (NVX's + default), or the image's own `/etc/machine-id`, as the adapter chooses. - The guest time status, bound read-only. +Every other workload-visible mount is `nodev`. For UID 0, `/proc/sys`, +`/proc/sysrq-trigger`, `/proc/bus`, `/proc/fs`, and `/proc/irq` are read-only. +`/proc/kcore`, `/proc/keys`, `/proc/timer_list`, and similar sensitive files +are masked. + The guest agents never pivot away from the initramfs, so they keep their own root and stay outside the workload's namespaces, where they can supervise and tear down the workload. The control device is never exposed inside the -workload's namespaces. +workload's namespaces. Before entry, the helper closes old-root, control, +namespace, and process descriptors. The workload inherits no control +terminal. The blockless managed root has no such isolation. Each execution runs in the guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS @@ -52,7 +69,9 @@ released. The guest agents' cgroup carries a default memory protection of 16 MiB, a soft reservation that shields the supervisor from reclaim under workload memory pressure but does not guarantee its survival. The workload cgroup carries optional memory and process-count limits and is the unit of -capture freeze, termination, and descendant cleanup. +capture freeze, termination, and descendant cleanup. In a managed lifecycle, +each execution's cgroup is a child of the workload cgroup, so those limits and +the capture freeze cover every execution. In the blockless managed root, each execution joins a dedicated cgroup before the workload runs, for termination and descendant accounting. This path does diff --git a/doc/design/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md index d5e8f4813..d9e8a033b 100644 --- a/doc/design/4.4-host-guest-control-interface.md +++ b/doc/design/4.4-host-guest-control-interface.md @@ -4,13 +4,13 @@ [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) Managed operation uses the dedicated control console rather than the boot -console. The VMM owns the outer transport: bounded framing, same-user local -endpoint authorization, capability authentication, reconnect epochs, and -receive-credit backpressure; see +console. The VMM owns the outer NVXS v1 transport: bounded framing, same-user +local endpoint authorization, capability authentication, reconnect epochs, +and receive-credit backpressure; see [Control Console](3.3-consoles.md#332-control-console). The guest agents own -the workload protocol carried inside that stream. Boot diagnostics and kernel -messages stay on the consoles, so framed control traffic never shares an -unstructured byte stream with them. +the NVXC v2 application protocol carried inside that stream. Boot diagnostics +and kernel messages stay on the consoles, so framed control traffic never +shares an unstructured byte stream with them. ## 4.4.1 Operations @@ -20,10 +20,9 @@ Table 4.2 lists the operations of the control protocol. | Operation | Purpose | | --- | --- | -| `READY` | Reports readiness, and is the success reply to requests that return no other data | -| `FEATURES` | Reports which control behaviors the guest image provides | +| `READY` | Reports the application version, guest build ID, and root profile, and is the success reply to requests that return no other data | | `MAPS` | Delivers a contiguous range of host mapping-table entries | -| `EXEC` | Runs one workload with a timeout, arguments, and an optional working directory and environment | +| `EXEC` | Runs one workload with a timeout, arguments, optional working directory and environment, and standard input | | `CANCEL` | Kills the running execution that the request names | | Shutdown | Shuts the VM down gracefully between executions | @@ -32,10 +31,11 @@ and optionally an absolute working directory of at most 4,096 bytes and up to 256 `KEY=VALUE` environment entries of at most 4,096 bytes each, with distinct, non-empty keys. The environment either replaces the default environment or is layered over it, each entry replacing the default variable -of the same name. Output returns as separate standard output and standard -error streams within the credit window, at most 1 MiB of them combined for -each execution. As Table 4.3 shows, an execution that runs ends with an exit -report, which carries an outcome category and a status. +of the same name. Standard input, standard output, and standard error stream +within per-execution credit windows. Only data in flight is bounded; there is +no total output limit. As Table 4.3 shows, an execution that runs ends with an +exit report, which carries an outcome category, status, and output-truncation +flag. **Table 4.3.** Exit outcomes. @@ -45,7 +45,6 @@ report, which carries an outcome category and a status. | `signal` | A signal ended the workload; the status is 128 plus the signal number | | `timeout` | The workload exceeded its timeout and was killed; the status is 124 | | `cancelled` | A cancellation killed the workload; the status is 137 | -| `output-limit` | The workload's combined output exceeded 1 MiB, and the guest agents killed it; the status is 125 | | `failed` | The guest agents could not supervise the workload; the status is 125 | Otherwise, the guest agents reply with an error, which carries one of the @@ -63,45 +62,42 @@ account for. | `containment-failed` | The guest agents could not prove that every workload process exited | | `mapping-failed` | A host mapping could not be mounted | | `mappings-incomplete` | Not every announced host mapping is mounted yet | -| `busy` | Another execution is running | +| `busy` | Another execution is running; NVXC runs one execution at a time | | `invalid-request` | The request is malformed or invalid in the current configuration | | `unsupported-operation` | The guest does not implement the operation | -## 4.4.2 Feature Negotiation +## 4.4.2 Readiness and Bundle Conformance -A successful readiness probe does not prove that an image enforces the -behaviors that a host depends on. As Table 4.5 shows, a `FEATURES` request -therefore returns a four-byte little-endian bit mask of control features. +NVXC has one application version, v2. `READY` carries that version, the guest +build ID, and the root profile. A managed layered root also carries its +activation-configuration identity. The build derives its ID at build time +from its build inputs. -**Table 4.5.** Control feature bits. - -| Bit | Feature | -| ---: | --- | -| 0 | Cancellation | -| 1 | Reserved; never set | -| 2 | Workload account creation | -| 3 | Per-execution cgroup containment | -| 4 | Per-execution environment | -| 5 | Working-directory refusal | -| 6 | Host mapping table | - -The blockless managed root provides all six features; the layered sandbox -root provides cancellation, workload account creation, and per-execution -environments. A feature bit is added together with the behavior it names. -Hosts ignore unknown bits and trailing bytes, read an `unsupported-operation` -refusal as no features, and terminate a guest that lacks a feature they need -instead of running workloads without the policy they requested. +On every fresh session, the host validates `READY` against the selected +bundle before it sends `MAPS` or `EXEC`. It terminates a guest whose +application version, build ID, root profile, or required activation identity +does not match. A restored guest reports the build captured in its snapshot, +so the host compares it with the bundle that built the template. ## 4.4.3 Session Semantics Requests are typed, length-bounded, ordered, and associated with one VM instance and epoch. Credit limits bound queued data in both directions. Malformed, unauthenticated, stale, duplicated, or out-of-order records fail -closed. Only one execution runs at a time; any request other than a -cancellation that arrives during an execution is refused as `busy`. Shutdown -is therefore graceful only between executions: a host cancels a running -execution before it requests shutdown, or else terminates the VM, which fails -the execution. +closed. NVXC runs one execution at a time: any request other than a +cancellation that arrives during an execution is refused as `busy`; standard +input and credit records belong to the running execution and are not +requests. Shutdown is therefore graceful only between executions: a host +cancels a running execution before it requests shutdown. Other adapters carry +their own framing over NVXS and set their own concurrency. + +Normal completion is lossless: the exit report follows the last output of +both streams. On cancellation, timeout, handle drop, disconnect, reset, or +Stop, pending output drains for a bounded grace period. The guest then +discards any remainder and sets the output-truncation flag, so teardown cannot +wait indefinitely for a reader. Control and timers stay live throughout. The +logical outcome, delivery completion and process settlement are separate: an +execution completes only after its descendants settle. The control session is process-local even when guest parser state is snapshotted. Restore establishes a fresh host capability, peer identity, @@ -110,6 +106,21 @@ any host data reaches them. A request from before the capture therefore can never be replayed into a clone or a resumed process. Secrets and per-execution environments belong on the authenticated control -channel, not in the kernel command line, image layers, scratch, or snapshot -metadata. Boot configuration that the machine profile owns contains only the -non-secret invariants needed before the control channel exists. +channel, not in the kernel command line, image layers, scratch, snapshot +metadata, or the restore packet's activation record. Boot configuration that +the machine profile owns contains only the non-secret invariants needed before +the control channel exists. + +## 4.4.4 Protocol Versions + +NVXC v2 uses piped standard input with an explicit end of input and gives each +execution its own credit windows in both directions. The host grants credit +for standard output and standard error as it consumes them, and the guest +grants credit for standard input as the workload drains it. A guest without +credit stops reading the workload's output, which blocks the workload instead +of killing it. Cancellation, credit grants, and session control still +progress while an execution's window is exhausted. + +NVXC execution credits are separate from the outer NVXS v1 transport's byte +credits. NVXC provides no terminal, window resizing, or signals other than +cancellation. diff --git a/doc/design/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md index 290ef835b..4e928e266 100644 --- a/doc/design/5.1-boot-protocol.md +++ b/doc/design/5.1-boot-protocol.md @@ -80,15 +80,27 @@ the [time ABI](3.2-clock-and-time.md#324-rates) MSRs, and `tsc_early_khz=` or `lapic_timer_hz=`, from the caller or a snapshot, is rejected. Launchers add runtime tokens of their own: one share token per host-share -child, the number of host mappings for a blockless managed root, and a request -to create the workload account. A caller may also declare the RAM-backed -overlay layout with exactly one `nvx_overlay_upper=ramfs` token; any other -form of that token is rejected. - -Embedded NULs are rejected, and the complete NUL-terminated command line must -fit in 64 KiB. The effective string and its SHA-256 digest become part of the -snapshot machine contract. A restore reuses that string verbatim and appends -no tokens. +child, the number of host mappings for a managed root, a blockless root's +request to create the workload account, and, for a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root), +its activation configuration, whose identity the guest agents report in +readiness. Activation tokens stay small: counts, flags, and identities, never +tables. A launch that declares a `platform` capture adds none of these +runtime tokens, because that tier consumes only invariants. The `distro` +token names the image format without probing fallback: +`nvx_layer=distro,MMIO,erofs[:UUID]` or +`nvx_layer=distro,MMIO,gpt[:DISK-GUID]`. A caller may declare the RAM-backed +overlay layout with exactly one `nvx_overlay_upper=ramfs` token, which selects +the tmpfs upper and permits no scratch block. Any other form or count of that +token is rejected. + +Launchers keep a 1,024-byte caller budget and refuse longer caller input +early. Embedded NULs are rejected. After appending its own tokens, the VMM +requires the effective command line to fit in 2,048 bytes including the +terminating NUL, at most 2,047 bytes of text, which is the guest kernel's +limit. It applies this check to cold boots and restores. The effective string +and its SHA-256 digest become part of the snapshot machine contract. A restore +reuses its saved command line verbatim and never rewrites or appends tokens. ## 5.1.3 Cold Boot Sequence @@ -106,5 +118,7 @@ A cold boot: A cold boot writes no TSC: every backend starts the processors' TSCs together. The kernel then discovers the fixed virtio-mmio devices and starts the guest agents, which step the guest clock to host UTC before readiness, -assemble the workload root, apply isolation, and start either the one-shot -workload or the managed control service. +assemble the workload root from boot tokens without waiting for the host, +apply isolation, and start either the one-shot workload or the managed control +service. A host that manages the VM attaches to the control console in +parallel and observes readiness once the control path is established. diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index 1bbac96a9..4e2565da2 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -6,7 +6,12 @@ Readiness is an explicit lifecycle boundary, not the observation of arbitrary console text. A cold boot is usable when the guest agents have initialized the required devices and either prepared the one-shot workload or established the -managed control path. +managed control path. In a +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root), +the guest agents assemble the workload root before they establish the control +path, so readiness also means that the root is assembled; a platform clone +reaches the same state through +[activation](5.5-restore-protocol.md#559-platform-clone-activation). ## 5.2.1 Control Frontends @@ -29,7 +34,10 @@ writes one fixed readiness record after every state unit has started and before it releases a restored processor. For a gated restore, host input is still gated at that point, and guest repair and its acknowledgement follow the event. A connection, write, or flush failure aborts startup, and the -endpoint is never saved in a snapshot. +endpoint is never saved in a snapshot. The event marks a started VM, not a +usable one: in a platform clone, activation also follows it, so a host runs +workloads only after the guest agents report readiness over the control +channel. ## 5.2.3 Management Exclusion @@ -50,7 +58,7 @@ operations. | Operation | Effect | | --- | --- | | Provision | Records the sandbox's configuration, including its host shares | -| Start | Launches the VMM with a freshly generated control capability, reattaches the recorded shares, and waits for guest readiness | +| Start | Launches the VMM with a freshly generated control capability, reattaches the recorded shares, waits for guest readiness, and delivers any host mapping table, completing once every announced mapping is mounted | | Exec | Runs one workload over the control channel | | Stop | Shuts down the VM and releases its runtime resources | | Deprovision | Removes the sandbox's state | @@ -61,11 +69,34 @@ of each sandbox, reconciles VMM processes with recorded state, and recovers interrupted launches. Teardown leaves no VMM process, control endpoint, capability, or runtime record, while supplied layers and scratch survive. +NVX's adapter admits one active execution per sandbox and keeps a FIFO of at +most eight waiters across CLI processes. The default queue wait is 60 seconds +and each invocation may override it. A full queue fails at once with +`queue-full`; an expired wait fails with `queue-timeout`. Other adapters choose +their own execution-concurrency policy. + +Stop bypasses the queue. It fences new admission, cancels the waiters, asks +the active execution's owner to cancel and release the slot, and then requests +graceful shutdown. If that sequence does not complete in time, NVX's CLI +reports the timeout and leaves the VM running; it never escalates. The +library's identity-verified forced termination remains available to other +callers. + +After an orchestrator restart, reattach is available only within the same host +boot and only for the same +[runtime identity](1.5-compatibility-and-versioning-model.md). A mismatch uses +the library's identity-verified forced termination without speaking the guest +protocol. + The sandbox lifecycle API exposes the same operations with request validation, backend capabilities, durable records, and stable error codes. Its default backend launches the VMM directly with the managed lifecycle and the -blockless managed root. An alternative backend delegates the lifecycle to a -separately supplied native library and an image-backed guest. +blockless managed root. Its image-backed backend runs the +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) +through the host control library and the guest runtime library. An external +orchestrator may drive the same host library through its own adapter, for +example in a separately supplied native library, and an external guest agent +may embed the same guest library. ## 5.2.5 Authority diff --git a/doc/design/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md index d400ec0e6..de5b1a077 100644 --- a/doc/design/5.3-snapshot-protocol.md +++ b/doc/design/5.3-snapshot-protocol.md @@ -26,7 +26,9 @@ Before they write the snapshot port, the guest agents: 2. Save and suppress RCU stall detection, and the debug stall detectors. 3. For paired scratch, freeze the workload cgroup with a bounded wait, synchronize, and freeze the mounted scratch filesystem, or, for fresh - scratch, require scratch to be unmounted. + scratch, require scratch to be unmounted. A `platform` capture, including + one in the RAM-backed overlay layout, which has no scratch, also requires + that neither the `distro` image nor the overlay is mounted. 4. Write the capture request, with its scratch policy, to the snapshot port. A committed capture terminates the source, so the write returns in the source diff --git a/doc/design/5.4-snapshot-artifact-and-attachment-model.md b/doc/design/5.4-snapshot-artifact-and-attachment-model.md index 5b8c131f3..692468361 100644 --- a/doc/design/5.4-snapshot-artifact-and-attachment-model.md +++ b/doc/design/5.4-snapshot-artifact-and-attachment-model.md @@ -122,7 +122,7 @@ lifecycle tiers. | Tier | Restore policy | Scratch | Consumed configuration | Sharing scope | | --- | --- | --- | --- | --- | -| `platform` | Clone | Fresh | Invariants only | Trusted platform build | +| `platform` | Clone | Fresh, or none in the RAM-backed overlay layout | Invariants only | Trusted platform build | | `workload-start` | Clone | Paired | Invariants, image binding, and sandbox | Tenant-scoped warm workload or runtime | | `instance-checkpoint` | Resume | Paired | Invariants, image binding, and sandbox | One live instance with one claimed continuation | @@ -130,7 +130,9 @@ Tier metadata requires sandbox blocks; blockless snapshots do not declare a tier. The saved host-owned tier token must agree with the manifest. Platform lower layers have unbound identities, whereas later tiers bind every consumed layer. The consumed-section set records which configuration the guest had -already consumed at capture. +already consumed at capture. A platform clone receives the image binding and +sandbox sections that it has not consumed through +[platform-clone activation](5.5-restore-protocol.md#559-platform-clone-activation). The platform point removes kernel and guest agent initialization from later launches. A workload-start snapshot also removes runtime initialization, but @@ -183,23 +185,34 @@ declarative reconstruction policy. | Resource | Saved | Reconstructed or supplied on restore | | --- | --- | --- | -| Byte channel | Pending RX and TX bytes | Host endpoint, a fresh generation ID, and an optional restore packet | +| Byte channel | Pending RX and TX bytes | Host endpoint, a fresh generation ID, and a restore packet, which may carry a platform clone's activation record | | Boot console | Queue progress, staged RX, partial TX, endpoint identity, and reconnect policy | An eligible listener at its captured endpoint or an approved fresh endpoint of the same kind, a required client connection, or a supplied inherited terminal | | Control console | Attachment identity, broker policy, and bounded broker state without queued records, host output, or receive credit; never the capability or peer identity | An approved same-user local endpoint, possibly at a fresh location, with a fresh capability; the broker restarts with a fresh instance and epoch | -| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy; published guest listeners and host forwards are rejected | -| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export | -| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | +| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy, or with a new policy for a `platform` snapshot captured with a deny-all policy; otherwise, published guest listeners and host forwards are rejected | +| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export or aggregate | +| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, or substitutes no longer than their recorded lengths for a `platform` snapshot, and, unless the snapshot has no scratch, private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | | Readiness endpoint | Nothing | An optional single-use endpoint for one restore | Attachment resolution happens before any processor starts. A supplied attachment must reproduce the saved identity, except that an eligible listener may replace only its endpoint while keeping its stable ID, attachment -and backend kinds, reconnect policy, required flag, length, and timeout. -Missing privileges, endpoint binding failures, a changed egress policy, -replaced filesystem objects, or a wrong attachment kind fail restore -explicitly. Orchestrators of reusable clones supply fresh private boot-console -and control endpoints, so that independent or concurrent restores never -collide with the terminated source. +and backend kinds, reconnect policy, required flag, length, and timeout, and +that a `platform` snapshot accepts substitute read-only layers and, when +captured with a deny-all policy, a new network policy. Missing privileges, +endpoint binding failures, any other changed egress policy, replaced +filesystem objects, or a wrong attachment kind fail restore explicitly. +Orchestrators of reusable clones supply fresh private boot-console and control +endpoints, so that independent or concurrent restores never collide with the +terminated source. + +For the attachments that a restore supplies, a host may name the identity +that it expects for each read-only layer, filesystem root, and pinned file +child. During attachment resolution, before any processor runs, the VMM +compares each with the object that it opened and fails the restore on a +mismatch, so an object replaced after the host's check never reaches the +guest. The VMM also reports a receipt to its launcher for each such +attachment, with its role and the opened object's identity and actual length. +Receipts identify objects, not contents, so producing them reads no data. ## 5.4.8 Memory Sharing diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index bf04443f0..5c898ad2b 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -43,9 +43,10 @@ Before worker construction, restore: 7. Resolves console, network, filesystem, policy, and read-only layer attachments by stable ID or role. 8. Validates the state, captured memory, selected RAM target, block geometry - and identities, and any paired scratch, and claims a resume-policy - snapshot. -9. Generates the restore entropy and generation ID, and sets the generation + and identities, any attachment identities that the host expects, and any + paired scratch, and claims a resume-policy snapshot. +9. Generates the restore entropy and generation ID, checks the size of any + activation record that the launcher supplied, and sets the generation counter to the captured value plus one. 10. Maps captured memory privately with copy-on-write, adds fresh private backing for selected expansion ranges, and materializes scratch according @@ -201,7 +202,7 @@ every tier and for untiered snapshots. ```text offset size field 0 3 magic "OVR" - 3 1 version = 4 + 3 1 version = 4, or 5 with an activation record 4 1 flags 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 6 1 memory_range_count 0 unless MEMORY_TARGET @@ -212,6 +213,8 @@ offset size field 24 8 utc_ns u64, little-endian, host UTC at selection 32 16*n ranges n x { u64 gpa_start, u64 length }, LE 32+16n 64 entropy + 96+16n 4 activation_length version 5 only: u32, little-endian, 1..1024 +100+16n m activation version 5 only: m = activation_length bytes ``` **Figure 5.7.** Restore packet layout. @@ -227,13 +230,21 @@ offset size field | 4-7 | | Zero; the guest rejects a packet with any of them set | The UTC field is latched when the guest first selects the packet; every other -field is sealed before the first restored processor runs. A packet without -ranges is 96 bytes. The packet is authoritative, and the status bits mirror -it; the acknowledgement flag spares the guest from inferring gating from the -tier and targets. A VM process exposes at most one packet, and reading its -last byte consumes it, so a guest captured again in the same process never -sees an earlier packet. The guest also ignores a packet whose generation -equals its recorded generation. +field, including any activation record, is sealed before the first restored +processor runs. A version-4 packet without ranges is 96 bytes, and version 5 +adds 4 bytes plus the record. The packet is authoritative, and the status bits +mirror it; the acknowledgement flag spares the guest from inferring gating +from the tier and targets. A VM process exposes at most one packet, and +reading its last byte consumes it, so a guest captured again in the same +process never sees an earlier packet. The guest also ignores a packet whose +generation equals its recorded generation. + +The activation record is opaque to the VMM. Only a restore of a `platform` +snapshot accepts one, of 1 to 1,024 bytes, and the VMM never stores it in a +snapshot. The VMM writes version 5 only when a record is present, so a guest +captured before version 5 restores unchanged. The guest agents and the host +own the record's schema, as +[Platform-Clone Activation](#559-platform-clone-activation) describes. ## 5.5.6 Generations @@ -259,7 +270,7 @@ runs before the acknowledgement: a restore. 2. Bracket the packet selection with realtime clock reads, read the packet four bytes per access, and validate its magic, version, reserved bits, - counts, generation, and generation ID. + counts, any activation record's length, generation, and generation ID. 3. Set the wall clock: derive the offset and its uncertainty from the bracket, take up to two further time samples if the uncertainty exceeds 1 ms, step the clock by the offset, and set the frequency correction to the @@ -273,6 +284,11 @@ runs before the acknowledgement: restore the detector settings, and restart the wall-clock discipline at its fast cadence. +A platform clone whose packet carries an activation record also validates the +record before the acknowledgement and activates after it, on the readiness +path rather than at idle priority, as +[Platform-Clone Activation](#559-platform-clone-activation) describes. + A failure in the time ABI's part of the repair, which validates the packet, steps the wall clock, and restores the stall-detector settings, emits a violation event and powers off the guest with status 195, and a failed rerun @@ -303,8 +319,9 @@ To prevent this, every restore delivers fresh entropy, which a clone of a 3. For a clone, the guest agents credit all 64 bytes to the kernel's entropy pool and force the kernel's random-number generator to reseed, so all later kernel randomness depends on the fresh bytes. -4. They refresh the workload's machine ID, which they derive from the fresh - bytes, and its UTS namespace. +4. When the adapter selects a generated machine ID, they generate a new one + with `getrandom` after the reseed; an image's own machine ID is left + unchanged. They also refresh the workload's UTS namespace. 5. The workload runtime's post-restore hook resets the runtime's own random-number state, which a kernel reseed cannot reach. 6. The guest agents acknowledge the restore and only then thaw scratch and @@ -318,8 +335,8 @@ As Table 5.7 shows, the refresh depends on the restore path. | Restore path | Guest random-number generator | Machine identity | | --- | --- | --- | -| `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it | -| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Refreshed, as for a clone | +| `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Generated machine ID and hostname refreshed; an image machine ID remains, and the runtime hook runs if present; `workload-start` requires the hook | +| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Generated machine ID and hostname refreshed, as for a clone | | `instance-checkpoint` tier (resume) | Not reseeded, by design; only the generation ID changes | Preserved | | Untiered, with nothing to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved | @@ -337,3 +354,56 @@ run, and gating host input does not hold it back. The snapshot's caller must therefore provide the execution barrier, for example by capturing before any workload starts, and, where the guest does not reseed, repair the kernel's and the runtime's random state with the entropy handed to it. + +## 5.5.9 Platform-Clone Activation + +A +[managed layered root](4.1-guest-filesystem-and-image-model.md#414-managed-layered-root) +reads its activation configuration from boot tokens on a cold boot and from +the activation record of the [restore packet](#555-restore-packet) in a +platform clone, because a restore never extends the captured command line. +Both deliveries use the same validation rules and report the same identity in +readiness. A launch that declares a `platform` capture carries no activation +configuration, and its guest agents capture before they assemble the root. + +The record is non-secret and small, because the guest reads it four bytes per +port exit. It carries: + +- The identity of the configuration, which readiness reports. +- The filesystem UUID or disk GUID that the substituted `distro` image must + have. +- The upper-layer settings that the template did not capture. +- The form of any export that the restore bound to the dormant filesystem + slot, and the number of host mappings. +- Whether to require an existing account or use numeric IDs. +- For a template captured with a deny-all network policy, the DNS server, the + ingress mode, and the guest's settings for any gateway proxy endpoint that + the new policy permits. + +Secrets, per-execution environments, and nonempty host mapping tables never +enter the record. They use the control channel after readiness, as they do +after a cold boot. + +Activation proceeds as follows: + +1. During [guest repair](#557-guest-repair), the guest agents read the record + with the packet and validate it before they acknowledge the restore. +2. After the acknowledgement, they flush the cached buffers of the + substituted `distro` image and assemble the root as steps 3 through 6 of + [Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) + describe, checking the image against the record. +3. They mount any export that the restore bound to the dormant filesystem slot + and apply the record's network settings. +4. They establish the control path with the restore's fresh capability, + instance, and epoch, and report readiness with the configuration's + identity. + +Readiness completes activation. The host then checks the reported identity +and delivers any host mapping table, which the guest agents mount as after a +cold boot, refusing executions until every announced mapping is mounted. + +The guest agents of a managed layered root captured at the `platform` tier +require the record and fail repair without one. An activation failure writes +an error line and powers off the guest with status 1, so a partially activated +clone never becomes ready. A platform clone of any other guest needs no record +and receives no configuration beyond its captured command line. diff --git a/doc/design/5.6-shutdown-and-failure-semantics.md b/doc/design/5.6-shutdown-and-failure-semantics.md index 1873d6e63..170e34378 100644 --- a/doc/design/5.6-shutdown-and-failure-semantics.md +++ b/doc/design/5.6-shutdown-and-failure-semantics.md @@ -49,7 +49,8 @@ describes. Failures are classified by whether execution can safely continue: -- Invalid configuration, artifacts, or attachments fail before guest entry. +- Invalid configuration, artifacts, or attachments fail before guest entry, + except an activation record, which the guest agents validate during repair. - Time-contract violations reject a cold boot, capture, or restore with a stable error code and never fall back. - A rejected capture preflight may return the still-consistent source to @@ -59,6 +60,9 @@ Failures are classified by whether execution can safely continue: - A failure after snapshot publication never resumes the source. - A post-restore repair failure before the acknowledgement, or a missing acknowledgement, leaves input gated and terminates the restored process. +- A platform-clone activation failure after the acknowledgement powers off + the guest before it reports readiness, so no workload runs in a partially + activated clone. - Malformed saved state or failed attachment reconstruction prevents any restored virtual processor from running.