Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 7 additions & 2 deletions crates/openshell-driver-docker/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-otel = { path = "../openshell-otel" }

opentelemetry = { workspace = true }
Expand All @@ -38,15 +39,19 @@ miette = { workspace = true }
toml = { workspace = true }
tower-http = { workspace = true }
http = { workspace = true }
rand = { workspace = true }
sha2 = { workspace = true }
rustix = { workspace = true }
libc = "0.2"
tar = "0.4"
tempfile = "3"

[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
opentelemetry = { workspace = true }
opentelemetry_sdk = { workspace = true, features = ["testing"] }
prost-types = { workspace = true }
tar = "0.4"
temp-env = "0.3"
tempfile = "3"
tracing-subscriber = { workspace = true }

[lints]
Expand Down
97 changes: 65 additions & 32 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,35 @@ no candidate responds.

## Runtime Model

The gateway runs as a host process. The Docker driver creates one container per
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
supervisor then creates the nested sandbox namespace for the agent process.
The gateway and Docker driver run on the host. For each sandbox, the driver
starts `openshell-sandbox --mode=control` beside the gateway and
`openshell-sandbox --mode=boundary` as PID 1 in the workload container. The two
roles use the shared RFC 0012 protocol over a private bind-mounted Unix socket.

Control owns the gateway session, policy, SSH relay, and network proxy. The
boundary owns the process tree, exec/signal/wait/PTY operations, loopback port
forwarding, procfs binary identity, and the workload network namespace. The
boundary receives no gateway JWT, client TLS private key, or policy authority.

The driver bind-mounts the boundary socket from a short, digest-keyed directory
under `$XDG_RUNTIME_DIR/openshell/docker-boundary-runtime`, falling back to the
XDG state directory when no runtime directory exists. The driver rejects
symlinked, incorrectly owned, or overly permissive pre-existing components and
checks the final path against the Unix-socket length limit. The full protected
topology and gateway material remain in the normal driver state directory, and
sandbox cleanup removes both locations.

## Stop and Start

Stop stops the managed container without removing it. Docker retains the
container writable layer, attached volumes, labels, token material, and restart
policy. Start starts that same container, so files in the resolved OCI
workspace remain available. A durably stopped sandbox is excluded from
gateway startup recovery and stays stopped across gateway restarts. Delete
continues to force-remove the container and clean up driver-owned material.
Stop terminates host control and stops the managed container without removing
it. Docker retains the container writable layer, attached volumes, labels, and
the protected driver topology. Stop removes the transient boundary and SSH
socket names; Start validates and removes any safe stale sockets before starting
that same container and recreating host control. Files in the resolved OCI
workspace remain available. A
durably stopped sandbox is excluded from gateway startup recovery and stays
stopped across gateway restarts. Delete force-removes the container and the
driver-owned boundary directory.
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
phase requires running compute without changing that persisted intent. On
startup, the gateway sends an idempotent `StartSandbox` request for the same
Expand All @@ -50,9 +67,9 @@ remain excluded.
Before creating the container, the driver inspects the final sandbox image and
captures its immutable image ID, raw OCI `Config.User`, and OCI
`Config.WorkingDir`. Container creation uses that image ID, preventing a
mutable tag from changing between inspection and launch. The supervisor runs as
root, resolves omitted policy identity fields from the image declaration, and
drops only agent children to the resulting identity. Named OCI components
mutable tag from changing between inspection and launch. The boundary runs as
root, resolves omitted policy identity fields from the image declaration inside
the image filesystem, and drops only agent children to the resulting identity. Named OCI components
remain names after validation; a missing group is filled with the user's
numeric primary GID. Explicit `process.run_as_user` and
`process.run_as_group` values take precedence independently.
Expand Down Expand Up @@ -83,14 +100,12 @@ are rejected, as are paths that overlap concrete OpenShell control resources.
The workspace is the child cwd and `HOME`. The supervisor starts from `/`, then
reports an invalid workdir as a readiness failure.

Docker containers join an OpenShell-managed bridge network. The driver injects
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
`host-gateway` alias. The driver requests a separate IPv4 loopback callback
listener when the primary listener does not already cover it. On native Linux
Docker, the gateway also binds the bridge gateway IP so containers can call
back to the host process.
Boundary containers use Docker `network_mode=none`. The nested agent network
namespace can reach only the boundary's mediation listener. Each accepted TCP
stream and its procfs-derived executable identity cross the private Unix socket
to host control, which applies policy before dialing the destination. Host
control reaches the gateway directly; the existing managed bridge listener is
retained for compatibility with legacy containers created before this split.

## Container Contract

Expand All @@ -100,13 +115,15 @@ contract:
| Setting | Purpose |
|---|---|
| `user = "0"` | The supervisor needs root inside the container to prepare namespaces, mounts, Landlock, and seccomp. |
| `network_mode = openshell` | Places the supervisor on the managed Docker bridge network. |
| `network_mode = none` | Removes direct container egress; the nested workload namespace can emit only through the authenticated boundary stream. |
| `cap_add` | Grants supervisor-only capabilities required for namespace setup and process inspection. |
| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. |
| `restart_policy = no` | A canonical main-process exit remains terminal and is not silently restarted by Docker. |
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
| `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. |
| Read-only supervisor/runtime mounts | Supplies the boundary binary and trusted `ip`/`nft` helper runtime without trusting workload-image tools. |
| Private boundary bind mount | Carries the protected boundary config and Unix control socket. The directory is host-owned mode `0700`; workload children cannot access it. |
| `policy-dns-transparent-tcp` capability | Declares that the shared boundary can own namespace-local DNS/TCP capture while host control retains authorization, pinned dialing, relaying, and OCSF decisions. The marker is not exposed to the workload environment. |

The agent child process does not retain these supervisor privileges.

Expand All @@ -127,8 +144,9 @@ mount types:

Host bind mounts are disabled by default because they expose gateway host
paths to sandbox requests. Image mounts are not part of the Docker
driver-config schema. The driver still uses internal bind mounts for
OpenShell-owned supervisor, token, and TLS material.
driver-config schema. The driver still uses internal bind mounts for the
OpenShell-owned boundary binary, trusted helper runtime, and private socket
directory. Gateway token and TLS material remain on the host.

Docker `bind` mounts accept `source`, `target`, optional `read_only`, and an
optional `selinux_label` of `shared` (applies `:z`) or `private` (applies
Expand All @@ -154,8 +172,9 @@ openshell sandbox create \

## Supervisor Binary Resolution

The Docker driver bind-mounts a host-side Linux `openshell-sandbox` binary into
each sandbox container. Resolution order is:
The Docker driver bind-mounts a host-side Linux `openshell-sandbox` boundary
binary and trusted helper runtime into each sandbox container. Resolution order
for the Linux artifact is:

1. `supervisor_bin` in `[openshell.drivers.docker]`.
2. `supervisor_image` in `[openshell.drivers.docker]`, extracting
Expand All @@ -164,22 +183,32 @@ each sandbox container. Resolution order is:
4. A local Linux cargo target build for the Docker daemon architecture.
5. The release-matched default supervisor image, extracting `/openshell-sandbox`.

When a selected binary has no valid sibling `openshell-runtime`, the driver
extracts `/openshell-runtime` from the configured or release-matched supervisor
image. Host control uses the native `openshell-sandbox` beside the gateway (or a
local native cargo build); Linux hosts can reuse the boundary binary.

Release and Docker-image gateway builds bake the matching supervisor image tag
into the binary at compile time. The default Docker supervisor image is not
`:latest` unless a custom build explicitly sets that tag.

## Callback and TLS

`OPENSHELL_ENDPOINT` is injected from the gateway's configured gRPC endpoint.
`OPENSHELL_ENDPOINT` is injected only into host control from the gateway's
configured gRPC endpoint.
When no endpoint is configured, the driver uses
`host.openshell.internal:<gateway-port>` with the appropriate HTTP or HTTPS
scheme. Set `host_gateway_ip` only when the host has an explicit, locally
assigned address that containers should use for callbacks; package-managed
macOS gateways should leave it unset.
macOS gateways should leave it unset. The current boundary topology requires
the Docker daemon and gateway to share bind-mounted Unix socket inodes. The
driver fails during initialization for Docker Desktop, Colima, Lima, Rancher
Desktop, OrbStack, and other detected VM-backed daemons instead of starting a
workload whose host control channel cannot become operational.

For HTTPS endpoints, the server certificate must include the endpoint host as a
subject alternative name. Docker sandboxes also need the client TLS bundle
mounted into the container and exposed with:
subject alternative name. The client TLS bundle is exposed only to host control
with:

- `OPENSHELL_TLS_CA`
- `OPENSHELL_TLS_CERT`
Expand All @@ -189,8 +218,12 @@ HTTP endpoints reject TLS material because the supervisor would not use it.

## Environment Ownership

The driver merges template environment and sandbox spec environment first, then
overwrites security-critical keys:
The driver writes template and sandbox environment into the protected boundary
config. Boundary mode exposes that map only to workload children. Host control
starts with an empty inherited environment and receives only driver-owned
values. It also arms Linux `PDEATHSIG` with a parent-PID race check so a killed
gateway cannot leave an orphan host-control process. Host control owns these
security-critical keys:

- `OPENSHELL_ENDPOINT`
- `OPENSHELL_SANDBOX_ID`
Expand Down
107 changes: 107 additions & 0 deletions crates/openshell-driver-docker/src/isolation.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

//! Docker provisioning for the shared authenticated boundary protocol.
//!
//! Docker owns only the container/socket topology and immutable OCI resource
//! claims. Lifecycle, process, network, identity, and wire behavior live in
//! `openshell-isolation-interface` and `openshell-sandbox`.

use std::collections::{BTreeMap, HashMap};
use std::net::IpAddr;
use std::path::PathBuf;

use openshell_isolation_interface::boundary_protocol::{
BOUNDARY_PROTOCOL_VERSION, BoundaryAgentIdentity, BoundaryConfig, BoundaryListener,
BoundaryTopology, BoundaryTransport,
};

/// Driver-owned inputs that bind one Docker container to one boundary.
pub struct DockerBoundarySpec {
pub boundary_id: String,
pub bootstrap_token: String,
pub container_id: String,
pub image_identity: String,
pub listener_socket: PathBuf,
pub control_socket: PathBuf,
pub host_gateway_ip: Option<IpAddr>,
pub oci_user: String,
pub trusted_runtime_root: PathBuf,
pub child_env: HashMap<String, String>,
}

/// Protected container config and matching host descriptor.
pub struct DockerBoundaryProvisioning {
pub boundary_config: BoundaryConfig,
pub topology: BoundaryTopology,
}

impl DockerBoundarySpec {
/// Produce both sides of the common protocol from the same immutable
/// Docker coordinates so attach cannot bind a different container.
#[must_use]
pub fn provision(self) -> DockerBoundaryProvisioning {
let resource_claims = BTreeMap::from([
("docker.container_id".to_string(), self.container_id),
("docker.image_identity".to_string(), self.image_identity),
]);
DockerBoundaryProvisioning {
boundary_config: BoundaryConfig {
protocol_version: BOUNDARY_PROTOCOL_VERSION,
boundary_id: self.boundary_id.clone(),
bootstrap_token: self.bootstrap_token.clone(),
listener: BoundaryListener::Unix {
socket_path: self.listener_socket,
},
resource_claims: resource_claims.clone(),
agent_identity: BoundaryAgentIdentity::OciUser {
declaration: self.oci_user,
},
protect_config_file: true,
trusted_runtime_root: self.trusted_runtime_root,
child_env: self.child_env,
},
topology: BoundaryTopology {
protocol_version: BOUNDARY_PROTOCOL_VERSION,
boundary_id: self.boundary_id,
transport: BoundaryTransport::Unix {
socket_path: self.control_socket,
},
host_gateway_ip: self.host_gateway_ip,
resource_claims,
bootstrap_token: self.bootstrap_token,
},
}
}
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn provisioning_binds_container_and_image_claims() {
let provisioned = DockerBoundarySpec {
boundary_id: "sandbox-1".to_string(),
bootstrap_token: "a".repeat(64),
container_id: "sha256:container".to_string(),
image_identity: "sha256:image".to_string(),
listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"),
control_socket: PathBuf::from("/host/control.sock"),
host_gateway_ip: Some(IpAddr::from([127, 0, 0, 1])),
oci_user: "1000:1000".to_string(),
trusted_runtime_root: PathBuf::from("/opt/openshell/bin/openshell-runtime"),
child_env: HashMap::new(),
}
.provision();

assert_eq!(
provisioned.boundary_config.resource_claims,
provisioned.topology.resource_claims
);
assert_eq!(
provisioned.topology.resource_claims["docker.container_id"],
"sha256:container"
);
}
}
Loading
Loading