Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 10 additions & 34 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,28 +69,12 @@ mise run helm:skaffold:dev
mise run helm:skaffold:run
```

**Supervisor sidecar topology** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar
```

**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar-mtls
```

Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which
must be at least `1000` and distinct from the workload UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. The default Skaffold values export gateway and Kubernetes-driver traces to
the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the
Optional Add-ons section below.
Both commands build the gateway and trusted runtime image and deploy the
OpenShell Helm chart. Kubernetes sandboxes always use the proxy-pod topology:
a capability-free sandbox Pod behind an empty-egress NetworkPolicy and a
separate capability-free supervisor Deployment. The default Skaffold values
export gateway and Kubernetes-driver traces to the collector service installed
by `helm:k3s:create`. Envoy Gateway is opt-in; see the Optional Add-ons section.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
the unified local forwarding task:
Expand All @@ -102,8 +86,8 @@ mise run helm:k3s:forward
The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to
`http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it
also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues
with the collector ports only. A successful plaintext `helm:skaffold:run` or
`helm:skaffold:run:sidecar` registers the gateway under the worktree-specific
with the collector ports only. A successful plaintext `helm:skaffold:run`
registers the gateway under the worktree-specific
k3d cluster name and selects it as the active gateway. Keep the forwarding
task running while using those endpoints.

Expand Down Expand Up @@ -134,8 +118,7 @@ create the Secret named `openshell-ha-pg` with a `uri` key, then run
### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
plaintext by default. To test sidecar topology with TLS enabled, use
`mise run helm:skaffold:run:sidecar-mtls`.
plaintext by default. Remove that override to test gateway TLS/mTLS locally.

| Mode | `server.disableTls` | Gateway scheme |
|------|---------------------|----------------|
Expand Down Expand Up @@ -188,12 +171,6 @@ openshell sandbox list --gateway-endpoint https://localhost:8090
mise run helm:skaffold:delete
```

For a sidecar-profile deployment:

```bash
mise run helm:skaffold:delete:sidecar
```

### Delete the cluster entirely

```bash
Expand Down Expand Up @@ -259,7 +236,7 @@ SPIFFE JWT-SVIDs for dynamic provider token grants:
`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
OpenShell mounts the SPIFFE CSI Workload API socket at
`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
`/spiffe-workload-api/spire-agent.sock` only into supervisor Pods for provider token
grants. Supervisor-to-gateway authentication remains on the Kubernetes
ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected
Kubernetes compute driver validates the projected token before the gateway
Expand Down Expand Up @@ -320,7 +297,6 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
Expand Down
15 changes: 7 additions & 8 deletions .github/workflows/branch-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -370,16 +370,12 @@ jobs:
include:
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
topology: combined
extra_helm_values: ""
topology: proxy-pod
extra_helm_values: deploy/helm/openshell/ci/values-proxy-pod.yaml
- agent_sandbox_api: v1alpha1
agent_sandbox_version: v0.4.6
topology: combined
extra_helm_values: ""
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
topology: sidecar
extra_helm_values: deploy/helm/openshell/ci/values-sidecar.yaml
topology: proxy-pod
extra_helm_values: deploy/helm/openshell/ci/values-proxy-pod.yaml
permissions:
actions: read
contents: read
Expand All @@ -404,6 +400,7 @@ jobs:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (workspace managed mode)
e2e-task: e2e:kubernetes:workspace-managed
extra-helm-values: deploy/helm/openshell/ci/values-proxy-pod.yaml
conformance-artifact-prefix: openshell-conformance

kubernetes-external-driver-e2e:
Expand All @@ -421,6 +418,7 @@ jobs:
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
external-driver-binary: openshell-driver-kubernetes
cluster-images: supervisor
extra-helm-values: deploy/helm/openshell/ci/values-proxy-pod.yaml
conformance-artifact-prefix: openshell-conformance

kubernetes-workspace-operator-e2e:
Expand All @@ -435,6 +433,7 @@ jobs:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (workspace operator mode)
e2e-task: e2e:kubernetes:workspace-operator
extra-helm-values: deploy/helm/openshell/ci/values-proxy-pod.yaml
conformance-artifact-prefix: openshell-conformance

kubernetes-ha-e2e:
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
| `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner |
| `crates/openshell-conformance-cli/` | Conformance CLI | Distributable `list` and `run` entrypoint for gateway conformance |
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
| `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
| `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
| `crates/openshell-binary-identity/` | Binary identity | Shared trusted procfs executable identity resolution for isolation backends |
| `crates/openshell-isolation-interface/` | Isolation backend interface | RFC 0012 `IsolationBackend` trait + types; the supervisor-facing runtime contract for the boundary |
| `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints |
Expand All @@ -59,7 +60,6 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
| `crates/openshell-driver-db-credstore/` | Database credential driver | In-process `CredentialDriver` backend for gateway database credential storage |
| `crates/openshell-driver-kubernetes/` | Kubernetes compute driver | In-process `ComputeDriver` backend for K8s sandbox pods |
| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers |
| `crates/openshell-driver-mxc/` | MXC compute driver | Windows in-process `ComputeDriver` backend for MXC sandbox execution |
| `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers |
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) |
| `crates/openshell-driver-mxc/` | Microsoft MXC compute driver | In-process Windows AppContainer and isolation-session compute backend |
Expand Down
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 0 additions & 8 deletions crates/openshell-core/src/container_paths.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ pub const OPT_ROOT: &str = "/opt/openshell";
pub const ETC_ROOT: &str = "/etc/openshell";
pub const TLS_ROOT: &str = "/etc/openshell-tls";
pub const RUN_ROOT: &str = "/run/openshell";
pub const SIDECAR_RUN_ROOT: &str = "/run/openshell-sidecar";
pub const NETNS_MOUNT_ROOT: &str = "/run/netns";
pub const NETNS_IPROUTE2_ROOT: &str = "/var/run/netns";

Expand All @@ -34,7 +33,6 @@ pub const CONTROL_ROOTS: &[&str] = &[
ETC_ROOT,
TLS_ROOT,
RUN_ROOT,
SIDECAR_RUN_ROOT,
NETNS_MOUNT_ROOT,
// The supervisor currently uses the conventional iproute2 spelling.
NETNS_IPROUTE2_ROOT,
Expand All @@ -52,9 +50,6 @@ pub const CONTAINER_POLICY_PATH: &str = "/etc/openshell/policy.yaml";
pub const POLICY_ADVISOR_SKILL_PATH: &str = "/etc/openshell/skills/policy_advisor.md";

pub const SSH_SOCKET_PATH: &str = "/run/openshell/ssh.sock";
pub const SIDECAR_CONTROL_SOCKET: &str = "/run/openshell-sidecar/control.sock";
pub const SIDECAR_TLS_DIR: &str = "/etc/openshell-tls/proxy";
pub const SIDECAR_CLIENT_TLS_DIR: &str = "/etc/openshell-tls/proxy/client";
pub const CLIENT_TLS_DIR: &str = "/etc/openshell-tls/client";
pub const SUPERVISOR_CA_CERT_PATH: &str = "/etc/openshell-tls/openshell-ca.pem";
pub const SUPERVISOR_CA_BUNDLE_PATH: &str = "/etc/openshell-tls/ca-bundle.pem";
Expand Down Expand Up @@ -92,9 +87,6 @@ mod tests {
CONTAINER_POLICY_PATH,
POLICY_ADVISOR_SKILL_PATH,
SSH_SOCKET_PATH,
SIDECAR_CONTROL_SOCKET,
SIDECAR_TLS_DIR,
SIDECAR_CLIENT_TLS_DIR,
CLIENT_TLS_DIR,
SUPERVISOR_CA_CERT_PATH,
SUPERVISOR_CA_BUNDLE_PATH,
Expand Down
1 change: 0 additions & 1 deletion crates/openshell-core/src/driver_mounts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,6 @@ mod tests {
"/etc/openshell-tls",
"/run",
"/run/openshell/cache",
"/run/openshell-sidecar/control.sock",
"/run/netns/project",
"/var/run/netns/project",
] {
Expand Down
4 changes: 2 additions & 2 deletions crates/openshell-core/src/proposals.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
//! Shared state controlling agent-driven policy proposals.
//!
//! Initialised once during sandbox start from the `agent_policy_proposals_enabled`
//! setting and updated by the policy poll loop or authoritative sidecar control
//! setting and updated by the policy poll loop or authoritative supervisor
//! when the setting changes. Read by the `policy.local` route handler and by
//! the skills installer to gate the agent-controlled mutation surface.

Expand All @@ -15,7 +15,7 @@ use std::sync::atomic::{AtomicBool, Ordering};
///
/// Clones point at the same atomic value, so the sandbox orchestrator can pass
/// this into the process and network supervisors and then update it from the
/// settings poll loop or sidecar control.
/// settings poll loop or supervisor control.
#[derive(Clone, Debug)]
pub struct AgentProposals {
enabled: Arc<AtomicBool>,
Expand Down
4 changes: 2 additions & 2 deletions crates/openshell-core/src/provider_credentials.rs
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,8 @@ impl ProviderCredentialState {
/// Build a static provider state from an already-prepared child
/// environment snapshot.
///
/// Kubernetes sidecar topology uses this in the process-only supervisor:
/// the network sidecar owns provider credential resolvers and sends the
/// The Kubernetes proxy-pod topology uses this in the sandbox process:
/// the supervisor owns provider credential resolvers and sends the
/// workload-facing env map over a local control channel. The process leaf
/// must inject that map into child processes without re-placeholderizing it
/// or holding the gateway-side resolver material.
Expand Down
21 changes: 0 additions & 21 deletions crates/openshell-core/src/sandbox_env.rs
Original file line number Diff line number Diff line change
Expand Up @@ -131,20 +131,13 @@ impl MainProcessConfig {
/// Deployment-controlled telemetry toggle propagated to the sandbox supervisor.
pub const TELEMETRY_ENABLED: &str = "OPENSHELL_TELEMETRY_ENABLED";

/// Supervisor pod/runtime topology. Kubernetes sidecar mode sets this to
/// `"sidecar"`; the default combined supervisor path omits it.
pub const SUPERVISOR_TOPOLOGY: &str = "OPENSHELL_SUPERVISOR_TOPOLOGY";

/// The isolation backend admitted by the deployment configuration (RFC 0012).
///
/// Delivered on a channel separate from the topology descriptor so descriptor
/// verification against the admitted backend is not self-referential. Required
/// whenever a topology descriptor is supplied.
pub const ADMITTED_ISOLATION_BACKEND: &str = "OPENSHELL_ADMITTED_ISOLATION_BACKEND";

/// Network enforcement backend selected by the compute driver.
pub const NETWORK_ENFORCEMENT_MODE: &str = "OPENSHELL_NETWORK_ENFORCEMENT_MODE";

/// Comma-separated runtime networking capabilities supplied by the compute
/// driver. Capabilities describe substrate the shared supervisor may activate;
/// they never move policy evaluation into the driver.
Expand All @@ -153,20 +146,6 @@ pub const NETWORK_RUNTIME_CAPABILITIES: &str = "OPENSHELL_NETWORK_RUNTIME_CAPABI
/// Driver capability for policy-gated DNS and transparent TCP interception.
pub const POLICY_DNS_TRANSPARENT_TCP_CAPABILITY: &str = "policy-dns-transparent-tcp";

/// Whether network policy evaluation must bind requests to the peer binary.
///
/// The default when unset is `"required"`. Kubernetes sidecar experiments may
/// set this to `"relaxed"` to enforce endpoint and L7 policy without per-binary
/// `/proc` identity binding.
pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY";

/// Unix socket used by Kubernetes sidecar topology for local coordination.
///
/// The network sidecar owns gateway credentials and serves policy/provider
/// state over this socket instead of exposing gateway credentials to the agent
/// container.
pub const SIDECAR_CONTROL_SOCKET: &str = "OPENSHELL_SIDECAR_CONTROL_SOCKET";

/// Optional TLS server name override used when connecting to the gateway.
pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";

Expand Down
3 changes: 3 additions & 0 deletions crates/openshell-driver-kubernetes/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-otel = { path = "../openshell-otel" }
openshell-policy = { path = "../openshell-policy" }

Expand All @@ -37,6 +38,8 @@ tracing-subscriber = { workspace = true }
thiserror = { workspace = true }
miette = { workspace = true }
notify = "8"
rand = { workspace = true }
rcgen = { workspace = true }

[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
Expand Down
Loading
Loading