diff --git a/TESTING.md b/TESTING.md index 199fa35416..14ec3c2055 100644 --- a/TESTING.md +++ b/TESTING.md @@ -232,6 +232,128 @@ Secrets and Vault storage backends one at a time: mise run e2e:kubernetes:credential-drivers ``` +### Kubernetes E2E (`e2e/rust/e2e-kubernetes.sh`) + +Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster +via Helm and run the Rust e2e suite against it. On vanilla Kubernetes the harness +reaches the gateway through `kubectl port-forward`; on OpenShift it instead uses a +passthrough Route secured with mandatory mTLS (see the OpenShift note below). + +Run with an ephemeral k3d cluster (macOS; created and torn down automatically): + +```shell +mise run e2e:kubernetes +``` + +Target an existing cluster (kind, k3d, or OpenShift): + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=my-context mise run e2e:kubernetes +``` + +Scope to a single test for local debugging: + +```shell +OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes +``` + +**OpenShift**: when the target cluster exposes the `route.openshift.io` API +group, the harness automatically applies SCC-compatible Helm overrides, grants +the required SCCs (`privileged` to `openshell-sandbox`, and `anyuid` to the +PostgreSQL fixture for DB scenarios), and drives the gateway through a +passthrough Route with mandatory mTLS instead of port-forward. No extra flags are +needed, but `oc` must be installed and authenticated against the target cluster +with permission to modify SCC bindings (`oc adm policy add-scc-to-user`) — the +harness exits early if `oc` is missing. The SCC grants and extracted client +mTLS material are removed during cleanup, including on failure or interrupt. + +On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely +on the sandbox-side `host.openshell.internal` alias reaching the machine running +the tests, which is unreachable from pods on a remote cluster, so they fail. +Left enabled, the `host_gateway_alias` suite fails because +`host.openshell.internal` does not resolve inside the pod, so the gateway +SSRF-denies the request (`DNS resolution failed` / `ssrf_denied`) — a networking +property of remote pods, not a gateway or transport fault. Override +`OPENSHELL_E2E_KUBERNETES_FEATURES` to exclude it: + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \ + OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \ + mise run e2e:kubernetes +``` + +On an existing cluster the harness builds the CLI from your branch but pulls the +**published** gateway/supervisor image (default tag `latest`). The CLI and the +image can therefore be different versions. If tests fail because of this version +difference — for example, sandbox tests fail with `Pod exists with phase: Failed` +or connect-based tests stall because the deployed image predates a feature your +branch CLI needs — set `IMAGE_TAG` to an image that matches your branch. + +The `latest` tag lags to the last semver release, so it is often older than +`main`. Two better choices: + +- `IMAGE_TAG=dev` — a floating tag that tracks the latest `main` build. Good for + an ad-hoc run when your branch is close to `main` HEAD. Because it floats, two + runs on different days can pull different images, so it is not reproducible. +- **Pin the exact commit your branch is based on** — deterministic and immune to + a floating tag moving. Published tags are the full 40-char git SHA (semver tags + without a `v` prefix also exist but only for released versions): + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \ + OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \ + IMAGE_TAG=$(git rev-parse "$(git merge-base HEAD upstream/main)") \ + mise run e2e:kubernetes +``` + +To pin a specific released version, use its semver tag without a `v` prefix +(`0.0.115`, not `v0.0.115`): + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \ + OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \ + IMAGE_TAG=0.0.115 \ + mise run e2e:kubernetes +``` + +A semver tag matches a released commit, which may be behind `main`; if your +branch CLI needs a newer feature, pin the SHA of your branch's base instead. + +Confirm a tag exists before relying on it (set `TAG` to the tag you plan to use): + +```shell +TAG=0.0.115 +skopeo inspect "docker://ghcr.io/nvidia/openshell/gateway:${TAG}" +``` + +`IMAGE_TAG` sets only the gateway/supervisor image; the CLI under test is always +built from your branch. To validate against images from your exact commit +instead, build and push them and point `OPENSHELL_REGISTRY`/`IMAGE_TAG` at them. + +Available task variants: + +| Task | Purpose | +|---|---| +| `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway | +| `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) | +| `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay | +| `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage | +| `e2e:kubernetes:workspace-managed` | Managed workspace mode (auto-created namespaces) | +| `e2e:kubernetes:workspace-operator` | Operator workspace mode (pre-provisioned namespaces) | +| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility | +| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar | + +Kubernetes e2e environment variables: + +| Variable | Purpose | +|---|---| +| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for an existing cluster (skips k3d creation) | +| `OPENSHELL_E2E_KUBE_TEST` | Scope to a single test (e.g. `smoke`) | +| `OPENSHELL_E2E_KUBE_EXTRA_VALUES` | Colon-separated additional Helm values files | +| `OPENSHELL_E2E_KUBERNETES_FEATURES` | Cargo feature flags (default: `e2e,e2e-host-gateway,e2e-kubernetes`) | +| `IMAGE_TAG` | Gateway/supervisor image tag (default: `latest` for existing clusters) | +| `OPENSHELL_REGISTRY` | Image registry prefix (default: `ghcr.io/nvidia/openshell`) | + Run a single test directly with cargo: ```shell @@ -263,3 +385,5 @@ The harness (`e2e/rust/src/harness/`) provides: | `OPENSHELL_GATEWAY_ENDPOINT` | Run E2E tests against an existing plaintext HTTP gateway endpoint | | `OPENSHELL_E2E_DRIVER` | Driver name exported by the e2e gateway wrapper (`docker`, `podman`, or `vm`) | | `OPENSHELL_E2E_CREDENTIAL_DRIVERS` | Enables the Kubernetes credential-driver fixture path in `e2e/with-kube-gateway.sh` | +| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for Kubernetes e2e (skips ephemeral k3d) | +| `OPENSHELL_E2E_KUBE_TEST` | Scope Kubernetes e2e to a single test by name | diff --git a/deploy/helm/openshell/ci/values-openshift-e2e.yaml b/deploy/helm/openshell/ci/values-openshift-e2e.yaml new file mode 100644 index 0000000000..d2aaf0fac8 --- /dev/null +++ b/deploy/helm/openshell/ci/values-openshift-e2e.yaml @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OpenShift overlay for the Kubernetes e2e harness. +# +# Bundles the OpenShift-specific settings the harness needs: the Route/mTLS +# transport, and an image pull policy that avoids stale cached images. +# +# Route/mTLS transport: on OpenShift, `kubectl port-forward` stalls the SSH-relay +# `sandbox connect` path (round-trip-heavy SSH over SPDY), so the harness drives +# the gateway through a passthrough OpenShift Route with mTLS instead. This +# overlay turns TLS back on (values-skaffold.yaml disables it), enables the Route, +# and promotes the cert-verified caller to a dev principal. +# +# Image pull policy: force `Always` so runs against the `latest` upstream image +# actually use it, instead of a stale copy cached on the cluster nodes. +# +# Layered by e2e/with-kube-gateway.sh AFTER ci/values-skaffold.yaml and +# ci/values-openshift-scc.yaml when an OpenShift cluster is detected. The harness +# supplies `openshiftRoute.host` and `pkiInitJob.serverDnsNames[0]` via --set at +# install time (both are the cluster-derived Route hostname). +# +# Security: this is NOT an open gateway. `server.tls.clientCaSecretName` defaults +# to `openshell-server-client-ca` and there is no OIDC, so `require_client_auth` +# is true and mTLS is MANDATORY at the TLS handshake — a caller with only the +# Route URL and no client certificate is rejected before any RPC. The passthrough +# Route terminates TLS at the gateway pod, so this holds end-to-end. +# `allowUnauthenticatedUsers` only promotes the already cert-verified caller to a +# dev principal at the app layer (mtls_auth is unsupported with the Kubernetes +# driver). Both are required together; the client certificate is the access gate. +image: + pullPolicy: Always + +supervisor: + image: + pullPolicy: Always + +server: + disableTls: false + auth: + allowUnauthenticatedUsers: true + +openshiftRoute: + enabled: true + # host is supplied via --set at install time (cluster-derived Route hostname). + # The default HAProxy Route timeout is 30s, which severs long-lived transfers + # (large sandbox upload/download, SSH-relay `sandbox connect`) mid-stream. Raise + # both the connection timeout and the passthrough tunnel timeout so these paths + # survive. Passthrough Routes proxy in TCP mode, so timeout-tunnel governs the + # established tunnel while timeout covers the pre-tunnel phase. + annotations: + haproxy.router.openshift.io/timeout: 300s + haproxy.router.openshift.io/timeout-tunnel: 300s diff --git a/deploy/helm/openshell/ci/values-openshift-scc.yaml b/deploy/helm/openshell/ci/values-openshift-scc.yaml new file mode 100644 index 0000000000..b7f37be6e0 --- /dev/null +++ b/deploy/helm/openshell/ci/values-openshift-scc.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and +# fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace- +# assigned UID/GID range. Layer after values.yaml: +# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml +# +# The e2e Kubernetes harness applies this automatically when it detects an +# OpenShift cluster (route.openshift.io API present). + +podSecurityContext: null + +securityContext: + runAsNonRoot: true + runAsUser: null + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 54d9e1ff99..25662518e8 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -34,8 +34,10 @@ spec: - host.docker.internal - host.openshell.internal {{- end }} + {{- with .Values.podSecurityContext }} securityContext: - {{- toYaml .Values.podSecurityContext | nindent 4 }} + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: openshell-gateway securityContext: diff --git a/e2e/rust/e2e-openshift.sh b/e2e/rust/e2e-openshift.sh deleted file mode 100755 index 639e6323a4..0000000000 --- a/e2e/rust/e2e-openshift.sh +++ /dev/null @@ -1,184 +0,0 @@ -#!/usr/bin/env bash -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Validates all OpenShift database-backend scenarios against a live cluster. -# -# Prerequisites: -# - oc CLI authenticated to an OpenShift cluster -# - helm 3.x installed -# -# Usage: -# mise run e2e:openshift -# e2e/rust/e2e-openshift.sh [--chart-path ./deploy/helm/openshell] [--image-tag dev] - -set -euo pipefail - -ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -# shellcheck source=e2e/support/gateway-common.sh -source "${ROOT}/e2e/support/gateway-common.sh" - -CHART_PATH="${CHART_PATH:-./deploy/helm/openshell}" -NAMESPACE="openshell" -RELEASE="openshell" -IMAGE_TAG="${IMAGE_TAG:-dev}" -WAIT_TIMEOUT="120s" -PASSED=0 -FAILED=0 -SCENARIOS=() -EXTERNAL_PG_SECRET="my-pg-credentials" -EXTERNAL_PG_SERVICE="openshell-e2e-postgres" -EXTERNAL_PG_SERVICE_ACCOUNT="openshell-e2e-postgres" -EXTERNAL_PG_PASSWORD="openshell-e2e-postgres" -EXTERNAL_PG_DATABASE="openshell" -EXTERNAL_PG_USERNAME="openshell" -EXTERNAL_PG_MANIFEST="${ROOT}/e2e/kubernetes/postgres-fixture.yaml" - -while [[ $# -gt 0 ]]; do - case $1 in - --chart-path) CHART_PATH="$2"; shift 2 ;; - --image-tag) IMAGE_TAG="$2"; shift 2 ;; - --namespace) NAMESPACE="$2"; shift 2 ;; - *) echo "Unknown option: $1" >&2; exit 1 ;; - esac -done - -# --- helpers ---------------------------------------------------------------- - -log() { echo "==> $*"; } -pass() { log "PASS: $1"; PASSED=$((PASSED + 1)); SCENARIOS+=("PASS $1"); } -fail() { log "FAIL: $1 — $2"; FAILED=$((FAILED + 1)); SCENARIOS+=("FAIL $1: $2"); } - -wait_for_ready() { - local label="$1" timeout="$2" - if oc wait pod -n "$NAMESPACE" -l "$label" --for=condition=Ready --timeout="$timeout" 2>/dev/null; then - return 0 - fi - return 1 -} - -cleanup_release() { - log "Cleaning up release $RELEASE" - helm uninstall "$RELEASE" -n "$NAMESPACE" --wait 2>/dev/null || true - # Wait for pods to terminate - for i in $(seq 1 30); do - if [ -z "$(oc get pods -n "$NAMESPACE" -l "app.kubernetes.io/instance=$RELEASE" --no-headers 2>/dev/null)" ]; then - break - fi - sleep 2 - done - # Clean up PVCs left by StatefulSets - oc delete pvc -n "$NAMESPACE" -l "app.kubernetes.io/instance=$RELEASE" --wait=false 2>/dev/null || true -} - -deploy_external_pg() { - local pg_uri - - log "Deploying standalone PostgreSQL as external database..." - oc create serviceaccount "$EXTERNAL_PG_SERVICE_ACCOUNT" -n "$NAMESPACE" 2>/dev/null || true - oc adm policy add-scc-to-user anyuid -z "$EXTERNAL_PG_SERVICE_ACCOUNT" -n "$NAMESPACE" >/dev/null - - oc apply -n "$NAMESPACE" -f "$EXTERNAL_PG_MANIFEST" - oc rollout status "deployment/${EXTERNAL_PG_SERVICE}" -n "$NAMESPACE" --timeout="$WAIT_TIMEOUT" - - pg_uri="postgresql://${EXTERNAL_PG_USERNAME}:${EXTERNAL_PG_PASSWORD}@${EXTERNAL_PG_SERVICE}.${NAMESPACE}.svc.cluster.local:5432/${EXTERNAL_PG_DATABASE}" - log "Creating existing Secret with PostgreSQL credentials..." - oc delete secret "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true - oc create secret generic "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" \ - --from-literal=uri="$pg_uri" -} - -cleanup_external_pg() { - oc delete -n "$NAMESPACE" -f "$EXTERNAL_PG_MANIFEST" --ignore-not-found 2>/dev/null || true - oc delete secret "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" --ignore-not-found 2>/dev/null || true - oc adm policy remove-scc-from-user anyuid -z "$EXTERNAL_PG_SERVICE_ACCOUNT" \ - -n "$NAMESPACE" 2>/dev/null || true -} - -verify_gateway() { - local scenario="$1" - if wait_for_ready "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=$RELEASE" "$WAIT_TIMEOUT"; then - # Check the pod is actually running (not CrashLoopBackOff) - local phase - phase=$(oc get pod -n "$NAMESPACE" -l "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=$RELEASE" \ - -o jsonpath='{.items[0].status.phase}' 2>/dev/null) - if [ "$phase" = "Running" ]; then - pass "$scenario" - else - fail "$scenario" "pod phase is $phase, expected Running" - fi - else - local status - status=$(oc get pods -n "$NAMESPACE" -l "app.kubernetes.io/name=openshell" --no-headers 2>/dev/null || echo "no pods found") - fail "$scenario" "gateway pod not ready within $WAIT_TIMEOUT ($status)" - fi -} - -# --- setup ------------------------------------------------------------------ - -log "Setting up namespace $NAMESPACE" -oc create ns "$NAMESPACE" 2>/dev/null || true -oc adm policy add-scc-to-user privileged -z "${RELEASE}-sandbox" -n "$NAMESPACE" - -OPENSHIFT_FLAGS=( - --set server.disableTls=true - --set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" - --set podSecurityContext.fsGroup=null - --set securityContext.runAsUser=null - --set image.tag="$IMAGE_TAG" -) - -# --- scenario 1: SQLite (default, no postgres) ----------------------------- - -SCENARIO="SQLite (default)" -log "Testing: $SCENARIO" -cleanup_release - -helm install "$RELEASE" "$CHART_PATH" -n "$NAMESPACE" \ - "${OPENSHIFT_FLAGS[@]}" - -verify_gateway "$SCENARIO" -cleanup_release - -# --- scenario 2: External PostgreSQL with existing Secret ------------------- - -SCENARIO="External PostgreSQL (externalDbSecret)" -log "Testing: $SCENARIO" -cleanup_release - -deploy_external_pg - -# Install OpenShell pointing at the existing Secret -helm install "$RELEASE" "$CHART_PATH" -n "$NAMESPACE" \ - "${OPENSHIFT_FLAGS[@]}" \ - --set server.externalDbSecret="$EXTERNAL_PG_SECRET" - -verify_gateway "$SCENARIO" - -# Cleanup external postgres and secret -cleanup_release -cleanup_external_pg - -# --- teardown --------------------------------------------------------------- - -log "Removing SCC binding and namespace" -oc adm policy remove-scc-from-user privileged -z "${RELEASE}-sandbox" -n "$NAMESPACE" 2>/dev/null || true -cleanup_external_pg -oc delete ns "$NAMESPACE" --wait=false 2>/dev/null || true - -# --- summary ---------------------------------------------------------------- - -echo "" -echo "========================================" -echo " Test Summary" -echo "========================================" -for s in "${SCENARIOS[@]}"; do - echo " $s" -done -echo "----------------------------------------" -echo " Passed: $PASSED Failed: $FAILED" -echo "========================================" - -if [ "$FAILED" -gt 0 ]; then - exit 1 -fi diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh index bd7f246d24..8a9eaf3df2 100755 --- a/e2e/with-kube-gateway.sh +++ b/e2e/with-kube-gateway.sh @@ -13,8 +13,20 @@ # Create a local k3d cluster via tasks/scripts/helm-k3s-local.sh, install # the chart, port-forward, and tear the cluster down on exit. # -# Helm e2e currently uses plaintext gateway traffic (ci/values-skaffold.yaml). -# The certgen hook still runs so the gateway has sandbox JWT signing keys. +# On vanilla Kubernetes, Helm e2e talks to the gateway in plaintext over +# `kubectl port-forward` (ci/values-skaffold.yaml). The certgen hook still runs +# so the gateway has sandbox JWT signing keys. +# +# On OpenShift, that port-forward is too slow for `sandbox connect`. That command +# opens an SSH session to the gateway, and SSH needs many small back-and-forth +# messages to set up. Each one has to travel through the port-forward tunnel, so +# the connection never finishes and the test times out. To avoid this, on +# OpenShift the harness reaches the gateway through a normal network path: a +# passthrough OpenShift Route, secured with mandatory mTLS +# (ci/values-openshift-e2e.yaml). +# +# Every OpenShift-specific branch below is gated on OPENSHIFT_DETECTED, so the +# vanilla-Kubernetes path stays exactly the same. # # Set OPENSHELL_E2E_KUBE_EXTRA_VALUES to one or more colon-separated Helm values # files, relative to the repository root or absolute, to layer additional chart @@ -94,6 +106,13 @@ VAULT_CHART_VERSION="${OPENSHELL_E2E_OPENBAO_CHART_VERSION:-0.28.3}" VAULT_DEV_ROOT_TOKEN="${OPENSHELL_E2E_VAULT_DEV_ROOT_TOKEN:-root}" CORPORATE_PROXY_FIXTURE_DEPLOYED=0 CORPORATE_PROXY_FIXTURE_SECRET="openshell-e2e-proxy-auth" +OPENSHIFT_DETECTED=0 +OPENSHIFT_SANDBOX_SCC_GRANTED=0 +OPENSHIFT_POSTGRES_SCC_GRANTED=0 +OPENSHIFT_ROUTE_HOST="" +# Temp dir holding the client mTLS material extracted from openshell-client-tls +# for the OpenShift Route transport. Removed by cleanup(). +OPENSHIFT_PKI_DIR="${WORKDIR}/openshift-pki" # Isolate CLI/SDK gateway metadata from the developer's real config. export XDG_CONFIG_HOME="${WORKDIR}/config" @@ -134,6 +153,16 @@ deploy_postgres_fixture() { kctl create namespace "${NAMESPACE}" fi + if [ "${OPENSHIFT_DETECTED}" = "1" ]; then + echo "Granting anyuid SCC to ${EXTERNAL_PG_FIXTURE_SERVICE} for OpenShift..." + oc adm policy add-scc-to-user anyuid \ + --context "${KUBE_CONTEXT}" \ + -z "${EXTERNAL_PG_FIXTURE_SERVICE}" -n "${NAMESPACE}" + # Record the grant before applying the fixture so cleanup revokes it even if + # the apply below fails and EXTERNAL_PG_FIXTURE_DEPLOYED is never set. + OPENSHIFT_POSTGRES_SCC_GRANTED=1 + fi + kctl -n "${NAMESPACE}" apply -f "${EXTERNAL_PG_FIXTURE_MANIFEST}" EXTERNAL_PG_FIXTURE_DEPLOYED=1 EXTERNAL_PG_FIXTURE_SECRET="${secret_name}" @@ -158,6 +187,14 @@ cleanup_postgres_fixture() { kctl -n "${NAMESPACE}" delete secret "${secret_name}" \ --ignore-not-found >/dev/null 2>&1 || true + if [ "${OPENSHIFT_POSTGRES_SCC_GRANTED}" = "1" ]; then + oc adm policy remove-scc-from-user anyuid \ + --context "${KUBE_CONTEXT}" \ + -z "${EXTERNAL_PG_FIXTURE_SERVICE}" -n "${NAMESPACE}" \ + 2>/dev/null || true + OPENSHIFT_POSTGRES_SCC_GRANTED=0 + fi + EXTERNAL_PG_FIXTURE_DEPLOYED=0 EXTERNAL_PG_FIXTURE_SECRET="" } @@ -242,7 +279,8 @@ cleanup() { fi fi - if [ "${EXTERNAL_PG_FIXTURE_DEPLOYED}" = "1" ]; then + if [ "${EXTERNAL_PG_FIXTURE_DEPLOYED}" = "1" ] \ + || [ "${OPENSHIFT_POSTGRES_SCC_GRANTED}" = "1" ]; then cleanup_postgres_fixture "${EXTERNAL_PG_FIXTURE_SECRET}" fi @@ -255,6 +293,20 @@ cleanup() { --ignore-not-found >/dev/null 2>&1 || true fi + if [ "${OPENSHIFT_SANDBOX_SCC_GRANTED}" = "1" ]; then + oc adm policy remove-scc-from-user privileged \ + --context "${KUBE_CONTEXT}" \ + -z openshell-sandbox -n "${NAMESPACE}" \ + 2>/dev/null || true + OPENSHIFT_SANDBOX_SCC_GRANTED=0 + fi + + # Remove the extracted client mTLS material (also covered by the WORKDIR sweep + # below, but drop the private key promptly and explicitly). + if [ -n "${OPENSHIFT_PKI_DIR}" ]; then + rm -rf "${OPENSHIFT_PKI_DIR}" 2>/dev/null || true + fi + # Sweep managed-mode and operator-mode workspace namespaces before # uninstalling the Helm release (ClusterRole still needed for deletion). if command -v kubectl >/dev/null 2>&1 && [ -n "${KUBE_CONTEXT}" ]; then @@ -327,6 +379,15 @@ scenario_cleanup_release() { -l "app.kubernetes.io/instance=${RELEASE_NAME}" --wait=false 2>/dev/null || true } +scenario_record_failure() { + local scenario_label="$1" + local reason="$2" + DB_FAILED=$((DB_FAILED + 1)) + DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: ${reason}") + scenario_stop_portforward + scenario_cleanup_release +} + scenario_deploy_external_pg() { echo "==> Deploying standalone PostgreSQL as external database..." deploy_postgres_fixture my-pg-credentials @@ -363,84 +424,33 @@ run_scenario() { --wait --timeout 5m HELM_INSTALLED=1 - LOCAL_PORT="$(e2e_pick_port)" - echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..." - kctl -n "${NAMESPACE}" port-forward "svc/openshell" \ - "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 & - PORTFORWARD_PID=$! - - local elapsed=0 pf_timeout=30 - while [ "${elapsed}" -lt "${pf_timeout}" ]; do - if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then - echo "ERROR: kubectl port-forward exited before becoming reachable" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - DB_FAILED=$((DB_FAILED + 1)) - DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: port-forward died") - scenario_stop_portforward - scenario_cleanup_release + if [ "${OPENSHIFT_DETECTED}" = "1" ]; then + # OpenShift: reach the gateway over the passthrough Route with mTLS instead + # of port-forward (which stalls the SSH-relay connect suites). + if ! openshift_register_route_gateway; then + scenario_record_failure "${scenario_label}" "Route/mTLS setup failed" return fi - if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then - break + else + # Vanilla Kubernetes: reach the gateway in plaintext over port-forward. + if ! start_grpc_portforward; then + scenario_record_failure "${scenario_label}" "port-forward failed" + return fi - sleep 1 - elapsed=$((elapsed + 1)) - done - if [ "${elapsed}" -ge "${pf_timeout}" ]; then - echo "ERROR: port-forward did not accept TCP within ${pf_timeout}s" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - DB_FAILED=$((DB_FAILED + 1)) - DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: port-forward timeout") - scenario_stop_portforward - scenario_cleanup_release - return + GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}" + GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}" + e2e_register_plaintext_gateway \ + "${XDG_CONFIG_HOME}" \ + "${GATEWAY_NAME}" \ + "${GATEWAY_ENDPOINT}" \ + "${LOCAL_PORT}" fi - HEALTH_LOCAL_PORT="$(e2e_pick_port)" - local workload_ref - workload_ref="$(kube_workload_ref "${RELEASE_NAME}")" - echo "Starting kubectl port-forward ${workload_ref} ${HEALTH_LOCAL_PORT}:health..." - kctl -n "${NAMESPACE}" port-forward "${workload_ref}" \ - "${HEALTH_LOCAL_PORT}:health" >"${PORTFORWARD_HEALTH_LOG}" 2>&1 & - PORTFORWARD_HEALTH_PID=$! - - elapsed=0 - while [ "${elapsed}" -lt "${pf_timeout}" ]; do - if ! kill -0 "${PORTFORWARD_HEALTH_PID}" 2>/dev/null; then - echo "ERROR: kubectl health port-forward exited before becoming reachable" >&2 - cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true - DB_FAILED=$((DB_FAILED + 1)) - DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: health port-forward died") - scenario_stop_portforward - scenario_cleanup_release - return - fi - if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${HEALTH_LOCAL_PORT}/healthz"; then - break - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - if [ "${elapsed}" -ge "${pf_timeout}" ]; then - echo "ERROR: health port-forward did not accept TCP within ${pf_timeout}s" >&2 - cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true - DB_FAILED=$((DB_FAILED + 1)) - DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: health port-forward timeout") - scenario_stop_portforward - scenario_cleanup_release + if ! start_health_portforward; then + scenario_record_failure "${scenario_label}" "health port-forward failed" return fi - export OPENSHELL_E2E_HEALTH_PORT="${HEALTH_LOCAL_PORT}" - - GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}" - GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}" - e2e_register_plaintext_gateway \ - "${XDG_CONFIG_HOME}" \ - "${GATEWAY_NAME}" \ - "${GATEWAY_ENDPOINT}" \ - "${LOCAL_PORT}" - export OPENSHELL_GATEWAY="${GATEWAY_NAME}" export OPENSHELL_E2E_DRIVER="kubernetes" # Kubernetes e2e runs against k3d/kind-style Docker-backed clusters. Host @@ -493,6 +503,148 @@ configure_fixture_container_engine() { export CONTAINER_ENGINE="${selected_engine}" } +# OpenShift only: extract the client mTLS material, wait for the passthrough +# Route to serve mTLS, assert that a certless caller is rejected at the TLS +# handshake, and register an mTLS CLI gateway pointing at the Route. +# +# Sets GATEWAY_NAME and GATEWAY_ENDPOINT on success. Returns non-zero on failure +# (unreachable Route or a certless request that was NOT rejected — a security +# hole). Reads OPENSHIFT_ROUTE_HOST and OPENSHIFT_PKI_DIR. +openshift_register_route_gateway() { + local pki_dir="${OPENSHIFT_PKI_DIR}" + + rm -rf "${pki_dir}" + mkdir -p "${pki_dir}/client" + + echo "Extracting client mTLS material from secret openshell-client-tls..." + kctl -n "${NAMESPACE}" get secret openshell-client-tls \ + -o jsonpath='{.data.ca\.crt}' | base64 -d >"${pki_dir}/ca.crt" + kctl -n "${NAMESPACE}" get secret openshell-client-tls \ + -o jsonpath='{.data.tls\.crt}' | base64 -d >"${pki_dir}/client/tls.crt" + kctl -n "${NAMESPACE}" get secret openshell-client-tls \ + -o jsonpath='{.data.tls\.key}' | base64 -d >"${pki_dir}/client/tls.key" + + # Wait until an mTLS request to the Route completes the TLS handshake. Helm + # --wait already made the gateway pod Ready; this only covers the short window + # while the OpenShift router loads the new Route. + echo "Waiting for Route https://${OPENSHIFT_ROUTE_HOST} to serve mTLS..." + local elapsed=0 timeout=180 + while [ "${elapsed}" -lt "${timeout}" ]; do + if curl -s --max-time 10 -o /dev/null \ + --cacert "${pki_dir}/ca.crt" \ + --cert "${pki_dir}/client/tls.crt" \ + --key "${pki_dir}/client/tls.key" \ + "https://${OPENSHIFT_ROUTE_HOST}/"; then + break + fi + sleep 3 + elapsed=$((elapsed + 3)) + done + if [ "${elapsed}" -ge "${timeout}" ]; then + echo "ERROR: Route ${OPENSHIFT_ROUTE_HOST} did not serve mTLS within ${timeout}s" >&2 + return 1 + fi + + # Security gate: a caller with no client certificate MUST be rejected during + # the TLS handshake (clientCaSecretName set + no OIDC => mTLS mandatory). + # + # Validate the server cert with --cacert (no -k) and inspect curl's exit code + # so an unrelated TLS/DNS/timeout failure is not silently accepted as "certless + # rejected". Only a handshake abort by the server (no client cert presented) + # counts as the expected rejection. + local certless_rc=0 + curl -s --max-time 10 -o /dev/null \ + --cacert "${pki_dir}/ca.crt" \ + "https://${OPENSHIFT_ROUTE_HOST}/" || certless_rc=$? + case "${certless_rc}" in + 0) + echo "ERROR: SECURITY HOLE — gateway accepted a certless request over the Route" >&2 + return 1 + ;; + 35 | 56) + # 35 CURLE_SSL_CONNECT_ERROR / 56 CURLE_RECV_ERROR: the server aborted the + # TLS handshake because no client certificate was presented — the expected + # mTLS rejection. + echo "OK: Route reachable over mTLS; certless request rejected at TLS (curl ${certless_rc})." + ;; + *) + echo "ERROR: certless probe to ${OPENSHIFT_ROUTE_HOST} failed with curl exit ${certless_rc}, not a TLS client-auth rejection; cannot confirm mTLS is enforced" >&2 + return 1 + ;; + esac + + GATEWAY_NAME="openshell-e2e-openshift" + GATEWAY_ENDPOINT="https://${OPENSHIFT_ROUTE_HOST}" + e2e_register_mtls_gateway \ + "${XDG_CONFIG_HOME}" \ + "${GATEWAY_NAME}" \ + "${GATEWAY_ENDPOINT}" \ + "$(e2e_endpoint_port "${GATEWAY_ENDPOINT}")" \ + "${pki_dir}" +} + +# Start `kubectl port-forward svc/openshell` for the gRPC endpoint and wait for +# it to accept TCP. Sets LOCAL_PORT and PORTFORWARD_PID. Prints the port-forward +# log and returns non-zero on failure. Used for the vanilla-Kubernetes transport +# (the OpenShift transport uses openshift_register_route_gateway instead). +start_grpc_portforward() { + LOCAL_PORT="$(e2e_pick_port)" + echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..." + kctl -n "${NAMESPACE}" port-forward "svc/openshell" \ + "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 & + PORTFORWARD_PID=$! + + local elapsed=0 timeout=30 + while [ "${elapsed}" -lt "${timeout}" ]; do + if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then + echo "ERROR: kubectl port-forward exited before becoming reachable" >&2 + cat "${PORTFORWARD_LOG}" >&2 || true + return 1 + fi + if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + echo "ERROR: port-forward did not accept TCP within ${timeout}s" >&2 + cat "${PORTFORWARD_LOG}" >&2 || true + return 1 +} + +# Start `kubectl port-forward` for the health endpoint and wait for /healthz. +# Sets HEALTH_LOCAL_PORT and PORTFORWARD_HEALTH_PID and exports +# OPENSHELL_E2E_HEALTH_PORT. Used on both cluster types: the OpenShift Route +# targets grpc only, and the health endpoint is not the SSH path so port-forward +# is fine for it. Prints the log and returns non-zero on failure. +start_health_portforward() { + HEALTH_LOCAL_PORT="$(e2e_pick_port)" + local workload_ref + workload_ref="$(kube_workload_ref "${RELEASE_NAME}")" + echo "Starting kubectl port-forward ${workload_ref} ${HEALTH_LOCAL_PORT}:health..." + kctl -n "${NAMESPACE}" port-forward "${workload_ref}" \ + "${HEALTH_LOCAL_PORT}:health" >"${PORTFORWARD_HEALTH_LOG}" 2>&1 & + PORTFORWARD_HEALTH_PID=$! + + local elapsed=0 timeout=30 + while [ "${elapsed}" -lt "${timeout}" ]; do + if ! kill -0 "${PORTFORWARD_HEALTH_PID}" 2>/dev/null; then + echo "ERROR: kubectl health port-forward exited before becoming reachable" >&2 + cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true + return 1 + fi + if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${HEALTH_LOCAL_PORT}/healthz"; then + export OPENSHELL_E2E_HEALTH_PORT="${HEALTH_LOCAL_PORT}" + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + echo "ERROR: health port-forward did not accept TCP within ${timeout}s" >&2 + cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true + return 1 +} + require_cmd helm require_cmd kubectl require_cmd curl @@ -738,6 +890,41 @@ if [ -n "${HOST_GATEWAY_IP}" ]; then fi helm_values_args=(--values "${ROOT}/deploy/helm/openshell/ci/values-skaffold.yaml") +if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | grep -q .; then + OPENSHIFT_DETECTED=1 + echo "OpenShift detected — applying SCC-compatible security context overrides." + helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-openshift-scc.yaml") + + if ! command -v oc >/dev/null 2>&1; then + echo "ERROR: oc CLI is required for OpenShift SCC management but was not found." >&2 + exit 2 + fi + + kctl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kctl apply -f - + + echo "Granting privileged SCC to openshell-sandbox in namespace ${NAMESPACE}..." + oc adm policy add-scc-to-user privileged \ + --context "${KUBE_CONTEXT}" \ + -z openshell-sandbox -n "${NAMESPACE}" + OPENSHIFT_SANDBOX_SCC_GRANTED=1 + + # Drive the gateway through a passthrough Route with mTLS instead of + # port-forward. The Route host is deterministic: OpenShift serves any name + # under the cluster ingress (apps) domain via the router's wildcard, so we + # bake "-." into the server cert SANs before + # the Route exists. + APPS_DOMAIN="$(kctl get ingresses.config/cluster -o jsonpath='{.spec.domain}')" + if [ -z "${APPS_DOMAIN}" ]; then + echo "ERROR: could not resolve the OpenShift cluster ingress domain." >&2 + exit 2 + fi + OPENSHIFT_ROUTE_HOST="${RELEASE_NAME}-${NAMESPACE}.${APPS_DOMAIN}" + echo "Using OpenShift Route host ${OPENSHIFT_ROUTE_HOST}." + + helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-openshift-e2e.yaml") + helm_extra_args+=(--set "openshiftRoute.host=${OPENSHIFT_ROUTE_HOST}") + helm_extra_args+=(--set "pkiInitJob.serverDnsNames[0]=${OPENSHIFT_ROUTE_HOST}") +fi if [ "${OPENSHELL_E2E_KUBE_CORPORATE_PROXY:-0}" = "1" ]; then if [ -z "${HOST_GATEWAY_IP}" ]; then echo "ERROR: corporate proxy e2e requires a host gateway IP for host.openshell.internal" >&2 @@ -866,68 +1053,23 @@ else --docker-password=e2e-password fi - LOCAL_PORT="$(e2e_pick_port)" - echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..." - kctl -n "${NAMESPACE}" port-forward "svc/openshell" \ - "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 & - PORTFORWARD_PID=$! - - elapsed=0 - timeout=30 - while [ "${elapsed}" -lt "${timeout}" ]; do - if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then - echo "ERROR: kubectl port-forward exited before becoming reachable" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - exit 1 - fi - if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then - break - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - if [ "${elapsed}" -ge "${timeout}" ]; then - echo "ERROR: port-forward did not accept TCP within ${timeout}s" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - exit 1 - fi - - HEALTH_LOCAL_PORT="$(e2e_pick_port)" - WORKLOAD_REF="$(kube_workload_ref "${RELEASE_NAME}")" - echo "Starting kubectl port-forward ${WORKLOAD_REF} ${HEALTH_LOCAL_PORT}:health..." - kctl -n "${NAMESPACE}" port-forward "${WORKLOAD_REF}" \ - "${HEALTH_LOCAL_PORT}:health" >"${PORTFORWARD_HEALTH_LOG}" 2>&1 & - PORTFORWARD_HEALTH_PID=$! - - elapsed=0 - timeout=30 - while [ "${elapsed}" -lt "${timeout}" ]; do - if ! kill -0 "${PORTFORWARD_HEALTH_PID}" 2>/dev/null; then - echo "ERROR: kubectl health port-forward exited before becoming reachable" >&2 - cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true - exit 1 - fi - if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${HEALTH_LOCAL_PORT}/healthz"; then - break - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - if [ "${elapsed}" -ge "${timeout}" ]; then - echo "ERROR: health port-forward did not accept TCP within ${timeout}s" >&2 - cat "${PORTFORWARD_HEALTH_LOG}" >&2 || true - exit 1 + if [ "${OPENSHIFT_DETECTED}" = "1" ]; then + # OpenShift: reach the gateway over the passthrough Route with mTLS so the + # SSH-relay `sandbox connect` suites work (port-forward stalls them). + openshift_register_route_gateway || exit 1 + else + # Vanilla Kubernetes: reach the gateway in plaintext over port-forward. + start_grpc_portforward || exit 1 + GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}" + GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}" + e2e_register_plaintext_gateway \ + "${XDG_CONFIG_HOME}" \ + "${GATEWAY_NAME}" \ + "${GATEWAY_ENDPOINT}" \ + "${LOCAL_PORT}" fi - export OPENSHELL_E2E_HEALTH_PORT="${HEALTH_LOCAL_PORT}" - - GATEWAY_NAME="openshell-e2e-kube-${LOCAL_PORT}" - GATEWAY_ENDPOINT="http://127.0.0.1:${LOCAL_PORT}" - e2e_register_plaintext_gateway \ - "${XDG_CONFIG_HOME}" \ - "${GATEWAY_NAME}" \ - "${GATEWAY_ENDPOINT}" \ - "${LOCAL_PORT}" + start_health_portforward || exit 1 export OPENSHELL_GATEWAY="${GATEWAY_NAME}" export OPENSHELL_E2E_DRIVER="kubernetes" diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 8e86bc0643..7d47b18065 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -74,6 +74,7 @@ Use gateway metadata, deployment values, or the user's setup notes to identify t | Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. | | Podman | Podman socket, rootless networking, sandbox containers, image pulls. | | Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. | +| OpenShift | Same as Kubernetes, plus SecurityContextConstraints (SCCs) and, for external access, an OpenShift `Route`. Detect OpenShift by the presence of the `route.openshift.io` API group (`oc api-resources --api-group=route.openshift.io`). | | VM | VM driver logs, rootfs availability, host virtualization support. | | Extension | External driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs. | @@ -686,6 +687,8 @@ configuration — check that the gateway spawned the driver binary you expect | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | +| OpenShift gateway pod fails to start with an SCC/`runAsUser` error (e.g. `unable to validate against any security context constraint`) | Chart's default `podSecurityContext`/`securityContext` hardcodes `runAsUser`/`fsGroup`, which the restricted-v2 SCC rejects; it must instead inject the namespace-assigned UID/GID range | `oc -n openshell describe pod `; deploy with `podSecurityContext: null` and clear `securityContext.runAsUser` (see `deploy/helm/openshell/ci/values-openshift-scc.yaml`) | +| OpenShift sandbox pod fails to start (`unable to validate against any security context constraint`) | The `openshell-sandbox` service account lacks the privileged SCC it needs | `oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell`; remove with `remove-scc-from-user` when done | | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login `, gateway auth logs | | Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; `binding_policy`, phases, and failure policy | diff --git a/tasks/test.toml b/tasks/test.toml index 4a5cda0890..bb1fa2e9ab 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -268,6 +268,3 @@ env = { OPENSHELL_E2E_DOCKER_GPU = "1", OPENSHELL_E2E_DOCKER_TEST = "gpu", OPENS depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh" -["e2e:openshift"] -description = "Run OpenShift database-backend integration scenarios against a live cluster (requires oc CLI authenticated to an OpenShift cluster)" -run = "e2e/rust/e2e-openshift.sh"