bug(mxc): sandbox CA bundle holds only the sandbox CA, so tls: skip endpoints fail certificate verification #5060

User Story

I'm integrating OpenShell into Hermes Agent at Nous Research, using the MXC driver on Windows. When a user allows a website from Hermes, the plugin adds it as a tls: skip endpoint so programs in the sandbox see the site's own certificate. On Windows, Python requests, git and curl --cacert then fail certificate verification for that site, because the CA bundle they read holds only the sandbox CA.

Problem Statement

On the windows branch, the variables OpenShell injects into an MXC sandbox for TLS trust (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, GIT_SSL_CAINFO) name a bundle that contains only the OpenShell sandbox CA. These variables replace a client's default trust store. For an endpoint with tls: skip, the proxy relays the TLS session without terminating it, so the client receives the server's own certificate, and a client that honors these variables rejects it because the bundle has no public roots.

On Linux the same bundle holds the distribution's root certificates followed by the sandbox CA, so the same clients verify both terminated and tls: skip endpoints. There read_system_ca_bundle() finds the distribution's bundle at one of the paths in SYSTEM_CA_PATHS (/etc/ssl/certs/ca-certificates.crt and the RHEL, openSUSE and Alpine equivalents): inside the sandbox's container for the Docker, Podman and Kubernetes drivers, and on the host for the VM driver. The MXC host supervisor runs on Windows, which keeps its root certificates in the certificate store, so none of those paths exist and the system part of the bundle is empty.

Impact / Why This Matters

Every client that honors these variables fails on every tls: skip endpoint in a Windows sandbox: Python requests (REQUESTS_CA_BUNDLE), Python ssl with cafile=$SSL_CERT_FILE, git with the OpenSSL backend (GIT_SSL_CAINFO), and Windows curl.exe given --cacert "%CURL_CA_BUNDLE%". The documentation recommends tls: skip for clients that must complete TLS themselves, such as client-certificate mTLS, so those workflows do not work on Windows.

The workarounds are to unset the variables or to disable certificate verification inside the sandbox. Unsetting them breaks every terminated endpoint in the same sandbox, because those clients then no longer trust the sandbox CA. Disabling verification removes certificate checks altogether; in our runs, an agent that hit CERTIFICATE_VERIFY_FAILED retried with curl -k.

Acceptance Criteria

• On Windows, the bundle named by SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and GIT_SSL_CAINFO contains the server-authentication roots of the Windows certificate store, followed by the sandbox CA.
• A client limited to that bundle verifies a server behind a tls: skip endpoint and sees the server's own certificate.
• The same client still verifies a terminated endpoint and sees a certificate issued by the sandbox CA.
• The bundle on Linux and macOS is unchanged.

Reproduction Steps

Prerequisites: an MXC gateway set up as described in WINDOWS.md (with allow_driver_config = true), selected in the openshell CLI, and Internet access. The only client used is Windows' own curl.exe.

1. Save this script as repro.ps1. It writes a policy with example.com:443 as tls: skip and example.org:443 terminated, starts one sandbox whose command runs curl.exe --cacert "%CURL_CA_BUNDLE%" against both, and prints the results.

   # Reproduces how a Windows (MXC) sandbox's injected CA bundle treats a `tls: skip`
   # endpoint and a terminated endpoint. Needs an MXC gateway selected in the
   # openshell CLI and Internet access. Inside the sandbox, Windows curl.exe trusts
   # only the bundle named by CURL_CA_BUNDLE (the same file as SSL_CERT_FILE,
   # REQUESTS_CA_BUNDLE and GIT_SSL_CAINFO):
   #   https://example.com/  tls: skip   the proxy relays TLS; curl sees the site's certificate
   #   https://example.org/  default     the proxy terminates TLS; curl sees a sandbox-CA certificate
   param(
       [string] $Cli = 'openshell',
       [string] $Work = (Join-Path $env:TEMP 'openshell-ca-bundle-repro')
   )
   # Policy binaries match the kernel-reported path exactly, including case.
   $cmd = 'C:\Windows\System32\cmd.exe'
   $curl = 'C:\Windows\System32\curl.exe'
   New-Item -ItemType Directory -Force $Work | Out-Null
   $w = $Work.Replace('\', '/')
   $policy = Join-Path $Work 'policy.yaml'
   $batch = Join-Path $Work 'probe.cmd'
   
   Set-Content -Encoding ASCII $policy @"
   version: 1
   filesystem_policy:
     include_workdir: false
     read_write:
       - "$w"
   network_policies:
     passthrough:
       name: passthrough
       endpoints:
         - host: example.com
           port: 443
           tls: skip
       binaries:
         - path: '$curl'
     terminated:
       name: terminated
       endpoints:
         - host: example.org
           port: 443
       binaries:
         - path: '$curl'
   "@
   
   # Schannel ignores CURL_CA_BUNDLE unless it is passed as --cacert. Best-effort
   # revocation: the sandbox CA has no CRL, and revocation hosts are not allowed.
   Set-Content -Encoding ASCII $batch @"
   @echo off
   findstr /c:"BEGIN CERTIFICATE" "%CURL_CA_BUNDLE%" | find /c /v "" > "$Work\bundle-count.txt"
   "$curl" -sS -o NUL --ssl-revoke-best-effort --cacert "%CURL_CA_BUNDLE%" --stderr "$Work\passthrough.err.txt" -w "exit=%%{exitcode}\n%%{certs}" https://example.com/ > "$Work\passthrough.txt"
   "$curl" -sS -o NUL --ssl-revoke-best-effort --cacert "%CURL_CA_BUNDLE%" --stderr "$Work\terminated.err.txt" -w "exit=%%{exitcode}\n%%{certs}" https://example.org/ > "$Work\terminated.txt"
   echo done > "$Work\done.txt"
   "@
   
   $config = @{ mxc = @{ command = @($cmd, '/d', '/c', $batch); cwd = $w } } | ConvertTo-Json -Compress
   # A proxy 403 here means the proxy could not attribute the connection to a process
   # (a separate, intermittent Windows issue, NVIDIA/OpenShell#5056), not a certificate result; run again.
   foreach ($try in 1..3) {
       & $Cli sandbox delete ca-bundle-repro 2>$null | Out-Null
       Remove-Item (Join-Path $Work '*.txt') -ErrorAction SilentlyContinue
       # One verbatim argument string: PowerShell versions differ in how they quote JSON for native programs.
       Start-Process -FilePath $Cli -NoNewWindow -Wait -ArgumentList ("sandbox create --name ca-bundle-repro --policy `"$policy`" " +
           "--driver-config-json `"$($config.Replace('"', '\"'))`" --detach")
       $deadline = (Get-Date).AddSeconds(120)
       while (-not (Test-Path (Join-Path $Work 'done.txt')) -and (Get-Date) -lt $deadline) { Start-Sleep 1 }
       & $Cli sandbox delete ca-bundle-repro | Out-Null
       if (-not (Select-String -Quiet -Pattern 'response 403' -Path (Join-Path $Work '*.err.txt'))) { break }
   }
   
   "runs: $try"
   "certificates in CURL_CA_BUNDLE: $((Get-Content (Join-Path $Work 'bundle-count.txt') -ErrorAction SilentlyContinue) -join '')"
   foreach ($case in 'passthrough', 'terminated') {
       $out = Get-Content (Join-Path $Work "$case.txt") -ErrorAction SilentlyContinue
       $issuer = ($out | Select-String -Pattern '^Issuer:' | Select-Object -First 1).Line
       $err = (Get-Content (Join-Path $Work "$case.err.txt") -ErrorAction SilentlyContinue) -join ' '
       "{0,-12} {1}  {2}  {3}" -f $case, ($out | Select-Object -First 1), $issuer, $err
   }

2. Run powershell -NoProfile -ExecutionPolicy Bypass -File repro.ps1.

Observed:

runs: 2
certificates in CURL_CA_BUNDLE: 1
passthrough  exit=60    curl: (60) schannel: the certificate chain is incomplete More details here: https://curl.se/docs/sslcerts.html  curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the webpage mentioned above.
terminated   exit=0  Issuer:CN=OpenShell Sandbox CA, O=OpenShell

Expected: the bundle holds the Windows roots plus the sandbox CA, and the tls: skip request exits 0 with the site's own issuer.

Suggested UX (if applicable)

None. No configuration change; the bundle contents change.

Environment

• OpenShell: windows branch at 5d1bfbf (openshell 0.1.5-dev.34+g5d1bfbf92), native x64 release build
• OS: Windows 11 Pro, build 26340.9767 (x64)
• Runtime: MXC driver, ProcessContainer, wxc-exec 1.0.1 (--probe: tier base-container, needsDaclAugmentation: false)

Logs

OCSF NET:OPEN [INFO] ALLOWED C:\Windows\System32\curl.exe(0) -> example.com:443 [policy:passthrough engine:opa] sandbox=ca-bundle-repro
OCSF NET:OPEN [INFO] ALLOWED C:\Windows\System32\curl.exe(0) -> example.org:443 [policy:terminated engine:opa] sandbox=ca-bundle-repro
OCSF HTTP:GET [INFO] ALLOWED GET http://example.org:443/ sandbox=ca-bundle-repro

The proxy allows both connections; the tls: skip request fails in the client's certificate check.