test(ci): verify Kubernetes trace receipt at a collector

.agents/skills/helm-dev-environment/SKILL.md

@@ -265,6 +265,14 @@ task running while using those endpoints.
 
 ### Viewing local traces
 
+For an automated collector receipt check, use `mise run e2e:kubernetes:tracing`
+with `OPENSHELL_E2E_KUBE_CONTEXT` pointing to a disposable test cluster and
+`OPENSHELL_BIN` pointing to the CLI under test. The task installs a temporary
+collector, uses the existing gateway harness, and fails when expected gateway
+or Kubernetes-driver spans do not arrive. The harness owns the test gateway
+release and namespace; do not run it against an existing working installation.
+Evidence is saved under `results/kubernetes-tracing/`.
+
 The gateway exports OTLP/gRPC to
 `http://openshell-collector.observability.svc.cluster.local:4317` through
 `ci/values-local-tracing.yaml`, which only `skaffold.yaml` layers. The Kubernetes

.github/workflows/branch-e2e.yml

@@ -501,6 +501,11 @@ jobs:
             agent_sandbox_version: v0.5.0
             job_name: Kubernetes E2E (additional destination CA, Agent Sandbox v1beta1)
             test_name: additional_ca
+          - agent_sandbox_api: v1beta1
+            agent_sandbox_version: v1.0.3
+            job_name: Kubernetes E2E (OTLP trace receipt)
+            test_name: ""
+            e2e_task: e2e:kubernetes:tracing
     permissions:
       actions: read
       contents: read
@@ -512,6 +517,7 @@ jobs:
       job-name: ${{ matrix.job_name }}
       agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
       test-name: ${{ matrix.test_name }}
+      e2e-task: ${{ matrix.e2e_task || 'e2e:kubernetes' }}
 
   kubernetes-app-listener-e2e:
     needs: [pr_metadata, build-binaries, build-conformance, build-images]

.github/workflows/e2e-kubernetes-test.yml

@@ -179,6 +179,14 @@ jobs:
           OPENSHELL_E2E_APP_TAKEOVER: ${{ inputs.listener-takeover }}
         run: mise run --no-deps --skip-deps "$E2E_TASK"
 
+      - name: Upload Kubernetes trace receipt evidence
+        if: always() && inputs.e2e-task == 'e2e:kubernetes:tracing'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: kubernetes-trace-receipt
+          path: results/kubernetes-tracing/
+          if-no-files-found: warn
+
       - name: Upload listener test evidence
         if: always() && inputs.e2e-task == 'e2e:kubernetes:app-listener'
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

CI.md

@@ -190,6 +190,13 @@ The GitHub ruleset should require the `OpenShell / ...` statuses published by
 `Required CI Gates` plus the direct `OpenShell / Trivy Changes` result, not the
 push-triggered workflow jobs themselves.
 
+The standard Kubernetes E2E matrix also includes an explicit OTLP trace-receipt
+lane. It starts a real collector before enabling export and requires gateway and
+Kubernetes compute-driver spans, their resource attributes, and shared validation
+context. Missing receipts fail the lane; ordinary lanes keep export disabled.
+Its collector logs and receipt evidence are uploaded as
+`kubernetes-trace-receipt`. See [Testing](TESTING.md#kubernetes-e2e-e2eruste2e-kubernetessh).
+
 ### K3s conformance version baseline
 
 The `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the

TESTING.md

@@ -752,6 +752,7 @@ Available task variants:
 | Task | Purpose |
 |---|---|
 | `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway |
+| `e2e:kubernetes:tracing` | Gateway and Kubernetes driver trace receipt at a real OTLP collector; requires a disposable cluster context and a built CLI |
 | `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) |
 | `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay |
 | `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage |
@@ -763,6 +764,18 @@ Available task variants:
 | `e2e:kubernetes:shared-supervisor-ha` | Two-replica shared supervisor pool behind two gateway replicas on PostgreSQL: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted; crashes and restarts gateways and checks each sandbox's session stays on its supervisor's gateway |
 | `e2e:kubernetes:shared-supervisor-scaling` | One-member shared supervisor pool: scales it to two members and back with `helm upgrade`, checking sandboxes stay running and the removed member hands its sandboxes off |
 
+Ordinary Kubernetes lanes do not configure OTLP export. The separate
+`e2e:kubernetes:tracing` lane installs a pinned OpenTelemetry Collector in its
+own temporary namespace before enabling gateway export. It creates and exercises
+a sandbox, then requires correctly attributed gateway and driver operations,
+valid trace IDs, and shared gateway-to-driver validation context within a bounded
+deadline. Missing spans fail the lane. Collector logs and receipt evidence are
+saved under `results/kubernetes-tracing/`, including on failure. This verifies
+collector receipt, not retention in a trace backend.
+
+See [the tracing fixture](e2e/kubernetes/tracing/README.md) for local commands,
+the disposable-cluster requirement, and the expected operations.
+
 Kubernetes e2e environment variables:
 
 | Variable | Purpose |

e2e/kubernetes/tracing/README.md

@@ -0,0 +1,72 @@
+# Kubernetes trace receipt
+
+This focused lane completes the collector-validation part of #3767. The missing
+collector configuration fix is already covered by #5348.
+
+Run the checker controls without a cluster:
+
+```shell
+uv run --no-project python -m unittest discover -s e2e/kubernetes/tracing -p '*_test.py'
+```
+
+Build the CLI, select a **disposable** kind or k3d test cluster with the supported
+Agent Sandbox controller, and run:
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=kind-my-test-cluster \
+  OPENSHELL_BIN="$PWD/target/debug/openshell" \
+  IMAGE_TAG=<full-source-commit> \
+  mise run e2e:kubernetes:tracing
+```
+
+The existing gateway harness installs and removes its test gateway in the
+`openshell` namespace. It must not target a working installation. Use matching
+gateway, supervisor, and sandbox images; a floating `latest` or `dev` tag
+does not establish exact-source validation.
+
+The lane installs OpenTelemetry Collector Contrib 0.164.0 in a fresh namespace,
+waits for its OTLP/gRPC receiver, then enables export only for the test gateway.
+The collector writes newline-delimited OTLP JSON to its container output through
+the file exporter. It has no service-account token, drops all capabilities, and
+uses a read-only root filesystem. The fixture runs as UID 0 to reopen the runtime
+stdout pipe; it is intended for disposable kind/k3d CI clusters, not restricted
+OpenShift namespaces or production deployments.
+
+A successful probe creates a sandbox and executes a small command. Receipt
+requires all of:
+
+- Gateway `openshell.v1.OpenShell/CreateSandbox`.
+- Kubernetes driver `openshell.compute.v1.ComputeDriver/ValidateSandboxCreate`,
+  sharing the gateway request's trace ID.
+- Kubernetes driver `openshell.compute.v1.ComputeDriver/CreateSandbox`.
+- Expected service names, the unique test gateway identity, the Kubernetes
+  compute-driver attribute, a nonempty service version, and valid nonzero IDs.
+
+Only successful or unset span statuses qualify. Asynchronous provisioning need
+not share the original gateway request's root. The dedicated fixture uses
+always-on sampling so this check is deterministic; production sampling and
+supervisor denial correlation remain separate tests.
+
+The receipt deadline is 60 seconds after the workload probe succeeds. An absent
+export, wrong service/resource identity, or broken validation context fails
+clearly. Checker tests include those negative controls. Readiness alone cannot
+pass the lane.
+
+The parent runner also requires the probe's receipt file and independently
+rechecks it against the captured collector data. A zero exit status from the
+gateway harness cannot pass a run that never reached the probe.
+
+Evidence under `results/kubernetes-tracing/<run-id>/` includes collector
+configuration, Helm values, gateway harness output, collector logs, a receipt
+summary, pod image IDs, and cleanup output. The collector namespace is removed
+on success and failure, including handled SIGINT/SIGTERM cancellation. Harness
+output is written as it arrives, so timeout diagnostics survive. On timeout or
+cancellation, the runner terminates its process group, allows up to 180 seconds
+for gateway teardown, then kills any remaining descendants. Collector cleanup
+verifies the run's namespace ownership label before deleting it. Cleanup failures
+are recorded and cannot turn a failed or cancelled run into a pass. SIGKILL,
+host loss, or an unreachable Kubernetes API can still require manual cleanup;
+the evidence records the namespace to inspect.
+
+Collector receipt does not establish trace-backend
+indexing, retention, loss-free delivery, or fleet reliability.