diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 56061385c3..b2a1e9baf6 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -265,6 +265,14 @@ task running while using those endpoints. ### Viewing local traces +For an automated collector receipt check, use `mise run e2e:kubernetes:tracing` +with `OPENSHELL_E2E_KUBE_CONTEXT` pointing to a disposable test cluster and +`OPENSHELL_BIN` pointing to the CLI under test. The task installs a temporary +collector, uses the existing gateway harness, and fails when expected gateway +or Kubernetes-driver spans do not arrive. The harness owns the test gateway +release and namespace; do not run it against an existing working installation. +Evidence is saved under `results/kubernetes-tracing/`. + The gateway exports OTLP/gRPC to `http://openshell-collector.observability.svc.cluster.local:4317` through `ci/values-local-tracing.yaml`, which only `skaffold.yaml` layers. The Kubernetes diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 0f59ab2a81..410a4b2f08 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -501,6 +501,11 @@ jobs: agent_sandbox_version: v0.5.0 job_name: Kubernetes E2E (additional destination CA, Agent Sandbox v1beta1) test_name: additional_ca + - agent_sandbox_api: v1beta1 + agent_sandbox_version: v1.0.3 + job_name: Kubernetes E2E (OTLP trace receipt) + test_name: "" + e2e_task: e2e:kubernetes:tracing permissions: actions: read contents: read @@ -512,6 +517,7 @@ jobs: job-name: ${{ matrix.job_name }} agent-sandbox-version: ${{ matrix.agent_sandbox_version }} test-name: ${{ matrix.test_name }} + e2e-task: ${{ matrix.e2e_task || 'e2e:kubernetes' }} kubernetes-app-listener-e2e: needs: [pr_metadata, build-binaries, build-conformance, build-images] diff --git a/.github/workflows/e2e-kubernetes-test.yml b/.github/workflows/e2e-kubernetes-test.yml index 1ae616e36e..b5362f12a4 100644 --- a/.github/workflows/e2e-kubernetes-test.yml +++ b/.github/workflows/e2e-kubernetes-test.yml @@ -179,6 +179,14 @@ jobs: OPENSHELL_E2E_APP_TAKEOVER: ${{ inputs.listener-takeover }} run: mise run --no-deps --skip-deps "$E2E_TASK" + - name: Upload Kubernetes trace receipt evidence + if: always() && inputs.e2e-task == 'e2e:kubernetes:tracing' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: kubernetes-trace-receipt + path: results/kubernetes-tracing/ + if-no-files-found: warn + - name: Upload listener test evidence if: always() && inputs.e2e-task == 'e2e:kubernetes:app-listener' uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 diff --git a/CI.md b/CI.md index a1a1b8bc77..c1d0b2f716 100644 --- a/CI.md +++ b/CI.md @@ -190,6 +190,13 @@ The GitHub ruleset should require the `OpenShell / ...` statuses published by `Required CI Gates` plus the direct `OpenShell / Trivy Changes` result, not the push-triggered workflow jobs themselves. +The standard Kubernetes E2E matrix also includes an explicit OTLP trace-receipt +lane. It starts a real collector before enabling export and requires gateway and +Kubernetes compute-driver spans, their resource attributes, and shared validation +context. Missing receipts fail the lane; ordinary lanes keep export disabled. +Its collector logs and receipt evidence are uploaded as +`kubernetes-trace-receipt`. See [Testing](TESTING.md#kubernetes-e2e-e2eruste2e-kubernetessh). + ### K3s conformance version baseline The `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the diff --git a/TESTING.md b/TESTING.md index a3936ad649..7505c0eb57 100644 --- a/TESTING.md +++ b/TESTING.md @@ -752,6 +752,7 @@ Available task variants: | Task | Purpose | |---|---| | `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway | +| `e2e:kubernetes:tracing` | Gateway and Kubernetes driver trace receipt at a real OTLP collector; requires a disposable cluster context and a built CLI | | `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) | | `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay | | `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage | @@ -763,6 +764,18 @@ Available task variants: | `e2e:kubernetes:shared-supervisor-ha` | Two-replica shared supervisor pool behind two gateway replicas on PostgreSQL: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted; crashes and restarts gateways and checks each sandbox's session stays on its supervisor's gateway | | `e2e:kubernetes:shared-supervisor-scaling` | One-member shared supervisor pool: scales it to two members and back with `helm upgrade`, checking sandboxes stay running and the removed member hands its sandboxes off | +Ordinary Kubernetes lanes do not configure OTLP export. The separate +`e2e:kubernetes:tracing` lane installs a pinned OpenTelemetry Collector in its +own temporary namespace before enabling gateway export. It creates and exercises +a sandbox, then requires correctly attributed gateway and driver operations, +valid trace IDs, and shared gateway-to-driver validation context within a bounded +deadline. Missing spans fail the lane. Collector logs and receipt evidence are +saved under `results/kubernetes-tracing/`, including on failure. This verifies +collector receipt, not retention in a trace backend. + +See [the tracing fixture](e2e/kubernetes/tracing/README.md) for local commands, +the disposable-cluster requirement, and the expected operations. + Kubernetes e2e environment variables: | Variable | Purpose | diff --git a/e2e/kubernetes/tracing/README.md b/e2e/kubernetes/tracing/README.md new file mode 100644 index 0000000000..32fdb1f778 --- /dev/null +++ b/e2e/kubernetes/tracing/README.md @@ -0,0 +1,72 @@ +# Kubernetes trace receipt + +This focused lane completes the collector-validation part of #3767. The missing +collector configuration fix is already covered by #5348. + +Run the checker controls without a cluster: + +```shell +uv run --no-project python -m unittest discover -s e2e/kubernetes/tracing -p '*_test.py' +``` + +Build the CLI, select a **disposable** kind or k3d test cluster with the supported +Agent Sandbox controller, and run: + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=kind-my-test-cluster \ + OPENSHELL_BIN="$PWD/target/debug/openshell" \ + IMAGE_TAG= \ + mise run e2e:kubernetes:tracing +``` + +The existing gateway harness installs and removes its test gateway in the +`openshell` namespace. It must not target a working installation. Use matching +gateway, supervisor, and sandbox images; a floating `latest` or `dev` tag +does not establish exact-source validation. + +The lane installs OpenTelemetry Collector Contrib 0.164.0 in a fresh namespace, +waits for its OTLP/gRPC receiver, then enables export only for the test gateway. +The collector writes newline-delimited OTLP JSON to its container output through +the file exporter. It has no service-account token, drops all capabilities, and +uses a read-only root filesystem. The fixture runs as UID 0 to reopen the runtime +stdout pipe; it is intended for disposable kind/k3d CI clusters, not restricted +OpenShift namespaces or production deployments. + +A successful probe creates a sandbox and executes a small command. Receipt +requires all of: + +- Gateway `openshell.v1.OpenShell/CreateSandbox`. +- Kubernetes driver `openshell.compute.v1.ComputeDriver/ValidateSandboxCreate`, + sharing the gateway request's trace ID. +- Kubernetes driver `openshell.compute.v1.ComputeDriver/CreateSandbox`. +- Expected service names, the unique test gateway identity, the Kubernetes + compute-driver attribute, a nonempty service version, and valid nonzero IDs. + +Only successful or unset span statuses qualify. Asynchronous provisioning need +not share the original gateway request's root. The dedicated fixture uses +always-on sampling so this check is deterministic; production sampling and +supervisor denial correlation remain separate tests. + +The receipt deadline is 60 seconds after the workload probe succeeds. An absent +export, wrong service/resource identity, or broken validation context fails +clearly. Checker tests include those negative controls. Readiness alone cannot +pass the lane. + +The parent runner also requires the probe's receipt file and independently +rechecks it against the captured collector data. A zero exit status from the +gateway harness cannot pass a run that never reached the probe. + +Evidence under `results/kubernetes-tracing//` includes collector +configuration, Helm values, gateway harness output, collector logs, a receipt +summary, pod image IDs, and cleanup output. The collector namespace is removed +on success and failure, including handled SIGINT/SIGTERM cancellation. Harness +output is written as it arrives, so timeout diagnostics survive. On timeout or +cancellation, the runner terminates its process group, allows up to 180 seconds +for gateway teardown, then kills any remaining descendants. Collector cleanup +verifies the run's namespace ownership label before deleting it. Cleanup failures +are recorded and cannot turn a failed or cancelled run into a pass. SIGKILL, +host loss, or an unreachable Kubernetes API can still require manual cleanup; +the evidence records the namespace to inspect. + +Collector receipt does not establish trace-backend +indexing, retention, loss-free delivery, or fleet reliability. diff --git a/e2e/kubernetes/tracing/lifecycle_test.py b/e2e/kubernetes/tracing/lifecycle_test.py new file mode 100644 index 0000000000..9723dbd7ce --- /dev/null +++ b/e2e/kubernetes/tracing/lifecycle_test.py @@ -0,0 +1,315 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Real process controls for timeout, cancellation, teardown and partial logs.""" + +import json +import os +import shlex +import signal +import subprocess +import sys +import tempfile +import time +import unittest +from contextlib import suppress +from pathlib import Path +from unittest import mock + +import run + + +def wait_for(path): + deadline = time.monotonic() + 5 + while not path.exists(): + if time.monotonic() > deadline: + raise AssertionError(f"process did not become ready: {path}") + time.sleep(0.01) + + +def harness(root, *, ignore_term=False): + child = root / "child.py" + ready = root / "ready" + heartbeat = root / "heartbeat" + pid = root / "child-pid" + cleanup = root / "cleanup" + child.write_text( + "import os,signal,time\nfrom pathlib import Path\n" + + ("signal.signal(signal.SIGTERM,signal.SIG_IGN)\n" if ignore_term else "") + + f"Path({str(pid)!r}).write_text(str(os.getpid()))\n" + + f"Path({str(heartbeat)!r}).write_text('starting')\n" + + f"Path({str(ready)!r}).write_text('ready')\n" + + f"while True:\n Path({str(heartbeat)!r}).write_text(str(time.monotonic_ns()))\n time.sleep(0.02)\n" + ) + # Exercise the signal/EXIT contract installed in the actual gateway harness. + traps = "\n".join( + line + for line in (run.ROOT / "e2e/with-kube-gateway.sh").read_text().splitlines() + if line.startswith("trap ") + ) + script = root / "harness.sh" + script.write_text( + f"cleanup() {{ printf cleaned > {shlex.quote(str(cleanup))}; }}\n" + + traps + + "\n" + + f"{shlex.quote(sys.executable)} {shlex.quote(str(child))} &\n" + + f"while [ ! -f {shlex.quote(str(ready))} ]; do sleep 0.01; done\n" + + "echo 'harness started'\nwait\n" + ) + return script, ready, heartbeat, pid, cleanup + + +def stop_test_child(pid_path): + if pid_path.exists(): + with suppress(ProcessLookupError): + os.kill(int(pid_path.read_text()), signal.SIGKILL) + + +class LifecycleTest(unittest.TestCase): + def assert_child_stopped(self, heartbeat): + before = heartbeat.read_text() + time.sleep(0.15) + self.assertEqual( + before, heartbeat.read_text(), "owned child is still doing work" + ) + + def test_timeout_runs_harness_cleanup_and_retains_partial_log(self): + with tempfile.TemporaryDirectory() as work: + root = Path(work) + script, _, heartbeat, pid, cleanup = harness(root) + output = root / "harness.log" + try: + with self.assertRaises(subprocess.TimeoutExpired): + run.run( + ["bash", str(script)], + timeout=0.3, + termination_grace=0.3, + log_path=output, + ) + self.assertTrue( + cleanup.exists(), "the harness EXIT cleanup did not run" + ) + self.assertIn("harness started", output.read_text()) + self.assert_child_stopped(heartbeat) + finally: + stop_test_child(pid) + + def test_completed_harness_reclaims_background_children_and_preserves_status(self): + for code in [0, 7]: + with self.subTest(exit_code=code), tempfile.TemporaryDirectory() as work: + root = Path(work) + script, _, heartbeat, pid, cleanup = harness(root, ignore_term=True) + script.write_text( + script.read_text().replace( + "echo 'harness started'\nwait\n", + f"echo 'harness started'\nexit {code}\n", + ) + ) + try: + result = run.run( + ["bash", str(script)], + check=False, + termination_grace=0.3, + log_path=root / "harness.log", + ) + self.assertEqual(result.returncode, code) + self.assertTrue(cleanup.exists()) + self.assert_child_stopped(heartbeat) + finally: + stop_test_child(pid) + + def test_timeout_escalates_for_a_child_ignoring_termination(self): + with tempfile.TemporaryDirectory() as work: + root = Path(work) + script, _, heartbeat, pid, cleanup = harness(root, ignore_term=True) + started = time.monotonic() + try: + with self.assertRaises(subprocess.TimeoutExpired): + run.run( + ["bash", str(script)], + timeout=0.3, + termination_grace=0.3, + log_path=root / "harness.log", + ) + self.assertLess(time.monotonic() - started, 3) + self.assertTrue(cleanup.exists()) + self.assert_child_stopped(heartbeat) + finally: + stop_test_child(pid) + + def test_handled_signals_stop_descendants_and_run_exit_cleanup(self): + for number in [signal.SIGINT, signal.SIGTERM]: + with self.subTest(signal=number), tempfile.TemporaryDirectory() as work: + root = Path(work) + script, ready, heartbeat, pid, cleanup = harness(root, ignore_term=True) + probe = root / "probe.py" + probe.write_text( + f"import sys\nsys.path.insert(0,{str(Path(run.__file__).parent)!r})\nimport run\n" + + "with run.termination_signals():\n" + + f" run.run(['bash',{str(script)!r}],timeout=30,termination_grace=0.3,log_path=__import__('pathlib').Path({str(root / 'harness.log')!r}))\n" + ) + process = subprocess.Popen( + [sys.executable, str(probe)], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + try: + wait_for(ready) + os.kill(process.pid, number) + stdout, stderr = process.communicate(timeout=5) + self.assertEqual(process.returncode, 128 + number, stdout + stderr) + self.assertTrue(cleanup.exists()) + self.assert_child_stopped(heartbeat) + finally: + if process.poll() is None: + process.kill() + process.wait(timeout=5) + stop_test_child(pid) + + def test_cancellation_cleans_created_namespace_and_preserves_failure(self): + for phase in ["create", "apply"]: + for cleanup_fails in [False, True]: + with ( + self.subTest(phase=phase, cleanup_fails=cleanup_fails), + tempfile.TemporaryDirectory() as work, + ): + root = Path(work) + cli = root / "cli" + cli.write_text("unused test CLI") + probe = root / "cancel.py" + probe.write_text( + "import json,os,signal,subprocess,sys\nfrom pathlib import Path\nfrom unittest import mock\n" + + f"sys.path.insert(0,{str(Path(run.__file__).parent)!r})\nimport run\n" + + f"root=Path({str(root)!r}); owner={{}}\n" + + "def fake(command,**kwargs):\n" + + " with (root/'calls.jsonl').open('a') as out: out.write(json.dumps(command)+'\\n')\n" + + " if 'create' in command: owner.update(json.loads(kwargs['input']))\n" + + f" if {phase!r} in command: os.kill(os.getpid(),signal.SIGTERM)\n" + + " text=json.dumps(owner) if 'get' in command and 'namespace' in command else ''\n" + + f" return subprocess.CompletedProcess(command,1 if {cleanup_fails!r} and 'delete' in command else 0,text,'')\n" + + "with mock.patch.object(run,'ROOT',root),mock.patch.object(run,'run',fake)," + + f"mock.patch.dict(os.environ,{{'OPENSHELL_BIN':{str(cli)!r},'OPENSHELL_E2E_KUBE_CONTEXT':'fake-disposable'}},clear=True)," + + "mock.patch.object(sys,'argv',['run.py']): run.main()\n" + ) + process = subprocess.run( + [sys.executable, str(probe)], + capture_output=True, + text=True, + timeout=5, + ) + self.assertEqual(process.returncode, 143, process.stderr) + calls = [ + json.loads(line) + for line in (root / "calls.jsonl").read_text().splitlines() + ] + self.assertTrue( + any( + "delete" in command and "namespace" in command + for command in calls + ) + ) + if cleanup_fails: + self.assertIn("collector cleanup failed", process.stderr) + + def test_cleanup_does_not_delete_a_namespace_owned_by_another_run(self): + with tempfile.TemporaryDirectory() as work: + result = subprocess.CompletedProcess( + [], + 0, + json.dumps( + {"metadata": {"labels": {"openshell.ai/trace-test-run": "other"}}} + ), + "", + ) + with mock.patch.object(run, "run", return_value=result) as invoked: + error = run.cleanup_namespace( + ["kubectl"], "test-namespace", "this-run", Path(work) + ) + self.assertIn("ownership", error) + self.assertFalse( + any("delete" in call.args[0] for call in invoked.call_args_list) + ) + + def test_forced_shutdown_reaches_nested_probe_cleanup(self): + for number in [None, signal.SIGINT, signal.SIGTERM]: + with self.subTest(signal=number), tempfile.TemporaryDirectory() as work: + root = Path(work) + cli = root / "cli" + ready = root / "create-ready" + heartbeat = root / "cleanup-heartbeat" + cli.write_text( + f"#!{sys.executable}\n" + "import os,signal,sys,time\nfrom pathlib import Path\n" + + f"root=Path({str(root)!r})\n" + + "operation=sys.argv[2]\n" + + "(root/(operation+'-pid')).write_text(str(os.getpid()))\n" + + "if operation=='create': (root/'create-ready').write_text('ready')\n" + + "if operation=='delete': signal.signal(signal.SIGTERM,signal.SIG_IGN)\n" + + "while True:\n" + + " if operation=='delete': (root/'cleanup-heartbeat').write_text(str(time.monotonic_ns()))\n" + + " time.sleep(0.02)\n" + ) + cli.chmod(0o755) + traps = "\n".join( + line + for line in (run.ROOT / "e2e/with-kube-gateway.sh") + .read_text() + .splitlines() + if line.startswith("trap ") + ) + script = root / "harness.sh" + script.write_text( + "cleanup() { :; }\n" + + traps + + "\necho 'probe starting'\n" + + f"{shlex.quote(sys.executable)} {shlex.quote(run.__file__)} --check\n" + ) + parent = root / "parent.py" + parent.write_text( + f"import sys\nsys.path.insert(0,{str(Path(run.__file__).parent)!r})\nimport run\n" + + "with run.termination_signals():\n" + + f" run.run(['bash',{str(script)!r}],timeout={0.7 if number is None else 30},termination_grace=0.5,log_path=__import__('pathlib').Path({str(root / 'harness.log')!r}))\n" + ) + env = os.environ.copy() + env.update( + OPENSHELL_BIN=str(cli), + OPENSHELL_E2E_KUBE_CONTEXT="fake-disposable", + OPENSHELL_E2E_KUBE_NAMESPACE="fake-gateway", + OPENSHELL_E2E_TRACING_EVIDENCE=str(root), + OPENSHELL_E2E_TRACING_NAMESPACE="fake-collector", + OPENSHELL_E2E_TRACING_GATEWAY="fake-gateway", + ) + process = subprocess.Popen( + [sys.executable, str(parent)], + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + try: + wait_for(ready) + if number is not None: + os.kill(process.pid, number) + stdout, stderr = process.communicate(timeout=5) + self.assertEqual( + process.returncode, + 1 if number is None else 128 + number, + stdout + stderr, + ) + self.assertTrue( + heartbeat.exists(), + "the actual probe did not enter nested cleanup", + ) + self.assert_child_stopped(heartbeat) + self.assertIn("probe starting", (root / "harness.log").read_text()) + finally: + if process.poll() is None: + process.kill() + process.wait(timeout=5) + for name in ["create-pid", "delete-pid"]: + stop_test_child(root / name) + + +if __name__ == "__main__": + unittest.main() diff --git a/e2e/kubernetes/tracing/run.py b/e2e/kubernetes/tracing/run.py new file mode 100644 index 0000000000..d5305053e4 --- /dev/null +++ b/e2e/kubernetes/tracing/run.py @@ -0,0 +1,590 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Verify gateway and Kubernetes compute-driver traces at a real OTLP collector.""" + +import argparse +import hashlib +import json +import os +import re +import signal +import subprocess +import sys +import tempfile +import time +import uuid +from contextlib import contextmanager, nullcontext, suppress +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[3] +HERE = Path(__file__).resolve().parent +COLLECTOR_IMAGE = "otel/opentelemetry-collector-contrib:0.164.0" +WORKLOAD_IMAGE = ( + "ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim" + "@sha256:9a59bb7206905ccaae4f7dab222fbac47c125a21e5fc16f43f427cd6c940ade3" +) +GATEWAY_SERVICE = "openshell-e2e-gateway" +GATEWAY_CREATE = "openshell.v1.OpenShell/CreateSandbox" +DRIVER_VALIDATE = "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate" +DRIVER_CREATE = "openshell.compute.v1.ComputeDriver/CreateSandbox" + + +def stop_process_group(process, grace): + """Give the owned harness time to tear down, then stop remaining children.""" + deadline = time.monotonic() + grace + with suppress(ProcessLookupError): + os.killpg(process.pid, signal.SIGTERM) + try: + process.wait(timeout=grace) + # The shell may exit before its probe finishes cleanup. Keep that + # cleanup within the same grace period, even after reaping the shell. + while time.monotonic() < deadline: + try: + os.killpg(process.pid, 0) + except ProcessLookupError: + break + time.sleep(min(0.05, max(0, deadline - time.monotonic()))) + except subprocess.TimeoutExpired: + pass + finally: + with suppress(ProcessLookupError): + os.killpg(process.pid, signal.SIGKILL) + process.wait(timeout=5) + + +def run( + command, + *, + timeout=60, + check=True, + log_path=None, + termination_grace=10, + own_group=True, + **kwargs, +): + input_text = kwargs.pop("input", None) + output = log_path.open("w", buffering=1) if log_path else nullcontext(None) + with output as log: + process = subprocess.Popen( + command, + text=True, + stdin=subprocess.PIPE if input_text is not None else None, + stdout=log if log else subprocess.PIPE, + stderr=subprocess.STDOUT if log else subprocess.PIPE, + start_new_session=own_group, + **kwargs, + ) + try: + stdout, stderr = process.communicate(input=input_text, timeout=timeout) + except BaseException: + if own_group: + stop_process_group(process, termination_grace) + else: + # Only the outer harness owns this group. A command's timeout + # must not terminate the probe or its teardown. + process.terminate() + try: + process.wait(timeout=termination_grace) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=5) + process.communicate(timeout=5) + raise + if own_group: + # A shell can finish while background commands still own sockets. + # Reclaim those descendants on successful and failing exits too. + stop_process_group(process, termination_grace) + result = subprocess.CompletedProcess( + command, process.returncode, stdout or "", stderr or "" + ) + if check and result.returncode: + raise RuntimeError( + f"{command[0]} exited {result.returncode}:\n" + f"{result.stdout}\n{result.stderr}" + ) + return result + + +@contextmanager +def termination_signals(): + """Turn handled cancellation into unwinding, including collector teardown.""" + signals = (signal.SIGINT, signal.SIGTERM) + previous = {number: signal.getsignal(number) for number in signals} + + def interrupted(number, _frame): + # A second normal cancellation must not interrupt bounded cleanup. + for signum in signals: + signal.signal(signum, signal.SIG_IGN) + raise SystemExit(128 + number) + + for number in signals: + signal.signal(number, interrupted) + try: + yield + finally: + for number, handler in previous.items(): + signal.signal(number, handler) + + +def capture(command, destination, *, own_group=True): + """Preserve diagnostics even when a failing cluster cannot answer.""" + try: + result = run(command, check=False, own_group=own_group) + destination.write_text(result.stdout + result.stderr) + except (OSError, subprocess.TimeoutExpired) as error: + destination.write_text(f"Diagnostic capture failed: {error}\n") + + +def attributes(items): + result = {} + for item in items: + key = item["key"] + if key in result: + raise ValueError(f"duplicate resource attribute {key}") + result[key] = item.get("value", {}).get("stringValue") + return result + + +def received_spans(logs): + """Read OTLP JSON batches, ignoring the collector's own diagnostic records.""" + spans = [] + for line in logs.splitlines(): + try: + batch = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(batch, dict) or "resourceSpans" not in batch: + continue + for resource in batch["resourceSpans"]: + attrs = attributes(resource.get("resource", {}).get("attributes", [])) + for scope in resource.get("scopeSpans", []): + spans.extend((span, attrs) for span in scope.get("spans", [])) + return spans + + +def valid_id(value, digits): + return ( + isinstance(value, str) + and re.fullmatch(rf"[0-9a-fA-F]{{{digits}}}", value) is not None + and int(value, 16) != 0 + ) + + +def witnesses(logs, gateway_name): + """Require correctly attributed successful operations, not arbitrary traffic.""" + expected = { + "gateway": (GATEWAY_SERVICE, GATEWAY_CREATE), + "driver_validation": ("openshell-driver-kubernetes", DRIVER_VALIDATE), + "driver_create": ("openshell-driver-kubernetes", DRIVER_CREATE), + } + found = {key: [] for key in expected} + for span, resource in received_spans(logs): + for role, (service, operation) in expected.items(): + if ( + resource.get("service.name") != service + or span.get("name") != operation + or resource.get("openshell.gateway.name") != gateway_name + or resource.get("openshell.gateway.compute_driver") != "kubernetes" + or not resource.get("service.version") + or not valid_id(span.get("traceId"), 32) + or not valid_id(span.get("spanId"), 16) + or span.get("status", {}).get("code", 0) not in (0, 1) + ): + continue + found[role].append( + { + "trace_id": span["traceId"].lower(), + "span_id": span["spanId"].lower(), + "parent_span_id": span.get("parentSpanId", "").lower(), + "operation": operation, + "resource": resource, + } + ) + if not all(found.values()): + missing = ", ".join(key for key, value in found.items() if not value) + raise ValueError(f"collector missing expected spans/resources: {missing}") + # Validation is a synchronous part of this public CreateSandbox request. + # Actual provisioning may continue asynchronously and need not share that root. + for gateway in found["gateway"]: + for driver in found["driver_validation"]: + if gateway["trace_id"] == driver["trace_id"]: + return { + "gateway": gateway, + "driver_validation": driver, + "driver_create": found["driver_create"][0], + } + raise ValueError("gateway CreateSandbox and driver validation have no shared trace") + + +def wait_for_witnesses(read_logs, gateway_name, timeout=60): + deadline = time.monotonic() + timeout + last_error = "collector has not been read" + while True: + try: + return witnesses(read_logs(), gateway_name) + except ValueError as error: + last_error = str(error) + if time.monotonic() >= deadline: + raise RuntimeError(f"trace receipt deadline exceeded: {last_error}") + time.sleep(1) + + +def confirm_receipt(evidence, gateway_name): + # A shell EXIT trap can return zero even if the probe was never reached. + # Require the child's receipt and independently recheck the captured data. + try: + receipt = json.loads((evidence / "receipt.json").read_text()) + received = witnesses((evidence / "collector.log").read_text(), gateway_name) + except (OSError, ValueError) as error: + raise RuntimeError(f"missing or invalid collector receipt: {error}") from error + if receipt != received: + raise RuntimeError("collector receipt does not match the captured spans") + + +def collector_manifest(namespace): + config = { + "receivers": {"otlp": {"protocols": {"grpc": {"endpoint": "0.0.0.0:4317"}}}}, + "exporters": { + "file": {"path": "/dev/stdout", "format": "json", "flush_interval": "1s"} + }, + "service": { + "pipelines": {"traces": {"receivers": ["otlp"], "exporters": ["file"]}} + }, + } + metadata = {"name": "collector", "namespace": namespace} + return { + "apiVersion": "v1", + "kind": "List", + "items": [ + { + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": metadata, + "data": {"collector.json": json.dumps(config)}, + }, + { + "apiVersion": "apps/v1", + "kind": "Deployment", + "metadata": metadata, + "spec": { + "replicas": 1, + "selector": {"matchLabels": {"app": "trace-receipt-collector"}}, + "template": { + "metadata": {"labels": {"app": "trace-receipt-collector"}}, + "spec": { + "automountServiceAccountToken": False, + "containers": [ + { + "name": "collector", + "image": COLLECTOR_IMAGE, + "args": ["--config=/etc/collector/collector.json"], + "ports": [{"name": "otlp", "containerPort": 4317}], + "readinessProbe": {"tcpSocket": {"port": "otlp"}}, + "resources": { + "requests": {"cpu": "100m", "memory": "64Mi"}, + "limits": {"cpu": "500m", "memory": "256Mi"}, + }, + # Reopening /dev/stdout needs access to the runtime's + # pipe. This disposable fixture has no token or caps. + "securityContext": { + "runAsUser": 0, + "allowPrivilegeEscalation": False, + "readOnlyRootFilesystem": True, + "capabilities": {"drop": ["ALL"]}, + }, + "volumeMounts": [ + { + "name": "config", + "mountPath": "/etc/collector", + "readOnly": True, + } + ], + } + ], + "volumes": [ + {"name": "config", "configMap": {"name": "collector"}} + ], + }, + }, + }, + }, + { + "apiVersion": "v1", + "kind": "Service", + "metadata": metadata, + "spec": { + "selector": {"app": "trace-receipt-collector"}, + "ports": [{"name": "otlp", "port": 4317, "targetPort": "otlp"}], + }, + }, + ], + } + + +def gateway_values(namespace, gateway_name): + return { + "server": { + "name": gateway_name, + "otlp": { + "endpoint": f"http://collector.{namespace}.svc.cluster.local:4317", + "serviceName": GATEWAY_SERVICE, + }, + "extraEnv": [{"name": "OTEL_TRACES_SAMPLER", "value": "always_on"}], + } + } + + +def kubectl(namespace): + context = os.environ.get("OPENSHELL_E2E_KUBE_CONTEXT") + if not context: + raise RuntimeError( + "set OPENSHELL_E2E_KUBE_CONTEXT to a disposable test cluster" + ) + command = ["kubectl", "--context", context] + return [*command, "-n", namespace] + + +def check_receipt(evidence, namespace, gateway_name, cli): + k = kubectl(namespace) + name = "trace-" + uuid.uuid4().hex[:12] + + def read_logs(): + logs = run( + [*k, "logs", "deployment/collector", "-c", "collector"], own_group=False + ).stdout + (evidence / "collector.log").write_text(logs) + return logs + + try: + create = run( + [ + cli, + "sandbox", + "create", + "--detach", + "--name", + name, + "--from", + WORKLOAD_IMAGE, + ], + timeout=360, + own_group=False, + ) + (evidence / "create.log").write_text(create.stdout + create.stderr) + executed = run( + [cli, "sandbox", "exec", name, "--", "sh", "-c", "printf collector-probe"], + timeout=360, + own_group=False, + ) + if "collector-probe" not in executed.stdout: + raise RuntimeError("sandbox probe did not execute successfully") + receipt = wait_for_witnesses(read_logs, gateway_name) + (evidence / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n") + print( + "Collector received gateway and Kubernetes driver spans with resource attribution." + ) + finally: + deleted = run( + [cli, "sandbox", "delete", name], timeout=120, check=False, own_group=False + ) + (evidence / "delete.log").write_text(deleted.stdout + deleted.stderr) + capture( + [ + *kubectl(os.environ["OPENSHELL_E2E_KUBE_NAMESPACE"]), + "get", + "pods", + "-o", + "json", + ], + evidence / "gateway-pods.json", + own_group=False, + ) + capture( + [*k, "logs", "deployment/collector", "-c", "collector"], + evidence / "collector-final.log", + own_group=False, + ) + + +def cleanup_namespace(k, namespace, owner, evidence): + """Delete only a namespace carrying this run's ownership marker.""" + try: + observed = run( + [*k, "get", "namespace", namespace, "--ignore-not-found", "-o", "json"], + timeout=15, + check=False, + ) + (evidence / "namespace-before-cleanup.json").write_text( + observed.stdout + observed.stderr + ) + if observed.returncode: + raise RuntimeError("could not verify collector namespace ownership") + if not observed.stdout.strip(): + (evidence / "collector-cleanup.log").write_text( + "Collector namespace is absent.\n" + ) + return None + metadata = json.loads(observed.stdout)["metadata"] + if metadata.get("labels", {}).get("openshell.ai/trace-test-run") != owner: + raise RuntimeError("collector namespace ownership does not match this run") + cleanup = run( + [ + *k, + "delete", + "namespace", + namespace, + "--ignore-not-found", + "--wait=true", + "--timeout=60s", + ], + timeout=90, + check=False, + ) + (evidence / "collector-cleanup.log").write_text(cleanup.stdout + cleanup.stderr) + if cleanup.returncode: + raise RuntimeError("collector namespace deletion failed") + except ( + OSError, + ValueError, + KeyError, + subprocess.TimeoutExpired, + RuntimeError, + ) as error: + (evidence / "collector-cleanup-error.log").write_text(str(error) + "\n") + return f"collector cleanup failed for {namespace}: {error}" + return None + + +def execute(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--check", action="store_true", help=argparse.SUPPRESS) + args = parser.parse_args() + cli = str( + Path(os.getenv("OPENSHELL_BIN", ROOT / "target/debug/openshell")).resolve() + ) + if not Path(cli).is_file(): + raise RuntimeError("build openshell-cli first or set OPENSHELL_BIN") + if args.check: + check_receipt( + Path(os.environ["OPENSHELL_E2E_TRACING_EVIDENCE"]), + os.environ["OPENSHELL_E2E_TRACING_NAMESPACE"], + os.environ["OPENSHELL_E2E_TRACING_GATEWAY"], + cli, + ) + return + if os.getenv("OPENSHELL_E2E_KUBE_DB_SCENARIOS", "0") != "0": + raise RuntimeError("the tracing probe uses one isolated gateway install") + owner = uuid.uuid4().hex + suffix = owner[:12] + namespace = "openshell-tracing-" + suffix + gateway_name = "tracing-" + suffix + evidence = ROOT / "results/kubernetes-tracing" / suffix + evidence.mkdir(parents=True) + k = kubectl(namespace) # Validate context before any cluster mutation. + manifest = json.dumps(collector_manifest(namespace), indent=2) + (evidence / "collector.json").write_text(manifest + "\n") + (evidence / "inputs.json").write_text( + json.dumps( + { + "cli_path": cli, + "cli_sha256": hashlib.sha256(Path(cli).read_bytes()).hexdigest(), + "source_head": run( + ["git", "-C", str(ROOT), "rev-parse", "HEAD"] + ).stdout.strip(), + "source_status": run( + ["git", "-C", str(ROOT), "status", "--short"] + ).stdout, + "images": { + key: os.environ.get(key) + for key in ( + "IMAGE_TAG", + "GATEWAY_IMAGE", + "SUPERVISOR_IMAGE", + "SANDBOX_IMAGE", + ) + }, + "collector_image": COLLECTOR_IMAGE, + "context": os.environ["OPENSHELL_E2E_KUBE_CONTEXT"], + }, + indent=2, + ) + + "\n" + ) + creation_started = False + try: + creation_started = True + namespace_manifest = { + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": namespace, + "labels": {"openshell.ai/trace-test-run": owner}, + }, + } + run([*k, "create", "-f", "-"], input=json.dumps(namespace_manifest)) + run([*k, "apply", "-f", "-"], input=manifest) + run( + [*k, "rollout", "status", "deployment/collector", "--timeout=120s"], + timeout=150, + ) + with tempfile.TemporaryDirectory(prefix="openshell-tracing-") as work: + values = Path(work) / "tracing.json" + values.write_text(json.dumps(gateway_values(namespace, gateway_name))) + (evidence / "values.json").write_text(values.read_text()) + env = os.environ.copy() + existing = env.get("OPENSHELL_E2E_KUBE_EXTRA_VALUES", "") + env.update( + OPENSHELL_BIN=cli, + OPENSHELL_E2E_KUBE_EXTRA_VALUES=":".join( + filter(None, [existing, str(values)]) + ), + OPENSHELL_E2E_TRACING_EVIDENCE=str(evidence), + OPENSHELL_E2E_TRACING_NAMESPACE=namespace, + OPENSHELL_E2E_TRACING_GATEWAY=gateway_name, + ) + result = run( + [ + "bash", + str(ROOT / "e2e/with-kube-gateway.sh"), + sys.executable, + str(HERE / "run.py"), + "--check", + ], + timeout=1200, + check=False, + log_path=evidence / "gateway-harness.log", + termination_grace=180, + env=env, + ) + if result.returncode: + raise RuntimeError( + f"Kubernetes trace validation failed; evidence: {evidence}" + ) + confirm_receipt(evidence, gateway_name) + finally: + failed = sys.exc_info()[0] is not None + if creation_started: + for filename, command in [ + ( + "collector-final.log", + ["logs", "deployment/collector", "-c", "collector"], + ), + ("collector-pods.json", ["get", "pods", "-o", "json"]), + ]: + capture([*k, *command], evidence / filename) + error = cleanup_namespace(k, namespace, owner, evidence) + if error and failed: + print(error, file=sys.stderr) + elif error: + raise RuntimeError(error) + print(f"Trace receipt evidence: {evidence}") + + +def main(): + with termination_signals(): + execute() + + +if __name__ == "__main__": + main() diff --git a/e2e/kubernetes/tracing/run_test.py b/e2e/kubernetes/tracing/run_test.py new file mode 100644 index 0000000000..8019171fc7 --- /dev/null +++ b/e2e/kubernetes/tracing/run_test.py @@ -0,0 +1,288 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""False-positive controls for the collector-backed trace receipt check.""" + +import copy +import json +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +import run + + +def batch(service, operation, trace="1" * 32, **overrides): + resource = { + "service.name": service, + "service.version": "test-version", + "openshell.gateway.name": "test-gateway", + "openshell.gateway.compute_driver": "kubernetes", + } + resource.update(overrides) + return { + "resourceSpans": [ + { + "resource": { + "attributes": [ + {"key": key, "value": {"stringValue": value}} + for key, value in resource.items() + ] + }, + "scopeSpans": [ + { + "spans": [ + {"name": operation, "traceId": trace, "spanId": "2" * 16} + ] + } + ], + } + ] + } + + +def successful_batches(): + return [ + batch(run.GATEWAY_SERVICE, run.GATEWAY_CREATE), + batch("openshell-driver-kubernetes", run.DRIVER_VALIDATE), + batch("openshell-driver-kubernetes", run.DRIVER_CREATE, trace="3" * 32), + ] + + +def logs(batches): + return "\n".join(json.dumps(item) for item in batches) + + +class ReceiptTest(unittest.TestCase): + @unittest.skipUnless( + shutil.which("kubectl"), + "kubectl is required for the local config-merge control", + ) + def test_kubeconfig_single_file_and_file_list_select_the_same_context(self): + with tempfile.TemporaryDirectory() as work: + root = Path(work) + cluster = { + "apiVersion": "v1", + "kind": "Config", + "clusters": [ + { + "name": "review-cluster", + "cluster": {"server": "http://127.0.0.1:9"}, + } + ], + } + context = { + "apiVersion": "v1", + "kind": "Config", + "contexts": [ + {"name": "review-context", "context": {"cluster": "review-cluster"}} + ], + "current-context": "review-context", + } + single, first, second = ( + root / "single.json", + root / "cluster.json", + root / "context.json", + ) + single.write_text(json.dumps(cluster | context)) + first.write_text(json.dumps(cluster)) + second.write_text(json.dumps(context)) + explicit = subprocess.run( + [ + "kubectl", + "--kubeconfig", + str(single), + "--context", + "review-context", + "config", + "view", + "--minify", + "-o", + "json", + ], + check=True, + capture_output=True, + text=True, + timeout=10, + ) + expected = json.loads(explicit.stdout) + for config in [str(single), os.pathsep.join([str(first), str(second)])]: + with ( + self.subTest(kubeconfig=config), + mock.patch.dict( + os.environ, + { + "KUBECONFIG": config, + "OPENSHELL_E2E_KUBE_CONTEXT": "review-context", + }, + ), + ): + # config view is client-only; the fake API endpoint is never contacted. + result = subprocess.run( + [ + *run.kubectl("fixture-only"), + "config", + "view", + "--minify", + "-o", + "json", + ], + check=True, + capture_output=True, + text=True, + timeout=10, + ) + self.assertEqual(json.loads(result.stdout), expected) + + def test_gateway_and_driver_receipt_with_shared_validation_context(self): + receipt = run.witnesses(logs(successful_batches()), "test-gateway") + self.assertEqual(receipt["gateway"]["trace_id"], "1" * 32) + self.assertEqual(receipt["driver_create"]["trace_id"], "3" * 32) + + def test_unrelated_collector_diagnostics_cannot_supply_spans(self): + for value in ("", "collector ready", '{"level":"info","msg":"ready"}'): + with self.subTest(log=value), self.assertRaisesRegex(ValueError, "missing"): + run.witnesses(value, "test-gateway") + + def test_each_required_operation_must_arrive(self): + batches = successful_batches() + for index in range(3): + with ( + self.subTest(missing=index), + self.assertRaisesRegex(ValueError, "missing"), + ): + run.witnesses( + logs(batches[:index] + batches[index + 1 :]), "test-gateway" + ) + + def test_wrong_service_cannot_supply_driver_operation(self): + batches = successful_batches() + batches[1] = batch(run.GATEWAY_SERVICE, run.DRIVER_VALIDATE) + with self.assertRaisesRegex(ValueError, "driver_validation"): + run.witnesses(logs(batches), "test-gateway") + + def test_unrelated_or_misattributed_gateway_cannot_satisfy_probe(self): + for key, value in [ + ("openshell.gateway.name", "previous-run"), + ("openshell.gateway.compute_driver", "docker"), + ("service.version", ""), + ]: + batches = successful_batches() + batches[2] = batch( + "openshell-driver-kubernetes", run.DRIVER_CREATE, **{key: value} + ) + with ( + self.subTest(attribute=key), + self.assertRaisesRegex(ValueError, "driver_create"), + ): + run.witnesses(logs(batches), "test-gateway") + + def test_ids_must_be_valid_and_nonzero(self): + for key, value in [ + ("traceId", "0" * 32), + ("traceId", "x" * 32), + ("spanId", "0" * 16), + ]: + batches = successful_batches() + batches[1]["resourceSpans"][0]["scopeSpans"][0]["spans"][0][key] = value + with ( + self.subTest(field=key), + self.assertRaisesRegex(ValueError, "driver_validation"), + ): + run.witnesses(logs(batches), "test-gateway") + + def test_failed_operations_cannot_satisfy_successful_probe(self): + batches = successful_batches() + batches[2]["resourceSpans"][0]["scopeSpans"][0]["spans"][0]["status"] = { + "code": 2 + } + with self.assertRaisesRegex(ValueError, "driver_create"): + run.witnesses(logs(batches), "test-gateway") + + def test_broken_gateway_to_driver_propagation_is_rejected(self): + batches = successful_batches() + batches[1] = batch( + "openshell-driver-kubernetes", run.DRIVER_VALIDATE, trace="4" * 32 + ) + with self.assertRaisesRegex(ValueError, "shared trace"): + run.witnesses(logs(batches), "test-gateway") + + def test_duplicate_resource_key_is_rejected(self): + batches = successful_batches() + attrs = batches[1]["resourceSpans"][0]["resource"]["attributes"] + attrs.append(copy.deepcopy(attrs[0])) + with self.assertRaisesRegex(ValueError, "duplicate resource"): + run.witnesses(logs(batches), "test-gateway") + + def test_missing_receipts_fail_at_deadline(self): + with self.assertRaisesRegex(RuntimeError, "deadline exceeded.*missing"): + run.wait_for_witnesses(lambda: "", "test-gateway", timeout=0) + + def test_persisted_receipt_is_rechecked_against_collector_data(self): + with tempfile.TemporaryDirectory() as work: + evidence = Path(work) + collected = logs(successful_batches()) + (evidence / "collector.log").write_text(collected) + receipt = run.witnesses(collected, "test-gateway") + (evidence / "receipt.json").write_text(json.dumps(receipt)) + run.confirm_receipt(evidence, "test-gateway") + + def test_receipt_with_different_trace_is_rejected(self): + with tempfile.TemporaryDirectory() as work: + evidence = Path(work) + collected = logs(successful_batches()) + (evidence / "collector.log").write_text(collected) + receipt = run.witnesses(collected, "test-gateway") + receipt["gateway"]["trace_id"] = "f" * 32 + (evidence / "receipt.json").write_text(json.dumps(receipt)) + with self.assertRaisesRegex(RuntimeError, "does not match"): + run.confirm_receipt(evidence, "test-gateway") + + def test_harness_exit_zero_without_running_probe_cannot_pass(self): + # Reproduces the macOS Bash failure: an EXIT trap can leave a zero + # status although the gateway harness never reached the probe. + with tempfile.TemporaryDirectory() as work: + root = Path(work) + cli = root / "openshell" + cli.write_text("unused test CLI") + environment = { + "OPENSHELL_BIN": str(cli), + "OPENSHELL_E2E_KUBE_CONTEXT": "kind-disposable-test", + } + owned_namespace = {} + + def fake_run(command, **kwargs): + if "create" in command and "-f" in command: + owned_namespace.update(json.loads(kwargs["input"])) + text = ( + json.dumps(owned_namespace) + if "get" in command and "namespace" in command + else "" + ) + return subprocess.CompletedProcess(command, 0, text, "") + + with ( + mock.patch.object(run, "ROOT", root), + mock.patch.dict(os.environ, environment, clear=True), + mock.patch.object(sys, "argv", ["run.py"]), + mock.patch.object( + run, + "run", + side_effect=fake_run, + ) as invoked, + self.assertRaisesRegex(RuntimeError, "receipt"), + ): + run.main() + self.assertTrue( + any("delete" in call.args[0] for call in invoked.call_args_list), + "collector cleanup must run when the receipt is absent", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh index 31c0771dc8..504622ab80 100755 --- a/e2e/with-kube-gateway.sh +++ b/e2e/with-kube-gateway.sh @@ -187,7 +187,7 @@ verify_gateway_config_rollout() { "${helm_values_args[@]}" \ --set "fullnameOverride=openshell" \ "${helm_extra_args[@]}" \ - "${helm_post_renderer_args[@]}" \ + ${helm_post_renderer_args[@]+"${helm_post_renderer_args[@]}"} \ --set-string 'gatewayConfig.openshell\.gateway.log_level=debug' \ --wait --timeout 5m @@ -671,6 +671,8 @@ cleanup() { rm -rf "${WORKDIR}" 2>/dev/null || true } trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM # --- DB-scenario helpers (used only when OPENSHELL_E2E_KUBE_DB_SCENARIOS=1) --- @@ -734,7 +736,7 @@ run_scenario() { "${GATEWAY_HELM_IMAGE_ARGS[@]}" \ "${SUPERVISOR_HELM_IMAGE_ARGS[@]}" \ "${SANDBOX_RUNTIME_HELM_IMAGE_ARGS[@]}" \ - "${helm_post_renderer_args[@]}" \ + ${helm_post_renderer_args[@]+"${helm_post_renderer_args[@]}"} \ "$@" \ --wait --timeout 5m HELM_INSTALLED=1 @@ -1542,7 +1544,7 @@ else "${SUPERVISOR_HELM_IMAGE_ARGS[@]}" \ "${SANDBOX_RUNTIME_HELM_IMAGE_ARGS[@]}" \ "${helm_extra_args[@]}" \ - "${helm_post_renderer_args[@]}" \ + ${helm_post_renderer_args[@]+"${helm_post_renderer_args[@]}"} \ --wait --timeout 5m HELM_INSTALLED=1 diff --git a/mise.toml b/mise.toml index 10c964e956..c2bed709bd 100644 --- a/mise.toml +++ b/mise.toml @@ -74,7 +74,7 @@ DOCKER_BUILDKIT = "1" [vars] # Python paths to include in formatting/linting -python_paths = "python/ tasks/scripts/*.py e2e/mcp-conformance/*.py e2e/kubernetes/app-listener/*.py deploy/sbom/*.py" +python_paths = "python/ tasks/scripts/*.py e2e/mcp-conformance/*.py e2e/kubernetes/app-listener/*.py e2e/kubernetes/tracing/*.py deploy/sbom/*.py" [task_config] includes = ["tasks/*.toml"] diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 23d58efa25..8ee69d2c65 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -1591,6 +1591,7 @@ credential failures. | Collector receives no OCSF log records, or `openshell.ocsf.gap` records | `[openshell.gateway.otlp.ocsf]` missing or without `enabled = true`, an unreachable endpoint or one that does not speak the table's `protocol`, or export failures and queue pressure | Check `[openshell.gateway.otlp.ocsf]` for `enabled = true` and the endpoint it resolves to, the gateway's `OTLP exporting enabled` startup log for `ocsf_endpoint` and `ocsf_protocol`, export warnings, and `openshell_ocsf_otlp_*` metrics. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). | | Gateway OCSF JSONL stops growing, has gaps, or is rejected by a SIEM | File errors, queue pressure, or shipper falling behind retention | Inspect gateway warnings and `openshell_ocsf_log_*` metrics; check `[openshell.gateway.ocsf_log]`, directory permissions, free space, per-replica paths, and shipper rotation checkpoints. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). | | `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with `OPENSHELL_DEV_OTLP_ENDPOINT` set but no collector is listening, for example because `helm:k3s:forward` stopped | Start the collector forwarding again, or restart the gateway task without `OPENSHELL_DEV_OTLP_ENDPOINT`; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` | +| Collector is ready but expected gateway or compute-driver traces are missing | Readiness confirms a listener, not receipt of the expected operations | Check the configured endpoint, the gateway and driver service identities and resource attributes, sampling, and export errors. Require an observed operation in the collector; distinguish collector receipt from backend indexing and retention. | | Sandbox log shows `OTLP agent trace relay inactive: not enabled by the gateway` at startup, then `agent OTLP export attempted but the agent trace relay is not enabled` when the agent exports | The gateway does not enable `[openshell.gateway.otlp.agent]`, so the supervisor refuses the workload's OTLP relay address | Set `enabled = true` in `[openshell.gateway.otlp.agent]` (Helm: `gatewayConfig."openshell.gateway.otlp.agent".enabled`) with an endpoint from that table or `[openshell.gateway.otlp]`, and recreate the sandbox; `openshell logs --source sandbox` then shows `OTLP agent trace relay enabled` | | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | | Docker or Podman sandbox never registers | Wrong gateway endpoint, unavailable host networking, or supervisor startup failure | Gateway logs and supervisor container logs | diff --git a/tasks/test.toml b/tasks/test.toml index 2839c1cf0c..6af117e0ea 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -254,6 +254,13 @@ description = "Run Kubernetes e2e against Agent Sandbox v1alpha1" env = { AGENT_SANDBOX_VERSION = "v0.4.6" } run = "e2e/rust/e2e-kubernetes.sh" +["e2e:kubernetes:tracing"] +description = "Verify gateway and Kubernetes driver trace receipt at an isolated OTLP collector on a disposable test cluster" +run = [ + "uv run --no-project python -m unittest discover -s e2e/kubernetes/tracing -p '*_test.py'", + "uv run --no-project python e2e/kubernetes/tracing/run.py", +] + ["e2e:kubernetes:agent-sandbox-versions"] description = "Run Kubernetes e2e against Agent Sandbox v1beta1 and v1alpha1" run = [