feat(kubernetes): configure sandbox-supervisor pod anti-affinity #4749

User Story

I need to run OpenShell sandboxes with the Kubernetes compute backend and keep each sandbox workload on a different node from its own supervisor. I want to configure this placement through the gateway and have it apply when sandboxes are created or restarted, without maintaining separate admission-webhook rules.

Problem Statement

The Kubernetes compute driver creates a sandbox workload pod and a separate supervisor pod, but currently exposes no operator setting to keep each pair on different nodes. Helm's top-level affinity setting only configures the gateway pod, and caller driver configuration rejects pod affinity fields.

Impact / Why This Matters

Operators who need node separation for each sandbox/supervisor pair must mutate pod admission or maintain downstream driver changes. This leaves placement policy outside the normal OpenShell gateway configuration and requires maintaining the same rule through sandbox creation and restart.

Proposed Design

Expose an operator-owned Kubernetes driver setting, sandbox_runtime.pod_anti_affinity, with off (default), preferred, and required modes. Each pair should match its own peer in the same namespace using node hostname topology. New runtime generations should use the current gateway setting. Required mode should wait when separation cannot be scheduled, without silently falling back to colocation.

This setting separates each sandbox from its own supervisor. It does not segregate all sandbox workloads from all supervisors or move already-running pods.

Suggested UX

[openshell.drivers.kubernetes.sandbox_runtime]
pod_anti_affinity = "required"

For Helm:

gatewayConfig:
  openshell.drivers.kubernetes:
    sandbox_runtime:
      pod_anti_affinity: required

Acceptance Criteria

• The default preserves existing placement behavior.
• Required mode places each sandbox and its own supervisor on different nodes, or waits for eligible capacity.
• Preferred mode permits colocation when only one eligible node exists.
• Stop/start applies the current gateway policy to both pods and preserves unrelated affinity.
• Caller driver configuration cannot weaken this operator-owned policy.
• Required rules and pair labels must survive admission before bootstrap scheduling gates are released.
• A verified unschedulable pair can recover after eligible capacity becomes available.
• Published configuration documentation explains scope, defaults, and Pending behavior.

Alternatives Considered

• An operator-managed mutating admission webhook can inject pod anti-affinity today, but requires configuring external admission infrastructure and maintaining its policy through both pod lifecycles.
• Node selectors alone choose a node pool but do not express separation from the paired pod. Helm affinity applies to the gateway workload.
• Middleware, gateway interceptors, and credential providers do not own Kubernetes pod scheduling.

Agent Investigation

The Kubernetes driver already labels workload and supervisor pods with a reserved pair identity and their respective roles. The implementation can render symmetric hostname anti-affinity without changing the portable sandbox API. A local implementation has passed driver unit tests and two-node k3d scenarios for creation, restart, mode transitions, capacity recovery, and preferred colocation.