bug(supervisor): gate rollout readiness on gateway attachment #4960

User Story

We are developing and testing OpenShell's shared-supervisor HA deployment in #4864. We need supervisor upgrades to retain serving members when their replacements cannot attach to the gateway, so running sandboxes keep their processes and in-memory state. This fast follow tracks the readiness gap identified during review; we have not directly reproduced the failed-attachment rollout in a live cluster.

Problem Statement

The shared supervisor's Kubernetes readiness probe checks the same TCP health listener as liveness. That listener opens before gateway authentication and pool attachment. A replacement Pod can therefore become Ready while its gateway connection is refused, unreachable, or unauthenticated.

The Deployment uses RollingUpdate with maxSurge: 1 and maxUnavailable: 0. False readiness allows Kubernetes to retire healthy serving supervisors before their replacements can participate in the pool. The PDB does not prevent Deployment-managed rolling replacement. The PR base used Recreate; rolling updates introduce an availability expectation that requires a stronger readiness signal.

Fast follow to #4864; inspected at f59e8dc94ce36225db81f88435de588b1b55ae72:

• Health listener starts before attachment.
• TCP readiness and liveness probes.
• Gateway acceptance updates connection state.

Impact / Why This Matters

A rollout can appear healthy while sandboxes lose supervision and exec/network access. Workloads may freeze. If an eligible supervisor takes over within the configured recovery deadline, their processes and memory survive. If recovery does not complete before the deadline (60 seconds by default), the runtime terminates the workload; later connectivity recovery does not restore that process or its memory.

Previously passing 1 → 2 → 1 scaling and abrupt-loss tests cover successful replacement attachment, not a replacement-only authentication/connectivity failure during rolling update.

Proposed Fix

Start with the simple approach agreed during review: readiness requires a currently accepted authenticated gateway stream and a member that is not draining. Before acceptance, after detected stream loss, and during drain, readiness fails. Keep process liveness independent of gateway availability so a gateway outage does not trigger supervisor restart loops.

Reuse the existing connection/drain state where practical. No compute-driver changes or new HA opt-in are needed. Durable membership-publication acknowledgement is outside this first fix: gateway acceptance is not a guarantee that durable membership has already been published.

Acceptance Criteria

• A newly started supervisor remains unready until the gateway accepts its stream.
• Readiness fails on detected disconnection/rejection and when draining begins; it recovers after a new stream is accepted.
• Liveness continues to succeed during gateway connection/authentication failures while the supervisor process is healthy.
• Unit/stream tests cover startup, acceptance, disconnect/rejection, reconnect, and drain; chart tests verify distinct readiness/liveness semantics.
• A rollout regression blocks only replacement attachment while existing supervisors remain connected. Replacements stay unready, serving members remain, and existing workloads retain working exec/network access, workload identity, and in-memory state throughout the stalled rollout.
• Restoring replacement attachment allows the rollout to complete and ownership to hand off without restarting workloads.
• Existing shared-supervisor scaling and abrupt-loss recovery tests still pass; update health-probe documentation and operational skills.

Reproduction Steps

Proposed reproduction, not yet executed:

1. Deploy OpenShell with supervisor_topology = "gateway", initially using a healthy supervisor pool. Create a sandbox with a long-running process and an in-memory counter; verify exec and policy-mediated network access.
2. Update only the supervisor Deployment's Pod template so replacements use an unreachable --gateway-endpoint or invalid supervisor credential. Keep the serving Pods' existing connections and the gateway unchanged.
3. Observe that replacements start their health listener and pass the existing TCP readiness probe despite never being accepted by the gateway.
4. Observe the Deployment retiring serving supervisors. Track workload access and process identity through the recovery deadline.
5. With the fix, repeat the scenario and verify the rollout stalls safely until replacement attachment is restored.

Environment

• OpenShell: shared-supervisor HA branch in feat(supervisor): recover shared instances with hash-based HA #4864, inspected at f59e8dc94.
• Deployment: OpenShell Helm chart on Kubernetes, gateway TLS enabled, supervisor_topology = "gateway".
• Existing local validation used Linux/k3d; the failed-attachment rollout itself has not been run.

Suggested UX (Notional)

The chart selects meaningful readiness by default, without an additional operator setting. For example, an HTTP readiness endpoint can report an unsuccessful status while the existing process-health probe continues to pass:

readinessProbe:
  httpGet:
    path: /readyz
    port: health
livenessProbe:
  tcpSocket:
    port: health

The exact endpoint is an implementation choice; the observable contract above is the requirement.