gha-runner-scale-set: no working way to protect in-flight jobs from node drain / cloud node auto-upgrade (PDBs non-functional, safe-to-evict doesn't apply to drains)

[aandreev-akamai]

What happened / the gap

A cloud node auto-upgrade (AKS) cordoned + drained the nodes running our gha-runner-scale-set runners mid-job. In-flight jobs died with ##[error]The runner has received a shutdown signal. A node drain is a voluntary eviction, so the expected protection is a PodDisruptionBudget — but there is currently no working pod-level mechanism to protect a busy ARC v2 runner from a drain:

1. PDBs are non-functional on ARC v2. The runner pod's controller owner is the per-pod EphemeralRunner CR, which implements only a status subresource, no scale. The K8s disruption controller resolves budgets via the pod's direct owner's scale subresource, can't find one, and marks the PDB SyncFailed (currentHealthy: 0, expectedPods: 0). Confirmed on controller 0.14.2 and still true on master (EphemeralRunner CRD subresources: {status: {}}). This was raised in Add scale subresource to EphemeralRunnerSet CRD for PDB compatibility #4493 and closed as architecturally won't-fix.
2. cluster-autoscaler.kubernetes.io/safe-to-evict and karpenter.sh/do-not-disrupt don't help — they're honored only by the Cluster Autoscaler / Karpenter when choosing to remove a node, not by kubectl drain or a cloud node auto-upgrade drain, which go through the eviction API and honor only PDBs. Add ability to add annotations to Runner Pods once they start running a job #2562 (busy-annotation request) is therefore CA-scoped and doesn't cover this.

Net: for the very common case of managed node auto-upgrade (AKS/EKS/GKE), there is no supported way to let a busy runner finish its job before its node is drained.

What I'd like

A supported mechanism to protect busy runners from voluntary node disruption. Options, roughly in order of preference:

1. Make PDBs work — give EphemeralRunner a scale subresource (reporting 1/1), or reparent runner pods to a set-level owner (EphemeralRunnerSet) that has one, so a standard PodDisruptionBudget can protect runner pods. This is the cleanest fit with how drains already work. (Reconsider Add scale subresource to EphemeralRunnerSet CRD for PDB compatibility #4493 with the node-drain use case, not just the API-compat framing.)
2. Native busy marker + graceful drain — controller sets a configurable label/annotation when a job lands and clears it on completion (Add ability to add annotations to Runner Pods once they start running a job #2562), and on receiving SIGTERM from a drain the runner finishes (or re-queues) the current job within terminationGracePeriodSeconds instead of dying immediately.
3. At minimum, document that ARC v2 runners cannot be protected from node drains and that the only mitigation is at the infrastructure layer (maintenance windows / controlled upgrade cadence).

Related: #4493 (PDB scale subresource, closed won't-fix), #2562 (busy annotations for CA), #4148 (EphemeralRunner stuck Running after node drain).

Environment

• gha-runner-scale-set / gha-runner-scale-set-controller 0.14.2
• AKS, node auto-upgrade (drain via eviction API)

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[humh25]

536975948763 humbertohlgl El sept 3, 2026, 10:40 -0600, aandreev-akamai ***@***.***>, escribió:

…

aandreev-akamai created an issue (actions/actions-runner-controller#4618) What happened / the gap A cloud node auto-upgrade (AKS) cordoned + drained the nodes running our gha-runner-scale-set runners mid-job. In-flight jobs died with ##[error]The runner has received a shutdown signal. A node drain is a voluntary eviction, so the expected protection is a PodDisruptionBudget — but there is currently no working pod-level mechanism to protect a busy ARC v2 runner from a drain: 1. PDBs are non-functional on ARC v2. The runner pod's controller owner is the per-pod EphemeralRunner CR, which implements only a status subresource, no scale. The K8s disruption controller resolves budgets via the pod's direct owner's scale subresource, can't find one, and marks the PDB SyncFailed (currentHealthy: 0, expectedPods: 0). Confirmed on controller 0.14.2 and still true on master (EphemeralRunner CRD subresources: {status: {}}). This was raised in #4493 and closed as architecturally won't-fix. 2. cluster-autoscaler.kubernetes.io/safe-to-evict and karpenter.sh/do-not-disrupt don't help — they're honored only by the Cluster Autoscaler / Karpenter when choosing to remove a node, not by kubectl drain or a cloud node auto-upgrade drain, which go through the eviction API and honor only PDBs. #2562 (busy-annotation request) is therefore CA-scoped and doesn't cover this. Net: for the very common case of managed node auto-upgrade (AKS/EKS/GKE), there is no supported way to let a busy runner finish its job before its node is drained. What I'd like A supported mechanism to protect busy runners from voluntary node disruption. Options, roughly in order of preference: 1. Make PDBs work — give EphemeralRunner a scale subresource (reporting 1/1), or reparent runner pods to a set-level owner (EphemeralRunnerSet) that has one, so a standard PodDisruptionBudget can protect runner pods. This is the cleanest fit with how drains already work. (Reconsider #4493 with the node-drain use case, not just the API-compat framing.) 2. Native busy marker + graceful drain — controller sets a configurable label/annotation when a job lands and clears it on completion (#2562), and on receiving SIGTERM from a drain the runner finishes (or re-queues) the current job within terminationGracePeriodSeconds instead of dying immediately. 3. At minimum, document that ARC v2 runners cannot be protected from node drains and that the only mitigation is at the infrastructure layer (maintenance windows / controlled upgrade cadence). Related: #4493 (PDB scale subresource, closed won't-fix), #2562 (busy annotations for CA), #4148 (EphemeralRunner stuck Running after node drain). Environment • gha-runner-scale-set / gha-runner-scale-set-controller 0.14.2 • AKS, node auto-upgrade (drain via eviction API) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>