From 82fbf246a001b2c85544dceb666496c74eb5b123 Mon Sep 17 00:00:00 2001 From: Srijan Upadhyay Date: Wed, 19 Aug 2026 13:34:51 +0530 Subject: [PATCH] docs(skypilot): document cluster teardown behaviour SkypilotExecutor exposes `autodown` and `idle_minutes_to_autostop`, which map onto `sky.launch(down=..., idle_minutes_to_autostop=...)`, but neither appeared anywhere in the docs. Both default to off and `cleanup()` only downloads logs, so a Kubernetes pod stays up holding its GPUs after the workload finishes, which reads as a NeMo Run bug rather than the SkyPilot default it is. Add the two parameters to the table and a short lifecycle section covering autodown alone, autodown with an idle timeout, stop-without-delete, and the case SkyPilot deliberately leaves up after a provisioning failure. Addresses #483 Signed-off-by: Srijan Upadhyay --- docs/guides/executors/skypilot.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/guides/executors/skypilot.md b/docs/guides/executors/skypilot.md index bbeca0c6..b8883597 100644 --- a/docs/guides/executors/skypilot.md +++ b/docs/guides/executors/skypilot.md @@ -44,6 +44,8 @@ Key parameters: | `container_image` | Docker image for the job | | `cluster_name` | Optional: name of an existing cluster to reuse | | `setup` | Shell commands to run once on the cluster before the job | +| `autodown` | Tear the cluster down once jobs finish. Defaults to `False`, so the cluster stays up | +| `idle_minutes_to_autostop` | Stop (or tear down, with `autodown=True`) after this many idle minutes | ## E2E workflow @@ -90,6 +92,28 @@ executor = SkypilotJobsExecutor( ) ``` +### Cluster lifecycle and teardown + +By default a `SkypilotExecutor` cluster outlives the job. `autodown` and `idle_minutes_to_autostop` are +both off, and `cleanup()` only downloads logs, so on Kubernetes the pod stays `Running` and keeps its GPUs +allocated until the cluster is brought down. That is SkyPilot's behaviour for an unmanaged cluster, not a +NeMo Run defect, but the knobs are worth knowing: + +```python +executor = SkypilotExecutor( + ..., + autodown=True, # tear down once all jobs finish + idle_minutes_to_autostop=10, # or wait for 10 idle minutes first +) +``` + +These map onto `sky.launch(down=..., idle_minutes_to_autostop=...)`. `autodown=True` on its own tears the +cluster down after all jobs reach a terminal state, and combining it with `idle_minutes_to_autostop` delays +that until the cluster has been idle for the given time. A cluster that fails during provisioning, data +sync or setup is deliberately left up by SkyPilot for debugging, so those need `sky down ` by hand. +To keep the cluster but stop paying for idle GPUs, set `idle_minutes_to_autostop` with `autodown=False`, +which stops rather than deletes it. + ### Package code from git ```python