Document Skypilot cluster teardown behaviour (#483) - #590
Open
CodersAcademy006 wants to merge 1 commit into
Open
Document Skypilot cluster teardown behaviour (#483)#590CodersAcademy006 wants to merge 1 commit into
CodersAcademy006 wants to merge 1 commit into
Conversation
SkypilotExecutor exposes `autodown` and `idle_minutes_to_autostop`, which map onto `sky.launch(down=..., idle_minutes_to_autostop=...)`, but neither appeared anywhere in the docs. Both default to off and `cleanup()` only downloads logs, so a Kubernetes pod stays up holding its GPUs after the workload finishes, which reads as a NeMo Run bug rather than the SkyPilot default it is. Add the two parameters to the table and a short lifecycle section covering autodown alone, autodown with an idle timeout, stop-without-delete, and the case SkyPilot deliberately leaves up after a provisioning failure. Addresses NVIDIA-NeMo#483 Signed-off-by: Srijan Upadhyay <srjnupadhyay@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #483.
@jesintharnold asked whether the pod staying alive after the workload finishes is SkyPilot behaviour or a NeMo Run integration gap. It is the former, but the reason it reads as a gap is that the two fields that control it are undocumented:
SkypilotExecutor.autodownandSkypilotExecutor.idle_minutes_to_autostop(nemo_run/core/execution/skypilot.py:120-121) are passed straight intosky.launchasdown=andidle_minutes_to_autostop=, both default to off, andSkypilotExecutor.cleanup()only downloads logs. So the default is a cluster that outlives the job with its GPUs still allocated, and nothing indocs/said so.grep -rn "autodown" docs/returned nothing before this change.This adds the two parameters to the key-parameters table and a short "Cluster lifecycle and teardown" section that covers the four cases worth knowing:
autodown=Truealone tearing down once jobs reach a terminal state, combining it with an idle timeout,idle_minutes_to_autostopwithautodown=Falseto stop rather than delete, and the provisioning/setup-failure case that SkyPilot deliberately leaves up for debugging. The semantics are quoted from thedownandidle_minutes_to_autostopdocstrings insky/client/sdk.pyin skypilot 0.12.3, not guessed.Docs only, no code change. If you would rather nemo_run also called
sky.downfromcleanup(), or defaultedautodowntoTrue, that is a behaviour change for existing users and I would want your call before writing it.@ko3n1g