Add ntasks to SlurmExecutor (#381) - #589
Open
CodersAcademy006 wants to merge 1 commit into
Open
Conversation
SBATCH_FLAGS already whitelisted "ntasks", but SlurmExecutor had no such field, so the flag could only be reached through additional_parameters, where it collided with the ntasks_per_node default of 1. Add `ntasks`, and drop `ntasks_per_node` whenever it is set, in both the sbatch parameter build and the interactive srun helper. Slurm treats --ntasks-per-node as a per-node maximum when --ntasks is present, so leaving the default in place would cap the request. Heterogeneous jobs size each group through resource_group, so combining ntasks with heterogeneous=True is rejected in __post_init__ rather than silently ignored. Closes NVIDIA-NeMo#381 Signed-off-by: Srijan Upadhyay <srjnupadhyay@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #381.
Why this was awkward before
SlurmExecutor.SBATCH_FLAGSalready contains"ntasks", andSlurmBatchRequestbuilds its sbatch parameters asasdict(executor)filtered through that list. So the flag was reachable in principle but the field never existed, which leftadditional_parameters={"ntasks": 8}as the only route, and that still emitted--ntasks-per-node=1alongside it from the field default. @activatedgeek's point in the issue is exactly that: the user ends up hand-deriving a per-node topology for what Slurm can size directly.Change
ntasks: Optional[int] = None. Being inSBATCH_FLAGSalready, it renders as#SBATCH --ntasks=<n>with no plumbing.ntasksis set,ntasks_per_nodeis dropped from the emitted flags. Slurm treats--ntasks-per-nodeas a per-node maximum when--ntasksis also present, so leaving the default of 1 in place would cap an explicit--ntasks=8at one task per node. The same exclusion is applied in the interactivesrun()helper, sinceSRUN_ARGSlists both names and would otherwise drift from the sbatch path.ntaskswithheterogeneous=Trueis rejected in__post_init__. Het groups get their sizes fromresource_groupentries, which carry their ownntasks_per_node, so a top-level total would be silently overridden. A clear assertion beats that.Default behaviour is unchanged: with
ntasksunset, the script still renders#SBATCH --ntasks-per-node=1and no--ntasks.Tests
Four in
test/core/execution/test_slurm_templates.py:test_dummy_batch_request_ntasksmaterializes withntasks=8and asserts--ntasks=8is present and--ntasks-per-nodeis gone.test_dummy_batch_request_ntasks_per_node_defaultpins the unchanged default path.test_srun_ntasks_drops_ntasks_per_nodeasserts the interactivesruncommand agrees with sbatch, so the two copies of the rule cannot diverge.test_ntasks_rejected_for_heterogeneouscovers the assertion.Verified locally: 42 passed in that file, 142 passed across
test_slurm.py,test_slurm_templates.py,test_slurm_job_name.pyandtest/run/test_experiment.py. Reverting only thenemo_run/half fails 3 of the 4 new tests.ruff format --diffandruff checkclean. No Slurm cluster here, so the verification is at the rendered-script level, and the flag semantics are per the sbatch and srun man pages rather than measured.@ko3n1g one design question worth your call: I made
ntaskswin overntasks_per_nodebecause the latter is a field default rather than an explicit choice. If you would rather haventasks_per_nodewin, or have both emitted and let Slurm reject the combination, say which and I will switch it.