Skip to content

Add ntasks to SlurmExecutor (#381) - #589

Open
CodersAcademy006 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
CodersAcademy006:feat/381-slurm-ntasks
Open

Add ntasks to SlurmExecutor (#381)#589
CodersAcademy006 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
CodersAcademy006:feat/381-slurm-ntasks

Conversation

@CodersAcademy006

Copy link
Copy Markdown
Contributor

Closes #381.

Why this was awkward before

SlurmExecutor.SBATCH_FLAGS already contains "ntasks", and SlurmBatchRequest builds its sbatch parameters as asdict(executor) filtered through that list. So the flag was reachable in principle but the field never existed, which left additional_parameters={"ntasks": 8} as the only route, and that still emitted --ntasks-per-node=1 alongside it from the field default. @activatedgeek's point in the issue is exactly that: the user ends up hand-deriving a per-node topology for what Slurm can size directly.

Change

  • New field ntasks: Optional[int] = None. Being in SBATCH_FLAGS already, it renders as #SBATCH --ntasks=<n> with no plumbing.
  • When ntasks is set, ntasks_per_node is dropped from the emitted flags. Slurm treats --ntasks-per-node as a per-node maximum when --ntasks is also present, so leaving the default of 1 in place would cap an explicit --ntasks=8 at one task per node. The same exclusion is applied in the interactive srun() helper, since SRUN_ARGS lists both names and would otherwise drift from the sbatch path.
  • ntasks with heterogeneous=True is rejected in __post_init__. Het groups get their sizes from resource_group entries, which carry their own ntasks_per_node, so a top-level total would be silently overridden. A clear assertion beats that.

Default behaviour is unchanged: with ntasks unset, the script still renders #SBATCH --ntasks-per-node=1 and no --ntasks.

Tests

Four in test/core/execution/test_slurm_templates.py:

  • test_dummy_batch_request_ntasks materializes with ntasks=8 and asserts --ntasks=8 is present and --ntasks-per-node is gone.
  • test_dummy_batch_request_ntasks_per_node_default pins the unchanged default path.
  • test_srun_ntasks_drops_ntasks_per_node asserts the interactive srun command agrees with sbatch, so the two copies of the rule cannot diverge.
  • test_ntasks_rejected_for_heterogeneous covers the assertion.

Verified locally: 42 passed in that file, 142 passed across test_slurm.py, test_slurm_templates.py, test_slurm_job_name.py and test/run/test_experiment.py. Reverting only the nemo_run/ half fails 3 of the 4 new tests. ruff format --diff and ruff check clean. No Slurm cluster here, so the verification is at the rendered-script level, and the flag semantics are per the sbatch and srun man pages rather than measured.

@ko3n1g one design question worth your call: I made ntasks win over ntasks_per_node because the latter is a field default rather than an explicit choice. If you would rather have ntasks_per_node win, or have both emitted and let Slurm reject the combination, say which and I will switch it.

SBATCH_FLAGS already whitelisted "ntasks", but SlurmExecutor had no such
field, so the flag could only be reached through additional_parameters,
where it collided with the ntasks_per_node default of 1.

Add `ntasks`, and drop `ntasks_per_node` whenever it is set, in both the
sbatch parameter build and the interactive srun helper. Slurm treats
--ntasks-per-node as a per-node maximum when --ntasks is present, so
leaving the default in place would cap the request. Heterogeneous jobs
size each group through resource_group, so combining ntasks with
heterogeneous=True is rejected in __post_init__ rather than silently
ignored.

Closes NVIDIA-NeMo#381

Signed-off-by: Srijan Upadhyay <srjnupadhyay@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support ntasks in SlurmExecutor

1 participant