[BUG] slurm on candide

[martinkilbinger]

I get an error when I run shapepipe_run on a compute node in the docker image. My commands:

srun --nodes=1 --ntasks-per-node=8 --time=24:00:00 --pty bash -i 
apptainer shell --bind /n17data,/n09data,/home /n17data/mkilbing/shapepipe_im_sims.sif 
shapepipe_run -h

The error message is:

--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[n08:752859] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!```

[martinkilbinger]

I get no error with mpirun:

mpirun -n 1 shapepipe_run works.

[cailmdaley]

Reproduced and root-caused on candide — with both your shapepipe_im_sims.sif and our ngmix_v2.0 image, so it's not image-specific.

The mechanism. shapepipe/run.py does from mpi4py import MPI at module import, which calls MPI_Init immediately — even for shapepipe_run -h. Inside an srun-launched shell, Open MPI sees the SLURM step environment and assumes it was direct-launched by srun, so it looks for a PMI server that srun never started (the container's OMPI isn't built with SLURM PMI support) and aborts the whole process. Bisecting the environment shows SLURM_STEP_ID alone is the trigger: unset it and everything works.

Workarounds, today:

• mpirun -n 1 shapepipe_run … (yours — works because mpirun brings its own PMIx server), or
• env -u SLURM_STEP_ID shapepipe_run … inside the srun shell — one variable, no mpirun wrapper, SMP mode untouched.

Fix: #747 gates the mpi4py import on an actual MPI-launcher environment (OMPI_COMM_WORLD_SIZE / PMI_RANK / PMIX_RANK), so a bare shapepipe_run never initializes MPI. Verified under a real srun allocation: bare run goes from OPAL abort to exit 0, mpirun -n 1 and -n 2 paths unchanged; regression test included.

— Claude on behalf of Cail.