Open MPI - spawn processes

Please consider Parallel.py=Case2_rotating_drum_mpi.py

Hi,

This:

"mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes"

is a message I've seen so many times. It isn't related to yade in most
cases, I would first try and run some dummy mpi4py scripts to see what
happens. There can be:

- some version conflicts between OpenMPI and mpi4py, or some bug in one
of them

- hardware-specific or infrastructure-specific problems (not all HPC may
enable spawning processes - see [1])

- also mind that the OpenMPI version on the host must be the same (or
sufficiently close to) the one in the singularity image, else it won't
connect communicate properly.

- on a cluster in Grenoble I found that the combination MPI/Singularity
would run properly with a ubuntu16.04 image but not with a 20.04 image
(no clue why at this point).

[1] If spawning is disabled on purpose on the cluster it might help to
run mpi explicitely, as in:

mpiexec -np N singularity run
/beegfs/common/singularity/yade/yade_debian_bookwarm_1.0.sif yade
Case2_rotating_drum_mpi.py

Last note: if you run "yade -j5" on 10 mpi processes it actually means
10x5=50 cores (mpi x openMP).

Bruno

Hi Bruno, thanks.

I just run a simple script[1] to try non-interactive execution (mpiexec -np NUMSUBD+1 yade script.py)

Got this message:

+ singularity run /beegfs/common/singularity/yade/yade_debian_bookwarm_1.0.sif mpiexec -n 4 yade Parallel.py
--------------------------------------------------------------------------
The SLURM process starter for OpenMPI was unable to locate a
usable "srun" command in its path. Please check your path
and try again.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
An internal error has occurred in ORTE:

[[58454,0],0] FORCE-TERMINATE AT (null):1 - error ../../../../../../orte/mca/plm/slurm/plm_slurm_module.c(475)

This is something that should be reported to the developers.
--------------------------------------------------------------------------

I will try this mpiexec -np N singularity run.

Kind regards and thanks for all insights.

[1]https://gitlab.com/yade-dev/trunk/blob/master/examples/mpi/vtkRecorderExample.py

Please check my command again. It's "mpiexec singularity" not
"singularity mpiexec".

That kind of message, if it persists may mean a local problem in your
installation (independent of Yade):
 > "Actually The SLURM process starter for OpenMPI was unable to locate
a usable "srun" command in its path."

Hence why I suggested to try with a real dummy program (you may find
some in mpi4py test suite) independently of Yade.

Or even more basic (but is will not test mpi4py):

echo "print('hello')" > test.py
mpiexec -np 10 singularity *.sif python test.py

Does it print "hello" 10 times?

Bruno

> "- also mind that the OpenMPI version on the host must be the same (or
sufficiently close to) the one in the singularity image, else it won't
connect communicate properly."

I run yade --test to check MPI:

| mpi | 3.1 | ompi:4.1.0 |
+---------------+-----------------------+---------------------+
| mpi4py | 3.0.3 |

My cluster has Intel(R) MPI Library, Version 2019 Update 8 installed. Do you know if this is compatible?

Hello all,

Since the OpenMPI and Intel MPI in my cluster may not be compatible and I dont think it is possible to get these two different MPI systems running.

Would you suggest some work arround?

I thought in creating a new docker image replacing OpenMPI with Intel MPI.
However, I don't think this is supported by Yade or would run in dependency problems.

Maybe I could run multiple scripts on different nodes? not sure how it would work acctually...

Any insight would be welcome.

Luis

Since different versions of openMPI is already a problem there's no way
it will work with intel+openMPI.

Indeed generating a new docker is an option, provided that you manage to
compile yade with intel compiler therein.

I see no escape and it's not really a problem with singularity : either
you get openMPI installed on the cluster or you compile yade with intel
on the cluster directly (or you compile yade with intel then you make a
docker->singularity of this, but it's only adding another layer of
possible trouble).

Is it a cluster you're admin of?

Hi Bruno,

We will try to compile yade with intel on the cluster directly.

Since I need IT for that it will take some time to make it.

I will close the issue for now, but when I finish this I will post the solution to keep documented here.

cheers,
Luis

A walkthrough of Intel compilation would be valuable, especially if it's less trivial than just changing the compiler (it needs linkage to more dynamic libraries iirc).

But in fact installing gcc on the cluster would make a lot of sense too. I don't know that a clusters can have only one compiler (not the case in Grenoble at least).

Bruno

Hi all, just to inform that we solved this:

Solution was to install OpenMpi 4.1.0 on my home directory on the cluster.

Here are the steps I performed:

Download
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.0.tar.gz

open a console on the cluster in your home folder:
cd ~
mkdir openMPI
cd openMPI

> copy or download openmpi-4.1.0.tar.gz tar ball into the openMPI folder, and unpack it

gtar zxf openmpi-4.1.0.tar.gz
cd openmpi-4.1.0

> create a output folder

mkdir /home/pires/bin/openmpi

> create a build folder, and compile openMPI

mkdir build
cd build
../configure --prefix=/home/pires/bin/openmpi --with-slurm --with-pmi
make all
make install

--------------

at this point you have openMPI 4.1.0 with slurm support installed into your home.
Next step is to make your own module.

mkdir /home/../privatemodules
cd /home/.../privatemodules

A module file is just a bunch of linux environment variables.

Hope it can help someone in this situation.
Luis

Hi, for clarity:
- did you compile openmpi with intel or gcc?
- then do you use that with singularity or with yade compiled on the cluster?

Bruno