which number to use when make the source

Hi Ning,

what processor do you use? It can be because of Hyperthreading technology in
your CPU.

I heard such opinion, that for this hard kind of tasks (simulation) it is
better to switch off the HT technology as it only slows down the process.
You can try to run the same simulation with different -np numbers. And then
you can find the best number.

Cheers,
Anton Gladkyy

2010/2/25 ceguo <email address hidden>

> New question #102361 on ESyS-Particle:
> https://answers.launchpad.net/esys-particle/+question/102361
>
> Hello everyone,
>
> As described in FAQ #261, when make the source using make -j num during
> install process, num is said to be "the number of CPUs/cores." Is this the
> number of CPU divided by the number of core? For instance, in my condition
> which number should I use?
>
> ~$ cat /proc/cpuinfo | grep "cores" | uniq
> cpu cores : 4
> ~$ cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc -l
> 2
> ~$ cat /proc/cpuinfo | grep "processor" | wc -l
> 16
>
> p.s. why here core number != processor number / physical number?
>
> Regards
>
> Ning
>
> --
> You received this question notification because you are an answer
> contact for ESyS-Particle.
>

Hi Ning,

Sorry for the confusion. The number after "make -j" should be equal to (or less than) the total number of cores on your PC. If you have 2 CPUs, each with 4 cores, use "make -j 8" for the fastest compile time possible. I usually use slightly less cores than the total available (e.g. "make -j 6") so I have a couple free to have fun while the code compiles! ;)

Regarding your second question, when I examine my own "/proc/cpuinfo" using your shell commands I get the following:

~$ cat /proc/cpuinfo | grep "cores" | uniq
cpu cores : 2
~$ cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc -l
1
~$ cat /proc/cpuinfo | grep "processor" | wc -l
4

which is a similar problem to your case (i.e. cores != processors/cpus). The reason is that each core of my PC is hyper-threaded and linux treats a hyper-threaded core as though it were 2 separate processors. I suspect the same applies for your PC.

For the most part, you should be safe to use the result of:
cat /proc/cpuinfo | grep "processor" | wc -l
as the number following "make -j".

Looks like you have a pretty nice PC! Have fun! ;)

Cheers,

Dion.

Thanks Anton and Dion!

Now I see the number will not influence how esys runs during simulation. Then I encounter this problem similar with Question #93757. But when I run "mpirun -np 2 `which mpipython` bingle_chk.py" (numWorkerProcesses=1, mpiDimList=[1,1,1]), it reports:

/usr/local/lib/python2.6/dist-packages/esys/lsm/util/Process.py:18: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module.
  import popen2
wrong number of processes !! aborting

Since "wrong number of processes !! aborting" appears only once, it seems not the case of conflicts between openmpi and mpich as that question. And I use "/usr/bin/mpirun -np 2 `which mpipython` SamplePrepare.py", the problem still exists.

Tks

Ning

Hi Ning,

I think you may have an old version of ESyS-Particle still installed on your PC somewhere. The latest development version doesn't use popen2 so the fact you got that warning would suggest you are running an old version of ESyS-Particle. Follow the instructions in the following FAQ to ensure you have completely removed the old version before re-installing the latest development version:

https://answers.launchpad.net/esys-particle/+faq/947

Cheers,

Dion.

P.S. Often /usr/bin/mpirun is just a symbolic link to an executable in /etc/alternatives which may be either the openMPI or mpich mpirun executable. You may need to check that /usr/bin/mpirun is pointing to the correct executable.

If you use both openmpi and mpich, maybe mpirun is only a simlink to one or
another MPI binaries.
Try to use:
mpirun.mpich-shmem -np 2 `which mpipython` bingle_chk.py
or
mpirun.openmpi -np 2 `which mpipython` bingle_chk.py

instead default mpirun simlink.
______________________________

Anton Gladkyy

2010/2/25 ceguo <email address hidden>

> Question #102361 on ESyS-Particle changed:
> https://answers.launchpad.net/esys-particle/+question/102361
>
> Status: Answered => Open
>
> ceguo is still having a problem:
> Thanks Anton and Dion!
>
> Now I see the number will not influence how esys runs during simulation.
> Then I encounter this problem similar with Question #93757. But when I
> run "mpirun -np 2 `which mpipython` bingle_chk.py"
> (numWorkerProcesses=1, mpiDimList=[1,1,1]), it reports:
>
> /usr/local/lib/python2.6/dist-packages/esys/lsm/util/Process.py:18:
> DeprecationWarning: The popen2 module is deprecated. Use the subprocess
> module.
> import popen2
> wrong number of processes !! aborting
>
> Since "wrong number of processes !! aborting" appears only once, it
> seems not the case of conflicts between openmpi and mpich as that
> question. And I use "/usr/bin/mpirun -np 2 `which mpipython`
> SamplePrepare.py", the problem still exists.
>
> Tks
>
> Ning
>
> --
> You received this question notification because you are an answer
> contact for ESyS-Particle.
>

Hi,

I reinstalled esys, but problem still exists. when run "mpirun -np 2 `which mpipython` bingle_chk.py", it only reports: wrong number of processes !! aborting.
while run "mpirun.openmpi -np 2 `which mpipython` bingle_chk.py", the error message reports two times. and run "mpirun.mpich-shmem -np 2 `which mpipython` bingle_chk.py" shows command not found.

$ mpirun -np 2 `which mpipython` bingle_chk.py
wrong number of processes !! aborting
$ mpirun.openmpi -np 2 `which mpipython` bingle_chk.py
wrong number of processes !! abortingwrong number of processes !! aborting
$ mpirun.mpich-shmem -np 2 `which mpipython` bingle_chk.py
The program 'mpirun.mpich-shmem' is currently not installed. You can install it by typing:
sudo apt-get install mpich-shmem-bin
mpirun.mpich-shmem: command not found

How did you configure your ESyS installation? With mpich or openmpi?
______________________________

Anton Gladkyy

2010/2/25 ceguo <email address hidden>

> Question #102361 on ESyS-Particle changed:
> https://answers.launchpad.net/esys-particle/+question/102361
>
> Status: Answered => Open
>
> ceguo is still having a problem:
> Hi,
>
> I reinstalled esys, but problem still exists. when run "mpirun -np 2 `which
> mpipython` bingle_chk.py", it only reports: wrong number of processes !!
> aborting.
> while run "mpirun.openmpi -np 2 `which mpipython` bingle_chk.py", the error
> message reports two times. and run "mpirun.mpich-shmem -np 2 `which
> mpipython` bingle_chk.py" shows command not found.
>
> $ mpirun -np 2 `which mpipython` bingle_chk.py
> wrong number of processes !! aborting
> $ mpirun.openmpi -np 2 `which mpipython` bingle_chk.py
> wrong number of processes !! abortingwrong number of processes !! aborting
> $ mpirun.mpich-shmem -np 2 `which mpipython` bingle_chk.py
> The program 'mpirun.mpich-shmem' is currently not installed. You can
> install it by typing:
> sudo apt-get install mpich-shmem-bin
> mpirun.mpich-shmem: command not found
>
> --
> You received this question notification because you are an answer
> contact for ESyS-Particle.
>

openmpi
I just follow the instruction posted in launchpad:
./configure CC=mpicc CXX=mpic++

I am clutching at straws, but try using the absolute path for bingle_chk.py on the command line, in case there is another copy of the script on your system which has been causing the "wrong number of processes" report.

Vince

Thanks Anton Gladky, that solved my question.