about the parallel calculation in 32 cores cluster

Hi Liqing,

to use a parallelism on 0.70, you just need to put "-jN" parameter to
the command line. Something like:

yade-0.70 -j4 yourScript.py

In this case 4 threads will be used. But the effectiveness of
parallelisation is strong depended on the task, you are using. For
some tasks it works better, for other you will get even a slowdown.
Just try.

> I read some relative documents about the
>  parallelism in yade. There are 3 levels. I don't know how to use others. If
>  some kind people realized already, could you please send me an example?

Well, the main "parallelism" is the "splitting" of some loops
(openmp), it works automatically.

Another opportunity is to use ParallelEngine [1]. But you should be
careful with that one.

Third, probably, is a batch-system [2], I am personally using it a
lot. You can start several Yade-processes in one time.t

[1] https://www.yade-dem.org/doc/yade.wrapper.html?highlight=parallelengine#yade.wrapper.ParallelEngine

[2] http://bazaar.launchpad.net/~yade-dev/yade/trunk/files/head:/scripts/test/batch/

Anton

Dear Anton,

May I know the relationship between the number of the thread and the the number of cores. For example, my computer has 32 CPUs, need I set the thread number to be 32? In fact, I set it, but all CPUs are almost occupied 50%. The effectiveness is just a little improved compared to the 4 cores desktop. And the effectiveness of 32 CPUs is 1.5 times that of the 4 CPUs. How can I improve it much more?

Looking forward to your reply.
Cheers, Liqing

At 2012-01-29 00:53:18,"Anton Gladky" <email address hidden> wrote: >Question #186108 on Yade changed: >https://answers.launchpad.net/yade/+question/186108 > > Status: Open => Answered > >Anton Gladky proposed the following answer: >Hi Liqing, > >to use a parallelism on 0.70, you just need to put "-jN" parameter to >the command line. Something like: > >yade-0.70 -j4 yourScript.py > >In this case 4 threads will be used. But the effectiveness of >parallelisation is strong depended on the task, you are using. For >some tasks it works better, for other you will get even a slowdown. >Just try. > >> I read some relative documents about the >> parallelism in yade. There are 3 levels. I don't know how to use others. If >> some kind people realized already, could you please send me an example? > >Well, the main "parallelism" is the "splitting" of some loops >(openmp), it works automatically. > >Another opportunity is to use ParallelEngine [1]. But you should be >careful with that one. > >Third, probably, is a batch-system [2], I am personally using it a >lot. You can start several Yade-processes in one time.t > >[1] https://www.yade- >dem.org/doc/yade.wrapper.html?highlight=parallelengine#yade.wrapper.ParallelEngine > >[2] http://bazaar.launchpad.net/~yade- >dev/yade/trunk/files/head:/scripts/test/batch/ > >Anton > >-- >You received this question notification because you are a member of >yade-users, which is an answer contact for Yade. > >_______________________________________________ >Mailing list: https://launchpad.net/~yade-users >Post to : <email address hidden> >Unsubscribe : https://launchpad.net/~yade-users >More help : https://help.launchpad.net/ListHelp

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Hi Liqing,

2012/2/2 Liqing Jiao <email address hidden>:
> ....
> May I know the relationship between the number of the thread and the the
> number of cores. For example, my computer has 32 CPUs, need I set the
> thread number to be 32? In fact, I set it, but all CPUs are almost
> occupied 50%. The effectiveness is just a little improved compared to
> the 4 cores desktop. And the effectiveness of 32 CPUs is 1.5 times that
> of the 4 CPUs. How can I improve it much more?
>...

How many bodies are participating in your simulation? If it is less
than, for example, 10 000, it will not be effective to use all 32
cores.

What yade-version do you use? The yade-daily version after rev.3000
has a better performance especially on many-thread simulations.

Anton.

Dear Anton,

I use yade-0.70.0. I think it should be very new. I just updated last month. About "How many bodies are participating in your simulation? If it is less than, for example, 10 000, it will not be effective to use all 32 cores." you said, does it mean that the more particle more cores will be better, right? What relationship between the particle amount and the cores? Thank you very much. Looking forward to your reply.

Cheers, Liqing

At 2012-02-18 06:30:54,"Anton Gladky" <email address hidden> wrote:
>Question #186108 on Yade changed:
>https://answers.launchpad.net/yade/+question/186108
>
>Anton Gladky posted a new comment:
>Hi Liqing,
>
>2012/2/2 Liqing Jiao <email address hidden>:
>> ....
>> May I know the relationship between the number of the thread and the the
>> number of cores. For example, my computer has 32 CPUs, need I set the
>> thread number to be 32? In fact, I set it, but all CPUs are almost
>> occupied 50%. The effectiveness is just a little improved compared to
>> the 4 cores desktop. And the effectiveness of 32 CPUs is 1.5 times that
>> of the 4 CPUs. How can I improve it much more?
>>...
>
>How many bodies are participating in your simulation? If it is less
>than, for example, 10 000, it will not be effective to use all 32
>cores.
>
>What yade-version do you use? The yade-daily version after rev.3000
>has a better performance especially on many-thread simulations.
>
>Anton.
>
>--
>You received this question notification because you are a member of
>yade-users, which is an answer contact for Yade.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~yade-users
>Post to : <email address hidden>
>Unsubscribe : https://launchpad.net/~yade-users
>More help : https://help.launchpad.net/ListHelp

Hi Liqing,

2012/2/18 Liqing Jiao <email address hidden>:
> ....
> I use yade-0.70.0. I think it should be very new. I just updated last month.

Not quite, this feature was added after 0.70 release. You should use yade-daily
version of yade to have it or download it from BZR. Or just wait a
little bit, we will probably release a newer version soon. I hope.

> About "How many bodies are participating in your simulation? If it is less
> than, for example, 10 000, it will not be effective to use all 32 cores."
> you said, does it mean that the more particle more cores will be better,
> right? What relationship between the particle amount and the cores?

Well, I can only say it from my experience and discussion the similar
problem with other DEM-guys from another software (esys-particle). And
it strongly depends on type of your simulation and number of
interactions (how many active contacts you have).

Roughly saying, you should have minimum 10 000 particles/core. Maybe
somebody else give you more precise answer.

Anton

Dear Anton,

I am sorry I can't download BZR from the yade website, it always show up the errors. And my computer tends to spend two days to download, and the end it showed the error. Could you please help me check it? Thank you very much.

By the way, I want to ask another question about the packing. I want to try different packing with same parameters (the model geometry and the particle size). When I re-pack my model without any change, Is the packing changed or not. I found when I re- pack it, the packing time is shorter than first packing. So I wonder it. Could you please help me? Thank you very much.

Best Regards, Liqing

At 2012-02-18 14:41:05,"Anton Gladky" <email address hidden> wrote:
>Question #186108 on Yade changed:
>https://answers.launchpad.net/yade/+question/186108
>
> Status: Open => Answered
>
>Anton Gladky proposed the following answer:
>Hi Liqing,
>
>2012/2/18 Liqing Jiao <email address hidden>:
>> ....
>> I use yade-0.70.0. I think it should be very new. I just updated last month.
>
>Not quite, this feature was added after 0.70 release. You should use yade-daily
>version of yade to have it or download it from BZR. Or just wait a
>little bit, we will probably release a newer version soon. I hope.
>
>> About "How many bodies are participating in your simulation? If it is less
>> than, for example, 10 000, it will not be effective to use all 32 cores."
>> you said, does it mean that the more particle more cores will be better,
>> right? What relationship between the particle amount and the cores?
>
>Well, I can only say it from my experience and discussion the similar
>problem with other DEM-guys from another software (esys-particle). And
>it strongly depends on type of your simulation and number of
>interactions (how many active contacts you have).
>
>Roughly saying, you should have minimum 10 000 particles/core. Maybe
>somebody else give you more precise answer.
>
>Anton
>
>--
>You received this question notification because you are a member of
>yade-users, which is an answer contact for Yade.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~yade-users
>Post to : <email address hidden>
>Unsubscribe : https://launchpad.net/~yade-users
>More help : https://help.launchpad.net/ListHelp

Hi Liqing,

> I am sorry I can't download BZR from the yade website, it always show up
> the errors. And my computer tends to spend two days to download, and the
> end it showed the error. Could you please help me check it? Thank you
> very much.

Sorry, I even cannot imagine, why it downloads 2 days.
Try git-version [1]:

git clone git://github.com/yade/trunk.git

Anyway, I strongly recommend you to use yade-daily built-version [2].
It solves a tons of further questions about dependencies, compilation etc.

> By the way, I want to ask another question about the packing. I want to
> try different packing with same parameters (the model geometry and the
> particle size). When I re-pack my model without any change, Is the
> packing changed or not.

Please, show your script and open a new question as it has another topic

Thanks.

[1] https://github.com/yade/trunk
[2] https://answers.launchpad.net/yade/+faq/1727

Anton

Hi, Anton,

Thank you for your answer, I can download the trunk version. But when I compile it, errors show up. The error message is :
scons: Building targets ...
> /usr/local/bin/yade-2012-04-27.git-3a6ba14-batch
> /usr/local/bin/yade-2012-04-27.git-3a6ba14
scons: *** [/usr/local/bin/yade-2012-04-27.git-3a6ba14-batch] /usr/local/bin/yade-2012-04-27.git-3a6ba14-batch: Permission denied
scons: *** [/usr/local/bin/yade-2012-04-27.git-3a6ba14] /usr/local/bin/yade-2012-04-27.git-3a6ba14: Permission denied
scons: building terminated because of errors.

May I get your help again, thank you!

Cheers, Liqing

At 2012-03-22 04:00:57,"Anton Gladky" <email address hidden> wrote:
>Your question #186108 on Yade changed:
>https://answers.launchpad.net/yade/+question/186108
>
> Status: Open => Answered
>
>Anton Gladky proposed the following answer:
>Hi Liqing,
>
>> I am sorry I can't download BZR from the yade website, it always show up
>> the errors. And my computer tends to spend two days to download, and the
>> end it showed the error. Could you please help me check it? Thank you
>> very much.
>
>Sorry, I even cannot imagine, why it downloads 2 days.
>Try git-version [1]:
>
>git clone git://github.com/yade/trunk.git
>
>Anyway, I strongly recommend you to use yade-daily built-version [2].
>It solves a tons of further questions about dependencies, compilation etc.
>
>> By the way, I want to ask another question about the packing. I want to
>> try different packing with same parameters (the model geometry and the
>> particle size). When I re-pack my model without any change, Is the
>> packing changed or not.
>
>Please, show your script and open a new question as it has another topic
>
>Thanks.
>
>[1] https://github.com/yade/trunk
>[2] https://answers.launchpad.net/yade/+faq/1727
>
>Anton
>
>--
>If this answers your question, please go to the following page to let us
>know that it is solved:
>https://answers.launchpad.net/yade/+question/186108/+confirm?answer_id=7
>
>If you still need help, you can reply to this email or go to the
>following page to enter your feedback:
>https://answers.launchpad.net/yade/+question/186108
>
>You received this question notification because you asked the question.

Hi,

> Permission denied
You are installing yade into the folder, where you do not have
permissions to write to.

Do:

scons PREFIX='/path/to/writable/folder'

Anton

type as root:

chmod -R 777 /usr/local/

Dear Anton,

Yes, I did that as your wrote the command as below. I compiled already. But I can't see what's the difference between Yade other package version and this trunk version?

Thank you, Liqing

At 2012-04-30 14:25:48,"Anton Gladky" <email address hidden> wrote:
>Your question #186108 on Yade changed:
>https://answers.launchpad.net/yade/+question/186108
>
> Status: Open => Answered
>
>Anton Gladky proposed the following answer:
>Hi,
>
>> Permission denied
>You are installing yade into the folder, where you do not have
>permissions to write to.
>
>Do:
>
>scons PREFIX='/path/to/writable/folder'
>
>Anton
>
>--
>If this answers your question, please go to the following page to let us
>know that it is solved:
>https://answers.launchpad.net/yade/+question/186108/+confirm?answer_id=9
>
>If you still need help, you can reply to this email or go to the
>following page to enter your feedback:
>https://answers.launchpad.net/yade/+question/186108
>
>You received this question notification because you asked the question.

Sorry, I am not sure, what you mean. After that you can start yade so:

path/to/writable/folder/bin/yade

Anton

2012/4/30 Liqing Jiao <email address hidden>:
> Question #186108 on Yade changed:
> https://answers.launchpad.net/yade/+question/186108
>
>    Status: Answered => Open
>
> Liqing Jiao is still having a problem:
> Dear Anton,
>
> Yes, I did that as your wrote the command as below. I compiled already.
> But I can't see what's the difference between Yade other package version
> and this trunk version?

Dear Anton,

I wonder whether the usage of this trunk version is the same as the package version like yade-0.70.0. If I use the trunk version, does it mean that I can see other person's update to the code immediately? Does this trunk version have other benefits compare to yade-0.70.0, or is it just a newer version with the same function as the release version?

Cheers, Liqing

At 2012-04-30 19:01:00,"Anton Gladky" <email address hidden> wrote:
>Your question #186108 on Yade changed:
>https://answers.launchpad.net/yade/+question/186108
>
> Status: Open => Answered
>
>Anton Gladky proposed the following answer:
>Sorry, I am not sure, what you mean. After that you can start yade so:
>
>path/to/writable/folder/bin/yade
>
>Anton
>
>
>2012/4/30 Liqing Jiao <email address hidden>:
>> Question #186108 on Yade changed:
>> https://answers.launchpad.net/yade/+question/186108
>>
>> Status: Answered => Open
>>
>> Liqing Jiao is still having a problem:
>> Dear Anton,
>>
>> Yes, I did that as your wrote the command as below. I compiled already.
>> But I can't see what's the difference between Yade other package version
>> and this trunk version?
>
>--
>If this answers your question, please go to the following page to let us
>know that it is solved:
>https://answers.launchpad.net/yade/+question/186108/+confirm?answer_id=12
>
>If you still need help, you can reply to this email or go to the
>following page to enter your feedback:
>https://answers.launchpad.net/yade/+question/186108
>
>You received this question notification because you asked the question.

Liqing,
We try to maintain backward compatibility when it is not too difficult. So, usage is the same as in 0.70.0 normally.
"Trunk" is generally better than the last release, provided devs work in the good direction. You can get immediate update if you wish, but you are not forced to update.
If you don't update your sources, then you will have a stable version frozen at a given revision.
0.70.0 is (almost) nothing more than that: a snapshot of trunk at a given time.

Hi Yade community (btw. you're doing an excellent job in developing this amazing DEM engine!)

Hi Liqing,
have you finally updated your Yade version and compared performance in simulation? Was CPU usage significantly better when using a 32 core system (a workstation i guess?) instead of a 4 core desktop pc?
it may be of help if you post your system specification.
At the moment I'm thinking about ways of fastening Yade simulations. the most obvious to me would be a better computer. so instead of a standard desktop with i5 processor a workstation with 32 maybe even 64 cores and a lot of Ram (128 or 256GB).
But if I get your problem right, that wouldn't really change anything... On the other hand the last post is some months ago.

Looking forward to all your answers,
Eugen

I would be interested in benchmarks too. :)

Actually we all go repeating that "Yade doesn't scale very well", but we are maybe demanding too much: we are always speaking of "strong" scaling (fixed number of bodies vs. increasing nb. of cores).
Nobody really measured the "weak" scaling properties afaik (fixed nb. of bodies per core).

Also, with the recent changes in the code and progress in hardware, our-years old benchmarks are irrelevant...

Hi Eugen,

Yade works definitely better on many-core systems. But it will not work
on "real" cluster (where nodes are connected through the network), in this
case you should have a look at LIGGGHTS or ESyS-Particle, which are
MPI-oriented by design.

Moreover, I think Yade should scale better on many-core systems, than
MPI-based code. Because MPI-programs usually divide the simulation
"scene" on equal parts, and each part is calculated by the same node.
Some of such parts can be completely empty, some of them can be
overfulled, so there always happens "dis-balance". Yade divides all active
interactions relatively equally between cores.

Back to hardware, it depends on how many particles you want to simulate,
how many of them will be in contact (dense or sparse packing), type of
the simulation (dynamic or quasi-static) and how much money you have.

The speed of calculation is highly depends on CPU-frequency and
RAM-frequency. More particles and contacts lead to higher RAM-usage.

Cheers,

Anton

2012/10/26 Eugen Kubowsky <email address hidden>:
> Question #186108 on Yade changed:
> https://answers.launchpad.net/yade/+question/186108
>
> Eugen Kubowsky requested more information:
> Hi Yade community (btw. you're doing an excellent job in developing this
> amazing DEM engine!)
>
> Hi Liqing,
> have you finally updated your Yade version and compared performance in simulation? Was CPU usage significantly better when using a 32 core system (a workstation i guess?) instead of a 4 core desktop pc?
> it may be of help if you post your system specification.
> At the moment I'm thinking about ways of fastening Yade simulations. the most obvious to me would be a better computer. so instead of a standard desktop with i5 processor a workstation with 32 maybe even 64 cores and a lot of Ram (128 or 256GB).
> But if I get your problem right, that wouldn't really change anything... On the other hand the last post is some months ago.
>
> Looking forward to all your answers,
> Eugen
>
> --
> You received this question notification because you are a member of
> yade-users, which is an answer contact for Yade.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~yade-users
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~yade-users
> More help : https://help.launchpad.net/ListHelp

Hi Bruno,

for a "quick and dirty" performance check there is an option
"--performance". It allows roughly to measure the productivity
of the given computer.

The result returns in "scores". The higher score means better performance.

Cheers,

Anton

2012/10/26 Chareyre <email address hidden>:
> Question #186108 on Yade changed:
> https://answers.launchpad.net/yade/+question/186108
>
> Chareyre posted a new comment:
> I would be interested in benchmarks too. :)
>
> Actually we all go repeating that "Yade doesn't scale very well", but we are maybe demanding too much: we are always speaking of "strong" scaling (fixed number of bodies vs. increasing nb. of cores).
> Nobody really measured the "weak" scaling properties afaik (fixed nb. of bodies per core).
>
> Also, with the recent changes in the code and progress in hardware, our-
> years old benchmarks are irrelevant...
>
> --
> You received this question notification because you are a member of
> yade-users, which is an answer contact for Yade.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~yade-users
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~yade-users
> More help : https://help.launchpad.net/ListHelp

Hi Anton,
thanks for your help so far!
Let me give yo some more technical details about the workstation solutions that I have in mind.
There are 4 of these solutions. Two of them are Intel and the other ones AMD.

INTEL
------

"low-price": (16 cores, 2ghz)
- 2x Intel® Xeon® Sandy Bridge- EP Series Processor E5- 2650, 2.00 GHz, 8-Core Socket 2011, 20MB Cache, max. 1600MHz (DDR3 Memory Speed), QPI 8.00 GT/s,
- 128 GB (16x 8GB) / PC1600 Reg. ECC, 2R, 1600mhz
- 2x 2TB SATA3- HDD Western Digital WD CAVIAR Black

"high-end": (16 cores, 3,1ghz)
- 2x Intel® Xeon® Sandy Bridge- EP Series Processor E5- 2687W, 3,10 GHz, 8- Core Socket 2011, 20MB Cache, max. 1600MHz (DDR3 Memory Speed), QPI 8.00 GT/s
- 256 GB (16x 16GB) / PC1600 Reg. ECC, 2R
- 120GB SSD SATAIII 6GPS 2.5” Intel 520 Series, 550/520 MB/S read/write
- 2x 2TB SATA3- HDD Western Digital WD CAVIAR Black

AMD
-----

"low-price": (32 cores, 2,1ghz)
- 2x AMD Opteron™ 6200 series (Interlagos) Processor 6272, 2,1 GHz, 16- Core, Socket G34, 16x1024KB/16MB L2/L3, max. 1600MHz (DDR3 Memory Speed)
Cache, max. 1600MHz (DDR3 Memory Speed), QPI 8.00 GT/s,
- 128 GB (16x 8GB) / PC1600 Reg. ECC, 2R, 1600mhz
- 2x 2TB SATA3- HDD Western Digital WD CAVIAR Black

"high-end": (32 cores, 2,7ghz)
- 2x AMD Opteron™ 6200 series (Interlagos) Processor 6284SE, 2,7 GHz, 16-Core, Socket G34, 16x1024KB/16MB L2/L3, max. 1 600MHz (DDR3 Memory Speed)
- 256 GB (16x 16GB) / PC1600 Reg. ECC, 2R
- 120GB SSD SATAIII 6GPS 2.5” Intel 520 Series, 550/520 MB/S read/write
- 2x 2TB SATA3- HDD Western Digital WD CAVIAR Black

graphics
---------
Do I need a graphics card like PCI-Express Nvidia QUADRO 600 1024MB DDR3, 1 GB DDR3, 128 - bit, DirectX 11 (or even a better one?) or is something like GF GT 640 ZONE 2GB DDR3 NVIDIA/ 2048 MB/ DDR3/ 128 Bit/ GPU-frequ: 900 MHz/ memory frequ: 1782 MHz sufficient?

There is also an option for 32 cores Intel respectively 64 cores AMD solution, but i guess they would be too expensive...

What do you think would be the best workstation for dynamic simulation with a lot of particles in dense packing?

Thanks for Help!
Eugen

Hey guys! could anybody please help me?

Hi,

I do not think, that somebody can help you a lot.
Everything depends on your budget and wishes.
AMD/Intel comparison: I would choose Intel.

Cheers,

Anton

Anton

2012/11/7 Eugen Kubowsky <email address hidden>

> Question #186108 on Yade changed:
> https://answers.launchpad.net/yade/+question/186108
>
> Eugen Kubowsky posted a new comment:
> Hey guys! could anybody please help me?
>
> --
> You received this question notification because you are a member of
> yade-users, which is an answer contact for Yade.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~yade-users
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~yade-users
> More help : https://help.launchpad.net/ListHelp
>