Yade

Parallelism inconsistency

Asked by JOHN

Good evening,
I recently tried yade in parallel. To test it, i created a scene and then loaded it with various thread numbers.
With 1 thread the data was consistently the same (positions times till the end etc)
with more, there were differences.
The particles werent that many at the beginning (1330) but the message saying
WARN /build/yade-KKgSmd/yade-1.20.0/pkg/common/InsertionSortCollider.cpp:80 insertionSortParallel: Parallel insertion: only 2 thread(s) used. The number of bodies is probably too small for allowing more threads, or the geometry is flat. The contact detection should succeed but not all available threads are used.

didnt appear for 2 threads(oviously), which also produced differences

One thing is that the number of particles decreases with the simulation time.
I cant think however why the results would vary for 2 threads anyway (openmp is safer than mpi)

Is this a known issue? Any suggestions?
Thank you very much in advance

John

Question information

Language:
English Edit question
Status:
Solved
For:
Yade Edit question
Assignee:
No assignee Edit question
Solved by:
Jan Stránský
Solved:
Last query:
Last reply:
Revision history for this message
Best Jan Stránský (honzik) said : 		#1

Hi John,
yes, this is a known issue, see [1]
cheers
Jan

[1] https://yade-dem.org/doc/formulation.html#result-indeterminism

Revision history for this message
JOHN (washingmachine) said : 		#2

Hi Jan,
thanks for the info!
That makes sense. Though it seems like there is no way around it, is it possible to decrease that effect? I am now seeing up to ~15% difference in some simulations
Thanks again
cheers
John

Revision history for this message
JOHN (washingmachine) said : 		#3

And now my new simulation is ~100% different. Reading the source i got the feeling that such large fluctuations are not normal

Revision history for this message
Jan Stránský (honzik) said : 		#4

Hi John,
in general it depends very much on the problem and how you measure fluctuation. I am afraid there is no way to decrease the effect with current implementation..
Can you please post your example (ideally in the minimal form)?
thanks
Jan

Revision history for this message
JOHN (washingmachine) said : 		#5

Hi,
thanks again. My example is unfortunately not easily given in minimal form.
The simulation itself just creates a hexagonal packing, imports an stl, substracts spheres that are in or partially in the walls and runs with the following engine
O.engines=[
     ForceResetter(),
     InsertionSortCollider([Bo1_Sphere_Aabb(),Bo1_Facet_Aabb()]),
     InteractionLoop(
  # handle sphere+sphere and facet+sphere collisions
        [Ig2_Sphere_Sphere_ScGeom(),Ig2_Facet_Sphere_ScGeom()],
        [Ip2_FrictMat_FrictMat_FrictPhys()],
        [Law2_ScGeom_FrictPhys_CundallStrack()]

     ),

     NewtonIntegrator(gravity=(0,-9.81,0),damping=0.4, label='newtonInt'),

the tricky part is an outside function that calculates rotations which is not easily given in minimal form

Revision history for this message
JOHN (washingmachine) said : 		#6

Thanks Jan Stránský, that solved my question.

Revision history for this message
Bruno Chareyre (bruno-chareyre) said : 		#7

Hi,
I would say this is not a misfeature that one wants to avoid. Instead it is an opportunity to assess indeterminism, which is in most cases inherent to the underlying physics.
Re-ordering the bodies (i.e renumbering) while keeping everything else the same would give similar fluctuations in single thread yet I don't think anyone would claim that one numbering is better than another.
The single thread result is like selecting one numbering and sticking to it, while parallel runs are like renumbering randomly each time.
If the fluctuations are about 100% it is an interesting indication that the physical problem could be somehow ill-posed. Like asking in which direction a sphere will fall if you put it right on the top of a needle.
Regards
Bruno