problem with parallel computatio with mpi

Asked by NgocSon-Nguyen

Deal all,

I am learning how to perform parallel computation with Yade as I run quite big simulations. I followed the guide in the Yade document, section "MPI parallelization" et run a simple example vtkRecorderExample.py with the command "yade vtkRecorderExample.py" as shown in the guide. I have installed Yade version 2021.01a . I got the following error

<WARNING> Shop:562 static yade::Real yade::Shop::PWaveTimeStep(boost::shared_ptr<yade::Scene>): PWaveTimeStep has not found any suitable spherical body to calculate dt. dt is set to 1.0
kinetic energy from workers: [2335799070.7638345, 2292260117.074841]

Could you please tell me why I got this error? Do I need perform some special things to run parallel computation with Yade?

Thank you in advance for your help.

Best regards
Son Nguyen

Here is the script vtkRecorderExample.py

import os
from yade import mpy as mp

NSTEPS=5000 #turn it >0 to see time iterations, else only initilization
numThreads = 4 # number of threads to be spawned, (in interactive mode).

#materials
young = 5e6
compFricDegree = 0.0
O.materials.append(FrictMat(young=young, poisson=0.5, frictionAngle = radians(compFricDegree), density= 2600, label='sphereMat'))
O.materials.append(FrictMat(young=young*100, poisson = 0.5, frictionAngle = compFricDegree, density =2600, label='wallMat'))

#add spheres

mn,mx=Vector3(0,0,0),Vector3(90,180,90)
pred = pack.inAlignedBox(mn,mx)
O.bodies.append(pack.regularHexa(pred,radius=2.80,gap=0, material='sphereMat'))

#walls (floor)

wallIds=aabbWalls([Vector3(-360,-1,-360),Vector3(360,360,360)],thickness=10.0, material='wallMat')
O.bodies.append(wallIds)

#engines
O.engines=[
        ForceResetter(),
        InsertionSortCollider([
                Bo1_Sphere_Aabb(),
                Bo1_Box_Aabb()], label = 'collider'), # always add labels.
        InteractionLoop(
                [Ig2_Sphere_Sphere_ScGeom(),Ig2_Box_Sphere_ScGeom()],
                [Ip2_FrictMat_FrictMat_FrictPhys()],
                [Law2_ScGeom_FrictPhys_CundallStrack()],
                label="interactionLoop"
        ),
        GlobalStiffnessTimeStepper(timestepSafetyCoefficient=0.3, timeStepUpdateInterval=100, parallelMode=True, label = 'timeStepper'),
        NewtonIntegrator(damping=0.1,gravity = (0, -0.1, 0), label='newton'),
        VTKRecorder(fileName='spheres/3d-vtk-', recorders=['spheres', 'intr', 'boxes'], parallelMode=True,iterPeriod=500), #use .pvtu to open spheres, .pvtp for ints, and .vtu for boxes.
]

#set a custom verletDist for efficiency.
collider.verletDist = 1.5

######### RUN ##########
# customize mpy
mp.ERASE_REMOTE_MASTER = True #keep remote bodies in master?
mp.DOMAIN_DECOMPOSITION= True #automatic splitting/domain decomposition
#mp.mpirun(NSTEPS) #passive mode run
mp.MERGE_W_INTERACTIONS = False
mp.mpirun(NSTEPS,numThreads,withMerge=True) # interactive run, numThreads is the number of workers to be initialized, see below for withMerge explanation.
mp.mergeScene() #merge scene after run.
if mp.rank == 0: O.save('mergedScene.yade')

#demonstrate getting stuff from workers, here we get kinetic energy from worker subdomains, notice that the master (mp.rank = 0), uses the sendCommand to tell workers to compute kineticEnergy.
if mp.rank==0:
        print("kinetic energy from workers: "+str(mp.sendCommand([1,2],"kineticEnergy()",True)))

Question information

Language:
English Edit question
Status:
Solved
For:
Yade Edit question
Assignee:
No assignee Edit question
Solved by:
NgocSon-Nguyen
Solved:
Last query:
Last reply:
Revision history for this message
Jérôme Duriez (jduriez) said :
#1

Hi,

This is not an error, it is just a warning (WARN) regarding the estimation of time step which was here attempted through PWaveTimeStep function. And this function needs spherical bodies to propose a time step, and here, probably due to MPI usage (which I have no experience of), spherical bodies can not be seen by PWaveTimeStep.

Maybe GlobalStiffnessTimeStepper could still do its job anyway (you can check asking for O.dt, and verifying whether it is still stuck to 1 or not). If not, you can also assign manually O.dt yourself.

Depending on the hardware you wanna use (and your exact simulations), you may also consider using OpenMP parallel executions of YADE. To do so, you just need to ask for e.g. yade -j2 vtkRecorderExample.py # to run on 2 CPU cores, without a need for modifying scripts.

That could be a simpler way, in case you would use a limited number of CPU cores anyway (?).

Revision history for this message
NgocSon-Nguyen (ngocson-nguyen) said :
#2

Hi Jérôme,

Thank you for your quick answer.

I will use the openMP to run my simulations without changing the script as you suggested. I have only 12 cores on my PC.

Best regards