gridSpacing and verletDist optimization

Hi David,

Thanks for this question. The choice of gridSpacing and verletDist is a complicated issue that can drastically affect simulation performance, either in terms of speed or memory usage. If the gridSpacing is too large, contact detection will take a long time and a lot of memory will be used. The reason for this is that each grid cell will contain a relatively large number of particles so contact detection starts to approach N^2 complexity instead of the optimal N log N complexity. On the flip-side, the verletDist can have a strong impact on accuracy of the simulation. If the verletDist is too large then contacts may not be detected or detected late. For those unfamiliar with the contact detection algorithm, the verletDist is defined as the distance any particle must move before an update of the neighbours table is enforced.

As far as I know, no-one has done a study on optimizing these values. I suspect it would depend on things like the bulk density of the particle assembly, the mean speed of particles, etc. In practice, I find the following rule-of-thumb works reasonably well in most circumstances:

gridSpacing = 2.2 * maxParticleRadius
verletDist = 0.2 * minParticleRadius

If the minParticleRadius is significantly smaller than the maxParticleRadius, you may be able to reduce the gridSpacing factor (2.2) a bit. The verletDist factor (0.2) should only be increased with extreme caution as its value is related to the Courant criterion governing numerical stability of the DEM algorithm. If the verletDist is too large you may see smaller particles flying out of the particle assembly in an unphysical manner, or in extreme cases, the particle assembly simply explodes.

David, if you have some measurements of simulation time for different gridSpacing and verletDist, I would be interested to see the results. If you wish to share these results, could you also provide a description of the type of simulations you are conducting (in terms of particle assembly geometry, spatial extents of the simulation domain, types of interactions etc.)?

Cheers,

Dion.

Hi Dion,

Thanks very much for the information. I ran a few, very small simulations (215 particles) for 100 time steps and varied the verletDist, gridSpacing and the mpiDimList.

Simulation Description
> Particles were bounded by a triMesh box of dimensions 0.01x0.01x0.01 m.
> Particle radii were selected from a normal distribution with a mean radius of 400E-6 m and a standard deviation of 100E-6 m
> minParticleRadius=127E-6 m
> maxParticleRadius=725E-6 m
> RotSphere particle type
> Interactions included GravityPrms, NRotElasticTriMeshPrms, RotFrictionPrms, and LinDampingPrms
> Particle packing was very loose and no particles were initially in contact---simple cubic seed lattice with a random perturbation of each lattice position
> timeStep=2.89E-8 s
> verletDist=Cv*minParticleRadius
> gridSpacing=Cg*maxParticleRadius
> mpiDimList[x,1,1]

Results
Cv,Cg,x,elapsed time [s],rate [time steps/s]
0.2,2.2,1,0.0799040794373,1251.50055797
0.2,2.2,2,0.137918949127,725.063529217
0.2,2.2,3,0.141515016556,706.638789535
0.2,22,1,0.0410671234131,2435.0378524
0.2,22,2,0.268573999405,372.33686143
0.2,22,3,error,size mismatch*
0.2,220,1,0.0405149459839,2468.22493703
0.2,220,2,0.267755031586,373.47570803
0.2,220,3,error,size mismatch*
0.02,220,1,0.0539438724518,1853.77866762
0.002,220,1,0.164556980133,607.692240822

Observations
> For x=1, speed increased with increasing gridSpacing
> For x=2, speed decreased with increasing gridSpacing
> Speed decreased with increasing number of processors. I hope this changes when I add more particles.
> Speed decreased with decreasing verletDist

*The full error observed: "rank = 0:ParallelParticleArray<T>::exchange_single ERROR: size mismatch in received data, recv_data.size()!=recv_slab.size() - (215!=0). dir = 0, dist = -1"

I will be generating larger simulations soon and will post any related observations. I wonder if increasing the number of particles may result in different trends. I would welcome any feedback regarding these results.

Thanks,
Dave

Hi David,

Thanks for the follow-up. I'm a bit suspicious that the large changes in run-time arise because of the very small number of particles you are currently using. The verlet-linked list neighbour search algorithm used in ESyS-Particle is optimal for relatively high bulk densities of particles with moderate velocities. It sounds like you have quite a low bulk density. Under those circumstances a lattice approach for neighbour searching is more suitable.

Regarding the use of multiple worker processes, I follow a rule-of-thumb that each worker process should handle at least 10000 particles. The rationale is that you want to ensure the amount of computation time for a worker exceeds communication time. In your test case I would not consider using more than one worker process (x=1).

I'm not surprised that speed decreases if you decrease the verlet distance. Doing this will cause the code to repopulate the neighbours table more frequently. Repopulating the neighbours table is a relatively expensive operation so one would prefer to minimise how often this is done by increasing the verlet distance. You don't want to increase it too much though because you will miss contacts if the verlet distance is too large.

Let me know if you notice significant run-time variations in your larger models.

Cheers,

Dion.

Hi Dion,

Thanks for the input. That makes sense. I was wondering if the small number of particles had something to do with it. I will see what I find with larger simulations.

Thanks,
Dave