SIGKILL and possible memory leakage

Hello Giulia.

could you please provide us with the script you use?

I personally did some simulations with memory leaks for no obvious reason..

Thanks
Jan
Dne 3.12.2012 17:56 "Giulia Macaro" <email address hidden>
napsal(a):

> New question #215877 on Yade:
> https://answers.launchpad.net/yade/+question/215877
>
> Hello,
>
> I am running a very long simulation, in which a big sphere is loaded on a
> packing of 50,000 smaller spheres settled under gravity force.
> After 4 days of running, the simulation ends without any particular
> reason. In the terminal it is only written "killed".
> I have rerun the same test in the debug mode, compiling Yade with debug=1,
> and then
>
> $ gdb /usr/bin/python
>
> GNU gdb (GDB) 7.1-ubuntu
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/bin/python...(no debugging symbols found)...done.
>
> (gdb) run
> /home/giulia/YetAnotherDynamicEngine/12.07.09_yade/YADE/bin/yade-bzr3040
> -j4 --debug 2012.11.28_1_realSph.py
>
> Starting program: /usr/bin/python
> /home/giulia/YetAnotherDynamicEngine/12.07.09_yade/YADE/bin/yade-bzr3040
> -j4 --debug 2012.11.28_1_realSph.py
> [Thread debugging using libthread_db enabled]
> Welcome to Yade bzr3040 (debug build)
> TCP python prompt on localhost:9000, auth cookie `cesaud'
> [New Thread 0x7fffd6f2c700 (LWP 7675)]
> [New Thread 0x7fffd672b700 (LWP 7676)]
>
> I did not get any information apart from [New Thread ... (LWP ... )] and
> [Thread ... (LWP ...) exited]
> and then at the end I got
>
> Program terminated with signal SIGKILL, Killed.
> The program no longer exists.
>
> I also tried
> (gdb) thread apply bt full
> But I did not get anything.
>
> And then
> (gdb) bt full
> which gave me
> No stack.
>
> I was recording the CPU and memory usage during the simulation. The
> machine has 8 cores and 12GB of memory. The CPU was constant around 40%
> (using 4 over 8 cores) but the total memory went from 31% to 99% in 24
> hours, then constant to 99% for the following 3 days, and then the test
> ended.
> I am not erasing any body. I am creating the spheres, apply gravity, wait
> that the deposition ends, create the big sphere, and push the big sphere in
> the packing very slowly.
> I was expecting to use a lot of memory since I have a lot of particles,
> but not that much... is that possible? I am wondering if the ending of the
> simulation is related to the memory.
>
>
> At this point I am also rerunning some periodic triaxial tests, with only
> 5,000 particles. The memory usage has increased from 22 to 32% in 3 hours
> of simulation, and it is still increasing... Has anybody encountered the
> same increasing of memory? Is it expected in Yade?
>
> Thank you
> Giulia
>
> --
> You received this question notification because you are a member of
> yade-users, which is an answer contact for Yade.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~yade-users
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~yade-users
> More help : https://help.launchpad.net/ListHelp
>

Your large simulation was definitely broken because of huge memory consumption (probably swap was used).
Why it leaks... difficult to say. It needs to be analysed on a small amount of spheres. It can be related
to a bug LP:1041084 [1].

Jan is right, a test script can be useful.

Anton

https://bugs.launchpad.net/yade/+bug/1041084

Hello,

I have run more tx tests with only 5,000 spheres:

I've restarted the computer, launched the test, and I have not opened any other program (not even a folder, firefox, mouse, or music).
After 3 hours of running with 4 out of 8 cores, the total memory usage increased from 11.13 to 11.51% (compared with the previous 22 to 32% which I had while running other programs).
At the end of the test, after 18 hours, the memory arrived up to 14.29%, which is considerably less than before.
It not a sceintific way, but it gives an idea of what is happening. Some observations:
- As soon as I start using the pc for other stuff, the memory used by Yade starts increasing faster
- Have you noticed the same behaviour?
- It is good news, I assume, that the memory does not increase much if I don't run anything else. But why, while I run other programs, does the memory used by Yade increase?

I am now trying to do the same (not using the pc at all) with the large simulation (which crashed with SIGKILL before). Can I decide how much memory Yade can use? Is there any command similar to -j4 given for the number of cores?

Here is the script of the large simulation. It took me 3 or 4 days only to settle the packing, and other 4 days in debug mode until it crashed.

Thanks for any suggestion
Giulia

#!/usr/bin/python
# -*- coding: utf-8 -*-
from yade import pack,log,params,timing,plot
if utils.runningInBatch()==False: from yade import qt
import os,sys,matplotlib
import numpy as np
from math import *
from yade import *

quick=False

#-----------------------------------------------------------------------
# D I M E N S I O N S
passing = [0, 5.72, 24.26, 35.07, 45.38, 55.17, 70.10, 79.58, 90.19, 100.0]
diameters= [0.600, 0.632, 0.710, 0.770, 0.810, 0.850, 0.940, 1.000, 1.060, 1.200]
radii = []; PSDup = 4
for i in range(len(diameters)): radii.append(diameters[i]*0.5e-3*PSDup)

Rmin = 0.632*0.5e-3*PSDup
Rmax = 1.200*0.5e-3*PSDup
AvgRadius= .400e-3*PSDup
Dpipe = 50e-3 # (m) pipe diameter

# Initial packing
dim0 = 5*2*AvgRadius
dim1 = 9*Dpipe
dim2 = 4*Dpipe
extra = 1*Dpipe # clearance between the packing and the periodic cell
# way of roughly estimating the initial packing
poroi = 0.76
porof = 0.5
nSph = int((1-porof)*(dim0*dim1*dim2)/(4/3*pi*AvgRadius**3))# approximate nSph to get the desired final packing
hi = nSph*(4/3*pi*AvgRadius**3)/((1-poroi)*dim0*dim1) # initial height of the packing
# Periodic cell dimensions
cell0 = dim0
cell1 = dim1+2*extra
cell2 = hi+2*extra
# Corners of the initial packing (in sp.particleSD)
minCorner = (0,extra,extra)
maxCorner = (dim0,extra+dim1,extra+hi)
#-----------------------------------------------------------------------
maxUnbalancedForce = 0.05
dporoMax = 0.001
dhppMax = 0.001
Damping = 0.5
#-----------------------------------------------------------------------
# M A T E R I A L S
O.materials.append( FrictMat(density=2650,young=70e10,poisson=0.3,frictionAngle=radians(0), label='bottomFloorMat'))
O.materials.append( FrictMat(density=2650,young=70e10,poisson=0.3,frictionAngle=radians(0), label='rightWallMat'))
O.materials.append( FrictMat(density=2650,young=70e10,poisson=0.3,frictionAngle=radians(0), label='leftWallMat'))
O.materials.append( FrictMat(density=2650,young=70e09,poisson=0.3,frictionAngle=radians(26), label='sandMat'))
O.materials.append( FrictMat(density=7850,young=20e10,poisson=0.3,frictionAngle=radians(26*2/3),label='pipeMat'))
#-----------------------------------------------------------------------
# G E O M E T R Y
n0 = 3; n1 = 5; n2 = 10 # lat direction 0; lat direction 1; vert direction 2
deltah0 = dim0/n0; deltah1 = dim1/n1; deltav = hi/n2
#"""
# ~~~ Periodic cell ~~~
O.periodic = True
O.cell.setBox(Vector3(cell0,cell1,cell2)) # set size of the cell (rectangular)

# ~~~ Wall ~~~

# bottom floor
si = 1./n1; sj = 1./n0
for i in range(0,n1):
 for j in range(0,n0):
  x1 = (j*sj*cell0,extra*1.0+i*si*(extra*0.0+dim1),extra); x2 = (j*sj*cell0,extra*1.0+(i+1)*si*(extra*0.0+dim1),extra); x3 = ((j+1)*sj*cell0,extra*1.0+(i+1)*si*(extra*0.0+dim1),extra); x4 = ((j+1)*sj*cell0,extra*1.0+i*si*(extra*0.0+dim1),extra)
  O.bodies.append([utils.facet([x1,x2,x3],wire=False,fixed=True,material='bottomFloorMat'),utils.facet([x1,x3,x4],wire=False,fixed=True,material='bottomFloorMat'),])
# right wall
si = 1./n2; sj = 1./n0
for i in range(0,n2):
 for j in range(0,n0):
  x1 = (j*sj*cell0,extra,extra+i*si*hi); x2 = ((j+1)*sj*cell0,extra,extra+i*si*hi); x3 = ((j+1)*sj*cell0,extra,extra+(i+1)*si*hi); x4 = ((j)*sj*cell0,extra,extra+(i+1)*si*hi)
  O.bodies.append([utils.facet([x1,x2,x3],fixed=True,material='rightWallMat'),utils.facet([x1,x3,x4],fixed=True,material='rightWallMat'),])
# left wall
si=1./n2;sj=1./n0
for i in range(0,n2):
 for j in range(0,n0):
  x1 = (j*sj*cell0,extra+dim1,extra+i*si*hi); x2 = ((j+1)*sj*cell0,extra+dim1,extra+i*si*hi); x3 = ((j+1)*sj*cell0,extra+dim1,extra+(i+1)*si*hi); x4 = ((j)*sj*cell0,extra+dim1,extra+(i+1)*si*hi)
  O.bodies.append([utils.facet([x1,x2,x3],fixed=True,material='leftWallMat'),utils.facet([x1,x3,x4],fixed=True,material='leftWallMat'),])

# ~~~ Particles ~~~
sp=pack.SpherePack()
sp.particleSD(minCorner,maxCorner,rMean=AvgRadius,periodic=False,name="",numSph=nSph,radii=radii,passing=passing)
packing = [utils.sphere(s[0],s[1],material='sandMat') for s in sp]
O.bodies.append(packing)

#"""
#-----------------------------------------------------------------------
# bodies ids
ids_bottomFloor=[]; ids_leftWall=[]; ids_rightWall=[]; ids_sand=[]
for b in O.bodies:
 if b.material.label=='bottomFloorMat': ids_bottomFloor.append(b.id)
 elif b.material.label=='leftWallMat': ids_leftWall.append(b.id)
 elif b.material.label=='rightWallMat': ids_rightWall.append(b.id)
 elif b.material.label=='sandMat': ids_sand.append(b.id)

#-----------------------------------------------------------------------
# E N G I N E S
O.engines=[
 ForceResetter(),
 InsertionSortCollider([Bo1_Box_Aabb(),Bo1_Sphere_Aabb(),Bo1_Facet_Aabb(),],allowBiggerThanPeriod=True),
 InteractionLoop(
  [Ig2_Sphere_Sphere_ScGeom(),Ig2_Box_Sphere_ScGeom(),Ig2_Facet_Sphere_ScGeom(),],
  [Ip2_FrictMat_FrictMat_MindlinPhys (gamma=0.0,betan=0,betas=0),],
  [Law2_ScGeom_MindlinPhys_Mindlin (includeAdhesion=False,preventGranularRatcheting=True,includeMoment=True,label='contactLaw'),] ),
 TranslationEngine(velocity=0,translationAxis=[0,0,1],ids=[0],dead=True,label='tEng'),
 GravityEngine (gravity= (0,0,0), label='gravEng'),

 NewtonIntegrator (damping= Damping, label='newton'),
 PyRunner(command='gravityfunction(-9.81)',iterPeriod=50000, label='gravityfunc'),
 PyRunner(command='deposition()', iterPeriod=1000, dead=False, label='mainfunction'),
]
#-----------------------------------------------------------------------
flag = 1
O.dt=0.5*utils.PWaveTimeStep()
#-----------------------------------------------------------------------
# F U N C T I O N S

def deposition():
 fName = sys._getframe().f_code.co_name
 global flag
 if quick==False:
  if O.iter<1e7: return
 if gravEng.gravity[2]!=-9.81: return
 if 2*O.interactions.countReal()*1./len(O.bodies)<=1: return
 mainfunction.command='waitForDeposition()'

poro_prev = poroi; poro_curr = poroi; dporo = poroi
def waitForDeposition():
 fName = sys._getframe().f_code.co_name
 global poro_prev,poro_curr,dporo
 poro_prev = poro_curr
 poro_curr = porosityfunction()
 dporo = poro_prev-poro_curr
 if utils.unbalancedForce()<maxUnbalancedForce and dporo<=dporoMax:
  mainfunction.command = 'depositionEnd()'

def depositionEnd():
 global flag
 fName = sys._getframe().f_code.co_name
 if flag==1:
  flag=2
  newton.damping = 0.0
  mainfunction.command = 'waitForDeposition()'
  return
 if flag==2:
  if utils.unbalancedForce()<maxUnbalancedForce and dporo<=dporoMax:
   flag=3
 if flag==3:
  generatePipe()
  flag=5
 if flag==4:
  mainfunction.command='pipeDeposition()'
 if flag==5:
  O.engines[3].velocity=0
  O.engines[3].ids = [pipe.id]
  O.engines[3].dead = False
  colorSpheres(htop()/20)
  mainfunction.command = 'loading()'

def colorSpheres(beta):
 for bi in ids_sand:
  bzp=(O.bodies[bi].state.pos[2]-extra)//beta
  if bzp%2==0:
   O.bodies[bi].shape.color=Vector3(0/255.,250.0/255.,154.0/255.)
  else:
   O.bodies[bi].shape.color=Vector3(1.0,0.0,0.0)

loadVel=-0.05
def loading():
 global flag, zpp0,wpp, flagInt,loadVel,normEmbedment, vStar,xpp0
 fName = sys._getframe().f_code.co_name
 if flag==5:
  O.engines[3].velocity = loadVel; flag=6
 flagInt=0
 if flag==6:
  for i in ids_sand:
   if flagInt==0:
    try:
     int=O.interactions[i,pipe.id]
     if int.isReal==True:
      zpp0 = pipe.state.pos[2]
      xpp0=0
      flagInt = 1
      flag = 7
    except StandardError:
     int=0
 if flag==7:
  wpp = zpp0-pipe.state.pos[2]
  normEmbedment = -(zpp0-pipe.state.pos[2])/Dpipe,
  if abs(wpp)/Dpipe>=0.5: O.pause()

#-----------------------------------------------------------------------
# P I P E
z_cyl=float('nan')
def generatePipe():
 wasRunning=True if O.running==True else False
 if wasRunning==True:O.pause()
 fName = sys._getframe().f_code.co_name
 global pipe, z_cyl, ids_pipe
 radius_cyl = Dpipe/2
 z_cyl = extra+htop()+radius_cyl
 center_cyl = Vector3(dim0/2.0,extra+2.5*Dpipe,z_cyl)
 length_cyl = dim0*2.
 radius_sph = dim0/2*0.9
 z_sph = extra+htop()+radius_sph
 center_sph = Vector3(dim0/2.0,extra+dim1/2,z_sph)
 pipe = utils.sphere(center=center_sph, radius=radius_sph,material='pipeMat', fixed=False, color=(0,0,1))
 pipe.state.blockedDOFs='xXYZ'
 O.bodies.append(pipe)
 pipe.dynamic= False
 ids_pipe = pipe.id
 if wasRunning==True:O.run()

def pipeInteractions():
 global pipeInt
 pipeInt=[]
 for i in O.interactions:
  if i.id2==pipe.id:
   pipeInt.append(i.id1)
 return pipeInt

def htop(): # height from the bottom floor
 top=0
 for itop in ids_sand:
  if O.bodies[itop].state.pos[2]>top:
   top=O.bodies[itop].state.pos[2]
 return (top-extra)

def porosityfunction():
 global currPoro
 volPack=htop()*dim0*dim1
 volSph=utils.getSpheresVolume()
 currPoro=(volPack-volSph)/volPack
 return currPoro

def gravityfunction(g0):
 nSteps=100 if quick==False else 1
 dg=g0*1.0/nSteps
 temp=gravEng.gravity[2]
 if temp<=g0:
  gravEng.gravity=(0,0,-9.81)
 else:
  gravEng.gravity=(0,0,temp+dg)

#-----------------------------------------------------------------------
O.trackEnergy=True
O.timingEnabled=True # to enable timing services
O.saveTmp()
#-----------------------------------------------------------------------

Regarding the memory leakage described in https://bugs.launchpad.net/yade/+bug/1041084, I understood it only happens when erasing a body, doesn't it?

Hi.

2012/12/6 Giulia Macaro <email address hidden>:
> I've restarted the computer, launched the test, and I have not opened any other program (not even a folder, firefox, mouse, or music).
> After 3 hours of running with 4 out of 8 cores, the total memory usage increased from 11.13 to 11.51% (compared with the previous 22 to 32% which I had while running other programs).
> At the end of the test, after 18 hours, the memory arrived up to 14.29%, which is considerably less than before.
> It not a sceintific way, but it gives an idea of what is happening. Some observations:
> - As soon as I start using the pc for other stuff, the memory used by Yade starts increasing faster

How do you measure, whether the memory, used by Yade, is increased?

> - Have you noticed the same behaviour?
> - It is good news, I assume, that the memory does not increase much if I don't run anything else. But why, while I run other programs, does the memory used by Yade increase?

At least for linux-based systems it should not be so at all. Can other
applications eat all the memory?

> I am now trying to do the same (not using the pc at all) with the large
> simulation (which crashed with SIGKILL before). Can I decide how much
> memory Yade can use? Is there any command similar to -j4 given for the
> number of cores?

No, memory is allocating dynamically depending on the number of bodies,
interactions etc.

Anton

What Yade version do you use?

Anton

2012/12/6 Giulia Macaro <email address hidden>:
> Question #215877 on Yade changed:
> https://answers.launchpad.net/yade/+question/215877
>
> Status: Answered => Open
>
> Giulia Macaro is still having a problem:
> Hello,
>
> I have run more tx tests with only 5,000 spheres:

Could you try without this?

O.trackEnergy=True
O.timingEnabled=True # to enable timing services

If it doesn't change the leak, then it would be also useful to remove all the postprocessing stuff, e.g. volSph=utils.getSpheresVolume(), to see if there is a leak in a very core part of the code.

The ultimate tool to detect leaks is valgrind:
valgrind --leak-check=full yade myscript

However, I didn't try recently, and it may well give a huge amount of results due to python now, independently of yade, which does not help to find the relevant data.

I was recording the total memory with sysstat, and the memory used only by Yade with the script given in the other bug, following the process id of Yade.
The other applications open were just firefox, nautilus and a music player, maybe open office sometimes. Not that much.

Actually I haven't updated yade in a while, I'm using bzr 3040.

Could you try without this?

O.trackEnergy=True
O.timingEnabled=True # to enable timing services

I'll try it. I've also tried to not plot or save anything, but it didn't make any difference.

I've never used Valgridnd, I'll try it.

Thanks,
Giulia

Not directly related to your question, but defining the limits of your problem with so many facets is really the source of potential problems. There are known bugs with such arrangement.
I would replace them by 3 boxes.

Well, some of my simulations have thousands of facets and I did not
see such difficulties.

Anton

2012/12/6 Chareyre <email address hidden>:
> Question #215877 on Yade changed:
> https://answers.launchpad.net/yade/+question/215877
>
> Chareyre posted a new comment:
> Not directly related to your question, but defining the limits of your problem with so many facets is really the source of potential problems. There are known bugs with such arrangement.
> I would replace them by 3 boxes.

When you need many facets, you have to live with them, and it's probably less a problem when they represent a complex geometry.
But Giulia is discretizing a plane into a large number of facets, which is really useless. It could be replaced by one box, or one big facet, even.
We know that coplanar facets are prone to give weird behavior when contact is on an edge.

B

At the beginning I decided to use more facets so that I could read the force acting on different layers of the vertical walls, and also the force at the centre and at the side of the bottom floor.
But if it might causes problems I can get the forces from the spheres, I guess.

Giulia

With the latest yade-daily and about 8h of simulation I do not see any
memory leakage in your script.

Cheers,

Anton

2012/12/6 Giulia Macaro <email address hidden>:
> Question #215877 on Yade changed:
> https://answers.launchpad.net/yade/+question/215877
>
> Giulia Macaro posted a new comment:
> At the beginning I decided to use more facets so that I could read the force acting on different layers of the vertical walls, and also the force at the centre and at the side of the bottom floor.
> But if it might causes problems I can get the forces from the spheres, I guess.
>
> Giulia

Thanks Anton for trying my script. I have found that if I restart the pc just before starting the test and I do not run other programs apart from Yade, it works fine for me too.
I have run some more tests on the same script, recording the total memory and also the memory used by Yade only, using sysstat. Every time I was restarting the pc before starting a new test.

I have found no increasing of memory (or only very slightly, from 2.38 to 2.55 % in 44 hours) if no other processes are running. But if then I start using other programs (firefox, nautilus, and geany) at the same time, the memory used by Yade increases faster (from 2.55 to 2.80 % in the following 3 hours).
The same script, but with more functions to periodically save the simulation, gives a memory usage increasing from 2.21 to 4.02 in the first 23 hours (with no other programs running) and then from 4.02 to 8.23 % in the following 3 hours (with others programs running too).

I understand if only the total memory would increase, but I do not see why Yade uses more memory if I open firefox, for example. Or maybe I am missing something about how the pc works.
I am wondering if someone has noticed the same behaviour before. Anyway, I am also happy to not use the pc at all while running Yade and keep the memory low.

>I understand if only the total memory would increase, but I do not see why Yade uses more memory if I open firefox, for example.

If you run -j4 on a quad-core CPU, I can imagine that other apps significantly disturb the timeline of the parallel execution. Then if some leakage was due to poor sharing of the workload between the cores. It could explain.
Still, it should not happen, I agree.

Thank you for reporting. Please let us know if you find more about this.