memory leak while assigning boost::python attributes

Hi,

thanks for the bugreport. After a quick look one can say, that you do
not have any steps between creating and removing facets (just
between removing and creating).

Not sure, but it should help to fix your problem.

Anton

When you insert another O.run(5,True) after creating the facets, the memory leak is still there and also much higher, than without it!

Also I get a crash :o

Running script memoryleak-erase-facets.py

Message from syslogd@debian at Aug 24 10:56:01 ...
 kernel:[1542691.744027] EDAC MC0: UE row 0, channel-a= 0 channel-b= 1 labels "-": NON-FATAL recoverable (Branch=0 DRAM-Bank=0 Buffer ID = 0 RDWR=Read RAS=0 CAS=0 NON-FATAL recoverable Err=0x2000 (FB-DIMM Configuration Write error on first attempt))

2012/8/24 Christian Jakob <email address hidden>:
> Message from syslogd@debian at Aug 24 10:56:01 ...
> kernel:[1542691.744027] EDAC MC0: UE row 0, channel-a= 0 channel-b= 1 labels "-": NON-FATAL recoverable (Branch=0 DRAM-Bank=0 Buffer ID = 0 RDWR=Read RAS=0 CAS=0 NON-FATAL recoverable Err=0x2000 (FB-DIMM Configuration Write error on first attempt))

In this case the memory is blowing up, because you get an enormous number
of interactions (see the distance between facets).

Actually, the memory leakage will be also with spheres, not only facets, but
with spheres it will be not so quickly. I should say, this is a
"normal" behavior.

When you erase a body, you do not destroy it as an object. You just "erase" it
from the simulation, remove it from there. So the object is continuing to exist.

That is why you get an increasing of memory consumption, because you
create more and more body-objects.

Anton

Thank you Anton for the explanation.

Why is memory not purged after erasing a body?
Is it possible to clear memory from erased bodies?

> Why is memory not purged after erasing a body?
> Is it possible to clear memory from erased bodies?

I think, it is "connected" with python-variables.
Removing the python variable should destroy an object (need to try):

b = utils.facet()
id = O.bodies.append(b)
O.bodies.erase(id)
b=0

You can try such approach. Maybe it will help.

Anton

>b = utils.facet()
>id = O.bodies.append(b)
>O.bodies.erase(id)
>b=0
>
>You can try such approach. Maybe it will help.

No, this is unfortunately not solving the problem...

The problem could be linked to this example:

f = utils.facet(((1,0,0),(0,1,0),(0,0,1)))
n = 100000000
for i in xrange(n):
  f.shape.vertices = Vector3(1,0,0),Vector3(0,1,0),Vector3(0,0,1)

that also leads to memory leaks, while it should not..

Jan

2012/8/27 Christian Jakob <email address hidden>

> >b = utils.facet()
> >id = O.bodies.append(b)
> >O.bodies.erase(id)
> >b=0
> >
> >You can try such approach. Maybe it will help.
>
> No, this is unfortunately not solving the problem...
>
> --
> You received this bug notification because you are a member of Yade
> developers, which is subscribed to Yade.
> https://bugs.launchpad.net/bugs/1041084
>
> Title:
> memory leak while erasing facets
>
> Status in Yet Another Dynamic Engine:
> New
>
> Bug description:
> I detected a memory leak in O.bodies.erase, when facets were erased
> (spheres are erased without memory leak!).
> In the attachment you will find two files, where you can reproduce the
> bug.
>
> 1. Open a terminal with three tabs.
> 2. Run top or htop in the first terminal tab
> 3. In the second terminal tab run yade input file
> "memoryleak-erase-facets.py"
> 4. Get the pid from first terminal tab (example 31667) and edit
> "get-memory-info.py"
> 5. Run "python get-memory-info.py" in the third terminal tab and wait
> (ca. 5 min) to the end of the calculation.
>
> You will get an output file called "fig-memoryleak-erase-facets.png",
> where you can see the effect of the memoryleak.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/yade/+bug/1041084/+subscriptions
>
> _______________________________________________
> Mailing list: https://launchpad.net/~yade-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~yade-dev
> More help : https://help.launchpad.net/ListHelp
>

I am afraid, we can nothing to do with that.
Yade itself does not "work" itself in this example.

Anton

2012/8/30 Jan Stránský <email address hidden>:
> The problem could be linked to this example:
>
> f = utils.facet(((1,0,0),(0,1,0),(0,0,1)))
> n = 100000000
> for i in xrange(n):
> f.shape.vertices = Vector3(1,0,0),Vector3(0,1,0),Vector3(0,0,1)
>
> that also leads to memory leaks, while it should not..

Hi all,

As asked by Anton, I report a similar problem. This is related to the question : https://answers.launchpad.net/yade/+question/253112

In the C++ code, I created an engine(HydroForceEngine) which use a variable vxFluct. In my python script, I actualize the variable vxFluct each period tau doing :
HydroForceEngine.vxFluct = X

When X is big and tau is small, there is a memory leakage : for example, with 8300 bodies (so for len(X) = 8300), after ~6500 actualization of the variable, the simulation takes about 40% of the memory of my computer; while the same code after the same time and without actualizing the variable takes 0.5% of the memory of my computer.

Raphaël

Yet another example leaking:

e=ForceEngine()
for k in range(1000):
   e.ids=range(0,1000000)

While this, of course gives no problem:
ids=[]
for k in range(1000):
   ids=range(0,1000000)

It is very annoying because it is a common situation (although fortunately we assign small things in most cases so the leak is unnoticed).

It sounds like a broken reference counting.
If so, the solution must be somewhere in Serializable.hpp, maybe _DEF_READWRITE_BY_VALUE?
But I'm still clueless overall.

Hi there,

trying in Woo, I see some increase in the original Christian's script adapted for Woo (it takes 3.5G RAM at the end), which is clearly an issue, I don't see anything like that when re-assigning vertices to Facets as Jan reported. Can someone reproduce that with Yade?

You should also check that there are no shared_ptr circles in Yade, since those will be never de-allocated.

Cheers, vaclav

PS just to add to the picture, I have simulations where then thousands of particles are routinely added and removed and never noticed issues. Perhaps the increase is not as bit as in this case, though, and I just did not notice.

@Vaclav
I can reproduce #8 with Yade 2014-08-01.git-ca4370a.
Did you try #11? If it works for you we can pinpoint the differences.

The problem when erasing/creating bodies like in the OP is less specific, probably misleading. Anton is also adding/deleting bodies intensively without troubles.

@Bruno: unable to reproduce #11, tried this with Woo:

from woo.dem import *
e=FlowAnalysis()
import sys
for k in range(10000):
 sys.stderr.write('.')
 e.flags=range(0,1000000)

(FlowAnalysis.flags is declared as vector<int>). Memory usage is sitting at 0.6% with no change at all.

PS the python definition is "probably" [1] (that is, passing by value and not by reference, read-write and without the postLoad hook when modified), if you want to check it out. v.

[1] http://bazaar.launchpad.net/~eudoxos/woo/trunk/view/head:/lib/object/Object.hpp#L161

Thanks Vaclav,
Actually I don't see a difference between Yade and Woo regarding [1]. So I am puzzled...
Also surprised by the fact that make_setter takes "return_by_value" [2], but clearly I need more boost-reading to really understand.

[2] make_setter(&thisClass::attr,py::return_value_policy<py::return_by_value>())

moved from https://bugs.launchpad.net/yade/+bug/1041084 to https://gitlab.com/yade-dev/trunk/issues/15