Yade

Abnormal increase in memory usage

Asked by Gao Xiang

Hi all:
I want to use many scenes in single simulations, so I used O.switchToScene() to achieve it, and used O.run(nsteps, True) to calculate for every step in a loop.
My question is that the memory usage will increase by about 3MB after every "O.run(nsteps, True) " code, because there are so many scenes and cycleindex , after executing "O.run(nsteps, True) " code many times， then Yade crashes .

Here are some part of my code:

.....
def shear2D(param)
    swap = psutil.swap_memory()
    O.switchToScene(param[0])
    ns = int(min(max(1e5 * numpy.max(numpy.abs(param[1])), 2),50))
    dstrain = utils.Matrix3(param[1][0], param[1][1], 0, param[1][3], param[1][2], 0, 0, 0, 0)
    Omega().cell.velGrad = dstrain / (ns * Omega().dt)

    print(u'Remaining memory_1：', psutil.virtual_memory().available / 1024.0 / 1024.0 / 1024.0)
    print(u'used swap_1：', swap.used / 1024.0 / 1024.0 / 1024.0)

    O.run(ns, True)

    print(u'Remaining memory_2：', psutil.virtual_memory().available / 1024.0 / 1024.0 / 1024.0)
    print(u'used swap_2：', swap.used / 1024.0 / 1024.0 / 1024.0)

    Omega().cell.velGrad = utils.Matrix3.Zero
    deformation = Omega().cell.getSmallStrain().norm()
    st_1 = utils.getStressAndTangent(symmetry=True)
    s_1 = st_1[0]
    s_1 = .5 * (s_1 + s_1.transpose())
    stress_dem = [s_1[0, 0], s_1[0, 1], s_1[1, 1], s_1[2, 2]] # deformed stress
.....
 # the loop likes followings
 for i in scenesID:
         shear2D((i,strain[i-1]))

I'm sure that the memory increases only relating to " O.run(ns, True)" by checking every each line of code，therefore I want to know the reason and how to fix it.
THANK YOU!!!

Question information

Language:
English Edit question
Status:
Solved
For:
Yade Edit question
Assignee:
No assignee Edit question
Solved by:
Robert Caulk
Solved:
Last query:
Last reply:
Revision history for this message
Best Robert Caulk (rcaulk) said : 		#1

Hello,

Maybe an explanation for why you believe you need to add new scenes would be helpful. There is likely a superior method for achieving your principal objective.

That said, if we read the doc[1]:

switchToScene((Omega)arg1, (int)arg2) → None :¶
    Switch to defined scene. Default scene has number 0, other scenes have to be created by addScene method.

We see that you should be using the addScene method if you wish to use switchToScene properly. I don't see addScene() in your script.

Cheers,

robert

[1]https://yade-dem.org/doc/yade.wrapper.html#yade.wrapper.Omega.switchToScene

Revision history for this message
Gao Xiang (gaoxiang22) said : 		#2

Hello, Robert
Thanks your reply, in fact, I have used the addScene method, seen the following code, firstly, adding the new secens by O.addScene(), and loading the yade packing from files, therefore, there are many scenes in a single simulation.

And for every scenes, I used switchToScene to convert to the corresponding scene， and started some calculation by O.run(nsteps, True) ， everything works well except the increasing memory.

I used the tools "psutil" to monitor memory for every line code, and I find the memory increasing occurs always after the code
“O.run(nsteps, True) ”，I'm really confused，maybe “O.run(nsteps, True) ” creat the new scene （allocate memory）and not delete the old scene（ free memory）？

forwaing to your reply！

Thank You！！！

###########################
def loadallRVE():
    inputPack(0)#load the initial pack as scene0
    pack_list=inputPack3() #load all the RVEs file and return list
    scenes=[0]
    for i in range(len(pack_list)):

       scenesID=O.addScene() #add new scenes and return the scenes' ID

       scenes.append(scenesID)
       #print(scenesID)
       O.switchToScene(scenes[i+1]) #switch to new scene
       scenestring=pack_list[i] #load the pack
       Omega().stringToScene(scenestring) #from pack string to scene
    return scenes
#############################################

Revision history for this message
Robert Caulk (rcaulk) said : 		#3

Hello,

Can you please explain WHY you believe this method is necessary as opposed to running yade-batch?

Cheers,

Robert

Revision history for this message
Gao Xiang (gaoxiang22) said : 		#4

Hello, Robert.
Thanks for your reply, I want to do some coupling calculation, like FEM×DEM in examples, so I need to use many RVEs in a single simulation, and allocate the RVE for a scene. In order to avoid high computing costs, I want to give up the "sceneToString" and "stringToScene" method in the examples of FEM×DEM, and I am trying to use switchToScene() to switch screen and run the simulation for some steps.

Therefore, I think the yade-batch may not achieve the goal? I have to solve the memory problem.

Sincerely,
Gao

Revision history for this message
Robert Caulk (rcaulk) said : 		#5

Ok.

Please provide an MWE - as we can only guess without seeing other interacting parts of your code.

Cheers,

Robert

Revision history for this message
Gao Xiang (gaoxiang22) said : 		#6

Hello, Robert.
Thanks for your reply, when I prepare an MWE, I might have found the problem through testing every line code. When I stopped the multiprocessing(Python) method, the script started working well.
Therefore, I think I need a more large memory, in my test script, I used 1024 RVEs with 400 particles, and my PC's memory is only 16GB.
When I am using "sceneToString" and "stringToScene" and multiprocessing method before, the script working well and the memory is enough, so I ingored it, and it seems like that the “switchToScene() ” needs much more memory, i still think it is a very good function to save many time.
Although, i can not explain why the memory increasing occurring after O.run(nsteps, True) at early stage, and it will be stable after several iterations.
I hope my experience can help others，now i think the MWE is not important, Should I turn the state of problem into "solved".

Again, thank Robert 's patient reply!

Sincerely,
Gao

Revision history for this message
Karol Brzezinski (kbrzezinski) said : 		#7

Hi Gao,

It is hard to tell without the MVE. I guess that maybe a 'clear scene' has no interactions, and when you run it interactions are created (which require some memory to be stored)?

Cheers,
Karol

Revision history for this message
Gao Xiang (gaoxiang22) said : 		#8

Hello, Karol.
Thank you for your reply and I thought that I have solve the problem, maybe not due to the inadequate interactions, but “switchToScene() ” needs much more memory compared with "sceneToString" and "stringToScene" method, especially when I have used the multiprocessing method in Python, so the memory is not enough.

I stopped the multiprocessing, the problem is solved, maybe I should add more memory to my PC.
Again，thanks all the help！

Sincerely,
Gao

Revision history for this message
Gao Xiang (gaoxiang22) said (last edit ): 		#9

Thanks Robert Caulk and Karol Brzezinski , that solved my question.