# Abnormal increase in memory usage

Asked by Gao Xiang

Hi all:
I want to use many scenes in single simulations, so I used O.switchToScene() to achieve it, and used O.run(nsteps, True) to calculate for every step in a loop.
My question is that the memory usage will increase by about 3MB after every "O.run(nsteps, True) " code, because there are so many scenes and cycleindex , after executing "O.run(nsteps, True) " code many times， then Yade crashes .

Here are some part of my code:

.....
def shear2D(param)
swap = psutil.swap_memory()
O.switchToScene(param[0])
ns = int(min(max(1e5 * numpy.max(numpy.abs(param[1])), 2),50))
dstrain = utils.Matrix3(param[1][0], param[1][1], 0, param[1][3], param[1][2], 0, 0, 0, 0)
Omega().cell.velGrad = dstrain / (ns * Omega().dt)

print(u'Remaining memory_1：', psutil.virtual_memory().available / 1024.0 / 1024.0 / 1024.0)
print(u'used swap_1：', swap.used / 1024.0 / 1024.0 / 1024.0)

O.run(ns, True)

print(u'Remaining memory_2：', psutil.virtual_memory().available / 1024.0 / 1024.0 / 1024.0)
print(u'used swap_2：', swap.used / 1024.0 / 1024.0 / 1024.0)

deformation = Omega().cell.getSmallStrain().norm()
st_1 = utils.getStressAndTangent(symmetry=True)
s_1 = st_1[0]
s_1 = .5 * (s_1 + s_1.transpose())
stress_dem = [s_1[0, 0], s_1[0, 1], s_1[1, 1], s_1[2, 2]] # deformed stress
.....
# the loop likes followings
for i in scenesID:
shear2D((i,strain[i-1]))

I'm sure that the memory increases only relating to " O.run(ns, True)" by checking every each line of code，therefore I want to know the reason and how to fix it.
THANK YOU!!!

## Question information

Language:
English Edit question
Status:
Solved
For:
Assignee:
No assignee Edit question
Solved by:
Robert Caulk
Solved:
Last query:
 Revision history for this message Robert Caulk (rcaulk) said on 2022-06-29: #1

Hello,

Maybe an explanation for why you believe you need to add new scenes would be helpful. There is likely a superior method for achieving your principal objective.

That said, if we read the doc[1]:

switchToScene((Omega)arg1, (int)arg2) → None :¶
Switch to defined scene. Default scene has number 0, other scenes have to be created by addScene method.

We see that you should be using the addScene method if you wish to use switchToScene properly. I don't see addScene() in your script.

Cheers,

robert

 Revision history for this message Gao Xiang (gaoxiang22) said on 2022-06-29: #2

Hello, Robert
Thanks your reply, in fact, I have used the addScene method, seen the following code, firstly, adding the new secens by O.addScene(), and loading the yade packing from files, therefore, there are many scenes in a single simulation.

And for every scenes, I used switchToScene to convert to the corresponding scene， and started some calculation by O.run(nsteps, True) ， everything works well except the increasing memory.

I used the tools "psutil" to monitor memory for every line code, and I find the memory increasing occurs always after the code
“O.run(nsteps, True) ”，I'm really confused，maybe “O.run(nsteps, True) ” creat the new scene （allocate memory）and not delete the old scene（ free memory）？

Thank You！！！

###########################
inputPack(0)#load the initial pack as scene0
pack_list=inputPack3() #load all the RVEs file and return list
scenes=[0]
for i in range(len(pack_list)):

scenesID=O.addScene() #add new scenes and return the scenes' ID

scenes.append(scenesID)
#print(scenesID)
O.switchToScene(scenes[i+1]) #switch to new scene
scenestring=pack_list[i] #load the pack
Omega().stringToScene(scenestring) #from pack string to scene
return scenes
#############################################

 Revision history for this message Robert Caulk (rcaulk) said on 2022-06-30: #3

Hello,

Can you please explain WHY you believe this method is necessary as opposed to running yade-batch?

Cheers,

Robert

 Revision history for this message Gao Xiang (gaoxiang22) said on 2022-06-30: #4

Hello, Robert.
Thanks for your reply, I want to do some coupling calculation, like FEM×DEM in examples, so I need to use many RVEs in a single simulation, and allocate the RVE for a scene. In order to avoid high computing costs, I want to give up the "sceneToString" and "stringToScene" method in the examples of FEM×DEM, and I am trying to use switchToScene() to switch screen and run the simulation for some steps.

Therefore, I think the yade-batch may not achieve the goal? I have to solve the memory problem.

Sincerely,
Gao

 Revision history for this message Robert Caulk (rcaulk) said on 2022-07-01: #5

Ok.

Please provide an MWE - as we can only guess without seeing other interacting parts of your code.

Cheers,

Robert

 Revision history for this message Gao Xiang (gaoxiang22) said on 2022-07-02: #6

Hello, Robert.
Thanks for your reply, when I prepare an MWE, I might have found the problem through testing every line code. When I stopped the multiprocessing(Python) method, the script started working well.
Therefore, I think I need a more large memory, in my test script, I used 1024 RVEs with 400 particles, and my PC's memory is only 16GB.
When I am using "sceneToString" and "stringToScene" and multiprocessing method before, the script working well and the memory is enough, so I ingored it, and it seems like that the “switchToScene() ” needs much more memory, i still think it is a very good function to save many time.
Although, i can not explain why the memory increasing occurring after O.run(nsteps, True) at early stage, and it will be stable after several iterations.
I hope my experience can help others，now i think the MWE is not important, Should I turn the state of problem into "solved".

Again, thank Robert 's patient reply!

Sincerely,
Gao

 Revision history for this message Karol Brzezinski (kbrzezinski) said on 2022-07-02: #7

Hi Gao,

It is hard to tell without the MVE. I guess that maybe a 'clear scene' has no interactions, and when you run it interactions are created (which require some memory to be stored)?

Cheers,
Karol

 Revision history for this message Gao Xiang (gaoxiang22) said on 2022-07-04: #8

Hello, Karol.
Thank you for your reply and I thought that I have solve the problem, maybe not due to the inadequate interactions, but “switchToScene() ” needs much more memory compared with "sceneToString" and "stringToScene" method, especially when I have used the multiprocessing method in Python, so the memory is not enough.

I stopped the multiprocessing, the problem is solved, maybe I should add more memory to my PC.
Again，thanks all the help！

Sincerely,
Gao

 Revision history for this message Gao Xiang (gaoxiang22) said on 2022-07-04 #9

Thanks Robert Caulk and Karol Brzezinski , that solved my question.

To post a message you must log in.