save/load precedure

Asked by azim

Dear all,
i want to run a simulation(scriptA.py) to reach a stable state and save it with O.save('stableState.yade.gz') then do some plots in the same script.
after that i want to load saved file with O.load('stableState.yade.gz') in another script(scrptB.py), do some runs,make some plots and again save it as O.save('firstcaculation.yade.gz') for loading in third script(ScriptC.py)
and finally run ScriptC.py and do the rest of the simulation and finally save my final state with O.save('finalState.yade.gz').

Take a look at [1] please.
here is my question:
what parameters are supposed to be saved in every save/load procedure? and what parameters show be defined again in new script??

[1] My Simulation Procedure:
##########################
ScrptA.py
.........(define parameters, packing, engines,......)
.........(some runs)
O.Save('StableState.yade.gz')
..............(some plots)
##########################
ScrptB.py
O.load('StableState.yade.gz')
.........(some runs)
O.Save('firstcalculation.yade.gz')
..............(some plots and......)
##########################
ScriptC.py
O.load('firstcalculations.yade.gz')
..........(some runs)
..........(some plots and .....)
O.save('finalsatate.yadegz.py')
##########################

thanks in advance
Azim

Question information

Language:
English Edit question
Status:
Solved
For:
Yade Edit question
Assignee:
No assignee Edit question
Solved by:
Bruno Chareyre
Solved:
Last query:
Last reply:
Revision history for this message
Bruno Chareyre (bruno-chareyre) said :
#1

>what parameters are supposed to be saved in every save/load procedure?

The content of "O" (bodies, engines, interactions).

>and what parameters show be defined again in new script??

Everything else.

You can escape the problem by not using two different scripts, and still avoid re-running the first part.
Start the script with:

if (saved):
     reload()
else:
    #create packing

All function definitions and parameters should be outside this if-else, of course, so they are re-defined consistently every time.

I hope it helps

Bruno

Revision history for this message
azim (mirzavand) said :
#2

if the below procedure is True? it means that my problem is solved:
for what i mentioned in [1], it will be something like this:

Script A&B&C in one ScriptABC:
##########################
ScrptABC.py

(define parameters,engines)

   if (saved=StableState.yade.gz):
        reload(StableState.yade.gz)
        (some runs and plots)
         O.save('firstcalculation.yade.gz')
         call y function

     elif (Saved= StableState.yade.gz and firstCalculation.yade.gz):
          reload('firstCalculation.yade.gz')
          call y function

   else:
        createpackings
         (some runs and plots)
         O.Save('StableState.yade.gz')
         call x function

def x function:
          (some runs and plots)
           O.save('firstcalculation.yadegz.py')
           call y function

def y function:
           (some runs and plots)
          O.save('finalsatate.yadegz.py')
###########################

Revision history for this message
azim (mirzavand) said :
#3

sorry dear bruno, def x function and def y function are outside the if-else conditions!!!

Revision history for this message
Jérôme Duriez (jduriez) said :
#4

For more details regarding what is saved / loaded, looking into the source code, you may give a look at
https://yade-dem.org/doc/prog.html#serialization
and
https://yade-dem.org/doc/prog.html#yade-class-base-doc

Revision history for this message
azim (mirzavand) said :
#5

Hi Jerome,
thanks for your references.
I really don't want to go so deep. i just wanted to solve THE problem and correct comment in #2.
Thanks

Revision history for this message
Best Bruno Chareyre (bruno-chareyre) said :
#6

Hi Azim, it seems I did not answer your last post.
The problem in the logic is that you defined the function after using them.
And the condition "if (saved=StableState.yade.gz)" is not a valid syntax.
I just commited [1] a new example script which shows how to workaround the issue, please check it.
Bruno

[1] https://github.com/yade/trunk/commit/f47f962d8b650a4e9923582457d352a58a51cbab

Revision history for this message
azim (mirzavand) said :
#7

Hi Bruno,
[1] really solved my problem.
thanks.

this is what i really was looking for, as i said in #2:

# -*- coding: utf-8 -*-
# Copyright (C) 2010 by Bruno Chareyre *
# bruno.chareyre_at_grenoble-inp.fr *

## This short modified version of examples/triax-tutorial/script-session1.py shows how to reload saved states and continue
## without the need to save user variables and functions separately (a typical issue as seen in e.g.
## https://answers.launchpad.net/yade/+question/665314).
## Here the technique is combined with batch execution, getting some parameters from readParamsFromTable(),
## but it is not a requirement.
## Comments on the simulation itself can be found in script-session1.py

nRead=readParamsFromTable(
 num_spheres=1001,
 compFricDegree = 30,
 key='_triax_base_',
 unknownOk=True
)
from yade.params import table

## Define the filename for saved state. By including paramater values in the name
## we make sure that a new state is generated for each parameter set - including a different 'key' name.
initStateFilename="confined_N="+table.key+str(table.num_spheres)+"_fric="+str(table.compFricDegree)+"_.yade.gz"
secondStateFilename="load_N="+table.key+str(table.num_spheres)+"_fric="+str(table.compFricDegree)+"_.yade.gz"
thirdStateFilename="final_N="+table.key+str(table.num_spheres)+"_fric="+str(table.compFricDegree)+"_.yade.gz"
## Check if a saved state exists and proceed to function/variable declarations as usual.
import os.path
savedState = os.path.exists(initStateFilename)
savedState0 = os.path.exists(secondStateFilename)

## user defined variables
num_spheres=table.num_spheres
key=table.key
targetPorosity = 0.43
compFricDegree = table.compFricDegree
finalFricDegree = 30
rate=-0.02
damp=0.2
stabilityThreshold=0.01
young=5e6
mn,mx=Vector3(0,0,0),Vector3(1,1,1)

import time
startTime=time.time()

## user function saving variables, it will not be saved with the simulation; which is ok since it is always re-defined here.
from yade import plot
def history():
   plot.addData(e11=-triax.strain[0], e22=-triax.strain[1], e33=-triax.strain[2],
        ev=-triax.strain[0]-triax.strain[1]-triax.strain[2],
      s11=-triax.stress(triax.wall_right_id)[0],
      s22=-triax.stress(triax.wall_top_id)[1],
      s33=-triax.stress(triax.wall_front_id)[2],
      i=O.iter)

O.materials.append(FrictMat(young=young,poisson=0.5,frictionAngle=radians(compFricDegree),density=2600,label='spheres'))
O.materials.append(FrictMat(young=young,poisson=0.5,frictionAngle=0,density=0,label='walls'))
walls=aabbWalls([mn,mx],thickness=0,material='walls')
wallIds=O.bodies.append(walls)
O.engines=[
 ForceResetter(),
 InsertionSortCollider([Bo1_Sphere_Aabb(),Bo1_Box_Aabb()]),
 InteractionLoop(
  [Ig2_Sphere_Sphere_ScGeom(),Ig2_Box_Sphere_ScGeom()],
  [Ip2_FrictMat_FrictMat_FrictPhys()],
  [Law2_ScGeom_FrictPhys_CundallStrack()]
 ),
 GlobalStiffnessTimeStepper(active=1,timeStepUpdateInterval=100,timestepSafetyCoefficient=0.8),
 TriaxialStressController(maxMultiplier=1.+2e4/young, finalMaxMultiplier=1.+2e3/young, thickness = 0, stressMask = 7, internalCompaction=True,label="triax"),
 PyRunner(iterPeriod=20,command='history()',label='recorder'),
 NewtonIntegrator(damping=damp,label="newton")
]
def continueRun():
 triax.internalCompaction=False
 setContactFriction(radians(finalFricDegree))
 triax.stressMask = 5
 triax.goal2=rate
 triax.goal1=-20000
 triax.goal3=-20000
 newton.damping=0.1
 O.run(100,True)
 print "Total execution time (savedState0=",str(savedState0),"): ",str(time.time()-startTime),"s"
 O.save(thirdStateFilename)

## the cloud generation is made conditional only because makeCloud() takes time for large num_spheres. Regenerating an initial
## cloud before reloading a saved state would be harmless but useless since the cloud will be replaced by the saved state anyway.
## materials defined hereabove will also be replaced by the the reloaded ones (which are identical) with no real impact on
## initialization time, hence no real need for a condition in that case
from yade import pack
sp=pack.SpherePack()
if not savedState0:
 if not savedState:
  sp.makeCloud(mn,mx,-1,0.3333,num_spheres,False, 0.95,seed=1)
  O.bodies.append([sphere(center,rad,material='spheres') for center,rad in sp])
 triax.goal1=triax.goal2=triax.goal3=-10000

 ## If no dense state has been generated previously proceed to confinement, else reload
 if not savedState:
  print "No saved state - running isotropic confinement for num_spheres=", str(table.num_spheres),", compFricDegree=", str(table.compFricDegree),", key='",str(table.key),"'"
  while 1:
   O.run(1000, True)
   unb=unbalancedForce()
   print 'unbalanced force:',unb,' mean stress: ',triax.meanStress
   if unb<stabilityThreshold and abs(-10000-triax.meanStress)/10000<0.001:
    break
  while triax.porosity>targetPorosity:
   compFricDegree = 0.95*compFricDegree
   setContactFriction(radians(compFricDegree))
   O.run(500,1)
  print "Confinement achieved, save then proceed to deviatoric loading"
  O.save(initStateFilename)
 else:
  print "Saved state found - reload then proceed to deviatoric loading for num_spheres=", str(table.num_spheres),", compFricDegree=", str(table.compFricDegree),", key='",str(table.key),"'"
  O.load(initStateFilename)

 ## Run triaxial simulation starting from a dense packing
 triax.internalCompaction=False
 setContactFriction(radians(finalFricDegree))
 triax.stressMask = 5
 triax.goal2=rate
 triax.goal1=-10000
 triax.goal3=-10000
 newton.damping=0.1
 O.run(100,True)
 print "Total execution time (savedState=",str(savedState),"): ",str(time.time()-startTime),"s"
 O.save(secondStateFilename)
 continueRun()
else:
 print "second Saved state found - reload then proceed to deviatoric loading for num_spheres=", str(table.num_spheres),", compFricDegree=", str(table.compFricDegree),", key='",str(table.key),"'"
 O.load(secondStateFilename)
 continueRun()

Revision history for this message
azim (mirzavand) said :
#8

Thanks Bruno Chareyre, that solved my question.

Revision history for this message
De zhang (dzlearnyade) said :
#9

Dear azim,
I have met the same problem as yours, and I could not understand your solutions as reading your answers, I saved a script A as 'SaveA.yade.gz' or 'SaveA.yade', when I opened a new yade to O.load('SaveA.yade.gz') or O.load('SaveA.yade.gz'), I can't continue to run the script A, it showed that the global parameters and def() functions are not defined at all. How to solve this problem?

Revision history for this message
Luc Scholtès (luc) said :
#10

Try to redefine the functions and global variables (copy/paste from initial script) into your new simulation before calling O.run().

I usually create a new python script when I want to load a previous simulation which include

O.load('nameOfSimulation')

and the custom functions

def customFunction():
  do this and that

Then, you should be able to call for

O.run()

directly in it.

Luc

Revision history for this message
Bruno Chareyre (bruno-chareyre) said :
#11

>copy/paste from initial script

If it fits in a file, why would you generate a second file with the same content?

Not a good idea. As the functions will be progressively modified and improved it will lead to different versions of them in different scripts in many cases (or painful editing to go change the same function in every scripts in the list, but pain is why people will forget that part). Same remark for parameters, instead of changing them in one place you have to change them in multiple places, it can make benchmark execution very difficult.
That's why I would suggest self-contained scripts like [1] in answer #6.
In my view the multiscript approach is not something we want to advise to standard yade users: no clear advantage over a single script, leads to bad practices such as duplicated code, generates a lot of questions on launchpad. In complex cases it may be required, but not here.
Bruno

Revision history for this message
Luc Scholtès (luc) said :
#12

I just gave a solution to solve De Zhang's problem which was, basically:

How to reload a simulation without suffering from the undefined functions and variables?

But of course, you are absolutely right, we should anticipate such problem by including the dedicated lines in the initial script.

Revision history for this message
Bruno Chareyre (bruno-chareyre) said :
#13

@Luc
I see, but copy/paste is not something to recommend, in general.
"import something" maybe. "copy/paste" no.

I think the best answer to "How to reload without suffering from undefined?" is "reload in the same script" in most cases - given current design.

Bruno

Revision history for this message
De zhang (dzlearnyade) said :
#14

Hi Bruno,
I actually come to a bug message using the copy/paste script problem.
I save a result as '1.yade', and in the second script , it could be loaded.
but when I finished the second script and saved as '2.yade', the new yade could not reload the '2.yade' with the message 'postcode and materials were unavaible and report the bug'.
Can I have your email and sent the saved file and script to you?
And I can make sense of your method but I was a little confused your procedure about calling the saved file by file name?
I was making a simulation which need four stages to accomplish, how to deal with it?

Revision history for this message
De zhang (dzlearnyade) said :
#15

Hi Bruno,
My e-mail is <email address hidden>, would you please send me a mail so that I can mail the files to you? Thank you~
Best regard!
De Zhang

Revision history for this message
azim (mirzavand) said :
#16

Hi De zhang,
I had the problem and it is fixed now.
I think, if you show a little part of your code so that covers the section causing errors, will be helpful for solving the problem.
it worked for me.
Azim

Revision history for this message
Bruno Chareyre (bruno-chareyre) said :
#17

>I actually come to a bug message using the copy/paste script problem

After I reiterated that copy/paste was a wrong approach I'm asked to help solving a copy/paste problem... is that a joke? :)

>would you please send me a mail

No sorry. Please use launchpad + external repository if you need to share big files (this being said, note that I'm not going to download them). Or better, organize everything in one single short script that you will post here.

Bruno