PicklingError

Hi,
I'm afraid it is impossible to help in any way with such question.

> worked around many issues from this morning and it seems that I resolved most of them.

Which yade version? Which linux version? Which compile options? What kind of issues? How did you solve them? Is it of general interest (maybe you found bugs that we should fix so next ones don't face the same problems)? Else why is it mentioned?

> PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

We have no line number and we don't know how you get this error message (plus it could be due to some of your workarounds).

Please clarify and let us know if you find that the code or the documentation need some update.

Cheers

Bruno

Hi Bruno,

Thanks for getting back to me and sorry for keeping it so brief. I thought that a similar issue has popped up previously for someone else since this kind of coupling is not easy to implement. My YADE version is yade-2019.01a and I am working on Ubuntu 18.04. Here is a detailed explanation of what I did. I followed the instruction in the biaxialSmooth.py by
1- sudo apt install python-escript (Although I had installed escript from the source--see 3)

2- cd examples/FEMxDEM (So I am in the folder for FEMxDEM examples)

3- export PYTHONPATH="/usr/lib/python-escript:/home/ngoudarz/Desktop/myYade/install/lib/x86_64-linux-gnu/yade-2019.01a/py/yade/FEMxDEM"

Note: /usr/lib/python-escrip is the address of my python-escript Installed by sudo apt install python-escript (not from source). I first addressed the location of the folder containing the executable from the one I had installed from the source code but I received an error: ImportError: No module named escript. I resolved this by changing the address as explained above.

4- export LD_LIBRARY_PATH='/usr/lib/python-escript/lib'
Note: This is the address of the lib folder of my python-escript installed by sudo apt install python-escript (not from source).

5- ln -s '/home/ngoudarz/Desktop/myYade/install/lib/x86_64-linux-gnu/yade-2019.01a/py/yade/FEMxDEM/yadeimport.py'. The path after the ln -s is the location where I put my yade.import.py file and is the place where the simDEM.py file is also located.

6- When I run '/home/ngoudarz/Desktop/myYade/install/bin/yade-2019.01a' ./biaxialSmooth.py, I first got an error: ImportError: cannot import name QtWebkit (even before the error explained in 3 above). I resolved this by sudo apt-get install python-pyqt5.qtwebkit. I then got the error mentioned in 3 which I resolved by changing the address and then another error. ImportError: No module named yadeimport. I traced the source of the error in the simDEM.py (line 15). To resolve this I first run sudo apt-get install python-yade and then changed line 15 in the simDEM.py from "from yadeimport import *" to "rom yade import *" as Jan Stránský had proposed in https://answers.launchpad.net/yade/+question/670527. This resolved this issue and finally, I was able to start running the script by inputting '/home/ngoudarz/Desktop/myYade/install/bin/yade-2019.01a' ./biaxialSmooth.py

Note: I have installed Yade from the source and /home/ngoudarz/Desktop/myYade/install/bin/yade-2019.01a is the path to its executable.

Here is my execution:
ngoudarz@Nima-VPCZ122GX:~/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM$ sudo apt install python-escript
ngoudarz@Nima-VPCZ122GX:~$ cd '/home/ngoudarz/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM'
ngoudarz@Nima-VPCZ122GX:~/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM$ export PYTHONPATH="/usr/lib/python-escript:/home/ngoudarz/Desktop/myYade/install/lib/x86_64-linux-gnu/yade-2019.01a/py/yade/FEMxDEM"
ngoudarz@Nima-VPCZ122GX:~/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM$ export LD_LIBRARY_PATH='/usr/lib/python-escript/lib'
ngoudarz@Nima-VPCZ122GX:~/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM$ ln -s '/home/ngoudarz/Desktop/myYade/install/lib/x86_64-linux-gnu/yade-2019.01a/py/yade/FEMxDEM/yadeimport.py'
ngoudarz@Nima-VPCZ122GX:~/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM$ ln -s '/home/ngoudarz/Desktop/myYade/install/lib/x86_64-linux-gnu/yade-2019.01a/py/yade/FEMxDEM/yadeimport.py'

Here is the output:

Welcome to Yade 2019.01a
TCP python prompt on localhost:9000, auth cookie `asused'
XMLRPC info provider on http://localhost:21000
Running script ./retainingSmooth.py
=======================================================================
For better performance compile python-escript with direct solver method
=======================================================================
Press Enter to continue...

When I hit enter, this is what I get:

Traceback (most recent call last):
  File "/home/ngoudarz/Desktop/myYade/install/bin/yade-2019.01a", line 241, in runScript
    execfile(script,globals())
  File "./retainingSmooth.py", line 45, in <module>
    prob = MultiScale(domain=mydomain,ng=numg,np=nump,random=False,rtol=1e-2,usePert=False,pert=-2.e-5,verbose=True)
  File "/home/ngoudarz/Desktop/myYade/install/lib/x86_64-linux-gnu/yade-2019.01a/py/yade/FEMxDEM/msFEM2D.py", line 85, in __init__
    self.__scenes=self.__pool.map(initLoad,range(ng))
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
[[ ^L clears screen, ^U kills line. F12 controller, F11 3d view (use h-key for showing help), F10 both, F9 generator, F8 plot. ]]

In [1]:

As you see, the only step I skipped is run `mv biaxialSmooth.yade.gz 0.yade.gz` as described in biaxialSmooth.py to generate initial RVE packing. Do I need to do so? If yes how?

Noteworthy that the result folder containing gauss, packing, and vtk subfolders are built in the directory ( FEMxDEM) but never occupied by any files due to the error.
Anyway, this is the final stage of my journey toward running this FEMxDEM and I am looking forward to a workaround to resolve the issue.

Thanks so much for your time and patience

Hi,
If I understand correctly you are linking one version of escript, then you load another one at runtime. Not a good idea, stick to one.
Also, it appears that you build for python2.7, which is not really supported any more. Better try with python3.
Regards
Bruno

On 12/9/19 5:12 AM, Bruno Chareyre wrote:
> Your question #686704 on Yade changed:
> https://answers.launchpad.net/yade/+question/686704
>
> Status: Open => Answered
>
> Bruno Chareyre proposed the following answer:
> Hi,
> If I understand correctly you are linking one version of escript, then you load another one at runtime. Not a good idea, stick to one.
> Also, it appears that you build for python2.7, which is not really supported any more. Better try with python3.
> Regards
> Bruno
>

Hi Bruno,

Thanks a lot.

1- If I understand correctly you are linking one version of escript, then you load another one at runtime

Indeed, this is not the case. Both are from non-source. Installed ny sudo apt-get install python-escript

2- Also, it appears that you build for python2.7, which is not really supported any more. Better try with python3.

Do you mean the escript? or YADE? when I enter python shell from the terminal my Python ver is 2.7. You are right but how I can enforce YADE compilation to use Python 3 (are higher versions
like 3.7 also acceptable?) Do I need to install YADE from the scratch if you mean python 3 should be used for YADE?

Much obliged

As for Python 3: I think all Python versions starting with 3 are acceptable.

I understood you installed YADE from source. To use a Python3-consistent YADE version, you just have to repeat this process, after grabbing a recent/the latest version of source code (2019.01a is too old in this aspect, it relies on Python 2) and follow instructions at https://yade-dem.org/doc/installation.html.

You will be able to check your new ecosystem looking at the following message when you launch yade
####
Welcome to Yade xxxx
Using python version: 3.6.9 # see this line !
####

Thanks so much Jerome,

I was able to run it on my laptop without the need for enforcing Yade using python 3. It is all about paths. Where and how you creat yadeimport.py, where you export your python path,...

I tried to repeat the steps on a desktop but encountered a new issue:
Here it is:

Welcome to Yade 2019.01a
TCP python prompt on localhost:9000, auth cookie `ssdkae&#39;
XMLRPC info provider on http://localhost:21000
Running script ./biaxialSmooth.py
usage: biaxialSmooth.py [options] [ TABLE [SIMULATION.py] | SIMULATION.py[/nCores] [...]
]
biaxialSmooth.py runs yade simulation multiple times with different
parameters. See https://yade-dem.org/sphinx/user.html#batch-queuing-and-
execution-yade-batch for details. Batch can be specified either with parameter
table TABLE (must not end in .py), which is either followed by exactly one
SIMULATION.py (must end in .py), or contains !SCRIPT column specifying the
simulation to be run. The second option is to specify multiple scripts, which
can optionally have /nCores suffix to specify number of cores for that
particular simulation (corresponds to !THREADS column in the parameter table),
e.g. sim.py/3.
optional arguments:
-h, --help show this help message and exit
-j NUM, --jobs NUM Maximum number of simultaneous threads to run
(default: number of cores, further limited by
OMP_NUM_THREADS if set by the environment: 1)
-v, --version Print version and exit.
--job-threads NUM Default number of threads for one job; can be
overridden by per-job with !THREADS (or
!OMP_NUM_THREADS) column. Defaults to 1.
--force-threads Force jobs to not use more cores than the maximum (see
\-j), even if !THREADS colums specifies more.
--log FORMAT Format of job log files: must contain a $, % or @,
which will be replaced by script name, line number or
by description column respectively (default: $.@.log)
--global-log FILE Filename where to redirect output of yade-batch itself
(as opposed to \-\-log); if not specified (default),
stdout/stderr are used
-l LIST, --lines LIST
Lines of TABLE to use, in the format 2,3-5,8,11-13
(default: all available lines in TABLE)
--nice NICE Nice value of spawned jobs (default: 10)
--cpu-affinity Bind each job to specific CPU cores; cores are
assigned in a quasi-random order, depending on
availability at the moment the jobs is started. Each
job can override this setting by setting AFFINE
column.
--executable FILE Name of the program to run (default: /home/ngoudarz/De
sktop/myYade/install/bin/yade-2019.01a). Jobs can
override with !EXEC column.
--gnuplot FILE Gnuplot file where gnuplot from all jobs should be put
together
--dry-run Do not actually run (useful for getting gnuplot only,

for instance)
--http-wait Do not quit if still serving overview over http
repeatedly
--plot-update TIME Interval (in seconds) at which job plots will be
updated even if not requested via HTTP. Non-positive
values will make the plots not being updated and saved
unless requested via HTTP (see \-\-plot-timeout for
controlling maximum age of those). Plots are saved at
exit under the same name as the log file, with the
.log extension removed. (default: 120 seconds)
--plot-timeout TIME Maximum age (in seconds) of plots served over HTTP;
they will be updated if they are older. (default: 30
seconds)
--refresh TIME Refresh rate of automatically reloaded web pages
(summary, logs, ...).
--timing COUNT Repeat each job COUNT times, and output a simple table
with average/variance/minimum/maximum job duration;
used for measuring how various parameters affect
execution time. Jobs can override the global value
with the !COUNT column.
--timing-output FILE With \-\-timing, save measured durations to FILE,
instead of writing to standard output.
--randomize Randomize job order (within constraints given by
assigned cores).
--disable-pynotify Disable screen notifications
ngoudarz@ngoudarz:~/Desktop/myYade/trunk-2019.01a/examples/FEMxDEM$

As you can see, it is not an error showing I am on the right track. It just jump out from running. This does not happen on my laptop. Everything is similar I use Ubuntu 18.04 on both systems and exactly follow the same procedure for running this coupled simulation. Do you think it is something related to MPI? has anyone encountered a similar issue?

My other question is about the last version of the Yade source code. If 2019.01a is too old where I can find the latest version?

Thanks so much for your patience,

I have no experience with biaxialSmooth.py and you did not tell us which exact command you launched.

However, it looks like (as the message says) an incorrect way of launching yade-batch, see https://yade-dem.org/doc/user.html#batch-queuing-and-execution-yade-batch for details, as the message suggests.
Are you otherwise able to launch another (simpler) script ?

As for (latest) source code, you can find it at https://gitlab.com/yade-dev/trunk, see https://yade-dem.org/doc/installation.html#source-code

Thanks again,

The commands are exactly those mentioned in biaxialSmooth.py (please see the top of the script for the commands). My surprise is that I can run the same script on my laptop by exactly the same command I use on the desktop (it does not work here as I mentioned). I have always run Yade scripts by dragging the Yade executable into my terminal and then dragging the python script after that. This has always worked.

The problem with the Yade-batch link is that I do not have any table of parameters and this is why I am surprised why an ordinary way for running a script does not work.

The answer to your question is YES. I can run other scripts.

Also, as I don't clone YADE (usually), I use the download page for downloading the last release of YADE. If you go to the download page in the link you sent me, you can see that the latest release is Yade 2019.01a. Do I have to clone for a newer version?

Thanks again

> I have always run Yade scripts by dragging the Yade executable into my terminal and then dragging the python script after that.

Interesting method... We could imagine any difference in Operating System / Terminal between your laptop and desktop would lead to different behaviors for this method. Under warranty method is the classical one in Linux (as said in the first "doc" lines of biaxialSmooth.py):
$ pathToYade pathToScript # Linux basics are necessary here..

Anyway, your output seems to be exactly the same than what one would get with an incorrect use of a batch version of YADE, replacing "yade-batch" with "biaxialSmooth". A wild guess (??) would be that you, on one of your computers, somehow assigned the "biaxialSmooth" name to the batch version of your YADE executable.

This would be a bloody mess, I do not know whether that's the case.

As for the YADE version, 2019.01a is a release of last January, it's old (there should be a new release in few weeks). Again, if you want more recent features of YADE (up to you to decide) such as Python3, you have to update.