Parallel Processing with Escript

Asked by Doug White


Is escript compatible with any other python parallel processing packages, or does it only work with MPI? For my specific problem I am using escript to run some diffusion simulations and I would like to have many of them running at once to increase efficiency. I was wondering if there was a way to have multiple different simulations and run them using a python parallel processing package like Parallel Python for example.


Question information

English Edit question
esys-escript Edit question
No assignee Edit question
Solved by:
Doug White
Last query:
Last reply:
Revision history for this message
Lutz Gross (l-gross) said :

I assume what you have in mind is to write a script which splits the available ranks into groups and run escript on each of the groups. This should no be a problem if you don't use the MPI version of escript (you still could use OpenMP parallelization in escript if your MPI allows you to do this).

If you still what to use the escript MPI version, this is possible but not out of the box as it needs some C++ programming. The problem is that the domain generators don't accept MPI communicators form python however the escript library supports them on the C++ level. the reason why we don't exposed this to the python level yet is that we didn't have the need and couldn't make up our mind which python interface for MPI to use.

There is also the possibility to execute escript as a separate process - via ssh if you need to perform escript remotely (with MPI and or OpenMP) and using sockets to transfer data (if there aren't many of them). There are other solution along these lines depending on your system,

Revision history for this message
Doug White (dewtennis222) said :

I am sorry I am not familiar with your terminology when you are referring to ranks. I have the following situation: I have 10 different simulations using escript which I would like to run at the same time. I would like to run each on a separate processor to start, and was hoping to accomplish this using the Parallel Python package. However, when I do this I get an error, and thus I was wondering if escript was compatible with other packages besides MPI. I tried this without using the MPI version. I am currently operating on Windows 7 64 bit machine with 12 available threads. I wanted to start just by writing parallel code for my own machine, but am not ruling out moving to a larger network if necessary.

Revision history for this message
Lutz Gross (l-gross) said :

I am not familiar with "Parallel Python". From my understanding it is not using MPI but I think it will be difficult to use the escript MPI version in connection with Parallel python. However, I would expect that the non-MPI version is compatible with Parallel python. What error message do you get?

Revision history for this message
Doug White (dewtennis222) said :

Ok this is what I thought. However, when I run the programs I get an error that says that one of modules does not support pickeling, or can't be pickled. This may not be a problem on your side, and I am in touch with the other developers to make sure my calls have the correct syntax and form to avoid any such errors. I will let you know when I get a resolution to this issue.

Revision history for this message
Joel Fenwick (j-fenwick1) said :

Unless it is buried somewhere in one of the libraries we use, we don't use pickling at all.
We don't implement any of the interfaces required for pickling so it might be that something in parallel python is trying to pickle a data object to send it to another rank.

Revision history for this message
Doug White (dewtennis222) said :

This does appear to be an issue with PP. Using another technique I was able to parellelize this just fine with no hassles.