Extending amcatnlo_configuration.txt for batch submission

Asked by Spyridon Argyropoulos

Hi,

due to batch system-specific requirements, we have to modify the submission code to include memory requirements and cpu time requirements.
Modifying the madgraph/various/cluster.py file does the job, however, we would like to be able to do this in a more general way instead of changing the source code for every process.
What we need is a way to read 2 strings from the run_card.dat through madgraph/interface/common_run_interface.py and pass it to cluster.py, e..g. in this case amcatnlo_configuration.txt would look like this:

cluster_type = sge
cluster_queue = 48h.q
cluster_time = h_cpu=00:10:00
cluster_memory = h_vmem=5500M

My attempt to modify the 2 python files failed (the strings were apparently not being read by the common_run_interface.py). Could you provide an example of how to pass these 2 strings from common_run_interface.py to cluster.py?

I could send the modified code if needed, however I couldn't find any functionality here to attach files.

Thanks,
Spyros

PS. I noticed that cluster.py in PROC/bin/internal/cluster.py is overwritten each time that someone launches the event generation from madgraph/various/cluster.py. I think it would be safer to use the file that is copied into the process directory, so that one doesn't have to mess around with the files that are provided with the original code bundle.

Question information

Language:
English Edit question
Status:
Solved
For:
MadGraph5_aMC@NLO Edit question
Assignee:
No assignee Edit question
Solved by:
Olivier Mattelaer
Solved:
Last query:
Last reply:
Revision history for this message
Best Olivier Mattelaer (olivier-mattelaer) said :
#1

Dear Spyros,

> PS. I noticed that cluster.py in PROC/bin/internal/cluster.py is overwritten each time that someone launches the event generation from madgraph/various/cluster.py. I think it would be safer to use the file that is copied into the process directory, so that one doesn't have to mess around with the files that are provided with the original code bundle.

In principle the behavior is the following:

If you launch your generation via the script ./bin/mg5 (and the command launch)
the file which is used is madgraph/various/cluster.py.

On the other hand if you launch your script via
PROC_PATH/bin/aMCatNLO or PROC_PATH/bin/generate_events
then the file which is used is PROC/bin/internal/cluster.py

They are two reasons for such behavior:
1) Python has trouble with loading multiple files with the same name
2) this behavior is practical for debugging/development since you can directly modify the python file
madgraph/various/cluster.py and directly test it in a process without the need to regenerate the process directory.

The content of PROC/bin/internal/ is overwritten only if you run the following command (inside ./bin/mg5):
output PROC

Is it coherent with the situation that you face? If not this should be consider as a bug.

> What we need is a way to read 2 strings from the run_card.dat through madgraph/interface/common_run_interface.py and pass it to cluster.py, e..g. in this case amcatnlo_configuration.txt

Here I'm confused, do you want to include those lines in the file run_card.dat or in the file amcatnlo_configuration.txt?
I would personally include those lines in amcatnlo_configuration.txt files.

I'll create a simple patch in order to pass any information that you add in the file amcatnlo_configuration.txt available in the cluster.py class.
I'm testing it right now, and will attach it to this as soon as this is fully tested (I'll created a bug report associate to this question, simply because we can't attach file for a question)

Cheers,

Olivier

On May 28, 2013, at 4:36 PM, Spyridon Argyropoulos <email address hidden> wrote:

> New question #229766 on MadGraph5:
> https://answers.launchpad.net/madgraph5/+question/229766
>
> Hi,
>
> due to batch system-specific requirements, we have to modify the submission code to include memory requirements and cpu time requirements.
> Modifying the madgraph/various/cluster.py file does the job, however, we would like to be able to do this in a more general way instead of changing the source code for every process.
> What we need is a way to read 2 strings from the run_card.dat through madgraph/interface/common_run_interface.py and pass it to cluster.py, e..g. in this case amcatnlo_configuration.txt would look like this:
>
> cluster_type = sge
> cluster_queue = 48h.q
> cluster_time = h_cpu=00:10:00
> cluster_memory = h_vmem=5500M
>
> My attempt to modify the 2 python files failed (the strings were apparently not being read by the common_run_interface.py). Could you provide an example of how to pass these 2 strings from common_run_interface.py to cluster.py?
>
> I could send the modified code if needed, however I couldn't find any functionality here to attach files.
>
> Thanks,
> Spyros
>
> PS. I noticed that cluster.py in PROC/bin/internal/cluster.py is overwritten each time that someone launches the event generation from madgraph/various/cluster.py. I think it would be safer to use the file that is copied into the process directory, so that one doesn't have to mess around with the files that are provided with the original code bundle.
>
> --
> You received this question notification because you are a member of
> MadTeam, which is an answer contact for MadGraph5.

Revision history for this message
Spyridon Argyropoulos (spyridon-argyropoulos) said :
#2

Thanks Olivier Mattelaer, that solved my question.

Revision history for this message
Spyridon Argyropoulos (spyridon-argyropoulos) said :
#3

Hi Olivier,

thanks for the clarifications.

Concerning the python behavior with respect to loading cluster.py, indeed what you describe is consistent with what I am seeing. I was using ./bin/mg5 instead of the executables in the PROC dir.

Concerning your second question, indeed these strings should better be in amcatnlo_configuration.txt. Sorry for the confusion. I am looking forward to testing your patch.

Thanks and best regards,
Spyros