madgraph spawns too many jobs

Asked by Heather Russell on 2018-12-03

Hi,

I'm running VBS event generation in cluster mode (slurm) and I've just gotten in a little trouble with the admins because madgraph has spawned over 14k jobs.

I can't seem to tell how this number is decided - is there some way of finding out how many jobs you'll create, a priori?

I ran with:
run_mode = 1
cluster_type = slurm
cluster_queue = None
cluster_size = 150

and then when I submitted, I submitted as:

#SBATCH --time=7-00:00:00
#SBATCH --mem=10G
#SBATCH --cpus-per-task=32

I would have thought, at the very most, i'd get 32*150 = 4800 jobs, but that is clearly not the case.

I'm running with MG5_aMC_v2_6_3, and had to slightly modify cluster.py in order to get the submission to work (added the necessary --mem, -t, -A options to the sbatch command).

It works fine with simpler processes I tested, e.g. p p > l+ l-.

Thanks in advance,
Heather

Question information

Language:
English Edit question
Status:
Answered
For:
MadGraph5_aMC@NLO Edit question
Assignee:
No assignee Edit question
Last query:
2018-12-03
Last reply:
2018-12-04

Hi,

The cluster_size attributes is actually only use when you generate loop-induced processes.

For LO and NLO jobs, it is difficult to know how many jobs are going to be generated.
Since it depends of the number of Feynman Diagram and the number of events that you request.

If the default behaviour of the slurm class is to submit too many jobs.
You should create your PLUGIN class for the cluster
(you should have done the same for the
> (added the necessary --mem, -t,
> -A options to the sbatch command).

https://cp3.irmp.ucl.ac.be/projects/madgraph/wiki/Plugin#Example3:NewCluster

Example 3 and/or 4 are stuff that you can consider to reduce the number of jobs created.

Cheers,

Olivier

> On 3 Dec 2018, at 23:47, Heather Russell <email address hidden> wrote:
>
> Question #676574 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/676574
>
> Description changed to:
> Hi,
>
> I'm running VBS event generation in cluster mode (slurm) and I've just
> gotten in a little trouble with the admins because madgraph has spawned
> over 14k jobs.
>
> I can't seem to tell how this number is decided - is there some way of
> finding out how many jobs you'll create, a priori?
>
> I ran with:
> run_mode = 1
> cluster_type = slurm
> cluster_queue = None
> cluster_size = 150
>
> and then when I submitted, I submitted as:
>
> #SBATCH --time=7-00:00:00
> #SBATCH --mem=10G
> #SBATCH --cpus-per-task=32
>
> I would have thought, at the very most, i'd get 32*150 = 4800 jobs, but
> that is clearly not the case.
>
> I'm running with MG5_aMC_v2_6_3, and had to slightly modify cluster.py
> in order to get the submission to work (added the necessary --mem, -t,
> -A options to the sbatch command).
>
> It works fine with simpler processes I tested, e.g. p p > l+ l-.
>
> Thanks in advance,
> Heather
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Can you help with this problem?

Provide an answer of your own, or ask Heather Russell for more information if necessary.

To post a message you must log in.