problem while generating 1M of event while generation of 0.5M works

Asked by Wojciech Kotlarski

I'm trying to generate 1M of events for user defined model (throught FeynRules interface) on a PBS cluster. The problem is that everything works if I set the number of events to 0.5M, while it fails for a 1M sample. I'm attaching an output from the cluster. I cut out parts that get repeated over & over and do not give any additional information (they are denoted by ...). The first part gives log from generation of a 1M of events that ends with an error
IOError : [Errno 2] No such file or directory: '/home/2/wkotlarski/sgluons_with_tau/SubProcesses/P0_qq_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq/G1f/results.dat'
I've checked that this file is indeed not present. While, as you can see from the last part of the log, when I set the number of events to 0.5M it finishes ok. What you can also see is that although I requested 0.5M events only 388709 got unweighted. I'm using MadGraph5 v 1.5.1.

No module named madgraph.interface.extended_cmd
No module named madgraph
No module named madgraph
************************************************************
* *
* W E L C O M E to M A D G R A P H 5 *
* M A D E V E N T *
* *
* * * *
* * * * * *
* * * * * 5 * * * * *
* * * * * *
* * * *
* *
* VERSION 5.1.5.1 *
* *
* The MadGraph Development Team - Please visit us at *
* https://server06.fynu.ucl.ac.be/projects/madgraph *
* *
* Type 'help' for in-line help. *
* *
************************************************************
load configuration from /home/2/wkotlarski/sgluons_with_tau/Cards/me5_configuration.txt
load configuration from /home/2/wkotlarski/sgluons_with_tau/Cards/me5_configuration.txt
Using default eps viewer "evince". Set another one in ./input/mg5_configuration.txt
Using default web browser "firefox". Set another one in ./input/mg5_configuration.txt
generate_events -f 7TeV_600GeV --cluster
Will run in mode parton
Generating 1000000 events with run name 7TeV_600GeV
survey 7TeV_600GeV
compile directory
/home/2/wkotlarski/sgluons_with_tau/bin/internal/ufomodel/particles.py
write ./param_card.dat
Using random number seed offset = 21
Running Survey
Creating Jobs
Working on SubProcesses
    P0_gg_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq
 ...
 ...
 ...
    P0_qq_xxc_x_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam_xc_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam
 Idle: 4 Running: 5 Finish: 219
 Idle: 0 Running: 3 Finish: 225
 Idle: 0 Running: 1 Finish: 227
 Idle: 0 Running: 1 Finish: 227
 Idle: 0 Running: 1 Finish: 227
 Idle: 0 Running: 1 Finish: 227
 Idle: 0 Running: 1 Finish: 227
INFO: All jobs finished
End survey
refine 1000000
compile directory
Creating Jobs
Refine results to 1000000
    P0_gg_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq
 ...
 ...
 ...
    P0_qq_xxc_x_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam_xc_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam
 Idle: 890 Running: 281 Finish: 15
 ...
 ...
 ...
 Idle: 0 Running: 1 Finish: 1185
INFO: All jobs finished
Combining runs
Start waiting for update on filesystem. (more info in debug mode)
Command "generate_events -f 7TeV_600GeV --cluster" interrupted with error:
IOError : [Errno 2] No such file or directory: '/home/2/wkotlarski/sgluons_with_tau/SubProcesses/P0_qq_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq/G1f/results.dat'
Please report this bug on https://bugs.launchpad.net/madgraph5
More information is found in '/home/2/wkotlarski/sgluons_with_tau/7TeV_600GeV_tag_1_debug.log'.
Please attach this file to your report.
quit

No module named madgraph.interface.extended_cmd
No module named madgraph
No module named madgraph
************************************************************
* *
* W E L C O M E to M A D G R A P H 5 *
* M A D E V E N T *
* *
* * * *
* * * * * *
* * * * * 5 * * * * *
* * * * * *
* * * *
* *
* VERSION 5.1.5.1 *
* *
* The MadGraph Development Team - Please visit us at *
* https://server06.fynu.ucl.ac.be/projects/madgraph *
* *
* Type 'help' for in-line help. *
* *
************************************************************
load configuration from /home/2/wkotlarski/sgluons_with_tau/Cards/me5_configuration.txt
load configuration from /home/2/wkotlarski/sgluons_with_tau/Cards/me5_configuration.txt
Using default eps viewer "evince". Set another one in ./input/mg5_configuration.txt
Using default web browser "firefox". Set another one in ./input/mg5_configuration.txt
generate_events -f 7TeV_600GeV --cluster
Will run in mode parton
Generating 500000 events with run name 7TeV_600GeV
survey 7TeV_600GeV
compile directory
Using random number seed offset = 27
Running Survey
Creating Jobs
Working on SubProcesses
    P0_gg_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq
 ...
 ...
 ...
    P0_qq_xxc_x_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam_xc_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam
 Idle: 24 Running: 49 Finish: 155
 ...
 ...
 ...
 Idle: 0 Running: 1 Finish: 227
INFO: All jobs finished
End survey
refine 500000
compile directory
Creating Jobs
Refine results to 500000
    P0_gg_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq
 ...
 ...
 ...
    P0_qq_xxc_x_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam_xc_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam
 Idle: 447 Running: 196 Finish: 19
 ...
 ...
 ...
 Idle: 0 Running: 1 Finish: 661
INFO: All jobs finished
Combining runs
finish refine
refine 500000
compile directory
Creating Jobs
Refine results to 500000
    P0_gg_xxc_x_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq_xc_ttx_t_wpb_wp_qq_tx_wmbx_wm_qq
 ...
 ...
 ...
    P0_qq_xxc_x_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam_xc_ttx_t_wpb_wp_vltap_tx_wmbx_wm_vltam
 Idle: 52 Running: 0 Finish: 0
 ...
 ...
 ...
 Idle: 0 Running: 1 Finish: 51
INFO: All jobs finished
Combining runs
finish refine
combine_events
Combining Events
  === Results Summary for run: 7TeV_600GeV tag: tag_1 ===

     Cross-section : 0.08392 +- 2.729e-05 pb
     Nb of events : 388709

store_events
Storing parton level results
End Parton
quit

Question information

Language:
English Edit question
Status:
Answered
For:
MadGraph5_aMC@NLO Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Olivier Mattelaer (olivier-mattelaer) said :
#1

Hi Wojciech,

In fact, you are not suppose to be allowed to generate more than 100k events by single run of generate_events.
This is a bug that you are allowed to do that.

A correct way to generate such huge number of events is to use the multi_run command:
./bin/madevent
then enter
multi_run 10
and ask in the run_card 100000 events.

At the end of this run one of the output file will be a file with 1M events.

Tell me if this works in this way,

Cheers,

Olivier

Revision history for this message
Johan Alwall (johan-alwall) said :
#2

Hello Wojciech,

The 100k limit was lifted to 1M in v. 1.4.4. However, for very complicated processes (as the one you are running), the number of events that can be generated from each integration channel might pose a more restrictive limit. As Olivier said, the suggested way to generate large number of events in this case is to use the multi_run functionality.

Note that if you want to generate really large numbers of events (say, 10M or more), I would suggest to instead use gridpacks.

All the best,
Johan

Revision history for this message
Wojciech Kotlarski (wojciech-kotlarski) said :
#3

Thank you all for a quick reply. I just have one more, minor, question. I'm not allowed to run interactive jobs on my cluster. I need to detach them from console though nohup linux command. I've checked that 'nohup ./bin/madevents multi_run 10 &'
does almost the job. The problem is that I do not know how to bypass question about editing. I would appreciate some pointers.

Best regards,
Wojciech

Revision history for this message
Johan Alwall (johan-alwall) said :
#4

Hello Wojciech,

Just add a "-f" at the end of the call to bypass all questions.
nohup ./bin/madevents multi_run 10 -f &

All the best,
Johan

Revision history for this message
Olivier Mattelaer (olivier-mattelaer) said :
#5

Hi Wojciech,

You can add -f to the command in order to bypass all questions.
i.e.:
> 'nohup ./bin/madevents multi_run 10 -f &'

Cheers,

Olivier

On Oct 9, 2012, at 5:36 AM, Wojciech Kotlarski <email address hidden> wrote:

> Question #210709 on MadGraph5 changed:
> https://answers.launchpad.net/madgraph5/+question/210709
>
> Wojciech Kotlarski posted a new comment:
> Thank you all for a quick reply. I just have one more, minor, question. I'm not allowed to run interactive jobs on my cluster. I need to detach them from console though nohup linux command. I've checked that 'nohup ./bin/madevents multi_run 10 &'
> does almost the job. The problem is that I do not know how to bypass question about editing. I would appreciate some pointers.
>
> Best regards,
> Wojciech
>
> --
> You received this question notification because you are a member of
> MadTeam, which is an answer contact for MadGraph5.

Can you help with this problem?

Provide an answer of your own, or ask Wojciech Kotlarski for more information if necessary.

To post a message you must log in.