Running Loop Induced on cluster mode gives error

Asked by Jay Sandesara on 2020-11-13

Hi,

I am trying to run gg->4lep+jets loop induced process gridpacks, using cluster mode. This processes run successfully while running on multicore mode. And tree level processes like pp>z also run successfully on cluster mode.

It is only the running of (any) loop induced process on cluster mode that gives me an error like the following:

INFO: P0_gg_zz/G1 is at 0.897 +- 0.0631 pb. Now submitting iteration #2.
INFO: P0_gg_zz/G2.2 is at 0.0172 +- 0.00577 pb. Now submitting iteration #2.
INFO: All jobs finished
INFO: Idle: 0, Running: 0, Completed: 16 [ 2m 2s ]
Error when reading /home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G1/results.dat
Command "generate_events run_02" interrupted with error:
IOError : [Errno 2] No such file or directory: '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G1/results.dat'
Please report this bug on https://bugs.launchpad.net/mg5amcnlo
More information is found in '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/run_02_tag_1_debug.log'.
Please attach this file to your report.
INFO:
quit

In the debug file it says the following:

#************************************************************
#* MadGraph5_aMC@NLO/MadEvent *
#* *
#* * * *
#* * * * * *
#* * * * * 5 * * * * *
#* * * * * *
#* * * *
#* *
#* *
#* VERSION 2.7.3 2020-06-21 *
#* *
#* The MadGraph5_aMC@NLO Development Team - Find us at *
#* https://server06.fynu.ucl.ac.be/projects/madgraph *
#* *
#************************************************************
#* *
#* Command File for MadEvent *
#* *
#* run as ./bin/madevent.py filename *
#* *
#************************************************************
generate_events run_02
Traceback (most recent call last):
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1515, in onecmd
    return self.onecmd_orig(line, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1464, in onecmd_orig
    return func(arg, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/madevent_interface.py", line 2468, in do_generate_events
    self.run_generate_events(switch_mode, args)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/common_run_interface.py", line 7011, in new_fct
    original_fct(obj, *args, **opts)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/madevent_interface.py", line 2496, in run_generate_events
    postcmd=False)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1544, in exec_cmd
    stop = Cmd.onecmd_orig(current_interface, line, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1464, in onecmd_orig
    return func(arg, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/madevent_interface.py", line 3376, in do_survey
    cross, error = self.make_make_all_html_results()
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/common_run_interface.py", line 681, in make_make_all_html_results
    return sum_html.make_all_html_results(self, folder_names, jobs)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 766, in make_all_html_results
    Presults = collect_result(cmd, folder_names=folder_names, jobs=jobs)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 747, in collect_result
    P_comb.add_results(os.path.basename(G), path, mfactors[G])
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 425, in add_results
    oneresult.read_results(filepath)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 279, in read_results
    finput = open(filepath)
IOError: [Errno 2] No such file or directory: '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G1/results.dat'

Related File: /home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G1/results.dat
                              Run Options
                              -----------
               stdout_level : 10 (user set)

                         MadEvent Options
                         ----------------
     automatic_html_opening : False (user set)
        notification_center : True
          cluster_temp_path : None
             cluster_memory : None (user set)
               cluster_size : 100
              cluster_queue : tier3 (user set)
                    nb_core : 80 (user set)
               cluster_time : 80 (user set)
                   run_mode : 1 (user set)

                      Configuration Options
                      ---------------------
                text_editor : None
         cluster_local_path : None
      cluster_status_update : (600, 30)
               pythia8_path : None (user set)
                  hwpp_path : None (user set)
            pythia-pgs_path : None (user set)
                    td_path : None (user set)
               delphes_path : None (user set)
                thepeg_path : None (user set)
               cluster_type : sge (user set)
          madanalysis5_path : None (user set)
           cluster_nb_retry : 3 (user set)
                 eps_viewer : None
                web_browser : None
               syscalc_path : None (user set)
           madanalysis_path : None (user set)
                     lhapdf : lhapdf-config
              f2py_compiler : None
                 hepmc_path : None (user set)
         cluster_retry_wait : 300 (user set)
           fortran_compiler : None
                auto_update : 7 (user set)
        exrootanalysis_path : None (user set)
                    timeout : 60
               cpp_compiler : None

Do you have any suggestions please?

Thanks,
Jay Sandesara

Question information

Language:
English Edit question
Status:
Answered
For:
MadGraph5_aMC@NLO Edit question
Assignee:
No assignee Edit question
Last query:
2020-11-16
Last reply:
2020-11-18

This question was reopened

Hi,

Did you set pt cut on the Z for that process? (otherwise you have an integrable singularity that makes impossible for the code to generate events).

This might not be your issue actualy.
Do you have a log file in /home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G1/
This should provide a hint of the problem.

Cheers,

Olivier

Jay Sandesara (jaysandesara) said : #2

Hey Olivier,

This same process runs fine on multicore mode, so it might not be a process fault.

I figured out after some trial and error that this issue only happened when re-'launch'ing the process from the same output directory twice, using cluster mode. Also note that these issues only happen for loop induced processes.

Making a fresh directory and launching that I get the following error now:

INFO: End survey
refine 10000
Creating Jobs
INFO: Refine results to 10000
INFO: Generating 10000.0 unweighted events.
Error when reading /home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G2.2/results.dat
Command "generate_events run_02" interrupted with error:
IOError : [Errno 2] No such file or directory: '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_z
z/G2.2/results.dat'
Please report this bug on https://bugs.launchpad.net/mg5amcnlo
More information is found in '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/run_02_tag_1_debug.log'.
Please attach this file to your report.
INFO:

And insude the G2.2 directory I only have a 'ftn25' file with the following text:

-0.0000071472770509 +0.0000591141206259 +0.0001118915191760 +0.0001586070965117
+0.0001994365145220 +0.0002421029975097 +0.0002926953314754 +0.0003523487460095
+0.0004236863352670 +0.0005052667349680 +0.0005992735090596 +0.0007104595265422
+0.0008391451560580 +0.0009829571308225 +0.0011476448366913 +0.0013381770143287
+0.0015554971619234 +0.0018044210464499 +0.0020817286656952 +0.0023957776249037
+0.0027574988763947 +0.0031563452966326 +0.0036079496484904 +0.0041142041451001
+0.0046682646098808 +0.0052921375532616 +0.0060071903786130 +0.0068121548619855
+0.0076895892725127 +0.0086403304029830 +0.0096706209410779 +0.0108243592167447
+0.0120823342676093 +0.0134291494846810 +0.0149085493099759 +0.0165694498392576
+0.0184115066212627 +0.0203742264510825 +0.0225402749439121 +0.0249928926104955
+0.0277203761138055 +0.0306143167750027 +0.0337140108676193 +0.0369555944944718
+0.0405272438142572 +0.0444133902408899 +0.0486875408901671 +0.0533060180057008
+0.0581305585991732 +0.0631589516688083 +0.0685135825633891 +0.0743436504844123
+0.0807263421637745 +0.0876346869841360 +0.0950173156952642 +0.1028330515710487
+0.1112490051114348 +0.1206252992725145 +0.1308070831726230 +0.1418407087175444
+0.1537754979023556 +0.1665051406441406 +0.1807489451192759 +0.1943091263901711
+0.2077724006608040 +0.2220985407950473 +0.2370793778325402 +0.2530348727361619
+0.2720555870179162 +0.2927611225930828 +0.3147813215513454 +0.3378464021072047
+0.3602675576122047 +0.3836429879873798 +0.4078555574671012 +0.4341521917186041
+0.4611883142773756 +0.4894891060584420 +0.5189450495193382 +0.5490129627486343
+0.5803121668810837 +0.6122113239806998 +0.6491135895599781 +0.6862803816361756
+0.7238997791747536 +0.7559450292020927 +0.7825129203857084 +0.8090808115693242
+0.8344001980933838 +0.8588481007022630 +0.8832960033111423 +0.9076218920051395
+0.9307164190038546 +0.9538109460025697 +0.9769054730012846 +1.0000000000000000
-0.0039725939432780 +0.0119417926133474 +0.0237292401204733 +0.0356403632462237
+0.0473623069674041 +0.0588905230044875 +0.0700342563562345 +0.0810027368804026
+0.0918823108434449 +0.1027378968350937 +0.1140353863779854 +0.1253712799878753
+0.1359236613654324 +0.1465024674160443 +0.1574887046290114 +0.1685117859149395
+0.1797218705439236 +0.1911918978104313 +0.2032510549996648 +0.2149640841176182
+0.2264249776841656 +0.2370589341510384 +0.2474328664019927 +0.2584717831546269
+0.2697293688050137 +0.2811572393311516 +0.2924559392744622 +0.3020981236641557
+0.3117403080538493 +0.3210750349313281 +0.3303834659242567 +0.3394446225155072
+0.3483910634655615 +0.3582237445552673 +0.3696706903260391 +0.3811429654216065
+0.3926373058727348 +0.4046525031790102 +0.4168527672822626 +0.4281272929992317
+0.4391809892681257 +0.4491463748077593 +0.4590748973618478 +0.4685683647171625
+0.4780618320724773 +0.4881184416524678 +0.4982492205314357 +0.5096355574775073
+0.5213415325981574 +0.5344711436771982 +0.5471416700272191 +0.5592555220117921
+0.5714414856805808 +0.5836358326015134 +0.5951624557867912 +0.6063060443241235
+0.6160823208098086 +0.6258141109890851 +0.6350838514845447 +0.6443535919800043
+0.6506643828060656 +0.6564131093612149 +0.6621618359163641 +0.6679953052626241
+0.6741356748570999 +0.6802760444515757 +0.6864164140460516 +0.6926629980322774
+0.6989323420239115 +0.7052016860155458 +0.7151543458921399 +0.7287832203652066
+0.7405970623289418 +0.7521975090460992 +0.7631566311816961 +0.7739838850317531
+0.7845027342424632 +0.7949983318956613 +0.8054442792602312 +0.8161186044989567
+0.8272682284954096 +0.8384951730048471 +0.8498143498380801 +0.8615446212024167
+0.8735317103648242 +0.8857617771026639 +0.8979968622081833 +0.9100991950156860
+0.9217427274717859 +0.9328418635616375 +0.9440048790921552 +0.9552140922886146
+0.9659993514583990 +0.9766211471728991 +0.9881652283997379 +1.0000000000000000
-0.0035811106363934 +0.0138043660495369 +0.0252523412960639 +0.0346894189778994
+0.0440425791316800 +0.0531577092669964 +0.0622728394023128 +0.0710146088867995
+0.0797468359411242 +0.0898117981710606 +0.1008056570873169 +0.1110456134600694
+0.1209536002682767 +0.1309387369095973 +0.1409771320046334 +0.1508827658614517
+0.1606639899932570 +0.1705984994018341 +0.1807765208915445 +0.1909050130260200
+0.2009371075632379 +0.2103250510243110 +0.2179055119397269 +0.2254859728551429
+0.2331093499775703 +0.2407732324464335 +0.2484371149152967 +0.2559537399349762
+0.2634326421928761 +0.2709274588204527 +0.2799281653844390 +0.2889288719484253
+0.2978186564674041 +0.3066599520731685 +0.3142415916527311 +0.3193721001190639
+0.3245026085853968 +0.3296331170517297 +0.3348096854077880 +0.3401054123215944
+0.3454011392354009 +0.3506968661492073 +0.3559868729631095 +0.3612660098943882
+0.3665451468256669 +0.3718242837569455 +0.3791289870622989 +0.3894918632151223
+0.3997818653723332 +0.4099569504429346 +0.4203113023161142 +0.4310127541451103
+0.4414356238396096 +0.4514296509933397 +0.4615194937894155 +0.4718233841425849
+0.4821990841059239 +0.4927528955537488 +0.5036993360333371 +0.5155062771863944
+0.5273435595256397 +0.5392057850856730 +0.5504344064388973 +0.5614971765132351
+0.5715742357284448 +0.5815530345809291 +0.5915774504650015 +0.6016117720566343
+0.6117564240978495 +0.6219387638752580 +0.6328560998215986 +0.6440893975813919
+0.6551352030886139 +0.6661465516489725 +0.6776379233847128 +0.6892424704271785
+0.7013802293744696 +0.7138600778414878 +0.7267987001322626 +0.7403065827164453
+0.7539420374957836 +0.7675776161337615 +0.7803597213138990 +0.7928750625828173
+0.8053991509321722 +0.8181372071199950 +0.8311554193635365 +0.8462341388937834
+0.8625221410929512 +0.8796482712797646 +0.8967976781932433 +0.9141764470692846
+0.9321951415721957 +0.9516664900638229 +0.9745712273542094 +1.0000000000000000
-0.0041558782259623 +0.0421210829654369 +0.0728036210201914 +0.0964663267121033
+0.1157922761454353 +0.1322328314546646 +0.1467179025401935 +0.1599616251150229
+0.1728319582954650 +0.1853194021140570 +0.1961294826203589 +0.2065846868709528
+0.2167843315205797 +0.2269326489519155 +0.2367954254405049 +0.2465776005433462
+0.2560711910291892 +0.2654094848675780 +0.2744377818386531 +0.2830365477081007
+0.2916353135775482 +0.2999912965872285 +0.3083463911148390 +0.3169000960621480
+0.3256501482242638 +0.3344084598533220 +0.3432262531331069 +0.3520440464128918
+0.3608517068371513 +0.3696561547672327 +0.3780321580857713 +0.3857465593958867
+0.3934609607060021 +0.4009208616562386 +0.4082677400595598 +0.4156146184628810
+0.4227843025072482 +0.4299243722489020 +0.4370644419905558 +0.4445563462280184
+0.4520711073622649 +0.4596506660793656 +0.4675541894388505 +0.4754577127983355
+0.4830600853147248 +0.4903961719615377 +0.4977322586083506 +0.5053600847047157
+0.5131184444549696 +0.5208757467489812 +0.5284453795289407 +0.5360150123089003
+0.5436996394377973 +0.5517231172574900 +0.5597465950771828 +0.5676937548207904
+0.5756010413841776 +0.5834952553718934 +0.5908118460306803 +0.5981284366894670
+0.6054653844367777 +0.6128984871925636 +0.6203315899483496 +0.6268274938007249
+0.6317408605506823 +0.6366542273006398 +0.6415675940505973 +0.6465203018872926
+0.6517321383552098 +0.6569439748231271 +0.6621558112910444 +0.6673837357680512
+0.6727151875485510 +0.6780466393290507 +0.6833780911095506 +0.6897851625278961
+0.6998577560078811 +0.7100324002996997 +0.7207486418348308 +0.7316109453455231
+0.7430082520765372 +0.7545608154742058 +0.7663597756456106 +0.7785937047703920
+0.7910932655392546 +0.8041629470929285 +0.8173578542208706 +0.8307240803681770
+0.8437087512894895 +0.8567396678851551 +0.8703668899596482 +0.8850413145182368
+0.9020076443296740 +0.9244792305687372 +0.9535514659715765 +1.0000000000000000
0.000155429914251 -1
  <DiscreteSampler_grid>
  Helicity
  1 # grid_type. 1=='ref', 2=='run'
  80 # Attribute 'min_bin_probing_points' of the grid.
  1 # Attribute 'grid_mode' of the grid. 1=='default',2=='initialization'
  0.030 # Attribute 'small_contrib_threshold' of the grid.
  0.333 # Attribute 'damping_power' of the grid.
# binID n_entries weight weight_sqr abs_weight
    1 1663 0.000396769505995 4.90690805858e-06 0.000396769505995
    36 1629 0.000344848080473 1.37027878946e-06 0.000344848080473
    12 261 4.43870559852e-05 7.65569155348e-09 4.43870559852e-05
    25 305 4.29785651091e-05 7.00240682762e-09 4.29785651091e-05
    16 80 2.39110016934e-05 4.03579270113e-08 2.39110016934e-05
    32 80 1.93159231401e-05 8.67081759439e-09 1.93159231401e-05
    5 80 1.10799499173e-05 1.29566108618e-09 1.10799499173e-05
    9 80 8.13914649118e-06 9.24038132287e-10 8.13914649118e-06
    28 80 7.72978581741e-06 7.97795515297e-10 7.72978581741e-06
    18 80 3.38390100641e-06 1.4492514864e-10 3.38390100641e-06
    19 80 1.92760543745e-06 2.83888030175e-11 1.92760543745e-06
    10 80 1.71712113548e-06 1.8230553568e-11 1.71712113548e-06
    27 80 1.70945637924e-06 1.8745065086e-11 1.70945637924e-06
    21 80 1.66814709166e-06 2.05936905057e-10 1.66814709166e-06
    33 80 8.13670754941e-07 6.14562961124e-12 8.13670754941e-07
    4 80 7.90962685235e-07 6.15517035847e-12 7.90962685235e-07
    22 80 7.83304632481e-07 5.36566495771e-12 7.83304632481e-07
    35 80 7.82836236393e-07 6.07847870116e-12 7.82836236393e-07
    2 80 7.81075962936e-07 6.07823072943e-12 7.81075962936e-07
    26 80 7.41268070756e-07 5.20224352984e-12 7.41268070756e-07
    11 80 7.32056131859e-07 5.13721571732e-12 7.32056131859e-07
    15 80 7.2318849244e-07 5.07219562019e-12 7.2318849244e-07
    7 80 7.16831031789e-07 6.91526497289e-12 7.16831031789e-07
    30 80 6.85248209061e-07 6.56823475446e-12 6.85248209061e-07
    3 80 6.42018838378e-07 6.44848911646e-12 6.42018838378e-07
    34 80 6.33895320192e-07 6.36686267277e-12 6.33895320192e-07
    23 80 6.30516089169e-07 1.49150740617e-11 6.30516089169e-07
    14 80 4.84749345721e-07 1.32152920187e-11 4.84749345721e-07
    24 80 2.09337832893e-07 9.93331926996e-13 2.09337832893e-07
    20 80 2.09337832893e-07 9.93331926988e-13 2.09337832893e-07
    13 80 2.09337832825e-07 9.93331928833e-13 2.09337832825e-07
    17 80 2.09337832824e-07 9.9333192883e-13 2.09337832824e-07
    31 80 5.25152115616e-08 1.6669773096e-13 5.25152115616e-08
    6 80 5.25152115607e-08 1.6669773095e-13 5.25152115607e-08
    29 80 5.25152115604e-08 1.66697730945e-13 5.25152115604e-08
    8 80 5.18666056052e-08 1.64614014233e-13 5.18666056052e-08
  </DiscreteSampler_grid>
  <DiscreteSampler_grid>
  grouped_processes
  1 # grid_type. 1=='ref', 2=='run'
  10 # Attribute 'min_bin_probing_points' of the grid.
  1 # Attribute 'grid_mode' of the grid. 1=='default',2=='initialization'
  0.030 # Attribute 'small_contrib_threshold' of the grid.
  0.333 # Attribute 'damping_power' of the grid.
# binID n_entries weight weight_sqr abs_weight
    10 10 290574.041619 1.16594623795e+11 290574.041619
  </DiscreteSampler_grid>

Jay Sandesara (jaysandesara) said : #3

Oh and the dubug log says the following:

#************************************************************
#* MadGraph5_aMC@NLO/MadEvent *
#* *
#* * * *
#* * * * * *
#* * * * * 5 * * * * *
#* * * * * *
#* * * *
#* *
#* *
#* VERSION 2.7.3 2020-06-21 *
#* *
#* The MadGraph5_aMC@NLO Development Team - Find us at *
#* https://server06.fynu.ucl.ac.be/projects/madgraph *
#* *
#************************************************************
#* *
#* Command File for MadEvent *
#* *
#* run as ./bin/madevent.py filename *
#* *
#************************************************************
generate_events run_01
Traceback (most recent call last):
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1515, in onecmd
    return self.onecmd_orig(line, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1464, in onecmd_orig
    return func(arg, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/madevent_interface.py", line 2468, in do_generate_events
    self.run_generate_events(switch_mode, args)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/common_run_interface.py", line 7011, in new_fct
    original_fct(obj, *args, **opts)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/madevent_interface.py", line 2496, in run_generate_events
    postcmd=False)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1544, in exec_cmd
    stop = Cmd.onecmd_orig(current_interface, line, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/extended_cmd.py", line 1464, in onecmd_orig
    return func(arg, **opt)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/madevent_interface.py", line 3376, in do_survey
    cross, error = self.make_make_all_html_results()
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/interface/common_run_interface.py", line 681, in make_make_all_html_results
    return sum_html.make_all_html_results(self, folder_names, jobs)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 766, in make_all_html_results
    Presults = collect_result(cmd, folder_names=folder_names, jobs=jobs)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 747, in collect_result
    P_comb.add_results(os.path.basename(G), path, mfactors[G])
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 425, in add_results
    oneresult.read_results(filepath)
  File "/home/net3/jsandesara/MG5_aMC_v2_7_3/madgraph/madevent/sum_html.py", line 279, in read_results
    finput = open(filepath)
IOError: [Errno 2] No such file or directory: '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_2/SubProcesses/P0_gg_zz/G1.2/results.dat'

Related File: /home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_2/SubProcesses/P0_gg_zz/G1.2/results.dat
                              Run Options
                              -----------
               stdout_level : 20 (user set)

                         MadEvent Options
                         ----------------
     automatic_html_opening : False (user set)
        notification_center : True
          cluster_temp_path : None
             cluster_memory : None (user set)
               cluster_size : 100
              cluster_queue : tier3 (user set)
                    nb_core : 20 (user set)
               cluster_time : 20 (user set)
                   run_mode : 1 (user set)

                      Configuration Options
                      ---------------------
                text_editor : None
         cluster_local_path : None
      cluster_status_update : (600, 30)
               pythia8_path : None (user set)
                  hwpp_path : None (user set)
            pythia-pgs_path : None (user set)
                    td_path : None (user set)
               delphes_path : None (user set)
                thepeg_path : None (user set)
               cluster_type : sge (user set)
          madanalysis5_path : None (user set)
           cluster_nb_retry : 3 (user set)
                 eps_viewer : None
                web_browser : None
               syscalc_path : None (user set)
           madanalysis_path : None (user set)
                     lhapdf : lhapdf-config
              f2py_compiler : None
                 hepmc_path : None (user set)
         cluster_retry_wait : 300 (user set)
           fortran_compiler : None
                auto_update : 7 (user set)
        exrootanalysis_path : None (user set)
                    timeout : 60
               cpp_compiler : None

I face some python3 issue related to slurm cluster for loop induced process (which should not be your issue since it seems that you still run python2).

Now the code is running with python3 and I will see if I can reproduce the issue on our cluster.

Cheers,

Olivier

> On 16 Nov 2020, at 06:35, Jay Sandesara <email address hidden> wrote:
>
> Question #694001 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/694001
>
> Status: Answered => Open
>
> Jay Sandesara is still having a problem:
> Hey Olivier,
>
> This same process runs fine on multicore mode, so it might not be a
> process fault.
>
> I figured out after some trial and error that this issue only happened
> when re-'launch'ing the process from the same output directory twice,
> using cluster mode. Also note that these issues only happen for loop
> induced processes.
>
> Making a fresh directory and launching that I get the following error
> now:
>
> INFO: End survey
> refine 10000
> Creating Jobs
> INFO: Refine results to 10000
> INFO: Generating 10000.0 unweighted events.
> Error when reading /home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_zz/G2.2/results.dat
> Command "generate_events run_02" interrupted with error:
> IOError : [Errno 2] No such file or directory: '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/SubProcesses/P0_gg_z
> z/G2.2/results.dat'
> Please report this bug on https://bugs.launchpad.net/mg5amcnlo
> More information is found in '/home/net3/jsandesara/MG5_aMC_v2_7_3/PROC_loop_sm_0/run_02_tag_1_debug.log'.
> Please attach this file to your report.
> INFO:
>
>
> And insude the G2.2 directory I only have a 'ftn25' file with the following text:
>
> -0.0000071472770509 +0.0000591141206259 +0.0001118915191760 +0.0001586070965117
> +0.0001994365145220 +0.0002421029975097 +0.0002926953314754 +0.0003523487460095
> +0.0004236863352670 +0.0005052667349680 +0.0005992735090596 +0.0007104595265422
> +0.0008391451560580 +0.0009829571308225 +0.0011476448366913 +0.0013381770143287
> +0.0015554971619234 +0.0018044210464499 +0.0020817286656952 +0.0023957776249037
> +0.0027574988763947 +0.0031563452966326 +0.0036079496484904 +0.0041142041451001
> +0.0046682646098808 +0.0052921375532616 +0.0060071903786130 +0.0068121548619855
> +0.0076895892725127 +0.0086403304029830 +0.0096706209410779 +0.0108243592167447
> +0.0120823342676093 +0.0134291494846810 +0.0149085493099759 +0.0165694498392576
> +0.0184115066212627 +0.0203742264510825 +0.0225402749439121 +0.0249928926104955
> +0.0277203761138055 +0.0306143167750027 +0.0337140108676193 +0.0369555944944718
> +0.0405272438142572 +0.0444133902408899 +0.0486875408901671 +0.0533060180057008
> +0.0581305585991732 +0.0631589516688083 +0.0685135825633891 +0.0743436504844123
> +0.0807263421637745 +0.0876346869841360 +0.0950173156952642 +0.1028330515710487
> +0.1112490051114348 +0.1206252992725145 +0.1308070831726230 +0.1418407087175444
> +0.1537754979023556 +0.1665051406441406 +0.1807489451192759 +0.1943091263901711
> +0.2077724006608040 +0.2220985407950473 +0.2370793778325402 +0.2530348727361619
> +0.2720555870179162 +0.2927611225930828 +0.3147813215513454 +0.3378464021072047
> +0.3602675576122047 +0.3836429879873798 +0.4078555574671012 +0.4341521917186041
> +0.4611883142773756 +0.4894891060584420 +0.5189450495193382 +0.5490129627486343
> +0.5803121668810837 +0.6122113239806998 +0.6491135895599781 +0.6862803816361756
> +0.7238997791747536 +0.7559450292020927 +0.7825129203857084 +0.8090808115693242
> +0.8344001980933838 +0.8588481007022630 +0.8832960033111423 +0.9076218920051395
> +0.9307164190038546 +0.9538109460025697 +0.9769054730012846 +1.0000000000000000
> -0.0039725939432780 +0.0119417926133474 +0.0237292401204733 +0.0356403632462237
> +0.0473623069674041 +0.0588905230044875 +0.0700342563562345 +0.0810027368804026
> +0.0918823108434449 +0.1027378968350937 +0.1140353863779854 +0.1253712799878753
> +0.1359236613654324 +0.1465024674160443 +0.1574887046290114 +0.1685117859149395
> +0.1797218705439236 +0.1911918978104313 +0.2032510549996648 +0.2149640841176182
> +0.2264249776841656 +0.2370589341510384 +0.2474328664019927 +0.2584717831546269
> +0.2697293688050137 +0.2811572393311516 +0.2924559392744622 +0.3020981236641557
> +0.3117403080538493 +0.3210750349313281 +0.3303834659242567 +0.3394446225155072
> +0.3483910634655615 +0.3582237445552673 +0.3696706903260391 +0.3811429654216065
> +0.3926373058727348 +0.4046525031790102 +0.4168527672822626 +0.4281272929992317
> +0.4391809892681257 +0.4491463748077593 +0.4590748973618478 +0.4685683647171625
> +0.4780618320724773 +0.4881184416524678 +0.4982492205314357 +0.5096355574775073
> +0.5213415325981574 +0.5344711436771982 +0.5471416700272191 +0.5592555220117921
> +0.5714414856805808 +0.5836358326015134 +0.5951624557867912 +0.6063060443241235
> +0.6160823208098086 +0.6258141109890851 +0.6350838514845447 +0.6443535919800043
> +0.6506643828060656 +0.6564131093612149 +0.6621618359163641 +0.6679953052626241
> +0.6741356748570999 +0.6802760444515757 +0.6864164140460516 +0.6926629980322774
> +0.6989323420239115 +0.7052016860155458 +0.7151543458921399 +0.7287832203652066
> +0.7405970623289418 +0.7521975090460992 +0.7631566311816961 +0.7739838850317531
> +0.7845027342424632 +0.7949983318956613 +0.8054442792602312 +0.8161186044989567
> +0.8272682284954096 +0.8384951730048471 +0.8498143498380801 +0.8615446212024167
> +0.8735317103648242 +0.8857617771026639 +0.8979968622081833 +0.9100991950156860
> +0.9217427274717859 +0.9328418635616375 +0.9440048790921552 +0.9552140922886146
> +0.9659993514583990 +0.9766211471728991 +0.9881652283997379 +1.0000000000000000
> -0.0035811106363934 +0.0138043660495369 +0.0252523412960639 +0.0346894189778994
> +0.0440425791316800 +0.0531577092669964 +0.0622728394023128 +0.0710146088867995
> +0.0797468359411242 +0.0898117981710606 +0.1008056570873169 +0.1110456134600694
> +0.1209536002682767 +0.1309387369095973 +0.1409771320046334 +0.1508827658614517
> +0.1606639899932570 +0.1705984994018341 +0.1807765208915445 +0.1909050130260200
> +0.2009371075632379 +0.2103250510243110 +0.2179055119397269 +0.2254859728551429
> +0.2331093499775703 +0.2407732324464335 +0.2484371149152967 +0.2559537399349762
> +0.2634326421928761 +0.2709274588204527 +0.2799281653844390 +0.2889288719484253
> +0.2978186564674041 +0.3066599520731685 +0.3142415916527311 +0.3193721001190639
> +0.3245026085853968 +0.3296331170517297 +0.3348096854077880 +0.3401054123215944
> +0.3454011392354009 +0.3506968661492073 +0.3559868729631095 +0.3612660098943882
> +0.3665451468256669 +0.3718242837569455 +0.3791289870622989 +0.3894918632151223
> +0.3997818653723332 +0.4099569504429346 +0.4203113023161142 +0.4310127541451103
> +0.4414356238396096 +0.4514296509933397 +0.4615194937894155 +0.4718233841425849
> +0.4821990841059239 +0.4927528955537488 +0.5036993360333371 +0.5155062771863944
> +0.5273435595256397 +0.5392057850856730 +0.5504344064388973 +0.5614971765132351
> +0.5715742357284448 +0.5815530345809291 +0.5915774504650015 +0.6016117720566343
> +0.6117564240978495 +0.6219387638752580 +0.6328560998215986 +0.6440893975813919
> +0.6551352030886139 +0.6661465516489725 +0.6776379233847128 +0.6892424704271785
> +0.7013802293744696 +0.7138600778414878 +0.7267987001322626 +0.7403065827164453
> +0.7539420374957836 +0.7675776161337615 +0.7803597213138990 +0.7928750625828173
> +0.8053991509321722 +0.8181372071199950 +0.8311554193635365 +0.8462341388937834
> +0.8625221410929512 +0.8796482712797646 +0.8967976781932433 +0.9141764470692846
> +0.9321951415721957 +0.9516664900638229 +0.9745712273542094 +1.0000000000000000
> -0.0041558782259623 +0.0421210829654369 +0.0728036210201914 +0.0964663267121033
> +0.1157922761454353 +0.1322328314546646 +0.1467179025401935 +0.1599616251150229
> +0.1728319582954650 +0.1853194021140570 +0.1961294826203589 +0.2065846868709528
> +0.2167843315205797 +0.2269326489519155 +0.2367954254405049 +0.2465776005433462
> +0.2560711910291892 +0.2654094848675780 +0.2744377818386531 +0.2830365477081007
> +0.2916353135775482 +0.2999912965872285 +0.3083463911148390 +0.3169000960621480
> +0.3256501482242638 +0.3344084598533220 +0.3432262531331069 +0.3520440464128918
> +0.3608517068371513 +0.3696561547672327 +0.3780321580857713 +0.3857465593958867
> +0.3934609607060021 +0.4009208616562386 +0.4082677400595598 +0.4156146184628810
> +0.4227843025072482 +0.4299243722489020 +0.4370644419905558 +0.4445563462280184
> +0.4520711073622649 +0.4596506660793656 +0.4675541894388505 +0.4754577127983355
> +0.4830600853147248 +0.4903961719615377 +0.4977322586083506 +0.5053600847047157
> +0.5131184444549696 +0.5208757467489812 +0.5284453795289407 +0.5360150123089003
> +0.5436996394377973 +0.5517231172574900 +0.5597465950771828 +0.5676937548207904
> +0.5756010413841776 +0.5834952553718934 +0.5908118460306803 +0.5981284366894670
> +0.6054653844367777 +0.6128984871925636 +0.6203315899483496 +0.6268274938007249
> +0.6317408605506823 +0.6366542273006398 +0.6415675940505973 +0.6465203018872926
> +0.6517321383552098 +0.6569439748231271 +0.6621558112910444 +0.6673837357680512
> +0.6727151875485510 +0.6780466393290507 +0.6833780911095506 +0.6897851625278961
> +0.6998577560078811 +0.7100324002996997 +0.7207486418348308 +0.7316109453455231
> +0.7430082520765372 +0.7545608154742058 +0.7663597756456106 +0.7785937047703920
> +0.7910932655392546 +0.8041629470929285 +0.8173578542208706 +0.8307240803681770
> +0.8437087512894895 +0.8567396678851551 +0.8703668899596482 +0.8850413145182368
> +0.9020076443296740 +0.9244792305687372 +0.9535514659715765 +1.0000000000000000
> 0.000155429914251 -1
> <DiscreteSampler_grid>
> Helicity
> 1 # grid_type. 1=='ref', 2=='run'
> 80 # Attribute 'min_bin_probing_points' of the grid.
> 1 # Attribute 'grid_mode' of the grid. 1=='default',2=='initialization'
> 0.030 # Attribute 'small_contrib_threshold' of the grid.
> 0.333 # Attribute 'damping_power' of the grid.
> # binID n_entries weight weight_sqr abs_weight
> 1 1663 0.000396769505995 4.90690805858e-06 0.000396769505995
> 36 1629 0.000344848080473 1.37027878946e-06 0.000344848080473
> 12 261 4.43870559852e-05 7.65569155348e-09 4.43870559852e-05
> 25 305 4.29785651091e-05 7.00240682762e-09 4.29785651091e-05
> 16 80 2.39110016934e-05 4.03579270113e-08 2.39110016934e-05
> 32 80 1.93159231401e-05 8.67081759439e-09 1.93159231401e-05
> 5 80 1.10799499173e-05 1.29566108618e-09 1.10799499173e-05
> 9 80 8.13914649118e-06 9.24038132287e-10 8.13914649118e-06
> 28 80 7.72978581741e-06 7.97795515297e-10 7.72978581741e-06
> 18 80 3.38390100641e-06 1.4492514864e-10 3.38390100641e-06
> 19 80 1.92760543745e-06 2.83888030175e-11 1.92760543745e-06
> 10 80 1.71712113548e-06 1.8230553568e-11 1.71712113548e-06
> 27 80 1.70945637924e-06 1.8745065086e-11 1.70945637924e-06
> 21 80 1.66814709166e-06 2.05936905057e-10 1.66814709166e-06
> 33 80 8.13670754941e-07 6.14562961124e-12 8.13670754941e-07
> 4 80 7.90962685235e-07 6.15517035847e-12 7.90962685235e-07
> 22 80 7.83304632481e-07 5.36566495771e-12 7.83304632481e-07
> 35 80 7.82836236393e-07 6.07847870116e-12 7.82836236393e-07
> 2 80 7.81075962936e-07 6.07823072943e-12 7.81075962936e-07
> 26 80 7.41268070756e-07 5.20224352984e-12 7.41268070756e-07
> 11 80 7.32056131859e-07 5.13721571732e-12 7.32056131859e-07
> 15 80 7.2318849244e-07 5.07219562019e-12 7.2318849244e-07
> 7 80 7.16831031789e-07 6.91526497289e-12 7.16831031789e-07
> 30 80 6.85248209061e-07 6.56823475446e-12 6.85248209061e-07
> 3 80 6.42018838378e-07 6.44848911646e-12 6.42018838378e-07
> 34 80 6.33895320192e-07 6.36686267277e-12 6.33895320192e-07
> 23 80 6.30516089169e-07 1.49150740617e-11 6.30516089169e-07
> 14 80 4.84749345721e-07 1.32152920187e-11 4.84749345721e-07
> 24 80 2.09337832893e-07 9.93331926996e-13 2.09337832893e-07
> 20 80 2.09337832893e-07 9.93331926988e-13 2.09337832893e-07
> 13 80 2.09337832825e-07 9.93331928833e-13 2.09337832825e-07
> 17 80 2.09337832824e-07 9.9333192883e-13 2.09337832824e-07
> 31 80 5.25152115616e-08 1.6669773096e-13 5.25152115616e-08
> 6 80 5.25152115607e-08 1.6669773095e-13 5.25152115607e-08
> 29 80 5.25152115604e-08 1.66697730945e-13 5.25152115604e-08
> 8 80 5.18666056052e-08 1.64614014233e-13 5.18666056052e-08
> </DiscreteSampler_grid>
> <DiscreteSampler_grid>
> grouped_processes
> 1 # grid_type. 1=='ref', 2=='run'
> 10 # Attribute 'min_bin_probing_points' of the grid.
> 1 # Attribute 'grid_mode' of the grid. 1=='default',2=='initialization'
> 0.030 # Attribute 'small_contrib_threshold' of the grid.
> 0.333 # Attribute 'damping_power' of the grid.
> # binID n_entries weight weight_sqr abs_weight
> 10 10 290574.041619 1.16594623795e+11 290574.041619
> </DiscreteSampler_grid>
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Jay Sandesara (jaysandesara) said : #5

Hey Olivier,

It seems your initial assumption might be right and this might be a singularity issue. To further investigate, I downloaded MG v2.5.5 and ran the same process on cluster mode. There, with all the default cuts of v2.5.5, sometimes it ran successfully and got the gridpack generated and sometimes it didnt. Interestingly though, this only happens in cluster mode and not multicore mode. With multicore, it always runs successfully.

Can you help with this problem?

Provide an answer of your own, or ask Jay Sandesara for more information if necessary.

To post a message you must log in.