Loop-induced processes on lxplus

Hi again,

Some additional information : if I setup the ATLAS framework (athena) *then* setup gcc 4.9.3, I can generate this process
athena brings a lot of additional packages, I'm not really sure which one is a game-changer here

Kind regards
Cyril

I've actually just found this thread, that seems to be on the same issue : https://answers.launchpad.net/mg5amcnlo/+question/265322

and where a solution is proposed

Sorry for the noise

Regards,
Cyril

Dear Cyril,
in the case the solution mentioned in the other thread does not work, and for your information.
I'm running standalone on a SLC6 desktop (must have the same OS as most lxplus machines), and am well
past the initialisation phase (it's generating events). I use gcc 4.7.0, and I don't do anything special.
In the shell where I launch the run, prior to running I execute:
 setenv PATH /afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc6-gcc47-opt/bin:${PATH}
 setenv LD_LIBRARY_PATH /afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc6-gcc47-opt/lib64:${LD_LIBRARY_PATH}
Furthermore, in mg5_configuration.txt I have
 fortran_compiler = /afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc6-gcc47-opt/bin/gfortran
The rest of my installation is standard CERN (in particular, python 2.6.6).
Regards, Stefano.

Hi Stefano,

If I follow your instructions on lxplus with a fresh installation of MG5_aMC_v2.3.3 then indeed it seems to compile.
However, if in the configuration card (MG5_aMC_v2_3_3/input/mg5_configuration.txt) I set:
output_dependencies = internal
then the compilation fails.

Unfortunately, we need this setting for our ATLAS installations as we work from a read-only filesystem.

If you are able to reproduce this, do think it can be fixed for the internal dependancy mode?

Best,

Josh.

Hi Josh,

I can indeed reproduce your issue with output_dependencies = internal.

The issue seems to have been that when CutTools is compiled from within a process output (as it is the case with the option choice above), then it inherits the compilation flags from make_opts in Source (contrary to when it is compiled directly from within vendor/CutTools).

This makes a difference for this option '-fbounds-check' which you might have in make_opts. It turns out that the scalar integral library 'avh_olo' (whose code is part of CutTools) issues a message when initialized which is longer than the statically allocated string that stores it. This is harmless when '-fbounds-check' is off, but it crashes the code when it is on.

Given that i am not sure if a similar issue could happen somewhere else, I added the following lines in
    <MG_Root>/CutTools/vendor/makefile
around line 20 or so, in the MG5aMC development version,

"""
# make sure no bound checks is used as it crashes avholo because of string message too long
tmp := $(FFLAGS)
FFLAGS = $(tmp:-fbounds-check=)
""""

This will make sure that '-fbounds-check' will be removed when compiling CutTools from within <proc_output>/Source/CutTools.
It will hopefully solve your problem as well.

Now regarding the need for 'output_dependencies = internal'. Why isn't it possible to make sure that the central installation of MG5_aMC has CutTools and IREGI compiled already in its vendor directory? (typically adding their compilation whenever a new distribution of MG5_aMC is installed there).
Is it because the user is not guaranteed to use the same compiler suite / architecture as the one active at the time of the central installation of MG5_aMC? I would have thought this was uniform within one ATLAS framework.

Recently, I have been implementing an interface to a new loop computation algorithm for MadLoop, called Ninja, which should become the default. Contrary to IREGI and CutTools, this time I made sure that its access is controlled via a path variable in the MG configuration file, so hopefully it will not cause similar issues on read-only setups.

Cheers

Hi Valentin,
Thanks a lot for looking into this!

Unfortunately, if make the change to CutTools/vendor/makefile you suggest I still see the same error:

Generating 10000 events with run name run_01
survey run_01
INFO: compile directory
initMadLoop -r -f

Initializing MadLoop loop-induced matrix elements (this can take some time)...
root: Error while executing make in /afs/cern.ch/work/m/mcfayden/mcgen/MadGraph5_aMCatNLO/testing/testForCyril/PROC_loop_sm_0/SubProcesses/PV0_0_1_gg_zz
root: Failed at running the process in /afs/cern.ch/work/m/mcfayden/mcgen/MadGraph5_aMCatNLO/testing/testForCyril/PROC_loop_sm_0/SubProcesses/PV0_0_1_gg_zz.
Error detected in "generate_events "
write debug file /afs/cern.ch/work/m/mcfayden/mcgen/MadGraph5_aMCatNLO/testing/testForCyril/PROC_loop_sm_0/run_01_tag_1_debug.log
If you need help with this issue please contact us on https://answers.launchpad.net/madgraph5
MadGraph5Error : Failed the initialization of loop-induced matrix element 'PV0_0_1_gg_zz' (trying with a maximum of 15 PS points).
quit

Any ideas?

Best,

Josh.

Did you clean CutTools, before trying again?
What I mean is going to '<MG_root>/vendor/CutTools' and running 'make clean' and then try again with an output using 'output_dependencies=internal'.

Also, if it still fails, you can go to <MG_root>/PROC_loop_sm_0/SubProcesses/PV0_0_1_gg_zz and check if you can run MadLoop by hand so that you get the details of what goes wrong, with:

cd <MG_root>/PROC_loop_sm_0/SubProcesses/PV0_0_1_gg_zz
make check
./check

With this, we should get a better idea of what goes wrong in your case.

Hi Valentin,

I'm afraid that doesn't seem to work either, I get the same error.

I can't compile in the subprocess directory either, I get this:
@lxplus028 PV0_0_1_gg_zz >: make check
gfortran -O -w -fbounds-check -fPIC -ffixed-line-length-132 -c loop_num.f -o loop_num.o
cts_mprec.h:1.9:
    Included at loop_num.f:38:

      USE MPMODULE
         1
Fatal Error: Can't open module file 'mpmodule.mod' for reading at (1): No such file or directory
make: *** [loop_num.o] Error 1

If it helps, this script recreates the issue from a fresh MG5_aMC tarball:
http://mcfayden.web.cern.ch/mcfayden/test/setup.sh
(with this proc_card: http://mcfayden.web.cern.ch/mcfayden/test/proc_card_mg5.dat)

Best,

Josh.

Thanks for this extra information Josh. So your issue is different, and somehow not affecting my compiling environment.
Anyway, it seems clear that the compiler needs to be specified where to find the 'mpmodule.mod' that I have placed in the 'lib'
directory.

It should be enough then to add the following in line 3 of '<PROC_OUTPUT_ROOT>/SubProcesses/MadLoop_makefile_definitions'

LOOP_INCLUDE =

Should become

LOOP_INCLUDE = -I$(LIBDIR)

Please give it a try. If this works for you, then I'll modify the development branch accordingly. This cannot break anything for the users for which it was already working, so it is fine. (I am however still curious to understand why it worked in my case then)

Cheers

Hi Valentin,

I'm afraid that doesn't seem to work either.

However, if go into PROC_loop_sm_0/Source/CutTools/ and run make by hand and then run generate_events it seems to work just fine. (Did you already say to try that? Sorry if I missed it.)

But shouldn't the CutTools compilation get triggered automatically when running generate_events?

I've updated my previous setup.sh script with this change so you can see how I made it work.

Best,

Josh.

Hi Josh,

Indeed this is what I was suggesting and I am happy to see that it indeed works.

Running your exact scripts without the preemptive CutTools compilation doesn't cause any problem in my case, so I cannot reproduce the issue. I agree however that normally CutTools compilation should be triggered automatically when launching the process (and it is in my case).
Also, in general it is preferable to always use the MG5aMC interface rather than calling the internal scripts directly like you did with 'generate_events'.
What I mean is that it would be better to do:

./bin/mg5_aMC
launch PROC_loop_sm_0
set nevents 1000

or
./bin/mg5_aMC
launch PROC_loop_sm_0 -i
generate_events

The interface supports scripting very well, so you can do put these commands in a file and then run
./bin/mg5_aMC my_MG5_commands
So basically exactly what you did for process generation should also be preferred to the actual running/steering of event generation.

Cheers

Hi everyone,

Coming back to this issue... We still see the same problem within the ATLAS framework ("Fatal Error: Can't open module file 'mpmodule.mod' for reading at (1): No such file or directory")
However it is not clear at all what it causing this issue, as it is fully solved outside the ATLAS framework

Let me be more clear : I ran the script provided by Josh several times. The first time I setup the full ATLAS framework then run the script (i.e. I do not actually use the ATLAS framework, only the different paths for python/gcc/etc matters here) and I see the Fatal error mentionned right above

Then I did two other runs for this script : one case where I only setup the version of gcc used in the ATLAS framework, a second case where I do the same for python. In both cases everything succeeds and I can generate events

Would you have an idea of what other software may interfere with the mg5 configuration ?

Thanks
Cyril

Hi all,

not sure, whether that might help, but I experienced similar problems (with p p > H > lnulnu).

It crashed and I had noticed a problem in the ninja compilation so I tried to re-compile, making sure to set beforehands
PATH=/afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc6-gcc47-opt/bin:${PATH}
LD_LIBRARY_PATH=/afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc5-gcc47-opt/lib64:${LD_LIBRARY_PATH}

it still crashed (and it was also connected to some library problem, which might have looked like an inconsistency). I removed HepTools in the hope, it would re-compile, but it did ignore this (I assume the library is already someplace else and I did not remove that one). re-compiling CutTools did not help --- and i could not figure out, how to make anything else re-compile.

i have now checked out a completely new and clean vresion of aMCatNLO and tried running as first process ever the gg one and it did work!!!!!!!!!!!!!!!!!! :-)

Not sure, how to achieve this now with the version, where I already HAD it installen (and on top of the ExRoot.. and PythiaPGS) but let's see ;)

Below follow the command and the settings of my libraries on lxplus.

Best
Kristin

 import model loop_sm #sm_Higgs
generate p p > H > e+ ve mu- vm~ [QCD]

>>> print (sys.version)
2.6.6 (r266:84292, Jul 23 2015, 00:03:09)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)]

[kristin@lxplus092]/afs/cern.ch/work/k/kristin/WWaMCatNLO/M2test/MG5_aMC_v2_4_0% echo $PATH
/afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc6-gcc47-opt/bin:/afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc6-gcc47-opt/bin:/afs/cern.ch/sw/lcg/external/gcc/4.7.0/x86_64-slc5-gcc47-opt/bin:/afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.07/x86_64-slc5-gcc43-opt/root/bin:/afs/cern.ch/atlas/scripts:/afs/cern.ch/user/k/kristin/scripts:/usr/sue/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/afs/cern.ch/user/k/kristin/bin:/usr/local/sbin:/usr/sbin:/sbin:/afs/cern.ch/user/k/kristin/bin:/opt/bin/

Update: I get the following error, but only if I try to generate > 5000 events

INFO: need to improve 2 channels
INFO: Idle: 0, Running: 1, Completed: -1
Exception AttributeError: "'Popen' object has no attribute '_child_created'" in <bound method Popen.__del__ of <subprocess.Popen object at 0x2128ed0>> ignored
Command "generate_events run_01" interrupted with error:
TypeError : __init__() got an unexpected keyword argument 'packet_member'
Please report this bug on https://bugs.launchpad.net/mg5amcnlo
More information is found in '/afs/cern.ch/work/k/kristin/WWaMCatNLO/M2test/MG5_aMC_v2_4_0/WW_HiggsSM2TeV_pp64/run_01_tag_1_debug.log'.
Please attach this file to your report.
INFO:

@ Kristin:

Olivier found out that your error is triggered because you are using the single core mode (i.e. run_mode 0) for loop-induced which is less tested and has an issue that I will fix in the next release. If you want to run on a single core, I suggest that you do:
set run_mode 2
set nb_core 1
Before running your loop-induced process (this will essentially be my fix by the way, so let me know if it still causes problems, though Olivier tried and it went smoothly [though very slowly of course....]).

@ Others:

Concerning the dependences to the couple of loop-reduction libraries such as CutTools and Ninja, the pattern of errors your report is a bit too erratic for me to be specific.
I will therefore explain here the work-flow of the compilation, hoping that it might help you better understand what goes wrong.

In MG5_aMC v2.4 and beyond, the following three dependencies are used by default for loop reduction and automatically interfaced to our tool (I will not mention other dependencies, since the ones below seems to be the problematic ones).

> CutTools
> IREGI
> Ninja

The tools CutTools and IREGI are shipped with MG5aMC and their source is in <MG5_aMC_install_dir>/vendor/[CutToos|IREGI].
The first time you output a process requiring loops, they are compiled, *within* their respective directory.

Ninja is a bit different since it is an actively developed code. MG5_aMC comes with a local copy of the Ninja installation tarball in
'MG5_aMC_install_dir>/vendor/ninja.tar.gz' and of our dependency installer in 'MG5_aMC_install_dir>/vendor/OfflineHEPToolsInstaller.tar.gz'. The first time you output a process requiring loops, and if an external version of Ninja is not already specified in the parameter 'ninja' in <MG5_aMC_install_dir>/input/mg5_configuration.txt, both of these local files will be used to install 'Ninja' in '<MG5_aMC_install_dir>/HEPTools/ninja/.
Then, the relevant include and library (only the *.a one so as to effectively force static linking) is soft-linked into the directories:

'<MG5_aMC_install_dir>/HEPTools/lib' and '<MG5_aMC_install_dir>/HEPTools/include'.

This is also the scheme we intend to use to all our automatic installation of future dependencies (LHAPDF, HepMC, Pythia8, Madanalysis5, etc.. can already be installed in this way in development versions).

Notice also that at any point the user can type the command 'install ninja' to overwrite the existing installation with the latest one found online (and also using the latest online version of our HEPToolsInstaller).

All of the above is to show you that the first output of a process with loops is a bit particular because it will compile/install new dependencies (and potentially modify the content of mg5_configuration.txt).
Whenever you setup a MG5_aMC distribution, it is therefore a good idea to run a trial loop_process to make sure everything is properly initialized and working. One can do this with the following minimal MadLoop standalone output which is very fast to run (except for the potential installation/compilation of tools that occurs the first time around of course):

./bin/mg5_aMC
MG5_aMC> generate u d~ > e+ ve [virt=QCD]; output InitializationTest; launch -f;

If everything goes according to plan you should see the numerical evaluation of the loops for one particular PS point. This can also be used to make sure that, as far as the loops are concerned your distribution is healthy, and if it is, you shouldn't find any related problems in other applications.

Finally, to be more specific about the linking of these dependencies in process output, the default scheme is as follows:

The following libraries will be soft-linked inside the directory <PROC_OUTPUT>/libs:

'ls -all <PROC_OUTPUT>/libs/*.a' gives (in part)
     libcts.a -> <MG5_aMC_install_dir>/vendor/CutTools/includects/libcts.a
     libiregi.a -> <MG5_aMC_install_dir>/vendor/IREGI/src/libiregi.a

The necessary include fortran module of CutTools, called 'mpmodule.mod' is also included and linked in the same directory, and our makefile make sure to specify '-I <PROC_OUTPUT>/libs' as well. That is:

'ls -all <PROC_OUTPUT>/libs/*.mod' gives
     mpmodule.mod -> <MG5_aMC_install_dir>/vendor/CutTools/includects/mpmodule.mod

Finally, for Ninja the more 'modern' approach is used and our makefiles also link the directory '<MG5_aMC_install_dir>/HepTools/lib' and include the directory ''<MG5_aMC_install_dir>/HepTools/include' where Ninja static library and includes will be found.
Therefore, by default and as far as loop-reduction dependencies go, they are all statically-linked so that the runtime behavior is independent of your LDLIBRARY environment path.

Let me also note that when using the MG5 option 'output_dependencies = internal' then 'Ninja' is disabled and the source directories of 'CutTools' and 'IREGI' are copied from the vendor directory '<MG5_aMC_install_dir>/vendor' into the 'Source' directory of your process output. The soft links to 'libiregi.a', 'libcts.a' and 'mpmodule.mod' in the '<PROC_OUTPUT>/libs' directory then point to this local copy instead, so that, as far as loop-dependencies are concerned, the process output can be moved around independently of MG5_aMC distribution and still compile successfully.
Beware however that this doesn't apply to all other dependencies (such as SysCalc for example) and that certain MG5_aMC features will no longer be available when the process is moved around (for example MadSpin).

I hope that the above technical details help you figure out the various erratic compiling issues you have been experiencing so far within ATLAS.

@ Kristin:

Olivier found out that your error is triggered because you are using the single core mode (i.e. run_mode 0) for loop-induced which is less tested and has an issue that I will fix in the next release. If you want to run on a single core, I suggest that you do:
set run_mode 2
set nb_core 1
Before running your loop-induced process (this will essentially be my fix by the way, so let me know if it still causes problems, though Olivier tried and it went smoothly [though very slowly of course....]).

@ Others:

Concerning the dependences to the couple of loop-reduction libraries such as CutTools and Ninja, the pattern of errors your report is a bit too erratic for me to be specific.
I will therefore explain here the work-flow of the compilation, hoping that it might help you better understand what goes wrong.

In MG5_aMC v2.4 and beyond, the following three dependencies are used by default for loop reduction and automatically interfaced to our tool (I will not mention other dependencies, since the ones below seems to be the problematic ones).

> CutTools
> IREGI
> Ninja

The tools CutTools and IREGI are shipped with MG5aMC and their source is in <MG5_aMC_install_dir>/vendor/[CutToos|IREGI].
The first time you output a process requiring loops, they are compiled, *within* their respective directory.

Ninja is a bit different since it is an actively developed code. MG5_aMC comes with a local copy of the Ninja installation tarball in
'MG5_aMC_install_dir>/vendor/ninja.tar.gz' and of our dependency installer in 'MG5_aMC_install_dir>/vendor/OfflineHEPToolsInstaller.tar.gz'. The first time you output a process requiring loops, and if an external version of Ninja is not already specified in the parameter 'ninja' in <MG5_aMC_install_dir>/input/mg5_configuration.txt, both of these local files will be used to install 'Ninja' in '<MG5_aMC_install_dir>/HEPTools/ninja/.
Then, the relevant include and library (only the *.a one so as to effectively force static linking) is soft-linked into the directories:

'<MG5_aMC_install_dir>/HEPTools/lib' and '<MG5_aMC_install_dir>/HEPTools/include'.

This is also the scheme we intend to use to all our automatic installation of future dependencies (LHAPDF, HepMC, Pythia8, Madanalysis5, etc.. can already be installed in this way in development versions).

Notice also that at any point the user can type the command 'install ninja' to overwrite the existing installation with the latest one found online (and also using the latest online version of our HEPToolsInstaller).

All of the above is to show you that the first output of a process with loops is a bit particular because it will compile/install new dependencies (and potentially modify the content of mg5_configuration.txt).
Whenever you setup a MG5_aMC distribution, it is therefore a good idea to run a trial loop_process to make sure everything is properly initialized and working. One can do this with the following minimal MadLoop standalone output which is very fast to run (except for the potential installation/compilation of tools that occurs the first time around of course):

./bin/mg5_aMC
MG5_aMC> generate u d~ > e+ ve [virt=QCD]; output InitializationTest; launch -f;

If everything goes according to plan you should see the numerical evaluation of the loops for one particular PS point. This can also be used to make sure that, as far as the loops are concerned your distribution is healthy, and if it is, you shouldn't find any related problems in other applications.

Finally, to be more specific about the linking of these dependencies in process output, the default scheme is as follows:

The following libraries will be soft-linked inside the directory <PROC_OUTPUT>/libs:

'ls -all <PROC_OUTPUT>/libs/*.a' gives (in part)
     libcts.a -> <MG5_aMC_install_dir>/vendor/CutTools/includects/libcts.a
     libiregi.a -> <MG5_aMC_install_dir>/vendor/IREGI/src/libiregi.a

The necessary include fortran module of CutTools, called 'mpmodule.mod' is also included and linked in the same directory, and our makefile make sure to specify '-I <PROC_OUTPUT>/libs' as well. That is:

'ls -all <PROC_OUTPUT>/libs/*.mod' gives
     mpmodule.mod -> <MG5_aMC_install_dir>/vendor/CutTools/includects/mpmodule.mod

Finally, for Ninja the more 'modern' approach is used and our makefiles also link the directory '<MG5_aMC_install_dir>/HepTools/lib' and include the directory ''<MG5_aMC_install_dir>/HepTools/include' where Ninja static library and includes will be found.
Therefore, by default and as far as loop-reduction dependencies go, they are all statically-linked so that the runtime behavior is independent of your LDLIBRARY environment path.

Let me also note that when using the MG5 option 'output_dependencies = internal' then 'Ninja' is disabled and the source directories of 'CutTools' and 'IREGI' are copied from the vendor directory '<MG5_aMC_install_dir>/vendor' into the 'Source' directory of your process output. The soft links to 'libiregi.a', 'libcts.a' and 'mpmodule.mod' in the '<PROC_OUTPUT>/libs' directory then point to this local copy instead, so that, as far as loop-dependencies are concerned, the process output can be moved around independently of MG5_aMC distribution and still compile successfully.
Beware however that this doesn't apply to all other dependencies (such as SysCalc for example) and that certain MG5_aMC features will no longer be available when the process is moved around (for example MadSpin).

I hope that the above technical details help you figure out the various erratic compiling issues you have been experiencing so far within ATLAS.