Can 3.1.X be built without pythia8?

No this is not mandatory.

The only step really mandatory is to run once at LO and once at NLO (if you plan to run both)
such that all the associated library are compiled.
All the tools that you install/compile in compile.py are technically optional and not needed.

Cheers,

Olivier

> On 4 Aug 2021, at 13:11, Dmitri Konstantinov <email address hidden> wrote:
>
> New question #698236 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/698236
>
> Dear experts,
>
> We are trying to build Madgraph 3.1.1 following ATLAS request.
> But we see that for 3.X.X - execution of .compile.py starts to build Pythia8 while initially we don't have it anywhere in mg5_configuration.txt or in .mg5_configuration_default.txt.
>
> And after unsuccessful run of .compily.py we see that mg5_configuration.txt is updated with:
>
>> # pythia8_path = ./HEPTools/pythia8
> 77c77
> < mg5amc_py8_interface_path = /tmp/dkonst/build/generators/madgraph5amc-3.1.1.atlas/src/madgraph5amc/3.1.1.atlas/HEPTools/MG5aMC_PY8_interface #
>
>
> Is Pythia8 mandatory in Madgraph 3.X.X ?
>
>
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hello, Olivier,

Thank you.

But then how to do it properly? Even not having pythia8 options in mg5_configuration.txt or in .mg5_configuration_default.txt it tries to download and install pythia8 and its dependencies when we run `python bin/.compile.py`

Kind Regards,
          Dmitri

So the default option of compile.py is to install those four optional packages:
['pythia8','Delphes','ExRootAnalysis','MadAnalysis5']

you can however specify the list of optional packages that you want to add via arguments so you can do
compile.py Delphes ExRootAnalysis MadAnalysis5

if you want to remove pythia8.

If you do not want any of those four
then you can do
compile.py ninja
(since ninja is likely to be installed anyway for NLO running)

Cheers,

Olivier

PS: Here is the list of optional program that can be installed:

boost hepmc MadAnalysis4 QCDLoop MadAnalysis5 ninja collier lhapdf5 MadAnalysis5 ninja zlib maddm oneloop zlib
Delphes lhapdf6 maddump oneloopExRootAnalysis MadSTR pythia8
Golem95 mg5amc_py8_interface

> On 4 Aug 2021, at 13:50, Dmitri Konstantinov <email address hidden> wrote:
>
> Question #698236 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/698236
>
> Status: Answered => Open
>
> Dmitri Konstantinov is still having a problem:
> Hello, Olivier,
>
> Thank you.
>
> But then how to do it properly? Even not having pythia8 options in
> mg5_configuration.txt or in .mg5_configuration_default.txt it tries to
> download and install pythia8 and its dependencies when we run `python
> bin/.compile.py`
>
> Kind Regards,
> Dmitri
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Dear Olivier,

This did not work as we expected.

In LCG we all previous Madgraph5amc version (2.X.X) were built against prebuilt its dependencies, i.e. we already have ninja, lhapdf etc. And now for 3.X.X we would like to follow the same straregy, something doesn't work.

I am still investigating...

Hi,

pythia-pgs is not supported anymore and not compatible with recent compiler so this is the reason I switch in compile.py from pythia-pgs to pythia8.

So my question is how in previous version you were handling those four packages that were installed by default?
pythia-pgs
Delphes
ExRootAnalysis
MadAnalysis5

Did you installed those in previous release?

Cheers,

Olivier

> On 5 Aug 2021, at 11:35, Dmitri Konstantinov <email address hidden> wrote:
>
> Question #698236 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/698236
>
> Dmitri Konstantinov posted a new comment:
> Dear Olivier,
>
> This did not work as we expected.
>
> In LCG we all previous Madgraph5amc version (2.X.X) were built against
> prebuilt its dependencies, i.e. we already have ninja, lhapdf etc. And
> now for 3.X.X we would like to follow the same straregy, something
> doesn't work.
>
> I am still investigating...
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi, Olivier,

For MG 2.9.X we apply the following patch partially provided by Zach Marshall (ATLAS):

https://gitlab.cern.ch/sft/lcgcmake/-/blob/LCG_88b/generators/patches/madgraph5amc-2.9.3.atlas.patch#L1068

In pre-build time we replace the placeholders for collier, lhapdf, ninja and SysCalc by actual paths. And then we ran compile.py.

As far as I see pythia-pgs, Delphes, ExRootAnalysis, MadAnalysis5 are not installed by default, at least I don't see them in our installations of 2.9.X. So I guess we did not install them,

But now in 3.X.X we have some problem to follow ATLAS recipe for 2.9.X.

Hi,

If I understand correctly your patch you are hacking compile.py to not install any external package.
So this means that you should actually not run the installation of pythia8.
So the first question is if your patching method went trough or not?

Additionally, I see some very dangerous patch (the one in banner.py) which impact what can be run within the code.
This is very dangerous to alter the code like that (especially for a LCG release). This mode is forbidden for some reason and one of my student is working to make that mode working as expected/needed in a near future.

Cheers,

Olivier

> On 5 Aug 2021, at 12:15, Dmitri Konstantinov <email address hidden> wrote:
>
> Question #698236 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/698236
>
> Dmitri Konstantinov posted a new comment:
> Hi, Olivier,
>
> For MG 2.9.X we apply the following patch partially provided by Zach
> Marshall (ATLAS):
>
> https://gitlab.cern.ch/sft/lcgcmake/-/blob/LCG_88b/generators/patches/madgraph5amc-2.9.3.atlas.patch#L1068
>
> In pre-build time we replace the placeholders for collier, lhapdf, ninja
> and SysCalc by actual paths. And then we ran compile.py.
>
> As far as I see pythia-pgs, Delphes, ExRootAnalysis, MadAnalysis5 are
> not installed by default, at least I don't see them in our installations
> of 2.9.X. So I guess we did not install them,
>
> But now in 3.X.X we have some problem to follow ATLAS recipe for 2.9.X.
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hello, Olivier,

Many thanks!

Right. I found one problem and now I am able to re-use collier, lhapdf, ninja and SysCalc from our installation.

There was a problem with lhapdf-config in 3_1_1. Even providing full path to lhapdf-config it tries to run lhapdf-config without full path...it is mitigated by adding lhapdf bin directory to PATH.

Please have a look at the successful output of 2_9_3 build:

https://pastebin.com/TXj6nCPx

And for 3.1.1 my build fails:

https://pastebin.com/kdAyFRp7

The simple script showing how I build it (not including LD_LIBRARY_PATH and PYTHONPATH for compiler and externals):

https://pastebin.com/YfpmrySU

Is there anything wrong with it?

Thank you!

P.S. Regarding dangerous patch provided by ATLAS - I guess it is not a problem if ATLAS people knows what they do. In addition this patched version is installed as ATLAS version and contains ATLAS in the version, i.e. 2.9.3.atlas

You can apply the following path to compile.py that fixes the issue.

=== modified file 'bin/compile.py'
--- bin/compile.py 2021-07-06 19:50:28 +0000
+++ bin/compile.py 2021-08-10 19:35:01 +0000
@@ -84,8 +84,8 @@

     def test_output_NLO(self):
         """do the output of a simple LO process to ensure that LO is correctly configure."""
- self.cmd.exec_cmd('generate p p > e+ ve [QCD]')
- self.cmd.exec_cmd('output %s/TESTNLO' %root_path)
+ self.cmd.run_cmd('generate p p > e+ ve [QCD]')
+ self.cmd.run_cmd('output %s/TESTNLO' %root_path)
         shutil.rmtree('%s/TESTNLO' % root_path)

Concerning the dangerous patch. If only Zach is using the patch then he knows that we do not support that and is likely aware of the issues of such computation. But then I do not see the point of a central release for one person.
At minimum a clear message should be printed such that this is not officially supported/...
I would suggest a message like:
logger.critical("Such type of computation are normally forbidden within the official version of MG5aMC. Use at your own risk. The authors of MG5aMC will not provide any support on such run and the result of such run might not be reproducible in the version of the code supporting officially such computation.")

If ATLAS hack the code for physics this should be clearly stated and not done in a hidden way.
If possible this message should be added to all hacked version of the code including your 2.9.3.atlas

Dear Olivier,

Thank you for the patch. It went through this place but now it says:

https://pastebin.com/h64aKxs4

Yes, agreed about disclaimer massage. I will add it.

Kind Regards,
         Dmitri

This I do not reproduce, which version of python do you use for running compile.py.
On my side I have tested with 2.7, 3.8 and 3.9.
Looks like that file does not check that the python version used is compatible with Mg5aMC.
So it is not impossible that you run with an unsupported version of MG5aMC.

Python requirement are either python2.7 or python3.7 (or any newer version of python3).
Note that you might want to run compile.py for all the python version that you want to run to allow the precompilation of all python file [creation of the .pyc and .pyo] (which will marginally speed-up the computation)

Cheers,

Olivier

> On 11 Aug 2021, at 11:41, Dmitri Konstantinov <email address hidden> wrote:
>
> Question #698236 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/698236
>
> Dmitri Konstantinov posted a new comment:
> Dear Olivier,
>
> Thank you for the patch. It went through this place but now it says:
>
> https://pastebin.com/h64aKxs4
>
> Yes, agreed about disclaimer massage. I will add it.
>
> Kind Regards,
> Dmitri
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

No, we use Python 3.8.6, so it should be fine...
It seems "order" is empty.

        firstprocess = history.get('generate')
        print("firstprocess = ", firstprocess)
        order = re.findall("\[(.*)\]", firstprocess)

Adding debug message I see that firstprocess stores "p p > t t~ "

I don't know if this can be helpful.

Thank you,
            Dmitri

My best bet here is that you did not apply my patch correctly, I do reproduce your bug if I change the line from the function
def test_output_LO(self):
and not
def test_output_NLO(self):

Can you double check?

Cheers,

Olivier

You are right, my bad. Thank you.

Should we replace exec_cmd to run_cmd for both, test_output_NLO and test_output_LO?

After replacing both it compiles but the output contains some error messages coming from different places:

Listing './tests/input_files/mssm'...
Compiling './tests/input_files/mssm/interactions.py'...
*** File "./tests/input_files/mssm/interactions.py", line 5
    'color': [1 f(0,1,2)],
                ^
SyntaxError: invalid syntax

Compiling './tests/input_files/mssm/particles.py'...
Listing './tests/input_files/sm'...
Compiling './tests/input_files/sm/interactions.py'...
*** File "./tests/input_files/sm/interactions.py", line 19
    'color': [1 f(0,1,2)],
                ^
SyntaxError: invalid syntax

Hi,

> Should we replace exec_cmd to run_cmd for both, test_output_NLO and test_output_LO?

For LO, looks like it does not matter. The difference between both is how post-processing is handle in the function.
Since 3.1 they do some check using the history of the command used which is done during the post-processing stage and this is why you need "run" and not "exec". For LO, the history is not used and then both are fine.

For the other error, I would just ignore them since they are related to test, and it can make sense to have those files that does not follow python convention (since we want to support some old UFO convention). I will take a deeper look to see if we can simply remove those files now, but in any case, this is not something that I would worry about for your release.

Cheers,

Olivier

Hi Olivier,

Great! Many thanks!

One more observation.

3.1.1 tarball contains one broken symlink:

pineappl_maxproc.inc -> ../../SubProcesses/pineappl_maxproc.inc

What do we do with it?

Cheers,
    Dmitri

Hi,

This should be fine and should be kept like that.
The symbolic link will be copied as is in the generated directory and then file SubProcesses/pineappl_maxproc.inc will be created.
Such that in the code generated by MG5aMC the symbolic link will not be dead anymore, and the code will be able to compile.
(In case you do not know, MG5aMC is a meta-code: a code generating code).

Cheers,

Olivier