installation of lhapdf5/6 fails

Hi,

I have updated the link, should be fixed at around 20h45.
(The time that all mirror update themself).
Not sure if this fix will also work for lhapdf5 but no one should use that one anymore.

Cheers,

Olivier

> On 3 Feb 2020, at 19:47, Wolfgang Waltenberger <email address hidden> wrote:
>
> New question #688482 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/688482
>
> In one of my setups, the installation procedures of lhapdf5/6 fail, because wget names the files
> "download?f=xxx". There is a http-redirect of hepforge.org/archive at works here, and it seems to depend
> on the specific network setup, whether wget uses the correct local filename or the redirected one. Could I please ask you to explicitly give the local file names, i.e. always use "wget -O xxx yyy" for all the automated HEPTools downloads? JFYI, for me it doesnt work in some specific singularity/docker setups.
>
> Thanks a lot,
>
> Wolfgang
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Dear Oliver,

I am encountering a very similar error, which causes the installation of lhapdf5 to fail, while that of lhapdf6 succeeds.

The error is `gzip: stdin: not in gzip format`, and seems to be caused by an extra `f=` in the URL pointing to the archive (which is then present twice). This extra `f=` is added on line 140 in `HEPToolInstaller.py`, and is absent for lhapdf6.

This causes the script to attempt to download `https://lhapdf.hepforge.org/downloads?f=f=LHAPDF-5.9.0.tar.gz` instead of `https://lhapdf.hepforge.org/downloads?f=lhapdf-5.9.0.tar.gz`. Unfortunately, this is difficult to fix locally, since `HEPToolInstaller.py` is downloaded during the install and cannot be edited before installing lhapdf5.

Could you please have a look at `HEPToolInstaller.py` to check if this extra `f=` is indeed what causes the issue?

Thanks!

Jean-Loup

JFYI, thats also what I am experiencing. And I agree with Jean-Loup,
its difficult to fix it locally since the installer itself gets downloaded.
Thanks in advance, guys,

Wolfgang

On Fri, 21 Feb 2020 at 19:43, Jean-Loup Tastet <
<email address hidden>> wrote:

> Your question #688482 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/688482
>
> Jean-Loup Tastet posted a new comment:
> Dear Oliver,
>
> I am encountering a very similar error, which causes the installation of
> lhapdf5 to fail, while that of lhapdf6 succeeds.
>
> The error is `gzip: stdin: not in gzip format`, and seems to be caused
> by an extra `f=` in the URL pointing to the archive (which is then
> present twice). This extra `f=` is added on line 140 in
> `HEPToolInstaller.py`, and is absent for lhapdf6.
>
> This causes the script to attempt to download
> `https://lhapdf.hepforge.org/downloads?f=f=LHAPDF-5.9.0.tar.gz`
> <https://lhapdf.hepforge.org/downloads?f=f=LHAPDF-5.9.0.tar.gz> instead
> of `https://lhapdf.hepforge.org/downloads?f=lhapdf-5.9.0.tar.gz`
> <https://lhapdf.hepforge.org/downloads?f=lhapdf-5.9.0.tar.gz>.
> Unfortunately, this is difficult to fix locally, since
> `HEPToolInstaller.py` is downloaded during the install and cannot be
> edited before installing lhapdf5.
>
> Could you please have a look at `HEPToolInstaller.py` to check if this
> extra `f=` is indeed what causes the issue?
>
> Thanks!
>
> Jean-Loup
>
> --
> You received this question notification because you asked the question.
>

HI,

I have remove the f= in the link but even with that it fails to download it.
I guess that this is linked to some internal mode ofhepforge/lhapdf.

At the same time, lhapdf5 should not be used anymore ( I plan to drop support for lhapdf5 and lhapdf6.1 in the next version of MG5aMC)

Cheers,

Olivier

Hi Oliver,

Did you update HEPToolsInstaller_V12X.tar.gz ? The latest version (V126, from 24-Feb-2020 03:33) still contains the `f=`.

It turns out the error I was seeing was not directly related to LHAPDF5. The symptoms are the same, but this time it looks like it is the `pdfsets/6.1/` directory which is missing on HEPForge.

```
INFO: Preparing MCatNLO run
INFO: Using LHAPDF v6.1.6 interface for PDFs
INFO: Trying to download NNPDF23_nlo_as_0119_qed
ERROR: Corrupted file on line 0: /home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_7_0/HEPTools/lhapdf6/share/LHAPDF/pdfsets.index
WARNING: No matching PDFs for pattern: NNPDF23_nlo_as_0119_qed
URL transformed to HTTPS due to an HSTS policy
--2020-02-24 10:44:16-- https://www.hepforge.org/archive/lhapdf/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
Resolving www.hepforge.org (www.hepforge.org)... 129.234.186.186
Connecting to www.hepforge.org (www.hepforge.org)|129.234.186.186|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.hepforge.org/downloads/lhapdf/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
--2020-02-24 10:44:16-- https://www.hepforge.org/downloads/lhapdf/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
Reusing existing connection to www.hepforge.org:443.
HTTP request sent, awaiting response... 302 Found
Location: https://lhapdf.hepforge.org/downloads/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
--2020-02-24 10:44:16-- https://lhapdf.hepforge.org/downloads/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
Resolving lhapdf.hepforge.org (lhapdf.hepforge.org)... 129.234.186.186
Connecting to lhapdf.hepforge.org (lhapdf.hepforge.org)|129.234.186.186|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /downloads?f=/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
--2020-02-24 10:44:17-- https://lhapdf.hepforge.org/downloads?f=/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
Reusing existing connection to lhapdf.hepforge.org:443.
HTTP request sent, awaiting response... 302 Found
Location: /downloads?f=pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
--2020-02-24 10:44:17-- https://lhapdf.hepforge.org/downloads?f=pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
Reusing existing connection to lhapdf.hepforge.org:443.
HTTP request sent, awaiting response... 200 OK
Length: 1799 (1,8K) [text/html]
Saving to: ‘/home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_7_0/HEPTools/lhapdf6//share/LHAPDF/NNPDF23_nlo_as_0119_qed.tar.gz’

/home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_ 100%[=====================================================================================================================>] 1,76K --.-KB/s in 0s

2020-02-24 10:44:17 (48,7 MB/s) - ‘/home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_7_0/HEPTools/lhapdf6//share/LHAPDF/NNPDF23_nlo_as_0119_qed.tar.gz’ saved [1799/1799]

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
WARNING: Trying to shower events using the same PDF in the shower as used in the generation of the events using LHAPDF. However, no valid LHAPDF installation found with the needed PDF set. Will use default internal PDF for the shower instead. To use the same set as was used in the event generation install LHAPDF and set the path using "set /path_to_lhapdf/bin/lhapdf-config" from the MadGraph5_aMC@NLO python shell
```

I don't know if the error comes from the corrupted `pdfsets.index` or from the missing directory. Could you have a look at it? Thanks!

Cheers,
Jean-Loup

Hi,

> Did you update HEPToolsInstaller_V12X.tar.gz ? The latest version (V126,
> from 24-Feb-2020 03:33) still contains the `f=`.

Sorry, I did the commit but I forget to push it on the server.
(at the same time since it does not help ...)

> It turns out the error I was seeing was not directly related to LHAPDF5.
> The symptoms are the same, but this time it looks like it is the
> `pdfsets/6.1/` directory which is missing on HEPForge.

This was an existing path last week.
Looks like LHPADF people have killed that link.
(Looks like they plan to not use hepforge for that anymore)

For the moment you can replace the line 4196 of the file madgraph/various/common_run_interface.py
wwwpath = "http://www.hepforge.org/archive/lhapdf/pdfsets/%s/%s.tar.gz"
to either
wwwpath = "http://www.hepforge.org/archive/lhapdf/pdfsets/v6.backup/%s/%s.tar.gz"
or to
wwwpath = "http://lhapdfsets.web.cern.ch/lhapdfsets/current/%s.tar.gz" % filename
(but then comment line #4197)

LHAPDF authors does not like the fact that we propose such feature of automatic download of pdfset. So I will not be able to convince them to fix their link. Also note that we try to use the tools provided by lhapdf to download the set but their own tool are not working either.

Cheers,

Olivier

> On 24 Feb 2020, at 10:57, Jean-Loup Tastet <email address hidden> wrote:
>
> Question #688482 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/688482
>
> Jean-Loup Tastet posted a new comment:
> Hi Oliver,
>
> Did you update HEPToolsInstaller_V12X.tar.gz ? The latest version (V126,
> from 24-Feb-2020 03:33) still contains the `f=`.
>
> It turns out the error I was seeing was not directly related to LHAPDF5.
> The symptoms are the same, but this time it looks like it is the
> `pdfsets/6.1/` directory which is missing on HEPForge.
>
> ```
> INFO: Preparing MCatNLO run
> INFO: Using LHAPDF v6.1.6 interface for PDFs
> INFO: Trying to download NNPDF23_nlo_as_0119_qed
> ERROR: Corrupted file on line 0: /home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_7_0/HEPTools/lhapdf6/share/LHAPDF/pdfsets.index
> WARNING: No matching PDFs for pattern: NNPDF23_nlo_as_0119_qed
> URL transformed to HTTPS due to an HSTS policy
> --2020-02-24 10:44:16-- https://www.hepforge.org/archive/lhapdf/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
> Resolving www.hepforge.org (www.hepforge.org)... 129.234.186.186
> Connecting to www.hepforge.org (www.hepforge.org)|129.234.186.186|:443... connected.
> HTTP request sent, awaiting response... 301 Moved Permanently
> Location: https://www.hepforge.org/downloads/lhapdf/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
> --2020-02-24 10:44:16-- https://www.hepforge.org/downloads/lhapdf/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
> Reusing existing connection to www.hepforge.org:443.
> HTTP request sent, awaiting response... 302 Found
> Location: https://lhapdf.hepforge.org/downloads/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
> --2020-02-24 10:44:16-- https://lhapdf.hepforge.org/downloads/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
> Resolving lhapdf.hepforge.org (lhapdf.hepforge.org)... 129.234.186.186
> Connecting to lhapdf.hepforge.org (lhapdf.hepforge.org)|129.234.186.186|:443... connected.
> HTTP request sent, awaiting response... 302 Found
> Location: /downloads?f=/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
> --2020-02-24 10:44:17-- https://lhapdf.hepforge.org/downloads?f=/pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
> Reusing existing connection to lhapdf.hepforge.org:443.
> HTTP request sent, awaiting response... 302 Found
> Location: /downloads?f=pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz [following]
> --2020-02-24 10:44:17-- https://lhapdf.hepforge.org/downloads?f=pdfsets/6.1/NNPDF23_nlo_as_0119_qed.tar.gz
> Reusing existing connection to lhapdf.hepforge.org:443.
> HTTP request sent, awaiting response... 200 OK
> Length: 1799 (1,8K) [text/html]
> Saving to: ‘/home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_7_0/HEPTools/lhapdf6//share/LHAPDF/NNPDF23_nlo_as_0119_qed.tar.gz’
>
> /home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_
> 100%[=====================================================================================================================>]
> 1,76K --.-KB/s in 0s
>
> 2020-02-24 10:44:17 (48,7 MB/s) -
> ‘/home/jl/Documents/PhD/Software/MadGraph/MG5_aMC_v2_7_0/HEPTools/lhapdf6//share/LHAPDF/NNPDF23_nlo_as_0119_qed.tar.gz’
> saved [1799/1799]
>
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> WARNING: Trying to shower events using the same PDF in the shower as used in the generation of the events using LHAPDF. However, no valid LHAPDF installation found with the needed PDF set. Will use default internal PDF for the shower instead. To use the same set as was used in the event generation install LHAPDF and set the path using "set /path_to_lhapdf/bin/lhapdf-config" from the MadGraph5_aMC@NLO python shell
> ```
>
> I don't know if the error comes from the corrupted `pdfsets.index` or
> from the missing directory. Could you have a look at it? Thanks!
>
> Cheers,
> Jean-Loup
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi,

Note that next version of madgraph, will be able to install lhapdf6.2
and that version of lhapdf fixes the issue they had with the automatic installation of the pdf set.
So you will not need the fallback that we put in place to download the pdf sets.

Cheers,

Olivier

Hi Oliver,

Thanks a lot for looking into this. I'll try your fix.

Cheers,
Jean-Loup

Your proposed fix is working (I used the CERN mirror). Thanks a lot! I'll remember to upgrade to the new version of MadGraph when it is released.

Cheers,
Jean-Loup