NLO Reweighting not using LHAPDF_DATA_PATH?

Hi,

Here is the code that we use to deternine the path to the directory

        if not pdfsets_dir:
            if 'LHAPATH' in os.environ:
                for p in os.environ['LHAPATH'].split(':'):
                    if os.path.exists(p):
                        pdfsets_dir = p
                        break
                else:
                    del os.environ['LHAPATH']
                    pdfsets_dir = subprocess.Popen([lhapdf_config, '--datadir'],
                        stdout = subprocess.PIPE).stdout.read().strip()
            else:
                pdfsets_dir = subprocess.Popen([lhapdf_config, '--datadir'],
                        stdout = subprocess.PIPE).stdout.read().strip()

according to your debugfile.

So if you have LHAPATH defined we check if it contains an existing path and if so use it.
In any other case we run
lhapdf-config --datadir
to get the information (where the path of that lhapdf-config can be modified in the preference file.

Cheers,

Olivier

> On 24 Jul 2019, at 18:57, Hannes <email address hidden> wrote:
>
> New question #682292 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/682292
>
> Hi,
>
> our LHAPDFsets and pdfsets.index are in a non-standard location and we usually tell MG5_aMC to look in the right place by setting LHAPDF_DATA_PATH appropriately, which works fine.
>
> However, when doing NLO-accurate reweighting, it seems the reweighting module tries to access pdfsets.index at a different place (I think at $LHAPATH/pdfsets.index).
> Is this behaviour desired? I would have hoped it also looks at LHAPDF_DATA_PATH (first).
>
> Maybe I am just confused regarding the meaning of LHAPATH and LHAPDF_DATA_PATH.
>
> Cheers,
> Hannes
>
> ---
>
> reweight run_01 -from_cards
> Traceback (most recent call last):
> File "/scratch/mildner/MGtests/reweightNLO/run/PROCNLO_loop_sm_0/bin/internal/extended_cmd.py", line 1514, in onecmd
> return self.onecmd_orig(line, **opt)
> File "/scratch/mildner/MGtests/reweightNLO/run/PROCNLO_loop_sm_0/bin/internal/extended_cmd.py", line 1463, in onecmd_orig
> return func(arg, **opt)
> File "/scratch/mildner/MGtests/reweightNLO/run/PROCNLO_loop_sm_0/bin/internal/common_run_interface.py", line 2142, in do_reweight
> reweight_cmd.import_command_file(path)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1660
> , in import_command_file
> self.exec_cmd(line, precmd=True)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1543
> , in exec_cmd
> stop = Cmd.onecmd_orig(current_interface, line, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1463
> , in onecmd_orig
> return func(arg, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", lin
> e 873, in do_set
> return self.exec_cmd("launch")
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1543, in exec_cmd
> stop = Cmd.onecmd_orig(current_interface, line, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1463, in onecmd_orig
> return func(arg, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/various/misc.py", line 100, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", line 487, in do_launch
> self.create_standalone_directory()
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/various/misc.py", line 100, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", line 1592, in create_standalone_directory
> self.create_standalone_virt_directory(data, second)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/various/misc.py", line 100, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", line 1441, in create_standalone_virt_directory
> mgcmd.options['lhapdf'], None, self.banner.run_card.get_lhapdf_id())
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/common_run_interface.py", line 4137, in install_lhapdf_pdfset_static
> pdf_info = CommonRunCmd.get_lhapdf_pdfsets_list_static(pdfsets_dir, lhapdf_version)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/common_run_interface.py", line 4267, in get_lhapdf_pdfsets_list_static
> [l for l in open(pjoin(pdfsets_dir, 'pdfsets.index')).read().split('\n') if l.strip()]
> IOError: [Errno 2] No such file or directory: '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc49-opt/20.7.9/sw/lcg/releases/LCG_81f/MCGenerators/lhapdf/6.1.5/x86_64-slc6-gcc49-opt/share/LHAPDF/pdfsets.index'
>
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

right, but at other times you look for LHAPDF_DATA_PATH instead, no?
For normal NLO generation it finds the right pdfsets directory, which is located at LHAPDF_DATA_PATH but reweighting crashes because it looks at LHAPATH.

Cheers,
Hannes

Hi,

That part of the code is actually not specific at all to the re-weighting module.
So I would expect the same handling at LO/NLO.

I checked on the lhapdf webpage and do not find the reference to which variable should be used in priority. In any case, I would advise you to clear the variable LHAPATH if this one is not the valid one.

Cheers,

Olivier

> On 24 Jul 2019, at 18:57, Hannes <email address hidden> wrote:
>
> New question #682292 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/682292
>
> Hi,
>
> our LHAPDFsets and pdfsets.index are in a non-standard location and we usually tell MG5_aMC to look in the right place by setting LHAPDF_DATA_PATH appropriately, which works fine.
>
> However, when doing NLO-accurate reweighting, it seems the reweighting module tries to access pdfsets.index at a different place (I think at $LHAPATH/pdfsets.index).
> Is this behaviour desired? I would have hoped it also looks at LHAPDF_DATA_PATH (first).
>
> Maybe I am just confused regarding the meaning of LHAPATH and LHAPDF_DATA_PATH.
>
> Cheers,
> Hannes
>
> ---
>
> reweight run_01 -from_cards
> Traceback (most recent call last):
> File "/scratch/mildner/MGtests/reweightNLO/run/PROCNLO_loop_sm_0/bin/internal/extended_cmd.py", line 1514, in onecmd
> return self.onecmd_orig(line, **opt)
> File "/scratch/mildner/MGtests/reweightNLO/run/PROCNLO_loop_sm_0/bin/internal/extended_cmd.py", line 1463, in onecmd_orig
> return func(arg, **opt)
> File "/scratch/mildner/MGtests/reweightNLO/run/PROCNLO_loop_sm_0/bin/internal/common_run_interface.py", line 2142, in do_reweight
> reweight_cmd.import_command_file(path)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1660
> , in import_command_file
> self.exec_cmd(line, precmd=True)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1543
> , in exec_cmd
> stop = Cmd.onecmd_orig(current_interface, line, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1463
> , in onecmd_orig
> return func(arg, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", lin
> e 873, in do_set
> return self.exec_cmd("launch")
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1543, in exec_cmd
> stop = Cmd.onecmd_orig(current_interface, line, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/extended_cmd.py", line 1463, in onecmd_orig
> return func(arg, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/various/misc.py", line 100, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", line 487, in do_launch
> self.create_standalone_directory()
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/various/misc.py", line 100, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", line 1592, in create_standalone_directory
> self.create_standalone_virt_directory(data, second)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/various/misc.py", line 100, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/reweight_interface.py", line 1441, in create_standalone_virt_directory
> mgcmd.options['lhapdf'], None, self.banner.run_card.get_lhapdf_id())
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/common_run_interface.py", line 4137, in install_lhapdf_pdfset_static
> pdf_info = CommonRunCmd.get_lhapdf_pdfsets_list_static(pdfsets_dir, lhapdf_version)
> File "/cvmfs/sft.cern.ch/lcg/releases/MCGenerators/madgraph5amc/2.6.5.atlas3-ba1c8/x86_64-slc6-gcc49-opt/madgraph/interface/common_run_interface.py", line 4267, in get_lhapdf_pdfsets_list_static
> [l for l in open(pjoin(pdfsets_dir, 'pdfsets.index')).read().split('\n') if l.strip()]
> IOError: [Errno 2] No such file or directory: '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc49-opt/20.7.9/sw/lcg/releases/LCG_81f/MCGenerators/lhapdf/6.1.5/x86_64-slc6-gcc49-opt/share/LHAPDF/pdfsets.index'
>
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

but you still just check whether the first path in LHAPATH exists and then stop, whether the pdfsets.index is located there or not, could at least that be improved?

And even in the absence of a clear convention, it is not super intuitive that different parts of the code look for the same thing with different environment variables.

Cheers,
Hannes

Looks like the issue is that NLO is using itself two convention.
(Ad said above the re-weighting it using the code of the LO/NLO interface).
But they are an option to force the pdfsetsdir that the NLO code used (I did not check but this is the only way your behavior makes sense). In top of that I found another method in the NLO code that check for the LHAPDF_DATA_PATH.

So I'm proposing to the NLO author a fix that should makes the code more consistent.
Now I will also discuss this more in details next week with lhapdf author and will try to fully re-write the lhapdf interface in MG5aMC. So not sure that such patch will finally be include or not in any future version.

Cheers,

Olivier

PS: The patch can be found here:
https://bazaar.launchpad.net/~mg5core1/mg5amcnlo/2.6.7/revision/289

> On 26 Jul 2019, at 00:37, Hannes <email address hidden> wrote:
>
> Question #682292 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/682292
>
> Hannes posted a new comment:
> Hi Olivier,
>
> but you still just check whether the first path in LHAPATH exists and
> then stop, whether the pdfsets.index is located there or not, could at
> least that be improved?
>
> And even in the absence of a clear convention, it is not super intuitive
> that different parts of the code look for the same thing with different
> environment variables.
>
> Cheers,
> Hannes
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

thanks a lot for the patch!

Cheers,
Hannes

Hi Olivier,

Actually that's the error I have encountered also today. It seems it has not been implemented yet to NLO?

Cheers,
Binish

It has been pushed to the development version.
So it is not yet in 2.6.6 but in the alpha version of 2.6.7 (now it is possible that we will jump directly to 2.7.0 around september)

Cheers,

Olivier

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: <email address hidden><mailto:<email address hidden>> <<email address hidden><mailto:<email address hidden>>> on behalf of Binish Batool <<email address hidden><mailto:<email address hidden>>>
Sent: Tuesday, August 6, 2019 3:17:26 PM
To: Olivier Mattelaer <<email address hidden><mailto:<email address hidden>>>
Subject: Re: [Question #682292]: NLO Reweighting not using LHAPDF_DATA_PATH?

Question #682292 on MadGraph5_aMC@NLO changed:
https://answers.launchpad.net/mg5amcnlo/+question/682292

Binish Batool posted a new comment:
Hi Olivier,

Actually that's the error I have encountered also today. It seems it has
not been implemented yet to NLO?

Cheers,
Binish

--
You received this question notification because you are an answer
contact for MadGraph5_aMC@NLO.