Reweighting at NLO

Hi,

> For a quick test, I studied p p > t t~ [QCD] and reweighted the mass to 180 GeV (I understand this is not what the reweighting is for :) )

Note that changing the mass of the final state particles is not only bad for efficiency point of view but just wrong. (since the kinematic of the events do not change)

> Now I am trying to understand the output, i.e. the parameters stored within the <mgrwgt> </mgrwgt> block.

Those store the information that are needed to perform the reweighting.

> Is there some documentation about it?

No. But you can look at the lhe_parser.py class on how such information is parse it contains all the information needed.

> I had expected that a weight for the additional hypothesis would be stored inside the <rwgt> </rwgt> section similar to the LO reweighting.

Yes it should.
I guess that your reweighting crash for some reason (maybe just some consistency check which make the code to stop.)

Cheers,

Olivier

> On May 10, 2016, at 09:27, Benedikt Maier <email address hidden> wrote:
>
> New question #293529 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Hi MG5_aMCatNLO team,
>
> I am currently digging into the newly available method of reweighting events at NLO. For a quick test, I studied p p > t t~ [QCD] and reweighted the mass to 180 GeV (I understand this is not what the reweighting is for :) )
>
> This is an example event of the run:
>
> https://www.dropbox.com/s/04wrv7qvg4je4f4/example_event.txt?dl=0
>
> Now I am trying to understand the output, i.e. the parameters stored within the <mgrwgt> </mgrwgt> block. Is there some documentation about it? I had expected that a weight for the additional hypothesis would be stored inside the <rwgt> </rwgt> section similar to the LO reweighting. But this seems not the case. Can you shed some light on the syntax?
>
>
> THANKS!
>
> Benedikt
>
>
>
>
>
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

Thanks for the reply. I reran with the default reweight card which does

launch
set sminputs 1 130 # modify 1/alpha_EW

But I still get the same kind of output format as previously: a massive <mgrwgt> block and 9 rwgt weights corresponding to the scale variations.

According to the shell output, the reweighting however works fine:

https://www.dropbox.com/s/h2gqv4hbil7fyrl/reweight_NLO.png?dl=0

Should we try with a different version of aMCatNLO? Currently we are using MG5_aMC_v2_4_0_beta.

Thanks,
Benedikt

Hi,

Indeed the code crashes in you case.
A normal run present like this:
> REWEIGHT: generating the square matrix element for reweighting
> REWEIGHT: generate p p > t t~ --no_warning=duplicate;define pert_QCD = -4 -3 -2 -1 1 2 3 4 21;add process p p > t t~ pert_QCD --no_warning=duplicate;
> REWEIGHT: Done 3.092
> REWEIGHT: When doing NLO reweighting, MG5aMC cannot use the loop reduction algorithms Golem and/or PJFry++
> REWEIGHT: generate p p > t t~ [ virt=QCD] ;
> REWEIGHT: Done 16.06
> REWEIGHT: starts to compute weight for events with the following modification to the param_card:
> REWEIGHT: set param_card sminputs 1 130.0 # orig: 132.507
>
> REWEIGHT: Event nb 0 current time: 10h31
> REWEIGHT: Event nb 10 31.1s
> REWEIGHT: Event nb 20 31.4s
> REWEIGHT: Event nb 30 33.9s
> …

I will check why the error message do not appear.
In principle, you should have a debug file created in your running directory with name ME5_debug.
Could you copy-paste the content of that file?

Cheers,

Olivier

> On May 10, 2016, at 10:17, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Hi Olivier,
>
> Thanks for the reply. I reran with the default reweight card which does
>
> launch
> set sminputs 1 130 # modify 1/alpha_EW
>
> But I still get the same kind of output format as previously: a massive
> <mgrwgt> block and 9 rwgt weights corresponding to the scale variations.
>
> According to the shell output, the reweighting however works fine:
>
> https://www.dropbox.com/s/h2gqv4hbil7fyrl/reweight_NLO.png?dl=0
>
> Should we try with a different version of aMCatNLO? Currently we are
> using MG5_aMC_v2_4_0_beta.
>
>
> Thanks,
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

Indeed from the ME5_debug file it looks like there is something wrong with the used python (which is external and which apparently works for normal event generation but not for the reweighting. (This python btw is what we atm are using for generating and creating gridpacks in CMS).

reweight run_02 -from_cards
Traceback (most recent call last):
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/ttbar_test/bin/internal/extended_cmd.py", line 1009, in onecmd
    return self.onecmd_orig(line, **opt)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/ttbar_test/bin/internal/extended_cmd.py", line 964, in onecmd_orig
    return func(arg, **opt)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/ttbar_test/bin/internal/common_run_interface.py", line 1179, in do_rewei
ght
    reweight_cmd.import_command_file(path)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/extended_cmd.py", line 1154, in import_command_file
    self.exec_cmd(line, precmd=True)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/extended_cmd.py", line 1035, in exec_cmd
    stop = Cmd.onecmd_orig(current_interface, line, **opt)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/extended_cmd.py", line 964, in onecmd_orig
    return func(arg, **opt)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/various/misc.py", line 162, in f_with_no_logger
    out = f(self, *args, **opt)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/reweight_interface.py", line 423, in do_launch
    self.create_standalone_directory()
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/various/misc.py", line 162, in f_with_no_logger
    out = f(self, *args, **opt)
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/reweight_interface.py", line 1464, in create_standalo
ne_directory
    mgcmd.options['lhapdf'], None, self.banner.run_card.get_lhapdf_id())
  File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/common_run_interface.py", line 2373, in install_lhapd
f_pdfset_static
    if os.path.exists(pjoin(pdfsets_dir, filename)):
  File "/cvmfs/cms.cern.ch/slc6_amd64_gcc481/cms/cmssw/CMSSW_7_1_20/external/slc6_amd64_gcc481/bin/../../../../../../external/python/2
.7.6/lib/python2.7/posixpath.py", line 75, in join
    if b.startswith('/'):
AttributeError: 'list' object has no attribute 'startswith'

Or am I misinterpreting the information above?

Thanks for your help,
Benedikt

Hi,

Your python version is perfectly fine.

This is actually a bug of the beta version which is already fixed.

Here is the associated patch:
I guess that just installing manually the pdf set should work as well but difficult to know for sure.

Cheers,

Olivier

=== modified file 'madgraph/various/banner.py'
--- madgraph/various/banner.py 2016-04-06 10:19:13 +0000
+++ madgraph/various/banner.py 2016-04-07 00:07:59 +0000
@@ -1544,7 +1544,11 @@

     def get_pdf_id(self, pdf):
         if pdf == "lhapdf":
- return self["lhaid"]
+ lhaid = self["lhaid"]
+ if isinstance(lhaid, list):
+ return lhaid[0]
+ else:
+ return lhaid
         else:
             return {'none': 0, 'mrs02nl':20250, 'mrs02nn':20270, 'cteq4_m': 19150,
                     'cteq4_l':19170, 'cteq4_d':19160, 'cteq5_m':19050,

> On May 10, 2016, at 12:17, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Hi Olivier,
>
> Indeed from the ME5_debug file it looks like there is something wrong
> with the used python (which is external and which apparently works for
> normal event generation but not for the reweighting. (This python btw is
> what we atm are using for generating and creating gridpacks in CMS).
>
>
> reweight run_02 -from_cards
> Traceback (most recent call last):
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/ttbar_test/bin/internal/extended_cmd.py", line 1009, in onecmd
> return self.onecmd_orig(line, **opt)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/ttbar_test/bin/internal/extended_cmd.py", line 964, in onecmd_orig
> return func(arg, **opt)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/ttbar_test/bin/internal/common_run_interface.py", line 1179, in do_rewei
> ght
> reweight_cmd.import_command_file(path)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/extended_cmd.py", line 1154, in import_command_file
> self.exec_cmd(line, precmd=True)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/extended_cmd.py", line 1035, in exec_cmd
> stop = Cmd.onecmd_orig(current_interface, line, **opt)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/extended_cmd.py", line 964, in onecmd_orig
> return func(arg, **opt)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/various/misc.py", line 162, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/reweight_interface.py", line 423, in do_launch
> self.create_standalone_directory()
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/various/misc.py", line 162, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/reweight_interface.py", line 1464, in create_standalo
> ne_directory
> mgcmd.options['lhapdf'], None, self.banner.run_card.get_lhapdf_id())
> File "/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0_beta/madgraph/interface/common_run_interface.py", line 2373, in install_lhapd
> f_pdfset_static
> if os.path.exists(pjoin(pdfsets_dir, filename)):
> File "/cvmfs/cms.cern.ch/slc6_amd64_gcc481/cms/cmssw/CMSSW_7_1_20/external/slc6_amd64_gcc481/bin/../../../../../../external/python/2
> .7.6/lib/python2.7/posixpath.py", line 75, in join
> if b.startswith('/'):
> AttributeError: 'list' object has no attribute 'startswith'
>
>
> Or am I misinterpreting the information above?
>
>
> Thanks for your help,
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

Thanks a lot for your support so far!

Two things:

I aplied your suggested changes in the banner.py, but unfortunately it is still not working ...

From what you wrote, I understood that in 2.4.1 it is fixed?
We downloaded it via bzr branch lp:~maddevelopers/mg5amcnlo/2.4.1 and did some first check without the reweighting, but even this one crashes with this ME_debug logfile:

https://www.dropbox.com/s/c110uiufo8l687u/ME_debug.txt?dl=0

Cheers,
Benedikt

Hi,

>> From what you wrote, I understood that in 2.4.1 it is fixed?

Actually I do not know if this was pushed in 2.4.1.
The 2.4.1 contains some code improvement which requires quite a lot of testing
before putting this branch even to beta. So I’m not surprised by the crashes (but thanks for reporting it, It will be useful).

The correct branch to use is the following one (confusing name but ok):
> lp:~maddevelopers/mg5amcnlo/2.3.4

> I aplied your suggested changes in the banner.py, but unfortunately it
> is still not working …

if this still happens with the above branch, could you put the debug file?

Thanks,

Olivier

> On May 10, 2016, at 15:57, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Hi Olivier,
>
> Thanks a lot for your support so far!
>
> Two things:
>
> I aplied your suggested changes in the banner.py, but unfortunately it
> is still not working ...
>
>> From what you wrote, I understood that in 2.4.1 it is fixed?
> We downloaded it via bzr branch lp:~maddevelopers/mg5amcnlo/2.4.1 and did some first check without the reweighting, but even this one crashes with this ME_debug logfile:
>
> https://www.dropbox.com/s/c110uiufo8l687u/ME_debug.txt?dl=0
>
>
> Cheers,
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

We tried it with 2.3.4, but it still crashes when compiling one of the SubProcesses:

https://www.dropbox.com/s/093o0fmkolt57ev/ME_debug_234.txt?dl=0

(more or less the same error as for 2.4.1)

Cheers,
Benedikt

Unfortunately I can not reproduce the problem. So the only way to help you is to ask you
to perform a lot of task to figure out what the problem is. It sounds related to your system in some way.

The problem seems that one file is missing:
maxparticles.inc
You should have this file in Source/maxparticles.inc
and symbolic link pointing to that file
in Subprocesses
Subprocesses/P0_gg_ttx/
Subprocesses/P0_uux_ttx/
Subprocesses/P0_uxu_ttx/

Can you check that all four of them are correctly existing and having the correct information in it?

Thanks,

Olivier

> On May 11, 2016, at 10:08, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Hi Olivier,
>
> We tried it with 2.3.4, but it still crashes when compiling one of the
> SubProcesses:
>
> https://www.dropbox.com/s/093o0fmkolt57ev/ME_debug_234.txt?dl=0
>
>
> (more or less the same error as for 2.4.1)
>
> Cheers,
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

The links in the SubProcesses/P0_* exist.

However, they are pointing to SubProcesses/maxparticles.inc, and this file is completely EMPTY (similar as the maxconfigs.inc).

The Source/maxparticles.inc has the following content:

      INTEGER MAX_PARTICLES, MAX_BRANCH
      PARAMETER (MAX_PARTICLES=5)
      PARAMETER (MAX_BRANCH=MAX_PARTICLES-1)

Thanks,
Benedikt

Dear Benedikt,

I went trough all the changes between 2.4.0.beta released version and the 2.3.4 branch
and no change seems related (even vaguely) with the handling of those files.

Could you check the following?

1) after the creation of the code (the output command) and before any run in that code. Could you check if the
SubProcesses/maxparticles.inc file is correctly a symbolic link to the Source/maxparticles.inc one?

2) could you double check that 2.4.0.beta is indeed working (sounds so weird).
One way to do this is to go inside the 2.3.4 branch and run the following command:
bzr revert -r 396
(actually if you do “bzr revert -r 397” correspond the 2.4.0.beta plus the patch that you tried before)

Sorry to not be very helpful so far, I’m really in the dark for this problem.

Cheers,

Olivier

> On May 11, 2016, at 14:22, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Hi Olivier,
>
>
> The links in the SubProcesses/P0_* exist.
>
> However, they are pointing to SubProcesses/maxparticles.inc, and this
> file is completely EMPTY (similar as the maxconfigs.inc).
>
> The Source/maxparticles.inc has the following content:
>
> INTEGER MAX_PARTICLES, MAX_BRANCH
> PARAMETER (MAX_PARTICLES=5)
> PARAMETER (MAX_BRANCH=MAX_PARTICLES-1)
>
> Thanks,
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Dear Olivier,

1) I can confirm that with 2.3.4, already after the output command (before any launching) the SubProcesses/maxparticles.inc is empty, i.e. there is NO symbolic link to the Source/maxparticles.inc

2) I have done checks with brand new installations of 2.3.3, 2.4.0.beta and 2.3.4.

2.3.3:
All the maxparticles.inc files in the SubProcesses have the correct symbolic link and are filled. Event generation of tt~ for 1000 events succeeds. The reweighting step however crashes with the following logfile

https://www.dropbox.com/s/laxu6oh0pl74hho/run_01_tag_1_debug.log.txt?dl=0

2.4.0.beta:
All the maxparticles.inc files in the SubProcesses have the correct symbolic link and are filled. Event generation of tt~ for 1000 events succeeds. The reweighting step does NOT crash, but the events.lhe.gz has this weird output format this thread started with.

2.3.4:
Only the Source/maxparticles.inc is filled. All the others are empty. Already the event generation crashes (with the error I linked three posts ago)

Can you make some sense of this behaviour? Which version did you use when you posted the

"> REWEIGHT: Event nb 0 current time: 10h31
> REWEIGHT: Event nb 10 31.1s
> REWEIGHT: Event nb 20 31.4s
> REWEIGHT: Event nb 30 33.9s
> …"

?

Thanks!
Benedikt

Hi Benedikt,

> The reweighting step however crashes with the following logfile
> https://www.dropbox.com/s/laxu6oh0pl74hho/run_01_tag_1_debug.log.txt?dl=0

The relevant information, is that this is a compilation problem of a missing library:
> cannot find -lpython2.7

Looks like this library is not always standard on linux.
if I’m correct this can be installed via
apt-get install python-dev
or
yum install python-devel

> 2.4.0.beta:
> All the maxparticles.inc files in the SubProcesses have the correct symbolic link and are filled. Event generation of tt~ for 1000 events succeeds. The reweighting step does NOT crash, but the events.lhe.gz has this weird output format this thread started with.

The crash occurs but the associated error message is muted for some reason.
The patch that I propose should solve the problem (then you should go back to the problem of 2.3.3).

> 2.3.4:
> Only the Source/maxparticles.inc is filled. All the others are empty. Already the event generation crashes (with the error I linked three posts ago)

This one does not make sense to me. 2.3.4 and 2.4.0.beta does not have any difference that I could (even vaguely) related to that stage of the code.
So I have basically no clue of what it can be.
One idea is that this is related in some way with the developer mode that is activate in 2.3.4 (since it comes from launchpad) and not in the two others (since, I guess, you are using the associated tarball for those). To test test, I have created the 2.4.0 tarball (which should in principle be released out of beta today):
madgraph.hep.uiuc.edu/Downloads/MG5_aMC_v2.4.0.tar.gz
If you can try we should be able to test such option.

Otherwise, the only way for me to figure it out is if you would be kind enough to search which build of MG5 introduces such effect.
This is a bit boring, since it correspond to do this inside the 2.3.4 version
1) get your current bzr version
bzr revno
(this will return on screen a given number. Depending of the exact time you load that version it should be around 404

2) change it older version
bzr revert -r XXX
./bin/mg5 # and test if the output command is working or not.

where XXX should be between 396 and your actual current version (404)
396 being the version 2.4.0.beta which we know is working.
With bissection method, three try should be enough to figure which version introduces such problem.

> Can you make some sense of this behaviour? Which version did you use when you posted the
>
> "> REWEIGHT: Event nb 0 current time: 10h31
>> REWEIGHT: Event nb 10 31.1s
>> REWEIGHT: Event nb 20 31.4s
>> REWEIGHT: Event nb 30 33.9s

It was 2.3.4.

Cheers,

Olivier

> On May 12, 2016, at 11:47, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Dear Olivier,
>
>
> 1) I can confirm that with 2.3.4, already after the output command (before any launching) the SubProcesses/maxparticles.inc is empty, i.e. there is NO symbolic link to the Source/maxparticles.inc
>
> 2) I have done checks with brand new installations of 2.3.3, 2.4.0.beta
> and 2.3.4.
>
> 2.3.3:
> All the maxparticles.inc files in the SubProcesses have the correct symbolic link and are filled. Event generation of tt~ for 1000 events succeeds. The reweighting step however crashes with the following logfile
>
> https://www.dropbox.com/s/laxu6oh0pl74hho/run_01_tag_1_debug.log.txt?dl=0
>
> 2.4.0.beta:
> All the maxparticles.inc files in the SubProcesses have the correct symbolic link and are filled. Event generation of tt~ for 1000 events succeeds. The reweighting step does NOT crash, but the events.lhe.gz has this weird output format this thread started with.
>
> 2.3.4:
> Only the Source/maxparticles.inc is filled. All the others are empty. Already the event generation crashes (with the error I linked three posts ago)
>
>
> Can you make some sense of this behaviour? Which version did you use when you posted the
>
> "> REWEIGHT: Event nb 0 current time: 10h31
>> REWEIGHT: Event nb 10 31.1s
>> REWEIGHT: Event nb 20 31.4s
>> REWEIGHT: Event nb 30 33.9s
>> …"
>
> ?
>
>
> Thanks!
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Dear Olivier,

Indeed the thing about the tarball vs. the bzr development versions is what I had realized as well previously. Now I checked with the newly available 2.4.0.tar.gz

I could manage to do the following:
In the first run, it looked again like the initial problem that the reweighting doesnt throw an error but the events.lhe.gz has the weird output format.

Then somebody from my institute who had a similar problem told me to do:

export LIBRARY_PATH=$LD_LIBRARY_PATH

and I reran the generation. And it seems to work "a bit more"! (My colleague had worked with the LO reweighting a few months ago, so he had invested quite some time to make the reweighting work within the environment typically used in CMS). So now the screen output looks like

REWEIGHT: starts to compute weight for events with the following modification to the param_card:
REWEIGHT: set param_card sminputs 1 130.0 # orig: 132.507

REWEIGHT: Event nb 0 0.04s
REWEIGHT: Event nb 10 1m 0s
REWEIGHT: Event nb 20 1m 0s
REWEIGHT: Event nb 30 1m 13s
REWEIGHT: Event nb 40 1m 22s
REWEIGHT: Event nb 50 1m 22s
REWEIGHT: Event nb 60 1m 22s
REWEIGHT: Event nb 70 1m 27s
REWEIGHT: Event nb 80 1m 27s
REWEIGHT: Event nb 90 1m 28s
REWEIGHT: Event nb 100 1m 28s
CRITICAL: fail compilation [reweight_interface.py at line 1070]
REWEIGHT: Original cross-section: 647.49959 +- 4.0568632 pb

So it really loops over the events now, but apparently at some event it crashes.

The debug log can be found here:
https://www.dropbox.com/s/1tlfscn8ntgt4lh/run_02_tag_2_debug.log.txt?dl=0

Thanks,
Benedikt

Dear Benedikt,

Looks like either a compilation problem and/or a pure linking problem (I think that this is the second problem).
It is not clear which process fail to be compiled and/or linked.
Could you apply the following patch and rerun:

=== modified file 'madgraph/interface/reweight_interface.py'
--- madgraph/interface/reweight_interface.py 2016-03-31 10:53:12 +0000
+++ madgraph/interface/reweight_interface.py 2016-05-18 15:01:29 +0000
@@ -1067,7 +1067,7 @@
                         os.system('install_name_tool -change libMadLoop.dylib %s/libMadLoop.dylib matrix%spy.so' % (Pdir,2*metag))
                         mymod = __import__('%s.SubProcesses.%s.matrix%spy' % (base, Pname, 2*metag), globals(), locals(), [],-1)
                     else:
- misc.sprint("fail compilation")
+ misc.sprint("fail compilation for '%s.SubProcesses.%s.matrix%spy" % (base, Pname, 2*metag))
                         raise
                 S = mymod.SubProcesses
                 P = getattr(S, Pname)

This should indicate to us which fortran code can not be compiled and/or linked.

Thanks,

Olivier

> On May 17, 2016, at 11:26, Benedikt Maier <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Status: Answered => Open
>
> Benedikt Maier is still having a problem:
> Dear Olivier,
>
> Indeed the thing about the tarball vs. the bzr development versions is
> what I had realized as well previously. Now I checked with the newly
> available 2.4.0.tar.gz
>
> I could manage to do the following:
> In the first run, it looked again like the initial problem that the reweighting doesnt throw an error but the events.lhe.gz has the weird output format.
>
> Then somebody from my institute who had a similar problem told me to do:
>
> export LIBRARY_PATH=$LD_LIBRARY_PATH
>
> and I reran the generation. And it seems to work "a bit more"! (My
> colleague had worked with the LO reweighting a few months ago, so he had
> invested quite some time to make the reweighting work within the
> environment typically used in CMS). So now the screen output looks like
>
>
> REWEIGHT: starts to compute weight for events with the following modification to the param_card:
> REWEIGHT: set param_card sminputs 1 130.0 # orig: 132.507
>
> REWEIGHT: Event nb 0 0.04s
> REWEIGHT: Event nb 10 1m 0s
> REWEIGHT: Event nb 20 1m 0s
> REWEIGHT: Event nb 30 1m 13s
> REWEIGHT: Event nb 40 1m 22s
> REWEIGHT: Event nb 50 1m 22s
> REWEIGHT: Event nb 60 1m 22s
> REWEIGHT: Event nb 70 1m 27s
> REWEIGHT: Event nb 80 1m 27s
> REWEIGHT: Event nb 90 1m 28s
> REWEIGHT: Event nb 100 1m 28s
> CRITICAL: fail compilation [reweight_interface.py at line 1070]
> REWEIGHT: Original cross-section: 647.49959 +- 4.0568632 pb
>
>
> So it really loops over the events now, but apparently at some event it crashes.
>
> The debug log can be found here:
> https://www.dropbox.com/s/1tlfscn8ntgt4lh/run_02_tag_2_debug.log.txt?dl=0
>
>
> Thanks,
> Benedikt
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Dear Olivier,

We changed the reweight_interface.py accordingly. Here is what it prints (btw it is a new run, so it didnt crash at the same event):

REWEIGHT: Event nb 300 56.9s
CRITICAL: fail compilation for 'rw_mevirt.SubProcesses.P4_ssx_ttx.matrix2py [reweight_interface.py at line 1070]
REWEIGHT: Original cross-section: 655.26214 +- 3.996254 pb
REWEIGHT: Computed cross-section:
REWEIGHT: gzipping output file: events.lhe

Here is the debug logfile:

https://www.dropbox.com/s/iskhh58bvjhyx9u/run_01_tag_2_debug.log.txt?dl=0

Thanks a lot for your help,
Kevin

ok then we need to investigate the directory:
> rw_mevirt/SubProcesses/P4_ssx_ttx

in that directory if you try:
python
then type
import matrix2py

You should get the error reported before.
you can then try to remove matrix2py.so
and recompile the code with
make matrix2py.so
(and retry the python linking step just in case)
I hope the log of this compilation can help us to understand what the problem is.

Cheers,

Olivier

> On May 23, 2016, at 13:52, Kevin Floeh <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Kevin Floeh posted a new comment:
> Dear Olivier,
>
>
> We changed the reweight_interface.py accordingly. Here is what it prints (btw it is a new run, so it didnt crash at the same event):
>
> REWEIGHT: Event nb 300 56.9s
> CRITICAL: fail compilation for 'rw_mevirt.SubProcesses.P4_ssx_ttx.matrix2py [reweight_interface.py at line 1070]
> REWEIGHT: Original cross-section: 655.26214 +- 3.996254 pb
> REWEIGHT: Computed cross-section:
> REWEIGHT: gzipping output file: events.lhe
>
>
> Here is the debug logfile:
>
> https://www.dropbox.com/s/iskhh58bvjhyx9u/run_01_tag_2_debug.log.txt?dl=0
>
> Thanks a lot for your help,
> Kevin
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

when I start python and import matrix2py I get the following error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named matrix2py

when I recompile the code I get:

cd ../SubProcesses; make OLP_static
make[1]: Entering directory `/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0/ttbar_test/rw_mevirt/SubProcesses'
ar rcs libMadLoop.a MadLoopParamReader.o MadLoopCommons.o P0_gg_ttx/polynomial.o P1_uux_ttx/polynomial.o P2_ccx_ttx/polynomial.o P3_ddx_ttx/polynomial.o P4_ssx_ttx/polynomial.o P0_gg_ttx/loop_matrix.o P1_uux_ttx/loop_matrix.o P2_ccx_ttx/loop_matrix.o P3_ddx_ttx/loop_matrix.o P4_ssx_ttx/loop_matrix.o P0_gg_ttx/improve_ps.o P1_uux_ttx/improve_ps.o P2_ccx_ttx/improve_ps.o P3_ddx_ttx/improve_ps.o P4_ssx_ttx/improve_ps.o P0_gg_ttx/born_matrix.o P1_uux_ttx/born_matrix.o P2_ccx_ttx/born_matrix.o P3_ddx_ttx/born_matrix.o P4_ssx_ttx/born_matrix.o P0_gg_ttx/CT_interface.o P1_uux_ttx/CT_interface.o P2_ccx_ttx/CT_interface.o P3_ddx_ttx/CT_interface.o P4_ssx_ttx/CT_interface.o P0_gg_ttx/loop_num.o P1_uux_ttx/loop_num.o P2_ccx_ttx/loop_num.o P3_ddx_ttx/loop_num.o P4_ssx_ttx/loop_num.o P0_gg_ttx/helas_calls_ampb_1.o P0_gg_ttx/helas_calls_uvct_1.o P1_uux_ttx/helas_calls_ampb_1.o P1_uux_ttx/helas_calls_uvct_1.o P2_ccx_ttx/helas_calls_ampb_1.o P2_ccx_ttx/helas_calls_uvct_1.o P3_ddx_ttx/helas_calls_ampb_1.o P3_ddx_ttx/helas_calls_uvct_1.o P4_ssx_ttx/helas_calls_ampb_1.o P4_ssx_ttx/helas_calls_uvct_1.o P0_gg_ttx/mp_compute_loop_coefs.o P1_uux_ttx/mp_compute_loop_coefs.o P2_ccx_ttx/mp_compute_loop_coefs.o P3_ddx_ttx/mp_compute_loop_coefs.o P4_ssx_ttx/mp_compute_loop_coefs.o P0_gg_ttx/mp_helas_calls_ampb_1.o P0_gg_ttx/mp_helas_calls_uvct_1.o P1_uux_ttx/mp_helas_calls_ampb_1.o P1_uux_ttx/mp_helas_calls_uvct_1.o P2_ccx_ttx/mp_helas_calls_ampb_1.o P2_ccx_ttx/mp_helas_calls_uvct_1.o P3_ddx_ttx/mp_helas_calls_ampb_1.o P3_ddx_ttx/mp_helas_calls_uvct_1.o P4_ssx_ttx/mp_helas_calls_ampb_1.o P4_ssx_ttx/mp_helas_calls_uvct_1.o P0_gg_ttx/coef_construction_1.o P1_uux_ttx/coef_construction_1.o P2_ccx_ttx/coef_construction_1.o P3_ddx_ttx/coef_construction_1.o P4_ssx_ttx/coef_construction_1.o P0_gg_ttx/loop_CT_calls_1.o P1_uux_ttx/loop_CT_calls_1.o P2_ccx_ttx/loop_CT_calls_1.o P3_ddx_ttx/loop_CT_calls_1.o P4_ssx_ttx/loop_CT_calls_1.o P0_gg_ttx/mp_coef_construction_1.o P1_uux_ttx/mp_coef_construction_1.o P2_ccx_ttx/mp_coef_construction_1.o P3_ddx_ttx/mp_coef_construction_1.o P4_ssx_ttx/mp_coef_construction_1.o P0_gg_ttx/TIR_interface.o P1_uux_ttx/TIR_interface.o P2_ccx_ttx/TIR_interface.o P3_ddx_ttx/TIR_interface.o P4_ssx_ttx/TIR_interface.o
mv libMadLoop.a ../lib/libMadLoop.a
make[1]: Leaving directory `/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0/ttbar_test/rw_mevirt/SubProcesses'
make: *** No rule to make target `f2py_wrapper.f', needed by `matrix2py.so'. Stop.

Cheers,

Kevin

Hi,

Sorry for the slow answer, did not had the time to look at it before.
The problem seems that you do not have the file f2py_wrapper.f
in that directory (you should).

In my case, If I do a “ls” in that directory, I have all the following files:
> CT_interface.f born_matrix.o f2py_wrapper.f loop_matrix.ps mp_coupl.inc param.log
> CT_interface.o born_matrix.ps helas_calls_ampb_1.f loop_max_coefs.inc mp_coupl_same_name.inc pmass.inc
> MadLoop5_resources check_sa.f helas_calls_ampb_1.o loop_num.f mp_helas_calls_ampb_1.f polynomial.f
> MadLoopCommons.f check_sa.py helas_calls_uvct_1.f loop_num.o mp_helas_calls_ampb_1.o polynomial.o
> MadLoopParamReader.f check_sa_born_splitOrders.f helas_calls_uvct_1.o makefile mp_helas_calls_uvct_1.f proc_prefix.txt
> MadLoopParams.inc coef_construction_1.f improve_ps.f matrix2py.so mp_helas_calls_uvct_1.o process_info.inc
> TIR_interface.f coef_construction_1.o improve_ps.o matrix3py.so mpmodule.mod
> TIR_interface.o coef_specs.inc loop_CT_calls_1.f mp_coef_construction_1.f nexternal.inc
> __init__.py coupl.inc loop_CT_calls_1.o mp_coef_construction_1.o ngraphs.inc
> __init__.pyc cts_mpc.h loop_matrix.f mp_compute_loop_coefs.f nsqso_born.inc
> born_matrix.f cts_mprec.h loop_matrix.o mp_compute_loop_coefs.o nsquaredSO.inc

Can you check that you have all the .f files as I do.
(Just to check if you misses some other file than f2py_wrapper.f)

That file is tailored to the matrix-element and therefore created specifically for that directory.
The only reason that I can see so far is that for some reason, you are automatically switch
to the not loop-optimized output mode of the loop computation which indeed do not create such file.
I temporarily assign this to Valentin which might have an idea why you suddenly switch in that mode (I have no idea)

Cheers,

Olivier

> On Jun 1, 2016, at 13:27, Kevin Floeh <email address hidden> wrote:
>
> Question #293529 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/293529
>
> Kevin Floeh posted a new comment:
> Hi Olivier,
>
> when I start python and import matrix2py I get the following error
> message:
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named matrix2py
>
> when I recompile the code I get:
>
> cd ../SubProcesses; make OLP_static
> make[1]: Entering directory `/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0/ttbar_test/rw_mevirt/SubProcesses'
> ar rcs libMadLoop.a MadLoopParamReader.o MadLoopCommons.o P0_gg_ttx/polynomial.o P1_uux_ttx/polynomial.o P2_ccx_ttx/polynomial.o P3_ddx_ttx/polynomial.o P4_ssx_ttx/polynomial.o P0_gg_ttx/loop_matrix.o P1_uux_ttx/loop_matrix.o P2_ccx_ttx/loop_matrix.o P3_ddx_ttx/loop_matrix.o P4_ssx_ttx/loop_matrix.o P0_gg_ttx/improve_ps.o P1_uux_ttx/improve_ps.o P2_ccx_ttx/improve_ps.o P3_ddx_ttx/improve_ps.o P4_ssx_ttx/improve_ps.o P0_gg_ttx/born_matrix.o P1_uux_ttx/born_matrix.o P2_ccx_ttx/born_matrix.o P3_ddx_ttx/born_matrix.o P4_ssx_ttx/born_matrix.o P0_gg_ttx/CT_interface.o P1_uux_ttx/CT_interface.o P2_ccx_ttx/CT_interface.o P3_ddx_ttx/CT_interface.o P4_ssx_ttx/CT_interface.o P0_gg_ttx/loop_num.o P1_uux_ttx/loop_num.o P2_ccx_ttx/loop_num.o P3_ddx_ttx/loop_num.o P4_ssx_ttx/loop_num.o P0_gg_ttx/helas_calls_ampb_1.o P0_gg_ttx/helas_calls_uvct_1.o P1_uux_ttx/helas_calls_ampb_1.o P1_uux_ttx/helas_calls_uvct_1.o P2_ccx_ttx/helas_calls_ampb_1.o P2_ccx_ttx/helas_calls_uvct_1.o P3_ddx_ttx/helas_calls_ampb_1.o P3_ddx_ttx/helas_calls_uvct_1.o P4_ssx_ttx/helas_calls_ampb_1.o P4_ssx_ttx/helas_calls_uvct_1.o P0_gg_ttx/mp_compute_loop_coefs.o P1_uux_ttx/mp_compute_loop_coefs.o P2_ccx_ttx/mp_compute_loop_coefs.o P3_ddx_ttx/mp_compute_loop_coefs.o P4_ssx_ttx/mp_compute_loop_coefs.o P0_gg_ttx/mp_helas_calls_ampb_1.o P0_gg_ttx/mp_helas_calls_uvct_1.o P1_uux_ttx/mp_helas_calls_ampb_1.o P1_uux_ttx/mp_helas_calls_uvct_1.o P2_ccx_ttx/mp_helas_calls_ampb_1.o P2_ccx_ttx/mp_helas_calls_uvct_1.o P3_ddx_ttx/mp_helas_calls_ampb_1.o P3_ddx_ttx/mp_helas_calls_uvct_1.o P4_ssx_ttx/mp_helas_calls_ampb_1.o P4_ssx_ttx/mp_helas_calls_uvct_1.o P0_gg_ttx/coef_construction_1.o P1_uux_ttx/coef_construction_1.o P2_ccx_ttx/coef_construction_1.o P3_ddx_ttx/coef_construction_1.o P4_ssx_ttx/coef_construction_1.o P0_gg_ttx/loop_CT_calls_1.o P1_uux_ttx/loop_CT_calls_1.o P2_ccx_ttx/loop_CT_calls_1.o P3_ddx_ttx/loop_CT_calls_1.o P4_ssx_ttx/loop_CT_calls_1.o P0_gg_ttx/mp_coef_construction_1.o P1_uux_ttx/mp_coef_construction_1.o P2_ccx_ttx/mp_coef_construction_1.o P3_ddx_ttx/mp_coef_construction_1.o P4_ssx_ttx/mp_coef_construction_1.o P0_gg_ttx/TIR_interface.o P1_uux_ttx/TIR_interface.o P2_ccx_ttx/TIR_interface.o P3_ddx_ttx/TIR_interface.o P4_ssx_ttx/TIR_interface.o
> mv libMadLoop.a ../lib/libMadLoop.a
> make[1]: Leaving directory `/afs/cern.ch/work/k/kfloeh/public/MG5_aMC_v2_4_0/ttbar_test/rw_mevirt/SubProcesses'
> make: *** No rule to make target `f2py_wrapper.f', needed by `matrix2py.so'. Stop.
>
>
> Cheers,
>
> Kevin
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi,

I accidentally ran the make comment in the wrong folder. Here is the correct error message:

https://www.dropbox.com/s/hiay2qcfbv4zon1/make_matrix2pyerror.txt?dl=0

in the rw_mevirt/SubProcesses/P4_ssx_ttx directory only

MadLoop5_resources check_sa.f

is missing.

Cheers,

Kevin

Hi Kevin,

I still see the following lines:

/cvmfs/cms.cern.ch/slc6_amd64_gcc481/external/gcc/4.8.1/bin/../lib/gcc/x86_64-redhat-linux-gnu/4.8.1/../../../../x86_64-redhat-linux-gnu/bin/ld: cannot find -lpython2.7
collect2: error: ld returned 1 exit status

I'm confused since I discuss how to get rid of those at the beginning of this thread.
Did you check that you have installed python-dev?

Cheers,

Olivier