Helas out of Memory for many leg final state

Hi Zach,

Currently the limit is indeed at p p > 5 jets.
But the reason of this limitation is not really the generation of the process but rather the compilation of the output files.
(even if you didn't reach that point)

The reason is of course linked to factorial growth in the number of diagram in the number of particles in the final states.

Part of the MG team is currently looking on relation recursion in order to generate/evaluate such processes.
More precisely, we are at the stage to optimize the associated phase-space integrator (But this is far to be trivial).

Cheers,

Olivier

On Nov 6, 2012, at 5:21 AM, Zachary Marshall <email address hidden> wrote:

> New question #213426 on MadGraph5:
> https://answers.launchpad.net/madgraph5/+question/213426
>
> Hi,
>
> I'm trying to push QCD multijet generation as far as it will go, and at the moment I have in MG 1.5.2:
>
> generate p p > j j @1
> add process p p > j j j @2
> add process p p > j j j j @3
> add process p p > j j j j j @4
> add process p p > j j j j j j @5
>
> This thing eventually crashes out with a memory error:
>
> File "/afs/cern.ch/atlas/offline/external/madgraph/MadGraph5_v1_5_2/madgraph/iolibs/group_subprocs.py", line 203, in generate_matrix_elements
> generate_matrix_elements(amplitudes))
> File "/afs/cern.ch/atlas/offline/external/madgraph/MadGraph5_v1_5_2/madgraph/core/helas_objects.py", line 4460, in generate_matrix_elements
> col_matrix = color_amp.ColorMatrix(col_basis)
> File "/afs/cern.ch/atlas/offline/external/madgraph/MadGraph5_v1_5_2/madgraph/core/color_amp.py", line 542, in __init__
> self.build_matrix(Nc, Nc_power_min, Nc_power_max, is_symmetric=True)
> File "/afs/cern.ch/atlas/offline/external/madgraph/MadGraph5_v1_5_2/madgraph/core/color_amp.py", line 587, in build_matrix
> self[(i1, i2)] = result
> MemoryError
>
> The standard output stream shows:
>
> Total: 1567 processes with 1392128 diagrams
> output -f
> INFO: initialize a new directory: PROC_sm_0
> INFO: remove old information in PROC_sm_0
> INFO: Organizing processes into subprocess groups
> INFO: Generating Helas calls for process: g g > g g g g g g WEIGHTED=6 @5
> Command "import /tmp/zmarshal/proc_card_mg5.dat" interrupted in sub-command:
> "output -f" with error:
> MemoryError :
>
> So ok, 1.4M diagrams in 1550 processes - 1.3M of which (775 of the processes) come from the six jet line. And the crash out is actually on the six gluon final state. Seems like this tells me that MadGraph in its current version will not do six jet final states.
>
> Is that the right conclusion, or is there something further I can do to help this behave?
>
>
> Thanks,
> Zach
>
> --
> You received this question notification because you are a member of
> MadTeam, which is an answer contact for MadGraph5.

Dear Olivier,

Ah, now I see the compilation problem, in fact. But I get an error already with gg>ggggg (below).

I'm using this version of gfortran:

/afs/cern.ch/sw/lcg/contrib/gcc/4.3.5/x86_64-slc5-gcc43-opt/bin/gfortran

from your message it seems like this should work if I have the right compiler options enabled? Or is this really borderline, and should I be giving up on doing 5 jet final states as well?

Thanks,
Zach

Working on SubProcesses
    P4_gg_ggggg
Error detected in "generate_events -f Test"
write debug file /tmp/zmarshal/test0/PROC_sm_0/Test_fermi_debug.log
If you need help with this issue please contact us on https://answers.launchpad.net/madgraph5
MadGraph5Error : A compilation Error occurs when trying to compile /tmp/zmarshal/test0/PROC_sm_0/SubProcesses/P4_gg_ggggg.
        The compilation fails with the following output message:
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o symmetry.o symmetry.f
            touch qmass.inc
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o setcuts.o setcuts.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o cuts.o cuts.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o cluster.o cluster.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o myamp.o myamp.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o genps.o genps.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o initcluster.o initcluster.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o setscales.o setscales.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o reweight.o reweight.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o get_color.o get_color.f
            gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o matrix1.o matrix1.f

            f951: out of memory allocating 139272 bytes after a total of 35012902912 bytes
            make: *** [matrix1.o] Error 1

        Please try to fix this compilations issue and retry.
        Help might be found at https://answers.launchpad.net/madgraph5.
        If you think that this is a bug, you can report this at https://bugs.launchpad.net/madgraph5
quit

This is probably border line but I compile it yesterday on my laptop and no problem at all.
( with gcc 4.6 and 16Go de RAM).
(note that I run the standalone output and didn't try the madevent output)

Maybe you can try with gcc4.6.
(I know that either gcc4.6 or gcc4.7 is available on the cern cluster
you can find instruction here:
http://amcatnlo.web.cern.ch/amcatnlo/list.htm)

Cheers,

Olivier

On Nov 7, 2012, at 11:16 AM, Zachary Marshall <email address hidden> wrote:

> Question #213426 on MadGraph5 changed:
> https://answers.launchpad.net/madgraph5/+question/213426
>
> Status: Answered => Open
>
> Zachary Marshall is still having a problem:
> Dear Olivier,
>
> Ah, now I see the compilation problem, in fact. But I get an error
> already with gg>ggggg (below).
>
> I'm using this version of gfortran:
>
> /afs/cern.ch/sw/lcg/contrib/gcc/4.3.5/x86_64-slc5-gcc43-opt/bin/gfortran
>
> from your message it seems like this should work if I have the right
> compiler options enabled? Or is this really borderline, and should I be
> giving up on doing 5 jet final states as well?
>
>
> Thanks,
> Zach
>
>
> Working on SubProcesses
> P4_gg_ggggg
> Error detected in "generate_events -f Test"
> write debug file /tmp/zmarshal/test0/PROC_sm_0/Test_fermi_debug.log
> If you need help with this issue please contact us on https://answers.launchpad.net/madgraph5
> MadGraph5Error : A compilation Error occurs when trying to compile /tmp/zmarshal/test0/PROC_sm_0/SubProcesses/P4_gg_ggggg.
> The compilation fails with the following output message:
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o symmetry.o symmetry.f
> touch qmass.inc
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o setcuts.o setcuts.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o cuts.o cuts.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o cluster.o cluster.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o myamp.o myamp.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o genps.o genps.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o initcluster.o initcluster.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o setscales.o setscales.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o reweight.o reweight.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o get_color.o get_color.f
> gfortran -O -w -ffixed-line-length-132 -m32 -w -c -o matrix1.o matrix1.f
>
> f951: out of memory allocating 139272 bytes after a total of 35012902912 bytes
> make: *** [matrix1.o] Error 1
>
> Please try to fix this compilations issue and retry.
> Help might be found at https://answers.launchpad.net/madgraph5.
> If you think that this is a bug, you can report this at https://bugs.launchpad.net/madgraph5
> quit
>
> --
> You received this question notification because you are a member of
> MadTeam, which is an answer contact for MadGraph5.

Hi Olivier,

Ok, yup, I get out of memory errors with 4.3 (32- and 64-bit). I don't get the same errors with 4.6 or 4.7, but we are (unfortunately) unable to arbitrarily choose the compiler for production running. Is there anything you can suggest, or should I just use 4-jet modes until we have 4.6 / 4.7 in production?

Thanks again,
Zach

Hi Zach,

I don't know how much "freedom" you have.

But maybe you can add the label
-mcmodel=medium
in the file Source/make_opts
of the generated process.

This might solves the problem for 4.3 (I'm not 100% sure)

Cheers,

Olivier

On Nov 8, 2012, at 12:11 PM, Zachary Marshall <email address hidden> wrote:

> Question #213426 on MadGraph5 changed:
> https://answers.launchpad.net/madgraph5/+question/213426
>
> Zachary Marshall posted a new comment:
> Hi Olivier,
>
> Ok, yup, I get out of memory errors with 4.3 (32- and 64-bit). I don't
> get the same errors with 4.6 or 4.7, but we are (unfortunately) unable
> to arbitrarily choose the compiler for production running. Is there
> anything you can suggest, or should I just use 4-jet modes until we have
> 4.6 / 4.7 in production?
>
> Thanks again,
> Zach
>
> --
> You received this question notification because you are a member of
> MadTeam, which is an answer contact for MadGraph5.

Hi Olivier,

For 32-bit compilation, that option isn't available, of course (just a note for anybody who runs across this later).

Unfortunately, for 64-bit compilation it still runs out of memory...

gfortran -O -w -ffixed-line-length-132 -mcmodel=medium -w -c -o matrix1.o matrix1.f

f951: out of memory allocating 14542320920 bytes after a total of 1933369344 bytes
gmake: *** [matrix1.o] Error 1

Best,
Zach

Hi zach (cc yoshitaro),

Thanks for the information.

So yes in this case you should limit yourself to 4 j for the time being, waiting that our projects using recursion is fully working.
(Except if someone else has an bright idea)

Cheers,

Olivier

On Nov 8, 2012, at 12:36 PM, Zachary Marshall <email address hidden> wrote:

> Question #213426 on MadGraph5 changed:
> https://answers.launchpad.net/madgraph5/+question/213426
>
> Zachary Marshall posted a new comment:
> Hi Olivier,
>
> For 32-bit compilation, that option isn't available, of course (just a
> note for anybody who runs across this later).
>
> Unfortunately, for 64-bit compilation it still runs out of memory...
>
>
> gfortran -O -w -ffixed-line-length-132 -mcmodel=medium -w -c -o matrix1.o matrix1.f
>
> f951: out of memory allocating 14542320920 bytes after a total of 1933369344 bytes
> gmake: *** [matrix1.o] Error 1
>
>
> Best,
> Zach
>
> --
> You received this question notification because you are a member of
> MadTeam, which is an answer contact for MadGraph5.

Ok, thanks Olivier!

Best,
Zach