MadGraph5_aMC@NLO

Number of files in a gridpack

Asked by Louis Helary

Dear all,

In ATLAS we are trying to generate LO signal samples of a mono-s(WW) search in the Dark Higgs scenario using gridpacks. The problem we are facing is that in some of our gridpacks we have about 1M files which represents too much files for the grid worker nodes to work with.

Most of the files are in each sub process directories:
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/run_01_log.txt
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/input_app.txt
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/run1_app.log
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/results.dat
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/default_log.txt
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/run_01_results.dat
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/default_results.dat

So the question I have is what files are really needed to generate events, and which ones could we remove?

Thanks in advance,

Best,

Louis

Question information

Language:
English Edit question
Status:
Answered
For:
MadGraph5_aMC@NLO Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Olivier Mattelaer (olivier-mattelaer) said : 		#1

Did you use the script that we provide to clean the gridpack of un-necessary files?

In addition, I believe that everything starting with run_01 can be removed as well as all the log file.
I would keep this file for sure:
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/default_results.dat

Those two might not be 100% needed but this need to be investigate:
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/input_app.txt
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/results.dat

You should also have a file which is important to keep that i do not see in your list:
 madevent/SubProcesses/P1_gs_n1n1hss_hs_wpwm_wp_cdx_wm_scx/G7.206/default_ftn26.gz

Cheers,

Olivier

Revision history for this message
Louis Helary (louis-helary) said : 		#2

Hi Olivier,

Thanks for your answer. Yes the script to clean the grid pack is run by default when we produce the gridpack, and it does reduce the number of files a bit but not by a large amount.

Alright so we will remove all the run_01 and log files, and make some tests to understand whether we can keep input_app.txt and results.dat.

For the last file you are mentioning default_ftn26.gz I see it in some directories but not all of them. Is this expected?

I propose to leave this ticket open for now and I'll update it with whether we can remove input_app.txt and results.dat or not.

Best,

Louis

Revision history for this message
Olivier Mattelaer (olivier-mattelaer) said : 		#3

Hi,

Since it seems that you have conflicting Breit-Wigner, it is possible that some channel of integration are indeed zero and in that case it makes sense that they are no phase-space integration grid.

Cheers,

Olivier

Revision history for this message
Louis Helary (louis-helary) said : 		#4

Hi Olivier,

Does that mean that we can also get rid of the sub-process directories that do not contain a default_ftn26.gz, which I guess is the phase-space integration grid?

Best,

Louis

Revision history for this message
Olivier Mattelaer (olivier-mattelaer) said : 		#5

I will not move in that direction.

Olivier

> On 23 Nov 2020, at 09:30, Louis Helary <email address hidden> wrote:
>
> Question #694071 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/694071
>
> Louis Helary posted a new comment:
> Hi Olivier,
>
> Does that mean that we can also get rid of the sub-process directories
> that do not contain a default_ftn26.gz, which I guess is the phase-space
> integration grid?
>
> Best,
>
> Louis
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Revision history for this message
Louis Helary (louis-helary) said : 		#6

Ok thanks for your guidance.

I'll make more tests and report back what we think can be removed.

Best,

Louis

Can you help with this problem?

Provide an answer of your own, or ask Louis Helary for more information if necessary.

To post a message you must log in.