Read-only version of model download

Hi Zach,

If you have in your PYTHONPATH a directory containing the three letter UFO.
(in my laptop I use UFOMODEL)
then the auto-download of the model will use that directory instead of the local directory.

I guess that this should be enough for you, isn't it?

Cheers,

Olivier

> On 9 Jul 2018, at 16:22, Zachary Marshall <email address hidden> wrote:
>
> New question #670770 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/670770
>
> In new MadGraph versions, it's possible to download and install models that are requested that were missing. I think that's great! When we have an installation in a read-only area, though, it seems to fail. The solution would be to download the model files locally or to a .madgraph directory and then add that directory to PYTHONPATH, so that the model would be picked up by the normal infrastructure. Could such functionality be added, please?
>
> Thanks,
> Zach
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

That seems to me a rather dangerous solution... if I have some other software that happens to be in my path and has "UFO" in the name, MadGraph will happily start downloading models into it? Why not go with something more controlled, with a user override option? Or even a "LocalUFOInstallArea" option in the config, which we could set to "." ?

Best,
Zach

Hi,

On server, you should anyway use lmod or modules to keep your environment variable clean.
And therefore such issue should not happen.
On laptop, this is quite rare people using PYTHONPATH such that we do not really need to worry about it.
(and typically under control)

I can see two bad scenario:
- the file are put in a weird place
- the file failed to be written since the wrong place is in RO

Both are not that bad to my point of view. So this is at least not a priority.

Actually, If we want to make something more controlled, then I would skip the idea of using PYTHONPATH
since this is not really well controlled to my point of view.

Cheers,

Olivier

> On 9 Jul 2018, at 23:37, Zachary Marshall <email address hidden> wrote:
>
> Question #670770 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/670770
>
> Status: Answered => Open
>
> Zachary Marshall is still having a problem:
> Hi Olivier,
>
> That seems to me a rather dangerous solution... if I have some other
> software that happens to be in my path and has "UFO" in the name,
> MadGraph will happily start downloading models into it? Why not go with
> something more controlled, with a user override option? Or even a
> "LocalUFOInstallArea" option in the config, which we could set to "." ?
>
> Best,
> Zach
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi Olivier,

If the files are put in a weird place the user may literally be unable to find them, which does seem pretty bad to me (to say nothing of just writing files into some directory on the user's disk without asking first). If the files fail to be written, the job crashes, so that's not great either.

Did you document this feature anywhere? I don't see any indication in the notes that this is the behavior that was implemented (that'd be a big help!).

On the PYTHONPATH -- that is literally the mechanism by which additional python modules should be used, so this seems exactly right, not at all uncontrolled. What makes you say that using PYTHONPATH is uncontrolled? Are you imagining users have multiple directories with models, and that they won't get the right versions because their path is in the wrong order or something?

Thanks,
Zach

BTW, a proposed patch for models/import_ufo.py :

    #get target directory
    # 1. PYTHONPATH containing UFO
    # 2. Current working directory
    # 3. models directory
    target = None
    if 'PYTHONPATH' in os.environ and not local_dir:
        for directory in os.environ['PYTHONPATH'].split(':'):
            if 'UFO' in os.path.basename(directory) and os.path.exists(directory):
                target= directory
    if target is None:
        try:
            target = os.getcwd()+'/local_UFO_models'
            if not os.access(target,os.R_OK):
                os.mkdir(target)
        except Exception:
            target = pjoin(MG5DIR, 'models')

Best,
Zach

If you insist on securing it, then I would do the following:

=== modified file 'models/import_ufo.py'
--- models/import_ufo.py 2018-07-11 14:13:42 +0000
+++ models/import_ufo.py 2018-07-17 08:24:10 +0000
@@ -138,7 +138,8 @@
     target = None
     if 'PYTHONPATH' in os.environ and not local_dir:
         for directory in os.environ['PYTHONPATH'].split(':'):
- if 'UFO' in os.path.basename(directory) and os.path.exists(directory):
+ if 'UFO' in os.path.basename(directory) and os.path.exists(directory) and\
+ misc.glob('*/couplings.py', path=directory):
                 target= directory
     if target is None:
         target = pjoin(MG5DIR, 'models')

This will ensure that models will be written only in a place where some other UFO model are already present.
This force you to put the first UFO model manually but this is an acceptable drawback for such feature.

Cheers,

Olivier

Hi Oliver,

I'm not sure how that actually solves the problem, which is a centralized installation of a release? If I'm there, then the MadGraph installation I have is read-only, and I don't have some directory in my path that's read/write that already has a UFO in it. With your patch, I'd have to actually make a directory, copy an arbitrary UFO there (or fake it), and only then would I have a place where the new model could go.

What's wrong with the patch I posted above, to make a local directory for the UFO if the other checks fail?

Best,
Zach

Hi,

Did you change what you wanted to do with that stuff?
My understanding of your use of PYTHONPATH (and the way I use it on my laptop) is to have a central way to have all my model independent of the MG5aMC version. So for that point of view having to have at least one to add additional model is not an issue here. Since I expect that you have already a lot of model in that directory. In practise this security should fix your issue of writting in unwanted place.

> What's wrong with the patch I posted above, to make a local directory for the UFO if the other checks fail?

Well maybe I do not understand your patch, but it seems to me that it do not use the model directory anymore but rather always write a local directory. This might be what you want but this is not something that I think is relevant for most user.
Additionally your patch does not test if "local_UFO_models" is inside PYTHONPATH if it is not this is pointless to write there.

Would something like this ok for you?

    if target is None:
        target = os.path.abspath('local_UFO_models')
        if target in os.environ['PYTHONPATH'].split(':'):
            if not os.access(target,os.R_OK):
                os.mkdir(target)
       else:
           target = pjoin(MG5DIR, 'models')

Cheers,

Olivier

Hi Oliver,

> Did you change what you wanted to do with that stuff?

No, but we're doing *distributed production*. You're right that we have a directory with a whole bunch of models in it. That directory is read-only to the outside world and production system, so random users or production jobs cannot download / install UFO files in it. The current mechanism means that it'll try -- and fail.

The last proposal means that if we have a directory in our PYTHONPATH that doesn't exist then it gets made? I'm not super sure how generally useful that is -- it sounds like we'd be the only ones doing something like that.

If the only option really is to stick a local UFO directory onto the front of PYTHONPATH, and there's no way we can easily get a user-specified directory (or default, as in the case of compiling externals like you have already), then I suppose we can go ahead and just make the local directory (without having to copy another UFO into it or fake a file location please).

I do think it'd be great if this behavior was documented somewhere; is it?

Best,
Zach

Hi,

> The last proposal means that if we have a directory in our PYTHONPATH
> that doesn't exist then it gets made?

No that's not correct.
This is only the case if ./local_UFO_models (to be precise the absolute version of that path)
is in PYTHONPATH and does not exists then it is created and used afterwards.

> I'm not super sure how generally
> useful that is -- it sounds like we'd be the only ones doing something
> like that.

Those features will be indeed dedicated to your setup and The only documentation
that would be dedicated to such feature will be this thread (or equivalent thread in the future if

> If the only option really is to stick a local UFO directory onto the
> front of PYTHONPATH, and there's no way we can easily get a user-
> specified directory (or default, as in the case of compiling externals
> like you have already), then I suppose we can go ahead and just make the
> local directory (without having to copy another UFO into it or fake a
> file location please).

Here I try to stay as close as possible of your proposal but making it more secure.
Looks like that I'm confused about what you try to achieve with your proposed patch.
If that does not match what you need, then I would go for the hidden configuration parameter.
(i.e. one of the input/mg5_configuration.txt)

Cheers,

Olivier

> On 22 Jul 2018, at 21:22, Zachary Marshall <email address hidden> wrote:
>
> Question #670770 on MadGraph5_aMC@NLO changed:
> https://answers.launchpad.net/mg5amcnlo/+question/670770
>
> Status: Answered => Open
>
> Zachary Marshall is still having a problem:
> Hi Oliver,
>
>> Did you change what you wanted to do with that stuff?
>
> No, but we're doing *distributed production*. You're right that we have
> a directory with a whole bunch of models in it. That directory is read-
> only to the outside world and production system, so random users or
> production jobs cannot download / install UFO files in it. The current
> mechanism means that it'll try -- and fail.
>
> The last proposal means that if we have a directory in our PYTHONPATH
> that doesn't exist then it gets made? I'm not super sure how generally
> useful that is -- it sounds like we'd be the only ones doing something
> like that.
>
> If the only option really is to stick a local UFO directory onto the
> front of PYTHONPATH, and there's no way we can easily get a user-
> specified directory (or default, as in the case of compiling externals
> like you have already), then I suppose we can go ahead and just make the
> local directory (without having to copy another UFO into it or fake a
> file location please).
>
> I do think it'd be great if this behavior was documented somewhere; is
> it?
>
> Best,
> Zach
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.