prepend a line to the PBS script

Hi Roberto,

The correct way for this would be to NOT modify cluster.py
But define your own cluster class:

https://cp3.irmp.ucl.ac.be/projects/madgraph/wiki/Plugin

Such that you do not have to modify madgraph each time you install a new version/...
This will also help to distribute your change to other member using the same cluster.

In your case, you can basically copy-paste the default way to submit and change the following line:

        command = ['qsub','-o', stdout,
                   '-N', me_dir,
                   '-e', stderr,
                   '-V']

To

        command = ['qsub','-o', stdout,
                   '-N', me_dir,
            '-l', 'walltime=36:00:00'
                   '-e', stderr,
                   '-V']

Cheers,

Olivier

> On 8 Aug 2017, at 01:29, Roberto Franceschini <email address hidden> wrote:
>
> New question #655441 on MadGraph5_aMC@NLO:
> https://answers.launchpad.net/mg5amcnlo/+question/655441
>
> Hi
> I need to prepend a line
> "#PBS -l walltime=36:00:00"
> to the scripts that will be submitted to PBS.
>
>> From cluster.py it seems that qsub is used reading the script from stdin (as opposed to from a "template" file). In this situation I am a bit lost in cluster.py looking for where I should stick my one-line at the beginning of the PBS script ...
> Is there a quick way to prepend this line to the PBS scripts submitted by MG?
>
> Thanks for reading,
> Roberto
>
>
> --
> You received this question notification because you are an answer
> contact for MadGraph5_aMC@NLO.

Hi I have passed the -l option, which does the same as adding the line I was trying to add to the script. Thanks for the tip!

Hi Olivier,
  I think back then I did it, but now I am unable to redo it : (

I am getting

CRITICAL: plugin directory PLUGIN/PBSslim fail to be loaded. Please check it
root: new_pbs is not recognized as a supported cluster format.

from this __init__.py

class MYcluster(cluster.PBSCluster):

    def modify_interface(self, run_interface):
        """
        routine which allow to modify the run_card/mg5cmd object to change the
        default behavior of the runs.
        This is called at the time of the compilation of the run_card.
        Note that this function can be called multiple times by run.
        """
        run_card = run_interface.run_card
        run_card.set('survey_nchannel_per_job',10, changeifuserset=False)
        run_card.set('refine_evt_by_job', 25000, changeifuserset=False)
        return super(MYcluster, self).modify_interface(run_interface)

    @multiple_try()
    def submit(self, prog, argument=[], cwd=None, stdout=None, stderr=None, log=None,
               required_output=[], nb_submit=0):
        """Submit a job prog to a PBS cluster"""

        me_dir = self.get_jobs_identifier(cwd, prog)

        if len(self.submitted_ids) > self.maximum_submited_jobs:
            fct = lambda idle, run, finish: logger.info('Waiting for free slot: %s %s %s' % (idle, run, finish))
            self.wait(me_dir, fct, self.maximum_submited_jobs)

        text = ""
        if cwd is None:
            cwd = os.getcwd()
        else:
            text = " cd %s;" % cwd
        if stdout is None:
            stdout = '/dev/null'
        if stderr is None:
            stderr = '/dev/null'
        elif stderr == -2: # -2 is subprocess.STDOUT
            stderr = stdout
        if log is None:
            log = '/dev/null'

        if not os.path.isabs(prog):
            text += "./%s" % prog
        else:
            text+= prog

        if argument:
            text += ' ' + ' '.join(argument)

        command = ['qsub','-o', stdout,
                   '-N', me_dir,
                   '-l', 'walltime=2:00:00',
                   '-e', stderr,
                   '-V']

        if self.cluster_queue and self.cluster_queue != 'None':
            command.extend(['-q', self.cluster_queue])

        a = misc.Popen(command, stdout=subprocess.PIPE,
                                      stderr=subprocess.STDOUT,
                                      stdin=subprocess.PIPE, cwd=cwd)

        output = a.communicate(text)[0]
        id = output.split('.')[0]
        if not id.isdigit() or a.returncode !=0:
            raise ClusterManagmentError, 'fail to submit to the cluster: \n%s' \
                                                                        % output

        self.submitted += 1
        self.submitted_ids.append(id)
        return id

How do I get a more detailed error message to see why this class does not work?

Thanks for helping,
Roberto

It turns out that this is necessary at the top of the __init__.py of the plugin because os is used by the submit function and also the decorator multiple_try is used and would otherwise not be defined.

import subprocess
import logging
import os
import time
import re
import glob
import inspect
import sys

logger = logging.getLogger('madgraph.cluster')

try:
    from madgraph import MadGraph5Error
    import madgraph.various.misc as misc
except Exception, error:
    if __debug__:
        print str(error)
    from internal import MadGraph5Error
    import internal.misc as misc

pjoin = os.path.join

multiple_try = misc.multiple_try