Job

Job Base Class

This class provides generic job definition functionality suitable for any VO.

Helper functions are documented with example usage for the DIRAC API. An example script (for a simple executable) would be:

from DIRAC.Interfaces.API.Dirac import Dirac
from DIRAC.Interfaces.API.Job import Job

j = Job()
j.setCPUTime(500)
j.setExecutable('/bin/echo hello')
j.setExecutable('yourPythonScript.py')
j.setExecutable('/bin/echo hello again')
j.setName('MyJobName')

dirac = Dirac()
jobID = dirac.submitJob(j)
print 'Submission Result: ',jobID

Note that several executables can be provided and wil be executed sequentially.

class DIRAC.Interfaces.API.Job.Job(script=None, stdout='std.out', stderr='std.err')

Bases: API

DIRAC jobs

__init__(script=None, stdout='std.out', stderr='std.err')

Instantiates the Workflow object and some default parameters.

execute()

Developer function. Executes the job locally.

runLocal(dirac=None)

The dirac (API) object is for local submission.

setBannedSites(sites)

Helper function.

Can specify a desired destination site for job. This can be useful for debugging purposes but often limits the possible candidate sites and overall system response time.

Example usage:

>>> job = Job()
>>> job.setBannedSites(['LCG.GRIDKA.de','LCG.CNAF.it'])
Parameters:

sites (str or list) – single site string or list

setCPUTime(timeInSecs)

Helper function.

Example usage:

>>> job = Job()
>>> job.setCPUTime(5000)
Parameters:

timeInSecs (int) – CPU time

setConfigArgs(cfgString)

Developer function. Allow to pass arbitrary settings to the payload configuration service environment.

setDestination(destination)

Helper function.

Can specify a desired destination site or sites for job. This can be useful for debugging purposes but often limits the possible candidate sites and overall system response time.

Example usage:

>>> job = Job()
>>> job.setDestination('LCG.CERN.ch')
Parameters:

destination (str or list) – site string

Returns:

S_OK/S_ERROR

setDestinationCE(ceName, diracSite=None)

Developer function.

Allows to direct a job to a particular Grid CE.

setExecutable(executable, arguments='', logFile='', modulesList=None, parameters=None, paramValues=None)

Helper function.

Specify executable script to run with optional arguments and log file for standard output.

These can be either:

  • Submission of a python or shell script to DIRAC
    • Can be inline scripts e.g. C{‘/bin/ls’}

    • Scripts as executables e.g. python or shell script file

Example usage:

>>> job = Job()
>>> job.setExecutable('myScript.py')
Parameters:
  • executable (str) – Executable

  • arguments (str) – Optional arguments to executable

  • logFile (str) – Optional log file name

  • modulesList (list) – Optional list of modules (to be used mostly when extending this method)

  • parameters (list of tuples) – Optional list of parameters (to be used mostly when extending this method)

  • paramValues – Optional list of parameters values (to be used mostly when extending this method)

setExecutionEnv(environmentDict)

Helper function.

Optionally specify a dictionary of key, value pairs to be set before the job executes e.g. {‘MYVAR’:3}

The standard application environment variables are always set so this is intended for user variables only.

Example usage:

>>> job = Job()
>>> job.setExecutionEnv({'<MYVARIABLE>':'<VALUE>'})
Parameters:

environmentDict (dictionary) – Environment variables

setInputData(lfns)

Helper function.

Specify input data by Logical File Name (LFN).

Example usage:

>>> job = Job()
>>> job.setInputData(['/lhcb/production/DC04/v2/DST/00000742_00003493_10.dst'])
Parameters:

lfns (Single LFN string or list of LFNs) – Logical File Names

setInputDataPolicy(policy, dataScheduling=True)

Helper function.

Specify a job input data policy, this takes precedence over any site specific or global settings.

Possible values for policy are ‘Download’ or ‘Protocol’ (case-insensitive). This requires that the module locations are defined for the VO in the CS.

Example usage:

>>> job = Job()
>>> job.setInputDataPolicy('download')
setInputSandbox(files)

Helper function.

Specify input sandbox files less than 10MB in size. If over 10MB, files or a directory may be uploaded to Grid storage, see C{dirac.uploadSandbox()}.

Paths to the options file and (if required) ‘lib/’ directory of the DLLs are specified here. Default is local directory. Executables may be placed in the lib/ directory if desired. The lib/ directory is transferred to the Grid Worker Node before the job executes.

Files / directories can be specified using the * character e.g. *.txt these are resolved correctly before job execution on the WN.

Example usage:

>>> job = Job()
>>> job.setInputSandbox(['DaVinci.opts'])
Parameters:

files (Single string or list of strings ['','']) – Input sandbox files, can specify full path

setJobGroup(jobGroup)

Helper function.

Allows to group certain jobs according to an ID.

Example usage:

>>> job = Job()
>>> job.setJobGroup('Bs2JPsiPhi')
Parameters:

jobGroup (string) – JobGroup name

setLogLevel(logLevel)

Helper function.

Optionally specify a DIRAC logging level for the job, e.g. ALWAYS, INFO, VERBOSE, WARN, DEBUG by default this is set to the info level.

Example usage:

>>> job = Job()
>>> job.setLogLevel('debug')
Parameters:

logLevel (string) – Logging level

setName(jobName)

Helper function.

A name for the job can be specified if desired. This will appear in the JobName field of the monitoring webpage. If nothing is specified a default value will appear.

Example usage:

>>> job=Job()
>>> job.setName("myJobName")
Parameters:

jobName (str) – Name of job

setNumberOfProcessors(numberOfProcessors=None, minNumberOfProcessors=None, maxNumberOfProcessors=None)

Helper function.

Example usage:

>>> job = Job()
>>> job.setNumberOfProcessors(numberOfProcessors=2)
means that the job needs 2 processors
>>> job = Job()
>>> job.setNumberOfProcessors(minNumberOfProcessors=4, maxNumberOfProcessors=8)
means that the job needs at least 4 processors, and that will use at most 8 processors
>>> job = Job()
>>> job.setNumberOfProcessors(minNumberOfProcessors=2)
means that the job needs at least 2 processors, and that will use all the processors available
>>> job = Job()
>>> job.setNumberOfProcessors(minNumberOfProcessors=1)
means that the job can run in SP mode, and that will use all the processors available
(so the job could run MP, but also SP)
>>> job = Job()
>>> job.setNumberOfProcessors(maxNumberOfProcessors=4)
is equivalent to
>>> job.setNumberOfProcessors(minNumberOfProcessors=1, maxNumberOfProcessors=4)
and it means that the job can run in SP mode, and that will use at most 4 processors
(so the job could run MP, but also SP)
>>> job = Job()
>>> job.setNumberOfProcessors(minNumberOfProcessors=6, maxNumberOfProcessors=4)
is a non-sense, and will lead to consider that the job can run exactly on 4 processors
>>> job = Job()
>>> job.setNumberOfProcessors(numberOfProcessors=3, maxNumberOfProcessors=4)
will lead to ignore the second parameter
>>> job = Job()
>>> job.setNumberOfProcessors(numberOfProcessors=3, minNumberOfProcessors=2)
will lead to ignore the second parameter
Parameters:
  • processors (int) – number of processors required by the job (exact number, unless a min/max are set)

  • minNumberOfProcessors (int) – optional min number of processors the job applications can use

  • maxNumberOfProcessors (int) – optional max number of processors the job applications can use

Returns:

S_OK/S_ERROR

setOutputData(lfns, outputSE=None, outputPath='')

Helper function.

For specifying output data to be registered in Grid storage. If a list of OutputSEs are specified the job wrapper will try each in turn until successful. If the OutputPath is specified this will appear only after / <VO> / user / <initial> / <username> directory.

The output data can be LFNs or local file names. If they are LFNs they should be pre-prended by “LFN:”, otherwise they will be interpreted as local files to be found. If local files are specified, then specifying the outputPath may become necessary, because if it’s not specified then it will be constructed starting from the user name.

Example usage:

>>> job = Job()
>>> job.setOutputData(['DVNtuple.root'])
Parameters:
  • lfns (Single string or list of strings ['','']) – Output data file or files.

  • outputSE (string or list) – Optional parameter to specify the Storage Element to store data or files, e.g. CERN-tape

  • outputPath (string) – Optional parameter to specify part of the path in the storage (see above)

setOutputSandbox(files)

Helper function.

Specify output sandbox files. If specified files are over 10MB, these may be uploaded to Grid storage with a notification returned in the output sandbox.

Example usage:

>>> job = Job()
>>> job.setOutputSandbox(['DaVinci_v19r12.log','DVNTuples.root'])
Parameters:

files (Single string or list of strings ['','']) – Output sandbox files

setOwner(ownerProvided)

Developer function.

Normally users should always specify their immutable DIRAC nickname.

setOwnerDN(ownerDN)

Developer function.

Allows to force expected owner DN of proxy.

setOwnerGroup(ownerGroup)

Developer function.

Allows to force expected owner group of proxy.

setParameterSequence(name, parameterList, addToWorkflow=False)

Function to define a sequence of values for parametric jobs.

Parameters:
  • name (str) – sequence parameter name

  • parameterList (list) – list of parameter values

  • addToWorkflow (bool) – flag to add parameter to the workflow on the fly, if str, then use as the workflow parameter

Returns:

S_OK/S_ERROR

setPlatform(platform)

Developer function: sets the target platform, e.g. Linux_x86_64_glibc-2.17. This platform is in the form of what it is returned by the dirac-platform script (or dirac-architecture if your extension provides it)

setPriority(priority)

Helper function.

Sets the job priority in the range 1-10. Priorities 1-9 are matched probabilistically with higher numbers being more likely. Priority 10 jobs are always started before lower priority jobs from the same user.

Example usage:

>>> job = Job()
>>> job.setPriority(5)
Parameters:

priority (int) – Job priority

setTag(tags)

Set the Tags job requirements

Example usage:

>>> job = Job()
>>> job.setTag( ['WholeNode','8GBMemory'] )
Parameters:

tags (str or list) – single tag string or a list of tags

setType(jobType)

Developer function.

Specify job type for testing purposes.