ModuleBase

ModuleBase - contains the base class for workflow modules. Defines several common utility methods.

The modules defined within this package are developed in a way to be executed by a DIRAC.Core.Worfklow.Worfklow. In particular, a DIRAC.Core.Workflow.Worfklow object will only call the “execute” function, that is defined here.

These modules, inspired by the LHCb experience, give the possibility to define simple user and production jobs. Many VOs might want to extend this package. And actually, for some cases, it will be necessary. For example, defining the LFN output at runtime (within the “UploadOutputs” module is a VO specific operation

The DIRAC APIs are used to create Jobs that make use of these modules.

exception DIRAC.Workflow.Modules.ModuleBase.GracefulTermination

Bases: Exception

__init__(*args, **kwargs)
args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class DIRAC.Workflow.Modules.ModuleBase.ModuleBase(loggerIn=None)

Bases: object

Base class for Modules - works only within DIRAC workflows

This module, inheriting by “object”, can use cooperative methods, very useful here.

__init__(loggerIn=None)

Initialization of module base.

loggerIn is a logger object that can be passed so that the logging will be more clear.

execute()

Function called by all super classes. This is the only function that Workflow will call automatically.

The design adopted here is that all the modules are inheriting from this class, and will NOT override this function. Instead, the inherited modules will override the following functions: _resolveInputVariables() _initialize() _setCommand() _executeCommand() _execute() that are called here exactly in this order. Each implementation of these functions, in the subclasses, should never return S_OK, S_ERROR. This choice has been made for convenience of coding, and for the high level of inheritance implemented here. Instead, they should return: - None when no issues arise - a RuntimeError exception when there are issues - a GracefulTermination exception (defined also here) when the module should be terminated gracefully

The various parameters in input to this method are used almost only for testing purposes.

finalize()

Just finalizing the module execution by flushing the logs. This will be done always.

generateFailoverFile()

Retrieve the accumulated reporting request, and produce a JSON file that is consumed by the JobWrapper

getCandidateFiles(outputList, outputLFNs, fileMask, stepMask='')

Returns list of candidate files to upload, check if some outputs are missing.

outputList has the following structure:

[ {‘outputDataType’:’’,’outputDataSE’:’’,’outputDataName’:’’} , {…} ]

outputLFNs is the list of output LFNs for the job

fileMask is the output file extensions to restrict the outputs to

returns dictionary containing type, SE and LFN for files restricted by mask

setApplicationStatus(status, sendFlag=True)

Wraps around setJobApplicationStatus of state update client

setJobParameter(name, value, sendFlag=True)

Wraps around setJobParameter of state update client