JobDB

The JobDB class is a front-end to the main WMS database containing

job definitions and status information. It is used in most of the WMS components

Configuration Parameters:

The following options can be set in Systems/WorkloadManagement/Databases/JobDB

• MaxRescheduling: Set the maximum number of times a job can be rescheduled, default 3.

• CompressJDLs: Enable compression of JDLs when they are stored in the database, default False.

class DIRAC.WorkloadManagementSystem.DB.JobDB.JobDB(parentLogger=None)

Bases: DB

Interface to MySQL-based JobDB

__init__(parentLogger=None)

Standard Constructor

buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None, useLikeQuery=False)

Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments For performing LIKE queries use the parameter useLikeQuery=True

countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of entries wit the given conditions

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

executeStoredProcedure(packageName, parameters, outputIds, *, conn=None)

executeStoredProcedureWithCursor(packageName, parameters, *, conn=None)

getAtticJobParameters(jobID, paramList=None, rescheduleCounter=-1)

Get Attic Job Parameters defined for a job with jobID. Returns a dictionary with the Attic Job Parameters per each rescheduling cycle. If parameterList is empty - all the parameters are returned. If recheduleCounter = -1, all cycles are returned.

getCSOption(optionName, defaultValue=None)

getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Get distinct values of a table attribute under specified conditions

getDistinctJobAttributes(attribute, condDict=None, older=None, newer=None, timeStamp='LastUpdateTime')

Get distinct values of the job attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None, useLikeQuery=False)

Select “outFields” from “tableName” with condDict N records can match the condition return S_OK(tuple(Field, Value)) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values. if useLikeQuery=True, then conDict can return matched rows if “%” is defined inside conDict.

getHeartBeatData(jobID)

Retrieve the job’s heart beat data

getInputData(jobID)

Get input data for the given job

getJobAttribute(jobID, attribute)

Get the given attribute of a job specified by its jobID

getJobAttributes(jobID, attrList=None)

Get all Job Attributes for a given jobID. Return a dictionary with all Job Attributes as value pairs

getJobCommand(jobID, status='Received')

Get a command to be passed to the job together with the next heart beat

getJobJDL(jobID, original=False)

Get JDL for job specified by its jobID. By default the current job JDL is returned. If ‘original’ argument is True, original JDL is returned

getJobOptParameter(jobID, parameter)

Get optimizer parameters for the given job.

getJobOptParameters(jobID, paramList=None)

Get optimizer parameters for the given job. If the list of parameter names is empty, get all the parameters then

getJobParameter(jobID, parameter)

Get the given parameter of a job specified by its jobID

Deprecated: Use JobParametersDB instead

getJobParameters(jobID, paramList=None)

Get Job Parameters defined for jobID. Returns a dictionary with the Job Parameters. If parameterList is empty - all the parameters are returned.

getJobsAttributes(jobIDs, attrList=None)

Get all Job(s) Attributes for a given list of jobIDs. Return a dictionary with all Job Attributes as value pairs

getSiteSummary()

Get the summary of jobs in a given status on all the sites

getSiteSummaryWeb(selectDict, sortList, startItem, maxItems)

Get the summary of jobs in a given status on all the sites in the standard Web form

getSummarySnapshot(requestedFields=False)

Get the summary snapshot for a given combination

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None)

Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

insertNewJobIntoDB(jdl, owner, ownerGroup, initialStatus='Received', initialMinorStatus='Job accepted', vo=None)

Insert the initial JDL into the Job database, Do initial JDL crosscheck, Set Initial job Attributes and Status

Parameters:

• jdl (str) – job description JDL

• owner (str) – job owner user name

• ownerGroup (str) – job owner group

• initialStatus (str) – optional initial job status (Received by default)

• initialMinorStatus (str) – optional initial minor job status

Returns:

new job ID

isValid()

Check if correctly initialised

property log

removeInfoFromHeartBeatLogging(status, delTime, maxLines)

Remove HeartBeatLoggingInfo from DB.

Parameters:

• status (str) – status of the jobs

• delTime (str) – timestamp of the age of the jobs

• maxLines (int) – maximum number of lines to be removed

Returns:

S_OK/S_ERROR

removeJobFromDB(jobIDs)

Remove jobs from the Job DB and clean up all the job related data in various tables

removeJobOptParameter(jobID, name)

Remove the specified optimizer parameter for jobID

rescheduleJob(jobID)

Reschedule the given job to run again from scratch. Retain the already defined parameters in the parameter Attic

selectJobs(condDict, older=None, newer=None, timeStamp='LastUpdateTime', orderAttribute=None, limit=None)

Select jobs matching the following conditions: - condDict dictionary of required Key = Value pairs; - with the last update date older and/or newer than given dates;

The result is ordered by JobID if requested, the result is limited to a given number of jobs if requested.

setAtticJobParameter(jobID, key, value, rescheduleCounter)

Set attic parameter for job specified by its jobID when job rescheduling for later debugging

setEndExecTime(jobID, endDate=None)

Set EndExecTime time stamp

setHeartBeatData(jobID, dynamicDataDict)

Add the job’s heart beat data to the database

setInputData(jobID, inputData)

Inserts input data for the given job

setJobAttribute(jobID, attrName, attrValue, update=False, myDate=None, force=False)

Set an attribute value for job specified by jobID. The LastUpdate time stamp is refreshed if explicitly requested

Parameters:

• jobID (int or str) – job ID

• attrName (str) – attribute name

• attrValue (str) – attribute value

• update (bool) – optional flag to update the job LastUpdateTime stamp

• myDate (str) – optional time stamp for the LastUpdateTime attribute

Returns:

S_OK/S_ERROR

setJobAttributes(jobID, attrNames, attrValues, update=False, myDate=None, force=False)

Set one or more attribute values for one or more jobs specified by jobID. The LastUpdate time stamp is refreshed if explicitly requested with the update flag

This method is also used for updating the Status, MinorStatus, ApplicationStatus of a job, as self.setJobsStatus also calls this method. If the status is already final, we don’t update it.

Parameters:

• jobID (int or str or list) – one or more job IDs

• attrNames (list) – names of attributes to update

• attrValues (list) – corresponding values of attributes to update

• update (bool) – optional flag to update the job LastUpdateTime stamp

• myDate (str) – optional time stamp for the LastUpdateTime attribute

• force (bool) – force update of Status (override State Machine decision)

Returns:

S_OK/S_ERROR

setJobCommand(jobID, command, arguments=None)

Store a command to be passed to the job together with the next heart beat

setJobCommandStatus(jobID, command, status)

Set the command status

setJobJDL(jobID, jdl=None, originalJDL=None)

Insert JDL’s for job specified by jobID

setJobOptParameter(jobID, name, value)

Set an optimzer parameter specified by name,value pair for the job JobID

setJobStatus(jobID, status='', minorStatus='', applicationStatus='', force=False)

Set status of the job specified by its jobID

setJobsMajorStatus(jIDList, candidateStatus, force=False)

Sets jobs major status, considering the JobStateMachine result

Parameters:

• jIDList (list) – list of one or more job IDs

• candidateStatus (str) – candidate major Status

setNextOptimizer(jobID, currentOptimizer)

Set the job status to be processed by the next optimizer in the chain

setOptimizerChain(jobID, optimizerList)

Set the optimizer chain for the given job. The ‘TaskQueue’ optimizer should be the last one in the chain, it is added if not present in the optimizerList

setStartExecTime(jobID, startDate=None)

Set StartExecTime time stamp and HeartBeatTime if not already set

transactionCommit()

transactionRollback()

transactionStart()

updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.