JobDB

The JobDB class is a front-end to the main WMS database containing

job definitions and status information. It is used in most of the WMS components

Configuration Parameters:

The following options can be set in Systems/WorkloadManagement/<Setup>/Databases/JobDB

• MaxRescheduling: Set the maximum number of times a job can be rescheduled, default 3.

• CompressJDLs: Enable compression of JDLs when they are stored in the database, default False.

class DIRAC.WorkloadManagementSystem.DB.JobDB.JobDB(parentLogger=None)

Bases: DIRAC.Core.Base.DB.DB

Interface to MySQL-based JobDB

__init__(parentLogger=None)

Standard Constructor

allowSiteInMask(site, authorDN='Unknown', comment='No comment')

Forbid the given site in the Site Mask

banSiteInMask(site, authorDN='Unknown', comment='No comment')

Forbid the given site in the Site Mask

buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None, useLikeQuery=False)

Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments For performing LIKE queries use the parameter useLikeQuery=True

countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of entries wit the given conditions

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

executeStoredProcedure(packageName, parameters, outputIds, *, conn=None)

executeStoredProcedureWithCursor(packageName, parameters, *, conn=None)

getAllSiteMaskStatus()

Get the everything from site mask status

getAtticJobParameters(jobID, paramList=None, rescheduleCounter=- 1)

Get Attic Job Parameters defined for a job with jobID. Returns a dictionary with the Attic Job Parameters per each rescheduling cycle. If parameterList is empty - all the parameters are returned. If recheduleCounter = -1, all cycles are returned.

getCSOption(optionName, defaultValue=None)

getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Get distinct values of a table attribute under specified conditions

getDistinctJobAttributes(attribute, condDict=None, older=None, newer=None, timeStamp='LastUpdateTime')

Get distinct values of the job attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None, useLikeQuery=False)

Select “outFields” from “tableName” with condDict N records can match the condition return S_OK(tuple(Field, Value)) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values. if useLikeQuery=True, then conDict can return matched rows if “%” is defined inside conDict.

getHeartBeatData(jobID)

Retrieve the job’s heart beat data

getInputData(jobID)

Get input data for the given job

getJobAttribute(jobID, attribute)

Get the given attribute of a job specified by its jobID

getJobAttributes(jobID, attrList=None)

Get all Job Attributes for a given jobID. Return a dictionary with all Job Attributes as value pairs

getJobCommand(jobID, status='Received')

Get a command to be passed to the job together with the next heart beat

getJobJDL(jobID, original=False)

Get JDL for job specified by its jobID. By default the current job JDL is returned. If ‘original’ argument is True, original JDL is returned

getJobOptParameter(jobID, parameter)

Get optimizer parameters for the given job.

getJobOptParameters(jobID, paramList=None)

Get optimizer parameters for the given job. If the list of parameter names is empty, get all the parameters then

getJobParameter(jobID, parameter)

Get the given parameter of a job specified by its jobID

getJobParameters(jobID, paramList=None)

Get Job Parameters defined for jobID. Returns a dictionary with the Job Parameters. If parameterList is empty - all the parameters are returned.

getJobsAttributes(jobIDs, attrList=None)

Get all Job(s) Attributes for a given list of jobIDs. Return a dictionary with all Job Attributes as value pairs

getSiteMask(siteState='Active')

Get the currently active site list

getSiteMaskLogging(siteList)

Get the site mask logging history for the list if site names

getSiteMaskStatus(sites=None)

Get the current site mask status

Parameters

sites – A string for a single site to check, or a list to check multiple sites.

Returns

If input was a list, a dictionary of sites, keys are site names and values are the site statuses. Unknown sites are not included in the output dictionary. If input was a string, then a single value with that site’s status, or S_ERROR if the site does not exist in the DB.

getSiteSummary()

Get the summary of jobs in a given status on all the sites

getSiteSummaryWeb(selectDict, sortList, startItem, maxItems)

Get the summary of jobs in a given status on all the sites in the standard Web form

getSummarySnapshot(requestedFields=False)

Get the summary snapshot for a given combination

getUserSitesTuple(sites)

Returns tuple of active/banned/invalid sties from a user provided list.

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None)

Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

insertNewJobIntoDB(jdl, owner, ownerDN, ownerGroup, initialStatus='Received', initialMinorStatus='Job accepted')

Insert the initial JDL into the Job database, Do initial JDL crosscheck, Set Initial job Attributes and Status

Parameters

• jdl (str) – job description JDL

• owner (str) – job owner user name

• ownerDN (str) – job owner DN

• ownerGroup (str) – job owner group

• initialStatus (str) – optional initial job status (Received by default)

• initialMinorStatus (str) – optional initial minor job status

Returns

new job ID

isValid()

Check if correctly initialised

removeInfoFromHeartBeatLogging(status, delTime, maxLines)

Remove HeartBeatLoggingInfo from DB.

Parameters

• status (str) – status of the jobs

• delTime (str) – timestamp of the age of the jobs

• maxLines (int) – maximum number of lines to be removed

Returns

S_OK/S_ERROR

removeJobFromDB(jobIDs)

Remove jobs from the Job DB and clean up all the job related data in various tables

removeJobOptParameter(jobID, name)

Remove the specified optimizer parameter for jobID

removeSiteFromMask(site=None)

Remove the given site from the mask

rescheduleJob(jobID)

Reschedule the given job to run again from scratch. Retain the already defined parameters in the parameter Attic

selectJobs(condDict, older=None, newer=None, timeStamp='LastUpdateTime', orderAttribute=None, limit=None)

Select jobs matching the following conditions: - condDict dictionary of required Key = Value pairs; - with the last update date older and/or newer than given dates;

The result is ordered by JobID if requested, the result is limited to a given number of jobs if requested.

setAtticJobParameter(jobID, key, value, rescheduleCounter)

Set attic parameter for job specified by its jobID when job rescheduling for later debugging

setEndExecTime(jobID, endDate=None)

Set EndExecTime time stamp

setHeartBeatData(jobID, dynamicDataDict)

Add the job’s heart beat data to the database

setInputData(jobID, inputData)

Inserts input data for the given job

setJobAttribute(jobID, attrName, attrValue, update=False, myDate=None, force=False)

Set an attribute value for job specified by jobID. The LastUpdate time stamp is refreshed if explicitly requested

Parameters

• jobID (int or str) – job ID

• attrName (str) – attribute name

• attrValue (str) – attribute value

• update (bool) – optional flag to update the job LastUpdateTime stamp

• myDate (str) – optional time stamp for the LastUpdateTime attribute

Returns

S_OK/S_ERROR

setJobAttributes(jobID, attrNames, attrValues, update=False, myDate=None, force=False)

Set one or more attribute values for one or more jobs specified by jobID. The LastUpdate time stamp is refreshed if explicitly requested with the update flag

This method is also used for updating the Status, MinorStatus, ApplicationStatus of a job, as self.setJobsStatus also calls this method. If the status is already final, we don’t update it.

Parameters

• jobID (int or str or list) – one or more job IDs

• attrNames (list) – names of attributes to update

• attrValues (list) – corresponding values of attributes to update

• update (bool) – optional flag to update the job LastUpdateTime stamp

• myDate (str) – optional time stamp for the LastUpdateTime attribute

• force (bool) – force update of Status (override State Machine decision)

Returns

S_OK/S_ERROR

setJobCommand(jobID, command, arguments=None)

Store a command to be passed to the job together with the next heart beat

setJobCommandStatus(jobID, command, status)

Set the command status

setJobJDL(jobID, jdl=None, originalJDL=None)

Insert JDL’s for job specified by jobID

setJobOptParameter(jobID, name, value)

Set an optimzer parameter specified by name,value pair for the job JobID

setJobParameter(jobID, key, value)

Set a parameter specified by name,value pair for the job JobID

setJobParameters(jobID, parameters)

Set parameters specified by a list of name/value pairs for the job JobID

Parameters

• jobID (int) – Job ID

• parameters (list) – list of tuples (name, value) pairs

Returns

S_OK/S_ERROR

setJobStatus(jobID, status='', minorStatus='', applicationStatus='')

Set status of the job specified by its jobID

setJobsMajorStatus(jIDList, candidateStatus, force=False)

Sets jobs major status, considering the JobStateMachine result

Parameters

• jIDList (list) – list of one or more job IDs

• candidateStatus (str) – candidate major Status

setNextOptimizer(jobID, currentOptimizer)

Set the job status to be processed by the next optimizer in the chain

setOptimizerChain(jobID, optimizerList)

Set the optimizer chain for the given job. The ‘TaskQueue’ optimizer should be the last one in the chain, it is added if not present in the optimizerList

setSiteMask(siteMaskList, authorDN='Unknown', comment='No comment')

Set the Site Mask to the given mask in a form of a list of tuples (site,status)

setStartExecTime(jobID, startDate=None)

Set StartExecTime time stamp and HeartBeatTime if not already set

transactionCommit()

transactionRollback()

transactionStart()

updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.

DIRAC.WorkloadManagementSystem.DB.JobDB.compressJDL(jdl)

Return compressed JDL string.

DIRAC.WorkloadManagementSystem.DB.JobDB.extractJDL(compressedJDL)

Return decompressed JDL string.