PilotAgentsDB

PilotAgentsDB class is a front-end to the Pilot Agent Database. This database keeps track of all the submitted grid pilot jobs. It also registers the mapping of the DIRAC jobs to the pilot agents.

Available methods are:

addPilotReferences() setPilotStatus() deletePilot() clearPilots() setPilotDestinationSite() storePilotOutput() getPilotOutput() setJobForPilot() getPilotsSummary() getGroupedPilotSummary()

class DIRAC.WorkloadManagementSystem.DB.PilotAgentsDB.PilotAgentsDB(parentLogger=None)

Bases: DB

__init__(parentLogger=None)

c’tor

Parameters:

self – self reference

addPilotReferences(pilotRef, VO, gridType='DIRAC', pilotStampDict={})

Add a new pilot job reference

buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None, useLikeQuery=False)

Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments For performing LIKE queries use the parameter useLikeQuery=True

clearPilots(interval=30, aborted_interval=7)

Delete all the pilot references submitted before <interval> days

countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of entries wit the given conditions

countPilots(condDict, older=None, newer=None, timeStamp='SubmissionTime')

Select pilot references according to the provided criteria. “newer” and “older” specify the time interval in minutes

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

deletePilot(pilotRef, conn=False)

Delete Pilot with the given reference from the PilotAgentsDB

deletePilots(pilotIDs, conn=False)

Delete Pilots with IDs in the given list from the PilotAgentsDB

executeStoredProcedure(packageName, parameters, outputIds, *, conn=None)

executeStoredProcedureWithCursor(packageName, parameters, *, conn=None)

getCSOption(optionName, defaultValue=None)

getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Get distinct values of a table attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None, useLikeQuery=False)

Select “outFields” from “tableName” with condDict N records can match the condition return S_OK(tuple(Field, Value)) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values. if useLikeQuery=True, then conDict can return matched rows if “%” is defined inside conDict.

getGroupedPilotSummary(columnList)

The simplified pilot summary based on getPilotSummaryWeb method. It calculates pilot efficiency based on the same algorithm as in the Web version, basically takes into account Done and Aborted pilots only from the last day. The selection is done entirely in SQL.

Parameters:

columnList (list) – A list of column to consider when grouping to calculate efficiencies. e.g. [‘GridSite’, ‘DestinationSite’] is used to calculate efficiencies for sites and CEs.

Returns:

S_OK/S_ERROR with a dict containing the ParameterNames and Records lists.

getJobsForPilot(pilotID)

Get IDs of Jobs that were executed by a pilot

getPilotCurrentJob(pilotRef)

The job ID currently executed by the pilot

getPilotInfo(pilotRef=False, conn=False, paramNames=[], pilotID=False)

Get all the information for the pilot job reference or reference list

getPilotMonitorSelectors()

Get distinct values for the Pilot Monitor page selectors

getPilotMonitorWeb(selectDict, sortList, startItem, maxItems)

Get summary of the pilot job information in a standard structure

getPilotOutput(pilotRef)

Retrieve standard output and error for pilot with pilotRef

getPilotSummary(startdate='', enddate='')

Get summary of the pilot jobs status by site

getPilotSummaryWeb(selectDict, sortList, startItem, maxItems)

Get summary of the pilot jobs status by CE/site in a standard structure

getPilotsForJobID(jobID)

Get ID of Pilot Agent that is running a given JobID

getSummarySnapshot(requestedFields=False)

Get the summary snapshot for a given combination

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None)

Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

selectPilots(condDict, older=None, newer=None, timeStamp='SubmissionTime', orderAttribute=None, limit=None)

Select pilot references according to the provided criteria. “newer” and “older” specify the time interval in minutes

setAccountingFlag(pilotRef, mark='True')

Set the pilot AccountingSent flag

setCurrentJobID(pilotRef, jobID)

Set the pilot agent current DIRAC job ID

setJobForPilot(jobID, pilotRef, site=None, updateStatus=True)

Store the jobID of the job executed by the pilot with reference pilotRef

setPilotBenchmark(pilotRef, mark)

Set the pilot agent benchmark

setPilotDestinationSite(pilotRef, destination, conn=False)

Set the pilot agent destination site

setPilotStatus(pilotRef, status, destination=None, statusReason=None, gridSite=None, queue=None, benchmark=None, currentJob=None, updateTime=None, conn=False)

Set pilot job status

storePilotOutput(pilotRef, output, error)

Store standard output and error for a pilot with pilotRef

transactionCommit()

transactionRollback()

transactionStart()

updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.

class DIRAC.WorkloadManagementSystem.DB.PilotAgentsDB.PivotedPilotSummaryTable(columnList)

Bases: object

The class creates a ‘pivoted’ table by combining records with the same group of self.columnList into a single row. It allows an easy calculation of pilot efficiencies.

__init__(columnList)

Initialise a table with columns to be grouped by.

Parameters:

columnList – i.e. [‘GridSite’, ‘DestinationSite’]

Returns:

buildSQL()

Build an SQL query to create a table with all status counts in one row, (“pivoted”) grouped by columns in the column list.

Parameters:

selectDict (dict)

Returns:

SQL query

getColumnList()

pstates = ['Submitted', 'Done', 'Failed', 'Aborted', 'Running', 'Waiting', 'Scheduled', 'Ready']