StorageManagementDB

StorageManagementDB is a front end to the Stager Database.

There are five tables in the StorageManagementDB: Tasks, CacheReplicas, TaskReplicas, StageRequests.

The Tasks table is the place holder for the tasks that have requested files to be staged. These can be from different systems and have different associated call back methods. The CacheReplicas table keeps the information on all the CacheReplicas in the system. It maps all the file information LFN, PFN, SE to an assigned ReplicaID. The TaskReplicas table maps the TaskIDs from the Tasks table to the ReplicaID from the CacheReplicas table. The StageRequests table contains each of the prestage request IDs for each of the replicas.

class DIRAC.StorageManagementSystem.DB.StorageManagementDB.StorageManagementDB(systemInstance='Default')

Bases: DIRAC.Core.Base.DB.DB

__init__(systemInstance='Default')

c’tor

Parameters

self – self reference

buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None, useLikeQuery=False)

Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments For performing LIKE queries use the parameter useLikeQuery=True

countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of entries wit the given conditions

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

executeStoredProcedure(packageName, parameters, outputIds)

executeStoredProcedureWithCursor(packageName, parameters)

getAssociatedReplicas(replicaIDs)

Retrieve the list of Replicas that belong to the same Tasks as the provided list

getCSOption(optionName, defaultValue=None)

getCacheReplicas(condDict={}, older=None, newer=None, timeStamp='LastUpdate', orderAttribute=None, limit=None, connection=False)

Get cache replicas for the supplied selection with support for the web standard structure

getCacheReplicasSummary(connection=False)

Reports breakdown of file number/size in different staging states across storage elements

getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Get distinct values of a table attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None, useLikeQuery=False)

Select “outFields” from “tableName” with condDict N records can match the condition return S_OK(tuple(Field, Value)) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values. if useLikeQuery=True, then conDict can return matched rows if “%” is defined inside conDict.

getOfflineReplicas(connection=False)

getStageRequests(condDict={}, older=None, newer=None, timeStamp='StageRequestSubmitTime', orderAttribute=None, limit=None, connection=False)

Get stage requests for the supplied selection with support for web standard structure

getStagedReplicas(connection=False)

getSubmittedStagePins()

getTaskInfo(taskID, connection=False)

Obtain all the information from the Tasks table for a supplied task.

getTaskStatus(taskID, connection=False)

Obtain the task status from the Tasks table.

getTaskSummary(jobID, connection=False)

Obtain the task summary from the database.

getTasks(condDict={}, older=None, newer=None, timeStamp='SubmitTime', orderAttribute=None, limit=None, connection=False)

Get stage requests for the supplied selection with support for web standard structure

getTasksWithStatus(status)

This method retrieves the TaskID from the Tasks table with the supplied Status.

getWaitingReplicas(connection=False)

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None)

Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

insertStageRequest(requestDict, pinLifeTime)

killTasksBySourceTaskID(sourceTaskIDs, connection=False)

Given SourceTaskIDs (jobs), this will cancel further staging of files for the corresponding tasks. The “cancel” is actually removing all stager DB records for these jobs. Care must be taken to NOT cancel staging of files that are requested also by other tasks.

removeStageRequests(replicaIDs, connection=False)

removeTasks(taskIDs, connection=False)

This will delete the entries from the TaskReplicas for the provided taskIDs.

removeUnlinkedReplicas(connection=False)

This will remove Replicas from the CacheReplicas that are not associated to any Task. If the Replica has been Staged, wait until StageRequest.PinExpiryTime and remove the StageRequest and CacheReplicas entries

setOldTasksAsFailed(daysOld, connection=False)

Set Tasks older than “daysOld” number of days to Failed These tasks have already been retried every day for staging

setRequest(lfnDict, source, callbackMethod, sourceTaskID, connection=False)

This method populates the StorageManagementDB Tasks table with the requested files.

setStageComplete(replicaIDs)

setTasksDone(taskIDs)

This will update the status for a list of taskIDs to Done.

transactionCommit()

transactionRollback()

transactionStart()

updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.

updateReplicaFailure(terminalReplicaIDs)

This method sets the status to Failure with the failure reason for the supplied Replicas.

updateReplicaInformation(replicaTuples)

This method set the replica size information and pfn for the requested storage element.

updateReplicaStatus(replicaIDs, newReplicaStatus, connection=False)

updateStageRequestStatus(replicaIDs, newStageStatus, connection=False)

updateTaskStatus(taskIDs, newTaskStatus, connection=False)

wakeupOldRequests(replicaIDs, retryInterval, connection=False)

get only StageRequests with StageRequestSubmitTime older than 1 day AND are still not staged delete these requests reset Replicas with corresponding ReplicaIDs to Status=’New’