TransformationDB

DIRAC Transformation DB

Transformation database is used to collect and serve the necessary information in order to automate the task of job preparation for high level transformations. This class is typically used as a base class for more specific data processing databases

class DIRAC.TransformationSystem.DB.TransformationDB.TransformationDB(dbname=None, dbconfig=None, dbIn=None, parentLogger=None)

Bases: DB

TransformationDB class

__init__(dbname=None, dbconfig=None, dbIn=None, parentLogger=None): The standard constructor takes the database name (dbname) and the name of the configuration section (dbconfig)

addDirectory(path, force=False): Adds all the files stored in a given directory in file catalog

addFile(fileDicts, force=False, connection=False): Add the supplied lfn to the Transformations and to the DataFiles table if it passes the filter

addFilesToTransformation(transName, lfns, connection=False): Add a list of LFNs to the transformation directly

addTaskForTransformation(transID, lfns=None, se='Unknown', connection=False): Create a new task with the supplied files for a transformation.

addTransformation(transName, description, longDescription, authorDN, authorGroup, transType, plugin, agentType, fileMask, transformationGroup='General', groupSize=1, inheritedFrom=0, body='', maxTasks=0, eventsPerTask=0, addFiles=True, connection=False, inputMetaQuery=None, outputMetaQuery=None): Add new transformation definition including its input streams

buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None, useLikeQuery=False): Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments For performing LIKE queries use the parameter useLikeQuery=True

cleanTransformation(transName, author='', connection=False): Clean the transformation specified by name or id

countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None): Count the number of entries wit the given conditions

createTransformationMetaQuery(transName, queryDict, queryType, author='', connection=False): Add a Meta Query to a given transformation

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None): Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

deleteTasks(transName, taskIDbottom, taskIDtop, author='', connection=False): Delete tasks with taskID range in transformation

deleteTransformation(transName, author='', connection=False): Remove the transformation specified by name or id

deleteTransformationMetaQuery(transName, queryType, author='', connection=False): Remove a Meta Query from the TransformationMetaQueries table

deleteTransformationParameter(transName, paramName, author='', connection=False): Delete a parameter from the additional parameters table

executeStoredProcedure(packageName, parameters, outputIds, *, conn=None)

executeStoredProcedureWithCursor(packageName, parameters, *, conn=None)

exists(lfns, connection=False): Check the presence of the lfn in the TransformationDB DataFiles table

extendTransformation(transName, nTasks, author='', connection=False): Extend SIMULATION type transformation by nTasks number of tasks

getAdditionalParameters(transName, connection=False)

getCSOption(optionName, defaultValue=None)

getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None): Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None): Get distinct values of a table attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None, useLikeQuery=False): Select “outFields” from “tableName” with condDict N records can match the condition return S_OK(tuple(Field, Value)) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values. if useLikeQuery=True, then conDict can return matched rows if “%” is defined inside conDict.

getFileSummary(lfns, connection=False): Get file status summary in all the transformations

getName(): Get the database name

getTableDistinctAttributeValues(table, attributes, selectDict, older=None, newer=None, timeStamp=None, connection=False)

getTaskInputVector(transName, taskID, connection=False): Get input vector for the given task

getTasksForSubmission(transName, numTasks=1, site='', statusList=None, older=None, newer=None, connection=False): Select tasks with the given status (and site) for submission

getTransformation(transName, extraParams=False, connection=False): Get Transformation definition and parameters of Transformation identified by TransformationID

getTransformationFiles(condDict=None, older=None, newer=None, timeStamp='LastUpdate', orderAttribute=None, limit=None, offset=None, connection=False, columns=None, include_web_records=True): Get files for the supplied transformations with support for the web standard structure

getTransformationFilesCount(transName, field, selection=None, connection=False): Get the number of files in the TransformationFiles table grouped by the supplied field

getTransformationLogging(transName, connection=False): Get logging info from the TransformationLog table

getTransformationMetaQuery(transName, queryType, connection=False)

Get the Meta Query for a given transformation.

Parameters:

transName (str or int) – transformation name or ID
queryType (str) – ‘Input’ or ‘Output’ query
connection – DB connection

Returns:

S_OK with query dictionary, S_ERROR, ENOENT if no query defined

getTransformationParameters(transName, parameters, connection=False): Get the requested parameters for a supplied transformation

getTransformationStats(transName, connection=False): Get number of files in Transformation Table for each status

getTransformationTaskStats(transName='', connection=False): Returns dictionary with number of jobs per status for the given production.

getTransformationTasks(condDict=None, older=None, newer=None, timeStamp='CreationTime', orderAttribute=None, limit=None, inputVector=False, offset=None, connection=False)

getTransformationWithStatus(status, connection=False): Gets a list of the transformations with the supplied status

getTransformations(condDict=None, older=None, newer=None, timeStamp='LastUpdate', orderAttribute=None, limit=None, extraParams=False, offset=None, connection=False, columns=None): Get parameters of all the Transformations with support for the web standard structure

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None): Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

removeFile(lfns, connection=False): Remove file specified by lfn from the ProcessingDB

reserveTask(transName, taskID, connection=False): Reserve the taskID from transformation for submission

setFileStatusForTransformation(transID, fileStatusDict=None, connection=False)

Set file status for the given transformation, based on fileStatusDict {fileID_A: (‘statusA’,errorA), fileID_B: (‘statusB’,errorB), …}

The ErrorCount is incremented if errorA flag is True

setMetadata(path, usermetadatadict): It can be applied to a file or to a directory (path). For a file, add the file to Transformations if the updated metadata dictionary passes the filter. For a directory, add the files contained in the directory to the Transformations if the the updated metadata dictionary passes the filter.

setTaskStatus(transName, taskID, status, connection=False): Set status for job with taskID in production with transformationID

setTaskStatusAndWmsID(transName, taskID, status, taskWmsID, connection=False): Set status and ExternalID for job with taskID in production with transformationID

setTransformationParameter(transName, paramName, paramValue, author='', connection=False): Add a parameter for the supplied transformations

transactionCommit()

transactionRollback()

transactionStart()

updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None): Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.