Utilities

Utilities for Transformation system

class DIRAC.TransformationSystem.Client.Utilities.PluginUtilities(plugin='Standard', transClient=None, dataManager=None, fc=None, debug=False, transID=None)

Bases: object

Utility class used by plugins

__init__(plugin='Standard', transClient=None, dataManager=None, fc=None, debug=False, transID=None)

c’tor

Setting defaults

clearCachedFileSize(lfns): Utility function

closerSEs(existingSEs, targetSEs, local=False): Order the targetSEs such that the first ones are closer to existingSEs. Keep all elements in targetSEs

createTasksBySize(lfns, replicaSE, fileSizes=None, flush=False): Split files in groups according to the size and create tasks for a given SE

getExistingCounters(normalise=False, requestedSites=[])

getPluginParam(name, default=None): Get plugin parameters using specific settings or settings defined in the CS Caution: the type returned is that of the default value

groupByReplicas(files, status)

Generates tasks based on the location of the input data

Parameters:

fileReplicas (dict) –

{‘/this/is/at.1’: [‘SE1’],: ’/this/is/at.12’: [‘SE1’, ‘SE2’], ‘/this/is/at.2’: [‘SE2’], ‘/this/is/at_123’: [‘SE1’, ‘SE2’, ‘SE3’], ‘/this/is/at_23’: [‘SE2’, ‘SE3’], ‘/this/is/at_4’: [‘SE4’]}

groupBySize(files, status): Generate a task for a given amount of data

isSameSE(se1, se2)

Check if 2 SEs are indeed the same.

Parameters:

se1 – name of the first StorageElement
se2 – name of the second StorageElement

Returns:

True/False if they are considered the same. See isSameSE

isSameSEInList(se1, seList): Check if an SE is the same as any in a list

logDebug(message, param=''): logger helper

logError(message, param=''): logger helper

logException(message, param='', lException=False): logger helper

logInfo(message, param=''): logger helper

logVerbose(message, param=''): logger helper

logWarn(message, param=''): logger helper

static seParamtoList(inputParam)

Transform inputParam to list.

Parameters:: inputParam – can be string, list, or string representation of list
Returns:: list

setParameters(params): Set the transformation parameters and extract transID

uniqueSEs(ses): return a list of SEs that are not physically the same

DIRAC.TransformationSystem.Client.Utilities.getActiveSEs(seList, access='Write'): Utility function - uses the StorageElement cached status

DIRAC.TransformationSystem.Client.Utilities.getFileGroups(fileReplicas, groupSE=True)

Group files by set of SEs

Parameters:

fileReplicas (dict) –

{‘/this/is/at.1’: [‘SE1’],: ’/this/is/at.12’: [‘SE1’, ‘SE2’], ‘/this/is/at.2’: [‘SE2’], ‘/this/is/at_123’: [‘SE1’, ‘SE2’, ‘SE3’], ‘/this/is/at_23’: [‘SE2’, ‘SE3’], ‘/this/is/at_4’: [‘SE4’]}

If groupSE == False, group by SE, in which case a file can be in more than one element

DIRAC.TransformationSystem.Client.Utilities.isArchive(se): Is the SE an archive

DIRAC.TransformationSystem.Client.Utilities.isFailover(se): Is the SE a failover SE

DIRAC.TransformationSystem.Client.Utilities.sortExistingSEs(lfnSEs, lfns=None): Sort SEs according to the number of files in each (most first)

DIRAC.TransformationSystem.Client.Utilities.sortSEs(ses): Returnes an ordered list of SEs, disk first