Utilities¶
Utilities for Transformation system
- class DIRAC.TransformationSystem.Client.Utilities.PluginUtilities(plugin='Standard', transClient=None, dataManager=None, fc=None, debug=False, transInThread=None, transID=None)¶
Bases:
object
Utility class used by plugins
- __init__(plugin='Standard', transClient=None, dataManager=None, fc=None, debug=False, transInThread=None, transID=None)¶
c’tor
Setting defaults
- clearCachedFileSize(lfns)¶
Utility function
- closerSEs(existingSEs, targetSEs, local=False)¶
Order the targetSEs such that the first ones are closer to existingSEs. Keep all elements in targetSEs
- createTasksBySize(lfns, replicaSE, fileSizes=None, flush=False)¶
Split files in groups according to the size and create tasks for a given SE
- getExistingCounters(normalise=False, requestedSites=[])¶
- getPluginParam(name, default=None)¶
Get plugin parameters using specific settings or settings defined in the CS Caution: the type returned is that of the default value
- groupByReplicas(files, status)¶
Generates tasks based on the location of the input data
- Parameters
fileReplicas (dict) –
- {‘/this/is/at.1’: [‘SE1’],
’/this/is/at.12’: [‘SE1’, ‘SE2’], ‘/this/is/at.2’: [‘SE2’], ‘/this/is/at_123’: [‘SE1’, ‘SE2’, ‘SE3’], ‘/this/is/at_23’: [‘SE2’, ‘SE3’], ‘/this/is/at_4’: [‘SE4’]}
- groupBySize(files, status)¶
Generate a task for a given amount of data
- isSameSE(se1, se2)¶
Check if 2 SEs are indeed the same.
- Parameters
se1 – name of the first StorageElement
se2 – name of the second StorageElement
- Returns
True/False if they are considered the same. See
isSameSE
- isSameSEInList(se1, seList)¶
Check if an SE is the same as any in a list
- logDebug(message, param='')¶
logger helper
- logError(message, param='')¶
logger helper
- logException(message, param='', lException=False)¶
logger helper
- logInfo(message, param='')¶
logger helper
- logVerbose(message, param='')¶
logger helper
- logWarn(message, param='')¶
logger helper
- static seParamtoList(inputParam)¶
Transform
inputParam
to list.- Parameters
inputParam – can be string, list, or string representation of list
- Returns
list
- setParameters(params)¶
Set the transformation parameters and extract transID
- uniqueSEs(ses)¶
return a list of SEs that are not physically the same
- DIRAC.TransformationSystem.Client.Utilities.getActiveSEs(seList, access='Write')¶
Utility function - uses the StorageElement cached status
- DIRAC.TransformationSystem.Client.Utilities.getFileGroups(fileReplicas, groupSE=True)¶
Group files by set of SEs
- Parameters
fileReplicas (dict) –
- {‘/this/is/at.1’: [‘SE1’],
’/this/is/at.12’: [‘SE1’, ‘SE2’], ‘/this/is/at.2’: [‘SE2’], ‘/this/is/at_123’: [‘SE1’, ‘SE2’, ‘SE3’], ‘/this/is/at_23’: [‘SE2’, ‘SE3’], ‘/this/is/at_4’: [‘SE4’]}
If groupSE == False, group by SE, in which case a file can be in more than one element
- DIRAC.TransformationSystem.Client.Utilities.isArchive(se)¶
Is the SE an archive
- DIRAC.TransformationSystem.Client.Utilities.isFailover(se)¶
Is the SE a failover SE
- DIRAC.TransformationSystem.Client.Utilities.sortExistingSEs(lfnSEs, lfns=None)¶
Sort SEs according to the number of files in each (most first)
- DIRAC.TransformationSystem.Client.Utilities.sortSEs(ses)¶
Returnes an ordered list of SEs, disk first