Utilities

Utilities for Transformation system

class DIRAC.TransformationSystem.Client.Utilities.PluginUtilities(plugin='Standard', transClient=None, dataManager=None, fc=None, debug=False, transID=None)

Bases: object

Utility class used by plugins

__init__(plugin='Standard', transClient=None, dataManager=None, fc=None, debug=False, transID=None)

c’tor

Setting defaults

clearCachedFileSize(lfns)

Utility function

closerSEs(existingSEs, targetSEs, local=False)

Order the targetSEs such that the first ones are closer to existingSEs. Keep all elements in targetSEs

createTasksBySize(lfns, replicaSE, fileSizes=None, flush=False)

Split files in groups according to the size and create tasks for a given SE

getExistingCounters(normalise=False, requestedSites=[])
getPluginParam(name, default=None)

Get plugin parameters using specific settings or settings defined in the CS Caution: the type returned is that of the default value

groupByReplicas(files, status)

Generates tasks based on the location of the input data

Parameters:

fileReplicas (dict) –

{‘/this/is/at.1’: [‘SE1’],

’/this/is/at.12’: [‘SE1’, ‘SE2’], ‘/this/is/at.2’: [‘SE2’], ‘/this/is/at_123’: [‘SE1’, ‘SE2’, ‘SE3’], ‘/this/is/at_23’: [‘SE2’, ‘SE3’], ‘/this/is/at_4’: [‘SE4’]}

groupBySize(files, status)

Generate a task for a given amount of data

isSameSE(se1, se2)

Check if 2 SEs are indeed the same.

Parameters:
  • se1 – name of the first StorageElement

  • se2 – name of the second StorageElement

Returns:

True/False if they are considered the same. See isSameSE

isSameSEInList(se1, seList)

Check if an SE is the same as any in a list

logDebug(message, param='')

logger helper

logError(message, param='')

logger helper

logException(message, param='', lException=False)

logger helper

logInfo(message, param='')

logger helper

logVerbose(message, param='')

logger helper

logWarn(message, param='')

logger helper

static seParamtoList(inputParam)

Transform inputParam to list.

Parameters:

inputParam – can be string, list, or string representation of list

Returns:

list

setParameters(params)

Set the transformation parameters and extract transID

uniqueSEs(ses)

return a list of SEs that are not physically the same

DIRAC.TransformationSystem.Client.Utilities.getActiveSEs(seList, access='Write')

Utility function - uses the StorageElement cached status

DIRAC.TransformationSystem.Client.Utilities.getFileGroups(fileReplicas, groupSE=True)

Group files by set of SEs

Parameters:

fileReplicas (dict) –

{‘/this/is/at.1’: [‘SE1’],

’/this/is/at.12’: [‘SE1’, ‘SE2’], ‘/this/is/at.2’: [‘SE2’], ‘/this/is/at_123’: [‘SE1’, ‘SE2’, ‘SE3’], ‘/this/is/at_23’: [‘SE2’, ‘SE3’], ‘/this/is/at_4’: [‘SE4’]}

If groupSE == False, group by SE, in which case a file can be in more than one element

DIRAC.TransformationSystem.Client.Utilities.isArchive(se)

Is the SE an archive

DIRAC.TransformationSystem.Client.Utilities.isFailover(se)

Is the SE a failover SE

DIRAC.TransformationSystem.Client.Utilities.sortExistingSEs(lfnSEs, lfns=None)

Sort SEs according to the number of files in each (most first)

DIRAC.TransformationSystem.Client.Utilities.sortSEs(ses)

Returnes an ordered list of SEs, disk first