StorageElement

This is the StorageElement class.

class DIRAC.Resources.Storage.StorageElement.StorageElementCache

Bases: object

__init__()
class DIRAC.Resources.Storage.StorageElement.StorageElementItem(name, plugins=None, vo=None, hideExceptions=False)

Bases: object

class StorageElement

common interface to the grid storage element

self.name is the resolved name of the StorageElement i.e CERN-tape self.options is dictionary containing the general options defined in the CS e.g. self.options[‘Backend] = ‘Castor2’ self.storages is a list of the stub objects created by StorageFactory for the protocols found in the CS. self.localPlugins is a list of the local protocols that were created by StorageFactory self.remotePlugins is a list of the remote protocols that were created by StorageFactory self.protocolOptions is a list of dictionaries containing the options found in the CS. (should be removed)

dynamic method:

retransferOnlineFile( lfn )
exists( lfn )
isFile( lfn )
getFile( lfn, localPath = False )
putFile( lfnLocal, sourceSize = 0 ) : {lfn:local}
replicateFile( lfn, sourceSize = 0 )
getFileMetadata( lfn )
getFileSize( lfn )
removeFile( lfn )
prestageFile( lfn, lifetime = 86400 )
prestageFileStatus( lfn )
pinFile( lfn, lifetime = 60 * 60 * 24 )
releaseFile( lfn )
isDirectory( lfn )
getDirectoryMetadata( lfn )
getDirectorySize( lfn )
listDirectory( lfn )
removeDirectory( lfn, recursive = False )
createDirectory( lfn )
putDirectory( lfn )
getDirectory( lfn, localPath = False )
__init__(name, plugins=None, vo=None, hideExceptions=False)

c’tor

Parameters

  • name (str) – SE name

  • plugins (list) – requested storage plugins

  • vo – vo

addAccountingOperation(lfns, startDate, elapsedTime, storageParameters, callRes)

Generates a DataOperation accounting if needs to be, and adds it to the DataStore client cache

Parameters

  • lfns – list of lfns on which we attempted the operation

  • startDate – datetime, start of the operation

  • elapsedTime – time (seconds) the operation took

  • storageParameters – the parameters of the plugins used to perform the operation

  • callRes – the return of the method call, S_OK or S_ERROR

The operation is generated with the OperationType “se.methodName” The TransferSize and TransferTotal for directory methods actually take into account the files inside the directory, and not the amount of directory given as parameter

checkOccupancy(occupancyDict, unit)

Validate occupancy dict given by getOccupancy

Parameters

  • occupancyDict (dict) – occupancy given by occupancy or storage plugins

  • unit (str) – Any of ( ‘B’, ‘kB’, ‘MB’, ‘GB’, ‘TB’, ‘PB’)

Returns

S_OK with updated occupancyDict

checksumType()

get specific /Resources/StorageElements/<SEName>/ChecksumType option if defined, otherwise global /Resources/StorageElements/ChecksumType

dump()

Dump to the logger a summary of the StorageElement items.

generateTransferURLsBetweenSEs(lfns, sourceSE, protocols=None)

This negotiate the URLs to be used for third party copy. This is mostly useful for FTS. If protocols is given, it restricts the list of plugins to use

Parameters

  • lfns – list/dict of lfns to generate the URLs

  • sourceSE – storageElement instance of the sourceSE

  • protocols – ordered protocol restriction list

Returns

dictionary with keys:

  • Successful: lfn indexed pair (src, dest) urls

  • Failed: lfn indexed with error

  • Protocols: tuple (srcProto, destProto)

getChecksumType()

Checksum type getter for backward compatibility

getLFNFromURL(urls)

Get the LFN from the PFNS .

Parameters

lfn – input lfn or lfns (list/dict)

getLocalPlugins()

Get the list of all the local access protocols defined for this Storage Element

getOccupancy(unit='MB', **kwargs)

Retrieves the space information about the storage. It returns the Total and Free space, and a SpaceReservation.

The SpaceReservation is just a name of a zone of the physical storage which can have some space reserved. It corresponds to the SpaceToken concept of SRM. If the StorageElement definition has a SpaceReservation option in the CS, this is returned, unless it is overwritten by the storage plugin.

It loops over the different Storage Plugins to query it.

Params occupancyLFN

(named param) LFN where to find the space reporting json file on the storage The json file should contain the Free and Total space in B. If not specified, the default path will be </vo/occupancy.json>

Params unit

(default MB)unit of the value returned. See convertSizeUnits() CAUTION: only the Total and Free field are converted ! Since the rest is whatever is returned by the plugin no conversion is performed

Returns

S_OK with dict (keys: Total, Free, SpaceReservation)

getPlugins()

Get the list of all the plugins defined for this Storage Element

getRemotePlugins()

Get the list of all the remote access protocols defined for this Storage Element

getStatus()

Return Status of the SE only if the SE is valid It returns an S_OK/S_ERROR structure

getStorageElementName()

SE name getter for backward compatibility

getStorageParameters(plugin=None, protocol=None)

Get plugin specific options

Parameters

  • plugin – plugin we are interested in

  • protocol – protocol we are interested in

Either plugin or protocol can be defined, not both, but at least one of them

getURL(lfn, protocol=False, replicaDict=None)

execute ‘getTransportURL’ operation.

Parameters

  • lfn (str) – string, list or dictionary of lfns

  • protocol – if no protocol is specified, we will request self.turlProtocols

  • replicaDict – optional results from the File Catalog replica query

isSameSE(otherSE)

Compares two SE together and tries to guess if the two SEs are pointing at the same location from the namespace point of view. This is primarily aimed at avoiding to overwrite a file with itself, in particular where the difference is only the SRM spacetoken.

Two SEs are considered to be the same if they have a couple (Host, Path) in common among their various protocols

Parameters

otherSE – the storage element to which we compare

Returns

boolean. True if the two SEs are the same.

isValid(operation=None)

check CS/RSS statuses for :operation:

Parameters

operation (str) – operation name

negociateProtocolWithOtherSE(sourceSE, protocols=None)

Negotiate what protocol could be used for a third party transfer between the sourceSE and ourselves. If protocols is given, the chosen protocol has to be among those

Parameters

  • sourceSE – storageElement instance of the sourceSE

  • protocols – ordered protocol restriction list

Returns

a list protocols that fits the needs, or None

status()

Return Status of the SE, a dictionary with:

  • Read: True (is allowed), False (it is not allowed)

  • Write: True (is allowed), False (it is not allowed)

  • Remove: True (is allowed), False (it is not allowed)

  • Check: True (is allowed), False (it is not allowed).

    Note

    Check is always allowed IF Read is allowed (regardless of what set in the Check option of the configuration)

  • DiskSE: True if TXDY with Y > 0 (defaults to True)

  • TapeSE: True if TXDY with X > 0 (defaults to False)

  • TotalCapacityTB: float (-1 if not defined)

  • DiskCacheTB: float (-1 if not defined)

It returns directly the dictionary

storageElementName()

SE name getter