StorageElement

This is the StorageElement module. It implements The StorageElementItem as well as the caching system

class DIRAC.Resources.Storage.StorageElement.StorageElementCache

Bases: object

The StorageElementCache keeps StorageElementItem instances in a cache to save on initialization cost. It keeps one instance per tuple (thread ID, seName, protocolSections, VO, proxy )

__init__()
class DIRAC.Resources.Storage.StorageElement.StorageElementItem(name, protocolSections=None, vo=None, hideExceptions=False)

Bases: object

class StorageElementItem

This class implements all the necessary logic to interact with the GRID storage Elements. Actual interaction with storages are delegated to StoragePlugins.

The role of the StorageElementItem is to:

  • Provide a single interface to all the grid storages

  • Ensure the status of the Storage in RSS

  • Select multiple protocols for various operations

  • Support multiple protocols as failover

  • Negociate protocols with other StorageElement for Third Party Copy

Variables:

  • name (str) – Resolved name of the StorageElement

  • options (dict) – dictionary containing the general options defined in the CS

  • storages (dict) – dict of the stub objects created by StorageFactory for the protocols found in the CS. Index by the protocol section name.

  • localProtocolSections (list) – list of the local protocols that were created by StorageFactory

  • remoteProtocolSections (list) – list of the remote protocols that were created by StorageFactory

  • protocolOptions (list) – list of dictionaries containing the options found in the CS. (should be removed)

dynamic method:

retransferOnlineFile( lfn )
exists( lfn )
isFile( lfn )
getFile( lfn, localPath = False )
putFile( lfnLocal, sourceSize = 0 ) : {lfn:local}
replicateFile( lfn, sourceSize = 0 )
getFileMetadata( lfn )
getFileSize( lfn )
removeFile( lfn )
prestageFile( lfn, lifetime = 86400 )
prestageFileStatus( lfn )
releaseFile( lfn )
isDirectory( lfn )
getDirectoryMetadata( lfn )
getDirectorySize( lfn )
listDirectory( lfn )
removeDirectory( lfn, recursive = False )
createDirectory( lfn )
putDirectory( lfn )
getDirectory( lfn, localPath = False )
__init__(name, protocolSections=None, vo=None, hideExceptions=False)

c’tor

Parameters:

  • name (str) – SE name

  • protocolSections (list) – requested storage protocolSections

  • vo – vo

addAccountingOperation(urlDict, startDate, elapsedTime, storageParameters, callRes)

Generates a DataOperationSender instance and sends the operation data filled in accountingDict.

Parameters:

  • urlDict – {url:lfn} on which we attempted the operation

  • startDate – datetime, start of the operation

  • elapsedTime – time (seconds) the operation took

  • storageParameters – the parameters of the plugins used to perform the operation

  • callRes – the return of the method call, S_OK or S_ERROR

The operation is generated with the OperationType “se.methodName” The TransferSize and TransferTotal for directory methods actually take into account the files inside the directory, and not the amount of directory given as parameter

checkOccupancy(occupancyDict, unit)

Validate occupancy dict given by getOccupancy

Parameters:

  • occupancyDict (dict) – occupancy given by occupancy or storage plugins

  • unit (str) – Any of ( ‘B’, ‘kB’, ‘MB’, ‘GB’, ‘TB’, ‘PB’)

Returns:

S_OK with updated occupancyDict

checksumType()

get specific /Resources/StorageElements/<SEName>/ChecksumType option if defined, otherwise global /Resources/StorageElements/ChecksumType

dump()

Dump to the logger a summary of the StorageElement items.

generateTransferURLsBetweenSEs(lfns, sourceSE, protocols=None)

This negotiate the URLs to be used for third party copy. This is mostly useful for FTS. If protocols is given, it restricts the list of plugins to use

Parameters:

  • lfns – list/dict of lfns to generate the URLs

  • sourceSE – storageElement instance of the sourceSE

  • protocols – ordered protocol restriction list

Returns:

dictionary with keys:

  • Successful: lfn indexed pair (src, dest) urls

  • Failed: lfn indexed with error

  • Protocols: tuple (srcProto, destProto)

getChecksumType()

Checksum type getter for backward compatibility

getLFNFromURL(urls)

Get the LFN from the PFNS .

Parameters:

lfn – input lfn or lfns (list/dict)

getLocalProtocolSections()

Get the list of all the local access protocols defined for this Storage Element

getOccupancy(unit='MB', **kwargs)

Retrieves the space information about the storage. It returns the Total and Free space, and a SpaceReservation.

The SpaceReservation is just a name of a zone of the physical storage which can have some space reserved. It corresponds to the SpaceToken concept of SRM. If the StorageElement definition has a SpaceReservation option in the CS, this is returned, unless it is overwritten by the storage plugin.

It loops over the different Storage Plugins to query it.

Params occupancyLFN:

(named param) LFN where to find the space reporting json file on the storage The json file should contain the Free and Total space in B. If not specified, the default path will be </vo/occupancy.json>

Params unit:

(default MB)unit of the value returned. See convertSizeUnits() CAUTION: only the Total and Free field are converted ! Since the rest is whatever is returned by the plugin no conversion is performed

Returns:

S_OK with dict (keys: Total, Free, SpaceReservation)

getProtocolSections()

Get the list of all the ProtocolSections defined for this Storage Element

getRemoteProtocolSections()

Get the list of all the remote access protocols defined for this Storage Element

getStatus()

Return Status of the SE only if the SE is valid It returns an S_OK/S_ERROR structure

getStorageElementName()

SE name getter for backward compatibility

getStorageParameters(protocolSection=None, protocol=None)

Get plugin specific options

Parameters:

  • protocolSection – protocolSection we are interested in

  • protocol – protocol we are interested in

Either protocolSection or protocol can be defined, not both, but at least one of them

getURL(lfn, protocol=False, replicaDict=None)

execute ‘getTransportURL’ operation.

Parameters:

  • lfn (str) – string, list or dictionary of lfns

  • protocol – if no protocol is specified, we will request self.turlProtocols

  • replicaDict – optional results from the File Catalog replica query

isSameSE(otherSE)

Compares two SE together and tries to guess if the two SEs are pointing at the same location from the namespace point of view. This is primarily aimed at avoiding to overwrite a file with itself, in particular where the difference is only the SRM spacetoken.

Two SEs are considered to be the same if they have a couple (Host, Path) in common among their various protocols

Parameters:

otherSE – the storage element to which we compare

Returns:

boolean. True if the two SEs are the same.

isValid(operation=None)

check CS/RSS statuses for :operation:

Parameters:

operation (str) – operation name

negociateProtocolWithOtherSE(sourceSE, protocols=None)

Negotiate what protocol could be used for a third party transfer between the sourceSE and ourselves. If protocols is given, the chosen protocol has to be among those

Parameters:

  • sourceSE – storageElement instance of the sourceSE

  • protocols – ordered protocol restriction list

Returns:

a list protocols that fits the needs, or None

status()

Return Status of the SE, a dictionary with:

  • Read: True (is allowed), False (it is not allowed)

  • Write: True (is allowed), False (it is not allowed)

  • Remove: True (is allowed), False (it is not allowed)

  • Check: True (is allowed), False (it is not allowed).

    Note

    Check is always allowed IF Read is allowed (regardless of what set in the Check option of the configuration)

  • DiskSE: True if TXDY with Y > 0 (defaults to True)

  • TapeSE: True if TXDY with X > 0 (defaults to False)

  • TotalCapacityTB: float (-1 if not defined)

  • DiskCacheTB: float (-1 if not defined)

It returns directly the dictionary

storageElementName()

SE name getter