StorageElement¶
This is the StorageElement class.
- class DIRAC.Resources.Storage.StorageElement.StorageElementItem(name, plugins=None, vo=None, hideExceptions=False)¶
Bases:
object
- class StorageElement¶
common interface to the grid storage element
self.name is the resolved name of the StorageElement i.e CERN-tape self.options is dictionary containing the general options defined in the CS e.g. self.options[‘Backend] = ‘Castor2’ self.storages is a list of the stub objects created by StorageFactory for the protocols found in the CS. self.localPlugins is a list of the local protocols that were created by StorageFactory self.remotePlugins is a list of the remote protocols that were created by StorageFactory self.protocolOptions is a list of dictionaries containing the options found in the CS. (should be removed)
dynamic method:
retransferOnlineFile( lfn ) exists( lfn ) isFile( lfn ) getFile( lfn, localPath = False ) putFile( lfnLocal, sourceSize = 0 ) : {lfn:local} replicateFile( lfn, sourceSize = 0 ) getFileMetadata( lfn ) getFileSize( lfn ) removeFile( lfn ) prestageFile( lfn, lifetime = 86400 ) prestageFileStatus( lfn ) pinFile( lfn, lifetime = 60 * 60 * 24 ) releaseFile( lfn ) isDirectory( lfn ) getDirectoryMetadata( lfn ) getDirectorySize( lfn ) listDirectory( lfn ) removeDirectory( lfn, recursive = False ) createDirectory( lfn ) putDirectory( lfn ) getDirectory( lfn, localPath = False )
- __init__(name, plugins=None, vo=None, hideExceptions=False)¶
c’tor
- addAccountingOperation(lfns, startDate, elapsedTime, storageParameters, callRes)¶
Generates a DataOperation accounting if needs to be, and adds it to the DataStore client cache
- Parameters
lfns – list of lfns on which we attempted the operation
startDate – datetime, start of the operation
elapsedTime – time (seconds) the operation took
storageParameters – the parameters of the plugins used to perform the operation
callRes – the return of the method call, S_OK or S_ERROR
The operation is generated with the OperationType “se.methodName” The TransferSize and TransferTotal for directory methods actually take into account the files inside the directory, and not the amount of directory given as parameter
- checkOccupancy(occupancyDict, unit)¶
Validate occupancy dict given by getOccupancy
- checksumType()¶
get specific /Resources/StorageElements/<SEName>/ChecksumType option if defined, otherwise global /Resources/StorageElements/ChecksumType
- dump()¶
Dump to the logger a summary of the StorageElement items.
- generateTransferURLsBetweenSEs(lfns, sourceSE, protocols=None)¶
This negotiate the URLs to be used for third party copy. This is mostly useful for FTS. If protocols is given, it restricts the list of plugins to use
- Parameters
lfns – list/dict of lfns to generate the URLs
sourceSE – storageElement instance of the sourceSE
protocols – ordered protocol restriction list
- Returns
dictionary with keys:
Successful: lfn indexed pair (src, dest) urls
Failed: lfn indexed with error
Protocols: tuple (srcProto, destProto)
- getChecksumType()¶
Checksum type getter for backward compatibility
- getLFNFromURL(urls)¶
Get the LFN from the PFNS .
- Parameters
lfn – input lfn or lfns (list/dict)
- getLocalPlugins()¶
Get the list of all the local access protocols defined for this Storage Element
- getOccupancy(unit='MB', **kwargs)¶
Retrieves the space information about the storage. It returns the Total and Free space, and a SpaceReservation.
The SpaceReservation is just a name of a zone of the physical storage which can have some space reserved. It corresponds to the
SpaceToken
concept of SRM. If the StorageElement definition has aSpaceReservation
option in the CS, this is returned, unless it is overwritten by the storage plugin.It loops over the different Storage Plugins to query it.
- Params occupancyLFN
(named param) LFN where to find the space reporting json file on the storage The json file should contain the Free and Total space in B. If not specified, the default path will be </vo/occupancy.json>
- Params unit
(default MB)unit of the value returned. See
convertSizeUnits()
CAUTION: only the Total and Free field are converted ! Since the rest is whatever is returned by the plugin no conversion is performed- Returns
S_OK with dict (keys: Total, Free, SpaceReservation)
- getPlugins()¶
Get the list of all the plugins defined for this Storage Element
- getRemotePlugins()¶
Get the list of all the remote access protocols defined for this Storage Element
- getStatus()¶
Return Status of the SE only if the SE is valid It returns an S_OK/S_ERROR structure
- getStorageElementName()¶
SE name getter for backward compatibility
- getStorageParameters(plugin=None, protocol=None)¶
Get plugin specific options
- Parameters
plugin – plugin we are interested in
protocol – protocol we are interested in
Either plugin or protocol can be defined, not both, but at least one of them
- getURL(lfn, protocol=False, replicaDict=None)¶
execute ‘getTransportURL’ operation.
- Parameters
lfn (str) – string, list or dictionary of lfns
protocol – if no protocol is specified, we will request self.turlProtocols
replicaDict – optional results from the File Catalog replica query
- isSameSE(otherSE)¶
Compares two SE together and tries to guess if the two SEs are pointing at the same location from the namespace point of view. This is primarily aimed at avoiding to overwrite a file with itself, in particular where the difference is only the SRM spacetoken.
Two SEs are considered to be the same if they have a couple (Host, Path) in common among their various protocols
- Parameters
otherSE – the storage element to which we compare
- Returns
boolean. True if the two SEs are the same.
- isValid(operation=None)¶
check CS/RSS statuses for :operation:
- Parameters
operation (str) – operation name
- negociateProtocolWithOtherSE(sourceSE, protocols=None)¶
Negotiate what protocol could be used for a third party transfer between the sourceSE and ourselves. If protocols is given, the chosen protocol has to be among those
- Parameters
sourceSE – storageElement instance of the sourceSE
protocols – ordered protocol restriction list
- Returns
a list protocols that fits the needs, or None
- status()¶
Return Status of the SE, a dictionary with:
Read: True (is allowed), False (it is not allowed)
Write: True (is allowed), False (it is not allowed)
Remove: True (is allowed), False (it is not allowed)
Check: True (is allowed), False (it is not allowed).
Note
Check is always allowed IF Read is allowed (regardless of what set in the Check option of the configuration)
DiskSE: True if TXDY with Y > 0 (defaults to True)
TapeSE: True if TXDY with X > 0 (defaults to False)
TotalCapacityTB: float (-1 if not defined)
DiskCacheTB: float (-1 if not defined)
It returns directly the dictionary
- storageElementName()¶
SE name getter