EchoStorage

Echo module based on the GFAL2_StorageBase class.

class DIRAC.Resources.Storage.EchoStorage.EchoStorage(storageName, parameters)

Bases: GFAL2_StorageBase

class EchoStorage

Interface to the Echo storage.

This plugin will work with both gsiftp and root protocol. According to the RAL admins, Echo best use is done:

using gsiftp for all WAN transfers, and LAN write.
using root from LAN read.

Note that there are still a few issues to be sorted out with xroot

This would translate in a configuration such as:

RAL-ECHO
{
  BackendType = Echo
  SEType = T0D1
  AccessProtocols = gsiftp,root
  WriteProtocols = gsiftp
  XRootConfig
  {
    Host = xrootd.echo.stfc.ac.uk
    PluginName = Echo
    Protocol = root
    Path = lhcb:user
    Access = remote
  }
  GidFTPConfig
  {
    Host = gridftp.echo.stfc.ac.uk
    PluginName = Echo
    Protocol = gsiftp
    Path = lhcb:user
    Access = remote
  }
}
Operations
{
  Defaults
  {
    DataManagement
    {
      ThirdPartyProtocols=srm,gsiftp,dips
      RegistrationProtocols=srm,gsiftp,dips
    }
  }
}

DYNAMIC_OPTIONS = {}

PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']

__init__(storageName, parameters): c’tor

changeDirectory(directory): Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False): Overwrite to add the double slash

createDirectory(path) → DOKReturnType[T] | DErrorReturnType

Create directory on the storage

Parameters:: path (str) – path to be created on the storage (pfn : srm://…)
Returns:: Successful dict {path : True } Failed dict {path : error message } S_ERROR in case of argument problems

exists(path) → DOKReturnType[T] | DErrorReturnType

Check if the path exists on the storage

Parameters:: path (str) – path or list of paths to be checked
Returns:: Failed dictionary: {pfn : error message} Successful dictionary: {pfn : bool} S_ERROR in case of argument problems

getCurrentDirectory(): Get the current directory

getCurrentURL(fileName): Overwrite to add the double slash

getDirectory(path, localPath=False)

Not available on Echo

Returns:: S_ERROR

getDirectoryMetadata(path)

Not available on Echo

Returns:: S_ERROR

getDirectorySize(path)

Not available on Echo

Returns:: S_ERROR

getEndpoint()

This will get endpoint of the storage. It basically is the same as getURLBase(): but without the basePath

Returns:: ‘proto://hostname<:port>’

getFile(path, localPath=False) → DOKReturnType[T] | DErrorReturnType

Make a local copy of storage :path:

Parameters:

path (str) – path (or list of paths) on storage (srm://…)
localPath – destination folder. Default is from current directory

Returns:

Successful dict: {path : size} Failed dict: {path : errorMessage} S_ERROR in case of argument problems

getFileMetadata(path) → DOKReturnType[T] | DErrorReturnType

Get metadata associated to the file(s)

Parameters:: path (str) – path (or list of paths) on the storage (srm://…)
Returns:: successful dict { path : metadata } failed dict { path : error message } S_ERROR in case of argument problems

getFileSize(path) → DOKReturnType[T] | DErrorReturnType

Get the physical size of the given file

Parameters:: path – path (or list of path) on storage (pfn : srm://…)
Returns:: Successful dict {path : size} Failed dict {path : error message } S_ERROR in case of argument problems

getName(): The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters:: occupancyLFN – (mandatory named argument) LFN of the json file.
Returns:: S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB

getParameters(): Get the parameters with which the storage was instantiated

getTransportURL(pathDict, protocols)

Get a transport URL for a given URL. For a simple storage plugin it is just returning input URL if the plugin protocol is one of the requested protocols

Parameters:

pathDict (dict) – URL obtained from File Catalog or constructed according to convention
protocols (python:list) – a list of acceptable transport protocols in priority order

getURLBase(withWSUrl=False): Overwrite to add the double slash

getWLCGTokenPath(lfn: str, wlcgTokenBasePath: str) → str: Returns the path expected to be in a WLCG token It basically consists of basepath - tokenBasePath + LFN The tokenBasePath is a configuration on the storage side.

isDirectory(path)

Not available on Echo

Returns:: S_ERROR

isFile(path) → DOKReturnType[T] | DErrorReturnType

Check if the path provided is a file or not

Parameters:: str – path or list of paths to be checked ( ‘srm://…’)
Returns:: Failed dict: {path : error message} Successful dict: {path : bool} S_ERROR in case of argument problems

isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters:

self – self reference
url (str) – URL

isURL(path)

Guess if the path looks like a URL

Parameters:

self – self reference
path (string) – input file LFN or URL

Returns boolean:

True if URL, False otherwise

listDirectory(path)

Not available on Echo

Returns:: S_ERROR

prestageFile(path, lifetime: int = 86400) → DOKReturnType[T] | DErrorReturnType

Issue prestage request for file(s)

Parameters:

path (str) – path or list of paths to be prestaged
lifetime (int) – prestage lifetime in seconds (default 24h)

Returns:

succesful dict { url : token } failed dict { url : message } S_ERROR in case of argument problems

prestageFileStatus(path) → DOKReturnType[T] | DErrorReturnType

Checking the staging status of file(s) on the storage

Parameters:: path (dict) – dict { url : token }
Returns:: succesful dict { url : bool } failed dict { url : message } S_ERROR in case of argument problems

putDirectory(path)

Not available on Echo

Returns:: S_ERROR

putFile(path, sourceSize: int = 0) → DOKReturnType[T] | DErrorReturnType

Put a copy of a local file or a file on another srm storage to a directory on the physical storage.

Parameters:

path – dictionary { lfn (srm://…) : localFile }
sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…

Returns:

Successful dict: { path : size } Failed dict: { path : error message } S_ERROR in case of argument problems

releaseFile(path) → DOKReturnType[T] | DErrorReturnType

Release a pinned file

Parameters:: path (str) – PFN path { pfn : token } - pfn can be an empty string, then all files that have that same token get released. Just as you can pass an empty token string and a directory as pfn which then releases all the files in the directory an its subdirectories
Returns:: successful dict {url : token}, failed dict {url : message} S_ERROR in case of argument problems

removeDirectory(path, recursive=False)

Not available on Echo

Returns:: S_ERROR

removeFile(path) → DOKReturnType[T] | DErrorReturnType

Physically remove the file specified by path

A non existing file will be considered as successfully removed

Parameters:: path (str) – path (or list of paths) on storage (srm://…)
Returns:: Successful dict {path : True} Failed dict {path : error message} S_ERROR in case of argument problems

resetCurrentDirectory(): Reset the working directory to the base dir

setParameters(parameterDict): Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)

updateURL(url, withWSUrl=False): Update the URL according to the current SE parameters