EchoStorage

Echo module based on the GFAL2_StorageBase class.

class DIRAC.Resources.Storage.EchoStorage.EchoStorage(storageName, parameters)

Bases: GFAL2_StorageBase

class EchoStorage

Interface to the Echo storage.

This plugin will work with both gsiftp and root protocol. According to the RAL admins, Echo best use is done:

  • using gsiftp for all WAN transfers, and LAN write.

  • using root from LAN read.

Note that there are still a few issues to be sorted out with xroot

This would translate in a configuration such as:

RAL-ECHO
{
  BackendType = Echo
  SEType = T0D1
  AccessProtocols = gsiftp,root
  WriteProtocols = gsiftp
  XRootConfig
  {
    Host = xrootd.echo.stfc.ac.uk
    PluginName = Echo
    Protocol = root
    Path = lhcb:user
    Access = remote
  }
  GidFTPConfig
  {
    Host = gridftp.echo.stfc.ac.uk
    PluginName = Echo
    Protocol = gsiftp
    Path = lhcb:user
    Access = remote
  }
}
Operations
{
  Defaults
  {
    DataManagement
    {
      ThirdPartyProtocols=srm,gsiftp,dips
      RegistrationProtocols=srm,gsiftp,dips
    }
  }
}
DYNAMIC_OPTIONS = {}
PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']
__init__(storageName, parameters)

c’tor

changeDirectory(directory)

Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False)

Overwrite to add the double slash

createDirectory(path) DOKReturnType[T] | DErrorReturnType

Create directory on the storage

Parameters:

path (str) – path to be created on the storage (pfn : srm://…)

Returns:

Successful dict {path : True } Failed dict {path : error message } S_ERROR in case of argument problems

exists(path) DOKReturnType[T] | DErrorReturnType

Check if the path exists on the storage

Parameters:

path (str) – path or list of paths to be checked

Returns:

Failed dictionary: {pfn : error message} Successful dictionary: {pfn : bool} S_ERROR in case of argument problems

getCurrentDirectory()

Get the current directory

getCurrentURL(fileName)

Overwrite to add the double slash

getDirectory(path, localPath=False)

Not available on Echo

Returns:

S_ERROR

getDirectoryMetadata(path)

Not available on Echo

Returns:

S_ERROR

getDirectorySize(path)

Not available on Echo

Returns:

S_ERROR

getEndpoint()
This will get endpoint of the storage. It basically is the same as getURLBase()

but without the basePath

Returns:

‘proto://hostname<:port>’

getFile(path, localPath=False) DOKReturnType[T] | DErrorReturnType

Make a local copy of storage :path:

Parameters:
  • path (str) – path (or list of paths) on storage (srm://…)

  • localPath – destination folder. Default is from current directory

Returns:

Successful dict: {path : size} Failed dict: {path : errorMessage} S_ERROR in case of argument problems

getFileMetadata(path) DOKReturnType[T] | DErrorReturnType

Get metadata associated to the file(s)

Parameters:

path (str) – path (or list of paths) on the storage (srm://…)

Returns:

successful dict { path : metadata } failed dict { path : error message } S_ERROR in case of argument problems

getFileSize(path) DOKReturnType[T] | DErrorReturnType

Get the physical size of the given file

Parameters:

path – path (or list of path) on storage (pfn : srm://…)

Returns:

Successful dict {path : size} Failed dict {path : error message } S_ERROR in case of argument problems

getName()

The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters:

occupancyLFN – (mandatory named argument) LFN of the json file.

Returns:

S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB

getParameters()

Get the parameters with which the storage was instantiated

getTransportURL(pathDict, protocols)

Get a transport URL for a given URL. For a simple storage plugin it is just returning input URL if the plugin protocol is one of the requested protocols

Parameters:
  • pathDict (dict) – URL obtained from File Catalog or constructed according to convention

  • protocols (python:list) – a list of acceptable transport protocols in priority order

getURLBase(withWSUrl=False)

Overwrite to add the double slash

isDirectory(path)

Not available on Echo

Returns:

S_ERROR

isFile(path) DOKReturnType[T] | DErrorReturnType

Check if the path provided is a file or not

Parameters:

str – path or list of paths to be checked ( ‘srm://…’)

Returns:

Failed dict: {path : error message} Successful dict: {path : bool} S_ERROR in case of argument problems

isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters:
  • self – self reference

  • url (str) – URL

isURL(path)

Guess if the path looks like a URL

Parameters:
  • self – self reference

  • path (string) – input file LFN or URL

Returns boolean:

True if URL, False otherwise

listDirectory(path)

Not available on Echo

Returns:

S_ERROR

prestageFile(path, lifetime: int = 86400) DOKReturnType[T] | DErrorReturnType

Issue prestage request for file(s)

Parameters:
  • path (str) – path or list of paths to be prestaged

  • lifetime (int) – prestage lifetime in seconds (default 24h)

Returns:

succesful dict { url : token } failed dict { url : message } S_ERROR in case of argument problems

prestageFileStatus(path) DOKReturnType[T] | DErrorReturnType

Checking the staging status of file(s) on the storage

Parameters:

path (dict) – dict { url : token }

Returns:

succesful dict { url : bool } failed dict { url : message } S_ERROR in case of argument problems

putDirectory(path)

Not available on Echo

Returns:

S_ERROR

putFile(path, sourceSize: int = 0) DOKReturnType[T] | DErrorReturnType

Put a copy of a local file or a file on another srm storage to a directory on the physical storage.

Parameters:
  • path – dictionary { lfn (srm://…) : localFile }

  • sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…

Returns:

Successful dict: { path : size } Failed dict: { path : error message } S_ERROR in case of argument problems

releaseFile(path) DOKReturnType[T] | DErrorReturnType

Release a pinned file

Parameters:

path (str) – PFN path { pfn : token } - pfn can be an empty string, then all files that have that same token get released. Just as you can pass an empty token string and a directory as pfn which then releases all the files in the directory an its subdirectories

Returns:

successful dict {url : token}, failed dict {url : message} S_ERROR in case of argument problems

removeDirectory(path, recursive=False)

Not available on Echo

Returns:

S_ERROR

removeFile(path) DOKReturnType[T] | DErrorReturnType

Physically remove the file specified by path

A non existing file will be considered as successfully removed

Parameters:

path (str) – path (or list of paths) on storage (srm://…)

Returns:

Successful dict {path : True} Failed dict {path : error message} S_ERROR in case of argument problems

resetCurrentDirectory()

Reset the working directory to the base dir

setParameters(parameterDict)

Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)
updateURL(url, withWSUrl=False)

Update the URL according to the current SE parameters