GFAL2_StorageBase

mod

GFAL2_StorageBase

synopsis

GFAL2 class from StorageElement using gfal2. Other modules can inherit from this use the gfal2 methods.

TODO

When we are totally migrated to python 3, we can explicitely remove the str cast of all the ctx calls. They were added because we would receive unicode from DIRAC https servers and boost python does not play well with that.

Environment Variables:

  • DIRAC_GFAL_GRIDFTP_ENABLE_IPV6: this should be exported and set to false on pure ipv4 nodes because of the globus bug

  • DIRAC_GFAL_GRIDFTP_SESSION_REUSE: This should be exported and set to true in server bashrc files for efficiency reasons.

class DIRAC.Resources.Storage.GFAL2_StorageBase.GFAL2_StorageBase(storageName, parameters)

Bases: StorageBase

class GFAL2_StorageBase

SRM v2 interface to StorageElement using gfal2

DYNAMIC_OPTIONS = {}
PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']
__init__(storageName, parameters)

c’tor

Parameters
  • self – self reference

  • storageName (str) – SE name

  • parameters (dict) – storage parameters

changeDirectory(directory)

Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False)

Construct URL from the given LFN according to the VO convention for the primary protocol of the storage plagin

Parameters
  • lfn (str) – file LFN

  • withWSUrl (boolean) – flag to include the web service part into the resulting URL

Return result

result[‘Value’] - resulting URL

createDirectory(path)

Create directory on the storage

Parameters
  • self – self reference

  • path (str) – path to be created on the storage (pfn : srm://…)

Returns

Successful dict {path : True } Failed dict {path : error message } S_ERROR in case of argument problems

exists(path)

Check if the path exists on the storage

Parameters
  • self – self reference

  • path (str) – path or list of paths to be checked

Returns

Failed dictionary: {pfn : error message} Successful dictionary: {pfn : bool} S_ERROR in case of argument problems

getCurrentDirectory()

Get the current directory

getCurrentURL(fileName)

Obtain the current file URL from the current working directory and the filename

Parameters
  • self – self reference

  • fileName (str) – path on storage

getDirectory(path, localPath=False)

get a directory from the SE to a local path with all its files and subdirectories

Parameters
  • path (str) – path (or list of paths) on the storage (srm://…)

  • localPath (str) – local path where the content of the remote directory will be saved, if not defined it takes current working directory.

Returns

successful and failed dictionaries. The keys are the paths, the values are dictionary {‘Files’: amount of files downloaded, ‘Size’ : amount of data downloaded} S_ERROR in case of argument problems

getDirectoryMetadata(path)

Get metadata for the directory(ies) provided

Parameters
  • self – self reference

  • path (str) – path (or list of paths) on storage (srm://…)

Returns

Successful dict {path : metadata} Failed dict {path : errStr} S_ERROR in case of argument problems

getDirectorySize(path)

Get the size of the directory on the storage

Warning

it is not recursive

Parameters
  • self – self reference

  • path (str) – path or list of paths on storage (srm://…)

Returns

list of successful and failed dictionaries, both indexed by the path

  • In the failed, the value is the error message

  • In the successful the values are dictionaries:

    • Files : amount of files in the dir

    • Size : summed up size of all files

    • subDirs : amount of sub dirs

  • S_ERROR in case of argument problems

getEndpoint()
This will get endpoint of the storage. It basically is the same as getURLBase()

but without the basePath

Returns

‘proto://hostname<:port>’

getFile(path, localPath=False)

Make a local copy of storage :path:

Parameters
  • self – self reference

  • path (str) – path (or list of paths) on storage (srm://…)

  • localPath – destination folder. Default is from current directory

Returns

Successful dict: {path : size} Failed dict: {path : errorMessage} S_ERROR in case of argument problems

getFileMetadata(path)

Get metadata associated to the file(s)

Parameters
  • self – self reference

  • path (str) – path (or list of paths) on the storage (srm://…)

Returns

successful dict { path : metadata } failed dict { path : error message } S_ERROR in case of argument problems

getFileSize(path)

Get the physical size of the given file

Parameters
  • self – self reference

  • path – path (or list of path) on storage (pfn : srm://…)

Returns

Successful dict {path : size} Failed dict {path : error message } S_ERROR in case of argument problems

getName()

The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters

occupancyLFN – (mandatory named argument) LFN of the json file.

Returns

S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB

getParameters()

Get the parameters with which the storage was instantiated

getTransportURL(pathDict, protocols)

Get a transport URL for a given URL. For a simple storage plugin it is just returning input URL if the plugin protocol is one of the requested protocols

Parameters
  • pathDict (dict) – URL obtained from File Catalog or constructed according to convention

  • protocols (python:list) – a list of acceptable transport protocols in priority order

getURLBase(withWSUrl=False)

This will get the URL base. This is then appended with the LFN in DIRAC convention.

Parameters
  • self – self reference

  • withWSUrl (bool) – flag to include Web Service part of the url

Returns

URL

isDirectory(path)

check if the path provided is a directory or not

Parameters
  • self – self reference

  • str – path or list of paths to be checked ( ‘srm://…’)

Returns

dict ‘Failed’ : failed, ‘Successful’ : succesful S_ERROR in case of argument problems

isFile(path)

Check if the path provided is a file or not

Parameters
  • self – self reference

  • str – path or list of paths to be checked ( ‘srm://…’)

Returns

Failed dict: {path : error message} Successful dict: {path : bool} S_ERROR in case of argument problems

isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters
  • self – self reference

  • url (str) – URL

isOK()
isURL(path)

Guess if the path looks like a URL

Parameters
  • self – self reference

  • path (string) – input file LFN or URL

Returns boolean

True if URL, False otherwise

listDirectory(path)

List the content of the path provided

Parameters

path (str) – single or list of paths (srm://…)

Returns

failed dict {path : message } successful dict { path : {‘SubDirs’ : subDirs, ‘Files’ : files} }. They keys are the paths, the values are the dictionary ‘SubDirs’ and ‘Files’. Each are dictionaries with path as key and metadata as values S_ERROR in case of argument problems

pinFile(path, lifetime=86400)

Pin a staged file

Parameters
  • self – self reference

  • path (str) – path of list of paths to be pinned

  • lifetime (int) – pinning time in seconds (default 24h)

Returns

successful dict {url : token}, failed dict {url : message} S_ERROR in case of argument problems

prestageFile(path, lifetime=86400)

Issue prestage request for file(s)

Parameters
  • self – self reference

  • path (str) – path or list of paths to be prestaged

  • lifetime (int) – prestage lifetime in seconds (default 24h)

Returns

succesful dict { url : token } failed dict { url : message } S_ERROR in case of argument problems

prestageFileStatus(path)

Checking the staging status of file(s) on the storage

Parameters
  • self – self reference

  • path (dict) – dict { url : token }

Returns

succesful dict { url : bool } failed dict { url : message } S_ERROR in case of argument problems

putDirectory(path)

Puts one or more local directories to the physical storage together with all its files

Parameters
  • self – self reference

  • path (str) – dictionary { srm://… (destination) : localdir (source dir) }

Returns

successful and failed dictionaries. The keys are the paths, the values are dictionary {‘Files’ : amount of files uploaded, ‘Size’ : amount of data upload } S_ERROR in case of argument problems

putFile(path, sourceSize=0)

Put a copy of a local file or a file on another srm storage to a directory on the physical storage.

Parameters
  • path – dictionary { lfn (srm://…) : localFile }

  • sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…

Returns

Successful dict: { path : size } Failed dict: { path : error message } S_ERROR in case of argument problems

releaseFile(path)

Release a pinned file

Parameters
  • self – self reference

  • path (str) – PFN path { pfn : token } - pfn can be an empty string, then all files that have that same token get released. Just as you can pass an empty token string and a directory as pfn which then releases all the files in the directory an its subdirectories

Returns

successful dict {url : token}, failed dict {url : message} S_ERROR in case of argument problems

removeDirectory(path, recursive=False)

Remove a directory on the physical storage together with all its files and subdirectories.

Parameters
  • path – single or list of path (srm://..)

  • recursive – if True, we recursively delete the subdir

Returns

successful and failed dictionaries. The keys are the pathes, the values are dictionary {‘Files’: amount of files deleted, ‘Size’: amount of data deleted} S_ERROR in case of argument problems

removeFile(path)

Physically remove the file specified by path

A non existing file will be considered as successfully removed

Parameters

path (str) – path (or list of paths) on storage (srm://…)

Returns

Successful dict {path : True} Failed dict {path : error message} S_ERROR in case of argument problems

resetCurrentDirectory()

Reset the working directory to the base dir

setParameters(parameterDict)

Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)
updateURL(url, withWSUrl=False)

Update the URL according to the current SE parameters