GFAL2_StorageBase¶
- mod
GFAL2_StorageBase
- synopsis
GFAL2 class from StorageElement using gfal2. Other modules can inherit from this use the gfal2 methods.
- TODO
When we are totally migrated to python 3, we can explicitely remove the str cast of all the ctx calls. They were added because we would receive unicode from DIRAC https servers and boost python does not play well with that.
Environment Variables:
DIRAC_GFAL_GRIDFTP_ENABLE_IPV6: this should be exported and set to false on pure ipv4 nodes because of the globus bug
DIRAC_GFAL_GRIDFTP_SESSION_REUSE: This should be exported and set to true in server bashrc files for efficiency reasons.
- class DIRAC.Resources.Storage.GFAL2_StorageBase.GFAL2_StorageBase(storageName, parameters)¶
Bases:
StorageBase
- class GFAL2_StorageBase¶
SRM v2 interface to StorageElement using gfal2
- DYNAMIC_OPTIONS = {}¶
- PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']¶
- __init__(storageName, parameters)¶
c’tor
- changeDirectory(directory)¶
Change the directory to the supplied directory
- constructURLFromLFN(lfn, withWSUrl=False)¶
Construct URL from the given LFN according to the VO convention for the primary protocol of the storage plagin
- Parameters
lfn (str) – file LFN
withWSUrl (boolean) – flag to include the web service part into the resulting URL
- Return result
result[‘Value’] - resulting URL
- createDirectory(path)¶
Create directory on the storage
- Parameters
self – self reference
path (str) – path to be created on the storage (pfn : srm://…)
- Returns
Successful dict {path : True } Failed dict {path : error message } S_ERROR in case of argument problems
- exists(path)¶
Check if the path exists on the storage
- Parameters
self – self reference
path (str) – path or list of paths to be checked
- Returns
Failed dictionary: {pfn : error message} Successful dictionary: {pfn : bool} S_ERROR in case of argument problems
- getCurrentDirectory()¶
Get the current directory
- getCurrentURL(fileName)¶
Obtain the current file URL from the current working directory and the filename
- Parameters
self – self reference
fileName (str) – path on storage
- getDirectory(path, localPath=False)¶
get a directory from the SE to a local path with all its files and subdirectories
- Parameters
- Returns
successful and failed dictionaries. The keys are the paths, the values are dictionary {‘Files’: amount of files downloaded, ‘Size’ : amount of data downloaded} S_ERROR in case of argument problems
- getDirectoryMetadata(path)¶
Get metadata for the directory(ies) provided
- Parameters
self – self reference
path (str) – path (or list of paths) on storage (srm://…)
- Returns
Successful dict {path : metadata} Failed dict {path : errStr} S_ERROR in case of argument problems
- getDirectorySize(path)¶
Get the size of the directory on the storage
Warning
it is not recursive
- Parameters
self – self reference
path (str) – path or list of paths on storage (srm://…)
- Returns
list of successful and failed dictionaries, both indexed by the path
In the failed, the value is the error message
In the successful the values are dictionaries:
Files : amount of files in the dir
Size : summed up size of all files
subDirs : amount of sub dirs
S_ERROR in case of argument problems
- getEndpoint()¶
- This will get endpoint of the storage. It basically is the same as
getURLBase()
but without the basePath
- Returns
‘proto://hostname<:port>’
- This will get endpoint of the storage. It basically is the same as
- getFile(path, localPath=False)¶
Make a local copy of storage :path:
- Parameters
self – self reference
path (str) – path (or list of paths) on storage (srm://…)
localPath – destination folder. Default is from current directory
- Returns
Successful dict: {path : size} Failed dict: {path : errorMessage} S_ERROR in case of argument problems
- getFileMetadata(path)¶
Get metadata associated to the file(s)
- Parameters
self – self reference
path (str) – path (or list of paths) on the storage (srm://…)
- Returns
successful dict { path : metadata } failed dict { path : error message } S_ERROR in case of argument problems
- getFileSize(path)¶
Get the physical size of the given file
- Parameters
self – self reference
path – path (or list of path) on storage (pfn : srm://…)
- Returns
Successful dict {path : size} Failed dict {path : error message } S_ERROR in case of argument problems
- getName()¶
The name with which the storage was instantiated
- getOccupancy(**kwargs)¶
Get the StorageElement occupancy info in MB.
This generic implementation download a json file supposed to contain the necessary info.
- Parameters
occupancyLFN – (mandatory named argument) LFN of the json file.
- Returns
S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB
- getParameters()¶
Get the parameters with which the storage was instantiated
- getTransportURL(pathDict, protocols)¶
Get a transport URL for a given URL. For a simple storage plugin it is just returning input URL if the plugin protocol is one of the requested protocols
- Parameters
pathDict (dict) – URL obtained from File Catalog or constructed according to convention
protocols (python:list) – a list of acceptable transport protocols in priority order
- getURLBase(withWSUrl=False)¶
This will get the URL base. This is then appended with the LFN in DIRAC convention.
- Parameters
self – self reference
withWSUrl (bool) – flag to include Web Service part of the url
- Returns
URL
- isDirectory(path)¶
check if the path provided is a directory or not
- Parameters
self – self reference
str – path or list of paths to be checked ( ‘srm://…’)
- Returns
dict ‘Failed’ : failed, ‘Successful’ : succesful S_ERROR in case of argument problems
- isFile(path)¶
Check if the path provided is a file or not
- Parameters
self – self reference
str – path or list of paths to be checked ( ‘srm://…’)
- Returns
Failed dict: {path : error message} Successful dict: {path : bool} S_ERROR in case of argument problems
- isNativeURL(url)¶
Check if URL :url: is valid for :self.protocol:
- Parameters
self – self reference
url (str) – URL
- isOK()¶
- isURL(path)¶
Guess if the path looks like a URL
- Parameters
self – self reference
path (string) – input file LFN or URL
- Returns boolean
True if URL, False otherwise
- listDirectory(path)¶
List the content of the path provided
- Parameters
path (str) – single or list of paths (srm://…)
- Returns
failed dict {path : message } successful dict { path : {‘SubDirs’ : subDirs, ‘Files’ : files} }. They keys are the paths, the values are the dictionary ‘SubDirs’ and ‘Files’. Each are dictionaries with path as key and metadata as values S_ERROR in case of argument problems
- pinFile(path, lifetime=86400)¶
Pin a staged file
- prestageFile(path, lifetime=86400)¶
Issue prestage request for file(s)
- prestageFileStatus(path)¶
Checking the staging status of file(s) on the storage
- Parameters
self – self reference
path (dict) – dict { url : token }
- Returns
succesful dict { url : bool } failed dict { url : message } S_ERROR in case of argument problems
- putDirectory(path)¶
Puts one or more local directories to the physical storage together with all its files
- Parameters
self – self reference
path (str) – dictionary { srm://… (destination) : localdir (source dir) }
- Returns
successful and failed dictionaries. The keys are the paths, the values are dictionary {‘Files’ : amount of files uploaded, ‘Size’ : amount of data upload } S_ERROR in case of argument problems
- putFile(path, sourceSize=0)¶
Put a copy of a local file or a file on another srm storage to a directory on the physical storage.
- Parameters
path – dictionary { lfn (srm://…) : localFile }
sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…
- Returns
Successful dict: { path : size } Failed dict: { path : error message } S_ERROR in case of argument problems
- releaseFile(path)¶
Release a pinned file
- Parameters
self – self reference
path (str) – PFN path { pfn : token } - pfn can be an empty string, then all files that have that same token get released. Just as you can pass an empty token string and a directory as pfn which then releases all the files in the directory an its subdirectories
- Returns
successful dict {url : token}, failed dict {url : message} S_ERROR in case of argument problems
- removeDirectory(path, recursive=False)¶
Remove a directory on the physical storage together with all its files and subdirectories.
- Parameters
path – single or list of path (srm://..)
recursive – if True, we recursively delete the subdir
- Returns
successful and failed dictionaries. The keys are the pathes, the values are dictionary {‘Files’: amount of files deleted, ‘Size’: amount of data deleted} S_ERROR in case of argument problems
- removeFile(path)¶
Physically remove the file specified by path
A non existing file will be considered as successfully removed
- Parameters
path (str) – path (or list of paths) on storage (srm://…)
- Returns
Successful dict {path : True} Failed dict {path : error message} S_ERROR in case of argument problems
- resetCurrentDirectory()¶
Reset the working directory to the base dir
- setParameters(parameterDict)¶
Set standard parameters, method can be overriden in subclasses to process specific parameters
- setStorageElement(se)¶
- updateURL(url, withWSUrl=False)¶
Update the URL according to the current SE parameters