FileStorage

This is the File StorageClass, only meant to be used localy

class DIRAC.Resources.Storage.FileStorage.FileStorage(storageName, parameters)

Bases: StorageBase

class FileStorage

File storage, for local file management

DYNAMIC_OPTIONS = {}

PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']

__init__(storageName, parameters)

c’tor

Parameters:

• self – self reference

• storageName (str) – SE name

• protocol (str) – protocol to use

• rootdir (str) – base path for vo files

changeDirectory(directory)

Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False)

The URL of this file protocol is very special: it has no protocol in front, etc. So we overwrite the method

createDirectory(path)

Make a/several new directory on the physical storage

This method creates all the intermediate directory

Parameters:

• self – self reference

• path (str) – path (or list of path) on storage

Returns:

Successful dict {path : True} Failed dict {path : error message }

exists(path)

Check if the given path exists.

Parameters:

• self – self reference

• path – path (or list of path) on storage

Returns:

Failed dictionary: {pfn : errorMsg} Successful dictionary: {pfn : bool}

getCurrentDirectory()

Get the current directory

getCurrentURL(fileName)

Obtain the current file URL from the current working directory and the filename

Parameters:

• self – self reference

• fileName (str) – path on storage

getDirectory(path, localPath=False)

Get locally a directory from the physical storage together with all its files and subdirectories.

Parameters:

• path – path (or list of path) on storage

• localPath – local path where to store what is downloaded

Returns:

successful and failed dictionaries. The keys are the pathes, the values are dictionary {‘Files’: amount of files downloaded, ‘Size’: amount of data downloaded}

getDirectoryMetadata(path)

Get metadata associated to the directory(ies)

Parameters:

• self – self reference

• path – url (or list of urls) on storage

Returns:

Successful dict {path : metadata} Failed dict {path : error message }

getDirectorySize(path)

Get the size of the directory on the storage

Warning

the size is not recursive, and does not go into subfolders

Parameters:

• self – self reference

• path – path (or list of path) on storage

Returns:

list of successfull and failed dictionary, both indexed by the path

• In the failed, the value is the error message

• In the successful the values are dictionnaries :

  • Files : amount of files in the directory

  • Size : summed up size of files

  • subDirs : amount of sub directories

getEndpoint()

This will get endpoint of the storage. It basically is the same as getURLBase()

but without the basePath

Returns:

‘proto://hostname<:port>’

getFile(path, localPath=False)

make a local copy of a storage :path:

Parameters:

• self – self reference

• path (str) – path on storage

• localPath – if not specified, self.cwd

Returns:

Successful dict {path : size} Failed dict {path : error message }

getFileMetadata(path)

Get metadata associated to the file(s)

Parameters:

• self – self reference

• path – path (or list of path) on storage

Returns:

Successful dict {path : metadata} Failed dict {path : error message }

getFileSize(path)

Get the physical size of the given file

Parameters:

• self – self reference

• path – path (or list of path) on storage

Returns:

Successful dict {path : size} Failed dict {path : error message }

getName()

The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters:

occupancyLFN – (mandatory named argument) LFN of the json file.

Returns:

S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB

getParameters()

Get the parameters with which the storage was instantiated

getTransportURL(pathDict, protocols)

Get a transport URL for a given URL. For a simple storage plugin it is just returning input URL if the plugin protocol is one of the requested protocols

Parameters:

• pathDict (dict) – URL obtained from File Catalog or constructed according to convention

• protocols (python:list) – a list of acceptable transport protocols in priority order

getURLBase(withWSUrl=False)

This will get the URL base. This is then appended with the LFN in DIRAC convention.

Parameters:

• self – self reference

• withWSUrl (bool) – flag to include Web Service part of the url

Returns:

URL

getWLCGTokenPath(lfn: str, wlcgTokenBasePath: str) → str

Returns the path expected to be in a WLCG token It basically consists of basepath - tokenBasePath  + LFN The tokenBasePath is a configuration on the storage side.

isDirectory(path)

Check if the given path exists and it is a directory

Parameters:

• self – self reference

• path – path (or list of path) on storage

Returns:

Successful dict {path : boolean} Failed dict {path : error message }

isFile(path)

Check if the given path exists and it is a file

Parameters:

• self – self reference

• path – path (or list of path) on storage

Returns:

Successful dict {path : boolean} Failed dict {path : error message }

isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters:

• self – self reference

• url (str) – URL

isURL(path)

Guess if the path looks like a URL

Parameters:

• self – self reference

• path (string) – input file LFN or URL

Returns boolean:

True if URL, False otherwise

listDirectory(path)

List the supplied path

Warning

It is not recursive!

Parameters:

path – single or list of url

Returns:

successful and failed dictionaries. The keys are the pathes, the values are dictionary ‘SubDirs’ and ‘Files’. Each are dictionaries with path as key and metadata as values (for Files only, SubDirs has just True as value)

prestageFile(*parms, **kws)

Issue prestage request for file

prestageFileStatus(*parms, **kws)

Obtain the status of the prestage request

putDirectory(path)

puts a or several local directory to the physical storage together with all its files and subdirectories

Parameters:

• self – self reference

• path (dict) – dictionary {url : local dir}

Returns:

successful and failed dictionaries. The keys are the pathes, the values are dictionary {‘Files’: amount of files uploaded, ‘Size’: amount of data uploaded}

putFile(path, sourceSize=0)

Put a copy of the local file to the current directory on the physical storage

Parameters:

• path – dictionary {pfn : localFile}

• sourceSize – compares the size of the local and remote. You obviously run into trouble if you use a list of path… If the size do not match, remove the remote file

Returns:

Successful dict {path : size} Failed dict {path : error message }

releaseFile(*parms, **kws)

Release the file on the destination storage element

removeDirectory(path, recursive=False)

Remove a directory on the physical storage together with all its files and

subdirectories.

param path:

single or list of path

param bool recursive:

if True, we recursively delete the subdir

return:

successful and failed dictionaries. The keys are the pathes, the values are dictionary {‘Files’: amount of files deleted, ‘Size’: amount of data deleted}

Note

It is known that if recursive is False, the removal of a non existing directory is successful, while it is failed for recursive = True. That’s stupid, but well… I guess I have to keep the interface

removeFile(path)

Remove physically the file specified by its path

A non existing file will be considered as successfully removed.

Parameters:

path – path (or list of path) on storage

Returns:

Successful dict {path : True} Failed dict {path : error message }

resetCurrentDirectory()

Reset the working directory to the base dir

setParameters(parameterDict)

Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)

updateURL(url, withWSUrl=False)

Update the URL according to the current SE parameters