S3Storage

Configuration of an S3 storage Like others, but in protocol S3 add:

  • SecureConnection: true if https, false otherwise

  • Aws_access_key_id

  • Aws_secret_access_key

if the Aws variables are not defined, it will try to go throught the S3Gateway

The key of the objects are the LFN without trailing path. The Path should be the BucketName

class DIRAC.Resources.Storage.S3Storage.S3Storage(storageName, parameters)

Bases: StorageBase

class StorageBase
DYNAMIC_OPTIONS = {}
PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']
__init__(storageName, parameters)
changeDirectory(directory)

Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False)

Construct URL from the given LFN according to the VO convention for the primary protocol of the storage plagin

Parameters:

  • lfn (str) – file LFN

  • withWSUrl (boolean) – flag to include the web service part into the resulting URL

Return result:

result[‘Value’] - resulting URL

createDirectory(urls)
Create directory on the storage.

S3 does not have such a concept, but we return OK for everything

Parameters:

urls – list of urls to be created on the storage

Returns:

Always Successful dict {path : True }

createPresignedUrl(urls, s3_method, expiration=3600)

Generate a presigned URL to share an S3 object

Parameters:

  • urls – urls for which to generate a presigned URL. If s3_method is put_object, it must be a dict <url:Fields> where fields are the metadata of the file (see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_post) # pylint: disable=line-too-long # noqa

  • s3_method – name of the method for which to generate a presigned URL

  • expiration – Time in seconds for the presigned URL to remain valid

Returns:

Presigned URL as string. If error, returns None.

exists(urls)

Check if the urls exists on the storage

Parameters:

urls – list of URLs

Returns:

Failed dictionary: {url : error message} Successful dictionary: {url : bool} S_ERROR in case of argument problems

getCurrentDirectory()

Get the current directory

getCurrentURL(fileName)

Obtain the current file URL from the current working directory and the filename

Parameters:

  • self – self reference

  • fileName (str) – path on storage

static getDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

static getDirectoryMetadata(*_args, **_kwargs)

Generic method for unavailable method on S3

static getDirectorySize(*_args, **_kwargs)

Generic method for unavailable method on S3

getEndpoint()
This will get endpoint of the storage. It basically is the same as getURLBase()

but without the basePath

Returns:

‘proto://hostname<:port>’

getFile(urls, localPath=False)

Make a local copy of the urls.

Parameters:

  • urls – list of urls on the storage

  • localPath – destination folder. Default is from current directory

Returns:

  • Successful dict: {path : size}

  • Failed dict: {path : errorMessage}

  • S_ERROR in case of argument problems

getFileMetadata(urls)

Get metadata associated to the file(s)

Parameters:

urls – list of urls on the storage

Returns:

  • successful dict { path : metadata }

  • failed dict { path : error message }

  • S_ERROR in case of argument problems

getFileSize(urls)

Get the physical size of the given file

Parameters:

urls – list of urls on the storage

Returns:

  • Successful dict {path : size}

  • Failed dict {path : error message }

  • S_ERROR in case of argument problem

getName()

The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters:

occupancyLFN – (mandatory named argument) LFN of the json file.

Returns:

S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB

getParameters()

Get the parameters with which the storage was instantiated

getTransportURL(urls, protocols)
Get a transport URL for given urls

If http/https is requested, the URLs will be valid for 24hours

Parameters:

  • urls (dict) – s3 urls

  • protocols (list) –

    a list of acceptable transport protocols in priority order. In practice, besides ‘s3’, it can only be:

    • ’https’ if secureConnection is True

    • ’http’ othewise

Returns:

succ/failed dict url with required protocol

getURLBase(withWSUrl=False)

This will get the URL base. This is then appended with the LFN in DIRAC convention.

Parameters:

  • self – self reference

  • withWSUrl (bool) – flag to include Web Service part of the url

Returns:

URL

getWLCGTokenPath(lfn: str, wlcgTokenBasePath: str) → str

Returns the path expected to be in a WLCG token It basically consists of basepath - tokenBasePath  + LFN The tokenBasePath is a configuration on the storage side.

static isDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

isFile(urls)

Check if the urls provided are a file or not

In practice, if the object exists, it is necessarily a file

Parameters:

urls – list of urls to be checked

Returns:

  • Failed dict: {path : error message}

  • Successful dict: {path : bool}

  • S_ERROR in case of argument problems

isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters:

  • self – self reference

  • url (str) – URL

isURL(path)

Guess if the path looks like a URL

Parameters:

  • self – self reference

  • path (string) – input file LFN or URL

Returns boolean:

True if URL, False otherwise

static listDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

static notAvailable(*_args, **_kwargs)

Generic method for unavailable method on S3

pluginName = 'S3'
prestageFile(*parms, **kws)

Issue prestage request for file

prestageFileStatus(*parms, **kws)

Obtain the status of the prestage request

static putDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

putFile(urls, sourceSize=0)

Upload a local file.

..warning:: no 3rd party copy possible

Parameters:

  • urls – dictionary { urls : localFile }

  • sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…

Returns:

  • Successful dict: { path : size }

  • Failed dict: { path : error message }

  • S_ERROR in case of argument problems

releaseFile(*parms, **kws)

Release the file on the destination storage element

static removeDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

removeFile(urls)

Physically remove the file specified by keys

A non existing file will be considered as successfully removed

Parameters:

urls – list of urls on the storage

Returns:

  • Successful dict {path : True}

  • Failed dict {path : error message}

  • S_ERROR in case of argument problems

resetCurrentDirectory()

Reset the working directory to the base dir

setParameters(parameterDict)

Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)
updateURL(url, withWSUrl=False)

Update the URL according to the current SE parameters