FileCatalogHandler

FileCatalogHandler is a simple Replica and Metadata Catalog service in the DIRAC framework

class DIRAC.DataManagementSystem.Service.FileCatalogHandler.FileCatalogHandler(handlerInitDict, trid)

Bases: FileCatalogHandlerMixin, RequestHandler

__init__(handlerInitDict, trid)

Constructor

Parameters:

• handlerInitDict (dictionary) – Information vars for the service

• trid (object) – Transport to use

auth_ping = ['all']

auth_refreshConfiguration = ['CSAdministrator']

auth_whoami = ['all']

export_addDataset(datasets)

Add a new dynamic dataset defined by its meta query

export_addDatasetAnnotation(datasetDict)

Add annotation to an already created dataset

export_addFile(lfns)

Register supplied files

export_addFileAncestors(lfns)

Add file ancestor information for the given list of LFNs

export_addGroup(groupName)

Add a new group to the File Catalog

export_addMetadataField(fieldName, fieldType, metaType='-d')

Add a new metadata field of the given type

export_addMetadataSet(setName, setDict)

Add a new metadata set

export_addReplica(lfns)

Register supplied replicas

export_addUser(userName)

Add a new user to the File Catalog

export_changePathGroup(lfns, recursive=False)

Get replica info for the given list of LFNs

export_changePathMode(lfns, recursive=False)

Get replica info for the given list of LFNs

export_changePathOwner(lfns, recursive=False)

Get replica info for the given list of LFNs

export_checkDataset(datasets)

Check the given dynamic dataset for changes since its definition

export_createDirectory(lfns)

Create the supplied directories

export_deleteGroup(groupName)

Delete group from the File Catalog

export_deleteMetadataField(fieldName)

Delete the metadata field

export_deleteUser(userName)

Delete user from the File Catalog

static export_echo(data)

This method is used for testing performance of the service

Parameters:

data (str) – data to be sent back to the caller

Returns:

S_OK, Value is the input data

export_exists(lfns)

Check whether the supplied paths exists

export_findDirectoriesByMetadata(metaDict, path='/')

Find all the directories satisfying the given metadata set

export_findFilesByMetadata(metaDict, path='/')

Find all the files satisfying the given metadata set

export_findFilesByMetadataDetailed(metaDict, path='/')

Find all the files satisfying the given metadata set

export_findFilesByMetadataWeb(metaDict, path, startItem, maxItems)

Find files satisfying the given metadata set

export_freezeDataset(datasets)

Freeze the contents of the dataset making it effectively static

export_getCatalogCounters()

Get the number of registered directories, files and replicas in various tables

export_getCompatibleMetadata(metaDict, path='/')

Get metadata values compatible with the given metadata subset

export_getDatasetAnnotation(datasets)

Get annotation of the given datasets

export_getDatasetFiles(datasets)

Get lfns in the given dataset

export_getDatasetParameters(datasets)

Get parameters of the given dynamic dataset as they are stored in the database

export_getDatasets(datasets)

Get parameters of the given dynamic dataset as they are stored in the database

export_getDirectoryDump(lfns)

Recursively list the contents of supplied directories

export_getDirectoryMetadata(lfns)

Get the metadata of the supplied directory

export_getDirectoryReplicas(lfns, allStatus=False)

Get replicas for files in the supplied directory

export_getDirectorySize(lfns, longOut=False, fromFiles=False, recursiveSum=True)

Get the size of the supplied directory

export_getDirectoryUserMetadata(path)

Get all the metadata valid for the given directory path

export_getFileAncestors(lfns, depths)

Get the status for the supplied replicas

export_getFileDescendents(lfns, depths)

Get the status for the supplied replicas

export_getFileDetails(lfns)

Get all the metadata associated to supplied lfns, including user metadata

export_getFileMetadata(lfns)

Get the metadata associated to supplied lfns

export_getFileSize(lfns)

Get the size associated to supplied lfns

export_getFileUserMetadata(path)

Get all the metadata valid for the given file

export_getGroups()

Get all the groups defined in the File Catalog

export_getLFNForGUID(guids)

Get the matching lfns for given guids

export_getMetadataFields()

Get all the metadata fields

export_getMetadataSet(setName, expandFlag)

Add a new metadata set

export_getPathPermissions(lfns)

Determine the ACL information for a supplied path

export_getReplicaStatus(lfns)

Get the status for the supplied replicas

export_getReplicas(lfns, allStatus=False)

Get replicas for supplied lfns

export_getReplicasByMetadata(metaDict, path='/', allStatus=False)

Find all the files satisfying the given metadata set

export_getUsers()

Get all the users defined in the File Catalog

export_hasAccess(paths, opType)

Determine if the given op can be performed on the paths The OpType is all the operations exported

The reason for the param types is backward compatibility. Between v6r14 and v6r15, the signature of hasAccess has changed, and the two parameters were swapped.

export_isDirectory(lfns)

Determine whether supplied path is a directory

export_isFile(lfns)

Check whether the supplied lfns are files

classmethod export_isOK()

returns S_OK if DB is connected

export_listDirectory(lfns, verbose)

List the contents of supplied directories

export_ping()

export_rebuildDirectoryUsage()

Rebuild DirectoryUsage table from scratch

static export_refreshConfiguration(fromMaster)

Force refreshing the configuration data

Parameters:

fromMaster (bool) – flag to refresh from the master configuration service

export_releaseDataset(datasets)

Release the contents of the frozen dataset allowing changes in its contents

export_removeDataset(datasets)

Check the given dynamic dataset for changes since its definition

export_removeDirectory(lfns)

Remove the supplied directories

export_removeFile(lfns)

Remove the supplied lfns

export_removeMetadata(pathMetadataDict)

Remove the specified metadata for the given path

export_removeReplica(lfns)

Remove the supplied replicas

export_repairCatalog()

Repair the catalog inconsistencies

export_setFileStatus(lfns)

Remove the supplied lfns

export_setMetadata(path, metadatadict)

Set metadata parameter for the given path

export_setMetadataBulk(pathMetadataDict)

Set metadata parameter for the given path

export_setReplicaHost(lfns)

Change the registered SE for the supplied replicas

export_setReplicaStatus(lfns)

Set the status for the supplied replicas

export_updateDataset(datasets)

Update the given dynamic dataset for changes since its definition

export_whoami()

A simple whoami, returns all credential dictionary, except certificate chain object.

findFilesByMetadataWeb(metaDict, path, startItem, maxItems)

Find all the files satisfying the given metadata set

classmethod getCSOption(optionName, defaultValue=False)

Get an option from the CS section of the services

Returns:

Value for serviceSection/optionName in the CS being defaultValue the default

getRemoteAddress()

Get the address of the remote peer.

Returns:

Address of remote peer.

getRemoteCredentials()

Get the credentials of the remote peer.

Returns:

Credentials dictionary of remote peer.

getSEDump(seNames)

Return all the files at given SEs, together with checksum and size

Parameters:

seNames – StorageElement names

Returns:

S_OK with list of tuples (SEName, lfn, checksum, size)

initialize()

Initialize this instance of the handler (to be overwritten)

classmethod initializeHandler(serviceInfo)

Handler initialization

srv_disconnect(trid=None)

classmethod srv_disconnectClient(trid)

srv_getActionTuple()

classmethod srv_getCSOption(optionName, defaultValue=False)

Get an option from the CS section of the services

Returns:

Value for serviceSection/optionName in the CS being defaultValue the default

srv_getClientSetup()

srv_getClientVO()

srv_getClientVersion()

srv_getFormattedRemoteCredentials()

classmethod srv_getMonitor()

srv_getRemoteAddress()

Get the address of the remote peer.

Returns:

Address of remote peer.

srv_getRemoteCredentials()

Get the credentials of the remote peer.

Returns:

Credentials dictionary of remote peer.

classmethod srv_getServiceName()

srv_getTransportID()

classmethod srv_getURL()

classmethod srv_msgCreate(msgName)

srv_msgReply(msgObj)

classmethod srv_msgSend(trid, msgObj)

transfer_bulkFromClient(bulkId, token, bulkSize, fileHelper)

transfer_bulkToClient(bulkId, token, fileHelper)

transfer_fromClient(fileId, token, fileSize, fileHelper)

transfer_listBulk(bulkId, token, fileHelper)

transfer_toClient(jsonSENames, token, fileHelper)

This method used to transfer the SEDump to the client, formated as CSV with ‘|’ separation

Parameters:

seName – name of the se to dump

Returns:

the result of the FileHelper

types_addDataset = [<class 'dict'>]

types_addDatasetAnnotation = [<class 'dict'>]

types_addFile = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_addFileAncestors = [<class 'dict'>]

types_addGroup = [<class 'str'>]

types_addMetadataField = [<class 'str'>, <class 'str'>, <class 'str'>]

types_addMetadataSet = [<class 'str'>, <class 'dict'>]

types_addReplica = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_addUser = [<class 'str'>]

types_changePathGroup = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_changePathMode = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_changePathOwner = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_checkDataset = [<class 'dict'>]

types_createDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_deleteGroup = [<class 'str'>]

types_deleteMetadataField = [<class 'str'>]

types_deleteUser = [<class 'str'>]

types_echo = [<class 'str'>]

types_exists = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_findDirectoriesByMetadata = [<class 'dict'>]

types_findFilesByMetadata = [<class 'dict'>, <class 'str'>]

types_findFilesByMetadataDetailed = [<class 'dict'>, <class 'str'>]

types_findFilesByMetadataWeb = [<class 'dict'>, <class 'str'>, <class 'int'>, <class 'int'>]

types_freezeDataset = [<class 'dict'>]

types_getCatalogCounters = []

types_getCompatibleMetadata = [<class 'dict'>, <class 'str'>]

types_getDatasetAnnotation = [<class 'dict'>]

types_getDatasetFiles = [<class 'dict'>]

types_getDatasetParameters = [<class 'dict'>]

types_getDatasets = [<class 'dict'>]

types_getDirectoryDump = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getDirectoryMetadata = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getDirectoryReplicas = [[<class 'list'>, <class 'dict'>, <class 'str'>], <class 'bool'>]

types_getDirectorySize = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getDirectoryUserMetadata = [<class 'str'>]

types_getFileAncestors = [[<class 'list'>, <class 'dict'>], (<class 'list'>, <class 'int'>)]

types_getFileDescendents = [[<class 'list'>, <class 'dict'>], (<class 'list'>, <class 'int'>)]

types_getFileDetails = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getFileMetadata = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getFileSize = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getFileUserMetadata = [<class 'str'>]

types_getGroups = []

types_getLFNForGUID = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getMetadataFields = []

types_getMetadataSet = [<class 'str'>, <class 'bool'>]

types_getPathPermissions = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getReplicaStatus = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getReplicas = [[<class 'list'>, <class 'dict'>, <class 'str'>], <class 'bool'>]

types_getReplicasByMetadata = [<class 'dict'>, <class 'str'>, <class 'bool'>]

types_getUsers = []

types_hasAccess = [[<class 'str'>, <class 'dict'>], [<class 'str'>, <class 'list'>, <class 'dict'>]]

types_isDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_isFile = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_isOK = []

types_listDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>], <class 'bool'>]

types_ping = []

types_rebuildDirectoryUsage = []

types_refreshConfiguration = [<class 'bool'>]

types_releaseDataset = [<class 'dict'>]

types_removeDataset = [<class 'dict'>]

types_removeDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_removeFile = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_removeMetadata = [<class 'dict'>]

types_removeReplica = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_repairCatalog = []

types_setFileStatus = [<class 'dict'>]

types_setMetadata = [<class 'str'>, <class 'dict'>]

types_setMetadataBulk = [<class 'dict'>]

types_setReplicaHost = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_setReplicaStatus = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_updateDataset = [<class 'dict'>]

types_whoami = []

class DIRAC.DataManagementSystem.Service.FileCatalogHandler.FileCatalogHandlerMixin

Bases: object

A simple Replica and Metadata Catalog service.

export_addDataset(datasets)

Add a new dynamic dataset defined by its meta query

export_addDatasetAnnotation(datasetDict)

Add annotation to an already created dataset

export_addFile(lfns)

Register supplied files

export_addFileAncestors(lfns)

Add file ancestor information for the given list of LFNs

export_addGroup(groupName)

Add a new group to the File Catalog

export_addMetadataField(fieldName, fieldType, metaType='-d')

Add a new metadata field of the given type

export_addMetadataSet(setName, setDict)

Add a new metadata set

export_addReplica(lfns)

Register supplied replicas

export_addUser(userName)

Add a new user to the File Catalog

export_changePathGroup(lfns, recursive=False)

Get replica info for the given list of LFNs

export_changePathMode(lfns, recursive=False)

Get replica info for the given list of LFNs

export_changePathOwner(lfns, recursive=False)

Get replica info for the given list of LFNs

export_checkDataset(datasets)

Check the given dynamic dataset for changes since its definition

export_createDirectory(lfns)

Create the supplied directories

export_deleteGroup(groupName)

Delete group from the File Catalog

export_deleteMetadataField(fieldName)

Delete the metadata field

export_deleteUser(userName)

Delete user from the File Catalog

export_exists(lfns)

Check whether the supplied paths exists

export_findDirectoriesByMetadata(metaDict, path='/')

Find all the directories satisfying the given metadata set

export_findFilesByMetadata(metaDict, path='/')

Find all the files satisfying the given metadata set

export_findFilesByMetadataDetailed(metaDict, path='/')

Find all the files satisfying the given metadata set

export_findFilesByMetadataWeb(metaDict, path, startItem, maxItems)

Find files satisfying the given metadata set

export_freezeDataset(datasets)

Freeze the contents of the dataset making it effectively static

export_getCatalogCounters()

Get the number of registered directories, files and replicas in various tables

export_getCompatibleMetadata(metaDict, path='/')

Get metadata values compatible with the given metadata subset

export_getDatasetAnnotation(datasets)

Get annotation of the given datasets

export_getDatasetFiles(datasets)

Get lfns in the given dataset

export_getDatasetParameters(datasets)

Get parameters of the given dynamic dataset as they are stored in the database

export_getDatasets(datasets)

Get parameters of the given dynamic dataset as they are stored in the database

export_getDirectoryDump(lfns)

Recursively list the contents of supplied directories

export_getDirectoryMetadata(lfns)

Get the metadata of the supplied directory

export_getDirectoryReplicas(lfns, allStatus=False)

Get replicas for files in the supplied directory

export_getDirectorySize(lfns, longOut=False, fromFiles=False, recursiveSum=True)

Get the size of the supplied directory

export_getDirectoryUserMetadata(path)

Get all the metadata valid for the given directory path

export_getFileAncestors(lfns, depths)

Get the status for the supplied replicas

export_getFileDescendents(lfns, depths)

Get the status for the supplied replicas

export_getFileDetails(lfns)

Get all the metadata associated to supplied lfns, including user metadata

export_getFileMetadata(lfns)

Get the metadata associated to supplied lfns

export_getFileSize(lfns)

Get the size associated to supplied lfns

export_getFileUserMetadata(path)

Get all the metadata valid for the given file

export_getGroups()

Get all the groups defined in the File Catalog

export_getLFNForGUID(guids)

Get the matching lfns for given guids

export_getMetadataFields()

Get all the metadata fields

export_getMetadataSet(setName, expandFlag)

Add a new metadata set

export_getPathPermissions(lfns)

Determine the ACL information for a supplied path

export_getReplicaStatus(lfns)

Get the status for the supplied replicas

export_getReplicas(lfns, allStatus=False)

Get replicas for supplied lfns

export_getReplicasByMetadata(metaDict, path='/', allStatus=False)

Find all the files satisfying the given metadata set

export_getUsers()

Get all the users defined in the File Catalog

export_hasAccess(paths, opType)

Determine if the given op can be performed on the paths The OpType is all the operations exported

The reason for the param types is backward compatibility. Between v6r14 and v6r15, the signature of hasAccess has changed, and the two parameters were swapped.

export_isDirectory(lfns)

Determine whether supplied path is a directory

export_isFile(lfns)

Check whether the supplied lfns are files

classmethod export_isOK()

returns S_OK if DB is connected

export_listDirectory(lfns, verbose)

List the contents of supplied directories

export_rebuildDirectoryUsage()

Rebuild DirectoryUsage table from scratch

export_releaseDataset(datasets)

Release the contents of the frozen dataset allowing changes in its contents

export_removeDataset(datasets)

Check the given dynamic dataset for changes since its definition

export_removeDirectory(lfns)

Remove the supplied directories

export_removeFile(lfns)

Remove the supplied lfns

export_removeMetadata(pathMetadataDict)

Remove the specified metadata for the given path

export_removeReplica(lfns)

Remove the supplied replicas

export_repairCatalog()

Repair the catalog inconsistencies

export_setFileStatus(lfns)

Remove the supplied lfns

export_setMetadata(path, metadatadict)

Set metadata parameter for the given path

export_setMetadataBulk(pathMetadataDict)

Set metadata parameter for the given path

export_setReplicaHost(lfns)

Change the registered SE for the supplied replicas

export_setReplicaStatus(lfns)

Set the status for the supplied replicas

export_updateDataset(datasets)

Update the given dynamic dataset for changes since its definition

findFilesByMetadataWeb(metaDict, path, startItem, maxItems)

Find all the files satisfying the given metadata set

getSEDump(seNames)

Return all the files at given SEs, together with checksum and size

Parameters:

seNames – StorageElement names

Returns:

S_OK with list of tuples (SEName, lfn, checksum, size)

classmethod initializeHandler(serviceInfo)

Handler initialization

types_addDataset = [<class 'dict'>]

types_addDatasetAnnotation = [<class 'dict'>]

types_addFile = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_addFileAncestors = [<class 'dict'>]

types_addGroup = [<class 'str'>]

types_addMetadataField = [<class 'str'>, <class 'str'>, <class 'str'>]

types_addMetadataSet = [<class 'str'>, <class 'dict'>]

types_addReplica = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_addUser = [<class 'str'>]

types_changePathGroup = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_changePathMode = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_changePathOwner = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_checkDataset = [<class 'dict'>]

types_createDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_deleteGroup = [<class 'str'>]

types_deleteMetadataField = [<class 'str'>]

types_deleteUser = [<class 'str'>]

types_exists = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_findDirectoriesByMetadata = [<class 'dict'>]

types_findFilesByMetadata = [<class 'dict'>, <class 'str'>]

types_findFilesByMetadataDetailed = [<class 'dict'>, <class 'str'>]

types_findFilesByMetadataWeb = [<class 'dict'>, <class 'str'>, <class 'int'>, <class 'int'>]

types_freezeDataset = [<class 'dict'>]

types_getCatalogCounters = []

types_getCompatibleMetadata = [<class 'dict'>, <class 'str'>]

types_getDatasetAnnotation = [<class 'dict'>]

types_getDatasetFiles = [<class 'dict'>]

types_getDatasetParameters = [<class 'dict'>]

types_getDatasets = [<class 'dict'>]

types_getDirectoryDump = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getDirectoryMetadata = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getDirectoryReplicas = [[<class 'list'>, <class 'dict'>, <class 'str'>], <class 'bool'>]

types_getDirectorySize = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getDirectoryUserMetadata = [<class 'str'>]

types_getFileAncestors = [[<class 'list'>, <class 'dict'>], (<class 'list'>, <class 'int'>)]

types_getFileDescendents = [[<class 'list'>, <class 'dict'>], (<class 'list'>, <class 'int'>)]

types_getFileDetails = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getFileMetadata = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getFileSize = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getFileUserMetadata = [<class 'str'>]

types_getGroups = []

types_getLFNForGUID = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getMetadataFields = []

types_getMetadataSet = [<class 'str'>, <class 'bool'>]

types_getPathPermissions = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getReplicaStatus = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_getReplicas = [[<class 'list'>, <class 'dict'>, <class 'str'>], <class 'bool'>]

types_getReplicasByMetadata = [<class 'dict'>, <class 'str'>, <class 'bool'>]

types_getUsers = []

types_hasAccess = [[<class 'str'>, <class 'dict'>], [<class 'str'>, <class 'list'>, <class 'dict'>]]

types_isDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_isFile = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_isOK = []

types_listDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>], <class 'bool'>]

types_rebuildDirectoryUsage = []

types_releaseDataset = [<class 'dict'>]

types_removeDataset = [<class 'dict'>]

types_removeDirectory = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_removeFile = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_removeMetadata = [<class 'dict'>]

types_removeReplica = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_repairCatalog = []

types_setFileStatus = [<class 'dict'>]

types_setMetadata = [<class 'str'>, <class 'dict'>]

types_setMetadataBulk = [<class 'dict'>]

types_setReplicaHost = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_setReplicaStatus = [[<class 'list'>, <class 'dict'>, <class 'str'>]]

types_updateDataset = [<class 'dict'>]