FileCatalogDB

DIRAC FileCatalog Database

class DIRAC.DataManagementSystem.DB.FileCatalogDB.FileCatalogDB(databaseLocation='DataManagement/FileCatalogDB')

Bases: DIRAC.Core.Base.DB.DB

class ConnectionPool(host, user, passwd, port=3306, graceTime=600)

Bases: object

Management of connections per thread

__init__(host, user, passwd, port=3306, graceTime=600)

x.__init__(…) initializes x; see help(type(x)) for signature

clean(now=False)
get(dbName, retries=10)
transactionCommit(dbName)
transactionRollback(dbName)
transactionStart(dbName)
__init__(databaseLocation='DataManagement/FileCatalogDB')

Standard Constructor

addFile(lfns, credDict)

Add a new File

Parameters:
  • lfns (dict) – indexed on file’s LFN, the values are dictionaries which contains the attributes of the files (PFN, SE, Size, GUID, Checksum)
  • creDict – credential
Returns:

Successful/Failed dict.

addFileAncestors(lfns, credDict)

Add ancestor information for the given LFNs

addGroup(groupName, credDict)

Add a new group

Parameters:
  • groupName (str) – Name of the group
  • credDict – credential
addReplica(lfns, credDict)
Add a replica to a File
Parameters:
  • lfns (dict) – keys are LFN. The values are dict with key PFN and SE (e.g. {myLfn : {“PFN” : “myPfn”, “SE” : “mySE”}})
  • creDict – credential
Returns:

Successful/Failed dict.

addSE(seName, credDict)

Add a new StorageElement

Parameters:
  • seName (str) – Name of the StorageElement
  • credDict – credential
addUser(userName, credDict)

Add a new user

Parameters:
  • userName (str) – Name of the User
  • credDict – credential
buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None)

Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments

changePathGroup(paths, credDict, recursive=False)

Bulk method to change Group for the given paths

Parameters:
  • paths (dict) – dictionary < lfn : group >
  • credDict (dict) – dictionary of the caller credentials
  • recursive (boolean) – flag to apply the operation recursively
changePathMode(paths, credDict, recursive=False)

Bulk method to change Mode for the given paths

Parameters:
  • paths (dict) – dictionary < lfn : mode >
  • credDict (dict) – dictionary of the caller credentials
  • recursive (boolean) – flag to apply the operation recursively
changePathOwner(paths, credDict, recursive=False)

Bulk method to change Owner for the given paths

Parameters:
  • paths (dict) – dictionary < lfn : owner >
  • credDict (dict) – dictionary of the caller credentials
  • recursive (boolean) – flag to apply the operation recursively
countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of entries wit the given conditions

createDirectory(lfns, credDict)

Create new directories

Parameters:
  • lfns (list) – list of directories
  • creDict – credential
Returns:

Successful/Failed dict.

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

deleteGroup(groupName, credDict)

Delete a group

Parameters:
  • groupName (str) – Name of the group
  • credDict – credential
deleteSE(seName, credDict)

Delete a StorageElement

Parameters:
  • seName (str) – Name of the StorageElement
  • credDict – credential
deleteUser(userName, credDict)

Delete a user

Parameters:
  • userName (str) – Name of the User
  • credDict – credential
executeStoredProcedure(packageName, parameters, outputIds)
executeStoredProcedureWithCursor(packageName, parameters)
exists(lfns, credDict)
getCSOption(optionName, defaultValue=None)
getCatalogCounters(credDict)
getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDirectoryMetadata(lfns, credDict)

Get standard directory metadata

Parameters:
  • lfns (list) – list of directory paths
  • credDict (dict) – credentials
Returns:

Successful/Failed dict.

getDirectoryReplicas(lfns, allStatus, credDict)
getDirectorySize(lfns, longOutput, fromFiles, credDict)

Get the sizes of a list of directories

Parameters:
  • lfns (list) – list of LFN to check
  • creDict – credential
Returns:

Successful/Failed dict. The successful values are dictionaries indexed “LogicalFiles” (nb of files), “LogicalDirectories” (nb of dir) and “LogicalSize” (sum of File’s sizes)

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Get distinct values of a table attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Select “outFields” from “tableName” with condDict N records can match the condition return S_OK( tuple(Field,Value) ) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values.

getFileAncestors(lfns, depths, credDict)
getFileDescendents(lfns, depths, credDict)
getFileDetails(lfnList, credDict)

Get all the metadata for the given files

getFileMetadata(lfns, credDict)

Gets the metadata of a list of lfns

Parameters:
  • lfns (list) – list of LFN to check
  • creDict – credential
Returns:

Successful/Failed dict.

getFileSize(lfns, credDict)

Gets the size of a list of lfns

Parameters:
  • lfns (list) – list of LFN to check
  • creDict – credential
Returns:

Successful/Failed dict.

getGroups(credDict)

Returns the list of groups

Parameters:credDict – credential
Returns:dictionary indexed on the group name
getLFNForGUID(guids, credDict)

Gets the lfns that match a list of guids

Parameters:
  • lfns (list) – list of guid to look for
  • creDict – credential
Returns:

S_OK({guid:lfn}) dict.

getPathPermissions(lfns, credDict)

Get permissions for the given user/group to manipulate the given lfns

getReplicaStatus(lfns, credDict)

Gets the status of a list of replicas

Parameters:
  • lfns (dict) – <lfn, se name>
  • creDict – credential
Returns:

Successful/Failed dict.

getReplicas(lfns, allStatus, credDict)

Gets the list of replicas of a list of lfns

Parameters:
  • lfns (list) – list of LFN to check
  • allStatus – if all the status are visible, or only those defined in config[‘ValidReplicaStatus’]
  • creDict – credential
Returns:

Successful/Failed dict. Successful is indexed on the LFN, and the values are dictionary with the SEName as keys

getSEDump(seName)
Return all the files at a given SE, together with checksum and size
Parameters:seName – name of the StorageElement
Returns:S_OK with list of tuples (lfn, checksum, size)
getUsers(credDict)

Returns the list of users

Parameters:credDict – credential
Returns:dictionary indexed on the user name
hasAccess(opType, paths, credDict)

Get permissions for the given user/group to execute the given operation on the given paths

returns Successful dict with True/False

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None)

Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

isDirectory(lfns, credDict)

Checks whether a list of LFNS are directories or not

Parameters:
  • lfns (list) – list of LFN to check
  • creDict – credential
Returns:

Successful/Failed dict. The values of the successful dict are True or False whether it’s a dir or not

isFile(lfns, credDict)

Checks whether a list of LFNS are files or not

Parameters:
  • lfns (list) – list of LFN to check
  • creDict – credential
Returns:

Successful/Failed dict. The values of the successful dict are True or False whether it’s a file or not

listDirectory(lfns, credDict, verbose=False)

List directories

Parameters:
  • lfns (list) – list of directories
  • creDict – credential
Returns:

Successful/Failed dict. The successful values are dictionaries indexed “Files”, “Datasets”, “Subdirs” and “Links”

rebuildDirectoryUsage()

Rebuild DirectoryUsage table from scratch

removeDirectory(lfns, credDict)

Remove directories

Parameters:
  • lfns (list) – list of directories
  • creDict – credential
Returns:

Successful/Failed dict.

removeFile(lfns, credDict)
Remove files
Parameters:
  • lfns (list) – list of LFNs to remove
  • creDict – credential
Returns:

Successful/Failed dict.

removeMetadata(pathMetadataDict, credDict)

Remove metadata for the given paths

removeReplica(lfns, credDict)
Remove replicas
Parameters:
  • lfns (dict) – keys are LFN. The values are dict with key PFN and SE (e.g. {myLfn : {“PFN” : “myPfn”, “SE” : “mySE”}})
  • creDict – credential
Returns:

Successful/Failed dict.

repairCatalog(directoryFlag=True, credDict={})

Repair catalog inconsistencies

setConfig(databaseConfig)
setFileStatus(lfns, credDict)

Set the status of a File

Parameters:
  • lfns (dict) – dict indexed on the LFNs. The values are the status (should be in config[‘ValidFileStatus’])
  • creDict – credential
Returns:

Successful/Failed dict.

setMetadata(path, metadataDict, credDict)

Add metadata to the given path

setMetadataBulk(pathMetadataDict, credDict)

Add metadata for the given paths

setReplicaHost(lfns, credDict)
setReplicaStatus(lfns, credDict)

Set the status of a Replicas

Parameters:
  • lfns (dict) – dict indexed on the LFNs. The values are dict with keys “SE” and “Status” (that has to be in config[‘ValidReplicaStatus’])
  • creDict – credential
Returns:

Successful/Failed dict.

setUmask(umask)
transactionCommit()
transactionRollback()
transactionStart()
updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.