FileCatalogDB

DIRAC FileCatalog Database

class DIRAC.DataManagementSystem.DB.FileCatalogDB.FileCatalogDB(databaseLocation='DataManagement/FileCatalogDB', parentLogger=None)

Bases: DB

__init__(databaseLocation='DataManagement/FileCatalogDB', parentLogger=None)

c’tor

Parameters:

self – self reference

addFile(lfns, credDict)

Add a new File

Parameters:

• lfns (dict) – indexed on file’s LFN, the values are dictionaries which contains the attributes of the files (PFN, SE, Size, GUID, Checksum)

• creDict – credential

Returns:

Successful/Failed dict.

addFileAncestors(lfns, credDict)

Add ancestor information for the given LFNs

addGroup(groupName, credDict)

Add a new group

Parameters:

• groupName (str) – Name of the group

• credDict – credential

addReplica(lfns, credDict)

Add a replica to a File

Parameters:

• lfns (dict) – keys are LFN. The values are dict with key PFN and SE (e.g. {myLfn : {“PFN” : “myPfn”, “SE” : “mySE”}})

• creDict – credential

Returns:

Successful/Failed dict.

addSE(seName, credDict)

Add a new StorageElement

Parameters:

• seName (str) – Name of the StorageElement

• credDict – credential

addUser(userName, credDict)

Add a new user

Parameters:

• userName (str) – Name of the User

• credDict – credential

buildCondition(condDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, limit=False, greater=None, smaller=None, offset=None, useLikeQuery=False)

Build SQL condition statement from provided condDict and other extra check on a specified time stamp. The conditions dictionary specifies for each attribute one or a List of possible values greater and smaller are dictionaries in which the keys are the names of the fields, that are requested to be >= or < than the corresponding value. For compatibility with current usage it uses Exceptions to exit in case of invalid arguments For performing LIKE queries use the parameter useLikeQuery=True

changePathGroup(paths, credDict, recursive=False)

Bulk method to change Group for the given paths

Parameters:

• paths (dict) – dictionary < lfn : group >

• credDict (dict) – dictionary of the caller credentials

• recursive (boolean) – flag to apply the operation recursively

changePathMode(paths, credDict, recursive=False)

Bulk method to change Mode for the given paths

Parameters:

• paths (dict) – dictionary < lfn : mode >

• credDict (dict) – dictionary of the caller credentials

• recursive (boolean) – flag to apply the operation recursively

changePathOwner(paths, credDict, recursive=False)

Bulk method to change Owner for the given paths

Parameters:

• paths (dict) – dictionary < lfn : owner >

• credDict (dict) – dictionary of the caller credentials

• recursive (boolean) – flag to apply the operation recursively

countEntries(table, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of entries wit the given conditions

createDirectory(lfns, credDict)

Create new directories

Parameters:

• lfns (list) – list of directories

• creDict – credential

Returns:

Successful/Failed dict.

deleteEntries(tableName, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Delete rows from “tableName” with N records can match the condition if limit is not False, the given limit is set String type values will be appropriately escaped, they can be single values or lists of values.

deleteGroup(groupName, credDict)

Delete a group

Parameters:

• groupName (str) – Name of the group

• credDict – credential

deleteSE(seName, credDict)

Delete a StorageElement

Parameters:

• seName (str) – Name of the StorageElement

• credDict – credential

deleteUser(userName, credDict)

Delete a user

Parameters:

• userName (str) – Name of the User

• credDict – credential

executeStoredProcedure(packageName, parameters, outputIds, *, conn=None)

executeStoredProcedureWithCursor(packageName, parameters, *, conn=None)

exists(lfns, credDict)

getCSOption(optionName, defaultValue=None)

getCatalogCounters(credDict)

getCounters(table, attrList, condDict, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Count the number of records on each distinct combination of AttrList, selected with condition defined by condDict and time stamps

getDirectoryDump(lfns, credDict)

Get a dump of the directories

Parameters:

• lfns (list) – list of directories

• creDict – credential

Returns:

Successful/Failed dict. The successful values are dictionaries indexed “Files”, “Subdirs”

getDirectoryMetadata(lfns, credDict)

Get standard directory metadata

Parameters:

• lfns (list) – list of directory paths

• credDict (dict) – credentials

Returns:

Successful/Failed dict.

getDirectoryReplicas(lfns, allStatus, credDict)

getDirectorySize(lfns, longOutput, fromFiles, recursiveSum, credDict)

Get the sizes of a list of directories

Parameters:

• lfns (list) – list of LFN to check

• longOutput – if True, get also the physical size per SE (takes longer)

• fromFiles – if True, recompute the size from the file tables instead of the precomputed values (takes longer)

• recursiveSum – if True (default), takes into account the subdirectories

• creDict – credential

Returns:

Successful/Failed dict. The successful values are dictionaries indexed “LogicalFiles” (nb of files), “LogicalDirectories” (nb of dir) and “LogicalSize” (sum of File’s sizes)

getDistinctAttributeValues(table, attribute, condDict=None, older=None, newer=None, timeStamp=None, connection=False, greater=None, smaller=None)

Get distinct values of a table attribute under specified conditions

getFields(tableName, outFields=None, condDict=None, limit=False, conn=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None, useLikeQuery=False)

Select “outFields” from “tableName” with condDict N records can match the condition return S_OK(tuple(Field, Value)) if outFields is None all fields in “tableName” are returned if limit is not False, the given limit is set inValues are properly escaped using the _escape_string method, they can be single values or lists of values. if useLikeQuery=True, then conDict can return matched rows if “%” is defined inside conDict.

getFileAncestors(lfns, depths, credDict)

getFileDescendents(lfns, depths, credDict)

getFileDetails(lfnList, credDict)

Get all the metadata for the given files

getFileDetailsPublic(lfns, credDict)

Return all the metadata, including user defined, for those lfns that exist.

Returns:

S_OK with a dictionary of LFNs to detailed information

getFileMetadata(lfns, credDict)

Gets the metadata of a list of lfns

Parameters:

• lfns (list) – list of LFN to check

• creDict – credential

Returns:

Successful/Failed dict.

getFileSize(lfns, credDict)

Gets the size of a list of lfns

Parameters:

• lfns (list) – list of LFN to check

• creDict – credential

Returns:

Successful/Failed dict.

getGroups(credDict)

Returns the list of groups

Parameters:

credDict – credential

Returns:

dictionary indexed on the group name

getLFNForGUID(guids, credDict)

Gets the lfns that match a list of guids

Parameters:

• lfns (list) – list of guid to look for

• creDict – credential

Returns:

S_OK({guid:lfn}) dict.

getPathPermissions(lfns, credDict)

Get permissions for the given user/group to manipulate the given lfns

getReplicaStatus(lfns, credDict)

Gets the status of a list of replicas

Parameters:

• lfns (dict) – <lfn, se name>

• creDict – credential

Returns:

Successful/Failed dict.

getReplicas(lfns, allStatus, credDict)

Gets the list of replicas of a list of lfns

Parameters:

• lfns (list) – list of LFN to check

• allStatus – if all the status are visible, or only those defined in config[‘ValidReplicaStatus’]

• creDict – credential

Returns:

Successful/Failed dict. Successful is indexed on the LFN, and the values are dictionary with the SEName as keys

getSEDump(seNames)

Return all the files at given SEs, together with checksum and size

Parameters:

seName – list of StorageElement names

Returns:

S_OK with list of tuples (SEName, lfn, checksum, size)

getUsers(credDict)

Returns the list of users

Parameters:

credDict – credential

Returns:

dictionary indexed on the user name

hasAccess(opType, paths, credDict)

Get permissions for the given user/group to execute the given operation on the given paths

returns Successful dict with True/False

insertFields(tableName, inFields=None, inValues=None, conn=None, inDict=None)

Insert a new row in “tableName” assigning the values “inValues” to the fields “inFields”. String type values will be appropriately escaped.

isDirectory(lfns, credDict)

Checks whether a list of LFNS are directories or not

Parameters:

• lfns (list) – list of LFN to check

• creDict – credential

Returns:

Successful/Failed dict. The values of the successful dict are True or False whether it’s a dir or not

isFile(lfns, credDict)

Checks whether a list of LFNS are files or not

Parameters:

• lfns (list) – list of LFN to check

• creDict – credential

Returns:

Successful/Failed dict. The values of the successful dict are True or False whether it’s a file or not

listDirectory(lfns, credDict, verbose=False)

List directories

Parameters:

• lfns (list) – list of directories

• creDict – credential

Returns:

Successful/Failed dict. The successful values are dictionaries indexed “Files”, “Datasets”, “Subdirs” and “Links”

rebuildDirectoryUsage()

Rebuild DirectoryUsage table from scratch

removeDirectory(lfns, credDict)

Remove directories

Parameters:

• lfns (list) – list of directories

• creDict – credential

Returns:

Successful/Failed dict.

removeFile(lfns, credDict)

Remove files

Parameters:

• lfns (list) – list of LFNs to remove

• creDict – credential

Returns:

Successful/Failed dict.

removeMetadata(pathMetadataDict, credDict)

Remove metadata for the given paths

removeReplica(lfns, credDict)

Remove replicas

Parameters:

• lfns (dict) – keys are LFN. The values are dict with key PFN and SE (e.g. {myLfn : {“PFN” : “myPfn”, “SE” : “mySE”}})

• creDict – credential

Returns:

Successful/Failed dict.

repairCatalog(credDict={})

Repair catalog inconsistencies

setConfig(databaseConfig)

setFileStatus(lfns, credDict)

Set the status of a File

Parameters:

• lfns (dict) – dict indexed on the LFNs. The values are the status (should be in config[‘ValidFileStatus’])

• creDict – credential

Returns:

Successful/Failed dict.

setMetadata(path, metadataDict, credDict)

Add metadata to the given path

setMetadataBulk(pathMetadataDict, credDict)

Add metadata for the given paths

setReplicaHost(lfns, credDict)

setReplicaStatus(lfns, credDict)

Set the status of a Replicas

Parameters:

• lfns (dict) – dict indexed on the LFNs. The values are dict with keys “SE” and “Status” (that has to be in config[‘ValidReplicaStatus’])

• creDict – credential

Returns:

Successful/Failed dict.

setUmask(umask)

transactionCommit()

transactionRollback()

transactionStart()

updateFields(tableName, updateFields=None, updateValues=None, condDict=None, limit=False, conn=None, updateDict=None, older=None, newer=None, timeStamp=None, orderAttribute=None, greater=None, smaller=None)

Update “updateFields” from “tableName” with “updateValues”. updateDict alternative way to provide the updateFields and updateValues N records can match the condition return S_OK( number of updated rows ) if limit is not False, the given limit is set String type values will be appropriately escaped.