DataManager
- mod:
DataManager
- synopsis:
DataManager links the functionalities of StorageElement and FileCatalog.
This module consists of DataManager and related classes.
- class DIRAC.DataManagementSystem.Client.DataManager.DataManager(catalogs=None, masterCatalogOnly=False, vo=False)
Bases:
object- class DataManager
A DataManager is taking all the actions that impact or require the FileCatalog and the StorageElement together
- __init__(catalogs=None, masterCatalogOnly=False, vo=False)
c’tor
- Parameters:
self – self reference
catalogs – the list of catalog in which to perform the operations. This list will be ignored if masterCatalogOnly is set to True
masterCatalogOnly – if set to True, the operations will be performed only on the master catalog. The catalogs parameter will be ignored.
vo – the VO for which the DataManager is created, get VO from the current proxy if not specified
- checkActiveReplicas(replicaDict)
Check a replica dictionary for active replicas, and verify input structure first
- cleanLogicalDirectory(lfnDir)
Clean the logical directory from the catalog and storage
- getActiveReplicas(lfns, getUrl=True, diskOnly=False, preferDisk=False, protocol=None)
Get all the replicas for the SEs which are in Active status for reading.
- getFile(lfn, destinationDir='', sourceSE=None, diskOnly=False)
Get local copy of LFN(s) from Storage Elements.
- Parameters:
lfn (mixed) – a single LFN or list of LFNs.
destinationDir (str) – directory to which the file(s) will be downnloaded. (Default: current working directory).
sourceSE (str) – source SE from which to download (Default: all replicas will be attempted).
diskOnly (bool) – chooses the disk ONLY replica(s). (Default: False)
- Returns:
S_OK({“Successful”: {}, “Failed”: {}})/S_ERROR(errMessage).
- getFilesFromDirectory(directory, days=0, wildcard='*')
get all files from :directory: older than :days: days matching to :wildcard:
- getReplica(lfn, storageElementName, localPath=False)
copy replicas from DIRAC SE to local directory
- getReplicaAccessUrl(lfn, storageElementName, protocol=False)
get the access url for lfns at the supplied StorageElement
- getReplicaIsFile(lfn, storageElementName)
determine whether the supplied lfns are files at the supplied StorageElement
- getReplicaMetadata(lfn, storageElementName)
get the file metadata for lfns at the supplied StorageElement
- getReplicaSize(lfn, storageElementName)
get the size of files for the lfns at the supplied StorageElement
- getReplicas(lfns, allStatus=True, getUrl=True, diskOnly=False, preferDisk=False, active=False, protocol=None)
get replicas from catalogue and filter if requested Warning: all filters are independent, hence active and preferDisk should be set if using forJobs
- getReplicasForJobs(lfns, allStatus=False, getUrl=True, diskOnly=False, protocol=None)
get replicas useful for jobs
- getReplicasFromDirectory(directory)
get all replicas from a given directory
- Parameters:
self – self reference
directory (mixed) – list of directories or one directory
- pinReplica(lfn, storageElementName, lifetime=86400)
pin the lfns at the supplied StorageElement
- prestageReplica(lfn, storageElementName, lifetime=86400)
issue a prestage requests for the lfns at the supplied StorageElement
- put(lfn, fileName, diracSE, path=None)
Put a local file to a Storage Element
- putAndRegister(lfn, fileName, diracSE, guid=None, path=None, checksum=None, overwrite=False)
Put a local file to a Storage Element and register in the File Catalogues
‘lfn’ is the file LFN ‘file’ is the full path to the local file ‘diracSE’ is the Storage Element to which to put the file ‘guid’ is the guid with which the file is to be registered (if not provided will be generated) ‘path’ is the path on the storage where the file will be put (if not provided the LFN will be used) ‘overwrite’ removes file from the file catalogue and SE before attempting upload
- registerFile(fileTuple, catalog='')
Register a file or a list of files
- registerReplica(replicaTuple, catalog='')
Register a replica (or list of) supplied in the replicaTuples.
‘replicaTuple’ is a tuple or list of tuples of the form (lfn,pfn,se)
- releaseReplica(lfn, storageElementName)
release pins for the lfns at the supplied StorageElement
- removeFile(lfn, force=None)
Remove the file (all replicas) from Storage Elements and file catalogue
‘lfn’ is the file to be removed
- removeReplica(storageElementName, lfn)
Remove replica at the supplied Storage Element from Storage Element then file catalogue
‘storageElementName’ is the storage where the file is to be removed ‘lfn’ is the file to be removed
- removeReplicaFromCatalog(storageElementName, lfn)
remove :lfn: replica from :storageElementName: SE
- Parameters:
self – self reference
storageElementName (str) – SE name
lfn (mixed) – a single LFN or list of LFNs
- replicate(lfn, destSE, sourceSE='', destPath='', localCache='')
Replicate a LFN to a destination SE and register the replica.
‘lfn’ is the LFN to be replicated ‘destSE’ is the Storage Element the file should be replicated to ‘sourceSE’ is the source for the file replication (where not specified all replicas will be attempted) ‘destPath’ is the path on the destination storage element, if to be different from LHCb convention ‘localCache’ is the local file system location to be used as a temporary cache
- replicateAndRegister(lfn, destSE, sourceSE='', destPath='', localCache='', catalog='')
Replicate a LFN to a destination SE and register the replica.
‘lfn’ is the LFN to be replicated ‘destSE’ is the Storage Element the file should be replicated to ‘sourceSE’ is the source for the file replication (where not specified all replicas will be attempted) ‘destPath’ is the path on the destination storage element, if to be different from LHCb convention ‘localCache’ is the local file system location to be used as a temporary cache
- setAccountingClient(client)
Set Accounting Client instance