DataManager

mod:

DataManager

synopsis:

DataManager links the functionalities of StorageElement and FileCatalog.

This module consists of DataManager and related classes.

class DIRAC.DataManagementSystem.Client.DataManager.DataManager(catalogs=None, masterCatalogOnly=False, vo=False)

Bases: object

class DataManager

A DataManager is taking all the actions that impact or require the FileCatalog and the StorageElement together

__init__(catalogs=None, masterCatalogOnly=False, vo=False)

c’tor

Parameters:
  • self – self reference

  • catalogs – the list of catalog in which to perform the operations. This list will be ignored if masterCatalogOnly is set to True

  • masterCatalogOnly – if set to True, the operations will be performed only on the master catalog. The catalogs parameter will be ignored.

  • vo – the VO for which the DataManager is created, get VO from the current proxy if not specified

checkActiveReplicas(replicaDict)

Check a replica dictionary for active replicas, and verify input structure first

cleanLogicalDirectory(lfnDir)

Clean the logical directory from the catalog and storage

getActiveReplicas(lfns, getUrl=True, diskOnly=False, preferDisk=False, protocol=None)

Get all the replicas for the SEs which are in Active status for reading.

getFile(lfn, destinationDir='', sourceSE=None, diskOnly=False)

Get local copy of LFN(s) from Storage Elements.

Parameters:
  • lfn (mixed) – a single LFN or list of LFNs.

  • destinationDir (str) – directory to which the file(s) will be downnloaded. (Default: current working directory).

  • sourceSE (str) – source SE from which to download (Default: all replicas will be attempted).

  • diskOnly (bool) – chooses the disk ONLY replica(s). (Default: False)

Returns:

S_OK({“Successful”: {}, “Failed”: {}})/S_ERROR(errMessage).

getFilesFromDirectory(directory, days=0, wildcard='*')

get all files from :directory: older than :days: days matching to :wildcard:

Parameters:
  • self – self reference

  • directory (mixed) – list of directories or directory name

  • days (int) – ctime days

  • wildcard (str) – pattern to match

getReplica(lfn, storageElementName, localPath=False)

copy replicas from DIRAC SE to local directory

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • localPath (mixed) – path in the local file system, if False, os.getcwd() will be used

  • singleFile (bool) – execute for the first LFN only

getReplicaAccessUrl(lfn, storageElementName, protocol=False)

get the access url for lfns at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • singleFile (bool) – execute for the first LFN only

getReplicaIsFile(lfn, storageElementName)

determine whether the supplied lfns are files at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • singleFile (bool) – execute for the first LFN only

getReplicaMetadata(lfn, storageElementName)

get the file metadata for lfns at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • singleFile (bool) – execute for the first LFN only

getReplicaSize(lfn, storageElementName)

get the size of files for the lfns at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • singleFile (bool) – execute for the first LFN only

getReplicas(lfns, allStatus=True, getUrl=True, diskOnly=False, preferDisk=False, active=False, protocol=None)

get replicas from catalogue and filter if requested Warning: all filters are independent, hence active and preferDisk should be set if using forJobs

getReplicasForJobs(lfns, allStatus=False, getUrl=True, diskOnly=False, protocol=None)

get replicas useful for jobs

getReplicasFromDirectory(directory)

get all replicas from a given directory

Parameters:
  • self – self reference

  • directory (mixed) – list of directories or one directory

pinReplica(lfn, storageElementName, lifetime=86400)

pin the lfns at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • lifetime (int) – 24h in seconds

  • singleFile (bool) – execute for the first LFN only

prestageReplica(lfn, storageElementName, lifetime=86400)

issue a prestage requests for the lfns at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • lifetime (int) – 24h in seconds

  • singleFile (bool) – execute for the first LFN only

put(lfn, fileName, diracSE, path=None)

Put a local file to a Storage Element

Parameters:
  • self – self reference

  • lfn (str) – LFN

  • fileName (str) – the full path to the local file

  • diracSE (str) – the Storage Element to which to put the file

  • path (str) – the path on the storage where the file will be put (if not provided the LFN will be used)

putAndRegister(lfn, fileName, diracSE, guid=None, path=None, checksum=None, overwrite=False)

Put a local file to a Storage Element and register in the File Catalogues

‘lfn’ is the file LFN ‘file’ is the full path to the local file ‘diracSE’ is the Storage Element to which to put the file ‘guid’ is the guid with which the file is to be registered (if not provided will be generated) ‘path’ is the path on the storage where the file will be put (if not provided the LFN will be used) ‘overwrite’ removes file from the file catalogue and SE before attempting upload

registerFile(fileTuple, catalog='')

Register a file or a list of files

Parameters:
  • self – self reference

  • fileTuple (tuple) – (lfn, physicalFile, fileSize, storageElementName, fileGuid, checksum )

  • catalog (str) – catalog name

registerReplica(replicaTuple, catalog='')

Register a replica (or list of) supplied in the replicaTuples.

‘replicaTuple’ is a tuple or list of tuples of the form (lfn,pfn,se)

releaseReplica(lfn, storageElementName)

release pins for the lfns at the supplied StorageElement

Parameters:
  • self – self reference

  • lfn (mixed) – LFN string, list if LFNs or dict with LFNs as keys

  • storageElementName (str) – DIRAC SE name

  • singleFile (bool) – execute for the first LFN only

removeFile(lfn, force=None)

Remove the file (all replicas) from Storage Elements and file catalogue

‘lfn’ is the file to be removed

removeReplica(storageElementName, lfn)

Remove replica at the supplied Storage Element from Storage Element then file catalogue

‘storageElementName’ is the storage where the file is to be removed ‘lfn’ is the file to be removed

removeReplicaFromCatalog(storageElementName, lfn)

remove :lfn: replica from :storageElementName: SE

Parameters:
  • self – self reference

  • storageElementName (str) – SE name

  • lfn (mixed) – a single LFN or list of LFNs

replicate(lfn, destSE, sourceSE='', destPath='', localCache='')

Replicate a LFN to a destination SE and register the replica.

‘lfn’ is the LFN to be replicated ‘destSE’ is the Storage Element the file should be replicated to ‘sourceSE’ is the source for the file replication (where not specified all replicas will be attempted) ‘destPath’ is the path on the destination storage element, if to be different from LHCb convention ‘localCache’ is the local file system location to be used as a temporary cache

replicateAndRegister(lfn, destSE, sourceSE='', destPath='', localCache='', catalog='')

Replicate a LFN to a destination SE and register the replica.

‘lfn’ is the LFN to be replicated ‘destSE’ is the Storage Element the file should be replicated to ‘sourceSE’ is the source for the file replication (where not specified all replicas will be attempted) ‘destPath’ is the path on the destination storage element, if to be different from LHCb convention ‘localCache’ is the local file system location to be used as a temporary cache

setAccountingClient(client)

Set Accounting Client instance