DataManagement
For an introduction about DataManagement concepts, please see the introduction
All the commands mentionned bellow can accept several StorageElements and LFNs as parameters. Please use –help for more details.
Basics
How to list one’s own files
For a user to know its own file list:
[Dirac prod] chaen $ dirac-dms-user-lfns
Will search for files in /lhcb/user/c/chaen
/lhcb/user/c/chaen: 5 files, 2 sub-directories
/lhcb/user/c/chaen/GangaInputFile: 0 files, 1 sub-directories
/lhcb/user/c/chaen/GangaInputFile/Job_2: 1 files, 0 sub-directories
/lhcb/user/c/chaen/subDir: 1 files, 0 sub-directories
7 matched files have been put in lhcb-user-c-chaen.lfns
How to see the various replicas of a file
To list the SE where a file is stored:
[DIRAC prod] chaen $ dirac-dms-lfn-replicas /lhcb/user/c/chaen/diracTutorial.txt
Successful :
/lhcb/user/c/chaen/diracTutorial.txt :
CERN-USER : srm://srm-eoslhcb.cern.ch:8443/srm/v2/server?SFN=/eos/lhcb/grid/user/lhcb/user/c/chaen/diracTutorial.txt
RAL-USER : srm://srm-lhcb.gridpp.rl.ac.uk:8443/srm/managerv2?SFN=/castor/ads.rl.ac.uk/prod/lhcb/user/c/chaen/diracTutorial.txt
How to get the xroot URL for my LFN
In order to get an xroot URL usable from for example ROOT:
[DIRAC prod] chaen $ dirac-dms-lfn-accessURL --Protocol=root,xroot /lhcb/user/c/chaen/diracTutorial.txt RAL-USER
Successful :
RAL-USER :
/lhcb/user/c/chaen/diracTutorial.txt : root://clhcbstager.ads.rl.ac.uk//castor/ads.rl.ac.uk/prod/lhcb/user/c/chaen/diracTutorial.txt?svcClass=lhcbUser
If you do not specify a StorageElement, DIRAC will check the URLs for all the replicas:
[DIRAC prod] chaen $ dirac-dms-lfn-accessURL --Protocol=root,xroot /lhcb/user/c/chaen/diracTutorial.txt
Using the following list of SEs: ['CERN-USER', 'RAL-USER']
Successful :
CERN-USER :
/lhcb/user/c/chaen/diracTutorial.txt : root://eoslhcb.cern.ch//eos/lhcb/grid/user/lhcb/user/c/chaen/diracTutorial.txt
RAL-USER :
/lhcb/user/c/chaen/diracTutorial.txt : root://clhcbstager.ads.rl.ac.uk//castor/ads.rl.ac.uk/prod/lhcb/user/c/chaen/diracTutorial.txt?svcClass=lhcbUser
How to upload a file to a grid storage
To put a local file to a GRID storage:
[DIRAC prod] chaen $ dirac-dms-add-file /lhcb/user/c/chaen/diracTutorial.txt ./diracTutorial.txt CERN-USER
Could not obtain GUID from file through Gaudi, using standard DIRAC method
Uploading ./diracTutorial.txt as /lhcb/user/c/chaen/diracTutorial.txt
Successfully uploaded ./diracTutorial.txt to CERN-USER (0.7 seconds)
How to replicate an LFN to another storage
The file has to be already on a grid storage:
[DIRAC prod] chaen $ dirac-dms-replicate-lfn /lhcb/user/c/chaen/diracTutorial.txt RAL-USER
Successful :
RAL-USER :
/lhcb/user/c/chaen/diracTutorial.txt :
register : 0.216005086899
replicate : 5.27293300629
How to have the metadata of an file
To get the metadata of an LFN as stored in the catalog:
[DIRAC prod] chaen $ dirac-dms-lfn-metadata /lhcb/user/c/chaen/diracTutorial.txt
Successful :
/lhcb/user/c/chaen/diracTutorial.txt :
Checksum : 2a810562
ChecksumType : Adler32
CreationDate : 2018-12-20 18:33:40
FileID : 390920814
GID : 2746
GUID : 15C4C7B2-47F3-9BDE-CA19-60A1E348EF90
Mode : 775
ModificationDate : 2018-12-20 18:33:40
Owner : chaen
OwnerGroup : lhcb_prmgr
Size : 14
Status : AprioriGood
UID : 20269
To get the metadata of the file actually stored, you can use the following command (with or without SE specification):
[DIRAC prod] chaen $ dirac-dms-pfn-metadata /lhcb/user/c/chaen/diracTutorial.txt
Getting replicas for 1 files : completed in 0.1 seconds
Getting SE metadata of 2 replicas : completed in 1.5 seconds
Successful :
/lhcb/user/c/chaen/diracTutorial.txt :
CERN-USER :
Accessible : True
Checksum : 2a810562
Directory : False
Executable : False
File : True
FileSerialNumber : 10376293541461674751
GroupID : 1470
LastAccess : 2018-12-20 19:33:39
Links : 1
ModTime : 2018-12-20 19:33:39
Mode : 400
Readable : True
Size : 14
StatusChange : 2018-12-20 19:33:39
UserID : 56212
Writeable : False
RAL-USER :
Accessible : True
Cached : 1
Checksum : 2a810562
Directory : False
Executable : False
File : True
FileSerialNumber : 0
GroupID : 46
LastAccess : Never
Links : 1
Lost : 0
Migrated : 0
ModTime : 2018-12-20 19:35:13
Mode : 644
Readable : True
Size : 14
StatusChange : 2018-12-20 19:35:13
Unavailable : 0
UserID : 45
Writeable : True
[DIRAC prod] chaen $ dirac-dms-pfn-metadata /lhcb/user/c/chaen/diracTutorial.txt CERN-USER
Getting replicas for 1 files : completed in 0.1 seconds
Getting SE metadata of 1 replicas : completed in 1.0 seconds
Successful :
/lhcb/user/c/chaen/diracTutorial.txt :
CERN-USER :
Accessible : True
Checksum : 2a810562
Directory : False
Executable : False
File : True
FileSerialNumber : 10376293541461674751
GroupID : 1470
LastAccess : 2018-12-20 19:33:39
Links : 1
ModTime : 2018-12-20 19:33:39
Mode : 400
Readable : True
Size : 14
StatusChange : 2018-12-20 19:33:39
UserID : 56212
Writeable : False
How to remove a replica of a file
In order to remove one of the replicas:
[DIRAC prod] chaen $ dirac-dms-remove-replicas /lhcb/user/c/chaen/diracTutorial.txt CERN-USER
Removing replicas : completed in 1.8 seconds
Successfully removed 1 replicas from CERN-USER
How to remove a file from the grid
Watch out, this will remove all the replicas of a file:
[DIRAC prod] chaen $ dirac-dms-remove-files /lhcb/user/c/chaen/diracTutorial.txt
Removing 1 files : completed in 1.9 seconds
Successfully removed 1 files
DFC as a metadata catalog
This section supposes that the DFC is used as a Metadata Catalog. This is for example not the case of LHCb. Please ask your administrator if you are unsure. The exercises are performed using the File Catalog CLI interface. You can start the CLI with the command:
dirac-dms-filecatalog-cli
How to add metadata to a directory
From the CLI:
meta set <directory> <metaname> <metavalue>
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg>meta set . ATMetaStr Test
FC:/vo.formation.idgrilles.fr/user/a/atsareg>mkdir testDir
Successfully created directory: /vo.formation.idgrilles.fr/user/a/atsareg/testDir
FC:/vo.formation.idgrilles.fr/user/a/atsareg>meta set testDir AnotherMeta AnotherTest
How to get directory metadata
From the CLI:
meta get <directory>
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg>meta get testDir
AnotherMeta : AnotherTest
ATMetaStr : Test
How to create metadata index
From the CLI:
meta index <metaname> <metatype>
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg>meta index NewMetaInt int
Added metadata field NewMetaInt of type int
Possible metadata types: int,float,string,date
How to show existing metadata indexes
From the CLI:
meta show
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg>meta show
ATMetaStr : VARCHAR(128)
ATMetaInt : INT
ATMetaDate : DATETIME
ATMetaSet : MetaSet
ATMetaInt1 : INT
NewMetaInt : INT
ATMetaFlt : float
How to find files with selection by metadata
From the CLI:
find <meta selection>
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg> find ATMetaInt=10,11 ATMetaInt1<15
Query: {'ATMetaInt': {'in': [10, 11]}, 'ATMetaInt1': {'<': 15}}
/vo.formation.idgrilles.fr/user/a/atsareg/newDir/wms_output.py
How to declare file’s ancestors
The ancestor declaration is done as following:
ancestorset <descendent> <ancestor>
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg> ancestorset file2 file1
FC:/vo.formation.idgrilles.fr/user/a/atsareg> ancestorset file3 file2
How to query file’s ancestors
It can be interrogated with the following commands:
ancestor <file> <depth>
descendent <file> <depth>
For example:
FC:/vo.formation.idgrilles.fr/user/a/atsareg> ancestor file3 2
/vo.formation.idgrilles.fr/user/a/atsareg/file3
1 /vo.formation.idgrilles.fr/user/a/atsareg/file2
2 /vo.formation.idgrilles.fr/user/a/atsareg/file1
FC:/vo.formation.idgrilles.fr/user/a/atsareg> descendent file1 2
/vo.formation.idgrilles.fr/user/a/atsareg/file1
1 /vo.formation.idgrilles.fr/user/a/atsareg/file2
2 /vo.formation.idgrilles.fr/user/a/atsareg/file3