3. Install a DIRAC Storage Element¶
3.1. Pre-requisite¶
You should have a machine setup as described in Basic Tutorial setup, and be able to install dirac components. For simple interaction with the StorageElement using dirac-dms-*
commands, you should also have a working FileCatalog.
3.2. Tutorial goal¶
The aim of the tutorial is to do a step by step guide to install a DIRAC StorageElement. By the end of the tutorial, you will be able to have a fully functional storage element that can be addressed like any other storage.
3.3. More links¶
StorageElement service documentation:
DIRAC.DataManagementSystem.Service.StorageElementHandler
3.4. Machine setup¶
This section is to be executed as dirac
user.
We will simply create a folder where the files will be stored:
[dirac@dirac-tuto ~]$ mkdir /opt/dirac/storageElementOne/
3.5. Installing the service¶
This section is to be executed as diracuser
user, with dirac_admin
proxy (reminder: dirac-proxy-init -g dirac_admin
).
Install the StorageElement service using dirac-admin-sysadmin-cli
. First we add the DataManagement system to the
Production instance on the configuration, then we restart all services so they pick up the new configuration, and finally we install the
StorageElement
pointing to the BasePath
we created earlier:
[diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto
Pinging dirac-tuto...
[dirac-tuto]$ add instance DataManagement Production
Adding DataManagement system as Production self.instance for MyDIRAC-Production self.setup to dirac.cfg and CS
DataManagement system instance Production added successfully
[dirac-tuto]$ restart *
All systems are restarted, connection to SystemAdministrator is lost
[dirac-tuto]$ install service DataManagement StorageElement -p BasePath=/opt/dirac/storageElementOne/
Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg
Adding to CS service DataManagement/StorageElement
service DataManagement_StorageElement is installed, runit status: Run
[dirac-tuto]$ quit
You now have a Service offering grid like storage. However, you still need to declare a StorageElement for it to be usable within DIRAC.
3.6. Adding the StorageElement¶
You need to add a StorageElement in the Resources/StorageElements
section (create the sections if necessary). Using the WebApp, just add the following:
StorageElementOne
{
BackendType = DISET
DIP
{
Host = dirac-tuto
Port = 9148
Protocol = dips
Path = /DataManagement/StorageElement
Access = remote
}
}
You now have a storage element that you can address as StorageElementOne
in all the dirac commands or in your code.
3.7. Test it¶
Create a dummy file:
[diracuser@dirac-tuto ~]$ echo "dummyFile" > /tmp/dummy.txt
Now create a file called /tmp/testSE.py
, with the following content
from DIRAC.Core.Base.Script import parseCommandLine
parseCommandLine()
localFile = '/tmp/dummy.txt'
lfn = '/tutoVO/myFirstFile.txt'
from DIRAC.Resources.Storage.StorageElement import StorageElement
se = StorageElement('StorageElementOne')
print "Putting file"
print se.putFile({lfn: localFile})
print "Listing directory"
print se.listDirectory('/tutoVO')
print "Getting file"
print se.getFile(lfn, '/tmp/')
print "Removing file"
print se.removeFile(lfn)
print "Listing directory"
print se.listDirectory('/tutoVO')
This script uploads /tmp/dummy.txt
on the StorageElement as myFirstFile.txt
, list the directory, downloads the uploaded file and removes it from the StorageElement. The output should be something like that:
[diracuser@dirac-tuto ~]$ python /tmp/testSE.py
Putting file
{'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': 10}, 'Failed': {}}}
Listing directory
{'OK': True, 'Value': {'Successful': {'/tutoVO': {'Files': {'myFirstFile.txt': {'Accessible': True, 'Migrated': 0, 'Unavailable': 0, 'Lost': 0, 'Exists': True, 'Cached': 1, 'Checksum': '166203b7', 'Mode': 420, 'File': True, 'Directory': True, 'TimeStamps': (1555342476, 1555342476, 1555342476), 'Type': 'File', 'Size': 10}}, 'SubDirs': {}}}, 'Failed': {}}}
Getting file
{'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': 10}, 'Failed': {}}}
Removing file
{'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': True}, 'Failed': {}}}
Listing directory
{'OK': True, 'Value': {'Successful': {'/tutoVO': {'Files': {}, 'SubDirs': {}}}, 'Failed': {}}}
The list of files within tmp
should also contain dummy.txt
as well as myFirstFile.txt
.
3.8. Adding a second DIRAC SE¶
It is often interesting to have a second SE.
As dirac
user, create a new directory:
[dirac@dirac-tuto ~]$ mkdir /opt/dirac/storageElementTwo/
Now the rest is to be installed with diracuser
and a proxy with dirac_admin
group.
We need another StorageElement service. However, it has to have a different name, Port and BasePath than the first one, so we will just call this service StorageElementTwo
:
[diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto
Pinging dirac-tuto...
[dirac-tuto]$ install service DataManagement StorageElementTwo -m StorageElement -p Port=9147 -p BasePath=/opt/dirac/storageElementTwo/
Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg
Adding to CS service DataManagement/StorageElementTwo
service DataManagement_StorageElementTwo is installed, runit status: Run
Using the WebApp, add the new StorageElement definition in the /Resources/StorageElements
section:
StorageElementTwo
{
BackendType = DISET
DIP
{
Host = dirac-tuto
Port = 9147
Protocol = dips
Path = /DataManagement/StorageElementTwo
Access = remote
}
}
In order to test it, just re-use /tmp/testSE.py
, replacing StorageElementOne
with StorageElementTwo