3. Install a DIRAC Storage Element

3.1. Pre-requisite

You should have a machine setup as described in Basic Tutorial setup, and be able to install dirac components. For simple interaction with the StorageElement using dirac-dms-* commands, you should also have a working FileCatalog.

3.2. Tutorial goal

The aim of the tutorial is to do a step by step guide to install a DIRAC StorageElement. By the end of the tutorial, you will be able to have a fully functional storage element that can be addressed like any other storage.

3.4. Machine setup

This section is to be executed as dirac user.

We will simply create a folder where the files will be stored:

[[email protected] ~]$ mkdir /opt/dirac/storageElementOne/

3.5. Installing the service

This section is to be executed as diracuser user, with dirac_admin proxy (reminder: dirac-proxy-init -g dirac_admin).

Install the StorageElement service using dirac-admin-sysadmin-cli. First we add the DataManagement system to the Production instance on the configuration, then we restart all services so they pick up the new configuration, and finally we install the StorageElement pointing to the BasePath we created earlier:

[[email protected] ~]$ dirac-admin-sysadmin-cli --host dirac-tuto
Pinging dirac-tuto...
[dirac-tuto]$ add instance DataManagement Production
Adding DataManagement system as Production self.instance for MyDIRAC-Production self.setup to dirac.cfg and CS
DataManagement system instance Production added successfully
[dirac-tuto]$ restart *
All systems are restarted, connection to SystemAdministrator is lost
[dirac-tuto]$ install service DataManagement StorageElement -p BasePath=/opt/dirac/storageElementOne/
Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg
Adding to CS service DataManagement/StorageElement
service DataManagement_StorageElement is installed, runit status: Run
[dirac-tuto]$ quit

You now have a Service offering grid like storage. However, you still need to declare a StorageElement for it to be usable within DIRAC.

3.6. Adding the StorageElement

You need to add a StorageElement in the Resources/StorageElements section (create the sections if necessary). Using the WebApp, just add the following:

StorageElementOne
{
  BackendType = DISET
  DIP
  {
    Host = dirac-tuto
    Port = 9148
    Protocol = dips
    Path = /DataManagement/StorageElement
    Access = remote
  }
}

You now have a storage element that you can address as StorageElementOne in all the dirac commands or in your code.

3.7. Test it

Create a dummy file:

[[email protected] ~]$ echo "dummyFile" > /tmp/dummy.txt

Now create a file called /tmp/testSE.py, with the following content

from DIRAC.Core.Base.Script import parseCommandLine
parseCommandLine()

localFile = '/tmp/dummy.txt'
lfn = '/tutoVO/myFirstFile.txt'

from DIRAC.Resources.Storage.StorageElement import StorageElement


se = StorageElement('StorageElementOne')

print "Putting file"
print se.putFile({lfn: localFile})

print "Listing directory"
print se.listDirectory('/tutoVO')

print "Getting file"
print se.getFile(lfn, '/tmp/')

print "Removing file"
print se.removeFile(lfn)

print "Listing directory"
print se.listDirectory('/tutoVO')

This script uploads /tmp/dummy.txt on the StorageElement as myFirstFile.txt, list the directory, downloads the uploaded file and removes it from the StorageElement. The output should be something like that:

[[email protected] ~]$ python /tmp/testSE.py
Putting file
{'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': 10}, 'Failed': {}}}
Listing directory
{'OK': True, 'Value': {'Successful': {'/tutoVO': {'Files': {'myFirstFile.txt': {'Accessible': True, 'Migrated': 0, 'Unavailable': 0, 'Lost': 0, 'Exists': True, 'Cached': 1, 'Checksum': '166203b7', 'Mode': 420, 'File': True, 'Directory': True, 'TimeStamps': (1555342476, 1555342476, 1555342476), 'Type': 'File', 'Size': 10}}, 'SubDirs': {}}}, 'Failed': {}}}
Getting file
{'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': 10}, 'Failed': {}}}
Removing file
{'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': True}, 'Failed': {}}}
Listing directory
{'OK': True, 'Value': {'Successful': {'/tutoVO': {'Files': {}, 'SubDirs': {}}}, 'Failed': {}}}

The list of files within tmp should also contain dummy.txt as well as myFirstFile.txt.

3.8. Adding a second DIRAC SE

It is often interesting to have a second SE.

As dirac user, create a new directory:

[[email protected] ~]$  mkdir /opt/dirac/storageElementTwo/

Now the rest is to be installed with diracuser and a proxy with dirac_admin group.

We need another StorageElement service. However, it has to have a different name, Port and BasePath than the first one, so we will just call this service StorageElementTwo:

[[email protected] ~]$ dirac-admin-sysadmin-cli --host dirac-tuto
Pinging dirac-tuto...
[dirac-tuto]$ install service DataManagement StorageElementTwo -m StorageElement -p Port=9147 -p BasePath=/opt/dirac/storageElementTwo/
Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg
Adding to CS service DataManagement/StorageElementTwo
service DataManagement_StorageElementTwo is installed, runit status: Run

Using the WebApp, add the new StorageElement definition in the /Resources/StorageElements section:

StorageElementTwo
{
  BackendType = DISET
  DIP
  {
    Host = dirac-tuto
    Port = 9147
    Protocol = dips
    Path = /DataManagement/StorageElementTwo
    Access = remote
  }
}

In order to test it, just re-use /tmp/testSE.py, replacing StorageElementOne with StorageElementTwo