Using DIRAC Clients From Python
Overview
For most tasks it is sufficient to use DIRAC from a terminal via the commands that are prefixed with dirac-
or via the WebApp.
In some cases however it becomes more convenient to use DIRAC directly from Python.
Before using the DIRAC’s Python API it is useful to understand a few DIRAC-specific concepts.
Architecture
DIRAC is built around a series of Systems
which are independent components and are each contained in a submodule under the main DIRAC
import.
Examples are:
DataManagementSystem
for accessing and managing dataWorkloadManagementSystem
for submitting and managing jobs
Each System
contains one or more Clients
which can be used for remotely requesting data from DIRAC services.
See Remote Procedure Calls.
Return values
Many DIRAC functions return either S_OK()
or S_ERROR()
.
In both of these cases the return value is a dictionary with an OK
key which determines if the call was successful.
True
the object corresponds toS_OK()
and contains aValue
key with the actual return value.False
the object corresponds toS_ERROR()
and contains aMessage
key with astr
which should contain more information.
For example, DIRAC contains a utility function for looking up the IP addresses of domains that returns S_OK()
or S_ERROR()
.
This function can be used as follows:
from DIRAC.Core.Utilities.Network import getIPsForHostName
for domain in ["diracgrid.org", "fake-domain.invalid"]:
ret = getIPsForHostName(domain)
if not ret["OK"]:
raise RuntimeError(f"Failed to find IPs for {domain}.org with error: {ret['Message']}")
ips = ret["Value"]
print(f"{domain} is running at {ips}")
This will print:
diracgrid.org is running at ['134.158.16.209']
Traceback (most recent call last):
File "return-values-example.py", line 6, in <module>
raise RuntimeError(f"Failed to find IPs for {domain}.org with error: {ret['Message']}")
RuntimeError: Failed to find IPs for fake-domain.invalid.org with error: Can't get info for host fake-domain.invalid: [Errno 8] nodename nor servname provided, or not known
Often when writing user scripts it is useful to immediately raise an exception when an error happens and otherwise just return the value.
This makes DIRAC behave more similarly to other Python functions and can be achieved using the helper function returnValueOrRaise()
.
Rewriting our previous example for looking up IP addressed, this would be:
from DIRAC.Core.Utilities.ReturnValues import returnValueOrRaise
from DIRAC.Core.Utilities.Network import getIPsForHostName
for domain in ["diracgrid.org", "fake-domain.invalid"]:
ips = returnValueOrRaise(getIPsForHostName(domain))
print(f"{domain} is running at {ips}")
Remote Procedure Calls
Remote Procedure Calls (RPC) describe the process of sending a request to other computers (i.e. DIRAC servers) that triggers them to process something. The result of this remotely ran function is then sent back to the origin of the call (i.e. your computer).
In DIRAC this is done automatically when using the Client
classes where calling a method MyClient().something(123)
triggers the server to execute a function named export_something(123)
.
See Using a client for an example of using a client.
Initializing DIRAC
Before using Python to contact a service DIRAC’s global state needs to be initialized.
Most importantly, this process contacts the Configuration Service and starts a refresher thread.
When writing a Python script this should be done using the DIRAC.initialize()
function.
import DIRAC
DIRAC.initialize()
Note
Currently it is essential to call initialize()
before importing clients.
Note
If you’re writing a dirac-
command initialization should be handled differently. See Developing Commands.
Using a client
Client classes can be found in the Client
submodule inside each System
module (see Architecture).
For example to list job IDs by calling the whoami
and getJobs
RPC calls:
import DIRAC
DIRAC.initialize()
from DIRAC.Core.Utilities.ReturnValues import returnValueOrRaise
from DIRAC.WorkloadManagementSystem.Client.JobMonitoringClient import JobMonitoringClient
jmc = JobMonitoringClient()
# Find out who the current user is
my_details = returnValueOrRaise(jmc.whoami())
username = my_details["username"]
# Find the job IDs for the given user
job_ids = returnValueOrRaise(jmc.getJobs({"owner": username}))
print(f"Job IDs for {username} are: {job_ids}")