Create a TransformationAgent Plugin

This page briefly explains the steps necessary to add a new TransformationPlugin to DIRAC, or an extension. It assumes the reader has some form of a development and testing setup available to them.

The file DIRAC/TransformationSystem/Agent/TransformationPlugin.py contains the TransformationPlugins for DIRAC. Plugins need to be registered in the configuration system, and the plugin function has to follow the expected return value structure explained in TransformationPlugin.

Note

If you create sufficiently generic plugins, they will be welcome in the vanilla DIRAC code base. If the plugin you create is too much focused on your own work you will need to create an extension of DIRAC and create an inherited TransformationPlugin class and tell the DIRAC.TransformationSystem.Agent.TransformationAgent where to find the plugin module of the extension.

Add New Plugin to the List of Allowed Plugins

The plugins that can be used inside the Transformation System, need to be added in the list of allowed plugins in the Operations/Transformations/AllowedPlugins option in the Configuration System.

If the option AllowedPlugins is already defined, simply add the new plugin:

Transformations
{
  AllowedPlugins = ...
  AllowedPlugins += ...
  AllowedPlugins += MyNewPlugin
}

Or, if it is not defined, you need to add the AllowedPlugins option including the list of default plugins, otherwise the other plugins would stop working:

Transformations
{
  AllowedPlugins = Broadcast
  AllowedPlugins += Standard
  AllowedPlugins += BySize
  AllowedPlugins += ByShare
  AllowedPlugins += MyNewPlugin
}

After adding to the AllowedPlugins option and the code for the plugin, the TransformationAgent should be restarted to reloads the configuration and source code.

The EvenOdd Plugin Example

This function shows an example plugin that groups files into even or odd numbers. See the comments to explain how to obtain LFNs and transformation parameters.

Should be part of DIRAC/TransformationSystem/Agent/TransformationPlugin.py
 1 def _EvenOdd(self):
 2   """Group files by Even (0,2,4,6,8) or Odd (1,3,5,7,9) numbering.
 3
 4   Use Transformation Parameter 'EvenOdd' to chose 'Even' or 'Odd', or 'Both'
 5
 6   :returns: S_OK with list of tuples. Each Tuple is a pair of SE and list of
 7             LFNs to treat in given task
 8   """
 9   # log parameters
10   self.util.logInfo('Running EvenOdd with parameters')
11   for param, value in self.params.iteritems():
12     self.util.logInfo('%s=%r' % (param, value))
13
14   # the transformation system tells us about the unused LFNs
15   lfns = self.data
16   self.util.logInfo('Treating the following LFNS:')
17   for lfn in lfns:
18     self.util.logInfo(lfn)
19
20   # Assuming files end with _[0-9]+
21   odd, even = [], []
22   for lfn in lfns:
23     self.util.logInfo(' odd or even?: %r' % lfn)
24     number = int(lfn.rsplit('_', 1)[1])
25     if number % 2 != 0:
26       self.util.logInfo(' odd')
27       odd.append(lfn)
28     else:
29       self.util.logInfo('even')
30       even.append(lfn)
31
32   # Treat only, even or odd numbers, or both
33   evenOrOdd = self.params.get('EvenOdd', 'Both')
34   if evenOrOdd == 'Both':
35     selection = [even, odd]
36   elif evenOrOdd == 'Even':
37     selection = [even]
38   elif evenOrOdd == 'Odd':
39     selection = [odd]
40   else:
41     return S_ERROR("Bad Parameter Value")
42
43   tasks = []
44   groupSize = self.params['GroupSize']
45   for chunks in selection:
46     for chunk in breakListIntoChunks(chunks, groupSize):
47       tasks.append(('', chunk))
48
49   self.util.logInfo('Tasks: %r' % tasks)
50   return S_OK(tasks)

Using the EvenOdd Plugin

When a transformation is created, set the EvenOdd plugin with setPlugin and set the ‘EvenOdd’ parameter to ‘Odd’ with setEvenOdd, and then execute this function to test it.

createEvenOdd.py
 1 from DIRAC import gLogger, initialize
 2
 3 initialize()
 4
 5 from DIRAC.TransformationSystem.Client.Transformation import Transformation
 6
 7 myTrans = Transformation()
 8 uniqueIdentifier = "OddOnly"
 9 myTrans.setTransformationName("ReplicateAndRegister_%s" % uniqueIdentifier)
10 myTrans.setDescription("Replicate only Odd files from StorageElementOne")
11 myTrans.setLongDescription("Replicate only Odd files from StorageElementOne")
12 myTrans.setType('Replication')
13 myTrans.setTransformationGroup('MyGroup')
14 myTrans.setGroupSize(2)
15
16 # Set the 'EvenOdd' plugin
17 myTrans.setPlugin('EvenOdd')
18 # set the 'EvenOdd' parameter to 'Odd', we can use python to
19 # automagically turn a myTrans.set<PARAMETER> function into a
20 # transformation parameter
21 myTrans.setEvenOdd('Odd')
22 myTrans.setSomeOtherParameter('Value')
23
24 targetSE = 'StorageElementOne'
25 myTrans.setBody([("ReplicateAndRegister", {"TargetSE": targetSE, "SourceSE": ''})])
26 myTrans.setTargetSE(targetSE)
27 res = myTrans.addTransformation()
28 if not res['OK']:
29   gLogger.error("Failed to add the transformation: %s" % res['Message'])
30   exit(1)
31
32 # now activate the transformation
33 myTrans.setStatus('Active')
34 myTrans.setAgentType('Automatic')
35 transID = myTrans.getTransformationID()['Value']
36 from DIRAC.TransformationSystem.Client.TransformationClient import TransformationClient
37 metadata = {'TransformationID': 2}
38 res = TransformationClient().createTransformationInputDataQuery(transID, metadata)
39 gLogger.notice('Added input data query', res)
40 gLogger.notice('Created EvenOdd transformation: %r' % transID)
41 exit(0)