ParallelLibrary

Parallel Library

Mainly used by Computing Elements to run several jobs/pilots in parallel within a same allocation. This is an abstract class and cannot be used directly. Use cases:

  • multiple nodes have been allocated and have to be used: a parallel library can execute the pilot wrapper on all the nodes in parallel.

  • multiple cpus have been allocated and have to be used: a parallel library could dispatch several jobs on the cores available (e.g. when sites have no external connectivity).

Parallel libraries implementations have to inherit from this class and override generateWrapper() and processOutput().

Parameters

The Parallel Libraries have to be defined in a CE section via the configuration system. See the page about configuring Resources / Computing for where the options can be placed.

WorkingDirectory:

Generally defined in the ComputingElement source code depending on how they deal with the executables. Indicates where the parallel library wrapper script has to be defined and retrieved.

class DIRAC.Resources.Computing.ParallelLibraries.ParallelLibrary.ParallelLibrary(name='undefined', workingDirectory='')

Bases: object

__init__(name='undefined', workingDirectory='')

Initialize the parallel library

Parameters
  • name (str) – name of the parallel library

  • workingDirectory (str) – directory that should contain the executable and its inputs

generateWrapper(executableFile)

Associate the executable with a wrapper script, generated by the parallel library, to execute the same command in parallel on multiple nodes. Wrap it in a new executable file.

Parameters

executableFile (str) – name of the executable file to wrap

Return str

name of the wrapper that runs the executable via the parallel library

processOutput(output, error, isFile=True)

Reorder the content of the output files that is possibly not easily readable after the execution of the parallel library. If isFile is enabled, then output and error have to be filenames and are modified in-place. Else the output and error contains content of the files, and the processed content is returned.

Parameters
  • output (str) – name of the output file, or its content

  • error (str) – name of the error file, or its content

  • isFile (bool) – indicates if the inputs represent files or content of the files

Return str

name of the files or their content depending on isFile value