PoolComputingElement

The Pool Computing Element is an “inner” CE (meaning it’s used by a jobAgent inside a pilot)

It’s used running several jobs simultaneously in separate processes, managed by a ProcessPool.

Configuration Parameters

LocalCEType:

Configuration for the PoolCE submission can be done via the CE configuration such as:

LocalCEType = Pool

The Pool Computing Element is specific: it embeds an additional “inner” CE (InProcess by default, Sudo, Singularity). The “inner” CE can be specified such as:

LocalCEType = Pool/Singularity
NumberOfProcessors:

Maximum number of processors that can be used to compute jobs.

Code Documentation

class DIRAC.Resources.Computing.PoolComputingElement.PoolComputingElement(ceUniqueID)

Bases: ComputingElement

__init__(ceUniqueID)

Standard constructor.

available(jobIDList=None)

This method returns the number of available slots in the target CE. The CE instance polls for waiting and running jobs and compares to the limits in the CE parameters.

Parameters:

jobIDList (list) – list of already existing job IDs to be checked against

finalizeJob(taskID, future)

Finalize the job by updating the process utilisation counters

Parameters:

future – evaluating the future result

getCEStatus()
Method to return information on running and waiting jobs,

as well as the number of processors (used, and available).

Returns:

dictionary of numbers of jobs per status and processors (used, and available)

getDescription()

Get a list of CEs descriptions (each is a dict)

This is called by the JobAgent.

getProcessorsInUse()

Get the number of currently allocated processor cores

Returns:

number of processors in use

initializeParameters()

Initialize the CE parameters after they are collected from various sources

isProxyValid(valid=1000)

Check if the stored proxy is valid

isValid()

Check the sanity of the Computing Element definition

loadBatchSystem(batchSystemName)

Instantiate object representing the backend batch system

Parameters:

batchSystemName (str) – name of the batch system

sendOutput(stdid, line)

Callback function such that the results from the CE may be returned.

setCPUTimeLeft(cpuTimeLeft=None)

Update the CPUTime parameter of the CE classAd, necessary for running in filling mode

setParameters(ceOptions)

Add parameters from the given dictionary overriding the previous values

Parameters:

ceOptions (dict) – CE parameters dictionary to update already defined ones

setProxy(proxy, valid=0)

Set proxy for this instance

setToken(token)
shutdown()

Wait for all futures (jobs) to complete

submitJob(executableFile, proxy=None, inputs=None, **kwargs)

Method to submit job. This method will submit to a ProcessPoolExecutor, which returns Future objects.

Parameters:
  • executableFile (str) – location of the executable file

  • proxy (str) – payload proxy

  • inputs (list) – dependencies of executableFile

Returns:

S_OK always. The result of the submission should be included in taskResults

writeProxyToFile(proxy)

CE helper function to write a CE proxy string to a file.

DIRAC.Resources.Computing.PoolComputingElement.executeJob(executableFile, proxy, taskID, inputs, **kwargs)

wrapper around ce.submitJob: decides which CE to use (Sudo or InProcess or Singularity)

Parameters:
  • executableFile (str) – location of the executable file

  • proxy (str) – proxy file location to be used for job submission

  • taskID (int) – local task ID of the PoolCE

Returns:

the result of the job submission (S_OK/S_ERROR)