PoolComputingElement¶
The Pool Computing Element is an “inner” CE (meaning it’s used by a jobAgent inside a pilot)
It’s used running several jobs simultaneously in separate processes, managed by a ProcessPool.
Configuration Parameters
- LocalCEType:
Configuration for the PoolCE submission can be done via the CE configuration such as:
LocalCEType = Pool
The Pool Computing Element is specific: it embeds an additional “inner” CE (InProcess by default, Sudo, Singularity). The “inner” CE can be specified such as:
LocalCEType = Pool/Singularity
- NumberOfProcessors:
Maximum number of processors that can be used to compute jobs.
Code Documentation
- class DIRAC.Resources.Computing.PoolComputingElement.PoolComputingElement(ceUniqueID)¶
Bases:
ComputingElement
- __init__(ceUniqueID)¶
Standard constructor.
- available(jobIDList=None)¶
This method returns the number of available slots in the target CE. The CE instance polls for waiting and running jobs and compares to the limits in the CE parameters.
- Parameters
jobIDList (list) – list of already existing job IDs to be checked against
- finalizeJob(future)¶
Finalize the job by updating the process utilisation counters
- Parameters
future – evaluating the future result
- getCEStatus()¶
- Method to return information on running and waiting jobs,
as well as the number of processors (used, and available).
- Returns
dictionary of numbers of jobs per status and processors (used, and available)
- getDescription()¶
Get a list of CEs descriptions (each is a dict)
This is called by the JobAgent.
- getProcessorsInUse()¶
Get the number of currently allocated processor cores
- Returns
number of processors in use
- initializeParameters()¶
Initialize the CE parameters after they are collected from various sources
- isProxyValid(valid=1000)¶
Check if the stored proxy is valid
- isValid()¶
Check the sanity of the Computing Element definition
- loadBatchSystem(batchSystemName)¶
Instantiate object representing the backend batch system
- Parameters
batchSystemName (str) – name of the batch system
- sendOutput(stdid, line)¶
Callback function such that the results from the CE may be returned.
- setCPUTimeLeft(cpuTimeLeft=None)¶
Update the CPUTime parameter of the CE classAd, necessary for running in filling mode
- setParameters(ceOptions)¶
Add parameters from the given dictionary overriding the previous values
- Parameters
ceOptions (dict) – CE parameters dictionary to update already defined ones
- setProxy(proxy, valid=0)¶
Set proxy for this instance
- shutdown()¶
Wait for all futures (jobs) to complete
- submitJob(executableFile, proxy=None, **kwargs)¶
Method to submit job. This method will submit to a ProcessPoolExecutor, which returns Future objects.
- writeProxyToFile(proxy)¶
CE helper function to write a CE proxy string to a file.
- DIRAC.Resources.Computing.PoolComputingElement.executeJob(executableFile, proxy, taskID, **kwargs)¶
wrapper around ce.submitJob: decides which CE to use (Sudo or InProcess or Singularity)