PoolComputingElement
The Pool Computing Element is an “inner” CE (meaning it’s used by a jobAgent inside a pilot)
It’s used running several jobs simultaneously in separate processes, managed by a ProcessPool.
Configuration Parameters
- LocalCEType:
Configuration for the PoolCE submission can be done via the CE configuration such as:
LocalCEType = Pool
The Pool Computing Element is specific: it embeds an additional “inner” CE (InProcess by default, Sudo, Singularity). The “inner” CE can be specified such as:
LocalCEType = Pool/Singularity
- NumberOfProcessors:
Maximum number of processors that can be used to compute jobs.
Code Documentation
- class DIRAC.Resources.Computing.PoolComputingElement.PoolComputingElement(ceUniqueID)
Bases:
ComputingElement- __init__(ceUniqueID)
Standard constructor.
- available(jobIDList=None)
This method returns the number of available slots in the target CE. The CE instance polls for waiting and running jobs and compares to the limits in the CE parameters.
- Parameters:
jobIDList (list) – list of already existing job IDs to be checked against
- finalizeJob(taskID, future)
Finalize the job by updating the process utilisation counters
- Parameters:
future – evaluating the future result
- getCEStatus()
- Method to return information on running and waiting jobs,
as well as the number of processors (used, and available).
- Returns:
dictionary of numbers of jobs per status and processors (used, and available)
- getDescription()
Get a list of CEs descriptions (each is a dict)
This is called by the JobAgent.
- getProcessorsInUse()
Get the number of currently allocated processor cores
- Returns:
number of processors in use
- initializeParameters()
Initialize the CE parameters after they are collected from various sources
- isProxyValid(valid=1000)
Check if the stored proxy is valid
- isValid()
Check the sanity of the Computing Element definition
- loadBatchSystem(batchSystemName)
Instantiate object representing the backend batch system
- Parameters:
batchSystemName (str) – name of the batch system
- sendOutput(stdid, line)
Callback function such that the results from the CE may be returned.
- setCPUTimeLeft(cpuTimeLeft=None)
Update the CPUTime parameter of the CE classAd, necessary for running in filling mode
- setParameters(ceOptions)
Add parameters from the given dictionary overriding the previous values
- Parameters:
ceOptions (dict) – CE parameters dictionary to update already defined ones
- setProxy(proxy, valid=0)
Set proxy for this instance
- setToken(token)
- shutdown()
Wait for all futures (jobs) to complete
- submitJob(executableFile, proxy=None, inputs=None, **kwargs)
Method to submit job. This method will submit to a ProcessPoolExecutor, which returns Future objects.
- writeProxyToFile(proxy)
CE helper function to write a CE proxy string to a file.
- DIRAC.Resources.Computing.PoolComputingElement.executeJob(executableFile, proxy, taskID, inputs, **kwargs)
wrapper around ce.submitJob: decides which CE to use (Sudo or InProcess or Singularity)