10.2.3. Pilots bootstrapping

Pilot3 needs some Python files, and a JSON file for bootstrapping. We simply call this file the pilot.json file. The pilot.json file is created starting from information found in the Configuration Service.

The pilot wrapper (the script that starts the pilot, which is effectively equivalent to what SiteDirectors send) expects to find and download such a pilot.json file from a known location, or a set of them. Such a location should be exposed via https:// by, for example, the DIRAC WebApp webserver (but not necessarily).

To make sure that the pilot.json is created and uploaded, there’s an agent that can be used for your convenience: the agent PilotsSyncAgent syncs CS and pilot files to a web server of your choice. The (list of) location(s) where the agent would upload the pilot files can be added in the CS agent option UploadLocations. If this location is not set, then the agent will try to use the Operations option Pilot/pilotFileServer.

If more than one location is used, add them as a list. We suggest to add at least the URL of the DIRAC WebApp webserver, but multiple locations are also possible, and advised.

Starting with DIRAC version v7r2, the file uploads is completely on the balancer (nginx) side. Make sure your balancer is set to load files by following the instructions.

Other options that can be set also in the Operations part of the CS include:

pilotRepo	Pointer to git repository of DIRAC pilot	pilotRepo = https://github.com/DIRACGrid/Pilot.git The value above is the default
pilotVORepo	Pointer to git repository of VO DIRAC extension of pilot. This option is needed in case you have an extension of the pilot	pilotVORepo = https://github.com/MyDIRAC/VOPilot.git
pilotScriptsPath	Path to the code, inside the Git repository	pilotScriptsPath = Pilot This value is the default
pilotVOScriptsPath	Path to the code, inside the Git repository of the pilot extension	pilotVOScriptsPath = VOPilot
pilotRepoBranch	Branch to use, inside the Git repository, of the pilot code to be used	pilotRepoBranch = master The value above is the default
pilotVORepoBranch	Branch to use, inside the Git repository, of the pilot code extension to be used	pilotVORepoBranch = master The value above is the default
uploadToWebApp	Whether to try to upload the files to the list of server specified	uploadToWebApp = True The value above is the default
workDir	Local directory of the master CS where the files will be downloaded before the upload	workDir = /tmp/pilotSyncDir There is no default (so /opt/dirac/runit/Configuration/Server)

10.2.3.1. Pilots environment

There is a small number of environment variables that can be set by the pilot to control the behaviour of DIRAC jobs.

DIRAC_PROCESSORS

The number of processors the pilot can use

DIRAC_JOB_PROCESSORS

The number of processors the jobs administered by the pilot can use (a subset of DIRAC_PROCESSORS)

DIRAC_WHOLENODE

A boolean flag indicating the pilot is exploiting the whole node (normally, False)

10.2.3.2. Install non-released code from within the Pilot

To install non-released code from within the pilot, the dirac-pilot command can be run with the –modules option. This option can be steered from the CS too.

For example:

dirac-pilot.py --modules=https://github.com/chaen/DIRAC.git:::DIRAC:::rel-v7r3_FEAT_proxyStrength

would end up installing a specific branch (rel-v7r3_FEAT_proxyStrength) pushed to github.

An easy way to try is to add the Modules option in the CS configuration of one of your ComputingElements:

Modules=https://github.com/chaen/DIRAC.git:::DIRAC:::rel-v7r3_FEAT_proxyStrength