DIRAC Projects¶
DIRAC is used by several user communities. Some of them are creating their own modules for DIRAC. These modules require a certain version of DIRAC in order to function properly. Virtual organizations have to be able to create their own releases of their modules and install them seamlessly with dirac-install. This is achieved by creating and releasing software projects in the DIRAC framework.
Preparing DIRAC distribution¶
Releases schema¶
DIRAC modules are released and distributed in projects. Each project has a releases.cfg configuration file where the releases, modules and dependencies are defined. A single releases.cfg can take care of one or more modules. releases.cfg file follows a simplified schema of DIRAC’s cfg format. It can have several sections, nested sections and options. Section Releases contains the releases definition. Each section in the Releases section defines a release. The name of the section will be the release name. Each release will contain a list of dependencies (if any) and a list of modules (if more than one). An example of a release.cfg for a single module is shown below:
DefaultModules = MyExt
Sources
{
MyExt = git://somerepohosting/MyExt.git
}
Releases
{
v1r2p3
{
depends = DIRAC:v5r12
}
v1r2p2
{
depends = DIRAC:v5r12p1
}
}
RequiredExternals
{
Server = tornado>=4.4.2, apache-libcloud==2.2.1
Client = apache-libcloud==2.2.1
}
The DefaultModules option (outside any section) defines what modules will be installed by default if there’s nothing explicitly specified at installation time. Because there is only one module defined in DefaultModules each release will try to install the MyExt module with the same version as the release name. Each release can require a certain version of any other project (DIRAC is also an project).
The RequiredExternals section contains lists of extra python modules that can be installed with a pip installer for different installation types. Each module in the lists is specified in a format suitable to pass to the pip command.
An example with more than one module follows:
DefaultModules = MyExt
RequiredExtraModules = WebApp
Sources
{
MyExt = git://somerepohosting/MyExt.git
MyExtExtra = svn | http://someotherrepohosting/repos/randomname/MyExtExtra/tags
}
Releases
{
v1r2p3
{
Modules = MyExt:v1r2p1, MyExtExtra:v1r1p1
Depends = DIRAC:v5r12p1
}
v1r2p2
{
Modules = MyExt:v1r2p1, MyExtExtra:v1r1
Depends = DIRAC:v5r12
}
}
If a project requires a module that is not installed by default from another project to be installed, it can be defined in the RequiredExtraModules option. For instance, DIRAC project contains DIRAC and Web. But by default DIRAC project only installs DIRAC module. If another project requires the DIRAC Web module to be installed it can be defined in this option. That way, when installing this other project, Web module will also be installed.
The Modules option can define explicitly which modules (and their version) to install. This is useful if a given VO is managing more than one module. In that scenario a release can be a combination of modules that can evolve independently. By defining releases as groups of modules with their versions the VO can ensure that a release is consistent for its modules. DIRAC uses this mechanism to ensure that the DIRAC Web will always be installed with a DIRAC version that it works with.
The Sources section defines where to extract the source code from for each module.
The releases are created using the dirac-distribution docker image, which can be found in GitHub package registry or in docker hub:
docker.pkg.github.com/diracgrid/management/dirac-distribution:latest (https://github.com/DIRACGrid/management/packages/79929)
diracgrid/dirac-distribution (https://hub.docker.com/r/diracgrid/dirac-distribution)
Pull it and run inside the dirac-distribution command:
docker pull diracgrid/dirac-distribution
python3 dirac-distribution.py -r v7r0p8
The above works also for DIRAC extensions, in this case just remember to specify the project name, e.g.:
python3 dirac-distribution.py --release v10r0-pre11 --project LHCb
The dirac-distribution image is re-created weekly starting from the management repository
You can also pass the releases.cfg to use via command line using the -relcfg switch. dirac-distribution will generate a set of tarballs, release notes in html and md5 files.
In the end of its execution, the dirac-distribution will print out a command that can be used to upload generated release files to a predefined repository, as you read above.
dirac-distribution knows how to handle several types of VCS. The ones supported are:
- file
A directory in the filesystem. dirac-distribution will assume that the directory especified contains the required module version of the module.
- svn
A subversion url that contains a directory with the same name as the version to be tagged. If the module version is v1r0 and the url is http://host/extName, dirac-distribution will check out http://host/extName/v1r0 and assume it contains the module contents.
- hg
A mercurial repository. dirac-distribution will check out the a tag with the same name as the module version and assume it contains the module contents.
- git
A git repository. dirac-distribution will clone the repository and check out to a tag with the same name as the module version and assume it contains the module contents.
Some of the VCS URLs may not explicitly define which VCS has to be used (for instance http://… it can
be a subversion or mercurial repository). In that case the option value can take the form <vcsName> | <vcsURL>
.
In that case dirac-distribution will use that VCS to check out the source code.
When installing, a project name can be given. If it is given dirac-install will try to install that project instead of the DIRAC project. dirac-install will have a mapping to discover where to find the releases.cfg based on the project name. Any VO can modify dirac-install to directly include their repositories inside dirac-install in their module source code, and use their modified version. DIRAC developers will also maintain a project name to releases.cfg location mapping in the DIRAC repository. Any VO can also notify the DIRAC developers to update the mapping in the DIRAC repository so dirac-install will automatically find the project’s releases.cfg without any change to dirac-install.
If a project is given, all modules inside that releases.cfg have to start with the same name as the project. For instance, if dirac-install is going to install project LHCb, all modules inside LHCb’s releases.cfg have to start with LHCb.
How to define how to make a project distribution¶
dirac-distribution needs to know where to find the releases.cfg file. dirac-distribution will load some global configuration from a DIRAC web server. That configuration can instruct dirac-distribution to load the project defaults file from a URL. Those defaults will define default values for dirac-distribution and dirac-install command line options. An example of a project defaults file would be::
#Where to load the release.cfg file from
Releases = https://github.com/DIRACGrid/DIRAC/raw/integration/releases.cfg
#Where to download the released tarballs from
BaseURL = http://diracproject.web.cern.ch/diracproject/tars/
#How to upload the release tarballs to the BaseURL
UploadCommand = ( cd %OUTLOCATION% ; tar -cf - %OUTFILENAMES% ) | ssh webuser@webhost 'cd /diracproject/tars && tar -xvf - && ls *.tar.gz > tars.list'
Once the tarballs and required files have been generated by dirac-distribution (see below), if UploadCommand is defined the variables will be substituted and the final command printed to be executed by the user.
dirac-install will download the project files from the BaseURL location.
The defaults file is defined per project and can live in any web server.
Installation¶
When installing, dirac-install requires a release version and optionally a project name. If the project
name is given dirac-install will try to load the project’s versioned release-<projectName>-<version>.cfg
instead of the DIRAC’s one (this file is generated by dirac-distribution when generating the release).
dirac-install has several mechanisms on how to find the URL where the released tarballs and releases
files for each project are. dirac-install will try the following steps:
Load DIRAC’s default global locations. This file contains the default values and paths for each project that DIRAC knows of and it’s maintained by DIRAC developers.
Load the required project’s defaults file. DIRAC’s default global locations has defined where this file is for each project. It can be in a URL that is maintained by the project’s developers/maintainers.
If an option called BaseURL is defined in the project’s defaults file then use that as the base URL to download the releases and tarballs files for the projects.
If it’s defined inside dirac-install, use it.
If not found then the installation is aborted.
The release-<projectName>-<version>.cfg
file will specify which module and version to install. All modules
that are defined inside a release-<projectName>-<version>.cfg
will be downloaded from the same parent URL.
For instance, if the release-<projectName>-<version>.cfg
is in http://diracgrid.org/releases/releases.cfg
and DIRAC v5r14 has to be installed, dirac-install will try to download it from
http://diracgrid.org/releases/DIRAC-v5r14.tar.gz
.
If nothing else is defined, dirac-install will only install the modules defined in DefaultModules option.
To install other modules that are defined in the release-<projectName>-<version>.cfg
the -e flag has to
be used.
Once all the modules defined in the release-<projectName>-<version>.cfg
are installed. dirac-install
will try to load the dependencies. The depends option defines on which projects the installed project
depends on. That will trigger loading that release-<projectName>-<version>.cfg
and process it as the
main one was processed. dirac-install will try to resolve recursively all the dependencies either until
all the required modules are installed or until there’s a mismatch in the requirements. If after resolving
all the release-<projectName>-<version>.cfg
an module is required to be installed with more than one
version, an error will be raised and the installation stopped.
The set of parameters used to install a project is called an installation. dirac-install also has support for installations. Each installation is a set of default values for dirac-install. If the -V switch is used dirac-install will try to load the defaults file for that installation and use those defaults for the arguments.
Reference of releases.cfg schema¶
#List of modules to be installed by default for the project
DefaultModules = MyExt
#Extra modules to be installed
RequiredExtraModules = WebApp
#Section containing where to find the source code to generate releases
Sources
{
#Source URL for module MyExt
MyExt = git://somerepohosting/MyExt.git
MyExtExtra = svn | http://someotherrepohosting/repos/randomname/MyExtExtra/tags
}
#Section containing the list of releases
Releases
{
#Release v1r2p3
v1r2p3
{
#(Optional) Contains a comma separated list of modules for this release and their version in format
# *extName(:extVersion)? (, extName(:extVersion)?)** .
#If this option is not defined, modules defined in *DefaultExtensions* will be installed
# with the same version as the release.
Modules = MyExt:v1r2p1, MyExtExtra:v1r1p1
#(Optional) Comma separated list of projects on which this project depends in format
# *projectName(:projectVersion)? (, projectName(:projectVersion)?)**.
#Defining this option triggers installation on the depended project.
#This is useful to install the proper version of DIRAC on which a set of modules depend.
Depends = DIRAC:v5r12p1
}
v1r2p2
{
Modules = MyExt:v1r2p1, MyExtExtra:v1r1
}
}
Reference of an installation’s defaults file¶
#(Everything in here is optional) Default values for dirac-install
LocalInstallation
{
#Install the requested project instead of this one
# Useful for setting defaults for VOs by defining them as projects and
# using this feature to install DIRAC instead of the VO name
Project = DIRAC
#Release to install if not defined via command line
Release = v1r4
#Modules to install by default
ModulesToInstall = MyExt
#Type of externals to install (client, client-full, server)
ExternalsType = client
#Version of lcg bundle to install
LcgVer = v14r2
#Install following DIRAC's pro/versions schema
UseVersionDir = False
#Force building externals
BuildExternals = False
#Build externals if the required externals is not available
BuildIfNotAvailable = False
#Enable debug logging
Debug = False
}
Reference of global default’s file¶
Global defaults is the file that dirac-install will try to load to discover where the each project’s
defaults.cfg
file is. The schema is as follows:
Projects
{
#Project name
ProjectName
{
#Where to find the defaults
DefaultsLocation = http://somehost/somepath/defaultsProject.cfg
#Release file location
ReleasesLocation = http://endoftheworld/releases.cfg
}
Project2Name
{
DefaultsLocation = http://someotherhost/someotherpath/chunkybacon.cfg
}
}
Installations
{
#Project name or installation name
InstallationName
{
#Location of the defaults for this installation
DefaultsLocation = http://somehost/somepath/defaultsProject.cfg
#Default values for dirac-install
LocalInstallation
{
#This section can contain the same as the LocalInstallation section in each project's defaults.cfg
}
}
#And repeat for each installation or project
OtherInstallation
{
....
}
#Alias with another names
ThisIsAnAlias = InstallationName
}
All the values in the defined defaults file file take precedence over the global ones. This file is useful for DIRAC maintainers to keep track of all the projects installable via native dirac-install.
Common pitfalls¶
Installation will find a given releases.cfg by looking up the project name. All modules defined inside a releases.cfg have to start with the same name as the project. For instance, if the project is MyVO, all modules inside have to start with MyVO. MyVOWeb, MyVOSomething and MyVO are all valid module names inside a MyVO releases.cfg