Editing DIRAC code

Warning

If you need to do legacy development you can refer to the v7r1 version of this page.

Introduction and prerequisites

What is this for?

Here we describe the suggested method for editing and unit-testing DIRAC code, and direct extensions of it.

What is this NOT for?

  • This method is NOT specific for the WebAppDIRAC or Pilot code development, although several things that are described here can be applied.

  • There will be no explanation of how to use Git. If you’re unfamiliar with Git there are many excellent resources online. A particularly nice interactive tutorial can be found at learngitbranching.js.org.

Notes before continuing

OS: Linux or macOS should both be fine. If you wish to develop on Windows the easiest solution is probably to use the Windows Subsystem for Linux.

shell: These instructions will work out of the box in any bourne compatible shell (bash/zsh/ksh) but other shells, such as fish/csh/tcsh, will work with only minor adjustments.

repository: DIRAC’s GIT repositories are hosted on GitHub and you will need a GitHub account to contribute.

Stuff you need to have installed

git: DIRAC’s version control system of choice is git and most computers will already have this installed. If not, you can easily install it by searching online for your operating system name and “install git”.

editor: Almost any text editor is fine so if you already have a preference for vim/emacs/sublime/eclipse/pycharm/atom use that. If you’re new to development, Visual Studio Code is free modern editor with an extensive plugin ecosystem and many nice IDE like features. Regardless of which editor you choose it’s worth investigating what plugins are available.

python: DIRAC is written in Python so you will need to have a Python environment available. While there are many possible ways of doing this, the simplest option for most people is to use either conda or DIRACOS2 (Linux only). Details about how to use both options are included below.

Creating a development environment with Conda

Note

The Python packaging ecosystem is very old and as a result there has been a lot of fragmentation. Here are a few points that might be worthwhile to clarify:

  • anaconda vs miniconda vs miniforge vs mambaforge: Anaconda Inc. is the original company behind the conda package manager, however the project has slowly evolved to be community project. Anaconda Inc. provides two free installers (anaconda/miniconda) as well as various paid support plans. Community efforts are centered around conda-forge, a NumFOCUS sponsored project to provide recipes, build infrastructure and distributions for the conda package manager. While anaconda and conda-forge provided packages are mostly compatible there can be issues when mixing them. As a result, it is often easier to use the official conda-forge installer bundles miniforge (conda only) or mambaforge (with both conda and mamba).

  • conda vs mamba: As conda has grown, some applications have had issues with conda being slow to solve and install some environments. Mamba is built upon conda and uses libsolv to provide a fast alternative install command. The two are interoperable and a common workflow is to replace calls to conda install/conda create/conda env create with mamba install/mamba create/mamba env create. The only command which cannot be substituted is conda activate as it is implemented as a shell function to allow it to change environment variables.

The conda package manager’s main selling point is that you can install binary packages in any location without needing system level privileges. Ideally users should create an “environment” for a given purpose and “activate” it whenever they want that software to be used. This has the advantage that unrelated activities (e.g. DIRAC development and physics analysis) can each have their own environments and any software installed won’t cause the other activity to break.

Installing conda

Note

This step can be skipped if you already have conda installed.

Download and run one of the “mambaforge” installers. The following snippet should work on most UNIX-like systems provided you adjust the URL on line one to download the installer for the appropriate platform:

1wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
2bash Mambaforge-Linux-x86_64.sh -b -p $HOME/mambaforge
3rm Mambaforge-Linux-x86_64.sh
4# Activate the environment manually the first time
5eval "$("$HOME/mambaforge/bin/conda" shell.bash hook)"
6# Make it so that conda doesn't automatically activate the base environment
7conda config --set auto_activate_base false
8# Automatically make the "conda" shell function available
9conda init bash

Assuming you have already cloned DIRAC, you can create an environment for DIRAC development by running:

1mamba env create --name dirac-development --file environment.yml

Whenever you open a new terminal you can then activate the development environment by running:

1conda activate dirac-development

Creating a development environment with DIRACOS2

You can create a development environment in a new directory named diracos/ by running:

1wget https://github.com/DIRACGrid/DIRACOS2/releases/latest/download/DIRACOS-Linux-x86_64.sh
2bash DIRACOS-Linux-x86_64.sh
3rm DIRACOS-Linux-x86_64.sh

Whenever you wish to use the new environment you just need to source the diracos/diracosrc file:

1source diracos/diracosrc

Setting up your development installation

The following steps will try to guide you on setting up a development installation for DIRAC.

Checking out the source

  1. Go to a directory where you would like to do your development work (replacing MY_DEV_DIRECTORY as appropriate:

1mkdir $HOME/MY_DEV_DIRECTORY/
2cd $HOME/MY_DEV_DIRECTORY/

  1. Check out DIRAC source code inside a directory of your choosing. DIRAC source is hosted on github.com. Fork it (online!), then:

1git clone https://github.com/YOUR_GITHUB_USERNAME/DIRAC.git

You must replace YOUR_GITHUB_USERNAME with the username that we have registered on github. This will create a folder named DIRAC containing the DIRAC source code.

  1. Now create a remote (in git terms) in the local repository called upstream that points to your source repository on GitHub.

1cd DIRAC
2git remote add upstream https://github.com/DIRACGrid/DIRAC.git
3git fetch upstream

  1. The next release of the DIRAC code lives on the integration branch and you can checkout it locally by running:

1git checkout upstream/integration

Maintenance branches for existing releases are named rel-vXrY.

Installing the local version

The locally cloned source code can be installed inside your conda or DIRACOS2 environment by running the following from inside your local repository:

1pip install -e .[testing]

This creates an editable installation meaning any changes you make will be automatically discovered whenever you next import DIRAC. Additionally the testing extra causes pip to install useful dependencies such as pytest and pycodestyle.

Formatting

The DIRAC code is formatted with black. This can be configured to run automatically before each commit using pre-commit. We recommend installing pre-commit system wide, using your operating system’s package manager, e.g. apt-get install pre-commit on Debian based system, dnf install pre-commit on Fedora like systems or brew install pre-commit on macOS. Once installed it can be enabled for your local clone using:

pre-commit install --allow-missing-config

Running unit tests

Unit tests are used to provide simple, quick to run, tests which don’t require any special environments. All of the unit tests are kept along side the DIRAC sources so you can run them with pytest by running:

1pytest src/

Note the src/ is important to avoid running the integration and certification tests. For more information about testing DIRAC please refer to Testing (VO)DIRAC.

Linting

A linter is a tool which statically (i.e. without executing it) makes checks on code for problems and bad practices. In Python they are especially useful for catching errors before running the code, similarly to how a compiler can find issues with C++ code before it is executed. Linters are also often used for enforcing stylistic standards, you can find more about the conventions used with DIRAC at Coding Conventions.

The main tools used in DIRAC are:

pylint

This looks for code which might be invalid for some reason (e.g. undefined variable names or missing methods). It can by ran with:

1pylint src/

pycodestyle

This helps ensure the code meets the DIRAC style guidelines and can be executed by running:

1pycodestyle

Repository structure

Looking at the root directory:

ls -al $DEVROOT/DIRAC/

will tell you a lot about the DIRAC code structure. Note that:

  • there is a tests/ directory

  • there is a docs/ directory

  • there is a src/ directory

  • there are some base files (README, LICENCE, etc.) and some dotfiles, which will become useful reading further.

Unsurprisingly:

  • tests contains tests - and specifically, it contains all the non-unit tests

  • docs contains… documentation (including this very same page!)

  • the src/ directory contain the (python) code of the DIRAC systems

Ready!

You’re ready for DIRAC development! (or at least, good part of it). What can you do with what you have just done?

  1. Editing code (this is the obvious!)

  2. Running unit tests

  3. Running linters

So, this is “pure code”! And what you CAN’T do (yet)?

  • You can’t get a proxy

  • you can’t interact with configuration files nor with the Configuration System

  • you can’t run services, nor agents (no DIRAC components)

Next?

This depends from you: if you are a casual developer, you can stop here, and look into sections Check your installation and the following Your first DIRAC code

Alternatively, if you want to do more, you may proceed to the section Developing stuff that runs.