Documenting your developments

Where should you document your developments? Well, at several places, indeed, depending on the documentation we are talking about:

Code documentation

This is quite easy indeed. It’s excellent practice to add docstring to your python code. The good part of it is that tools like pyDev can automatically read it. Also your python shell can (try help()), and so does iPython (just use ? for example). Python stores every docstring in the special attribute __doc__.

Pylint will, by default, complain for every method/class/function left without a docstring.

Release documentation

Releases documentation can be found in 2 places: release notes, and github wiki:

  • release notes are automatically created from the first comment in the pull requests, please describe the changes between BEGRINRELEASENOTES and ENDRELEASENOTES as presented by the template provided
  • The github wiki can contain a section, for each DIRACGrid repository, highlighting update operations, for example the DIRAC releases notes are linked from the DIRAC wiki main page.

Full development documentation

As said at the beginning of this guide, this documentation is in git at DIRAC/docs. It is very easy to contribute to it, and you are welcome to do that. You don’t even have to clone the repository: github lets you edit it online. This documentation is written in RST and it is compiled using sphinx.

Some parts of the documentation can use UML diagrams. They are generated from .uml files with plantuml. Sphinx support plantuml but ReadTheDocs didn’t, so you have to convert .uml in .png with java -jar plantuml.jar file.uml.

Component Options documentation

The agent, service and executor options are documented in their respective module docstring via literal include of their options in the ConfigTemplate.cfg:

.. literalinclude:: ../ConfigTemplate.cfg
  :start-after: ##BEGIN MyComponent
  :end-before: ##END
  :dedent: 2
  :caption: MyComponent options

Around the section in the ConfigTemplate.cfg configuring the component the ##BEGIN MyComponent and ##END tags need set so that the include is restricted to the section belonging to the component. The options :dedent: and :caption: are optional, but create a nicer output.

Building the Documentation

The DIRAC documentation is created using sphinx and makes use of the diracdoctools to create the code documentation, command references, and concatenate the ConfigTemplates into one file. The diracdoctools are located in the DIRAC/docs folder, but they can be pip installed for use outside of DIRAC. In ReadTheDocs we do not install DIRAC via dirac-install, so we have to install dependencies via pip, or Mock the packages if they cannot be installed.

This requirements.txt shows how the diracdoctools can be installed from the GitHub folder directly, and how other dependencies can also be installed.

The docs/requirements.txt used in readthedocs
# cannot use DIRAC requirements because of inability to install pycurl (from FTS dependency) on RTD
#-r https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/requirements.txt
#-e git+https://github.com/DIRACGrid/DIRAC/@integration#egg=diracdoctools&subdirectory=docs
M2Crypto==0.32
Sphinx>=1.8.0
boto3
elasticsearch_dsl
future
futures
matplotlib
mock
mysql-python
psutil
pyasn1>0.4.1
pyasn1_modules
pyparsing
pytz
recommonmark
sqlalchemy
subprocess32
suds

For DIRAC extensions you also have to install DIRAC as a requirement, so add this line to your requirements.txt

git+https://github.com/DIRACGrid/DIRAC/@integration

Some packages (e.g., FTS3) cannot be installed in READTHEDOCS, so we let sphinx mock these packages in the source/conf.py add the import at the top and set the autodoc_mock_imports variable.

from diracdoctools import fakeEnvironment, DIRAC_DOC_MOCK_LIST … autodoc_mock_imports = DIRAC_DOC_MOCK_LIST

Using the DIRAC_DOC_MOCK_LIST you can inherit the packages that are mocked for DIRAC, and extend it with packages needed to be mocked for your extension. The fakeEnvironment sets up a special mock for GSI and simply needs to be imported.

In the sphinx configuration file (source/conf.py), the functionality can then be called to create code reference, command reference and concatenated CFG files.

docs/source/conf.py
if os.environ.get('READTHEDOCS') == 'True':
  setUpReadTheDocsEnvironment(moduleName='DIRAC')

  # re-create the RST files for the command references
  LOG.info('Building command reference')
  from diracdoctools.cmd.commandReference import run as buildCommandReference
  buildCommandReference(configFile='../docs.conf')

  # singlehtml build needs too much memory, so we need to create less code documentation
  buildType = 'limited' if any('singlehtml' in arg for arg in sys.argv) else 'full'
  LOG.info('Chosing build type: %r', buildType)
  from diracdoctools.cmd.codeReference import run as buildCodeDoc
  buildCodeDoc(configFile='../docs.conf', buildType=buildType)

  # Update dirac.cfg
  LOG.info('Concatenating dirac.cfg')
  from diracdoctools.cmd.concatcfg import run as updateCompleteDiracCFG
  updateCompleteDiracCFG(configFile='../docs.conf')

The configuration for diracdoctools is done via a configuration file located in the docs folder. Just copy the file DIRAC/docs/docs.conf and adapt it to your needs. All options are mandatory unless otherwise stated. If certain features are not used, simply leave the values empty

docs/docs.conf
[Docs]

# name of the module to documented, DIRAC, voDIRAC, ExtensionDIRAC
module_name = DIRAC

# path to the source folder relative to docs.conf
source_folder = ../

[Code]
# building code reference
# where to place the code reference
docs_target_path = ./source/CodeDocumentation
# where Custom docstrings can be found, see
# DIRAC/docs/diracdoctools/CustomizedDocs for an example
customdocs_folder = ./diracdoctools/CustomizedDocs

# add :private-members: to autodoc for matching module
document_private_members = FCConditionsParser
# add :no-inherited: to autodoc for matching module
no_inherited_members =

# only creating dummy files, because they cannot be safely imported due to sideEffects
create_dummy_files = lfc_dfc_copy, lfc_dfc_db_copy, JobWrapperTemplate, PlotCache,
                     PlottingHandler

# do not include these files in the documentation tree
ignore_folders = diracdoctools, /test, /scripts
ignore_files = setup.py

[CFG]
# concatenating ConfigTemplates
# which file to use as a base
base_file = ../dirac.cfg
# where to place the resulting file
target_file = source/AdministratorGuide/Configuration/ExampleConfig.rst

[Commands]
# list of commands which are not executed to get doc files, rst files have to be
# created by hand
ignore_commands=
# does not have --help, deploys scripts
  dirac-deploy-scripts,
# does not have --help, starts compiling externals
  dirac-compile-externals,
# does not have --help
  dirac-install-client,
# does not have --help
  dirac-framework-self-ping,
# obsolete
  dirac-dms-add-files,
# just prints version, no help
  dirac-version,
# just prints platform, no help
  dirac-platform,
# no doc, purely internal use
  dirac-agent,
# no doc, purely internal use
  dirac-executor,
# no doc, purely internal use
  dirac-service,

# scripts where the module docstring should be added to the documentation
add_module_docstring = dirac-install

# arbitrary number of these sections can be added, they result in folders with
# rst files for each command matching the pattern. all options are mandatory,
# except indexFile and prefix
[commands.admin]
# pattern to match in the full path of the command names
pattern = admin, accounting, FrameworkSystem, framework, install, utils,
        dirac-repo-monitor, dirac-jobexec, dirac-info, ConfigurationSystem, Core, rss,
        transformation, stager
# title of the section
title = Admin
# this list of patterns will reject scripts that are matched by the patterns above
exclude =
# the scripts in this list are added to the indexFile, but the rst file has to be provided manually
manual = dirac-cert-convert.sh, dirac-platform, dirac-version
# where to place the rst files
sectionPath = source/AdministratorGuide/CommandReference
# optional: only if a hand curated rst file exists
indexFile = source/AdministratorGuide/CommandReference/index.rst
# optional: prefix to add when creating reference anchors, anchor will be prefix_scriptName
prefix = admin

[commands.dms]
pattern = dms
title = Data Management
manual =
exclude =
sectionPath = source/UserGuide/CommandReference/%(title)s

[commands.wms]
pattern = wms
title = Workload Management
manual =
exclude =
sectionPath = source/UserGuide/CommandReference/%(title)s

[commands.z_section3]
pattern = dirac-proxy, dirac-info, myproxy, -resource-
title = Others
exclude =
manual = dirac-cert-convert.sh, dirac-platform, dirac-version
sectionPath = source/UserGuide/CommandReference/%(title)s

For local testing of the documentation, the scripts can also be called directly, like this example from the Makefile shows.

docs/Makefile
htmlall:
	diracdoctools/scripts/dirac-docs-build-commands.py
	diracdoctools/scripts/dirac-docs-build-code.py
	diracdoctools/scripts/dirac-docs-concatenate-diraccfg.py
	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

Code Reference

The code reference is either created by calling run from diracdoctools.cmd.codeReference or by invoking the script dirac-docs-build-code.py. This creates an rst file for each python file, using autodoc to document all classes inside those modules. The actual documentation is build when sphinx is invoked, which must be able to import the modules and all their dependencies

Command Reference

The command references can be created by calling run from diracdoctools.cmd.commandReference, or by calling the dirac-docs-build-command.py script. [commands.section] will result in a list of commands with links to their documentation, which is based on the output of their --help. The resulting index.rst has to be included explicitly in the documentation. If indexFile is specified it has to contain all the links itself, or warnings are generated for missing and superfluous entries.

Note

The parsing of the --help output is extremely limited and naive. You must not end a line with a colon : unless you intend to create a verbatim block after it.

CFG File

If you developed your own systems, you can concatenate all the settings defined in ConfigTemplate.cfg files into one large file. You need a base dirac.cfg file and a location where to put the final result