Documenting your developments 

Where should you document your developments? Well, at several places, indeed, depending on the documentation we are talking about:

Code documentation 

This is quite easy indeed. It’s excellent practice to add docstring to your python code. The good part of it is that tools like pyDev can automatically read it. Also your python shell can (try help()), and so does iPython (just use ? for example). Python stores every docstring in the special attribute __doc__.

Pylint will, by default, complain for every method/class/function left without a docstring.

Release documentation 

Releases documentation can be found in 2 places: release notes, and github wiki:

release notes are automatically created from the first comment in the pull requests, please describe the changes between BEGRINRELEASENOTES and ENDRELEASENOTES as presented by the template provided

The github wiki can contain a section, for each DIRACGrid repository, highlighting update operations, for example the DIRAC releases notes are linked from the DIRAC wiki main page.

Full development documentation 

As said at the beginning of this guide, this documentation is in git at DIRAC/docs. It is very easy to contribute to it, and you are welcome to do that. You don’t even have to clone the repository: github lets you edit it online. This documentation is written in RST and it is compiled using sphinx.

Some parts of the documentation can use UML diagrams. They are generated from .uml files with plantuml. Sphinx support plantuml but ReadTheDocs didn’t, so you have to convert .uml in .png with java -jar plantuml.jar file.uml.

Component Options documentation 

The agent, service and executor options are documented in their respective module docstring via literal include of their options in the ConfigTemplate.cfg:

.. literalinclude:: ../ConfigTemplate.cfg
  :start-after: ##BEGIN MyComponent
  :end-before: ##END
  :dedent: 2
  :caption: MyComponent options

Around the section in the ConfigTemplate.cfg configuring the component the ##BEGIN MyComponent and ##END tags need set so that the include is restricted to the section belonging to the component. The options :dedent: and :caption: are optional, but create a nicer output.

Building the Documentation 

The DIRAC documentation is created using sphinx and makes use of the diracdoctools to create the code documentation, command references, and concatenate the ConfigTemplates into one file. The diracdoctools are located in the DIRAC/docs folder, but they can be pip installed for use outside of DIRAC.

For local development the easiest way to build the docs is by running these commands from a standard Python 3 DIRAC development environment (see Creating a development environment with Conda.)

pip install docs/
cd docs
make htmlall
python -m http.server -d build/html/

You can now browse the docs locally by going to the URL shown in your terminal.

In the sphinx configuration file (source/conf.py), the functionality can then be called to create code reference, command reference and concatenated CFG files.

docs/source/conf.py

if os.environ.get("READTHEDOCS") == "True":
    setUpReadTheDocsEnvironment(moduleName="DIRAC", location="../../src")

    # re-create the RST files for the command references
    LOG.info("Building command reference")
    from diracdoctools.cmd.commandReference import run as buildCommandReference

    if buildCommandReference(configFile="../docs.conf") != 0:
        raise RuntimeError("Something went wrong with the documentation creation")

    # singlehtml build needs too much memory, so we need to create less code documentation
    buildType = "limited" if any("singlehtml" in arg for arg in sys.argv) else "full"
    LOG.info("Chosing build type: %r", buildType)
    from diracdoctools.cmd.codeReference import run as buildCodeDoc

    buildCodeDoc(configFile="../docs.conf", buildType=buildType)

    # Update dirac.cfg
    LOG.info("Concatenating dirac.cfg")
    from diracdoctools.cmd.concatcfg import run as updateCompleteDiracCFG

    updateCompleteDiracCFG(configFile="../docs.conf")

The configuration for diracdoctools is done via a configuration file located in the docs folder. Just copy the file DIRAC/docs/docs.conf and adapt it to your needs. Note that some options are mandatory, this is indicated in the comments to them.

docs/docs.conf

# See more options:
# https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/docs/docs.conf

[Docs] # general documentation settings

# [mandatory] name of the module to documented, DIRAC, voDIRAC, ExtensionDIRAC
module_name = DIRAC

# path to the source folder relative to docs.conf
# By default: ../src/{module_name}
# source_folder = ../src/DIRAC

[Code] # building code reference

# Directory where to place the code reference
# By default: source/CodeDocumentation
# docs_target_path = source/CodeDocumentation

# Directory where Custom docstrings can be found
# See DIRAC/docs/diracdoctools/CustomizedDocs for an example
# By default: diracdoctools/CustomizedDocs
# customdocs_folder = diracdoctools/CustomizedDocs

# add :private-members: to autodoc for matching module
document_private_members = FCConditionsParser

# add :no-inherited: to autodoc for matching module
# Useful to avoid markup errors where the inherited members come from a module
# whose docstrings are not reST formatted
no_inherited_members =
  DIRAC.Core.Utilities.Graphs.GraphUtilities,
  DIRAC.DataManagementSystem.private.HttpStorageAccessHandler,
  DIRAC.FrameworkSystem.private.standardLogging.LogLevels,

# only creating dummy files, because they cannot be safely imported due to sideEffects
create_dummy_files = lfc_dfc_copy, lfc_dfc_db_copy, JobWrapperTemplate, JobWrapperOfflineTemplate

# do not include these files in the documentation tree
ignore_folders = diracdoctools, /test, /scripts
ignore_files = setup.py

# To add section "Commands" with all commands description to newly created "code documentation" (described in the [Code] section).
# By default: false
# add_commands_section = true

[CFG] # concatenating ConfigTemplates

# which file to use as a base
# By default: ../dirac.cfg
# base_file = ../dirac.cfg

# where to place the resulting file
# By default: source/ExampleConfig.rst
target_file = source/AdministratorGuide/Configuration/ExampleConfig.rst

[Commands] # building commands references

# List of commands which are not executed to get doc files
# ignore_commands = command_to_ignore

# Directory where to place the RST file with all commands specification.
# By default: source/Commands
# sectionPath = source/CodeDocumentation/Commands

# Name of the RST file with all commands specification.
# By default: index.rst
# fileName = index.rst

[commands.admin] # grouping commands by scope

# [mandatory] title of the section
title = Admin

# [mandatory] pattern to match in the full path of the command names.
pattern = admin, accounting, FrameworkSystem, framework, install, utils,
        dirac-repo-monitor, dirac-jobexec, dirac-info, ConfigurationSystem, Core, rss,
        transformation, stager

# this list of patterns will reject scripts that are matched by the patterns above
# exclude = user

# [mandatory] path to the target RST file
sectionPath = source/AdministratorGuide/CommandReference

# Name of the target RST file
# By default: index.rst
# fileName = index.rst

# reference anchors prefix, e.g.: admin (anchor will be .. _admin_cmd)
# prefix = admin

[commands.admin.g0] # subgrouping commands by scope in group

# [mandatory] title of the subsection
title = General information

# [mandatory] pattern to match in the full path of the command names.
pattern = dirac-admin-service-ports, dirac-platform

# this list of patterns will reject scripts that are matched by the patterns above
# exclude = user

# reference anchors prefix
# prefix = general

[commands.admin.g1]
title = Managing Registry
pattern = group, user, host, shifter, voms-sync
prefix = admin_registry

[commands.admin.g2]
title = Managing Resources
pattern = -site, resources, -allow-, -ban-, bdii-info, ce-info,
exclude = setup-site, cs-sites
prefix = admin_resources

[commands.admin.g3]
title = Workload management commands
pattern = job, pilot, task
exclude = update-pilot, sync-pilot
prefix = admin_wms

[commands.admin.g4]
title = Transformation management commands
pattern = transformation
prefix = admin_ts

[commands.admin.g5]
title = Managing DIRAC installation
pattern = install, component, ping-service, dirac-service, setup-site, configure, get-CAs, dirac-info, dirac-version, check
prefix = admin_install

[commands.admin.g6]
title = Managing DIRAC software
pattern = deploy, requirements
prefix = admin_software

[commands.admin.g7]
title = User convenience
pattern = utils, cert, accounting-d
prefix = admin_user

[commands.admin.g8]
title = ProxyManager management commands
pattern = -proxy
exclude = with
prefix = admin_proxy

[commands.admin.g9]
title = Other commands
pattern = dirac, install
prefix = admin_other

[commands.dms]
pattern = dms
title = Data Management
sectionPath = source/UserGuide/CommandReference/%(title)s

[commands.wms]
pattern = wms
title = Workload Management
sectionPath = source/UserGuide/CommandReference/%(title)s

[commands.z_section3]
pattern = dirac-proxy, dirac-info, -resource-
title = Others
sectionPath = source/UserGuide/CommandReference/%(title)s

[commands.comdirac]
prefix = shorthand
pattern = d
exclude = dirac
title = Shorthand
sectionPath = source/UserGuide/CommandReference/%(title)s

For local testing of the documentation, the scripts can also be called directly, like this example from the Makefile shows.

docs/Makefile

htmlall:
	diracdoctools/scripts/dirac-docs-build-commands.py
	diracdoctools/scripts/dirac-docs-build-code.py
	diracdoctools/scripts/dirac-docs-concatenate-diraccfg.py
	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

Code Reference 

The code reference is either created by calling run from diracdoctools.cmd.codeReference or by invoking the script dirac-docs-build-code.py. This creates an rst file for each python file, using autodoc to document all classes inside those modules. The actual documentation is build when sphinx is invoked, which must be able to import the modules and all their dependencies

Command Reference 

The command references can be created by calling run from diracdoctools.cmd.commandReference, or by calling the dirac-docs-build-command.py script. [commands.section] will result in a list of commands with links to their documentation, which is based on the output of their --help. By adding subsections like a [commands.section.g1] you can additionally group commands. The resulting index.rst has to be included explicitly in the documentation.

Note

starting with 8.0, to add a link to the command description use command name in a reference, e.g.: dirac-admin-sysadmin-cli.

Warning

Starting with 8.0, the option indexFile is no longer available. If you created an RST file and decided to add commands yourself, you simply do not need to describe it in the docs.conf.

Note

The parsing of the --help output is extremely limited and naive. You must not end a line with a colon : unless you intend to create a verbatim block after it.

CFG File 

If you developed your own systems, you can concatenate all the settings defined in ConfigTemplate.cfg files into one large file. You need a base dirac.cfg file and a location where to put the final result