Documenting your developments
Where should you document your developments? Well, at several places, indeed, depending on the documentation we are talking about:
Code documentation
This is quite easy indeed. It’s excellent practice to add
docstring to your
python code. The good part of it is that tools like pyDev can automatically
read it. Also your python shell can (try help()
), and so does iPython
(just use ?
for example). Python stores every docstring in
the special attribute __doc__
.
Pylint will, by default, complain for every method/class/function left without a docstring.
Release documentation
Releases documentation can be found in 2 places: release notes, and github wiki:
release notes are automatically created from the first comment in the pull requests, please describe the changes between BEGRINRELEASENOTES and ENDRELEASENOTES as presented by the template provided
The github wiki can contain a section, for each DIRACGrid repository, highlighting update operations, for example the DIRAC releases notes are linked from the DIRAC wiki main page.
Full development documentation
As said at the beginning of this guide, this documentation is in git at
DIRAC/docs.
It is very easy to contribute to it, and you are welcome to do that. You don’t
even have to clone the repository: github lets you edit it online.
This documentation is written in RST
and it is compiled using
sphinx.
Some parts of the documentation can use UML diagrams. They are generated from .uml files
with plantuml. Sphinx support plantuml but ReadTheDocs
didn’t, so you have to convert .uml in .png with java -jar plantuml.jar file.uml
.
Component Options documentation
The agent, service and executor options are documented in their respective module docstring via literal include of their options in the ConfigTemplate.cfg:
.. literalinclude:: ../ConfigTemplate.cfg
:start-after: ##BEGIN MyComponent
:end-before: ##END
:dedent: 2
:caption: MyComponent options
Around the section in the ConfigTemplate.cfg configuring the component the ##BEGIN MyComponent and ##END tags need set so that the include is restricted to the section belonging to the component. The options :dedent: and :caption: are optional, but create a nicer output.
Building the Documentation
The DIRAC documentation is created using sphinx
and makes use of the
diracdoctools
to create the code documentation, command references, and
concatenate the ConfigTemplates
into one file. The diracdoctools
are
located in the DIRAC/docs
folder, but they can be pip installed
for use
outside of DIRAC.
For local development the easiest way to build the docs is by running these commands from a standard Python 3 DIRAC development environment (see Creating a development environment with Conda.)
pip install docs/
cd docs
make htmlall
python -m http.server -d build/html/
You can now browse the docs locally by going to the URL shown in your terminal.
In the sphinx configuration file (source/conf.py
), the functionality can
then be called to create code reference, command reference and concatenated CFG
files.
if os.environ.get("READTHEDOCS") == "True":
setUpReadTheDocsEnvironment(moduleName="DIRAC", location="../../src")
# re-create the RST files for the command references
LOG.info("Building command reference")
from diracdoctools.cmd.commandReference import run as buildCommandReference
if buildCommandReference(configFile="../docs.conf") != 0:
raise RuntimeError("Something went wrong with the documentation creation")
# singlehtml build needs too much memory, so we need to create less code documentation
buildType = "limited" if any("singlehtml" in arg for arg in sys.argv) else "full"
LOG.info("Chosing build type: %r", buildType)
from diracdoctools.cmd.codeReference import run as buildCodeDoc
buildCodeDoc(configFile="../docs.conf", buildType=buildType)
# Update dirac.cfg
LOG.info("Concatenating dirac.cfg")
from diracdoctools.cmd.concatcfg import run as updateCompleteDiracCFG
updateCompleteDiracCFG(configFile="../docs.conf")
The configuration for diracdoctools
is done via a configuration file
located in the docs folder. Just copy the file DIRAC/docs/docs.conf
and
adapt it to your needs. Note that some options are mandatory, this is indicated in the comments to them.
# See more options:
# https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/docs/docs.conf
[Docs] # general documentation settings
# [mandatory] name of the module to documented, DIRAC, voDIRAC, ExtensionDIRAC
module_name = DIRAC
# path to the source folder relative to docs.conf
# By default: ../src/{module_name}
# source_folder = ../src/DIRAC
[Code] # building code reference
# Directory where to place the code reference
# By default: source/CodeDocumentation
# docs_target_path = source/CodeDocumentation
# Directory where Custom docstrings can be found
# See DIRAC/docs/diracdoctools/CustomizedDocs for an example
# By default: diracdoctools/CustomizedDocs
# customdocs_folder = diracdoctools/CustomizedDocs
# add :private-members: to autodoc for matching module
document_private_members = FCConditionsParser
# add :no-inherited: to autodoc for matching module
# Useful to avoid markup errors where the inherited members come from a module
# whose docstrings are not reST formatted
no_inherited_members =
DIRAC.Core.Utilities.Graphs.GraphUtilities,
DIRAC.DataManagementSystem.private.HttpStorageAccessHandler,
DIRAC.FrameworkSystem.private.standardLogging.LogLevels,
# only creating dummy files, because they cannot be safely imported due to sideEffects
create_dummy_files = lfc_dfc_copy, lfc_dfc_db_copy, JobWrapperTemplate
# do not include these files in the documentation tree
ignore_folders = diracdoctools, /test, /scripts
ignore_files = setup.py
# To add section "Commands" with all commands description to newly created "code documentation" (described in the [Code] section).
# By default: false
# add_commands_section = true
[CFG] # concatenating ConfigTemplates
# which file to use as a base
# By default: ../dirac.cfg
# base_file = ../dirac.cfg
# where to place the resulting file
# By default: source/ExampleConfig.rst
target_file = source/AdministratorGuide/Configuration/ExampleConfig.rst
[Commands] # building commands references
# List of commands which are not executed to get doc files
# ignore_commands = command_to_ignore
# Directory where to place the RST file with all commands specification.
# By default: source/Commands
# sectionPath = source/CodeDocumentation/Commands
# Name of the RST file with all commands specification.
# By default: index.rst
# fileName = index.rst
[commands.admin] # grouping commands by scope
# [mandatory] title of the section
title = Admin
# [mandatory] pattern to match in the full path of the command names.
pattern = admin, accounting, FrameworkSystem, framework, install, utils,
dirac-repo-monitor, dirac-jobexec, dirac-info, ConfigurationSystem, Core, rss,
transformation, stager
# this list of patterns will reject scripts that are matched by the patterns above
# exclude = user
# [mandatory] path to the target RST file
sectionPath = source/AdministratorGuide/CommandReference
# Name of the target RST file
# By default: index.rst
# fileName = index.rst
# reference anchors prefix, e.g.: admin (anchor will be .. _admin_cmd)
# prefix = admin
[commands.admin.g0] # subgrouping commands by scope in group
# [mandatory] title of the subsection
title = General information
# [mandatory] pattern to match in the full path of the command names.
pattern = dirac-admin-service-ports, dirac-platform
# this list of patterns will reject scripts that are matched by the patterns above
# exclude = user
# reference anchors prefix
# prefix = general
[commands.admin.g1]
title = Managing Registry
pattern = group, user, host, shifter, voms-sync
prefix = admin_registry
[commands.admin.g2]
title = Managing Resources
pattern = -site, resources, -allow-, -ban-, bdii-info, ce-info,
exclude = setup-site, cs-sites
prefix = admin_resources
[commands.admin.g3]
title = Workload management commands
pattern = job, pilot, task
exclude = update-pilot, sync-pilot
prefix = admin_wms
[commands.admin.g4]
title = Transformation management commands
pattern = transformation
prefix = admin_ts
[commands.admin.g5]
title = Managing DIRAC installation
pattern = install, component, ping-service, dirac-service, setup-site, configure, get-CAs, dirac-info, dirac-version, check
prefix = admin_install
[commands.admin.g6]
title = Managing DIRAC software
pattern = deploy, requirements
prefix = admin_software
[commands.admin.g7]
title = User convenience
pattern = utils, myproxy, cert, accounting-d
prefix = admin_user
[commands.admin.g8]
title = ProxyManager management commands
pattern = -proxy
exclude = with
prefix = admin_proxy
[commands.admin.g9]
title = Other commands
pattern = dirac, install
prefix = admin_other
[commands.dms]
pattern = dms
title = Data Management
sectionPath = source/UserGuide/CommandReference/%(title)s
[commands.wms]
pattern = wms
title = Workload Management
sectionPath = source/UserGuide/CommandReference/%(title)s
[commands.z_section3]
pattern = dirac-proxy, dirac-info, myproxy, -resource-
title = Others
sectionPath = source/UserGuide/CommandReference/%(title)s
[commands.comdirac]
prefix = shorthand
pattern = d
exclude = dirac
title = Shorthand
sectionPath = source/UserGuide/CommandReference/%(title)s
For local testing of the documentation, the scripts can also be called
directly, like this example from the Makefile
shows.
htmlall:
diracdoctools/scripts/dirac-docs-build-commands.py
diracdoctools/scripts/dirac-docs-build-code.py
diracdoctools/scripts/dirac-docs-concatenate-diraccfg.py
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
Code Reference
The code reference is either created by calling run
from
diracdoctools.cmd.codeReference
or by invoking the script
dirac-docs-build-code.py
. This creates an rst
file for each python
file, using autodoc
to document all classes inside those modules. The
actual documentation is build when sphinx is invoked, which must be able to
import the modules and all their dependencies
Command Reference
The command references can be created by calling run
from
diracdoctools.cmd.commandReference
, or by calling the
dirac-docs-build-command.py
script. [commands.section]
will result in a
list of commands with links to their documentation, which is based on the output
of their --help
. By adding subsections like a [commands.section.g1]
you can additionally group commands.
The resulting index.rst
has to be included explicitly in the documentation.
Note
starting with 8.0, to add a link to the command description use command name in a reference, e.g.: dirac-admin-sysadmin-cli.
Warning
Starting with 8.0, the option indexFile
is no longer available. If you created an RST file and
decided to add commands yourself, you simply do not need to describe it in the docs.conf.
Note
The parsing of the --help
output is extremely limited and naive. You must not end a line with a
colon :
unless you intend to create a verbatim block after it.
CFG File
If you developed your own systems, you can concatenate all the settings defined
in ConfigTemplate.cfg
files into one large file. You need a base
dirac.cfg
file and a location where to put the final result