Documenting your developments¶
Contents
Where should you document your developments? Well, at several places, indeed, depending on the documentation we are talking about:
Code documentation¶
This is quite easy indeed. It’s excellent practice to add
docstring to your
python code. The good part of it is that tools like pyDev can automatically
read it. Also your python shell can (try help()
), and so does iPython
(just use ?
for example). Python stores every docstring in
the special attribute __doc__
.
Pylint will, by default, complain for every method/class/function left without a docstring.
Release documentation¶
Releases documentation can be found in 2 places: release notes, and github wiki:
release notes are automatically created from the first comment in the pull requests, please describe the changes between BEGRINRELEASENOTES and ENDRELEASENOTES as presented by the template provided
The github wiki can contain a section, for each DIRACGrid repository, highlighting update operations, for example the DIRAC releases notes are linked from the DIRAC wiki main page.
Full development documentation¶
As said at the beginning of this guide, this documentation is in git at
DIRAC/docs.
It is very easy to contribute to it, and you are welcome to do that. You don’t
even have to clone the repository: github lets you edit it online.
This documentation is written in RST
and it is compiled using
sphinx.
Some parts of the documentation can use UML diagrams. They are generated from .uml files
with plantuml. Sphinx support plantuml but ReadTheDocs
didn’t, so you have to convert .uml in .png with java -jar plantuml.jar file.uml
.
Component Options documentation¶
The agent, service and executor options are documented in their respective module docstring via literal include of their options in the ConfigTemplate.cfg:
.. literalinclude:: ../ConfigTemplate.cfg
:start-after: ##BEGIN MyComponent
:end-before: ##END
:dedent: 2
:caption: MyComponent options
Around the section in the ConfigTemplate.cfg configuring the component the ##BEGIN MyComponent and ##END tags need set so that the include is restricted to the section belonging to the component. The options :dedent: and :caption: are optional, but create a nicer output.
Building the Documentation¶
The DIRAC documentation is created using sphinx
and makes use of the
diracdoctools
to create the code documentation, command references, and
concatenate the ConfigTemplates
into one file. The diracdoctools
are
located in the DIRAC/docs
folder, but they can be pip installed
for use
outside of DIRAC.
For local development the easiest way to build the docs is by running these commands from a standard Python 3 DIRAC development environment (see Creating a development environment with Conda.)
pip install docs/
cd docs
make htmlall
python -m http.server -d build/html/
You can now browse the docs locally by going to the URL shown in your terminal.
In the sphinx configuration file (source/conf.py
), the functionality can
then be called to create code reference, command reference and concatenated CFG
files.
if os.environ.get("READTHEDOCS") == "True":
setUpReadTheDocsEnvironment(moduleName="DIRAC", location="../../src")
# re-create the RST files for the command references
LOG.info("Building command reference")
from diracdoctools.cmd.commandReference import run as buildCommandReference
buildCommandReference(configFile="../docs.conf")
# singlehtml build needs too much memory, so we need to create less code documentation
buildType = "limited" if any("singlehtml" in arg for arg in sys.argv) else "full"
LOG.info("Chosing build type: %r", buildType)
from diracdoctools.cmd.codeReference import run as buildCodeDoc
buildCodeDoc(configFile="../docs.conf", buildType=buildType)
# Update dirac.cfg
LOG.info("Concatenating dirac.cfg")
from diracdoctools.cmd.concatcfg import run as updateCompleteDiracCFG
updateCompleteDiracCFG(configFile="../docs.conf")
The configuration for diracdoctools
is done via a configuration file
located in the docs folder. Just copy the file DIRAC/docs/docs.conf
and
adapt it to your needs. All options are mandatory unless otherwise stated. If
certain features are not used, simply leave the values empty
[Docs]
# name of the module to documented, DIRAC, voDIRAC, ExtensionDIRAC
module_name = DIRAC
# path to the source folder relative to docs.conf
source_folder = ../src/DIRAC
[Code]
# building code reference
# where to place the code reference
docs_target_path = ./source/CodeDocumentation
# where Custom docstrings can be found, see
# DIRAC/docs/diracdoctools/CustomizedDocs for an example
customdocs_folder = ./diracdoctools/CustomizedDocs
# add :private-members: to autodoc for matching module
document_private_members = FCConditionsParser
# add :no-inherited: to autodoc for matching module
# Useful to avoid markup errors where the inherited members come from a module
# whose docstrings are not reST formatted
no_inherited_members =
DIRAC.Core.Utilities.Graphs.GraphUtilities,
DIRAC.DataManagementSystem.private.HttpStorageAccessHandler
# only creating dummy files, because they cannot be safely imported due to sideEffects
create_dummy_files = lfc_dfc_copy, lfc_dfc_db_copy, JobWrapperTemplate, PlotCache,
PlottingHandler
# do not include these files in the documentation tree
ignore_folders = diracdoctools, /test, /scripts
ignore_files = setup.py
[CFG]
# concatenating ConfigTemplates
# which file to use as a base
base_file = ../dirac.cfg
# where to place the resulting file
target_file = source/AdministratorGuide/Configuration/ExampleConfig.rst
[Commands]
# list of commands which are not executed to get doc files, rst files have to be
# created by hand
ignore_commands=
# does not have --help, starts compiling externals
dirac-compile-externals,
# does not have --help
dirac-install-client,
# does not have --help
dirac-framework-self-ping
# scripts where the module docstring should be added to the documentation
add_module_docstring = dirac-install
# arbitrary number of these sections can be added, they result in folders with
# rst files for each command matching the pattern. all options are mandatory,
# except indexFile and prefix
[commands.admin]
# pattern to match in the full path of the command names
pattern = admin, accounting, FrameworkSystem, framework, install, utils,
dirac-repo-monitor, dirac-jobexec, dirac-info, ConfigurationSystem, Core, rss,
transformation, stager
# title of the section
title = Admin
# this list of patterns will reject scripts that are matched by the patterns above
exclude =
# the scripts in this list are added to the indexFile, but the rst file has to be provided manually
manual = dirac-cert-convert.sh
# where to place the rst files
sectionPath = source/AdministratorGuide/CommandReference
# optional: only if a hand curated rst file exists
indexFile = source/AdministratorGuide/CommandReference/index.rst
# optional: prefix to add when creating reference anchors, anchor will be prefix_scriptName
prefix = admin
[commands.dms]
pattern = dms
title = Data Management
manual =
exclude =
sectionPath = source/UserGuide/CommandReference/%(title)s
[commands.wms]
pattern = wms
title = Workload Management
manual =
exclude =
sectionPath = source/UserGuide/CommandReference/%(title)s
[commands.z_section3]
pattern = dirac-proxy, dirac-info, myproxy, -resource-
title = Others
exclude =
manual = dirac-cert-convert.sh
sectionPath = source/UserGuide/CommandReference/%(title)s
For local testing of the documentation, the scripts can also be called
directly, like this example from the Makefile
shows.
htmlall:
diracdoctools/scripts/dirac-docs-build-commands.py
diracdoctools/scripts/dirac-docs-build-code.py
diracdoctools/scripts/dirac-docs-concatenate-diraccfg.py
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
Code Reference¶
The code reference is either created by calling run
from
diracdoctools.cmd.codeReference
or by invoking the script
dirac-docs-build-code.py
. This creates an rst
file for each python
file, using autodoc
to document all classes inside those modules. The
actual documentation is build when sphinx is invoked, which must be able to
import the modules and all their dependencies
Command Reference¶
The command references can be created by calling run
from
diracdoctools.cmd.commandReference
, or by calling the
dirac-docs-build-command.py
script. [commands.section]
will result in a
list of commands with links to their documentation, which is based on the output
of their --help
. The resulting index.rst
has to be included explicitly
in the documentation. If indexFile
is specified it has to contain all the
links itself, or warnings are generated for missing and superfluous entries.
Note
The parsing of the --help
output is extremely limited and naive. You must not end a line with a
colon :
unless you intend to create a verbatim block after it.
CFG File¶
If you developed your own systems, you can concatenate all the settings defined
in ConfigTemplate.cfg
files into one large file. You need a base
dirac.cfg
file and a location where to put the final result