Test your documentation
This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post for an introduction.
Consider a docstring
Suppose you’re writing a Python function and, as you’re supposed to do, you give it a docstring, and you even provide some examples of how the function is supposed to work. Like this:
def is_even(number: int) -> bool:
"""
Determine whether the given number is even.
Examples:
>>> is_even(4)
False
>>> is_even(7)
False
"""
return number % 2 == 0
There’s a bug here — the first example shows the wrong return value. Is there a way you could catch that?
It turns out there is — the doctest module in the Python standard library can examine your code, extract examples like the ones above from your docstrings, run them, and verify that they work. And doctest does indeed catch the bug here:
python -m doctest is_even.py
**********************************************************************
File "is_even.py", line 9, in is_even.is_even
Failed example:
is_even(7)
Expected:
False
Got:
True
**********************************************************************
1 items had failures:
1 of 2 in is_even.is_even
***Test Failed*** 1 failures.
While you can do more comprehensive testing with doctest, it’s not a great idea; the doctest module really is geared toward checking examples given in documentation, rather than as a full replacement for a unit-testing framework like unittest or `pytest.
But what about other documentation?
Many Python projects now use Sphinx as their documentation builder, and while Sphinx can insert docstring contents into your generated documentation, it also lets you (and encourages you to) write documentation that isn’t in docstrings. For example, tutorials often need to be longer, coherent prose documents rather than just a collection of docstrings displayed in order (see Diátaxis for a more comprehensive breakdown of types of documentation — docstrings mostly belong to the “reference” quadrant).
Fortunately, Sphinx has doctest-like support via the sphinx.ext.doctest extension. Any code block in your documentation marked for testing will be extracted and run as a test by this extension, and then you can add a python -m sphinx -b doctest (with appropriate customization for your setup) to your CI, ensuring that any code samples in your documentation are automatically checked on each commit and will break your build if they stop working.