There is a convenient way to write unit tests in every major programming language. In Python and other languages, there is a very convenient way to test the documentation strings of a method.

def square(x):
    """
    Returns the product of the argument with itself.

    >>> square(4)
    16
    >>> square(-5)
    25
    """
    return x * x

The code snippets in the docstring serve as an example and add tiny test cases to the software. Performing doctests then also ensures that the code snippets are in-sync with the code.

In most cases, a code base includes documentation that is not directly built from comments in the code. A prime example is a README file. The README file in most repositories gives instructions on how to install the library, how to do the first steps, and where to find the full documentation. How can we test code snippets in these files? It can be annoying for first-time users if an example in the README file is outdated.

Another example of documentation not being in-sync with the code because of a lack of continuous testing are software tutorials. I think it is not uncommon to attend a software tutorial and a couple of pages or chapters into the tutorial you cannot reproduce the result. In my experience, this happens most frequently because the documentation is outdated or because of last-minute changes to the tutorial and nobody had the time to perform all the steps from start to end and verify the material.

Both issues can be mitigated with continuous testing using the Python package doxec.

The solution: Doxec

Doxec executes special instructions in markdown and compares the result to the expected result. The instruction set is tailored to common use-cases. For example, if you write

Create the file squares.py and add the following content. […]

or

Run the program with python square.py. You should see the following output. […]

you can add instructions such that doxec mirrors these actions described in plain text during testing. Currently, the package only supports Markdown, but the parser is extensible.

If you add doxec to your CI setup, you will immediately notice if a code change breaks the examples in the README, or if a last-minute change to the tutorial breaks the entire thing. Doxec runs the whole tutorial from front to end within seconds/minutes.

Example

A primitive README can look as follows.

This new program is cool and does the specific task that you need at the
moment.  
<!-- console_output -->
```bash
$ echo "Hello World!"
Hello World!
```

When executed with doxec README, it will test whether the console command echo "Hello World!" generates the expected output. More information about the use of doxec is located in the repository.