10.14 Unit Testing with Docstrings and doctest

A key aspect of software development is testing your code to ensure that it works correctly. Even with extensive testing, however, your code may still contain bugs. According to the famous Dutch computer scientist Edsger Dijkstra, “Testing shows the presence, not the absence of bugs.”¹⁷

17. J. N. Buxton and B. Randell, eds, Software Engineering Techniques, April 1970, p. 16. Report on a conference sponsored by the NATO Science Committee, Rome, Italy, 27–31 October 1969

Module doctest and the testmod Function

The Python Standard Library provides the doctest module to help you test your code and conveniently retest it after you make modifications. When you execute the doctest module’s testmod function, it inspects your functions’, methods’ and classes' docstrings looking for sample Python statements preceded by >>>, each followed on the next line by the given statement’s expected output (if any).¹⁸ The testmod function then executes those statements and confirms that they produce the expected output. If they do not, testmod reports errors indicating which tests failed so you can locate and fix the problems in your code. Each test you define in a docstring typically tests a specific unit of code, such as a function, a method or a class. Such tests are called unit tests.

18. The notation >>> mimics the standard python interpreter’s input prompts.

Modified Account Class

The file accountdoctest.py contains the class Account from this chapter’s first example. We modified the __init__ method’s docstring to include four tests which can be used to ensure that the method works correctly:

• The test in line 11 creates a sample Account object named account1. This statement does not produce any output.

• The test in line 12 shows what the value of account1’s name attribute should be if line 11 executed successfully. The sample output is shown in line 13.

• The test in line 14 shows what the value of account1’s balance attribute should be if line 11 executed successfully. The sample output is shown in line 15.

• The test in line 18 creates an Account object with an invalid initial balance. The sample output shows that a ValueError exception should occur in this case. For exceptions, the doctest module’s documentation recommends showing just the first and last lines of the traceback.¹⁹

  19. https://docs.python.org/3/library/doctest.html?highlight=doctest#module-doctest.

You can intersperse your tests with descriptive text, such as line 17.

 1 # accountdoctest.py
 2 """Account class definition."""
 3 from decimal import Decimal
 4
 5 class Account:
 6     """Account class for demonstrating doctest."""
 7
 8     def __init__(self, name, balance):
 9         """Initialize an Account object.
10
11         >>> account1 = Account('John Green', Decimal('50.00'))
12         >>> account1.name
13         'John Green'
14         >>> account1.balance
15         Decimal('50.00')
16
17         The balance argument must be greater than or equal to 0.
18         >>> account2 = Account('John Green', Decimal('-50.00'))
19         Traceback (most recent call last):
20         ...
21         ValueError: Initial balance must be >= to 0.00.
22         """
23
24         # if balance is less than 0.00, raise an exception
25         if balance < Decimal('0.00'):
26         raise ValueError('Initial balance must be >= to 0.00.')
27
28         self.name = name
29         self.balance = balance
30
31     def deposit(self, amount):
32         """Deposit money to the account."""
33
34         # if amount is less than 0.00, raise an exception
35         if amount < Decimal('0.00'):
36             raise ValueError('amount must be positive.')
37
38         self.balance += amount
39
40 if __name__ == '__main__':
41     import doctest
42     doctest.testmod(verbose=True)

Module __main__

When you load any module, Python assigns a string containing the module’s name to a global attribute of the module called __name__. When you execute a Python source file (such as accountdoctest.py) as a script, Python uses the string '__main__' as the module’s name. You can use __name__ in an if statement like lines 40–42 to specify code that should execute only if the source file is executed as a script. In this example, line 41 imports the doctest module and line 42 calls the module’s testmod function to execute the docstring unit tests.

Running Tests

Run the file accountdoctest.py as a script to execute the tests. By default, if you call testmod with no arguments, it does not show test results for successful tests. In that case, if you get no output, all the tests executed successfully. In this example, line 42 calls testmod with the keyword argument verbose=True. This tells testmod to produce verbose output showing every test’s results:

Trying:
    account1 = Account('John Green', Decimal('50.00'))
Expecting nothing
ok
Trying:
    account1.name
Expecting:
    'John Green'
ok
Trying:
    account1.balance
Expecting:
    Decimal('50.00')
ok
Trying:
    account2 = Account('John Green', Decimal('-50.00'))
Expecting:
    Traceback (most recent call last):
        ...
    ValueError: Initial balance must be >= to 0.00.
ok
3 items had no tests:
    __main__
    __main__.Account
    __main__.Account.deposit
1 items passed all tests:
    4 tests in __main__.Account.__init__
4 tests in 4 items.
4 passed and 0 failed.
Test passed.

In verbose mode, testmod shows for each test what it’s "Trying" to do and what it’s "Expecting" as a result, followed by "ok" if the test is successful. After completing the tests in verbose mode, testmod shows a summary of the results.

To demonstrate a failed test, “comment out” lines 25–26 in accountdoctest.py by preceding each with a #, then run accountdoctest.py as a script. To save space, we show just the portions of the doctest output indicating the failed test:

...
**********************************************************************
File "accountdoctest.py", line 18, in __main__.Account.__init__
Failed example:
    account2 = Account('John Green', Decimal('-50.00'))
Expected:
    Traceback (most recent call last):
        ...
    ValueError: Initial balance must be >= to 0.00.
Got nothing
**********************************************************************
1 items had failures:
   1 of 4 in __main__.Account.__init__
4 tests in 4 items.
3 passed and 1 failed.
***Test Failed*** 1 failures.

In this case, we see that line 18’s test failed. The testmod function was expecting a traceback indicating that a ValueError was raised due to the invalid initial balance. That exception did not occur, so the test failed. As the programmer responsible for defining this class, this failing test would be an indication that something is wrong with the validation code in your __init__ method.

IPython %doctest_mode Magic

A convenient way to create doctests for existing code is to use an IPython interactive session to test your code, then copy and paste that session into a docstring. IPython’s In [] and Out[] prompts are not compatible with doctest, so IPython provides the magic %doctest_mode to display prompts in the correct doctest format. The magic toggles between the two prompt styles. The first time you execute %doctest_mode, IPython switches to >>> prompts for input and no output prompts. The second time you execute %doctest_mode, IPython switches back to In [] and Out[] prompts.

Self Check

1. (Fill-In) When you execute a Python source file as a script, Python creates a global attribute __name__ and assigns it the string _________.
   Answer: '__main__'.

2. (True/False) When you execute the doctest module’s testmod function, it inspects your code and automatically creates tests for you.
   Answer: False. When you execute the doctest module’s testmod function, it inspects your code’s function, method and class docstrings looking for sample Python statements preceded by >>>, each followed on the next line by the given statement’s expected output (if any).

3. (IPython Session) Add tests to the deposit method’s docstring, then execute the tests. Your test should create an Account object, deposit a valid amount into it, then attempt to deposit an invalid negative amount, which raises a ValueError.
   Answer: The updated docstring for method deposit is shown below, followed by the verbose doctest results:

   """Deposit money to the account.
   
   >>> account1 = Account('John Green', Decimal('50.00'))
   >>> account1.deposit(Decimal('10.55'))
   >>> account1.balance
   Decimal('60.55')
   
   >>> account1.deposit(Decimal('-100.00'))
   Traceback (most recent call last):
       ...
   ValueError: amount must be positive.
   """

   Trying:
       account1 = Account('John Green', Decimal('50.00'))
   Expecting nothing
   ok
   Trying:
       account1.name
   Expecting:
       'John Green'
   ok
   Trying:
       account1.balance
   Expecting:
       Decimal('50.00')
   ok
   Trying:
       account2 = Account('John Green', Decimal('-50.00'))
   Expecting:
       Traceback (most recent call last):
           ...
       ValueError: Initial balance must be >= to 0.00.
   ok
   Trying:
       account1 = Account('John Green', Decimal('50.00'))
   Expecting nothing
   ok
   Trying:
       account1.deposit(Decimal('10.55'))
   Expecting nothing
   ok
   Trying:
       account1.balance
   Expecting:
   Decimal('60.55')
   ok
   Trying:
       account1.deposit(Decimal('-100.00'))
   Expecting:
       Traceback (most recent call last):
           ...
       ValueError: amount must be positive.
   ok
   2 items had no tests:
       __main__
       __main__.Account
   2 items passed all tests:
       4 tests in __main__.Account.__init__
       4 tests in __main__.Account.deposit
   8 tests in 4 items.
   8 passed and 0 failed.
   Test passed.