Chapter 22. Finishing “My Lists”: Outside-In TDD

In this chapter I’d like to talk about a technique called Outside-In TDD. It’s pretty much what we’ve been doing all along. Our “double-loop” TDD process, in which we write the functional test first and then the unit tests, is already a manifestation of outside-in—we design the system from the outside, and build up our code in layers. Now I’ll make it explicit, and talk about some of the common issues involved.

The Alternative: “Inside-Out”

The alternative to “outside-in” is to work “inside-out”, which is the way most people intuitively work before they encounter TDD. After coming up with a design, the natural inclination is sometimes to implement it starting with the innermost, lowest-level components first.

For example, when faced with our current problem, providing users with a “My Lists” page of saved lists, the temptation is to start by adding an “owner” attribute to the List model object, reasoning that an attribute like this is “obviously” going to be required. Once that’s in place, we would modify the more peripheral layers of code, such as views and templates, taking advantage of the new attribute, and then finally add URL routing to point to the new view.

It feels comfortable because it means you’re never working on a bit of code that is dependent on something that hasn’t yet been implemented. Each bit of work on the inside is a solid foundation on which to build the next layer out.

But working inside-out like this also has some weaknesses.

The FT for “My Lists”

As we work through the following functional test, we start with the most outward-facing (presentation layer), through to the view functions (or “controllers”), and lastly the innermost layers, which in this case will be model code.

We know our create_pre_authenticated_session code works now, so we can just write our FT to look for a “My Lists” page:

functional_tests/test_my_lists.py (ch19l001-1)

    def test_logged_in_users_lists_are_saved_as_my_lists(self):
        # Edith is a logged-in user
        self.create_pre_authenticated_session('edith@example.com')

        # She goes to the home page and starts a list
        self.browser.get(self.live_server_url)
        self.add_list_item('Reticulate splines')
        self.add_list_item('Immanentize eschaton')
        first_list_url = self.browser.current_url

        # She notices a "My lists" link, for the first time.
        self.browser.find_element_by_link_text('My lists').click()

        # She sees that her list is in there, named according to its
        # first list item
        self.wait_for(
            lambda: self.browser.find_element_by_link_text('Reticulate splines')
        )
        self.browser.find_element_by_link_text('Reticulate splines').click()
        self.wait_for(
            lambda: self.assertEqual(self.browser.current_url, first_list_url)
        )

We create a list with a couple of items, and then we check that this list appears on a new “My Lists” page, and that it’s “named” after the first item in the list.

Let’s validate that it really works by creating a second list, and seeing that appear on the My Lists page as well. The FT continues, and while we’re at it, we check that only logged-in users can see the “My Lists” page:

functional_tests/test_my_lists.py (ch19l001-2)

        [...]
        self.wait_for(
            lambda: self.assertEqual(self.browser.current_url, first_list_url)
        )

        # She decides to start another list, just to see
        self.browser.get(self.live_server_url)
        self.add_list_item('Click cows')
        second_list_url = self.browser.current_url

        # Under "my lists", her new list appears
        self.browser.find_element_by_link_text('My lists').click()
        self.wait_for(
            lambda: self.browser.find_element_by_link_text('Click cows')
        )
        self.browser.find_element_by_link_text('Click cows').click()
        self.wait_for(
            lambda: self.assertEqual(self.browser.current_url, second_list_url)
        )

        # She logs out.  The "My lists" option disappears
        self.browser.find_element_by_link_text('Log out').click()
        self.wait_for(lambda: self.assertEqual(
            self.browser.find_elements_by_link_text('My lists'),
            []
        ))

Our FT uses a new helper method, add_list_item, which abstracts away entering text into the right input box. We define it in base.py:

functional_tests/base.py (ch19l001-3)

from selenium.webdriver.common.keys import Keys
[...]

    def add_list_item(self, item_text):
        num_rows = len(self.browser.find_elements_by_css_selector('#id_list_table tr'))
        self.get_item_input_box().send_keys(item_text)
        self.get_item_input_box().send_keys(Keys.ENTER)
        item_number = num_rows + 1
        self.wait_for_row_in_list_table(f'{item_number}: {item_text}')

And while we’re at it we can use it in a few of the other FTs, like this:

functional_tests/test_list_item_validation.py

    self.add_list_item('Buy wellies')

I think it makes the FTs a lot more readable. I made a total of six changes—see if you agree with me.

A quick run of all FTs, a commit, and then back to the FT we’re working on. The first error should look like this:

$ python3 manage.py test functional_tests.test_my_lists
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: My lists

Moving Down One Layer to View Functions (the Controller)

That will cause a template error, so we’ll start to move down from the presentation layer and URLs down to the controller layer, Django’s view functions.

As always, we start with a test:

lists/tests/test_views.py (ch19l003)

class MyListsTest(TestCase):

    def test_my_lists_url_renders_my_lists_template(self):
        response = self.client.get('/lists/users/a@b.com/')
        self.assertTemplateUsed(response, 'my_lists.html')

That gives:

AssertionError: No templates used to render the response

And we fix it, still at the presentation level, in urls.py:

lists/urls.py

urlpatterns = [
    url(r'^new$', views.new_list, name='new_list'),
    url(r'^(\d+)/$', views.view_list, name='view_list'),
    url(r'^users/(.+)/$', views.my_lists, name='my_lists'),
]

That gives us a test failure, which informs us of what we should do as we move down to the next level:

AttributeError: module 'lists.views' has no attribute 'my_lists'

We move in from the presentation layer to the views layer, and create a minimal placeholder:

lists/views.py (ch19l005)

def my_lists(request, email):
    return render(request, 'my_lists.html')

And a minimal template:

lists/templates/my_lists.html

{% extends 'base.html' %}

{% block header_text %}My Lists{% endblock %}

That gets our unit tests passing, but our FT is still at the same point, saying that the “My Lists” page doesn’t yet show any lists. It wants them to be clickable links named after the first item:

$ python3 manage.py test functional_tests.test_my_lists
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: Reticulate splines

Another Pass, Outside-In

At each stage, we still let the FT drive what development we do.

Starting again at the outside layer, in the template, we begin to write the template code we’d like to use to get the “My Lists” page to work the way we want it to. As we do so, we start to specify the API we want from the code at the layers below.

Designing Our API Using the Template

Meanwhile, in my_lists.html we override the list_form and say it should be empty…

lists/templates/my_lists.html

{% extends 'base.html' %}

{% block header_text %}My Lists{% endblock %}

{% block list_form %}{% endblock %}

And then we can just work inside the extra_content block:

lists/templates/my_lists.html

[...]

{% block list_form %}{% endblock %}

{% block extra_content %}
    <h2>{{ owner.email }}'s lists</h2>  1
    <ul>
        {% for list in owner.list_set.all %}  2
            <li><a href="{{ list.get_absolute_url }}">{{ list.name }}</a></li>  3
        {% endfor %}
    </ul>
{% endblock %}

We’ve made several design decisions in this template which are going to filter their way down through the code:

1

We want a variable called owner to represent the user in our template.

2

We want to be able to iterate through the lists created by the user using owner.list_set.all (I happen to know we get this for free from the Django ORM).

3

We want to use list.name to print out the “name” of the list, which is currently specified as the text of its first element.

Note

Outside-In TDD is sometimes called “programming by wishful thinking”, and you can see why. We start writing code at the higher levels based on what we wish we had at the lower levels, even though it doesn’t exist yet!

We can rerun our FTs, to check that we didn’t break anything, and to see whether we’ve got any further:

$ python manage.py test functional_tests
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: Reticulate splines

 ---------------------------------------------------------------------
Ran 8 tests in 77.613s

FAILED (errors=1)

Well, no further, but at least we didn’t break anything. Time for a commit:

$ git add lists
$ git diff --staged
$ git commit -m "url, placeholder view, and first-cut templates for my_lists"

The Next “Requirement” from the Views Layer: New Lists Should Record Owner

Before we move down to the model layer, there’s another part of the code at the views layer that will need to use our model: we need some way for newly created lists to be assigned to an owner, if the current user is logged in to the site.

Here’s a first crack at writing the test:

lists/tests/test_views.py (ch19l014)

class NewListTest(TestCase):
    [...]

    def test_list_owner_is_saved_if_user_is_authenticated(self):
        user = User.objects.create(email='a@b.com')
        self.client.force_login(user)  1
        self.client.post('/lists/new', data={'text': 'new item'})
        list_ = List.objects.first()
        self.assertEqual(list_.owner, user)
1

force_login() is the way you get the test client to make requests with a logged-in user.

The test fails as follows:

AttributeError: 'List' object has no attribute 'owner'

To fix this, we can try writing code like this:

lists/views.py (ch19l015)

def new_list(request):
    form = ItemForm(data=request.POST)
    if form.is_valid():
        list_ = List()
        list_.owner = request.user
        list_.save()
        form.save(for_list=list_)
        return redirect(list_)
    else:
        return render(request, 'home.html', {"form": form})

But it won’t actually work, because we don’t know how to save a list owner yet:

    self.assertEqual(list_.owner, user)
AttributeError: 'List' object has no attribute 'owner'

Moving Down to the Model Layer

Our outside-in design has driven out two requirements for the model layer: we want to be able to assign an owner to a list using the attribute .owner, and we want to be able to access the list’s owner with the API owner.list_set.all.

Let’s write a test for that:

lists/tests/test_models.py (ch19l018)

from django.contrib.auth import get_user_model
User = get_user_model()
[...]

class ListModelTest(TestCase):

    def test_get_absolute_url(self):
        [...]

    def test_lists_can_have_owners(self):
        user = User.objects.create(email='a@b.com')
        list_ = List.objects.create(owner=user)
        self.assertIn(list_, user.list_set.all())

And that gives us a new unit test failure:

    list_ = List.objects.create(owner=user)
    [...]
TypeError: 'owner' is an invalid keyword argument for this function

The naive implementation would be this:

from django.conf import settings
[...]

class List(models.Model):
    owner = models.ForeignKey(settings.AUTH_USER_MODEL)

But we want to make sure the list owner is optional. Explicit is better than implicit, and tests are documentation, so let’s have a test for that too:

lists/tests/test_models.py (ch19l020)

    def test_list_owner_is_optional(self):
        List.objects.create()  # should not raise

The correct implementation is this:

lists/models.py

from django.conf import settings
[...]

class List(models.Model):
    owner = models.ForeignKey(settings.AUTH_USER_MODEL, blank=True, null=True)

    def get_absolute_url(self):
        return reverse('view_list', args=[self.id])

Now running the tests gives the usual database error:

    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such column: lists_list.owner_id

Because we need to make some migrations:

$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0006_list_owner.py
    - Add field owner to list

We’re almost there; a couple more failures:

ERROR: test_redirects_after_POST (lists.tests.test_views.NewListTest)
[...]
ValueError: Cannot assign "<SimpleLazyObject:
<django.contrib.auth.models.AnonymousUser object at 0x7f364795ef90>>":
"List.owner" must be a "User" instance.
ERROR: test_can_save_a_POST_request (lists.tests.test_views.NewListTest)

[...]
ValueError: Cannot assign "<SimpleLazyObject:
<django.contrib.auth.models.AnonymousUser object at 0x7f364795ef90>>":
"List.owner" must be a "User" instance.

We’re moving back up to the views layer now, just doing a little tidying up. Notice that these are in the old test for the new_list view, when we haven’t got a logged-in user. We should only save the list owner when the user is actually logged in. The .is_authenticated attribute we defined in Chapter 19 comes in useful now (when they’re not logged in, Django represents users using a class called AnonymousUser, whose .is_authenticated is always False):

lists/views.py (ch19l023)

    if form.is_valid():
        list_ = List()
        if request.user.is_authenticated:
            list_.owner = request.user
        list_.save()
        form.save(for_list=list_)
        [...]

And that gets us passing!

$ python manage.py test lists
[...]
.......................................
 ---------------------------------------------------------------------
Ran 39 tests in 0.237s

OK

This is a good time for a commit:

$ git add lists
$ git commit -m "lists can have owners, which are saved on creation."

Final Step: Feeding Through the .name API from the Template

The last thing our outside-in design wanted came from the templates, which wanted to be able to access a list “name” based on the text of its first item:

lists/tests/test_models.py (ch19l024)

    def test_list_name_is_first_item_text(self):
        list_ = List.objects.create()
        Item.objects.create(list=list_, text='first item')
        Item.objects.create(list=list_, text='second item')
        self.assertEqual(list_.name, 'first item')

lists/models.py (ch19l025)

    @property
    def name(self):
        return self.item_set.first().text

And that, believe it or not, actually gets us a passing test, and a working “My Lists” page (Figure 22-1)!

$ python manage.py test functional_tests
[...]
Ran 8 tests in 93.819s

OK

But we know we cheated to get there. The Testing Goat is eyeing us suspiciously. We left a test failing at one layer while we implemented its dependencies at the lower layer. Let’s see how things would play out if we were to use better test isolation…

Screenshot of new My Lists page
Figure 22-1. The “My Lists” page, in all its glory (and proof I did test on Windows)