Chapter 15. More Advanced Forms

Now let’s look at some more advanced forms usage. We’ve helped our users to avoid blank list items, so now let’s help them avoid duplicate items.

This chapter goes into more intricate details of Django’s form validation, and you have my official permission to skip it if you already know all about customising Django forms, or if you’re reading this book for the TDD rather than for the Django.

If you’re still learning Django, there’s good stuff in here. If you want to skip ahead, that’s OK too. Make sure you take a quick look at the aside on developer stupidity, and the recap on testing views at the end.

Another FT for Duplicate Items

We add a second test method to ItemValidationTest:

functional_tests/test_list_item_validation.py (ch13l001)

def test_cannot_add_duplicate_items(self):
    # Edith goes to the home page and starts a new list
    self.browser.get(self.live_server_url)
    self.get_item_input_box().send_keys('Buy wellies')
    self.get_item_input_box().send_keys(Keys.ENTER)
    self.wait_for_row_in_list_table('1: Buy wellies')

    # She accidentally tries to enter a duplicate item
    self.get_item_input_box().send_keys('Buy wellies')
    self.get_item_input_box().send_keys(Keys.ENTER)

    # She sees a helpful error message
    self.wait_for(lambda: self.assertEqual(
        self.browser.find_element_by_css_selector('.has-error').text,
        "You've already got this in your list"
    ))

Why have two test methods rather than extending one, or having a new file and class? It’s a judgement call. These two feel closely related; they’re both about validation on the same input field, so it feels right to keep them in the same file. On the other hand, they’re logically separate enough that it’s practical to keep them in different methods:

$ python manage.py test functional_tests.test_list_item_validation
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error

Ran 2 tests in 9.613s

OK, so we know the first of the two tests passes now. Is there a way to run just the failing one, I hear you ask? Why, yes indeed:

$ python manage.py test functional_tests.\
test_list_item_validation.ItemValidationTest.test_cannot_add_duplicate_items
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error

Preventing Duplicates at the Model Layer

Here’s what we really wanted to do. It’s a new test that checks that duplicate items in the same list raise an error:

lists/tests/test_models.py (ch09l028)

def test_duplicate_items_are_invalid(self):
    list_ = List.objects.create()
    Item.objects.create(list=list_, text='bla')
    with self.assertRaises(ValidationError):
        item = Item(list=list_, text='bla')
        item.full_clean()

And, while it occurs to us, we add another test to make sure we don’t overdo it on our integrity constraints:

lists/tests/test_models.py (ch09l029)

def test_CAN_save_same_item_to_different_lists(self):
    list1 = List.objects.create()
    list2 = List.objects.create()
    Item.objects.create(list=list1, text='bla')
    item = Item(list=list2, text='bla')
    item.full_clean()  # should not raise

I always like to put a little comment for tests which are checking that a particular use case should not raise an error; otherwise, it can be hard to see what’s being tested:

AssertionError: ValidationError not raised

If we want to get it deliberately wrong, we can do this:

lists/models.py (ch09l030)

class Item(models.Model):
    text = models.TextField(default='', unique=True)
    list = models.ForeignKey(List, default=None)

That lets us check that our second test really does pick up on this problem:

Traceback (most recent call last):
  File "/.../superlists/lists/tests/test_models.py", line 62, in
test_CAN_save_same_item_to_different_lists
    item.full_clean()  # should not raise
    [...]
django.core.exceptions.ValidationError: {'text': ['Item with this Text already
exists.']}

Just like ModelForms, models have a class Meta, and that’s where we can implement a constraint which says that an item must be unique for a particular list, or in other words, that text and list must be unique together:

lists/models.py (ch09l031)

class Item(models.Model):
    text = models.TextField(default='')
    list = models.ForeignKey(List, default=None)

    class Meta:
        unique_together = ('list', 'text')

You might want to take a quick peek at the Django docs on model Meta attributes at this point.

A Little Digression on Queryset Ordering and String Representations

When we run the tests they reveal an unexpected failure:

======================================================================
FAIL: test_saving_and_retrieving_items
(lists.tests.test_models.ListAndItemModelsTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../superlists/lists/tests/test_models.py", line 31, in
test_saving_and_retrieving_items
    self.assertEqual(first_saved_item.text, 'The first (ever) list item')
AssertionError: 'Item the second' != 'The first (ever) list item'
- Item the second
[...]
Note

Depending on your platform and its SQLite installation, you may not see this error. You can follow along anyway; the code and tests are interesting in their own right.

That’s a bit of a puzzler. A bit of print-based debugging:

lists/tests/test_models.py

    first_saved_item = saved_items[0]
    print(first_saved_item.text)
    second_saved_item = saved_items[1]
    print(second_saved_item.text)
    self.assertEqual(first_saved_item.text, 'The first (ever) list item')

will show us…

.....Item the second
The first (ever) list item
F.....

It looks like our uniqueness constraint has messed with the default ordering of queries like Item.objects.all(). Although we already have a failing test, it’s best to add a new test that explicitly tests for ordering:

lists/tests/test_models.py (ch09l032)

    def test_list_ordering(self):
        list1 = List.objects.create()
        item1 = Item.objects.create(list=list1, text='i1')
        item2 = Item.objects.create(list=list1, text='item 2')
        item3 = Item.objects.create(list=list1, text='3')
        self.assertEqual(
            Item.objects.all(),
            [item1, item2, item3]
        )

That gives us a new failure, but it’s not a very readable one:

AssertionError: <QuerySet [<Item: Item object>, <Item: Item object>, <Item:
Item object>]> != [<Item: Item object>, <Item: Item object>, <Item: Item
object>]

We need a better string representation for our objects. Let’s add another unit test:

Note

Ordinarily you would be wary of adding more failing tests when you already have some—it makes reading test output that much more complicated, and just generally makes you nervous. Will we ever get back to a working state? In this case, they’re all quite simple tests, so I’m not worried.

lists/tests/test_models.py (ch13l008)

def test_string_representation(self):
    item = Item(text='some text')
    self.assertEqual(str(item), 'some text')

That gives us:

AssertionError: 'Item object' != 'some text'

As well as the other two failures. Let’s start fixing them all now:

lists/models.py (ch09l034)

class Item(models.Model):
    [...]

    def __str__(self):
        return self.text

Now we’re down to two failures, and the ordering test has a more readable failure message:

AssertionError: <QuerySet [<Item: i1>, <Item: item 2>, <Item: 3>]> != [<Item:
i1>, <Item: item 2>, <Item: 3>]

We can fix that in the class Meta:

lists/models.py (ch09l035)

    class Meta:
        ordering = ('id',)
        unique_together = ('list', 'text')

Does that work?

AssertionError: <QuerySet [<Item: i1>, <Item: item 2>, <Item: 3>]> != [<Item:
i1>, <Item: item 2>, <Item: 3>]

Urp? It has worked; you can see the items are in the same order, but the tests are confused. I keep running into this problem actually—Django querysets don’t compare well with lists. We can fix it by converting the queryset to a list1 in our test:

lists/tests/test_models.py (ch09l036)

    self.assertEqual(
        list(Item.objects.all()),
        [item1, item2, item3]
    )

That works; we get a fully passing test suite:

OK

Rewriting the Old Model Test

That long-winded model test did serendipitously help us find an unexpected bug, but now it’s time to rewrite it. I wrote it in a very verbose style to introduce the Django ORM, but in fact, now that we have the explicit test for ordering, we can get the same coverage from a couple of much shorter tests. Delete test_saving_and_retrieving_items and replace with this:

lists/tests/test_models.py (ch13l010)

class ListAndItemModelsTest(TestCase):

    def test_default_text(self):
        item = Item()
        self.assertEqual(item.text, '')


    def test_item_is_related_to_list(self):
        list_ = List.objects.create()
        item = Item()
        item.list = list_
        item.save()
        self.assertIn(item, list_.item_set.all())

    [...]

That’s more than enough really—a check of the default values of attributes on a freshly initialized model object is enough to sanity-check that we’ve probably set some fields up in models.py. The “item is related to list” test is a real “belt and braces” test to make sure that our foreign key relationship works.

While we’re at it, we can split this file out into tests for Item and tests for List (there’s only one of the latter, test_get_absolute_url):

lists/tests/test_models.py (ch13l011)

class ItemModelTest(TestCase):

    def test_default_text(self):
        [...]



class ListModelTest(TestCase):

    def test_get_absolute_url(self):
        [...]

That’s neater and tidier:

$ python manage.py test lists
[...]
Ran 29 tests in 0.092s

OK

Some Integrity Errors Do Show Up on Save

A final aside before we move on. Do you remember I mentioned in Chapter 13 that some data integrity errors are picked up on save? It all depends on whether the integrity constraint is actually being enforced by the database.

Try running makemigrations and you’ll see that Django wants to add the unique_together constraint to the database itself, rather than just having it as an application-layer constraint:

$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0005_auto_20140414_2038.py
    - Change Meta options on item
    - Alter unique_together for item (1 constraint(s))

Now if we change our duplicates test to do a .save instead of a .full_clean…

lists/tests/test_models.py

    def test_duplicate_items_are_invalid(self):
        list_ = List.objects.create()
        Item.objects.create(list=list_, text='bla')
        with self.assertRaises(ValidationError):
            item = Item(list=list_, text='bla')
            # item.full_clean()
            item.save()

It gives:

ERROR: test_duplicate_items_are_invalid (lists.tests.test_models.ItemModelTest)
[...]
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: lists_item.list_id,
lists_item.text
[...]
django.db.utils.IntegrityError: UNIQUE constraint failed: lists_item.list_id,
lists_item.text

You can see that the error bubbles up from SQLite, and it’s a different error from the one we want, an IntegrityError instead of a ValidationError.

Let’s revert our changes to the test, and see them all passing again:

$ python manage.py test lists
[...]
Ran 29 tests in 0.092s
OK

And now it’s time to commit our model-layer changes:

$ git status # should show changes to tests + models and new migration
# let's give our new migration a better name
$ mv lists/migrations/0005_auto* lists/migrations/0005_list_item_unique_together.py
$ git add lists
$ git diff --staged
$ git commit -am "Implement duplicate item validation at model layer"

Experimenting with Duplicate Item Validation at the Views Layer

Let’s try running our FT, just to see where we are:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error

In case you didn’t see it as it flew past, the site is 500ing.2 A quick unit test at the view level ought to clear this up:

lists/tests/test_views.py (ch13l014)

class ListViewTest(TestCase):
    [...]

    def test_for_invalid_input_shows_error_on_page(self):
        [...]


    def test_duplicate_item_validation_errors_end_up_on_lists_page(self):
        list1 = List.objects.create()
        item1 = Item.objects.create(list=list1, text='textey')
        response = self.client.post(
            f'/lists/{list1.id}/',
            data={'text': 'textey'}
        )

        expected_error = escape("You've already got this in your list")
        self.assertContains(response, expected_error)
        self.assertTemplateUsed(response, 'list.html')
        self.assertEqual(Item.objects.all().count(), 1)

Gives:

django.db.utils.IntegrityError: UNIQUE constraint failed: lists_item.list_id,
lists_item.text

We want to avoid integrity errors! Ideally, we want the call to is_valid to somehow notice the duplication error before we even try to save, but to do that, our form will need to know in advance what list it’s being used for.

Let’s put a skip on that test for now:

lists/tests/test_views.py (ch13l015)

from unittest import skip
[...]

    @skip
    def test_duplicate_item_validation_errors_end_up_on_lists_page(self):

A More Complex Form to Handle Uniqueness Validation

The form to create a new list only needs to know one thing, the new item text. A form which validates that list items are unique needs to know the list too. Just as we overrode the save method on our ItemForm, this time we’ll override the constructor on our new form class so that it knows what list it applies to.

We duplicate our tests for the previous form, tweaking them slightly:

lists/tests/test_forms.py (ch13l016)

from lists.forms import (
    DUPLICATE_ITEM_ERROR, EMPTY_ITEM_ERROR,
    ExistingListItemForm, ItemForm
)
[...]

class ExistingListItemFormTest(TestCase):

    def test_form_renders_item_text_input(self):
        list_ = List.objects.create()
        form = ExistingListItemForm(for_list=list_)
        self.assertIn('placeholder="Enter a to-do item"', form.as_p())


    def test_form_validation_for_blank_items(self):
        list_ = List.objects.create()
        form = ExistingListItemForm(for_list=list_, data={'text': ''})
        self.assertFalse(form.is_valid())
        self.assertEqual(form.errors['text'], [EMPTY_ITEM_ERROR])


    def test_form_validation_for_duplicate_items(self):
        list_ = List.objects.create()
        Item.objects.create(list=list_, text='no twins!')
        form = ExistingListItemForm(for_list=list_, data={'text': 'no twins!'})
        self.assertFalse(form.is_valid())
        self.assertEqual(form.errors['text'], [DUPLICATE_ITEM_ERROR])

Next we iterate through a few TDD cycles until we get a form with a custom constructor, which just ignores its for_list argument. (I won’t show them all, but I’m sure you’ll do them, right? Remember, the Goat sees all.)

lists/forms.py (ch09l071)

DUPLICATE_ITEM_ERROR = "You've already got this in your list"
[...]
class ExistingListItemForm(forms.models.ModelForm):
    def __init__(self, for_list, *args, **kwargs):
        super().__init__(*args, **kwargs)

At this point our error should be:

ValueError: ModelForm has no model class specified.

Then let’s see if making it inherit from our existing form helps:

lists/forms.py (ch09l072)

class ExistingListItemForm(ItemForm):
    def __init__(self, for_list, *args, **kwargs):
        super().__init__(*args, **kwargs)

Yes, that takes us down to just one failure:

FAIL: test_form_validation_for_duplicate_items
(lists.tests.test_forms.ExistingListItemFormTest)
    self.assertFalse(form.is_valid())
AssertionError: True is not false

The next step requires a little knowledge of Django’s internals, but you can read up on it in the Django docs on model validation and form validation.

Django uses a method called validate_unique, both on forms and models, and we can use both, in conjunction with the instance attribute:

lists/forms.py

from django.core.exceptions import ValidationError
[...]

class ExistingListItemForm(ItemForm):

    def __init__(self, for_list, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.instance.list = for_list


    def validate_unique(self):
        try:
            self.instance.validate_unique()
        except ValidationError as e:
            e.error_dict = {'text': [DUPLICATE_ITEM_ERROR]}
            self._update_errors(e)

That’s a bit of Django voodoo right there, but we basically take the validation error, adjust its error message, and then pass it back into the form.

And we’re there! A quick commit:

$ git diff
$ git commit -a

Using the Existing List Item Form in the List View

Now let’s see if we can put this form to work in our view.

We remove the skip, and while we’re at it, we can use our new constant. Tidy.

lists/tests/test_views.py (ch13l049)

from lists.forms import (
    DUPLICATE_ITEM_ERROR, EMPTY_ITEM_ERROR,
    ExistingListItemForm, ItemForm,
)
[...]

    def test_duplicate_item_validation_errors_end_up_on_lists_page(self):
        [...]
        expected_error = escape(DUPLICATE_ITEM_ERROR)

That brings back our integrity error:

django.db.utils.IntegrityError: UNIQUE constraint failed: lists_item.list_id,
lists_item.text

Our fix for this is to switch to using the new form class. Before we implement it, let’s find the tests where we check the form class, and adjust them:

lists/tests/test_views.py (ch13l050)

class ListViewTest(TestCase):
[...]

    def test_displays_item_form(self):
        list_ = List.objects.create()
        response = self.client.get(f'/lists/{list_.id}/')
        self.assertIsInstance(response.context['form'], ExistingListItemForm)
        self.assertContains(response, 'name="text"')

    [...]

    def test_for_invalid_input_passes_form_to_template(self):
        response = self.post_invalid_input()
        self.assertIsInstance(response.context['form'], ExistingListItemForm)

That gives us:

AssertionError: <ItemForm bound=False, valid=False, fields=(text)> is not an
instance of <class 'lists.forms.ExistingListItemForm'>

So we can adjust the view:

lists/views.py (ch13l051)

from lists.forms import ExistingListItemForm, ItemForm
[...]
def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    form = ExistingListItemForm(for_list=list_)
    if request.method == 'POST':
        form = ExistingListItemForm(for_list=list_, data=request.POST)
        if form.is_valid():
            form.save()
            [...]

And that almost fixes everything, except for an unexpected fail:

TypeError: save() missing 1 required positional argument: 'for_list'

Our custom save method from the parent ItemForm is no longer needed. Let’s make a quick unit test for that:

lists/tests/test_forms.py (ch13l053)

def test_form_save(self):
    list_ = List.objects.create()
    form = ExistingListItemForm(for_list=list_, data={'text': 'hi'})
    new_item = form.save()
    self.assertEqual(new_item, Item.objects.all()[0])

We can make our form call the grandparent save method:

lists/forms.py (ch13l054)

    def save(self):
        return forms.models.ModelForm.save(self)
Note

Personal opinion here: I could have used super, but I prefer not to use super when it requires arguments, say, to get a grandparent method. I find Python 3’s super() with no args awesome to get the immediate parent. Anything else is too error-prone, and I find it ugly besides. YMMV.

And we’re there! All the unit tests pass:

$ python manage.py test lists
[...]
Ran 34 tests in 0.082s

OK

And so does our FT for validation:

$ python manage.py test functional_tests.test_list_item_validation
[...]
..
 ---------------------------------------------------------------------
Ran 2 tests in 12.048s

OK

As a final check, we rerun all the FTs:

$ python manage.py test functional_tests
[...]
.....
 ---------------------------------------------------------------------
Ran 5 tests in 19.048s

OK

Hooray! Time for a final commit, and a wrap-up of what we’ve learned about testing views over the last few chapters.

Wrapping Up: What We’ve Learned About Testing Django

We’re now at a point where our app looks a lot more like a “standard” Django app, and it implements the three common Django layers: models, forms, and views. We no longer have any “training wheels”-style tests, and our code looks pretty much like code we’d be happy to see in a real app.

We have one unit test file for each of our key source code files. Here’s a recap of the biggest (and highest-level) one, test_views (the listing shows just the key tests and assertions):

Why these points? Skip ahead to Appendix B, and I’ll show how they are sufficient to ensure that our views are still correct if we refactor them to start using class-based views.

Next we’ll try to make our data validation more friendly by using a bit of client-side code. Uh-oh, you know what that means…

1 You could also check out assertSequenceEqual from unittest, and assertQuerysetEqual from Django’s test tools, although I confess when I last looked at assertQuerysetEqual I was quite baffled…

2 It’s showing a server error, code 500. Gotta get with the jargon!