Chapter 13. Validation at the Database Layer

Over the next few chapters we’ll talk about testing and implementing validation of user inputs.

In terms of content, there’s going to be quite a lot of material here that’s more about the specifics of Django, and less discussion of TDD philosophy. That doesn’t mean you won’t be learning anything about testing—there are plenty of little testing tidbits in here, but perhaps it’s more about really getting into the swing of things, the rhythm of TDD, and how we get work done.

Once we get through these three short chapters, I’ve saved a bit of fun with JavaScript (!) for the end of Part II. Then it’s on to Part III, where I promise we’ll get right back into some of the real nitty-gritty discussions in TDD methodology—unit tests versus integrated tests, mocking, and more. Stay tuned!

But for now, a little validation. Let’s just remind ourselves where our FT is pointing us:

$ python3 manage.py test functional_tests.test_list_item_validation
[...]
======================================================================
ERROR: test_cannot_add_empty_list_items
(functional_tests.test_list_item_validation.ItemValidationTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../superlists/functional_tests/test_list_item_validation.py", line
15, in test_cannot_add_empty_list_items
    self.wait_for(lambda: self.assertEqual(
[...]
  File "/.../superlists/functional_tests/test_list_item_validation.py", line
16, in <lambda>
    self.browser.find_element_by_css_selector('.has-error').text,
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error

It’s expecting to see an error message if the user tries to input an empty item.

Model-Layer Validation

In a web app, there are two places you can do validation: on the client side (using JavaScript or HTML5 properties, as we’ll see later), and on the server side. The server side is “safer” because someone can always bypass the client side, whether it’s maliciously or due to some bug.

Similarly on the server side, in Django, there are two levels at which you can do validation. One is at the model level, and the other is higher up at the forms level. I like to use the lower level whenever possible, partially because I’m a bit too fond of databases and database integrity rules, and partially because, again, it’s safer—you can sometimes forget which form you use to validate input, but you’re always going to use the same database.

The self.assertRaises Context Manager

Let’s go down and write a unit test at the models layer. Add a new test method to ListAndItemModelsTest, which tries to create a blank list item. This test is interesting because it’s testing that the code under test should raise an exception:

lists/tests/test_models.py (ch11l018)

from django.core.exceptions import ValidationError
[...]

class ListAndItemModelsTest(TestCase):
    [...]

    def test_cannot_save_empty_list_items(self):
        list_ = List.objects.create()
        item = Item(list=list_, text='')
        with self.assertRaises(ValidationError):
            item.save()

Tip

If you’re new to Python, you may never have seen the with statement. It’s used with what are called “context managers”, which wrap a block of code, usually with some kind of setup, cleanup, or error-handling code. There’s a good write-up in the Python 2.5 release notes.

This is a new unit testing technique: when we want to check that doing something will raise an error, we can use the self.assertRaises context manager. We could have used something like this instead:

try:
    item.save()
    self.fail('The save should have raised an exception')
except ValidationError:
    pass

But the with formulation is neater. Now, we can try running the test, and see its expected failure:

    item.save()
AssertionError: ValidationError not raised

A Django Quirk: Model Save Doesn’t Run Validation

And now we discover one of Django’s little quirks. This test should already pass. If you take a look at the docs for the Django model fields, you’ll see that TextField actually defaults to blank=False, which means that it should disallow empty values.

So why is the test still failing? Well, for slightly counterintuitive historical reasons, Django models don’t run full validation on save. As we’ll see later, any constraints that are actually implemented in the database will raise errors on save, but SQLite doesn’t support enforcing emptiness constraints on text columns, and so our save method is letting this invalid value through silently.

There’s a way of checking whether the constraint will happen at the database level or not: if it was at the database level, we would need a migration to apply the constraint. But Django knows that SQLite doesn’t support this type of constraint, so if we try to run makemigrations, it will report there’s nothing to do:

$ python manage.py makemigrations
No changes detected

Django does have a method to manually run full validation, however, called full_clean (more info in the docs). Let’s hack it in to see it work:

lists/tests/test_models.py

    with self.assertRaises(ValidationError):
        item.save()
        item.full_clean()

That gets the test to pass:

OK

Good. That taught us a little about Django validation, and the test is there to warn us if we ever forget our requirement and set blank=True on the text field (try it!).

Surfacing Model Validation Errors in the View

Let’s try to enforce our model validation in the views layer and bring it up through into our templates, so the user can see them. Here’s how we can optionally display an error in our HTML—we check whether the template has been passed an error variable, and if so, we display it next to the form:

lists/templates/base.html (ch11l020)

  <form method="POST" action="{% block form_action %}{% endblock %}">
    <input name="item_text" id="id_new_item"
           class="form-control input-lg"
           placeholder="Enter a to-do item" />
    {% csrf_token %}
    {% if error %}
      <div class="form-group has-error">
        <span class="help-block">{{ error }}</span>
      </div>
    {% endif %}
  </form>

Take a look at the Bootstrap docs for more info on form controls.

Passing this error to the template is the job of the view function. Let’s take a look at the unit tests in the NewListTest class. I’m going to use two slightly different error-handling patterns here.

In the first case, our URL and view for new lists will optionally render the same template as the home page, but with the addition of an error message. Here’s a unit test for that:

lists/tests/test_views.py (ch11l021)

class NewListTest(TestCase):
    [...]

    def test_validation_errors_are_sent_back_to_home_page_template(self):
        response = self.client.post('/lists/new', data={'item_text': ''})
        self.assertEqual(response.status_code, 200)
        self.assertTemplateUsed(response, 'home.html')
        expected_error = "You can't have an empty list item"
        self.assertContains(response, expected_error)

As we’re writing this test, we might get slightly offended by the /lists/new URL, which we’re manually entering as a string. We’ve got a lot of URLs hardcoded in our tests, in our views, and in our templates, which violates the DRY principle. I don’t mind a bit of duplication in tests, but we should definitely be on the lookout for hardcoded URLs in our views and templates, and make a note to refactor them out. But we won’t do them straight away, because right now our application is in a broken state. We want to get back to a working state first.

Back to our test, which is failing because the view is currently returning a 302 redirect, rather than a “normal” 200 response:

AssertionError: 302 != 200

Let’s try calling full_clean() in the view:

lists/views.py

def new_list(request):
    list_ = List.objects.create()
    item = Item.objects.create(text=request.POST['item_text'], list=list_)
    item.full_clean()
    return redirect(f'/lists/{list_.id}/')

As we’re looking at the view code, we find a good candidate for a hardcoded URL to get rid of. Let’s add that to our scratchpad:

SCRATCHPAD:

• Remove hardcoded URLs from views.py

Now the model validation raises an exception, which comes up through our view:

[...]
  File "/.../superlists/lists/views.py", line 11, in new_list
    item.full_clean()
[...]
django.core.exceptions.ValidationError: {'text': ['This field cannot be
blank.']}

So we try our first approach: using a try/except to detect errors. Obeying the Testing Goat, we start with just the try/except and nothing else. The tests should tell us what to code next…

lists/views.py (ch11l025)

from django.core.exceptions import ValidationError
[...]

def new_list(request):
    list_ = List.objects.create()
    item = Item.objects.create(text=request.POST['item_text'], list=list_)
    try:
        item.full_clean()
    except ValidationError:
        pass
    return redirect(f'/lists/{list_.id}/')

That gets us back to the 302 != 200:

AssertionError: 302 != 200

Let’s return a rendered template then, which should take care of the template check as well:

lists/views.py (ch11l026)

    except ValidationError:
        return render(request, 'home.html')

And the tests now tell us to put the error message into the template:

AssertionError: False is not true : Couldn't find 'You can't have an empty list
item' in response

We do that by passing a new template variable in:

lists/views.py (ch11l027)

    except ValidationError:
        error = "You can't have an empty list item"
        return render(request, 'home.html', {"error": error})

Hmm, it looks like that didn’t quite work:

AssertionError: False is not true : Couldn't find 'You can't have an empty list
item' in response

A little print-based debug…

lists/tests/test_views.py

expected_error = "You can't have an empty list item"
print(response.content.decode())
self.assertContains(response, expected_error)

…will show us the cause—Django has HTML-escaped the apostrophe:

[...]
<span class="help-block">You can&#39;t have an empty list
item</span>

We could hack something like this into our test:

    expected_error = "You can't have an empty list item"

But using Django’s helper function is probably a better idea:

lists/tests/test_views.py (ch11l029)

from django.utils.html import escape
[...]

        expected_error = escape("You can't have an empty list item")
        self.assertContains(response, expected_error)

That passes!

Ran 11 tests in 0.047s

OK

Checking That Invalid Input Isn’t Saved to the Database

Before we go further though, did you notice a little logic error we’ve allowed to creep into our implementation? We’re currently creating an object, even if validation fails:

lists/views.py

    item = Item.objects.create(text=request.POST['item_text'], list=list_)
    try:
        item.full_clean()
    except ValidationError:
        [...]

Let’s add a new unit test to make sure that empty list items don’t get saved:

lists/tests/test_views.py (ch11l030-1)

class NewListTest(TestCase):
    [...]

    def test_validation_errors_are_sent_back_to_home_page_template(self):
        [...]

    def test_invalid_list_items_arent_saved(self):
        self.client.post('/lists/new', data={'item_text': ''})
        self.assertEqual(List.objects.count(), 0)
        self.assertEqual(Item.objects.count(), 0)

That gives:

[...]
Traceback (most recent call last):
  File "/.../superlists/lists/tests/test_views.py", line 40, in
test_invalid_list_items_arent_saved
    self.assertEqual(List.objects.count(), 0)
AssertionError: 1 != 0

We fix it like this:

lists/views.py (ch11l030-2)

def new_list(request):
    list_ = List.objects.create()
    item = Item(text=request.POST['item_text'], list=list_)
    try:
        item.full_clean()
        item.save()
    except ValidationError:
        list_.delete()
        error = "You can't have an empty list item"
        return render(request, 'home.html', {"error": error})
    return redirect(f'/lists/{list_.id}/')

Do the FTs pass?

$ python manage.py test functional_tests.test_list_item_validation
[...]
File "/.../superlists/functional_tests/test_list_item_validation.py", line
29, in test_cannot_add_empty_list_items
    self.wait_for(lambda: self.assertEqual(
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error

Not quite, but they did get a little further. Checking line 29, we can see that we’ve got past the first part of the test, and are now onto the second check—that submitting a second empty item also shows an error.

We’ve got some working code though, so let’s have a commit:

$ git commit -am "Adjust new list view to do model validation"

Django Pattern: Processing POST Requests in the Same View as Renders the Form

This time we’ll use a slightly different approach, one that’s actually a very common pattern in Django, which is to use the same view to process POST requests as to render the form that they come from. Whilst this doesn’t fit the REST-ful URL model quite as well, it has the important advantage that the same URL can display a form, and display any errors encountered in processing the user’s input.

The current situation is that we have one view and URL for displaying a list, and one view and URL for processing additions to that list. We’re going to combine them into one. So, in list.html, our form will have a different target:

lists/templates/list.html (ch11l030)

{% block form_action %}/lists/{{ list.id }}/{% endblock %}

Incidentally, that’s another hardcoded URL. Let’s add it to our to-do list, and while we’re thinking about it, there’s one in home.html too:

SCRATCHPAD:

• Remove hardcoded URLs from views.py

• Remove hardcoded URL from forms in list.html and home.html

This will immediately break our original functional test, because the view_list page doesn’t know how to process POST requests yet:

$ python manage.py test functional_tests
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error
[...]
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']

Note

In this section we’re performing a refactor at the application level. We execute our application-level refactor by changing or adding unit tests, and then adjusting our code. We use the functional tests to tell us when our refactor is complete and things are back to working as before. Have another look at the diagram from the end of Chapter 4 if you need to get your bearings.

Refactor: Transferring the new_item Functionality into view_list

Let’s take all the old tests from NewItemTest, the ones that are about saving POST requests to existing lists, and move them into ListViewTest. As we do so, we also make them point at the base list URL, instead of …/add_item:

lists/tests/test_views.py (ch11l031)

class ListViewTest(TestCase):

    def test_uses_list_template(self):
        [...]

    def test_passes_correct_list_to_template(self):
        [...]

    def test_displays_only_items_for_that_list(self):
        [...]

    def test_can_save_a_POST_request_to_an_existing_list(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()

        self.client.post(
            f'/lists/{correct_list.id}/',
            data={'item_text': 'A new item for an existing list'}
        )

        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new item for an existing list')
        self.assertEqual(new_item.list, correct_list)


    def test_POST_redirects_to_list_view(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()

        response = self.client.post(
            f'/lists/{correct_list.id}/',
            data={'item_text': 'A new item for an existing list'}
        )
        self.assertRedirects(response, f'/lists/{correct_list.id}/')

Note that the NewItemTest class disappears completely. I’ve also changed the name of the redirect test to make it explicit that it only applies to POST requests.

That gives:

FAIL: test_POST_redirects_to_list_view (lists.tests.test_views.ListViewTest)
AssertionError: 200 != 302 : Response didn't redirect as expected: Response
code was 200 (expected 302)
[...]
FAIL: test_can_save_a_POST_request_to_an_existing_list
(lists.tests.test_views.ListViewTest)
AssertionError: 0 != 1

We change the view_list function to handle two types of request:

lists/views.py (ch11l032-1)

def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'], list=list_)
        return redirect(f'/lists/{list_.id}/')
    return render(request, 'list.html', {'list': list_})

That gets us passing tests:

Ran 12 tests in 0.047s

OK

Now we can delete the add_item view, since it’s no longer needed…oops, an unexpected failure:

[...]
AttributeError: module 'lists.views' has no attribute 'add_item'

It’s because we’ve deleted the view, but it’s still being referred to in urls.py. We remove it from there:

lists/urls.py (ch11l033)

urlpatterns = [
    url(r'^new$', views.new_list, name='new_list'),
    url(r'^(\d+)/$', views.view_list, name='view_list'),
]

And that gets us to the OK. Let’s try a full FT run:

$ python manage.py test
[...]
ERROR: test_cannot_add_empty_list_items
[...]

Ran 16 tests in 15.276s
FAILED (errors=1)

We’re back to the one failure in our new functional test. Our refactor of the add_item functionality is complete. We should commit there:

$ git commit -am "Refactor list view to handle new item POSTs"

Note

So did I break the rule about never refactoring against failing tests? In this case, it’s allowed, because the refactor is required to get our new functionality to work. You should definitely never refactor against failing unit tests. But in my book it’s OK for the FT for the current story you’re working on to be failing.¹

Enforcing Model Validation in view_list

We still want the addition of items to existing lists to be subject to our model validation rules. Let’s write a new unit test for that; it’s very similar to the one for the home page, with just a couple of tweaks:

lists/tests/test_views.py (ch11l034)

class ListViewTest(TestCase):
    [...]

    def test_validation_errors_end_up_on_lists_page(self):
        list_ = List.objects.create()
        response = self.client.post(
            f'/lists/{list_.id}/',
            data={'item_text': ''}
        )
        self.assertEqual(response.status_code, 200)
        self.assertTemplateUsed(response, 'list.html')
        expected_error = escape("You can't have an empty list item")
        self.assertContains(response, expected_error)

That should fail, because our view currently does not do any validation, and just redirects for all POSTs:

    self.assertEqual(response.status_code, 200)
AssertionError: 302 != 200

Here’s an implementation:

lists/views.py (ch11l035)

def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    error = None

    if request.method == 'POST':
        try:
            item = Item(text=request.POST['item_text'], list=list_)
            item.full_clean()
            item.save()
            return redirect(f'/lists/{list_.id}/')
        except ValidationError:
            error = "You can't have an empty list item"

    return render(request, 'list.html', {'list': list_, 'error': error})

It’s not deeply satisfying, is it? There’s definitely some duplication of code here; that try/except occurs twice in views.py, and in general things are feeling clunky.

Ran 13 tests in 0.047s

OK

Let’s wait a bit before we do more refactoring though, because we know we’re about to do some slightly different validation coding for duplicate items. We’ll just add it to our scratchpad for now:

SCRATCHPAD:

• Remove hardcoded URLs from views.py

• Remove hardcoded URL from forms in list.html and home.html

• Remove duplication of validation logic in views

Note

One of the reasons that the “three strikes and refactor” rule exists is that, if you wait until you have three use cases, each might be slightly different, and it gives you a better view for what the common functionality is. If you refactor too early, you may find that the third use case doesn’t quite fit with your refactored code…

At least our functional tests are back to passing:

$ python manage.py test functional_tests
[...]
OK

We’re back to a working state, so we can take a look at some of the items on our scratchpad. This would be a good time for a commit. And possibly a tea break.

$ git commit -am "enforce model validation in list view"

Refactor: Removing Hardcoded URLs

Do you remember those name= parameters in urls.py? We just copied them across from the default example Django gave us, and I’ve been giving them some reasonably descriptive names. Now we find out what they’re for:

lists/urls.py

    url(r'^new$', views.new_list, name='new_list'),
    url(r'^(\d+)/$', views.view_list, name='view_list'),

The {% url %} Template Tag

We can replace the hardcoded URL in home.html with a Django template tag which refers to the URL’s “name”:

lists/templates/home.html (ch11l036-1)

{% block form_action %}{% url 'new_list' %}{% endblock %}

We check that this doesn’t break the unit tests:

$ python manage.py test lists
OK

Let’s do the other template. This one is more interesting, because we pass it a parameter:

lists/templates/list.html (ch11l036-2)

{% block form_action %}{% url 'view_list' list.id %}{% endblock %}

See the Django docs on reverse URL resolution for more info. We run the tests again, and check that they all pass:

$ python manage.py test lists
OK
$ python manage.py test functional_tests
OK

Excellent:

$ git commit -am "Refactor hard-coded URLs out of templates"

SCRATCHPAD:

• Remove hardcoded URLs from views.py

• Remove hardcoded URL from forms in list.html and home.html

• Remove duplication of validation logic in views

Using get_absolute_url for Redirects

Now let’s tackle views.py. One way of doing it is just like in the template, passing in the name of the URL and a positional argument:

lists/views.py (ch11l036-3)

def new_list(request):
    [...]
    return redirect('view_list', list_.id)

That would get the unit and functional tests passing, but the redirect function can do even better magic than that! In Django, because model objects are often associated with a particular URL, you can define a special function called get_absolute_url which says what page displays the item. It’s useful in this case, but it’s also useful in the Django admin (which I don’t cover in the book, but you’ll soon discover for yourself): it will let you jump from looking at an object in the admin view to looking at the object on the live site. I’d always recommend defining a get_absolute_url for a model whenever there is one that makes sense; it takes no time at all.

All it takes is a super-simple unit test in test_models.py:

lists/tests/test_models.py (ch11l036-4)

    def test_get_absolute_url(self):
        list_ = List.objects.create()
        self.assertEqual(list_.get_absolute_url(), f'/lists/{list_.id}/')

Which gives:

AttributeError: 'List' object has no attribute 'get_absolute_url'

The implementation is to use Django’s reverse function, which essentially does the reverse of what Django normally does with urls.py (see the docs):

lists/models.py (ch11l036-5)

from django.core.urlresolvers import reverse


class List(models.Model):

    def get_absolute_url(self):
        return reverse('view_list', args=[self.id])

And now we can use it in the view—the redirect function just takes the object we want to redirect to, and it uses get_absolute_url under the hood automagically!

lists/views.py (ch11l036-6)

def new_list(request):
    [...]
    return redirect(list_)

There’s more info in the Django docs. Quick check that the unit tests still pass:

OK

Then we do the same to view_list:

lists/views.py (ch11l036-7)

def view_list(request, list_id):
    [...]

            item.save()
            return redirect(list_)
        except ValidationError:
            error = "You can't have an empty list item"

And a full unit test and functional test run to assure ourselves that everything still works:

$ python manage.py test lists
OK
$ python manage.py test functional_tests
OK

Cross off our to-dos…

SCRATCHPAD:

• Remove hardcoded URLs from views.py

• Remove hardcoded URL from forms in list.html and home.html

• Remove duplication of validation logic in views

And a commit…

$ git commit -am "Use get_absolute_url on List model to DRY urls in views"

And we’re done with that bit! We have working model-layer validation, and we’ve taken the opportunity to do a few refactors along the way.

That final scratchpad item will be the subject of the next chapter…

On Database-Layer Validation

I always like to push my validation logic down as low as possible.

Validation at the database layer is the ultimate guarantee of data integrity

It can ensure that, no matter how complex your code at the layers above gets, you have guarantees at the lowest level that your data is valid and consistent.

But it comes at the expense of flexibility

This benefit doesn’t come for free! It’s now impossible, even temporarily, to have inconsistent data. Sometimes you might have a good reason for temporarily storing data that breaks the rules rather than storing nothing at all. Perhaps you’re importing data from an external source in several stages, for example.

And it’s not designed for user-friendliness

Trying to store invalid data will cause a nasty IntegrityError to come back from your database, and possibly the user will see a confusing 500 error page. As we’ll see in later chapters, forms-layer validation is designed with the user in mind, anticipating the kinds of helpful error messages we want to send them.

1 If you really want a “clean” test run, you could add a skip or an early return to the current FT, but you’d need to make sure you didn’t accidentally forget it.