Appendix F. Building a REST API: JSON, Ajax, and Mocking with JavaScript

Representational State Transfer (REST) is an approach to designing a web service to allow a user to retrieve and update information about “resources”. It’s become the dominant approach when designing APIs for use over the web.

We’ve built a working web app without needing an API so far. Why might we want one? One motivation might be to improve the user experience by making the site more dynamic. Rather than waiting for the page to refresh after each addition to a list, we can use JavaScript to fire off those requests asynchronously to our API, and give the user a more interactive feeling.

Perhaps more interestingly, once we’ve built an API, we can interact with our back-end application via other mechanisms than the browser. A mobile app might be one new candidate client application, another might be some sort of command-line application, or other developers might be able to build libraries and tools around your backend.

In this chapter we’ll see how to build an API “by hand”. In the next, I’ll give an overview of how to use a popular tool from the Django ecosystem called Django-Rest-Framework.

Our Approach for This Appendix

I won’t convert the entirety of the app for now; we’ll start by assuming we have an existing list. REST defines a relationship between URLs and the HTTP methods (GET and POST, but also the more funky ones like PUT and DELETE) which will guide us in our design.

The Wikipedia entry on REST has a good overview. In brief:

  • Our new URL structure will be /api/lists/{id}/

  • GET will give you details of a list (including all its items) in JSON format

  • POST lets you add an item

We’ll take the code from its state at the end of Chapter 25.

Basic Piping

We start with a unit test that just checks that our new URL structure returns a 200 response to GET requests, and that it uses the JSON format (instead of HTML):

lists/tests/test_api.py

import json
from django.test import TestCase

from lists.models import List, Item


class ListAPITest(TestCase):
    base_url = '/api/lists/{}/'  1

    def test_get_returns_json_200(self):
        list_ = List.objects.create()
        response = self.client.get(self.base_url.format(list_.id))
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response['content-type'], 'application/json')
1

Using a class-level constant for the URL under test is a new pattern we’ll introduce for this appendix. It’ll help us to remove duplication of hardcoded URLs. You could even use a call to reverse to reduce duplication even further.

First we wire up a couple of urls files:

superlists/urls.py

from django.conf.urls import include, url
from accounts import urls as accounts_urls
from lists import views as list_views
from lists import api_urls
from lists import urls as list_urls

urlpatterns = [
    url(r'^$', list_views.home_page, name='home'),
    url(r'^lists/', include(list_urls)),
    url(r'^accounts/', include(accounts_urls)),
    url(r'^api/', include(api_urls)),
]

and:

lists/api_urls.py

from django.conf.urls import url
from lists import api

urlpatterns = [
    url(r'^lists/(\d+)/$', api.list, name='api_list'),
]

And the actual core of our API can live in a file called api.py. Just three lines should be enough:

lists/api.py

from django.http import HttpResponse

def list(request, list_id):
    return HttpResponse(content_type='application/json')

The tests should pass, and we have the basic piping together:

$ python manage.py test lists
[...]
..................................................
 ---------------------------------------------------------------------
Ran 50 tests in 0.177s

OK

Actually Responding with Something

Our next step is to get our API to actually respond with some content—specifically, a JSON representation of our list items:

lists/tests/test_api.py (ch36l002)

    def test_get_returns_items_for_correct_list(self):
        other_list = List.objects.create()
        Item.objects.create(list=other_list, text='item 1')
        our_list = List.objects.create()
        item1 = Item.objects.create(list=our_list, text='item 1')
        item2 = Item.objects.create(list=our_list, text='item 2')
        response = self.client.get(self.base_url.format(our_list.id))
        self.assertEqual(
            json.loads(response.content.decode('utf8')),  1
            [
                {'id': item1.id, 'text': item1.text},
                {'id': item2.id, 'text': item2.text},
            ]
        )
1

This is the main thing to notice about this test. We expect our response to be in JSON format; we use json.loads() because testing Python objects is easier than messing about with raw JSON strings.

And the implementation, conversely, uses json.dumps():

lists/api.py

import json
from django.http import HttpResponse
from lists.models import List, Item


def list(request, list_id):
    list_ = List.objects.get(id=list_id)
    item_dicts = [
        {'id': item.id, 'text': item.text}
        for item in list_.item_set.all()
    ]
    return HttpResponse(
        json.dumps(item_dicts),
        content_type='application/json'
    )

A nice opportunity to use a list comprehension!

Adding POST

The second thing we need from our API is the ability to add new items to our list by using a POST request. We’ll start with the “happy path”:

lists/tests/test_api.py (ch36l004)

    def test_POSTing_a_new_item(self):
        list_ = List.objects.create()
        response = self.client.post(
            self.base_url.format(list_.id),
            {'text': 'new item'},
        )
        self.assertEqual(response.status_code, 201)
        new_item = list_.item_set.get()
        self.assertEqual(new_item.text, 'new item')

And the implementation is similarly simple—basically the same as what we do in our normal view, but we return a 201 rather than a redirect:

lists/api.py (ch36l005)

def list(request, list_id):
    list_ = List.objects.get(id=list_id)
    if request.method == 'POST':
        Item.objects.create(list=list_, text=request.POST['text'])
        return HttpResponse(status=201)
    item_dicts = [
        [...]

And that should get us started:

$ python manage.py test lists
[...]

Ran 52 tests in 0.177s

OK
Note

One of the fun things about building a REST API is that you get to use a few more of the full range of HTTP status codes.

Testing the Client-Side Ajax with Sinon.js

Don’t even think of doing Ajax testing without a mocking library. Different test frameworks and tools have their own; Sinon is generic. It also provides JavaScript mocks, as we’ll see…

Start by downloading it from its site, http://sinonjs.org/, and putting it into our lists/static/tests/ folder.

Then we can write our first Ajax test:

lists/static/tests/tests.html (ch36l007)

  <div id="qunit-fixture">
    <form>
      <input name="text" />
      <div class="has-error">Error text</div>
    </form>
    <table id="id_list_table">  1
    </table>
  </div>

  <script src="../jquery-3.1.1.min.js"></script>
  <script src="../list.js"></script>
  <script src="qunit-2.0.1.js"></script>
  <script src="sinon-1.17.6.js"></script>  2

  <script>
/* global sinon */

var server;
QUnit.testStart(function () {
  server = sinon.fakeServer.create();  3
});
QUnit.testDone(function () {
  server.restore();  3
});

QUnit.test("errors should be hidden on keypress", function (assert) {
[...]


QUnit.test("should get items by ajax on initialize", function (assert) {
  var url = '/getitems/';
  window.Superlists.initialize(url);

  assert.equal(server.requests.length, 1); 4
  var request = server.requests[0];
  assert.equal(request.url, url);
  assert.equal(request.method, 'GET');
});

  </script>
1

We add a new item to the fixture div to represent our list table.

2

We import sinon.js (you’ll need to download it and put it in the right folder).

3

testStart and testDone are the QUnit equivalents of setUp and tearDown. We use them to tell Sinon to start up its Ajax testing tool, the fakeServer, and make it available via a globally scoped variable called server.

4

That lets us make assertions about any Ajax requests that were made by our code. In this case, we test what URL the request went to, and what HTTP method it used.

To actually make our Ajax request, we’ll use the jQuery Ajax helpers, which are much easier than trying to use the low-level browser standard XMLHttpRequest objects:

lists/static/list.js

@@ -1,6 +1,10 @@
 window.Superlists = {};
-window.Superlists.initialize = function () {
+window.Superlists.initialize = function (url) {
   $('input[name="text"]').on('keypress', function () {
     $('.has-error').hide();
   });
+
+  $.get(url);
+
 };
+

That should get our test passing:

5 assertions of 5 passed, 0 failed.
1. errors should be hidden on keypress (1)
2. errors aren't hidden if there is no keypress (1)
3. should get items by ajax on initialize (3)

Well, we might be pinging out a GET request to the server, but what about actually doing something? How do we test the actual “async” part, where we deal with the (eventual) response?

Sinon and Testing the Asynchronous Part of Ajax

This is a major reason to love Sinon. server.respond() allows us to exactly control the flow of the asynchronous code.

lists/static/tests/tests.html (ch36l009)

QUnit.test("should fill in lists table from ajax response", function (assert) {
  var url = '/getitems/';
  var responseData = [
    {'id': 101, 'text': 'item 1 text'},
    {'id': 102, 'text': 'item 2 text'},
  ];
  server.respondWith('GET', url, [
    200, {"Content-Type": "application/json"}, JSON.stringify(responseData) 1
  ]);
  window.Superlists.initialize(url); 2

  server.respond(); 3

  var rows = $('#id_list_table tr');  4
  assert.equal(rows.length, 2);
  var row1 = $('#id_list_table tr:first-child td');
  assert.equal(row1.text(), '1: item 1 text');
  var row2 = $('#id_list_table tr:last-child td');
  assert.equal(row2.text(), '2: item 2 text');
});
1

We set up some response data for Sinon to use, telling it what status code, headers, and importantly what kind of response JSON we want to simulate coming from the server.

2

Then we call the function under test.

3

Here’s the magic. Then we can call server.respond(), whenever we like, and that will kick off all the async part of the Ajax loop—that is, any callback we’ve assigned to deal with the response.

4

Now we can quietly check whether our Ajax callback has actually populated our table with the new list rows…

The implementation might look something like this:

lists/static/list.js (ch36l010)

  if (url) {
    $.get(url).done(function (response) {  1
      var rows = '';
      for (var i=0; i<response.length; i++) {  2
        var item = response[i];
        rows += '\n<tr><td>' + (i+1) + ': ' + item.text + '</td></tr>';
      }
      $('#id_list_table').html(rows);
    });
  }
Tip

We’re lucky because of the way jQuery registers its callbacks for Ajax when we use the .done() function. If you want to switch to the more standard JavaScript Promise .then() callback, we get one more “level” of async. QUnit does have a way of dealing with that. Check out the docs for the async function. Other test frameworks have something similar.

Wiring It All Up in the Template to See If It Really Works

We break it first, by removing the list table {% for %} loop from the lists.html template:

lists/templates/list.html

@@ -6,9 +6,6 @@

 {% block table %}
   <table id="id_list_table" class="table">
-    {% for item in list.item_set.all %}
-      <tr><td>{{ forloop.counter }}: {{ item.text }}</td></tr>
-    {% endfor %}
   </table>

   {% if list.owner %}
Note

This will cause one of the unit tests to fail. It’s OK to delete that test at this point.

That causes our basic FT to fail:

$ python manage.py test functional_tests.test_simple_list_creation
[...]
FAIL: test_can_start_a_list_for_one_user
[...]
  File "/.../superlists/functional_tests/test_simple_list_creation.py", line
32, in test_can_start_a_list_for_one_user
    self.wait_for_row_in_list_table('1: Buy peacock feathers')
[...]
AssertionError: '1: Buy peacock feathers' not found in []
[...]
FAIL: test_multiple_users_can_start_lists_at_different_urls

FAILED (failures=2)

Let’s add a block called {% scripts %} to the base template, which we can selectively override later in our lists page:

lists/templates/base.html

    <script src="/static/list.js"></script>

    {% block scripts %}
      <script>
$(document).ready(function () {
  window.Superlists.initialize();
});
      </script>
    {% endblock scripts %}

  </body>

And now in list.html we add a slightly different call to initialize, with the correct URL:

lists/templates/list.html (ch36l016)

{% block scripts %}
  <script>
$(document).ready(function () {
  var url = "{% url 'api_list' list.id %}";
  window.Superlists.initialize(url);
});
  </script>
{% endblock scripts %}

And guess what? The test passes!

$ python manage.py test functional_tests.test_simple_list_creation
[...]
Ran 2 test in 11.730s

OK

That’s a pretty good start!

Now if you run all the FTs you’ll see we’ve got some failures in other FTs, so we’ll have to deal with them. Also, we’re using an old-fashioned POST from the form, with page refresh, so we’re not at our trendy hipster single-page app yet. But we’ll get there!

Implementing Ajax POST, Including the CSRF Token

First we give our list form an id so we can pick it up easily in our JS:

lists/templates/base.html

  <h1>{% block header_text %}{% endblock %}</h1>
  {% block list_form %}
    <form id="id_item_form" method="POST" action="{% block form_action %}{% endblock %}">
      {{ form.text }}
      [...]

Next tweak the fixture in our JS test to reflect that ID, as well as the CSRF token that’s currently on the page:

lists/static/tests/tests.html

@@ -9,9 +9,14 @@
 <body>
   <div id="qunit"></div>
   <div id="qunit-fixture">
-    <form>
+    <form id="id_item_form">
       <input name="text" />
-      <div class="has-error">Error text</div>
+      <input type="hidden" name="csrfmiddlewaretoken" value="tokey" />
+      <div class="has-error">
+        <div class="help-block">
+          Error text
+        </div>
+      </div>
     </form>

And here’s our test:

lists/static/tests/tests.html (ch36l019)

QUnit.test("should intercept form submit and do ajax post", function (assert) {
  var url = '/listitemsapi/';
  window.Superlists.initialize(url);

  $('#id_item_form input[name="text"]').val('user input');  1
  $('#id_item_form input[name="csrfmiddlewaretoken"]').val('tokeney');  1
  $('#id_item_form').submit();  1

  assert.equal(server.requests.length, 2);  2
  var request = server.requests[1];
  assert.equal(request.url, url);
  assert.equal(request.method, "POST");
  assert.equal(
    request.requestBody,
    'text=user+input&csrfmiddlewaretoken=tokeney'  3
  );
});
1

We simulate the user filling in the form and hitting Submit.

2

We now expect that there should be a second Ajax request (the first one is the GET for the list items table).

3

We check our POST requestBody. As you can see, it’s URL-encoded, which isn’t the most easy value to test, but it’s still just about readable.

And here’s how we implement it:

lists/static/list.js

[...]
  $('#id_list_table').html(rows);
});

var form = $('#id_item_form');
form.on('submit', function(event) {
  event.preventDefault();
  $.post(url, {
    'text': form.find('input[name="text"]').val(),
    'csrfmiddlewaretoken': form.find('input[name="csrfmiddlewaretoken"]').val(),
  });
});

That gets our JS tests passing but it breaks our FTs, because, although we’re doing our POST all right, we’re not updating the page after the POST to show the new list item:

$ python manage.py test functional_tests.test_simple_list_creation
[...]
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']

Mocking in JavaScript

We want our client side to update the table of items after the Ajax POST completes. Essentially it’ll do the same work as we do as soon as the page loads, retrieving the current list of items from the server, and filling in the item table.

Sounds like a helper function is in order!

lists/static/list.js

window.Superlists = {};

window.Superlists.updateItems = function (url) {
  $.get(url).done(function (response) {
    var rows = '';
    for (var i=0; i<response.length; i++) {
      var item = response[i];
      rows += '\n<tr><td>' + (i+1) + ': ' + item.text + '</td></tr>';
    }
    $('#id_list_table').html(rows);
  });
};

window.Superlists.initialize = function (url) {
  $('input[name="text"]').on('keypress', function () {
    $('.has-error').hide();
  });

  if (url) {
    window.Superlists.updateItems(url);

    var form = $('#id_item_form');
    [...]

That was just a refactor; now we check that the JS tests all still pass:

12 assertions of 12 passed, 0 failed.
1. errors should be hidden on keypress (1)
2. errors aren't hidden if there is no keypress (1)
3. should get items by ajax on initialize (3)
4. should fill in lists table from ajax response (3)
5. should intercept form submit and do ajax post (4)

Now how to test that our Ajax POST calls updateItems on POST success? We don’t want to dumbly duplicate the code that simulates a server response and checks the items table manually…how about a mock?

First we set up a thing called a “sandbox”. It will keep track of all the mocks we create, and make sure to un-monkeypatch all the things that have been mocked after each test:

lists/static/tests/tests.html (ch36l023)

var server, sandbox;
QUnit.testStart(function () {
  server = sinon.fakeServer.create();
  sandbox = sinon.sandbox.create();
});
QUnit.testDone(function () {
  server.restore();
  sandbox.restore(); 1
});
1

This .restore() is the important part; it undoes all the mocking we’ve done in each test.

lists/static/tests/tests.html (ch36l024)

QUnit.test("should call updateItems after successful post", function (assert) {
  var url = '/listitemsapi/';
  window.Superlists.initialize(url); 1
  var response = [
    201,
    {"Content-Type": "application/json"},
    JSON.stringify({}),
  ];
  server.respondWith('POST', url, response); 1
  $('#id_item_form input[name="text"]').val('user input');
  $('#id_item_form input[name="csrfmiddlewaretoken"]').val('tokeney');
  $('#id_item_form').submit();

  sandbox.spy(window.Superlists, 'updateItems');  2
  server.respond();  2

  assert.equal(
    window.Superlists.updateItems.lastCall.args,  3
    url
  );
});
1

First important thing to notice: We only set up our server response after we do the initialize. We want this to be the response to the POST request that happens on form submit, not the response to the initial GET request. (Remember our lesson from Chapter 16? One of the most challenging things about JS testing is controlling the order of execution.)

2

Similarly, we only start mocking our helper function after we know the first call for the initial GET has already happened. The sandbox.spy call is what does the job that patch does in Python tests. It replaces the given object with a mock version.

3

Our updateItems function has now grown some mocky extra attributes, like lastCall and lastCall.args, which are like the Python mock’s call_args.

To get it passing, we first make a deliberate mistake, to check that our tests really do test what we think they do:

lists/static/list.js

$.post(url, {
  'text': form.find('input[name="text"]').val(),
  'csrfmiddlewaretoken': form.find('input[name="csrfmiddlewaretoken"]').val(),
}).done(function () {
  window.Superlists.updateItems();
});

Yep, we’re almost there but not quite:

12 assertions of 13 passed, 1 failed.
[...]
6. should call updateItems after successful post (1, 0, 1)
    1. failed
        Expected: "/listitemsapi/"
        Result: []
        Diff: "/listitemsapi/"[]
        Source: file:///.../superlists/lists/static/tests/tests.html:124:15

And we fix it thusly:

lists/static/list.js

      }).done(function () {
        window.Superlists.updateItems(url);
      });

And our FT passes! Or at least one of them does. The others have problems, and we’ll come back to them shortly.

Finishing the Refactor: Getting the Tests to Match the Code

First, I’m not happy until we’ve seen through this refactor, and made our unit tests match the code a little more:

lists/static/tests/tests.html

@@ -50,9 +50,19 @@ QUnit.testDone(function () {
 });


-QUnit.test("should get items by ajax on initialize", function (assert) {
+QUnit.test("should call updateItems on initialize", function (assert) {
   var url = '/getitems/';
+  sandbox.spy(window.Superlists, 'updateItems');
   window.Superlists.initialize(url);
+  assert.equal(
+    window.Superlists.updateItems.lastCall.args,
+    url
+  );
+});
+
+QUnit.test("updateItems should get correct url by ajax", function (assert) {
+  var url = '/getitems/';
+  window.Superlists.updateItems(url);

   assert.equal(server.requests.length, 1);
   var request = server.requests[0];
@@ -60,7 +70,7 @@ QUnit.test("should get items by ajax on initialize", function (assert) {
   assert.equal(request.method, 'GET');
 });

-QUnit.test("should fill in lists table from ajax response", function (assert) {
+QUnit.test("updateItems should fill in lists table from ajax response", function (assert) {
   var url = '/getitems/';
   var responseData = [
     {'id': 101, 'text': 'item 1 text'},
@@ -69,7 +79,7 @@ QUnit.test("should fill in lists table from ajax response", function [...]
   server.respondWith('GET', url, [
     200, {"Content-Type": "application/json"}, JSON.stringify(responseData)
   ]);
-  window.Superlists.initialize(url);
+  window.Superlists.updateItems(url);

   server.respond();

And that should give us a test run that looks like this instead:

14 assertions of 14 passed, 0 failed.
1. errors should be hidden on keypress (1)
2. errors aren't hidden if there is no keypress (1)
3. should call updateItems on initialize (1)
4. updateItems should get correct url by ajax (3)
5. updateItems should fill in lists table from ajax response (3)
6. should intercept form submit and do ajax post (4)
7. should call updateItems after successful post (1)

Data Validation: An Exercise for the Reader?

If you do a full test run, you should find two of the validation FTs are failing:

$ python manage.py test
[...]
ERROR: test_cannot_add_duplicate_items
(functional_tests.test_list_item_validation.ItemValidationTest)
[...]
ERROR: test_error_messages_are_cleared_on_input
(functional_tests.test_list_item_validation.ItemValidationTest)
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: .has-error

I won’t spell this all out for you, but here’s at least the unit tests you’ll need:

lists/tests/test_api.py (ch36l027)

from lists.forms import DUPLICATE_ITEM_ERROR, EMPTY_ITEM_ERROR
[...]
    def post_empty_input(self):
        list_ = List.objects.create()
        return self.client.post(
            self.base_url.format(list_.id),
            data={'text': ''}
        )


    def test_for_invalid_input_nothing_saved_to_db(self):
        self.post_empty_input()
        self.assertEqual(Item.objects.count(), 0)


    def test_for_invalid_input_returns_error_code(self):
        response = self.post_empty_input()
        self.assertEqual(response.status_code, 400)
        self.assertEqual(
            json.loads(response.content.decode('utf8')),
            {'error': EMPTY_ITEM_ERROR}
        )


    def test_duplicate_items_error(self):
        list_ = List.objects.create()
        self.client.post(
            self.base_url.format(list_.id), data={'text': 'thing'}
        )
        response = self.client.post(
            self.base_url.format(list_.id), data={'text': 'thing'}
        )
        self.assertEqual(response.status_code, 400)
        self.assertEqual(
            json.loads(response.content.decode('utf8')),
            {'error': DUPLICATE_ITEM_ERROR}
        )

And on the JS side:

lists/static/tests/tests.html (ch36l029-2)

QUnit.test("should display errors on post failure", function (assert) {
  var url = '/listitemsapi/';
  window.Superlists.initialize(url);
  server.respondWith('POST', url, [
    400,
    {"Content-Type": "application/json"},
    JSON.stringify({'error': 'something is amiss'})
  ]);
  $('.has-error').hide();

  $('#id_item_form').submit();
  server.respond(); // post

  assert.equal($('.has-error').is(':visible'), true);
  assert.equal($('.has-error .help-block').text(), 'something is amiss');
});

QUnit.test("should hide errors on post success", function (assert) {
    [...]

You’ll also want some modifications to base.html to make it compatible with both displaying Django errors (which the home page still uses for now) and errors from JavaScript:

lists/templates/base.html (ch36l031)

@@ -51,17 +51,21 @@
         <div class="col-md-6 col-md-offset-3 jumbotron">
           <div class="text-center">
             <h1>{% block header_text %}{% endblock %}</h1>
+
             {% block list_form %}
               <form id="id_item_form" method="POST" action="{% block [...]
                 {{ form.text }}
                 {% csrf_token %}
-                {% if form.errors %}
-                  <div class="form-group has-error">
-                    <div class="help-block">{{ form.text.errors }}</div>
+                <div class="form-group has-error">
+                  <div class="help-block">
+                    {% if form.errors %}
+                      {{ form.text.errors }}
+                    {% endif %}
                   </div>
-                {% endif %}
+                </div>
               </form>
             {% endblock %}
+
           </div>
         </div>
       </div>

By the end you should get to a JS test run a bit like this:

20 assertions of 20 passed, 0 failed.
1. errors should be hidden on keypress (1)
2. errors aren't hidden if there is no keypress (1)
3. should call updateItems on initialize (1)
4. updateItems should get correct url by ajax (3)
5. updateItems should fill in lists table from ajax response (3)
6. should intercept form submit and do ajax post (4)
7. should call updateItems after successful post (1)
8. should not intercept form submit if no api url passed in (1)
9. should display errors on post failure (2)
10. should hide errors on post success (1)
11. should display generic error if no error json (2)

And a full test run should pass, including all the FTs:

$ python manage.py test
[...]
Ran 81 tests in 62.029s
OK

Laaaaaahvely.1

And there’s your hand-rolled REST API with Django. If you need a hint finishing it off yourself, check out the repo.

But I would never suggest building a REST API in Django without at least checking out Django-Rest-Framework. Which is the topic of the next appendix! Read on, Macduff.

1 Put on your best cockney accent for this one.