Chapter 3. Writing Asynchronous Code with Deferreds

Callbacks are a fundamental part of event-driven programming and are the way that the reactor indicates to an application that an event has arrived. As event-driven programs grow, handling both the success and error cases for the events in one’s application becomes increasingly complex. Failing to register an appropriate callback can leave a program blocking on event processing that will never happen, and errors might have to propagate up a chain of callbacks from the networking stack through the layers of an application.

Twisted provides an elegant abstraction called a Deferred to manage these callbacks. This chapter will give you practice using Deferreds by themselves and then demonstrate their real-world utility by integrating Deferreds into some client and server examples.

We’ll use Deferreds while writing asynchronous servers and clients throughout the remainder of this book.

What Deferreds Do and Don’t Do

It’s worth heading off a common misconception up front:

• Deferreds do help you write asynchronous code.

• Deferreds do not automatically make code asynchronous or nonblocking. To turn a synchronous function into an asynchronous function, it’ll need to be refactored to return a Deferred with which callbacks are registered.

Practice will help you develop an intuition for how to structure asynchronous code. Let’s look at a Deferred so you can start getting some of that practice.

The Structure of a Deferred Object

Deferreds have a pair of callback chains, one for success (callbacks) and one for errors (errbacks). Deferreds start out with two empty chains. You add pairs of callbacks and errbacks to the Deferred to handle successes and failures at each point in the event processing. When an asynchronous result arrives, the Deferred is “fired” and the appropriate callbacks or errbacks are invoked in the order in which they were added to the chains. Figure 3-1 diagrams a Deferred and its callback chains.

Figure 3-1. A Deferred with its callback and errback chains

To get a feel for how Deferreds work, we can create them, register callbacks and errbacks with them, and fire them without involving the reactor.

Example 3-1 creates a Deferred d and uses its addCallback method to register the function myCallback with the callback chain. d.callback “fires” d and invokes the callback chain, which only contains myCallback. The argument passed to callback is propagated as an argument to the first function in the callback chain.

Example 3-1. Using addCallback

from twisted.internet.defer import Deferred

def myCallback(result):
    print result

d = Deferred()
d.addCallback(myCallback)
d.callback("Triggering callback.")

Running Example 3-1 produces the following:

    Triggering callback.

Example 3-2 creates a Deferred d and uses its addErrback method to register the function myErrback with the errback chain. d.errback “fires” d and invokes the first function in the errback chain, which only contains myErrback. The argument passed to errback is wrapped in a Failure object before getting passed to the errback function.

Example 3-2. Using addErrback

from twisted.internet.defer import Deferred

def myErrback(failure):
    print failure

d = Deferred()
d.addErrback(myErrback)
d.errback("Triggering errback.")

A Failure is Twisted’s implementation of a dressed-up Exception suitable for asynchronous communication. It contains a stack trace for where an asynchronous error actually happened (which might not be the current stack trace).

Running Example 3-2 produces the following:

    [Failure instance: Traceback (failure with no frames):
    <class 'twisted.python.failure.DefaultException'>:
    Triggering errback.
    ]

An asynchronous event may have many processing steps, each requiring a level of callbacks and errbacks. For example, a web request might need to be deserialized, formatted, and then cause a database insert, and each of those steps might possibly fail. Deferreds make it easy to manage these multiple levels of success and error handling in one place.

To register multiple levels of callbacks and errbacks with a Deferred, simply attach them to their callback chains in the order you want them invoked using addCallback and addErrback, as illustrated in Example 3-3. The result returned by a callback or errback in a Deferred chain is passed as the first argument to the next callback or errback in the chain.

Example 3-3. Registering multiple callbacks

from twisted.internet.defer import Deferred

def addBold(result):
    return "<b>%s</b>" % (result,)

def addItalic(result):
    return "<i>%s</i>" % (result,)

def printHTML(result):
    print result

d = Deferred()
d.addCallback(addBold)
d.addCallback(addItalic)
d.addCallback(printHTML)
d.callback("Hello World")

Running Example 3-3 produces:

    <i><b>Hello World</b></i>

Note that registering a callback with addCallback also registers a “pass-through” for that level of the errback chain. Similarly, registering an errback with addErrback also registers a “pass-through” for that level of the callback chain. The chains always have the same length.

Deferreds also sport an addCallbacks method, which registers both a callback and an errback at the same level in their respective callback chains. For example:

d = Deferred()
d.addCallbacks(myCallback, myErrback)
d.callback("Triggering callback.")

Callback Chains and Using Deferreds in the Reactor

Now that we have experience playing with callbacks and errbacks outside the reactor, let’s use them inside the reactor.

Example 3-4 retrieves a headline and then processes it, either converting it to HTML and then printing it or printing an error to stderr if the headline is too long.

Example 3-4. An asynchronous headline retriever

from twisted.internet import reactor, defer

class HeadlineRetriever(object):
    def processHeadline(self, headline):
        if len(headline) > 50:
            self.d.errback(
                "The headline ``%s'' is too long!" % (headline,))
        else:
            self.d.callback(headline)

    def _toHTML(self, result):
        return "<h1>%s</h1>" % (result,)

    def getHeadline(self, input):
        self.d = defer.Deferred()
        reactor.callLater(1, self.processHeadline, input)
        self.d.addCallback(self._toHTML)
        return self.d

def printData(result):
    print result
    reactor.stop()

def printError(failure):
    print failure
    reactor.stop()

h = HeadlineRetriever()
d = h.getHeadline("Breaking News: Twisted Takes Us to the Moon!")
d.addCallbacks(printData, printError)

reactor.run()

Running Example 3-4 produces:

    <h1>Breaking News: Twisted Takes Us to the Moon!</h1>

Because the provided headline is fewer than 50 characters long, HeadlineRetriever fires the callback chain, invoking _toHTML and then printData, which prints the HTML headline.

Example 3-4 uses a helpful reactor method called callLater, which you can use to schedule events. In this example, we use callLater in getHeadline to fake an asynchronous event arriving after one second.

What happens when we replace the three lines before reactor.run() with the following?

    h = HeadlineRetriever()
    d = h.getHeadline("1234567890"*6)
    d.addCallbacks(printData, printError)

Running this version of the example, we get:

    [Failure instance: Traceback (failure with no frames):
    <class 'twisted.python.failure.DefaultException'>:
    The headline ``1234567890123456789<...>01234567890'' is too long!
    ]

In this version, HeadlineRetriever encounters a headline that is too long and fires the errback chain: a pass-through (from the call to addCallback(self._toHTML)), then printError. Figure 3-2 traces the path followed through the Deferred.

Figure 3-2. Error path through HeadlineRetriever’s Deferred

Practice: What Do These Deferred Chains Do?

In this section, we’ll look at a series of examples where the functions from Example 3-5 are chained together in various ways as callbacks and errbacks in a Deferred that is then fired. For each example, think about what sequence of callbacks and errbacks is executed and what the resulting output is. In examples where the output includes a traceback, the middle of the traceback has been elided for brevity and clarity.

Example 3-5. Various functions for use as callbacks and errbacks

from twisted.internet.defer import Deferred

def callback1(result):
    print "Callback 1 said:", result
    return result

def callback2(result):
    print "Callback 2 said:", result

def callback3(result):
    raise Exception("Callback 3")

def errback1(failure):
    print "Errback 1 had an an error on", failure
    return failure

def errback2(failure):
    raise Exception("Errback 2")

def errback3(failure):
    print "Errback 3 took care of", failure
    return "Everything is fine now."

Exercise 1

d = Deferred()
d.addCallback(callback1)
d.addCallback(callback2)
d.callback("Test")

When this Deferred fires, execution starts at the top of the callback chain; callback1 is executed, followed by callback2. The result is:

Callback 1 said: Test
Callback 2 said: Test

Exercise 2

d = Deferred()
d.addCallback(callback1)
d.addCallback(callback2)
d.addCallback(callback3)
d.callback("Test")

When this Deferred fires, execution starts at the top of the callback chain; callback1 is executed, followed by callback2, followed by callback3. callback3 raises an Exception, and because there is no registered errback to handle the Exception, the program terminates and reports an Unhandled Error to the user. The result is:

Callback 1 said: Test
Callback 2 said: Test
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/tmp/test.py", line 33, in <module>
    d.callback("Test")
<...>
  File "/tmp/test.py", line 11, in callback3
    raise Exception("Callback 3")
exceptions.Exception: Callback 3

Exercise 3

d = Deferred()
d.addCallback(callback1)
d.addCallback(callback2)
d.addCallback(callback3)
d.addErrback(errback3)
d.callback("Test")

This Deferred has the same callbacks as the previous example, except that errback3 is also registered before firing. errback3 handles the Exception raised by callback3. The result is:

Callback 1 said: Test
Callback 2 said: Test
Errback 3 took care of [Failure instance:
Traceback: <type 'exceptions.Exception'>: Callback 3
test.py:40:<module>
<...>
test.py:11:callback3

Exercise 4

d = Deferred()
d.addErrback(errback1)
d.errback("Test")

This Deferred fires its errback chain. The first argument to an errback is always a Failure (being wrapped in one if necessary, as is the case with the “Test” string); errback1 returns the Failure, so that Failure is passed along as the argument to the next errback in the chain for processing. Because there is no additional errback to handle the Failure, execution stops with an Unhandled Error:

Errback 1 had an an error on [Failure instance:
Traceback (failure with no frames):
<class 'twisted.python.failure.DefaultException'>: Test
]
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
Failure: twisted.python.failure.DefaultException: Test

Exercise 5

d = Deferred()
d.addErrback(errback1)
d.addErrback(errback3)
d.errback("Test")

This Deferred fires its errback chain, and errback1 propagates a Failure to errback3. errback3 handles the Failure by virtue of not raising an Exception or returning a Failure. It instead returns a string; because there is no callback at the next level to process the result, the Deferred is done firing.

Errback 1 had an an error on [Failure instance:
Traceback (failure with no frames):
<class 'twisted.python.failure.DefaultException'>: Test
]
Errback 3 took care of [Failure instance:
Traceback (failure with no frames):
<class 'twisted.python.failure.DefaultException'>: Test
]

Exercise 6

d = Deferred()
d.addErrback(errback2)
d.errback("Test")

This Deferred fires its errback chain, starting with errback2, which raises an Exception. Since raising an Exception passes control to the next errback in the chain, and there is no errback to handle the Exception, an Unhandled Error is raised:

Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "test.py", line 59, in <module>
    d.errback("Test")
<...>
  File "test.py", line 18, in errback2
    raise Exception("Errback 2")
exceptions.Exception: Errback 2

The Truth About addCallbacks

Now that you have some Deferred practice under your belt, a somewhat subtle point needs to be made: addCallbacks is not the same as sequential calls to addCallback and addErrback.

What’s the difference?

addCallbacks

Registers a callback with the callback chain and an errback with the errback chain, at the same level

addCallback

Registers a callback with the callback chain and a pass-through with the errback chain, which simply returns the result passed to it

addErrback

Registers an errback with the errback chain and a pass-through with the callback chain

The salient difference is that callbacks and errbacks registered together using addCallbacks do not interact. Put another way, when a callback and an errback are registered together using addCallbacks, that errback can’t handle exceptions raised by that callback: exceptions raised at level N in the callback chain are processed by the errback at level N + 1.

Figures 3-3 and 3-4 depict the difference between a call to addCallbacks and sequential calls to addCallback and addErrback.

Figure 3-3. A single call to addCallbacks

Figure 3-4. Sequential calls to addCallback and addErrback

Given this distinction, what do the following Deferred chains do?

Exercise 7

d = Deferred()
d.addCallback(callback1)
d.addCallback(callback2)
d.addCallbacks(callback3, errback3)
d.callback("Test")

This Deferred chain is the same as the one in Exercise 3, except that instead of calling addCallback(callback3) and addErrback(errback3) sequentially, they are registered together using addCallbacks. These code fragments are not equivalent! In Exercise 3, callback3 and a pass-through were registered as the callback and errback at level 3 for this Deferred, and then a pass-through and errback3 were registered as the callback and errback at level 4. This meant that an Exception raised on level 3 could be handled by the errback at level 4.

In Exercise 7, callback3 and errback3 are registered together as the callback and errback on level 3. This means there is no errback at level 4 to handle Exceptions raised at level 3. The result is:

Callback 1 said: Test
Callback 2 said: Test
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "test.py", line 46, in <module>
    d.callback("Test")
<...>
  File "test.py", line 11, in callback3
    raise Exception("Callback 3")
exceptions.Exception: Callback 3

Exercise 8

d = Deferred()
d.addCallback(callback3)
d.addCallbacks(callback2, errback3)
d.addCallbacks(callback1, errback2)
d.callback("Test")

This Deferred fires its callback chain. callback3 raises an Exception, so control passes to the next errback in the chain, errback3. errback3 handles the Exception, so control passes back to the callback chain and callback1 is invoked. The result is:

Errback 3 took care of [Failure instance:
Traceback: <type 'exceptions.Exception'>: Callback 3
test.py:75:<module>
<...>
test.py:11:callback3
]
Callback 1 said: Everything is fine now.

Key Facts About Deferreds

This section reiterates some important points about Deferreds and introduces a few new ones:

1. A Deferred is “fired” by invoking its callback or errback method.

2. A Deferred can only be fired once. Attempting to fire it again results in an AlreadyCalledError. This helps prevent accidentally processing an event more than once.

3. Exceptions at level N in the callback and errback chains are handled by the errback at level N + 1.

   If a callback or errback raises an Exception or returns a Failure at level N, the errback at level N + 1 is invoked to handle that error. If there is no errback, program execution stops and an Unhandled Error is reported.

   If a callback or errback at level N doesn’t raise an Exception or return a Failure, control is passed to the callback at level N + 1. Note that this applies to errbacks! If an errback doesn’t produce an error, control passes to the callback chain. Control will criss-cross between the errback and callback chains depending on the results of processing the event.

4. The result returned by a callback in a Deferred chain is passed as the first argument to the next callback in the chain. This is what allows chaining processing of results. Don’t forget to return the result from your callbacks for further processing!

5. If the object passed to an errback is not already a Failure, it is first wrapped in one. This includes objects passed to the errback chain when firing a Deferred and Exceptions raised by callbacks, which switch control to the errback chain for processing.

Summary of the Deferred API

The Deferred API has one last method for adding callbacks, addBoth, which adds the same callback to both the callback and errback chains for the Deferred. Note that while we haven’t been passing arguments to our callback yet, that is supported by the API. The supported methods are:

addCallback

Add a callback to the callback chain for the Deferred and add a pass-through to the errback chain.

addErrback

Add an errback to the errback chain for the Deferred and add a pass-through to the callback chain. The analogous synchronous logic is the except part of a try/except block.

addCallbacks

Add a callback and errback parallel to each other in the callback chains for the Deferred.

addBoth

Add the same callback to both the callback and errback chains for the Deferred. The analogous synchronous logic is the finally part of a try/except/finally block.

More Practice and Next Steps

This chapter introduced the Deferred, an abstraction that simplifies and centralizes the management of callbacks for success and error handling in your asynchronous programs.

We’ll use Deferreds while writing HTTP servers and clients in the next two chapters.

The Twisted Core HOWTO has two main documents on Deferreds, an overview of using them, and a guide to writing functions that generate them.