Chapter 6. Deploying Twisted Applications

Twisted is an engine for producing scalable, cross-platform network servers and clients. Making it easy to deploy these applications in a standardized fashion in production environments is an important part of a platform like this getting wide-scale adoption.

To that end, Twisted provides an application infrastructure: a reusable and configurable way to deploy a Twisted application. It allows a programmer to avoid boilerplate code by hooking an application into existing tools for customizing the way it is run, including daemonization, logging, using a custom reactor, profiling code, and more.

The Twisted Application Infrastructure

The application infrastructure has five main components: services, applications, TAC files, plugins, and the twistd command-line utility. To illustrate this infrastructure, we’ll turn the echo server from Chapter 2 into an application. Example 6-1 reproduces the server code.

Example 6-1. echoserver.py from Chapter 2
from twisted.internet import protocol, reactor

class Echo(protocol.Protocol):
    def dataReceived(self, data):
        self.transport.write(data)

class EchoFactory(protocol.Factory):
    def buildProtocol(self, addr):
        return Echo()

reactor.listenTCP(8000, EchoFactory())
reactor.run()

Services

A service is anything that can be started and stopped and that implements the IService interface. Twisted comes with service implementations for TCP, FTP, HTTP, SSH, DNS, and many other protocols. Many services can register with a single application.

The core of the IService interface is:

startService

Start the service. This might include loading configuration data, setting up database connections, or listening on a port.

stopService

Shut down the service. This might include saving state to disk, closing database connections, or stopping listening on a port.

Our echo service uses TCP, so we can use Twisted’s default TCPServer implementation of this IService interface.

An application is the top-level container for one or more services that are deployed together. Services register themselves with an application, and the twistd deployment utility described shortly searches for and runs applications.

We’ll create an echo application with which the echo service can register.

When writing a Twisted program as a regular Python file, the developer is responsible for writing code to start and stop the reactor and to configure the program. Under the Twisted application infrastructure, protocol implementations live in a module, services using those protocols are registered in a Twisted application configuration (TAC) file, and the reactor and configuration are managed by an external utility.

To turn our echo server into an echo application, we can follow a simple algorithm:

  1. Move the Protocol and Factory for the service into their own module.

  2. Inside a TAC file:

    1. Create an instance of twisted.application.service.Application.

    2. Instead of registering the Protocol Factory with a reactor, like in Chapter 2, register the factory with a service, and register that service with the Application.

In our case, this means creating an instance of the TCPServer service, which will use our EchoFactory to create instances of the Echo protocol on port 8000.

The code for managing the reactor will be taken care of by twistd, which we’ll discuss next. The application code is now split into two files: echo.py, shown in Example 6-2; and echo_server.tac, shown in Example 6-3.

Example 6-2. echo.py, a module containing the Protocol and Factory definitions
from twisted.internet import protocol, reactor

class Echo(protocol.Protocol):
    def dataReceived(self, data):
        self.transport.write(data)

class EchoFactory(protocol.Factory):
    def buildProtocol(self, addr):
        return Echo()
Example 6-3. echo_server.tac, a Twisted application configuration file
from twisted.application import internet, service
from echo import EchoFactory

application = service.Application("echo")
echoService = internet.TCPServer(8000, EchoFactory())
echoService.setServiceParent(application)

twistd (pronounced “twist-dee”) is a cross-platform utility for deploying Twisted applications. It runs TAC files and handles starting and stopping the application. As part of Twisted’s batteries-included approach to network programming, twistd comes with a number of useful configuration flags, including flags for daemonizing the application, specifying the location of log files, dropping privileges, running in a chroot, running under a non-default reactor, or even running the application under a profiler.

We can run our echo server application with:

twistd -y echo_server.tac

In this simplest case, twistd starts a daemonized instance of the application, logging to twistd.log, with a PID stored in twisted.pid. After starting and stopping the application, the log looks like this:

2012-11-19 22:23:07-0500 [-] Log opened.
2012-11-19 22:23:07-0500 [-] twistd 12.1.0 (/usr/bin/python 2.7.1) ...
2012-11-19 22:23:07-0500 [-] reactor class: twisted.internet.select...
2012-11-19 22:23:07-0500 [-] echo.EchoFactory starting on 8000
2012-11-19 22:23:07-0500 [-] Starting factory <echo.EchoFactory ...
2012-11-19 22:23:20-0500 [-] Received SIGTERM, shutting down.
2012-11-19 22:23:20-0500 [-] (TCP Port 8000 Closed)
2012-11-19 22:23:20-0500 [-] Stopping factory <echo.EchoFactory ...
2012-11-19 22:23:20-0500 [-] Main loop terminated.
2012-11-19 22:23:20-0500 [-] Server Shut Down.

To suppress daemonization and log to stdout, pass -n (--nodaemon). For a full list of twistd’s capabilities, run twistd --help or consult the manpage.

Without writing any code ourselves, we got free daemonization and logging. Running a service using the Twisted application infrastructure allows developers to skip writing boilerplate code for common server functionalities.

An alternative to the TAC-based system for running Twisted applications is the plugin system. While the TAC system makes it easy to register simple hierarchies of predefined services within an application configuration file, the plugin system makes it easy to register custom services as subcommands of the twistd utility and to extend the command-line interface to an application.

Using this system:

To extend a program using the Twisted plugin system, all you have to do is create objects that implement the IPlugin interface and put them in a particular location where the plugin system knows to look for them.

Having already converted our echo server to a Twisted application, transformation into a Twisted plugin is straightforward. Alongside the echo module from before, which contains the Echo protocol and EchoFactory definitions, we add a directory called twisted, containing a subdirectory called plugins containing our echo plugin definition. Graphically, the directory structure is:

echoproject/
├── echo.py
└── twisted
    └── plugins
        └── echo_plugin.py

Let’s make the port our echo service uses configurable through twistd. Example 6-4 shows the necessary logic.

Example 6-4. echo_plugin.py
from zope.interface import implements

from twisted.application.service import IServiceMaker
from twisted.application import internet
from twisted.plugin import IPlugin
from twisted.python import usage

from echo import EchoFactory

class Options(usage.Options):
    optParameters = [["port", "p", 8000, "The port number to listen on."]]

class EchoServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echo"
    description = "A TCP-based echo server."
    options = Options

    def makeService(self, options):
        """
        Construct a TCPServer from a factory defined in echo.py.
        """
        return internet.TCPServer(int(options["port"]), EchoFactory())

serviceMaker = EchoServiceMaker()

A service plugin needs a minimum of two components:

  1. A subclass of twisted.python.usage.Options, with a class variable optParameters describing each of the command-line options to the service.

    In our case, optParameters describes a single -p/--port configuration option, which has a default of 8000.

  2. An implementor of both IPlugin and IServiceMaker. This class implements a makeService method that passes the command-line configuration options to the service. It also defines the name and description of the service as displayed by twistd.

    In our case, as with the TAC implementation, we’ll create instances of the TCPServer service, but with a port pulled from the command-line options instead of hardcoding 8000.

With this plugin defined, if we run twistd from the top-level project directory our echo server will now show up as a server option in the output of twistd --help, and running twistd echo --port=1235 will start an echo server on port 1235.

More twistd Examples

twistd ships with many commands that make it easy to spin up simple services with zero lines of code. Here are some examples:

twistd web --port 8080 --path .

Run an HTTP server on port 8080, serving both static and dynamic content out of the current working directory. Visit http://localhost:8080 to see the directory listing.

twistd dns -v -p 5553 --hosts-file=hosts

Run a DNS server on port 5553, resolving domains out of a file called hosts in the format of /etc/hosts.

For example, say you’d like to run your own Twisted DNS resolver and are also trying to cut back on social media. Create a hosts file that resolves facebook.com, twitter.com, and reddit.com to localhost, 127.0.0.1:

127.0.0.1 facebook.com
127.0.0.1 twitter.com
127.0.0.1 reddit.com

Then run your twistd DNS resolver, configure your operating system to try that resolver first, and effectively disable your ability to view those sites.

A quick command-line way to prove that the resolver is working is to use the dig DNS lookup utility. First, query the default resolver, then query the twistd resolver:

$ dig +short twitter.com
199.59.150.7
199.59.148.10
199.59.150.39
$ dig @localhost -p 5553 +short twitter.com
127.0.0.1
sudo twistd conch -p tcp:2222

Run an ssh server on port 2222. ssh keys must be set up independently.

twistd mail -E -H localhost -d localhost=emails

Run an ESMTP POP3 server, accepting email for localhost and saving it to the emails directory.

I don’t know about you, but I get pretty excited by the networking power of these simple twistd one-liners.

More Practice and Next Steps

This chapter introduced the Twisted application infrastructure for configuring and deploying Twisted programs in a standardized fashion.

There are two main ways of deploying applications using this infrastructure: TAC files and plugins. TAC files are simpler but less extensible, making them ideal for simple server deployments that want to take advantage of Twisted’s built-in deployment features, like logging and daemonization. Plugins have a higher initial development cost but expose a clear API for extending your application. Plugins are ideal for applications that need a stable interface for third-party developers or more control over plugin discovery and loading.

The Twisted Core HOWTO provides an overview of the application framework and TAC files, as well as information about the plugin philosophy and twistd plugins specifically.

Twisted comes with a pluggable authentication system for servers called Twisted Cred, and a common use of the plugin system is to add authentication to an application. Twisted Cred is discussed in detail in Chapter 9.

  1. Converting a Twisted program into a TAC-based or plugin-based service follows a straightforward algorithm that you can practice on any of the servers we build in this book.

    Try converting the chat server from Example 2-5 to a Twisted application, and converting the nonblocking web server from Example 4-8 to a plugin-based service.

  2. All of the commands listed in twistd --help are plugins that you can browse in the Twisted source code at twisted/plugins/. Pick one and read through the service definition.