Chapter 2. Building Basic Clients and Servers

The best way to learn about the components of a Twisted application is to dive right into some examples. This chapter will introduce you to the reactor event loop, transports, and protocols through implementations of a few basic TCP servers and clients.

A TCP Echo Server and Client

Skim the code for the TCP echo server and client pair in Examples 2-1 and 2-2. The server’s job is to listen for TCP connections on a particular port and echo back anything it receives. The client’s job is to connect to the server, send it a message, receive a response, and terminate the connection.

Example 2-1. echoserver.py

from twisted.internet import protocol, reactor

class Echo(protocol.Protocol):
    def dataReceived(self, data):
        self.transport.write(data)

class EchoFactory(protocol.Factory):
    def buildProtocol(self, addr):
        return Echo()

reactor.listenTCP(8000, EchoFactory())
reactor.run()

Example 2-2. echoclient.py

from twisted.internet import reactor, protocol

class EchoClient(protocol.Protocol):
   def connectionMade(self):
       self.transport.write("Hello, world!")

   def dataReceived(self, data):
       print "Server said:", data
       self.transport.loseConnection()

class EchoFactory(protocol.ClientFactory):
   def buildProtocol(self, addr):
       return EchoClient()

   def clientConnectionFailed(self, connector, reason):
       print "Connection failed."
       reactor.stop()

   def clientConnectionLost(self, connector, reason):
       print "Connection lost."
       reactor.stop() 

reactor.connectTCP("localhost", 8000, EchoFactory())
reactor.run()

To test this pair of scripts, first run the server in one terminal with python echoserver.py. This will start a TCP server listening for connections on port 8000. Then run the client in a second terminal with python echoclient.py.

A transcript from the session looks like this:

    $ python echoserver.py # In Terminal 1

    $ python echoclient.py # In Terminal 2
    Server said: Hello, world!
    Connection lost.

Ta-da! You’ve just completed your first asynchronous, event-driven communication with Twisted. Let’s look at each of the components of these scripts in more detail.

Event-Driven Programming

The echo server and echo client are event-driven programs, and more generally Twisted is an event-driven networking engine. What does that mean?

In an event-driven program, program flow is determined by external events. It is characterized by an event loop and the use of callbacks to trigger actions when events happen. Contrast this structure with two other common models: single-threaded (synchronous) and multithreaded programming.

Figure 2-1 summarizes these three models visually by showing the work done by a program over time under each of them. The program has three tasks to complete, each of which blocks while waiting for I/O to finish. Time spent blocking on I/O is grayed out.

Figure 2-1. Comparing single-threaded, multithreaded, and event-driven program flow

In the single-threaded synchronous version of the program, tasks are performed serially. If one task blocks on I/O, all of the other tasks must also wait. Single-threaded programs are thus easy to reason about but can be unnecessarily slow.

In the multithreaded version, the three blocking tasks are performed in separate threads of control, which may run interleaved on one or many processors. This allows progress to be made by some threads while others are blocking on resources and is often more time-efficient than the analogous synchronous program. However, one has to write code that protects shared resources that could be accessed concurrently from multiple threads, which when implemented improperly can lead to notoriously subtle and painful threading bugs.

The event-driven version of the program interleaves the execution of the three tasks, but in a single thread of control. When performing I/O or other expensive operations, a callback is registered with an event loop, and then execution continues while the I/O completes. The callback describes how to handle an event once it has completed. The event loop polls for events and dispatches them as they arrive to the callbacks that are waiting for them. This allows the program to make progress without the use of additional threads.

Event-driven programs enjoy both the parallelism of multithreaded programs and the ease of reasoning of single-threaded programs.

The Reactor

The core of Twisted is the reactor event loop. The reactor knows about network, filesystem, and timer events. It waits on and demultiplexes these events and dispatches them to waiting event handlers. Twisted takes care of abstracting away platform-specific behavior and using the underlying nonblocking APIs correctly. Twisted presents a common interface to the various event sources so that responding to events anywhere in the network stack is easy.

The reactor essentially accomplishes the following:

    while True:
        timeout = time_until_next_timed_event()
        events = wait_for_events(timeout)
        events += timed_events_until(now())
        for event in events:
            event.process()

In our echo server and client from Examples 2-1 and 2-2, the reactor’s listenTCP and connectTCP methods take care of registering callbacks with the reactor to get notified when data is available to read from a TCP socket on port 8000.

After those callbacks have been registered, we start the reactor’s event loop with reactor.run. Once running, the reactor will poll for and dispatch events forever or until reactor.stop is called.

Transports

A transport represents the connection between two endpoints communicating over a network. Transports describe connection details: for example, is this connection stream-oriented (like TCP) or datagram-oriented (like UDP)? TCP, UDP, Unix sockets, and serial ports are examples of transports. Transports implement the ITransport interface, which has the following methods:

write: Write data to the physical connection in a nonblocking manner.
writeSequence: Write a list of strings to the physical connection. Useful when working with line-oriented protocols.
loseConnection: Write all pending data and then close the connection.
getPeer: Get the remote address of the connection.
getHost: Like getPeer, but returns the address of the local side of the connection.

In the echo server and client examples from earlier, the two endpoints send each other data using their transport’s write method. The client terminates the TCP connection after receiving a response from the server by calling loseConnection.

Protocols

Protocols describe how to process network events asynchronously. Twisted maintains implementations for many popular application protocols, including HTTP, Telnet, DNS, and IMAP. Protocols implement the IProtocol interface, which has the following methods:

makeConnection: Create a connection between two endpoints across a transport.
connectionMade: Called when a connection to another endpoint is made.
dataReceived: Called when data is received across a transport.
connectionLost: Called when the connection is shut down.

In our echo server, we create our own Echo protocol by subclassing protocol.Protocol. To echo data back to the client, we take the data received from the client and simply write it back out through the transport in dataReceived.

In the echo client, we create our own EchoClient protocol by subclassing protocol.Protocol. The call to connectTCP creates a TCP connection to the server on port 8000 and registers callbacks for the various stages of the connection. For example, a callback is registered to invoke dataReceived when new data is available on the transport. Once the connection is established, we write data out to the server through the transport in connectionMade. When we receive data back from the server in dataReceived, we print that data and close the TCP connection.

Protocol Factories

A new instance of our Echo protocol class is instantiated for every connection and goes away when the connection terminates. This means that persistent configuration information is not saved in the protocol.

Persistent configuration information is instead kept in an EchoFactory class, which inherits from protocol.Factory in the server and protocol.ClientFactory in the client. A factory’s buildProtocol method creates a protocol for each new connection, which gets passed to the reactor to register callbacks.

Decoupling Transports and Protocols

A major design decision in Twisted is that transports and protocols are completely decoupled. This decoupling makes it easy for many protocols to reuse the same type of transport. It is also hugely important for testing: to test a protocol implementation you can have it use a mock transport that simply writes data to a string for inspection. You’ll experience this first-hand in Chapter 11.

A TCP Quote Server and Client

Let’s reiterate some of the core ideas discussed in the previous sections with a slightly more complicated quote exchange service.

The quote server in Example 2-3 is seeded with an initial quote. Upon receiving a quote from a client, it will send the client its current quote and store the client’s quote to share with the next client. It also keeps track of the number of concurrent client connections.

The client in Example 2-4 creates several TCP connections, each of which exchanges a quote with the server.

Example 2-3. quoteserver.py

from twisted.internet.protocol import Factory
from twisted.internet import reactor, protocol

class QuoteProtocol(protocol.Protocol):
    def __init__(self, factory):
        self.factory = factory

    def connectionMade(self):
        self.factory.numConnections += 1

    def dataReceived(self, data):
        print "Number of active connections: %d" % (
            self.factory.numConnections,)
        print "> Received: ``%s''\n>  Sending: ``%s''" % (
            data, self.getQuote())
        self.transport.write(self.getQuote())
        self.updateQuote(data)

    def connectionLost(self, reason):
        self.factory.numConnections -= 1

    def getQuote(self):
        return self.factory.quote

    def updateQuote(self, quote):
        self.factory.quote = quote

class QuoteFactory(Factory):
    numConnections = 0

    def __init__(self, quote=None):
        self.quote = quote or "An apple a day keeps the doctor away"

    def buildProtocol(self, addr):
        return QuoteProtocol(self)

reactor.listenTCP(8000, QuoteFactory())
reactor.run()

Example 2-4. quoteclient.py

from twisted.internet import reactor, protocol

class QuoteProtocol(protocol.Protocol):
    def __init__(self, factory):
        self.factory = factory

    def connectionMade(self):
        self.sendQuote()

    def sendQuote(self):
        self.transport.write(self.factory.quote)

    def dataReceived(self, data):
        print "Received quote:", data
        self.transport.loseConnection()

class QuoteClientFactory(protocol.ClientFactory):
    def __init__(self, quote):
        self.quote = quote

    def buildProtocol(self, addr):
        return QuoteProtocol(self)

    def clientConnectionFailed(self, connector, reason):
        print 'connection failed:', reason.getErrorMessage()
        maybeStopReactor()

    def clientConnectionLost(self, connector, reason):
        print 'connection lost:', reason.getErrorMessage()
        maybeStopReactor()

def maybeStopReactor():
    global quote_counter
    quote_counter -= 1
    if not quote_counter:
        reactor.stop()

quotes = [
    "You snooze you lose",
    "The early bird gets the worm",
    "Carpe diem"
]
quote_counter = len(quotes)

for quote in quotes:
    reactor.connectTCP('localhost', 8000, QuoteClientFactory(quote))
reactor.run()

Start the server in one terminal with python quoteserver.py and then run the client in another terminal with python quoteclient.py. Transcripts from these sessions will look something like the following—note that because this communication is asynchronous, the order in which connections are made and terminated may vary between runs:

$ python quoteserver.py 
Number of active connections: 2
> Received: ``You snooze you lose''
>  Sending: ``An apple a day keeps the doctor away.''
Number of active connections: 2
> Received: ``The early bird gets the worm''
>  Sending: ``You snooze you lose''
Number of active connections: 3
> Received: ``Carpe diem''
>  Sending: ``The early bird gets the worm''

$ python quoteclient.py
Received quote: The early bird gets the worm
Received quote: You snooze you lose
connection lost: Connection was closed cleanly.
connection lost: Connection was closed cleanly.
Received quote: Carpe diem
connection lost: Connection was closed cleanly.

This quote server and client pair highlight some key points about client/server communication in Twisted:

Persistent protocol state is kept in the factory.
Because a new instance of a protocol class is created for each connection, protocols can’t contain persistent state; that information must instead be stored in a protocol factory. In the echo server, the number of current connections is stored in numConnections in QuoteFactory.
It is common for a factory’s buildProtocol method to do nothing beyond return an instance of a Protocol. For that simple case, Twisted provides a shortcut: instead of implementing buildProtocol, just define a protocol class variable for the factory; the default implementation of buildProtocol will take care of creating an instance of your Protocol and setting a factory attribute on the protocol pointing back to the factory (making it easy for protocol instances to access the shared state stored in the factory).
For example, you could get rid of QuoteProtocol’s __init__ method and QuoteFactory could be rewritten as:
```
class QuoteFactory(Factory):
    numConnections = 0
    protocol = QuoteProtocol

    def __init__(self, quote=None):
        self.quote = quote or "An apple a day keeps the doctor away."
```
This is a common idiom in Twisted programs, so keep an eye out for it!
Protocols can retrieve the reason why a connection was terminated.
The reason is passed as an argument to clientConnectionLost and clientConnectionFailed. If you run quoteclient.py without a server waiting for its connections, you’ll get:
```
$ python quoteclient.py
connection failed: Connection was refused by other side...
connection failed: Connection was refused by other side...
connection failed: Connection was refused by other side...
```
Clients can make make many simultaneous connections to a server.
To do this, simply call connectTCP repeatedly, as was done in the quote client before starting the reactor.

Lastly, our use of maybeStopReactor is hinting at a general client design issue of how to determine when all of the connections you wanted to make have terminated (often so that you can shut down the reactor). maybeStopReactor gets the job done here, but we’ll explore a more idiomatic way of accomplishing this using objects called Deferreds later in the next book.

Protocol State Machines

Protocols typically have different states and can be expressed in client and server code as a state machine. Example 2-5 is a chat server that implements a small state machine. It also subclasses the LineReceiver class, which is a convenience class that makes it easy to write line-based protocols. When using LineReceiver, a client should send messages with sendLine and a server should process received messages in lineReceived.

Example 2-5. chatserver.py

from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor

class ChatProtocol(LineReceiver):
    def __init__(self, factory):
        self.factory = factory
        self.name = None
        self.state = "REGISTER"

    def connectionMade(self):
        self.sendLine("What's your name?")

    def connectionLost(self, reason):
        if self.name in self.factory.users:
            del self.factory.users[self.name]
            self.broadcastMessage("%s has left the channel." % (self.name,))

    def lineReceived(self, line):
        if self.state == "REGISTER":
            self.handle_REGISTER(line)
        else:
            self.handle_CHAT(line)

    def handle_REGISTER(self, name):
        if name in self.factory.users:
            self.sendLine("Name taken, please choose another.")
            return
        self.sendLine("Welcome, %s!" % (name,))
        self.broadcastMessage("%s has joined the channel." % (name,))
        self.name = name
        self.factory.users[name] = self
        self.state = "CHAT"

    def handle_CHAT(self, message):
        message = "<%s> %s" % (self.name, message)
        self.broadcastMessage(message)

    def broadcastMessage(self, message):
        for name, protocol in self.factory.users.iteritems():
            if protocol != self:
                protocol.sendLine(message)

class ChatFactory(Factory):
    def __init__(self):
        self.users = {}

    def buildProtocol(self, addr):
        return ChatProtocol(self)

reactor.listenTCP(8000, ChatFactory())
reactor.run()

Run the chat server with python chatserver.py. You can then connect to the chat server with the telnet utility. Example 2-6 shows a sample transcript of two users chatting.

Example 2-6. Using the chat server

$ telnet localhost 8000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
What's your name?
Jessica
Welcome, Jessica!
Adam has joined the channel.
Hey Adam!
<Adam> How's it going?
I've got a working Twisted chat server now, so pretty great!
^]
telnet> quit
Connection closed.

$ telnet localhost 8000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
What's your name?
Adam
Welcome, Adam!
<Jessica> Hey Adam!
How's it going?
<Jessica> I've got a working Twisted chat server now, so pretty great!
Jessica has left the channel.

Tip

To terminate a telnet connection, hold down the Control key and press the right-bracket key. That will drop you to a telnet> prompt; from there, type quit and press the Return key to terminate the connection.

ChatProtocol has two states, REGISTER and CHAT. lineReceived calls the correct handler based on the current state of the protocol.

Note that the persistent protocol state—the dictionary of connected users—is stored in ChatFactory.

Tip

Avoid mixing application-specific logic with protocol code. This will make testing your protocol and application easier and facilitate protocol reuse.

As you can see, the servers and clients for the echo, quote, and chat services are all structurally very similar. The shared recipe is:

Define a protocol class, subclassing twisted.internet.protocol.Protocol for arbitrary data or twisted.protocols.basic.LineReceiver for line-oriented protocols.
Define a factory class, subclassing twisted.internet.protocol.Factory for servers and twisted.internet.protocol.ClientFactory for clients. That factory creates instances of the protocol and stores state shared across protocol instances.
Clients use reactor.connectTCP to initiate a connection to a server. Invoking connectTCP registers callbacks with the reactor to notify your protocol when new data has arrived across a socket for processing. Servers use reactor.listenTCP to listen for and respond to client connections.
Communication doesn’t start until reactor.run is called, which starts the reactor event loop.

More Practice and Next Steps

This chapter introduced the core components of Twisted servers and clients: the reactor, transports, protocols, and protocol factories. Because a new instance of a protocol class is created for each connection, persistent state is kept in a protocol factory. Protocols and transports are decoupled, which makes transport reuse and protocol testing easy.

The Twisted Core examples directory has many additional examples of basic servers and clients, including implementations for UDP and SSL.

The Twisted Core HOWTO index has an extended “Twisted from Scratch” tutorial that builds a finger service from scratch.

One real-world example of building a protocol in Twisted is AutobahnPython, a WebSockets implementation.

Twisted has been developing a new higher-level endpoints API for creating a connection between a client and server. The endpoints API wraps lower-level APIs like listenTCP and connectTCP, and provides greater flexibility because it decouples constructing a connection from initiating use of the connection, allowing parameterization of the endpoint. You’ll start seeing the endpoints API in more documentation and examples through the next couple of Twisted releases, so keep an eye out for it. You can read more about that at the Twisted endpoints API page.