In the last part of this chapter, we’re going to implement a usable subset of the Hypertext Transfer Protocol (HTTP). HTTP is indisputably the most important application protocol on the internet. Not only does it power the websites we use every day, but it’s rapidly becoming the universal protocol for all kinds of programs to request data from one another, via so-called RESTful APIs.
What does HTTP provide that TCP doesn’t? First, it defines a predictable request/response workflow, with well-defined message boundaries. Second, every request must refer to a specific resource, or object, named by a Uniform Resource Identifier, or URI. Third, each request must use one of eight methods on the resource, such as GET, POST, PUT, or DELETE, with well-defined semantics. Finally, HTTP transmits all of the above metadata in a message header, but allows any data format to be used for the content of either the request or the response—not just HTML or JSON!
Even though HTTP was originally designed only to deliver hypertext documents to web browsers, its generic and broadly supported capabilities have made it an essential block for all sorts of distributed systems and applications.
The standards body that governs the web, the W3C,[21] continues to promulgate extensions of the basic protocol, and even the giant tech firms that build web browsers can struggle to implement all of the standards correctly. But the core of HTTP is surprisingly simple and straightforward to implement.
An HTTP message contains either a request line or a status line, line-delimited headers, and optionally a body. The newlines are in the carriage-return line-feed style, which is written like \r\n in Scala and C strings. There is a single empty line after the last header. To see this in practice, let’s make a simple request. Instead of netcat, we’ll use the command-line utility curl:
| | $ curl -v www.example.com |
| | * Rebuilt URL to: www.example.com/ |
| | * Trying 93.184.216.34... |
| | * TCP_NODELAY set |
| | * Connected to www.example.com (93.184.216.34) port 80 (#0) |
| | > GET / HTTP/1.1 |
| | > Host: www.example.com |
| | > User-Agent: curl/7.54.0 |
| | > Accept: */* |
| | > |
| | < HTTP/1.1 200 OK |
| | < Cache-Control: max-age=604800 |
| | < Content-Type: text/html; charset=UTF-8 |
| | ... |
| | < Content-Length: 1270 |
| | < |
| | <!doctype html> |
| | <html> |
| | <head> |
| | <title>Example Domain</title> |
| | <meta charset="utf-8" /> |
| | ... |
| | </head> |
| | |
| | <body> |
| | <div> |
| | <h1>Example Domain</h1> |
| | <p>This domain is established to be used for illustrative examples ... |
| | domain in examples without prior coordination or asking for permission.</p> |
| | <p><a href="http://www.iana.org/domains/example">More information...</a></p> |
| | </div> |
| | </body> |
| | </html> |
| | * Connection #0 to host www.example.com left intact |
That’s a lot of output, but it’s a great representation of an actual HTTP exchange. Normally curl just prints out the HTTP response body, which in this case is the HTML content at the bottom, but in -v, or verbose, mode it’s also printing out the request headers prefixed by >, the response headers prefixed by <, and TCP connection status prefixed by \*. The three headers are Host, indicating the intended recipient of the request; User-Agent, identifying the program making the request; and Accept, indicating the response formats that may be received. \*/\* means any type the server prefers.
The response looks pretty similar, but instead of a request line, it has a status line. The status line consists of the HTTP version again, a response code, and a text description of the response code. You’ve probably seen response codes before if you’ve used the web at all: common codes include 200 OK, 301 moved permanently, 404 not found, and the dreaded 500 internal server error. The response has a few more headers; we won’t go through them all, but the most important ones are Content-Type, indicating the format of the response body, and Content-Length, indicating the length of the body in bytes. The other interesting header is Connection: keep-alive, which tells us that the client, if it wishes, may keep the TCP connection open and make another request, instead of closing the connection and reconnecting. This can be very helpful for high-performance HTTP clients!
Now, a lot more can go on in an HTTP exchange than what we’ve seen here, but it all falls within this framework. Most of the subtleties consist of choosing the correct combination of headers and data format for a specific server. The good news for us is that the protocol specifies relatively little of these requirements; for better or for worse, the burden falls on the server provider to document requirements, and client developer to implement them. As we go on, I’ll make a note of a few general best-practices that are usually worth following.