Modern Systems Programming with Scala Native

Implementing HTTP

How should we go about implementing HTTP on top of our TCP codebase? Although there are a few moving parts, the fundamental requirement is that we need to implement a handleConnection() function that does four things:

Generate an HTTP request matching the format described in the previous section.
Transmit the request.
Receive an HTTP response.
Decode the HTTP response into a useful form.

Transmitting and receiving the request can be performed with read() and write(), which we looked at in the first chapter. The tricky part is in representing HTTP requests and responses as Scala objects, and in encoding and decoding those to the wire protocol we described in the previous section. If we model the request and response as regular Scala case classes, they could look something like this:

HTTPClient/httpclient/HTTPClient.scala

	case class HttpRequest(
	method:String,
	uri:String,
	headers:collection.Map[String, String],
	body:String)
	case class HttpResponse(
	code:Int,
	headers:collection.Map[String, String],
	body:String)

First, we’ll write out the request in three steps:

Write out the header line with the method and the URI.
Write out each header line, followed by an empty line.
If a body is present, write it out.

We can implement these steps in a few compact bits of Scala code:

HTTPClient/httpclient/HTTPClient.scala

	def writeRequestLine(socket_fd:Ptr[FILE], method:CString,
	uri:CString):Unit = {
	stdio.fprintf(socket_fd, c"%s %s %s\r\n", method, uri, c"HTTP/1.1")
	}

	def writeHeader(socket_fd:Ptr[FILE], key:CString, value:CString):Unit = {
	stdio.fprintf(socket_fd, c"%s: %s\r\n", key, value)
	}

	def writeBody(socket_fd:Ptr[FILE], body:CString):Unit = {
	stdio.fputs(body, socket_fd)
	}

	def writeRequest(socket_fd:Ptr[FILE], request:HttpRequest):Unit = {
	Zone { implicit z =>
	writeRequestLine(socket_fd, toCString(request.method),
	toCString(request.uri))
	for ( (key, value) <- request.headers) {
	writeHeader(socket_fd, toCString(key), toCString(value))
	}
	stdio.fputs(c"\n", socket_fd)
	writeBody(socket_fd, toCString(request.body))
	}
	}

Reading the response is going to be a bit trickier. As you may remember from Chapter 1, The Basics: Input and Output, any time we read data, we need to have space allocated to hold it, whether we use read(), fgets(), fscanf(), or other related I/O function; but in our case, we don’t know how big the HTTP response is going to be.

Many binary protocols solve this problem elegantly: they define a message header of a fixed-size number of bytes and store the length of the message body in a header field. With this strategy, a program that wants to read such a message can always read the header into a fixed-size chunk of memory, and each time it gets a header, it knows how many additional bytes to allocate and read for the rest of the message. Lots of protocols use this strategy, including IP itself, UDP, as well as application-level protocols like HTTP/2 and FastCGI.

Unfortunately for us, HTTP1.1 is a text-based protocol; although it has distinct headers and bodies, we cannot know the length of the header ahead of time. Since a header can have any number of key/value pairs, we instead have to scan through the message until we find the empty line that indicates the boundary between the header and the body. We could use a few different strategies, but the simplest is something like this:

Read a line of input and validate that it’s an HTTP response status line.
Read another line of input. If it’s not blank, it’s a header. Validate it and repeat. If it’s blank, we have a complete header. Proceed to step 3.
Check whether we have a content-length in the header. If we have a content-length, we can read exactly that number of bytes. Otherwise, raise an error (FN: chunked transfer, http 1.0, and so on).

We can implement this algorithm in Scala step by step, like this:

HTTPClient/httpclient/HTTPClient.scala

	def parseStatusLine(line:CString):Int = {
	println("parsing status")
	val protocol_ptr = stackalloc[Byte](64)
	val code_ptr = stackalloc[Int]
	val desc_ptr = stackalloc[Byte](128)
	val scan_result = stdio.sscanf(line,
	c"%s %d %s\n",
	protocol_ptr, code_ptr, desc_ptr)
	if (scan_result < 3) {
	throw new Exception("bad status line")
	} else {
	val code = !code_ptr
	return code
	}
	}

	def parseHeaderLine(line:CString):(String, String) = {
	val key_buffer = stackalloc[Byte](64)
	val value_buffer = stackalloc[Byte](64)
	stdio.printf(c"about to sscanf line: '%s'\n", line)
	val scan_result = stdio.sscanf(line, c"%s %s\n", key_buffer, value_buffer)
	if (scan_result < 2) {
	throw new Exception("bad header line")
	} else {
	val key_string = fromCString(key_buffer)
	val value_string = fromCString(value_buffer)
	return (key_string, value_string)
	}
	}

	def readResponse(socket_fd:Ptr[FILE]):HttpResponse = {
	val line_buffer = stdlib.malloc(4096)
	println("reading status line?")
	var read_result = stdio.fgets(line_buffer, 4096, socket_fd)
	val code = parseStatusLine(line_buffer)
	var headers = mutable.Map[String, String]()
	println("reading first response header")
	read_result = stdio.fgets(line_buffer, 4096, socket_fd)
	var line_length = string.strlen(line_buffer)

	while (line_length > 2) {
	val (k,v) = parseHeaderLine(line_buffer)
	println(s"${(k,v)}")
	headers(k) = v
	println("reading header")
	read_result = stdio.fgets(line_buffer, 4096, socket_fd)
	line_length = string.strlen(line_buffer)
	}

	val content_length = if (headers.contains("Content-Length:")) {
	println("saw content-length")
	headers("Content-Length:").toInt
	} else {
	65535
	}
	val body_buffer = stdlib.malloc(content_length + 1)
	val body_read_result = stdio.fread(body_buffer, 1, content_length,
	socket_fd)
	val body_length = string.strlen(body_buffer)
	if (body_length != content_length) {
	println("""Warning: saw ${body_length} bytes, but expected
	${content_length}""")
	}
	val body = fromCString(body_buffer)
	return HttpResponse(code, headers, body)
	}

For a full implementation of HTTP, we’d want to provide readRequest() and writeResponse() as well; I’ll omit the implementation here, but they’re present in the code for this chapter, and we’ll use them later on when we implement an HTTP server. For now, we can just stitch together our writeRequest() and readResponse() with our makeConnection() and handleConnection() from earlier in this chapter. We’ll slightly modify our main() function to take a URL string from the command line, make the request, and print out the response. We’ll also supply a few headers in our request. The standard is vague, but most servers require a User-agent header to return a valid response:

HTTPClient/httpclient/HTTPClient.scala

	def main(args: Array[String]):Unit = {
	if (args.length != 3) {
	println(s"${args.length} {args}")
	println("Usage: ./tcp_test [address] [port] [path]")
	return ()
	}

	Zone { implicit z =>
	val address = toCString(args(0))
	val host = args(0)
	val port = toCString(args(1))
	val path = args(2)
	stdio.printf(c"looking up address: %s port: %s\n", address, port)

	val sock = makeConnection(address, port)
	handleConnection(sock,host,path)
	}
	}

Now, we’re ready to test it!