All the programs we wrote in the first part of the book used blocking I/O operations exclusively; whether we were reading from a file, or writing to a TCP socket, we always stopped the progress of our program until the I/O system call returned. And in the last two chapters, we used multiprocess-based concurrency techniques to allow our programs to keep operating while waiting for an incoming socket, or other long-running system calls. However, the benchmarks for our fork-based web server also pointed at the limitations of this technique, especially for programs that need to handle thousands of simultaneous network connections at once.
Traditional UNIX operating systems provided alternative techniques for nonblocking I/O in the form of two system calls: poll() and select(). Both allow a program to poll, or check the readiness of, a file handle for reading or writing; thus, instead of starting a call to read or write data and blocking, we can wait until we know that the call can be completed and—if we are entirely disciplined—ensure that our program never blocks.
When properly executed, this pattern can lead to dramatic performance improvements, since all of the time spent waiting on I/O can be spent on other tasks. However, the traditional implementations of poll() and select() also had to deal with some severe bottlenecks: since the set of file handles to poll is kept in the user-space processes’ memory, the OS kernel has to walk through and check each handle, one at a time. In other words, the amount of time required to process a call to poll() or select() scales in proportion to the number of file handles. This is fine for programs like our stream-merging code that might have to handle only a few dozen files, but it starts to impose a very noticeable overhead on a busy web server with thousands of simultaneous connections.
The other issue is program complexity. Not only do we have to find something to do while we wait for I/O to happen, but we also have to track all the potential pending I/O requests that have yet to complete and react to them accordingly. These constraints generally lead to an architecture known as an event loop, in which a program continuously checks for new things to do, while the programmer defines tasks as callback functions that can be executed at some point in the future. Although this is a powerful programming style, it can also be tricky to get right, especially if you have to write everything from scratch.
The good news is that starting in the early 2000s, operating systems began to introduce new interfaces for high-performance asynchronous I/O with the capability to handle tens of thousands of sockets (or more) at once. The bad news is that the major OSs implemented radically different, incompatible interfaces. Linux gave us epoll,[32] FreeBSD and Mac OS X both implemented kqueue,[33] and Windows implemented I/O Completion Ports.[34] Although projects that embrace these capabilities may perform ten times or more better than their predecessors, serious issues with portability and complexity have made them challenging for developers to adopt.
Fortunately, a variety of open-source libraries have emerged to fill the gap, wrapping the complexity and variability of each OS-specific interface behind a consistent (if not necessary simple) surface. Most notably, libuv—extracted from the node.js server-side JavaScript engine—provides an excellent, portable wrapper for all kinds of high-performance I/O tasks, and its C API is exceedingly well documented.[35] To make use of libuv, though, we’ll need to extend Scala Native’s capability with C libraries, as well as create function pointers for libuv’s celebrated callback programming style.