When it comes to IPC, we face a range of choices that can at first seem bewildering. In later chapters that describe each IPC facility, we include sections that compare each facility against other similar facilities. In the following pages, we consider a number of general points that may determine the choice of IPC facility.
In order to access an IPC object, a process must have some means of identifying the object, and once the object has been “opened,” the process must use some type of handle to refer to the open object. Table 43-1 summarizes these properties for the various types of IPC facilities.
Table 43-1. Identifiers and handles for various types of IPC facilities
Facility type | Name used to identify object | Handle used to refer to object in programs |
|---|---|---|
Pipe | no name | file descriptor |
pathname | file descriptor | |
pathname | file descriptor | |
IP address + port number | file descriptor | |
System V IPC key | System V IPC identifier | |
System V IPC key | System V IPC identifier | |
System V IPC key | System V IPC identifier | |
POSIX IPC pathname | mqd_t (message queue descriptor) | |
POSIX IPC pathname | sem_t * (semaphore pointer) | |
no name | sem_t * (semaphore pointer) | |
POSIX IPC pathname | file descriptor | |
Anonymous mapping | no name | none |
Memory-mapped file | pathname | file descriptor |
flock() lock | pathname | file descriptor |
fcntl() lock | pathname | file descriptor |
There are functional differences between the various IPC facilities that can be relevant in determining which facility to use. We begin by summarizing the differences between data-transfer facilities and shared memory:
Data-transfer facilities involve read and write operations, with transferred data being consumable by just one reader process. Flow control between writer and reader, as well as synchronization (so that a reader is blocked when trying to read data from a facility that is currently empty) is automatically handled by the kernel. This model fits well with many application designs.
Other application designs more naturally suit a shared-memory model. Shared memory allows one process to make data visible to any number of other processes sharing the same memory region. Communication “operations” are simple—a process can access data in shared memory in the same manner as it accesses any other memory in its virtual address space. On the other hand, the need to handle synchronization (and perhaps flow control) can add to the complexity of a shared-memory design. This model fits well with application designs that need to maintain shared state (e.g., a shared data structure).
With respect to the various data-transfer facilities, the following points are worth noting:
Some data-transfer facilities transfer data as a byte stream (pipes, FIFOs, and stream sockets); others are message-oriented (message queues and datagram sockets). Which approach is preferable depends on the application. (An application can also impose a message-oriented model on a byte-stream facility, by using delimiter characters, fixed-length messages, or message headers that encode the length of the total message; see Section 44.8.)
A distinctive feature of System V and POSIX message queues, compared with other data-transfer facilities, is the ability to assign a numeric type or priority to a message, so that messages can be delivered in a different order from that in which they were sent.
Pipes, FIFOs, and sockets are implemented using file descriptors. These IPC facilities all support a range of alternative I/O models that we describe in Chapter 63: I/O multiplexing (the select() and poll() system calls), signal-driven I/O, and the Linux-specific epoll API. The primary benefit of these techniques is that they allow an application to simultaneously monitor multiple file descriptors to see whether I/O is possible on any of them. By contrast, System V message queues don’t employ file descriptors and don’t support these techniques.
On Linux, POSIX message queues are also implemented using file descriptors and support the alternative I/O techniques described above. However, this behavior is not specified in SUSv3, and is not supported on most other implementations.
POSIX message queues provide a notification facility that can send a signal to a process, or instantiate a new thread, when a message arrives on a previously empty queue.
UNIX domain sockets provide a feature that allows a file descriptor to be passed from one process to another. This allows one process to open a file and make it available to another process that otherwise might not be able to access the file. We briefly describe this feature in Passing File Descriptors.
UDP (Internet domain datagram) sockets allow a sender to broadcast or multicast a message to multiple recipients. We briefly describe this feature in TCP Versus UDP.
With respect to process-synchronization facilities, the following points are worth noting:
Record locks placed using fcntl() are considered to be owned by the process placing the lock. The kernel uses this ownership property to detect deadlocks (situations where two or more processes are holding locks that block each other’s further lock requests). If a deadlock situation occurs, the kernel denies the lock request of one of the processes, returning an error from the fcntl() call to indicate that a deadlock occurred. System V and POSIX semaphores don’t have an ownership property; no deadlock detection occurs for semaphores.
Record locks placed using fcntl() are automatically released when the process that owns the locks terminates. System V semaphores provide a similar feature in the form of an “undo” feature, but this feature is not reliable in all circumstances (Semaphore Undo Values). POSIX semaphores don’t provide an analog of this feature.
Of all of the IPC methods shown in Figure 43-1, only sockets permit processes to communicate over a network. Sockets are generally used in one of two domains: the UNIX domain, which allows communication between processes on the same system, and the Internet domain, which allows communication between processes on different hosts connected via a TCP/IP network. Often, only minor changes are required to convert a program that uses UNIX domain sockets into one that uses Internet domain sockets, so an application that is built using UNIX domain sockets can be made network-capable with relatively little effort.
Modern UNIX implementations support most of the IPC facilities shown in Figure 43-1. However, the POSIX IPC facilities (message queues, semaphores, and shared memory) are not quite as widely available as their System V IPC counterparts, especially on older UNIX systems. (An implementation of POSIX message queues and full support for POSIX semaphores have appeared on Linux only in the 2.6.x kernel series.) Therefore, from a portability point of view, System V IPC may be preferable to POSIX IPC.
The System V IPC facilities were designed independently of the traditional UNIX I/O model, and consequently suffer a few peculiarities that make their programming interfaces more complicated to use. The corresponding POSIX IPC facilities were designed to address these problems. The following points are of particular note:
The System V IPC facilities are connectionless. These facilities provide no notion of a handle (like a file descriptor) referring to an open IPC object. In later chapters, we’ll sometimes talk of “opening” a System V IPC object, but this is really just shorthand to describe the process of obtaining a handle to refer to the object. The kernel does not record the process as having “opened” the object (unlike other types of IPC objects). This means that the kernel can’t maintain a reference count of the number of processes that are currently using an object. Consequently, it can require additional programming effort for an application to be able to know when an object can safely be deleted.
The programming interfaces for the System V IPC facilities are inconsistent with the traditional UNIX I/O model (they use integer key values and IPC identifiers instead of pathnames and file descriptors). The programming interfaces are also overly complex. This last point applies particularly to System V semaphores (refer to Disadvantages of System V Semaphores and Comparisons with Other Synchronization Techniques).
By contrast, the kernel counts open references for POSIX IPC objects. This simplifies decisions about when an object can be deleted. Furthermore, the POSIX IPC facilities provide an interface that is simpler and more consistent with the traditional UNIX model.
The second column of Table 43-2 summarizes an important characteristic of each type of IPC object: the permissions scheme that governs which processes can access the object. The following list adds some details on the various schemes:
For some IPC facilities (e.g., FIFOs and sockets), object names live in the file system, and accessibility is determined according to the associated file permissions mask, which specifies permissions for owner, group, and other (File Permissions). Although System V IPC objects don’t reside in the file system, each object has an associated permissions mask whose semantics are similar to those for files.
A few IPC facilities (pipes, anonymous memory mappings) are marked as being accessible only by related processes. Here, related means related via fork(). In order for two processes to access the object, one of them must create the object and then call fork(). As a consequence of the fork(), the child process inherits a handle referring to the object, allowing both processes to share the object.
The accessibility of a POSIX unnamed semaphore is determined by the accessibility of the shared memory region containing the semaphore.
In order to place a lock on a file, a process must have a file descriptor referring to the file (i.e., in practice, it must have permission to open the file).
There are no restrictions on accessing (i.e., connecting or sending a datagram to) an Internet domain socket. If necessary, access control must be implemented within the application.
Table 43-2. Accessibility and persistence for various types of IPC facilities
Facility type | Accessibility | Persistence |
|---|---|---|
Pipe | only by related processes | process |
permissions mask | process | |
permissions mask | process | |
by any process | process | |
permissions mask | kernel | |
permissions mask | kernel | |
permissions mask | kernel | |
permissions mask | kernel | |
permissions mask | kernel | |
permissions of underlying memory | depends | |
permissions mask | kernel | |
Anonymous mapping | only by related processes | process |
Memory-mapped file | permissions mask | file system |
open() of file | process | |
fcntl() file lock | open() of file | process |
The term persistence refers to the lifetime of an IPC object. (Refer to the third column of Table 43-2.) We can distinguish three types of persistence:
Process persistence: A process-persistent IPC object remains in existence only as long as it is held open by at least one process. If the object is closed by all processes, then all kernel resources associated with the object are freed, and any unread data is destroyed. Pipes, FIFOs, and sockets are examples of IPC facilities with process persistence.
The persistence of a FIFO’s data is not the same as the persistence of its name. A FIFO has a name in the file system that persists even after all file descriptors referring to the FIFO have been closed.
Kernel persistence: A kernel-persistent IPC object exists until either it is explicitly deleted or the system is shut down. The lifetime of the object is independent of whether any process holds the object open. This means that, for example, one process can create an object, write data to it, and then close it (or terminate). At a later point, another process can open the object and read the data. Examples of facilities with kernel persistence are System V IPC and POSIX IPC. We exploit this property in the example programs that we present when describing these facilities in later chapters: for each facility, we implement separate programs that create an object, delete an object, and perform communication or synchronization.
File-system persistence: An IPC object with file-system persistence retains its information even when the system is rebooted. The object exists until it is explicitly deleted. The only type of IPC object that demonstrates file-system persistence is shared memory based on a memory-mapped file.
In some circumstances, different IPC facilities may show notable differences in performance. However, in later chapters, we generally refrain from making performance comparisons, for the following reasons:
The performance of an IPC facility may not be a significant factor in the overall performance of an application, and it may not be the only factor in determining the choice of an IPC facility.
The relative performance of the various IPC facilities may vary across UNIX implementations or between different versions of the Linux kernel.
Most importantly, the performance of an IPC facility will vary depending on the precise manner and environment in which it is used. Relevant factors include the size of the data units exchanged in each IPC operation, the amount of unread data that may be outstanding on the IPC facility, whether or not a process context switch is required for each unit of data exchanged, and other load on the system.
If IPC performance is crucial, there is no substitute for application-specific benchmarks run under an environment that matches the target system. To this end, it may be worth writing an abstract software layer that hides details of the IPC facility from the application and then testing performance when different IPC facilities are substituted underneath the abstract layer.