Location of Shared Memory in Virtual Memory

In Memory Layout of a Process, we considered the layout of the various parts of a process in virtual memory. It is useful to revisit this topic in the context of attaching System V shared memory segments. If we follow the recommended approach of allowing the kernel to choose where to attach a shared memory segment, then (on the x86-32 architecture) the memory layout appears as shown in Figure 48-2, with the segment being attached in the unallocated space between the upwardly growing heap and the downwardly growing stack. To allow space for heap and stack growth, shared memory segments are attached starting at the virtual address 0x40000000. Mapped mappings (Chapter 49) and shared libraries (Chapter 41 and Chapter 42) are also placed in this area. (There is some variation in the default location at which shared memory mappings and memory segments are placed, depending on the kernel versions and the setting of the process’s RLIMIT_STACK resource limit.)

Note

The address 0x40000000 is defined as the kernel constant TASK_UNMAPPED_BASE. It is possible to change this address by defining this constant with a different value and rebuilding the kernel.

A shared memory segment (or memory mapping) can be placed at an address below TASK_UNMAPPED_BASE, if we employ the unrecommended approach of explicitly specifying an address when calling shmat() (or mmap()).

Using the Linux-specific /proc/PID/maps file, we can see the location of the shared memory segments and shared libraries mapped by a program, as we demonstrate in the shell session below.

Note

Starting with kernel 2.6.14, Linux also provides the /proc/PID/smaps file, which exposes more information about the memory consumption of each of a process’s mappings. For further details, see the proc(5) manual page.

Figure 48-2. Locations of shared memory, memory mappings, and shared libraries (x86-32)

In the shell session below, we employ three programs that are not shown in this chapter, but are provided in the svshm subdirectory in the source code distribution for this book. These programs perform the following tasks:

• The svshm_create.c program creates a shared memory segment. This program takes the same command-line options as the corresponding programs that we provide for message queues (Example 46-1, in Creating or Opening a Message Queue) and semaphores, but includes an additional argument that specifies the size of the segment.

• The svshm_attach.c program attaches the shared memory segments identified by its command-line arguments. Each of these arguments is a colon-separated pair of numbers consisting of a shared memory identifier and an attach address. Specifying 0 for the attach address means that the system should choose the address. The program displays the address at which the memory is actually attached. For informational purposes, the program also displays the value of the SHMLBA constant and the process ID of the process running the program.

• The svshm_rm.c program deletes the shared memory segments identified by its command-line arguments.

We begin the shell session by creating two shared memory segments (100 kB and 3200 kB in size):

$ ./svshm_create -p 102400
9633796
$ ./svshm_create -p 3276800
9666565
$ ./svshm_create -p 102400
1015817
$ ./svshm_create -p 3276800
1048586

We then start a program that attaches these two segments at addresses chosen by the kernel:

$ ./svshm_attach 9633796:0 9666565:0
SHMLBA = 4096 (0x1000), PID = 9903
1: 9633796:0 ==> 0xb7f0d000
2: 9666565:0 ==> 0xb7bed000
Sleeping 5 seconds

The output above shows the addresses at which the segments were attached. Before the program completes sleeping, we suspend it, and then examine the contents of the corresponding /proc/PID/maps file:

Type Control-Z to suspend program
[1]+  Stopped           ./svshm_attach 9633796:0 9666565:0
$ cat /proc/9903/maps

The output produced by the cat command is shown in Example 48-4.

Example 48-4. Example of contents of /proc/PID/maps

$ cat /proc/9903/maps

    08048000-0804a000 r-xp 00000000 08:05 5526989  /home/mtk/svshm_attach
    0804a000-0804b000 r--p 00001000 08:05 5526989  /home/mtk/svshm_attach
    0804b000-0804c000 rw-p 00002000 08:05 5526989  /home/mtk/svshm_attach
 b7bed000-b7f0d000 rw-s 00000000 00:09 9666565  /SYSV00000000 (deleted)
    b7f0d000-b7f26000 rw-s 00000000 00:09 9633796  /SYSV00000000 (deleted)
    b7f26000-b7f27000 rw-p b7f26000 00:00 0
 b7f27000-b8064000 r-xp 00000000 08:06 122031   /lib/libc-2.8.so
    b8064000-b8066000 r--p 0013d000 08:06 122031   /lib/libc-2.8.so
    b8066000-b8067000 rw-p 0013f000 08:06 122031   /lib/libc-2.8.so
    b8067000-b806b000 rw-p b8067000 00:00 0
    b8082000-b8083000 rw-p b8082000 00:00 0
 b8083000-b809e000 r-xp 00000000 08:06 122125   /lib/ld-2.8.so
    b809e000-b809f000 r--p 0001a000 08:06 122125   /lib/ld-2.8.so
    b809f000-b80a0000 rw-p 0001b000 08:06 122125   /lib/ld-2.8.so
 bfd8a000-bfda0000 rw-p bffea000 00:00 0        [stack]
 ffffe000-fffff000 r-xp 00000000 00:00 0        [vdso]

In the output from /proc/PID/maps shown in Example 48-4, we can see the following:

• Three lines for the main program, shm_attach. These correspond to the text and data segments of the program . The second of these lines is for a read-only page holding the string constants used by the program.

• Two lines for the attached System V shared memory segments .

• Lines corresponding to the segments for two shared libraries. One of these is the standard C library (libc-version.so) . The other is the dynamic linker (ld-version.so), which we describe in Using a Shared Library .

• A line labeled [stack]. This corresponds to the process stack .

• A line containing the tag [vdso] . This is an entry for the linux-gate virtual dynamic shared object (DSO). This entry appears only in kernels since 2.6.12. See http://www.trilithium.com/johan/2005/08/linux-gate/ for further information about this entry.

The following columns are shown in each line of /proc/PID/maps, in order from left to right:

1. A pair of hyphen-separated numbers indicating the virtual address range (in hexadecimal) at which the memory segment is mapped. The second of these numbers is the address of the next byte after the end of the segment.

2. Protection and flags for this memory segment. The first three letters indicate the protection of the segment: read (r), write (w), and execute (x). A hyphen (-) in place of any of these letters indicates that the corresponding protection is disabled. The final letter indicates the mapping flag for the memory segment; it is either private (p) or shared (s). For an explanation of these flags, see the description of the MAP_PRIVATE and MAP_SHARED flags in Section 49.2. (A System V shared memory segment is always marked shared.)

3. The hexadecimal offset (in bytes) of the segment within the corresponding mapped file. The meanings of this and the following two columns will become clearer when we describe the mmap() system call in Chapter 49. For a System V shared memory segment, the offset is always 0.

4. The device number (major and minor IDs) of the device on which the corresponding mapped file is located.

5. The i-node number of the mapped file, or, for System V shared memory segments, the identifier for the segment.

6. The filename or other identifying tag associated with this memory segment. For a System V shared memory segment, this consists of the string SYSV concatenated with the shmget() key of the segment (expressed in hexadecimal). In this example, SYSV is followed by zeros because we created the segments using the key IPC_PRIVATE (which has the value 0). The string (deleted) that appears after the SYSV field for a System V shared memory segment is an artifact of the implementation of shared memory segments. Such segments are created as mapped files in an invisible tmpfs file system (A Virtual Memory File System: tmpfs), and then later unlinked. Shared anonymous memory mappings are implemented in the same manner. (We describe mapped files and shared anonymous memory mappings in Chapter 49.)