Single-user mode
unscheduled in the nighttime?
Something just went “boom”!
In order to properly manage any computing platform, you must understand the boot process. Many systems administration tasks cannot be done while the system is running. OpenBSD specifically requires that certain tasks be done before the boot process has completed. And, of course, on any operating system, sometimes a process starting up prevents the system from completing its boot. The only way to fix these problems is to interrupt the boot partway through.
First, we’ll look at the key to OpenBSD’s booting process: the boot loader. Then we’ll move on to single-user mode, and finally multiuser startup. You can perform useful work at any of these stages.
I recommend playing with the OpenBSD boot process on a test machine before one of your machines won’t boot. That way, when something breaks in the wee hours of the morning, you can spend your time fixing the problem instead of fumbling around with unfamiliar commands.
In general, when a PC-style computer first boots, it starts the BIOS. The BIOS is a small piece of software that figures out things like which drives are attached and what they’re attached to, what sort of CPU is installed, and how much memory is available. After getting that information, the BIOS loads a minimal boot loader from some kind of storage device.[11]
The boot loader is a small program that handles initial system configuration and boots the kernel. It finds and starts the kernel, which in turn detects hardware, attaches device drivers, and performs other core setup. Finally, the kernel calls init(8)
, which starts processes and enables user programs, network interfaces, server software, and so on.
While most of this process cannot be managed—no one actually configures init
!—there’s plenty you can do before the system finishes booting and dumps you at the login screen.
The OpenBSD boot loader lets you interrupt the boot process, configure the system before it boots, adjust kernel settings, and even boot an alternate kernel.
When the hardware hands control of the boot process over to the OpenBSD partition, you’ll see the boot loader prompt, which looks something like this:
>> OpenBSD/amd64 BOOT 3.18 boot>
The boot loader’s main purpose is to find the kernel, load it into memory, and start it. Because it runs before the kernel starts, the boot loader can pass instructions to the kernel itself.
Here are some of the things you can do before booting is complete:
Use built-in helpUse the
help
function to print a brief list of commands that the boot loader supports.boot> help commands: # boot echo env help ls machine reboot set stty time machine: boot diskinfo memory
Delay the boot processBy default, the loader waits five seconds for instructions, and then boots the kernel. To pause the boot at the prompt, press the spacebar.
Set the boot timeoutTo set a new boot idle timeout, specify a number of seconds with the
set timeout
command.boot> set timeout 10
After the boot prompt is idle for 10 seconds, the system should boot.
Boot the systemIf you’ve paused the boot process, the system won’t boot until you tell it to. When you’re ready to boot, use the
boot
command:> boot
We’ll use various permutations of boot
to configure the kernel, boot single-user mode, and so on. I’ll cover other boot commands in the appropriate sections. For full details on what you can do at the boot loader prompt, read the boot(8)
man page.
Single-user mode is the earliest point when OpenBSD can give you a Unix-style shell prompt. At this point, the kernel has probed all the hardware, attached drivers to all the hardware that it’s going to acknowledge, and started init
. The system hasn’t mounted any filesystems except for the root partition, which is mounted in read-only mode. The network isn’t started, no services are running, security is not implemented, and filesystem permissions are ignored.
To boot OpenBSD in single-user mode, enter boot -s
at the loader prompt.
boot> boot -s
Why would you want to boot into single-user mode? If your computer has a problem that is preventing it from booting, you should be able to access single-user mode and fix the problem. Suppose a failed disk is preventing the system from booting during a multiuser boot, or you changed your terminal settings in /etc/ttys and now you can’t log on to the system. Or maybe you put a daft setting in rc.conf.local, and the boot process hangs because it’s trying to do something impossible. At times like these, single-user mode is your best friend.
Also, some system administration tasks, such as clearing filesystem flags (see Chapter 8), can be done only in single-user mode.
Usually, you should have a fully functional filesystem before doing anything in single-user mode. If your system crashed, be sure to check the integrity of your filesystems before mounting them:
# fsck -p /dev/sd0a (e4bf0318329fe596.a): file system is clean; not checking /dev/sd0h (e4bf0318329fe596.h): file system is clean; not checking … # mount -a
fsck
and mount
have many more options. We’ll cover them in more detail in Chapter 8.
Once you’ve mounted all of your filesystems, all usual command-line software should be available. You should be able to edit configuration files, start and stop programs, and generally do whatever you like to the system (including destroy it).
Use the shell script /etc/netstart to configure the network while in singleuser mode. (You could run all the appropriate commands by hand, but /etc/netstart will read your system’s configuration files and do the grunt work for you.) You must explicitly run this script through sh
:
# sh /etc/netstart
If you’re booting into single-user mode because of network problems, this script will conveniently reproduce the issue for you.
As we’ll cover in tedious detail in Chapter 18, you can configure the OpenBSD kernel, but before you do so, be sure that you can boot alternate kernels. You’ll need to be able to boot a different kernel if, say, you hose your filesystem so badly that it won’t even boot to single-user mode, and you need to recover using the installation kernel.
An OpenBSD installation includes three kernels out of the box: the single-processor kernel /bsd, the multiprocessor kernel /bsd.mp, and the upgrade and install kernel /bsd.rd. (If your machine has multiple processors, the installer renames /bsd to /bsd.sp and /bsd.mp to /bsd.)
To boot a nonstandard kernel, first reboot and interrupt the boot process at the boot loader prompt. Run boot
, and give the full path to the kernel you want to boot:
boot> boot /bsd.rd
This will start the system using your chosen kernel. You can use other boot options as well, such as booting the alternate kernel in single-user mode:
boot> boot -s /bsd.sp
This will let you recover from a bad kernel, try a new kernel, or anything in between.
Suppose you’ve really fouled everything up beyond all recognition, and you don’t have a usable kernel on your root partition. Fortunately, if you have a usable kernel on a different hard drive, you can boot from that. (Usually, this kernel lives on an alternate root partition, /altroot, as discussed in Chapter 8.) In this section, I’ll break the task of booting from that alternate kernel into a few steps: finding the hard disk with the partition, finding the partition with the file, and booting the right file on that partition.
Once you’re familiar with OpenBSD, you may begin to think of the hard drives in your system by their device names, such as /dev/sd0, /dev/wd1, and so on. Unfortunately, those are the kernel’s names for the disks; the boot loader recognizes only the BIOS’s disk names.
To ask the boot loader about disks, use the machine diskinfo
command:
boot> machine diskinfo
Disk BIOS# Type Cyls Heads Secs Flags Checksum
fd0 0x0 *none* 80 2 18 0x4 0x0
hd0 0x80 label 1024 255 63 0x2 0x51db843d
hd1 0x81 label 1024 255 63 0x2 0x9329b723
hd2 0x82 label 1024 255 63 0x2 0xcfadb343
boot>
Here, the boot loader has found four disk devices. The first, fd0
, is a floppy disk drive. This drive might or might not have a disk in it, but whatever it has, it’s almost certainly not your alternate kernel. (It might be an installation disk, though, so don’t automatically rule out using it for disaster recovery.)
The other three devices—hd0
, hd1
, and hd2
—are hard disks. The first, hd0
, is the default system boot disk. If you can’t boot from that disk, you need to find the hard disk that contains your kernel.
Vague stirrings of memory in this output lead me to think that hd2
might be the disk that holds my backup root partition. To try it, tell the loader that disk partition hd2a
is the new root partition:
boot> set device hd2a
Before trying to boot from this partition, look at its contents:
boot> ls
stat(hd2a:/.): Invalid argument
boot>
Apparently, disk hd2
has no partition a
. After service is restored, I’ll take this disk out behind my garage and beat its weakness out of it. For now, let’s try the only remaining disk, hd1
.
boot> set device hd1a boot> ls drwxr-xr-x 0,0 512 . drwxr-xr-x 0,0 512 .. drwxr-xr-x 0,0 512 altroot drwxr-xr-x 0,0 512 home drwxr-xr-x 0,0 512 tmp …
This looks like an actual root partition (altroot
offers a hint).
At this point, we could boot a different kernel, but we’ll just boot the /bsd kernel on this partition in single-user mode, because the filesystem table would have the incorrect entry for the root filesystem, which would mess up all sorts of stuff.
boot> boot -s
booting hd1a:/bsd: 5669864+1601484+935608+0+617568 [89+499848+323884]=0xd351b8
…
Alternatively, you could give the device name at the boot prompt:
boot> boot -s hd1a:/bsd
As a general rule, you should mount the actual root partition on /mnt, make the necessary changes for normal operation, and reboot into the proper root partition. You could also boot the /bsd.rd kernel, giving you a cleaner boot at the cost of having fewer tools available.
To make boot loader options permanent, edit /etc/boot.conf. The boot loader reads and runs entries from this file before giving you the boot>
prompt, which means you can use it to automatically run boot loader commands every time your computer boots. (Although if you would rather sit at your computer and enter your settings every time you reboot, don’t let me stop you.)
Any command you might give at the loader prompt is a valid boot.conf entry. For example, if the default boot speed is too slow for your liking, you can set your boot timeout to two seconds by adding this line to boot.conf:
set timeout 2
You can also tell the system to boot a different kernel with the correct boot.conf command.
set image /bsd.mp
By far, boot.conf is most often used to configure a serial console.
All of these nifty boot functions let you do useful stuff when your system is in trouble, but how can you use them when your computer isn’t right in front of you? If your computer is in a data center on the other side of the country, or sitting in the basement behind the last decade of payroll records, a serial console will make your life far more pleasant.
A hardware serial console allows you to run a serial cable between a computer and a terminal server (on another computer) to access BIOS messages and operating system boot and startup messages which simplifies managing remote systems. Serial consoles are invaluable when debugging system crashes, too; error messages come over the serial port, where you can easily capture them.[12]
True UNIX hardware has serial console capabilities, as does most server-grade i386 and amd64 hardware. Most desktop-grade hardware, however, does not. But fortunately, even if you don’t have a hardware serial console, you can access all of OpenBSD’s startup messages with a serial port and a software serial console. While OpenBSD’s software serial console won’t give you access to the hardware BIOS, it will let you interface with the boot loader and remotely access the system console, even when the network is down.
Every hardware platform has its own standards for serial consoles. If you’re running a less common platform, check your hardware’s documentation.
If your hardware supports a real serial console, you should usually configure it in the BIOS. OpenBSD supports whatever the hardware supports, so your Sparc64 hardware will support OpenBSD’s serial console just as well as it supports any other operating system’s serial console.
A serial console requires a null modem cable, which you should be able to get from any computer store or an online vendor. While gold-plated cables aren’t worth the money, don’t buy the cheapest cable you can find either. If you have an emergency and need the serial console right now, you won’t be in the mood to deal with a defective cable.
Plug one end of the null modem cable into your OpenBSD machine’s first serial port. (The serial console is supported on only the first serial port, or com0
on i386 and amd64 hardware.) Plug the null modem cable’s other end into an open serial port on another system. (For simplicity’s sake, use either another OpenBSD or Unix-like system.)
If you have two OpenBSD machines at a remote location and you want to use serial consoles on both, you can have each machine act as the console client for the other. Attach the first serial port on each server to the second serial port on the other. If you have three machines, you can daisy-chain them in a loop. If you have four or more machines, pick up a used terminal server from your favorite auction site.
You can also use two DB9-to-RJ45 converters, one standard and one crossover, which will allow you to run your console connections over a standard CAT5 cable. If you have a lights-out data center where human beings are forbidden unless they are installing or removing equipment, you can stretch your serial console cables about 12 meters, which should reach into your warm room. (Most modern data facilities are better equipped to handle CAT5 cables than serial cables.)
Now that you have the console physically ready, the next step is to configure your client to access the serial console. Then you can set up the serial console.
The following are the default settings for an OpenBSD i386 or amd64 system:
9600 baud
8 bits
No parity
1 stop bit
Enter these values into any terminal emulator on the client computer, and the serial console should Just Work. You can find terminal emulators for Microsoft platforms (I recommend PuTTY), OS X, and just about any other operating system.
OpenBSD includes the terminal emulator tip(1)
, which reads its configuration from /etc/remote. The configuration tty00
in /etc/remote matches the default OpenBSD serial console configuration for i386 and amd64 systems (as well as several other platforms). If you’ve attached your null modem cable to the first serial port on the client, connect with this command:
# tip tty00
connected
If it doesn’t say connected
, your serial client is misconfigured. Fix your client before enabling your serial console on the server. You want your serial client ready before configuring the console.
OpenBSD normally uses the local physical keyboard, video, and mouse as the console, but it can also use the first serial port as a serial console.
To set the console, use the boot loader. You must know the loader’s device name for your preferred console: com0
for the first serial port or pc0
for the physically attached video and keyboard.
The first time you try to use a serial console, use a local test machine. Set up your client beforehand and start your terminal emulator, and then boot your test machine. At the boot loader prompt, enter this command:
boot> set tty com0
Your server’s monitor and keyboard should stop responding, and if you’ve set up everything correctly, you should see the boot loader prompt in your terminal emulator.
To switch back to the physical console, tell the boot loader to use the pc0
device:
boot> set tty pc0
Poof! The server’s keyboard and monitor should work again.
To have your machine use the serial console at every boot, add this statement in /etc/boot.conf:
set tty com0
Be sure to test your serial console after the machine is installed in its permanent location, and always screw the serial cables to the server. A loose serial cable provides only a comforting illusion that betrays you when it will hurt the most.
Newer serial ports (meaning anything made within the past 10 years) can run at speeds far above 9600 baud. I have servers with serial consoles that run only at 115,200 baud. The BIOS messages display at 115,200 baud, but then the OpenBSD console runs at 9600 baud. My client displays one or the other as gibberish. (A lot of OpenBSD folks think that anything that won’t do serial at 9600 baud is broken, but you won’t always have control over the hardware you work with.)
To use these ports, I can either change my connection speed in my serial console client when switching between the BIOS messages and the OpenBSD messages, or change the speed of my OpenBSD console to match the hardware.
At the boot loader, tell the serial console to run at 115,200 baud:
boot> stty com0 115200 boot> set tty com0
If these settings work, copy them to /etc/boot.conf.
Now configure your serial client. Modify tip
to use the higher speed. First, find the entry for tty00
in /etc/remote:
tty00|For hp300,i386,mac68k,macppc,vax:\ :dv=/dev/tty00:tc=direct:tc=unixhost:
But don’t modify this entry! We’ll use it to illustrate the style of /etc/remote entries.
/etc/remote is designed much like a termcap(5)
database. If you ever need to write your own termcap
entries from scratch, you’re living your life wrong. But you can recognize the contents and modify existing entries without much pain. If you really want to learn everything about these entries, read the remote(5)
man page.
Backslashes (\
) in this entry mean “continued on the next line.” Colons separate fields. Each line after the first must start with a colon, and each field is a key/value pair.
Now, to create a console entry that runs at 115,200 baud, use the following:
console:br#115200:tc=tty00:
The first field in an /etc/remote entry is the name, and every entry must have a unique name. I named this entry console
. The second field is the br
value. According to remote(5)
, br
stands for bit rate. I’ve set the bit rate to 115,200 baud. The third field is tc
, for “table continues,” which is equal to tty00
. This means that the description of this entry continues in entry tty00
.
Taken as a whole, this entry means “copy the tty00
entry, and add a bit rate of 115,200.”
If you have two OpenBSD machines, each sending its serial console out its first serial port to the other machine’s second serial port, you must tell tip
to connect to the second serial port. The command tip tty00
doesn’t actually connect to the serial port named tty00
—it connects to a port defined by the /etc/remote entry named tty00
. That means that you can’t run, say, tip tty03
and connect to serial port tty03 unless you have an /etc/remote entry named tty03
. By default, there isn’t one, but you can define one easily, as follows:
tty03:dv=/dev/tty03:tc=tty00:
This entry is named tty01
. The dv
setting tells /etc/remote the physical device to use. Other than this, all settings are copied from the entry called tty00
.
With these examples, you should be able to use OpenBSD’s tip
to connect to almost any serial console.
The serial console lets you interact with the boot process. Once your machine is fully multiuser, however, a default serial console will not let you actually log in to OpenBSD. In multiuser mode, OpenBSD uses getty(8)
to initialize terminals and handle logins, and in order to log in to your machine over a serial port, you will need to tell getty
to take charge of the serial line by configuring /etc/ttys.
We’ll discuss /etc/ttys further in Chapter 14, but for now, here’s how to allow logins over the first serial port. Find the entry for tty00
, which should look like this:
tty00 "/usr/libexec/getty std.9600" unknown off
Remove the last two words, and replace them to match the following:
tty00 "/usr/libexec/getty std.9600" vt220 on secure
Now run kill -1 1
, and you should get a login prompt over your serial line.
When the kernel finishes its core setup and hands control over to userland, init(8)
runs the shell script /etc/rc. This script handles all system setup, including mounting filesystems, configuring device nodes, identifying shared libraries, and any other task required to make the system usable. Some tasks are delegated to separate scripts; for example, /etc/netstart is used to configure the network.
In this section, we’ll cover how /etc/rc and other startup scripts function, and the flow of the startup process. Armed with this understanding, you should be able to easily configure your OpenBSD machine to start exactly what you need—no more, no less.
The startup system includes the scripts /etc/rc, /etc/rc.conf, /etc/rc.conf.local, /etc/netstart, /etc/rc.securelevel, /etc/rc.local, /etc/rc.shutdown, /etc/rc.firsttime, /etc/fastboot, and the contents of the /etc/rc.d directory.
On OpenBSD, everything outside the kernel is configured with a shell command, from setting the hostname to starting server daemons. The master script is /etc/rc, and it runs all of these commands in the correct order, ensuring that the system is configured exactly the same way at every boot. As a final step, /etc/rc runs getty(8)
to present login prompts on all the appropriate terminals.
Never edit /etc/rc unless you’re a very experienced systems administrator with truly unique needs. This is one of the several files in /etc that is technically editable, but mere mortals are well advised to treat as binary. Instead, whenever you need to disable functions, deactivate them in /etc/rc.conf.local. To add new functionality to the startup process, use the shell scripts /etc/rc.securelevel and /etc/rc.local, or write a shell script for /etc/rc.d.
The /etc/rc.conf file contains nothing but the default values for all other startup scripts. Read this file to see the configuration options for different system services. Here’s a small snippet of what you’ll find in rc.conf:
… bgpd_flags=NO # for normal use: "" rarpd_flags=NO # for normal use: "-a" bootparamd_flags=NO # for normal use: "" rbootd_flags=NO # for normal use: "" sshd_flags="" # for normal use: "" named_flags=NO # for normal use: "" …
If a variable is set to NO
, the associated service is disabled by default.
As you can see, OpenBSD turns off almost everything by default, with one exception: the SSH daemon. Setting the variable to a pair of quotes, as shown after each entry in the preceding snippet, is enough to enable most daemons, and most daemons will run just fine without any command-line flags. However, if a daemon requires a command-line argument in order to run, that argument will be shown as it is in the -a
attached to rarpd_flags
.
I’ve mentioned this before, but I’m going to beat you over the head with it: Place your changes to rc.conf in rc.conf.local. Entries in rc.conf.local override the defaults in rc.conf.
For example, say that on a particular machine, you want to run sshd(8)
with extra debugging, and you also want to run named(8)
. Additionally, you want to run the time server ntpd(8)
, and have it correct the time at boot by using the -s
flag. After consulting the documentation for those programs, you add the following lines to rc.conf.local:
sshd_flags="-D" ntpd_flags="-s" named_flags=""
OpenBSD will start the programs with the flags specified here. If you specify invalid, incorrect, or incompatible flags, the daemon will print error messages to the console.
While its name differs from the other scripts, /etc/netstart is definitely a system startup script. It reads /etc/mygate, /etc/myname, and all the /etc/hostname.if files, and uses the information in them to configure all network interfaces, bridges, routing, and so forth. The file /etc/rc runs this script before starting any server daemons, network filesystems, and so on. In single-user mode, you’ll run this script by hand to bring up the network.
The /etc/rc.securelevel shell script runs early in the boot process, before /etc/rc raises the system securelevel, but after starting the network. Many programs, particularly those that touch the kernel or intimately affect the filesystem, will not run once the securelevel is raised. If you run such a program, you can add the command to start it to this script. If your local program doesn’t need to run before the system securelevel is raised, you’re better off starting it from rc.local or writing a proper rc.d script for it.
One important entry in rc.securelevel is the definition of the system securelevel. We’ll discuss securelevels in Chapter 10. For now, don’t touch the line that sets the securelevel unless you’re already familiar with BSD-based systems and know exactly which toe you’re shooting off.
After /etc/rc does just about everything else, it runs /etc/rc.local. You can put commands to start local daemons in rc.local, but you’re better off writing an rc.d script to start local daemons so you can easily and consistently restart them later. Of course, if you’re lazy, you can get by with rc.local.
Whenever you use reboot(8)
or halt(8)
, OpenBSD runs the /etc/rc.shutdown script, which you can count on to run extra commands needed to safely shut down your server. Most server software shuts down cleanly without any special intervention, but software that requires data integrity (like databases) may need help shutting down without losing data. Again, if at all possible, write an rc.d script to manage your software.
/etc/rc runs the script /etc/rc.firsttime once, mails the output to root, and deletes rc.firsttime. The installer uses rc.firsttime for tasks such as fetching firmware that can’t be legally redistributed. While you won’t normally use rc.firsttime, you should know that it exists and that you can use it to perform one-time tasks when a machine boots.
If the /etc/fastboot file exists, OpenBSD assumes that all filesystems are clean (see Chapter 8), and the boot process skips checking filesystem integrity.
The /etc/rc.d directory contains shell scripts for managing software, as discussed in the next section. While the system comes with scripts for software included in OpenBSD, add-on packages can provide their own scripts (see Chapter 13).
OpenBSD uses shell scripts to start, stop, restart, check, and reconfigure server software. These scripts are found in the directory /etc/rc.d. Every piece of server software that comes with OpenBSD has a script in this directory, as do most ports and packages that need scripts for proper startup and shutdown. Use these scripts to manage integrated software without rebooting the server.
The rc.d scripts read their configuration from rc.conf and rc.conf.local. Most servers run the SSH daemon sshd
, which can be enabled by adding the line sshd_enable=""
to rc.conf.local. Look in /etc/rc.d, and you’ll find the shell script sshd.
If you change your sshd
configuration, you must restart the daemon. Use the shell script to do this consistently.
# cd /etc/rc.d/ # ./sshd restart sshd(ok) sshd(ok)
Of course, you could do the same thing without the shell script simply by identifying the currently running sshd(8)
process, reading the man page to see how to shut it down properly, and then restarting it with the same command-line flags. In the case of sshd
, that’s easy: Running pkill -1 sshd
would tell the daemon to reread its configuration file. But restarting a daemon that requires all sorts of flags is a big deal. Automating these system administration tasks ensures that your daemons run consistently.
To see if a daemon is running, use the check
command to check your shell for the return value. The script will return a 0
if the daemon is running and a 1
if it isn’t, as shown here:
# ./nfsd check # echo $? 1
As you can see by the 1
, nfsd
is not running.
The most common use for check
is in shell scripts. You can start the daemon with the argument start
and terminate it with stop
. Use the restart
argument to tell the daemon to reload its configuration.
In OpenBSD, rc.d scripts run when the system boots and again when it shuts down. (Something needs to unmount all those hard drives, shut down daemons, and clean up.) At shutdown, every script in the /etc/rc.d folder is called with the stop
argument.
OpenBSD packages for third-party software include rc.d scripts as necessary. For example, the popular database server MySQL mysql-server
package installs the script /etc/rc.d/mysqld. To use the package, you must enable it in rc.conf.local:
mysqld_flags=""
Once the package is enabled, you can manage your MySQL server just like any other OpenBSD daemon. However, packaged software will still not start automatically at boot, so you must tell OpenBSD to run this particular rc.d script at boot and shut down with the pkg_scripts
variable in rc.conf.local:
pkg_scripts="mysqld"
The startup process runs the scripts in this variable, in the order given, at boot. The order is important for certain daemons. For example, if you have a database-driven website, you should start the database before the web server. At shutdown, it runs these scripts in reverse order.
Sometimes you don’t want to enable software globally; you just want to run a certain daemon for a short time or to address a specific situation. You can use rc.d scripts to manage this software using the -f
flag to force the software to run.
Now for a real-life example. I previously ran PostgreSQL on my server, but someone kidnapped my pet rats and blackmailed me into using MySQL in exchange for their safe return. I needed to check some data in the old database, however, so I force-started the disabled PostgreSQL server:
# ./postgresql -f start
postgresql(ok)
If you package or install your own software, I strongly recommend writing your own rc.d script. A few minutes spent reading the existing scripts will tell you most of what you need to know. For the rest, read the rc.d(8)
and rc.subr(8)
man pages.
Now that you can start OpenBSD, let’s set up some user accounts.