In these modern times, the hardworking admin might be tempted to turn her back on the Old Ways and indulge in increasingly exotic methods of interfacing with servers: Ethernet, USB, Firewire, Wireless, Infrared, KVM switches—next stop: direct neural implants.
There is one old-timer that still has a useful place in the network admininstrator's toolkit: the serial console. It's simple and cheap—you don't need to install drivers or expansion cards, it's just there. It's the lowest-level means of interfacing with your system. Configure your servers to accept serial logins, set up a laptop as a portable console, and you have an instant cheap rescue device when everything else fails.
Your portable serial console will also serve you well when you need it to connect to routers and switches.
When you're troubleshooting headless systems, it saves the hassle of hooking up a keyboard and monitor. You can capture kernel and logging messages that otherwise would be lost, reboot the system and get a boot menu, edit network settings and stop/start networking, restart or tweak SSH, and tweak network card settings.
I don't recommend it as your sole means of server administration—Ethernet is a lot faster. But, when nothing else works, the serial console will save the day.
There are a number of ways to make the physical connection. You can connect a hardware-controller modem, the kind old-timers fondly refer to as real modems, and do remote administration via dial-up. It couldn't be any simpler, just dial direct. This makes a nice inexpensive backup for Ethernet failures. Or, grab a null modem cable, connect to a laptop or a nearby workstation, and you have a directly connected serial console. (See Recipe 19.1 to learn about turning a laptop into a portable network diagnostic and repair tool.) You can turn any feeble old PC into a perfectly good serial console.
Users of x86 hardware, which is the majority of the Linux universe, must deal with the limitations of the x86 PC BIOS. Unlike real Unix hardware, it was not designed to support a serial console. This means you can't use the serial port to poke around in the BIOS to make changes or check the settings, and you don't see the Power On Self-Test (POST) messages. The Unix BIOS and bootloaders come with a lot of neat features not available in x86. They can do diskless netbooting, hard resets, reboots, suspend the boot process and then restart it, walk step-by-step through the boot process, and reconfigure the BIOS—all remotely. It's like having a little computer to jump-start and control the big computer.
There are some ways to get around these limitations. One is to purchase x86 server hardware with an advanced BIOS. For example, rack mount and blade units are usually equipped for serial port administration, and come with advanced management features, just like real Unix hardware. Another way is to purchase an expansion card like the PC Weasel. The PC Weasel is a PCI or ISA expansion card that emulates an attached video and keyboard, and presents a serial port for administration. At $250 to $350, it's not an inexpensive option, but when you consider that it's less hassle than a motherboard upgrade and comes with a great feature set, it looks like a pretty good deal.
A lot of data centers rely on commercial serial consoles such as those sold by Cyclades, Lantronix, and Digi, which fit nicely in racks, have up to 48 ports, and include all sorts of management software, remote logging and kernel message capture, and security features. (Don't be confused by all those RJ-45 ports—those are real, genuine serial RS-232 ports that use nice RJ-45 connectors.)
Or, you can build your own, using multiport serial expansion cards from vendors like Comtrol, Moxa, and Axxon. You can even add some management software—Conserver (http://www.conserver.com) is a great open source and free-of-cost console server. It includes excellent logging, SSL integration, user authentication, message broadcasting, spy mode, and system monitoring. This chapter covers how to set up an ordinary x86 PC as a headless server, and how to set up an ordinary x86 PC as a serial console. You could say this is Linux serial consoling on the cheap.