A lot has changed in the world of Unix system administration since the first publication of this book in 2004. At that time, few casual computer users ran Unix-like operating systems—but as beginner-friendly desktop Linux distributions like Ubuntu gained popularity, that began to change. Then came OS X, the next iteration of Apple’s Unix-based operating system, as well as a slew of technologies based on iOS. Today, Unix-like operating systems are more widely adopted than ever. Indeed, they are perhaps the most ubiquitous operating systems in the world, if we take Android smartphones into account.
Needless to say, much has changed, but one thing that has persisted is the Bourne-again shell, or bash, as the prevailing system shell available to Unix users. Utilizing the full power of bash scripting has never been a more needed skill in a system admin’s, engineer’s, or hobbyist’s toolbox.
This book focuses on common challenges you might face when writing portable automation, such as when building software or providing orchestration, by making common tasks easily automatable. But the way to get the most out of this book is to take the solution you create for each problem and extrapolate it to other, similar problems you may encounter. For instance, in Chapter 1, we write a portable echo implementation by creating a small wrapper script. While many system admins will get some benefit from this specific script, the important takeaway is the general solution of creating a wrapper script to ensure consistent behavior across platforms. Later on in the book, we delve into some wicked cool features of bash scripting and common utilities available for Unix systems, putting great versatility and power right at your fingertips.
Bash remains a staple tool for anyone working on Unix-like servers or workstations, including web developers (many of whom develop on OS X and deploy to Linux servers), data analysts, mobile app developers, and software engineers—to name just a few! On top of that, more hobbyists are running Linux on their open source microcomputers, like the Raspberry Pi, to automate their smart homes. For all of these uses, shell scripts are perfect.
The applications of these scripts are endlessly useful for both those looking to develop their already substantial bash skills with some cool shell scripts and those who may only use a terminal or shell script every once in a while. Individuals in the latter camp may want to brush up on a few shortcuts or supplement their reading with an introduction to more advanced bash concepts.
This book isn’t a tutorial, though! We aim to bring you practical technical applications of bash scripting and common utilities in (mostly) short, compact scripts, but we don’t provide line-by-line explanations. We explain the core parts of each script, and more seasoned shell scripters might be able to tell how the rest of the script works by reading the code. But we expect you as the reader to play with the script—breaking it and fixing it and altering it to meet your needs—to figure it out. The spirit of these scripts is all about solving common challenges, such as web management or syncing files—problems every techie needs to solve regardless of the tools they’re using.
This second edition updates and modernizes the original 12 chapters and adds 3 new chapters. Each chapter will demonstrate new features or use cases for shell scripts, and together they cover a wide range of ways shell scripts can be used to streamline your use of Unix. OS X users should rest assured that most of the scripts in the book will work across Linux or OS X; it is called out explicitly when this is not the case.
Chapter 0: A Shell Scripts Crash Course
This brand-new chapter for the second edition gives new Unix users a quick introduction to the syntax of bash scripts and how to use them. From the very basics of what shell scripts are to building and executing simple shell scripts, this short and no-nonsense chapter gets you up to speed on bash scripts so you can hit the ground running in Chapter 1.
Chapter 1: The Missing Code Library
Programming languages in the Unix environment, particularly C, Perl, and Python, have extensive libraries of useful functions and utilities to validate number formats, calculate date offsets, and perform many other useful tasks. When working with the shell, we’re left much more on our own, so this first chapter focuses on various tools and hacks to make shell scripts more friendly. What you learn in this chapter will help both with the scripts you find throughout the book and with your own scripts. We’ve included various input validation functions, a simple but powerful scriptable frontend to bc, a tool for quickly adding commas to improve the presentation of very large numbers, a technique for sidestepping Unixes that don’t support the helpful -n flag to echo, and a script for using ANSI color sequences in scripts.
Chapters 2 and 3: Improving on User Commands and Creating Utilities
These two chapters feature new commands that extend and expand Unix in various helpful ways. Indeed, one wonderful aspect of Unix is that it’s always growing and evolving. We’re just as guilty of aiding this evolution as the next hacker, so this pair of chapters offers scripts that implement a friendly interactive calculator, an unremove facility, two reminder/event-tracking systems, a reimplementation of the locate command, a multi–time zone date command, and a new version of ls that increases the usefulness of the directory listings.
This may be heresy, but there are aspects of Unix that seem broken, even after decades of development. If you move between different flavors of Unix, particularly between open source Linux distributions and commercial Unixes such as OS X, Solaris, or Red Hat, you’ll become aware of missing flags, missing commands, inconsistent commands, and similar issues. Therefore, this chapter includes both rewrites and frontends to Unix commands that will make them a bit more friendly or more consistent with other Unixes. Included here is a method of adding GNU-style full-word command flags to non-GNU commands. You’ll also find a couple of smart scripts to make working with various file compression utilities considerably easier.
Chapters 5 and 6: System Administration: Managing Users and System Maintenance
If you’ve picked up this book, chances are that you have both administrative access and administrative responsibility on one or more Unix systems, even if it’s just a personal Ubuntu or BSD box. These two chapters offer quite a few scripts to improve your life as an admin, including disk usage analysis tools, a disk quota system that automatically emails users who are over their allotted quota, a killall reimplementation, a crontab validator, a log file rotation tool, and a couple of backup utilities.
Chapter 7: Web and Internet Users
This chapter includes a bunch of really cool shell script hacks that show that the Unix command line offers some wonderful—and simple— methods of working with resources on the internet. Included here are a tool for extracting URLs from any web page, a weather tracker, a movie database search tool, and a website change tracker that automatically sends email notifications when changes occur.
Maybe you run a website, either from your own Unix system or on a shared server elsewhere on the network. If you’re a webmaster, the scripts in this chapter offer interesting tools for building web pages on the fly, creating a web-based photo album, and even logging web searches.
Chapters 9 and 10: Web and Internet Administration and Internet Server Administration
These two chapters address the challenges facing the administrator of an internet-facing server. They include two scripts that analyze different aspects of a web server traffic log, tools for identifying broken internal or external links across a website, and a slick Apache web password management tool that makes it easy to maintain the accuracy of a .htaccess file. Techniques for mirroring directories and entire websites are also explored.
OS X, with its attractive, commercially successful graphical user interface, is a tremendous leap forward in the integration of Unix into user-friendly operating systems. More importantly, because OS X includes a complete Unix hidden behind the pretty interface, there are a number of useful and educational scripts that can be written for it, and that’s exactly what this chapter explores. In addition to an automated screen capture tool, there are scripts in this chapter that explore how iTunes stores its music library, how to change the Terminal window titles, and how to improve the useful open command.
Chapter 12: Shell Script Fun and Games
What’s a programming book without at least a few games? This chapter integrates many of the most sophisticated techniques and ideas in the book to create six fun and challenging games. While the goal of this chapter is to entertain, the code for each game is also well worth studying. Of special note is the hangman game, which shows off some smart coding techniques and shell script tricks.
Chapter 13: Working with the Cloud
Since the first edition of this book, the internet has taken on more and more responsibilities in our daily lives, many of which revolve around synchronizing devices and files with cloud services such as iCloud, Dropbox, and Google Drive. This chapter covers shell scripts that enable you to take full advantage of these services by ensuring files and directories are backed up and synchronized. You’ll also find a couple of shell scripts that show off specific features of OS X for working with photos or text-to-speech.
Chapter 14: ImageMagick and Working with Graphics Files
Command line applications don’t have to be limited to text-based data or graphics. This chapter is dedicated to identifying and manipulating images from the command line using the suite of image-processing tools included in the open source software ImageMagick. From identifying image types to framing and watermarking images, the shell scripts in this chapter accomplish common image tasks, plus a few more use cases.
The final chapter simplifies the tedious details of dealing with dates and appointments: figuring out how far apart two dates are, what day a given date was, or how many days there are until a specified date. We solve these problems with easy-to-use shell scripts.
Appendix A: Installing Bash on Windows 10
During the development of the second edition, Microsoft began to heavily change its stance on open source software, going so far as to release a full bash system for Windows 10 in 2016. While the examples in the book have not been tested against this version of bash, many of the concepts and solutions should be very portable. In this appendix, we cover installing bash on Windows 10 so you can try your hand at writing some wicked cool shell scripts on your Windows machines!
Every good girl or boy scout knows you should always have a backup plan! In our case, we wanted to make sure we had backup shell scripts during the development of this book in case anything came up and we needed to replace some scripts. As it turned out, we didn’t need our backups, but it’s no fun keeping secrets from your friends. This appendix includes three extra scripts—for bulk-renaming files, bulk-running commands, and finding the phase of the moon—that we couldn’t just keep to ourselves once we had the first 101 scripts ready to go.
The source files for all the shell scripts, plus a few of the hacked scripts, are available to download from https://www.nostarch.com/wcss2/. You’ll also find resource files for examples we use in scripts, like a list of words for the hangman game in Script #84 on page 277, and the excerpt from Alice in Wonderland in Script #27 on page 98.
We hope you enjoy the updates we’ve made and new scripts we’ve added to this classic book on shell scripting. Having fun is an integral part of learning, and the examples in this book were chosen because they were fun to write and fun to hack. We want readers to have as much fun exploring the book as we did writing it. Enjoy!