In this section, you will learn why every command can use three standard streams for accessing its input and output. Also, you will learn how to work with those input and output streams and how to use redirection. Finally, we will learn how to use pipes and why they are so important. One philosophy of the Linux operating system is that every command has exactly one functionality in the system, nothing more, and nothing less. For example, there's one command to list files, another to sort text, and one to print the file's content, and so on.
Now, one of the most important features of the shell is to connect different commands to create custom tailored solutions and tools for all kinds of problems and workflows. But before we can show you how to concatenate different commands together to build something powerful, we first need to know how a command uses its input and output and what input and output redirection is. Most Linux commands follow a similar pattern when processing data. Most of the commands we are using do get some kind of input, for example, they read the content of a file and then they process this information, and afterward almost all of them do output some kind of results on the computer screen. Because every command uses some kind of input and returns some kind of output on Linux, three standard channels are defined and are available for every command. They are used for communication between the operating system and the command during execution. They are called standard input or stdin, standard output or stdout, and standard error or stderr.
Normal program output goes to the stdout channel, while stderr is also an output stream and it can be used for showing and processing any kind of error messages occurring while a command is executing. These are also called standard streams. They are called streams because the data is flowing continuously through a specific channel and gets processed or generated consecutively by the command, although they have an open end, which means the command working with them cannot predict when this flow of data will stop or finish. Now, we can change the stdin and stdout locations using certain files; this is called redirection.
Here, in this section, we will also explain the concept of pipes, which is one of the most fundamental concepts and major features of the Linux shell, and how to work with them. For example, if you type ls /var/lib/system/, the result random-seed will be printed to the screen because it is defined as an stdout device by default for every Linux command. But if you type cat /var/log/messages, an error message is printed to the same screen as both stdout and stderr are connected to the same output device, the screen.
On Linux, your physical input and output devices, such as your keyboard or screen, like any other hardware devices, are abstracted and represented by special system files. All of these special files reside in a system directory called /dev, which is also called the system devices directory. But what can we do with a system like that? Its beauty is that we can redirect the input and output of a command to another location other than the default keyboard and screen source or destinations, which also must be of the filetype. This is also very useful to separate stdout and stderr to two different locations, which especially helps to keep the overview of a command running if it produces a lot of output.
For output channel redirection, we use the greater than sign (>), for input redirection we use the smaller than sign (<). To address a specific channel, such as stdin, stdout, and stderr, we use the corresponding numbers 0, 1, and 2. When using output redirection, the stdout channel is expected, so we don't have to write it explicitly. For 99% of all cases you only redirect stdout and stderr, so let's focus on those examples.
To redirect the stdout stream output of a command to a file, use the greater than sign. As said before, the stdout channel is expected, so the last command can also be typed as follows:
ls /var/lib/systemd/ > /tmp/stdout-output.txt ls /var/lib/systemd/ 1> /tmp/stdout-output.txt
Use the card command to print out the content of the file that we just created with the redirection to stdout. To redirect the stderr channel, use number 2 as the standard stream descriptor. The following screenshot shows the output of the previous commands:
As you can see, the error message has been redirected to a file. To redirect stdout and stderr to two different files, type the commands shown in the following screenshot:
Another notation, using the ampersand char allows the redirecting of one channel another one. To redirect stderr to the stdout channel, type the commands shown in the following screenshot:
Sometimes, you are only interested in one output stream, therefore a special device file exists in any Linux system, which is called the null device, and it consumes and vanishes any kind of streaming data that gets redirected to it into the void. If you don't want any output at all for any command, for example, you can use the command shown in the following screenshot:
Finally, to redirect stdin, you use the smaller than sign [<]. For example, this can be very useful because some of the available shell commands can directly read a file's content as stdin, such as the grep command, which we will learn about later.
Now, let's discuss pipes. Besides redirecting a command's default input and output streams, stdin, stdout, and stderr, to files, we can also use the concept of shell pipes to get one command output as the input for another command. There are no limits to this system and it's very easy to build multi-command chains to answer very complex questions for you. As mentioned previously, this shell feature lets you create very powerful command pipelines and workflows for creating custom tailored solutions for all kinds of Linux command-line work, and to answer very complex questions for you.
To chain commands together, which means to use stdout from the first command as stdin to the next command, we use the vertical bar symbol [|] on our keyboard, which in Linux is called the pipe symbol. For example, if you've got a very long directory content list that you want to read without scrolling through the Terminal window forever, you can use the pipe to output the directory content from the ls command, not on the screen, but directly as input for the file viewer, as we learned before. Often, pipes are used to avoid intermediate result files and are more efficient without them. The use cases for this are endless, for example, if we got a file with unsorted names of people in it, we could sort them using cat names.txt | sort:
You can also get a list of all the unique names in this file. We will use the unique command to do so, which only works on a sorted list. So, we need to sort using cat names.text | sort | uniq:
You can also count the number of unique lines using the word count command-line tool using cat names.text | sort | uniq | wc:
There are quite a few unique names in this file. The sky's the limit when it comes to pipe examples, and there are just too many examples. Ideally, this should be run with the root user account. Please ignore the errors. The following screenshot shows the core summary of the filesystem:
Also, another useful pipe command is to print out the used files in a directory. If you are using a Windows system, you may know of a utility called ZIP, which compresses files. On Linux, you can do something very similar, but here we need two tools to work together. For compression, we use the gzip tool. Because gzip can only work on single files, we first need to create an archive that will concatenate multiple files to a single file. For archiving, we use the tar command. So, to create a compressed archive of your home directory in the /tmp directory, first create an archive of your home directory using the tar command: tar -cv /home/olip/ | gzip. The archive will be output to the stdout stream, so we pipe it into the gzip command as stdin. As gzip itself outputs the compressed file to stdout, we will redirect it to a file. The result of the compression versus the uncompressed data amount is as follows:
A lot more piping examples will be shown throughout this book. If you redirect stdout or stderr into a file, normally the file will be erased if it already exists, or a new file will be created before any content is written to it. So as not to delete a file, but append the content instead, use the greater than sign. For example, to create a new output file, execute the command shown in the following screenshot:
Now, to append the string, Hello World, to the output file, we will use the greater than sign. This will not delete the file's content when we start to redirect content to it. Instead, it will append the content to the end of the file. As said before, pipes are one of the most important concepts of the shell and it is so much fun working with them.