Objectives
How SSH works to create and maintain secure connections.
Advanced SSH usage and techniques.
How to use SSH for secure file transfers.
How to use SSH to perform remote command execution.
How to generate and use Public/Private Key Pairs (PPKP) for authentication.
To use X-forwarding to run GUI programs on the remote host so that the windows of their GUI interface are displayed on the local host.
An easy and elegant way to perform a centralized backup of remote hosts.
Introduction
We looked at using Secure Shell (SSH) in Volume 2, Chapter 17, of this course. We set up the SSH server and used it to connect to itself as the localhost for some simple testing.
SSH is an important mechanism for secure connections between Linux hosts. SSH is a software-based virtual private network (VPN) tool that can create a secure VPN whenever needed. It can be used to securely log in to any remote host so long as you have proper credentials, and it can be used to enhance tools such as tar and other backup programs like rsync so that remote hosts can be easily backed up to a local system. The scp (secure copy) program uses the SSH encrypted tunnel to copy files between a remote host and a local one.
In this chapter, we will explore SSH in more depth and use it to communicate between two separate VM hosts. We will also use Public/Private Key Pairs for authentication that is more secure than using passwords.
The name Secure Shell is misleading. SSH is not a shell; it is a set of connection protocols that enables secure, encrypted links between two computers. An SSH login to a remote host uses the default shell of the user on the remote host. In most cases, this would be Bash, but some users prefer other shells and have designated them as their default shell. Whichever shell the use as their default will be the shell used for the SSH connection.
- 1.
It provides reliable authentication for the identities of the hosts as well as the users. It ensures that the hosts and users are who they claim to be.
- 2.
It encrypts the communication between the hosts, including the transmission of any login ID and password or Public/Private Key Pairs (PPKP).
- 3.
It ensures the integrity of the transmissions and can detect and notify the user if data is missing, added, or changed.
Using SSH for inter-host communications is an excellent security precaution and can prevent data that is transmitted over any part of a public network from being intercepted, blocked, or altered. As with all security tools, SSH is not the complete solution and other security precautions must also be taken. But SSH ensures secure communications can be easily accomplished.
Starting the SSH server
We started the SSH server on the StudentVM1 host in the first book of this course but we need to start it on StudentVM2, too. Any Linux host can be an SSH server, and it makes sense to do so at least some of the time in order to facilitate easy communications and the ability to work on remote hosts.
The SSH server should already be running on StudentVM1. If so, the first set of commands in Experiment 5-1 will emit errors, but nothing bad will happen. However, do be sure to complete the rest of the experiment as well.
Experiment 5-1
The first time an SSH connection is made to any host the authenticity message is displayed along with the fingerprint of the private key of the remote host. In a very security conscious environment, we would have already received a copy of the remote host’s key fingerprint. This allows comparison so that we know we are connecting to the correct remote host. This is not the security key; it is a fingerprint that is unique to that host’s private key. It is impossible to reconstruct the original private key from which the fingerprint was generated.
You must type “yes” – the full word – in order to continue the login. Then you must enter the password for the remote host.
After accepting this key during the first connection to the remote host, the connections initialize a little faster because the two computers now know each other and can identify themselves via the keys.
The host keys are stored in /etc/ssh along with the SSH client and server configuration files. List the contents of that directory to view the various key files.
Now, as the root user on StudentVM1, connect to StudentvM2 and verify that connection works as well.
Type exit to disconnect the SSH connections. Exit from all SSH connections if there are more than just one.
Now we know that the SSH server on StudentVM2 is working as it should and can accept connections from remote hosts. But there is so much more.
How SSH works – briefly
- 1.
Enter the ssh studentvm2 command.
- 2.
The local host establishes an unencrypted TCP connection to remote host.
- 3.
The remote host sends its own public key to the local host which compares it to the one for that host in ~/.ssh/known_hosts. This authenticates the remote host.
- 4.
The two hosts negotiate the encryption algorithm to use and start it so that all further communications are performed through the encrypted channel.
- 5.
The local host prompts the user for their password and sends it to the remote server over the encrypted channel.
- 6.
The remote server verifies the password is correct and permits the login to proceed.
- 7.
The remote host launches the user’s default shell – usually Bash.
If no user is specified as part of the SSH command in user@host format, the user issuing the command is assumed by SSH to be the user to connect with on the remote host.
As an alternative to using passwords, a Public/Private Key pair (PPKP) could be used. The details of this will be covered later in this chapter. The PPKP may also use an arbitrarily long optional passphrase for an additional level of security.
SSH can multiplex many different concurrent channels over the authenticated connection. This allows tunneling of login sessions and TCP forwarding so that other protocols such as Telnet and the X Window System, that are not normally encrypted can use this encrypted channel.
SSH today is normally implemented with OpenSSH1. Until September of 2000, SSH was encumbered with patents and other proprietary restriction, but those all expired years ago. There are still commercial versions of SSH available but there is no reason to use them on Linux. Fedora, and at least some other distributions I have tried, installs both the client and server by default and allows root access by default.
Public/Private Key Pairs
PPKPs are used to enhance security by – mostly – removing the need for passwords to initiate SSH connections to a remote host. For the user, this is more secure because it eliminates the need to memorize and the temptation to write down long but good passwords.
Each host already has a PPKP that was generated during first boot after install. As we have seen, those key host pairs are stored in the /etc/ssh directory. The host’s public key is swapped at the first SSH connection during the initial handshaking protocols. These host keys are used to positively identify the hosts to each other and are used to launch the initial encryption of the connection so that the authentication sequences are secure.
How PPKPs work
Suppose that I want to send you encrypted messages that only you – and others with the public key – can read. I need to be able to encrypt it and you need to be able to decrypt it. Cryptology texts are full of ways to do this that involve various types of keys and varying levels of security. Shared keys are fine until the key is compromised. I may not know that the key we share has been compromised and will keep sending messages that can be intercepted and read by the very people I might want to keep them from.
- 1.
Create a Public/Private Key Pair.
- 2.
Send the public key to the remote computer which will be decrypting my messages and encrypting the reply messages back to me.
- 3.
Encrypt messages with the private key and send then to the remote computer.
- 4.
The encrypted message is decrypted by the remote computer using public key.
- 5.
To respond, the remote computer encrypts the message using the public key from my host and sends it to my host.
- 6.
The message encrypted by the public key is decrypted on the local host using the private key.
Of course, these messages are the data contained in the TCP packets sent between the computers.
There are some interesting implications from all this. First, anyone who has the public key can decrypt the messages (data packets) that I send. Therefore, I can send my one public key to many different computers and then use SSH to connect to them all using the single private key. I do not need a separate set of keys for each computer.
Another inference that we can make is that anyone with the public key can send me messages. However, only I can initiate a conversation by using the private key. If another host wishes to initiate a conversation, they must create a PPKP and send a copy of their public key to me. So someone cannot just obtain a copy of my public key and then use it to initiate encrypted connections to my host computer.
I could send my public key to the user of the host on the other end in an email, and the user at that end could then append it to their ~/.ssh/authorized_keys file. However, there is a tool that I can use to install my public key on the remote host so long as I have the password of the user account with which I want to communicate. Therefore, without the cooperation of a friendly user at the remote host, or already having my own user account and password, I cannot just push my public key across the network and login to any random remote host.
This is all very nice and secure which is, of course, the intention.
So I am limited to SSH connections to remote hosts on which I have an account and know the original password. This reduces to the student user (or the root user) having an account on both StudentVM1 and StudentVM2 and sending the public key from one host to another.
Experiment 5-2
Perform this experiment as the student user on both StudentVM1 and StudentVM2 hosts.
The host key’s fingerprint and/or the randomart image can be used to verify the validity of a public key for the host. It cannot be used to recreate the original public or private key and it cannot be used for communication. It is used only to verify the validity of the key.
Now that we have generated our key pair, look again at the contents of the ~/.ssh directory for the student user on StudentVM1. You should see two new files: id_rsa which is the private key and id_rsa.pub which is the public key. The .pub extension kind of gives that away.
You could do some tests on the remote host like listing files and so on. Let’s copy a file from StudentVM1 to StudentVM2.
Use the terminal session on StudentVM1 that is logged in to the student user on StudentVM2 via SSH to verify that the file has been copied to StudentVM2. There is a possibility that it has not been copied. If the random.txt file is not present in the student user’s home directory on StudentVM2, it is likely the unintended consequence of the echo statements we added to various Bash configuration files. You can see that line earlier, in which it states that .bashrc is running. Even if you did not encounter this problem, be aware that disruption of the expected protocol stream by added comments like this can cause SSH and SCP to fail without an error.
- 1.
~/.bashrc
- 2.
~/.bash_profile
- 3.
/etc/bashrc
- 4.
/etc/profile
- 5.
/etc/profile.d/myBashConfig.sh
Create a PPKP for the root user on StudentVM1 and copy the public key to StudentVM2. Also create PPKPs for both the root and student users on the StudentVM2 host and copy them to studentVM1. This sets up a situation where users can SSH easily from one host to another.
Even without a passphrase, using a PPKP is more secure than a basic SSH connection using a password.
X-forwarding
We now have SSH working and tested. There is more fun stuff ahead. Let’s start by running a GUI program on the remote host with the display of the program’s window on the local host. Most GUI desktop systems use Wayland2 windowing system, or the X Window System,3 a.k.a, X, as their underlying windowing engines. X-forwarding works in either event because they both use the same protocols.
Experiment 5-3
Perform this experiment as the student user on the StudentVM1 host.
Using X-forwarding via SSH to display the Thunar file manager running on StudentVM2 in a window on the desktop of StudentVM1
Now, from the StudentVM1 desktop, use the SSH connection to StudentVM2 that has X-forwarding enabled. Enter the same command. Move the mouse pointer to see the eyes follow it.
The ampersand (&) after the command specifies that the command is to run in the background; that returns the terminal session to a command prompt while leaving the GUI program to run. That will not affect running the specified program like Thunar or xeyes, but if you run Thunar without the &, the terminal session does not return to a command prompt. You would need to terminate Thunar or log in remotely again to get a command line from which to launch xeyes.
Have some fun with this for a minute or two and then close down both Thunar and xeyes. The xeyes program must be moved or closed from the application bar because it has no window frame to manipulate.
This capability can be both fun and useful.
The X Window System
Because X-forwarding over SSH is a client/server type of operation, let’s look at the details a little more closely.
Tip
Although some people call it X Windows or X-Windows, to be technically and legally correct, it should be called the “X Window System” or just “X”.
It seems pretty clear that a standard SSH connection takes place from the client on the local host to a server on the remote host. We have, in fact, done that over the course of these experiments. But is that also true of X-forwarding over SSH?
To understand this, we need to know more about the X Window System.4 Wikipedia has a rather old article describing the X Window System and some of its history. The short version is that the X Window System is a windowing system for Unix-like operating systems such as Linux. X does not do anything other than provide the primitive graphical tools to create and manipulate windows and objects on a bit-mapped display. It does not impose any aspect of the user interface such as how it looks or how users and application programs can interact with it.
- 1.
We use the mouse on StudentVM1 to select a folder in Thunar.
- 2.
Thunar running on StudentVM2 opens the folder and generates a series of graphical commands that cause the redrawing of the Thunar window. This is a client request to the X server.
- 3.
Those commands are sent to StudentVM1 where the X server translates them into the new images in the Thunar window. This is the X server fulfilling the request from the client.
Most of the time, the server and the client are located on the same host but they can be located on different hosts as seen in the preceding experiment. This is only possible because the client and server functions are separate.
Remote commands
Although using SSH to perform remote commands may sound like logging in to a remote computer using SSH and then typing in commands on the remote Bash shell, there is a significant difference. And that little difference is what makes SSH such a powerful tool.
Let’s start with a simple task like checking the contents of a directory on the remote host.
Experiment 5-4
Perform this experiment as the student user. Our objective is to determine the contents of the student user’s home directory on the remote host.
As the student user on the StudentVM1 host, run the following command. This can only be accomplished without using a password if a PPKP is in place.
So that works as we expect. But try the same command without quotes. You will see that the remote command ends at the semicolon that the local shell uses to delimit the final command. The quotes are required so that the shell can properly send the entire command to the remote host.
Tip Bash shell aliases such as the ll command are not available when using remote commands. So be careful when using scripts containing remote commands to not use command aliases.
Remote backups
The term “remote backups” may be a bit misleading, even with what we now know about running commands remotely. For many years, I used a script to perform backups on my main workstation and several remote hosts. I used remote commands to perform the remote backups to the local workstation.
Making backups of remote hosts is much easier than you might think. We used the tar command to create backups in Volume 2, Chapter 18. We will use it again for this.
Experiment 5-5
Now verify that the tarball /tmp/studentvm2.tgz contains the files we expect. But do it from StudentVM1.
We have now used an SSH remote tar command to create a stream consisting of the backup data from the remote host. That data stream is sent across the SSH connection to STDOUT on the local host where it can be used with other commands through pipes or redirected to a file.
We have now performed a backup of a remote host and stored the backup file on the local host, all with a simple command line program.
Once I had created an easy and elegant method for creating backups of remote hosts using tar and SSH, the next step for me was to create a script that would perform that same backup on several hosts and then set up a cron job or a systemd timer to do those backups every night.
Chapter summary
SSH uses two levels of authentication, first authenticating the hosts themselves and then user authentication. It encrypts the entire session including the authentication and all of the data transmission. SSH is very secure and can be used to transmit data securely over public networks.
SSH features such as remote command execution and data stream transmission over the encrypted connection enable powerful solutions for things like backups using simple tools like tar. SSH also provides X-forwarding so that we can run graphical programs on the remote host with their windows on our local host.
Exercises
- 1.
What are the permissions on the ~/.ssh/id_rsa file? Why?
- 2.
What are the permissions on the ~/.ssh/id_rsa.pub file? Why?
- 3.
The StudentVM1 host should have several user accounts including one named “student1.” If it does not, create that account. As the student1 user on StudentVM1, create a PPKP and copy the public key to the student (NOT student1) account on StudentVM2. SSH to the student account on the StudentVM1 host.
- 4.
You should already have created PPKPs for both root and the student user on both hosts, StudentVM1 and StudentVM2. Copy the public key of the student user on StudentVM1 to the root account of StudentVM2. As the student user on StudentVM1, SSH to root at StudentVM2.
- 5.
Suppose you, as the student user, have created a PPKP and copied the public key to the remote host using your password. A few days or weeks later, as is the policy, you change the password on both the local and remote hosts. Can you still login to the remote host using the PPKP?
- 6.
Write a Bash script on StudentVM1 to back up the /home, /root, and /etc of it and to also back up /home, /root, /etc, and /var of the StudentVM2 host.
- 7.
After you have tested the backup script from the previous exercise, create a cron job or systemd timer to run the script every morning at 2 a.m.
- 8.
When using X-forwarding over SSH, which host is the X server?
- 9.
When using X-forwarding over SSH, which host is the X Client?