Keep it simple.Security is definitely a hot topic both inside and outside the computer world. It can be difficult to distinguish legitimate threats from basic paranoia, but as anyone who has connected to a high-speed connection and monitored the logs knows, these days there are armies of servers out there trying to attack you.
Even though other operating systems and products seem to get the majority of the press for their security breaches, Ubuntu users aren’t completely in the clear. Even though Ubuntu has good security out of the box, the moment you set up new services you risk opening holes to attack. This chapter discusses some common security practices and simple steps you can take to keep your Ubuntu server secure. Just in case you still get attacked, I also include a section at the end on how to respond to a security breach.
There is a saying in security circles: “Security is a process, not a product.” What that means is that despite what your vendor might tell you, you can’t solve all your security problems with some appliance or software. Instead, you find real security when you follow sound security principles and develop sound security procedures. While I cover some specific tools and options you can use to increase your system’s security later in the chapter, there’s no way I can discuss how to lock down every major service under Ubuntu. These principles, though, are something that you can apply no matter what software you might run:
Keep it simple.
Another saying you will hear in security circles is “Complexity is the enemy of security.” The more complex a system, the more difficult it is to understand every part of it and the greater the likelihood that the security of some aspect of the system was overlooked. Whenever you design a system, try to keep the number of interoperating pieces as small as you can. Not only will it help with security, it will help with troubleshooting and overall administration as well.
Follow the principle of least privilege.
The principle of least privilege is the idea that programs and people should operate with the lowest possible level of power. This is the concept behind the separation of root and the regular users. Since most daily tasks don’t require full system privileges, give users fewer privileges. Programs like Apache and Postfix follow this principle; they use the root privileges only when they are absolutely necessary, and then the bulk of the work is done via child processes owned by a different user. When these practices are in place, and you do get attacked, the amount of damage an attacker can do is limited.
Provide layers of protection.
Some refer to this as “defense in depth.” The best security occurs in layers. Slapping a firewall in front of your servers won’t automatically make them secure, but it will help increase their security. Instead, you want multiple layers of defense, such as a firewall between you and the outside world, a local software firewall, strong passwords, and sudo roles.
Avoid security by obscurity.
On the surface it might seem as if moving the SSH server from port 22 to port 257 would add extra security. After all, no one will think to look for it there. Unfortunately, steps like this slow down, but don’t stop, an attacker. The real danger of these sorts of security methods is that they create a false sense of security. This isn’t to say that moving ports around and using other means of obscurity are completely bad, just that they should be recognized for what they are—things that only slow down attackers and that must be combined with other security procedures.
Keep on top of security patches.
You can have all sorts of security procedures in place and still be attacked if a vulnerability is found in a service and you fail to patch it. It’s important to monitor security updates and use Ubuntu’s package management to keep your systems up to date.
Out of the box Ubuntu implements a number of practices to make the default install more secure. One of these is the disabling of the root account and the use of sudo for superuser privileges. The sudo program provides a much more robust set of features to increase user privileges compared to the traditional su program. Here is a list of some of the more interesting features of sudo:
Uses the user’s password for authentication
When the sudo command is run and the user is prompted for a password, each user enters his or her own password instead of the password for the user he or she wants to become. This helps with security because it means that you can give people superuser privileges without having to tell them the root password (if it is enabled) or the password for any other users on the system. It also means that a user can’t directly log in to the system as the root user since there is no password—since all Linux machines have a root user, it’s the most common account someone will try to brute force.
Limitation on superuser access
With sudo a user doesn’t have to have complete root access. Instead, you can define a list of programs a user can execute with superuser privileges. With this feature you can better follow the principle of least privilege. If a user really needs root privileges only to run apache2ctl, why give the user root access over the entire system? With sudo you can easily limit the user’s access strictly to apache2ctl.
Support for group-based and host-based access
The sudo configuration file allows you to put users into groups and then assign access to that group. If you find that you have a number of different sudo rules for the same set of users, this makes it much easier to organize the file and add or remove users from the group. You can also define rules based on the host. With this feature you can maintain a single sudoers file (sudo configuration file) that can be copied to all hosts on your network (or even shared via LDAP).
Auto-expiration of sudo access
Once you pass the password check once, you can continue to run sudo commands and sudo won’t prompt you for a password for a configurable amount of time. This certainly saves time when you need to run multiple sudo commands in a row and provides extra protection in case you forget to lock your terminal and leave your desk.
Every time users run sudo they generate a new log entry in /var/log/auth.log that lists the time, the users who ran sudo, and what commands they ran. This gives an administrator a nice forensics trail. Also, sudo can send the administrator an e-mail whenever a user fails the password check.
Can configure passwordless access to rules
While you wouldn’t necessarily want to enable this for every sudo rule, there is sometimes a need for a script (particularly cron scripts) to gain root access to run a particular command. Since it’s a cron script, it’s much easier if it can run without your having to code support for the interactive password prompt. It might be tempting to give complete passwordless sudo access to a user, but I recommend you limit rules like this to individual commands.
sudo’s configuration file can be found at /etc/sudoers. Ubuntu provides a basic file by default that allows the root user to do anything as any other user and allows members of the admin group to become root (the user you create at install time is automatically added to this group). Your natural inclination might be to open the file with your favorite text editor; however, this is not recommended. The sudo package provides a tool called visudo that you should use whenever you want to make changes to the file, so to view and edit the /etc/sudoers file, type
$ sudo visudo
The reason you want to use visudo is that it automatically checks your sudoers file for mistakes. Since a mistake in the sudoers file could potentially lock you out of root access, this syntax check is pretty important. If you do make a mistake, visudo will tell you about it after you save and exit. You will have the option to go back and fix your mistake, exit without saving, or ignore its warnings and save anyway (which is not recommended).
While the basics of sudoers syntax are pretty straightforward, the full set of features and syntax for things like user or command groups can get complicated rather quickly, especially without an example to work from. Luckily you can access the sudoers manual by typing man sudoers. This manual defines this file’s syntax and gives a number of examples you can use for your own sudoers file. Instead of documenting that manual again here, to start let’s look at two lines from the default /etc/sudoers file and break down each of the fields:
root ALL=(ALL) ALL
%admin ALL=(ALL) ALL
The first column defines which user or group this rule applies to. To refer to a user, you can just list the user, or you can list multiple users separated by commas. To reference a group add the % sign in front of the group’s name. The second column before the = sign defines which hosts this rule applies to. In this case it is set to ALL, which means it applies to all hosts. The value within the parentheses sets which user this sudo rule will be run as. In this case it is also set to ALL so it can be run as any user. The final column defines which commands the user can run. In this example it is set to ALL so any command can be run.
It’s easier to see how all of these options fit in when you see an example that isn’t set to ALL. Let’s assume I have a Web server named web1. I have an administrator with a username of jorge who manages only this Web server. I don’t want to give him full root access to the machine because he has a history of accidents with the rm -rf command. All I really want to give him is the ability to use apache2ctl so he can reload and restart Apache on this one server. I copy my sudoers file to all of the servers on my network, so I need to restrict the rule to web1 only. The resulting rule would look like this:
jorge web1 = (root) /usr/sbin/apache2ctl
Notice that I specified the full path to apache2ctl. If I had just listed apache2ctl without the path, I would have opened up a way to full root access on the system. The user could have created a bash script in his home directory named apache2ctl and run whatever he wanted as root.
Now let’s say that the jorge user never really used apache2ctl except to reload changes he made to configuration files. He wanted to automate this process and wrote a script to deploy his configuration files, but he still has to manually log in to run apache2ctl because it asks for a password. To remove the password requirement for this rule, add the NOPASSWD: statement before the command:
jorge web1 = (root) NOPASSWD: /usr/sbin/apache2ctl
Keep in mind that you want to restrict how often you use NOPASSWD since it bypasses one of sudo’s main security measures. You also want to avoid using it if the command is set to ALL if you can help it. Even in this case I may not want the full apache2ctl command available without a password. If I wanted to restrict this further, I could create my own shell script that ran /usr/sbin/apache2ctl graceful and then give the jorge user NOPASSWD access to that script instead.
As your sudoers file gets larger and more complicated, or as you deploy it to a larger number of servers, you will see that it easily becomes disorganized and difficult to manage. A new developer gets hired and needs sudo access to a machine, and you find yourself poring over line after line in the sudoers file looking for all of the different rules that might apply. Aliases save you from that trouble and allow you to define groups for each of the columns in an entry. The basic syntax for each of these aliases is as follows:
User_Alias ALIASNAME = user1,user2,user3
Host_Alias ALIASNAME = host1,host2,host3
Runas_Alias ALIASNAME = user4,user5,user6
Cmnd_Alias ALIASNAME = /bin/command1,/sbin/command2
There are some restrictions given to the alias name. You can use only uppercase letters, 0–9, and the _ symbol for them. That not only helps you distinguish them from regular users, it helps sudo distinguish them as well.
So let’s extend our previous scenario to incorporate groups. Instead of just the jorge user, I have a number of users, jorge, allan, and ben, who work on our Web cluster, web1, web2, and web3. I want to grant them access both to apache2ctl and also some of the other useful Apache commands Ubuntu includes, such as a2enmod, a2dismod, a2ensite, and a2dissite, so they can enable and disable modules and sites. Finally, I want them to be able to run any command as the webadmin and apache users—two special users we set up just for Web administrators. Here are the resulting rules I would add to /etc/sudoers:
User_Alias WEB_ADMIN = jorge,allan,ben
Host_Alias WEB_CLUSTER1 = web1,web2,web3
Cmnd_Alias WEB_COMMANDS = /usr/sbin/apache2ctl, \
/usr/sbin/a2enmod, \
/usr/sbin/a2dismod, \
/usr/sbin/a2ensite, \
/usr/sbin/a2edissite
RunAs_Alias WEB_ACCOUNTS = webadmin,apache
WEB_ADMIN WEB_CLUSTER1 = (root) WEB_COMMANDS
WEB_ADMIN WEB_CLUSTER1 = (WEB_ACCOUNTS) ALL
Notice with the WEB_COMMANDS alias I can span multiple lines as long as I use a \ at the end of the line. With these aliases in place, if we hire new users, all I have to do is add them to the WEB_ADMIN alias. If we add a fourth Web server to the cluster, I need to update only the WEB_CLUSTER1 alias.
The UNIX permissions model has long been used to lock down access to users and programs. Even though it works well, there are still areas where extra access control can come in handy. For instance, many services still run as the root user, and therefore if they are exploited, the attacker potentially can run commands throughout the rest of the system as the root user. There are a number of ways to combat this problem, including sandboxes, chroot jails, and so on, but Ubuntu has included a system called AppArmor, installed by default, that adds access control to specific system services.
AppArmor is based on the security principle of least privilege; that is, it attempts to restrict programs to the minimal set of permissions they need to function. It works through a series of rules assigned to particular programs. These rules define, for instance, which files or directories a program is allowed to read and write to or only read from. When an application that is being managed by AppArmor violates these access controls, AppArmor steps in and prevents it and logs the event. A number of services include AppArmor profiles that are enforced by default, and more are being added in each Ubuntu release. In addition to the default profiles, the universe repository has an apparmor-profiles package you can install to add more profiles for other services. Once you learn the syntax for AppArmor rules, you can even add your own profiles.
Probably the simplest way to see how AppArmor works is to use an example program. The BIND DNS server is one program that is automatically managed by AppArmor under Ubuntu, so first I install the BIND package with sudo apt-get install bind9. Once the package is installed, I can use the aa-status program to see that AppArmor is already managing it:
$ sudo aa-status
apparmor module is loaded.
5 profiles are loaded.
5 profiles are in enforce mode.
/sbin/dhclient3
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/connman/scripts/dhclient-script
/usr/sbin/named
/usr/sbin/tcpdump
0 profiles are in complain mode.
2 processes have profiles defined.
1 processes are in enforce mode :
/usr/sbin/named (5020)
0 processes are in complain mode.
1 processes are unconfined but have a profile defined.
/sbin/dhclient3 (607)
Here you can see that the /usr/sbin/named profile is loaded and in enforce mode, and that my currently running /usr/sbin/named process (PID 5020) is being managed by AppArmor.
The AppArmor profiles are stored within /etc/apparmor.d/ and are named after the binary they manage. For instance, the profile for /usr/sbin/named is located at /etc/apparmor.d/usr.sbin.named. If you look at the contents of the file, you can get an idea of how AppArmor profiles work and what sort of protection they provide:
# vim:syntax=apparmor
# Last Modified: Fri Jun 1 16:43:22 2007
#include <tunables/global>
/usr/sbin/named {
#include <abstractions/base>
#include <abstractions/nameservice>
capability net_bind_service,
capability setgid,
capability setuid,
capability sys_chroot,
# /etc/bind should be read-only for bind
# /var/lib/bind is for dynamically updated zone (and journal) files.
# /var/cache/bind is for slave/stub data, since we're not the origin
#of it.
# See /usr/share/doc/bind9/README.Debian.gz
/etc/bind/** r,
/var/lib/bind/** rw,
/var/lib/bind/ rw,
/var/cache/bind/** rw,
/var/cache/bind/ rw,
# some people like to put logs in /var/log/named/
/var/log/named/** rw,
# dnscvsutil package
/var/lib/dnscvsutil/compiled/** rw,
/proc/net/if_inet6 r,
/usr/sbin/named mr,
/var/run/bind/run/named.pid w,
# support for resolvconf
/var/run/bind/named.options r,
}
For instance, take a look at the following excerpt from that file:
/etc/bind/** r,
/var/lib/bind/** rw,
/var/lib/bind/ rw,
/var/cache/bind/** rw,
/var/cache/bind/ rw,
The syntax is pretty straightforward for these files. First there is a file or directory path, followed by the permissions that are allowed. Globs are also allowed, so, for instance, /etc/bind/** applies to all of the files below the /etc/bind directory recursively. A single * would apply only to files within the current directory. In the case of that rule you can see that /usr/sbin/named is allowed only to read files in that directory and not write there. This makes sense, since that directory contains only BIND configuration files—the named program should never need to write there. The second line in the excerpt allows named to read and write to files or directories under /var/lib/bind/. This also makes sense because BIND might (among other things) store slave zone files here, and since those files are written to every time the zone changes, named needs permission to write there.
You might have noticed that the aa-status output mentions two modes: enforce and complain modes. In enforce mode, AppArmor actively blocks any attempts by a program to violate its profile. In complain mode, AppArmor simply logs the attempt but allows it to happen. The aa-enforce and aa-complain programs allow you to change a profile to be in enforce or complain mode, respectively. So if my /usr/sbin/named program did need to write to a file in /etc/bind or some other directory that wasn’t allowed, I could either modify the AppArmor profile to allow it or I could set it to complain mode:
$ sudo aa-complain /usr/sbin/named
Setting /usr/sbin/named to complain mode
If later on I decided that I wanted the rule to be enforced again, I would use the aa-enforce command in the same way:
$ sudo aa-enforce /usr/sbin/named
Setting /usr/sbin/named to enforce mode
If I had decided to modify the default rule set at /etc/apparmor.d/usr .sbin.named, I would need to be sure to reload AppArmor so it would see the changes. You can run AppArmor’s init script and pass it the reload option to accomplish this:
$ sudo /etc/init.d/apparmor reload
Be careful when you modify AppArmor rules. When you first start to modify rules, you might want to set that particular rule into complain mode and then monitor /var/log/syslog for any violations. For instance, if /usr/sbin/named were in enforce mode and I had commented out the line in the /usr/sbin/named profile that granted read access to /etc/bind/**, then reloaded AppArmor and restarted BIND, not only would BIND not start (since it couldn’t read its config files), I would get a nice log entry in /var/log/syslog from the kernel to report the denied attempt:
Jan 7 19:03:02 kickseed kernel: [ 2311.120236]
audit(1231383782.081:3): type=1503 operation="inode_permission"
requested_mask="::r" denied_mask="::r" name="/etc/bind/named.conf"
pid=5225 profile="/usr/sbin/named" namespace="default"
The following list details the common directories and files AppArmor uses, including where it stores configuration files and where it logs:
/etc/apparmor/
This directory contains the main configuration files for the AppArmor program, but note that it does not contain AppArmor rules.
/etc/apparmor.d/
You will find all of the AppArmor rules under this directory along with subdirectories that contain different sets of include files to which certain rule sets refer.
/etc/init.d/apparmor
This is the AppArmor init script. By default AppArmor is enabled.
/var/log/apparmor/
AppArmor stores its logs under this directory.
/var/log/syslog
When an AppArmor rule is violated in either enforce or complain mode, the kernel generates a log entry under the standard system log.
If you are going to run services on your servers, these days it’s a safe bet that one of them will be SSH. SSH provides a secure, encrypted channel between your desktop and a server so that you can run commands and manage the machine without having to physically be there with a keyboard and mouse. Even though SSH was designed with security at the forefront, poor management of the service can open you up to attack. In fact, one of the most common ways that Linux servers are attacked at the moment is via SSH brute-force attacks. I cover how to manage those attacks, but first I discuss a few other methods to enhance the security of SSH.
The /etc/ssh/sshd_config file is where you will find all of the settings for the SSH server. The default Ubuntu sshd_config file is pretty secure out of the box, as it allows only SSH protocol 2, uses privilege separation, and allows authentication keys to be used. The only questionable setting is PermitRootLogin yes. This option allows the root user to log in via SSH. In a way this setting is useless on a default Ubuntu install, since the root account is disabled, but if you decide to enable the root account, you might want to set this option to no and run sudo service ssh reload to save the settings. This way you force users to log in with their regular accounts and sudo up to root, and you also prevent a user from being able to guess the root password and gain access.
If there is a weak link in SSH security, password authentication would probably be it. I know plenty of people who have been hacked simply because of a weak user password. There are many brute-force SSH scripts active in the wild that constantly scan for new machines and run through a dictionary full of passwords until one works. I know of a honeypot server intentionally set with weak passwords that was hacked and used as part of a botnet within hours of showing up online.
The good news is that you don’t need password authentication to log in to an SSH server. SSH supports key-based authentication. In this approach the user generates a public and private key. The public key is then placed in a special file on the remote server. When the user logs in, these keys are used to authenticate the user instead of a password. It’s certainly more convenient to be able to log in to a machine without typing a password every time, although if you want an extra layer of security, you can set a passphrase on your keys as well.
It is relatively simple to set up key-based authentication. In the following example we have a user named ubuntu on desktop1 who wants to set up key authentication on server1. The first step is to use the ssh-keygen program to create an RSA public and private key on desktop1. At each prompt you can press Enter to accept the defaults.
ubuntu@desktop1:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa):
Created directory '/home/ubuntu/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
The key fingerprint is:
91:ae:0c:ff:16:a2:67:98:19:34:71:5b:71:e3:d2:2c ubuntu@ubuntu
The script creates the keys in the .ssh directory under your home directory, in this case /home/ubuntu/.ssh. The private key and public key are named id_rsa and id_rsa.pub respectively. It’s very important (especially if you chose an empty passphrase) to keep the private key (id_rsa) safe! If anyone else gets access to this file, he or she can copy it and will be able to log in to any machines you have set up with this key.
Once you have created the keys, the next step is to copy the id_rsa.pub key to the server and then append it to the ~/.ssh/authorized_keys file. There are a number of ways you can do this. You could SSH into the remote machine, open ~/.ssh/authorized_keys with a text editor, and paste in the contents of id_rsa.pub, for instance. Here are two other ways to do this. The first way is simple to understand but takes multiple steps. The second method does the entire operation in one command. In method one I use the scp command to copy the id_rsa.pub file to the home directory on the remote server, then I append it to the ~/.ssh/authorized_keys file:
ubuntu@desktop1:~$ scp ~/.ssh/id_rsa.pub
ubuntu@server1:/home/ubuntu/id_rsa-desktop1.pub
ubuntu@desktop1:~$ ssh ubuntu@server1
ubuntu@server1:~$ mkdir ~/.ssh
ubuntu@server1:~$ chmod 700 ~/.ssh
ubuntu@server1:~$ cat ~/id_rsa-desktop1.pub >>
~/.ssh/authorized_keys
It turns out that you can skip all of the preceding steps with an included script named ssh-copy-id, which safely copies your local public key to the remote host:
ubuntu@desktop1:~$ ssh-copy-id ubuntu@server1
Once you have keys set up on a machine, you should be able to log in without a password prompt, unless you set a passphrase for your key, in which case you will need to type it. After your keys work, you might want to disable SSH password authentication altogether. Just make sure that your SSH keys work first or you could lock yourself out! To disable password authentication, edit /etc/ssh/sshd_config and locate the line that says
#PasswordAuthentication yes
Uncomment that line and set it to no:
PasswordAuthentication no
Finally, run sudo service sshd reload to load the new change.
As mentioned earlier in this chapter, SSH brute-force attacks have become a very common threat to Linux servers. Even if your password is hard to guess, unless you impose strong password restrictions on the server, there’s no way of knowing that every other user has a strong password. The best way to combat SSH brute-force attacks is to simply disable password authentication and use SSH keys. Unfortunately, that isn’t an option for every administrator. If you must use password authentication, there is another way to protect against these attacks: a package named denyhosts.
The way that denyhosts works is to monitor for failed SSH logins. When a host attempts to log in either as a user that doesn’t exist or too many times, that host is added to /etc/hosts.deny and blocked from future SSH access. Also, if a host tries to log in as a valid user but fails too many times, the host is blocked.
A number of administrators use this tool or tools like it to protect against brute-force attacks and like the results, but I find it hard to recommend. I mention it so that you know it is available, since you might disagree with my opinion. In case you do decide to deploy it, here are some things to watch out for:
Any program that automatically modifies firewall (or TCP wrappers) rules is dangerous. If an attacker can detect that such a tool exists, he or she can remotely modify your firewall rules. What happens if the attacker can appear to come from a different host, such as your desktop, and lock you out?
Set your thresholds carefully. Even with reasonably large thresholds, such as ten failed attempts for a valid user, you might still lock out valid users who forgot their password. I’ve even seen this happen with a user who set up keys and had a cron script log in and perform various tasks. When the server got overloaded and the SSH connections timed out, the failed SSH connections crossed the threshold and locked out the script.
Set whitelists for trusted hosts. Be sure to add any hosts or networks that you can’t risk being blocked into /etc/hosts.allow. Be sure to keep your whitelists up-to-date with new hosts or networks. Just keep in mind that if attackers do manage to hack into another machine on any of these networks, they will be able to attack these machines.
Botnets know about denyhosts and can work around it. It’s true that denyhosts makes a brute-force attack more difficult, but a large-enough botnet can work around this problem by having a particular host attack only a few times, or shift to a different host once the first is locked out.
One of the most common ways to protect machines on a network is with a firewall. Essentially a firewall gives you the ability to restrict access to services over the network. With a firewall you could limit access to SSH, for instance, to hosts only within your internal network, while allowing HTTP access to everyone. There are two major types of firewalls used in an organization: so-called hardware and software firewalls. A hardware firewall is generally a stand-alone machine that sits between your hosts and another network (often the Internet). This machine is then configured with a set of rules to control what access is allowed. A software firewall is a program that is run on a host itself and has a similar ability to restrict access, except in this case it applies only to that specific host.
Many organizations deploy both hardware and software firewalls, which is in line with the “Provide layers of protection” security principle. The hardware firewalls often double as the gateways for a particular network and help restrict access in and out of the network, while software firewalls on each host help reinforce the rules from the hardware firewall and can provide additional protection from hosts inside their own network—something a separate hardware firewall can’t do. Honestly, many “hardware firewalls” out there are simply stand-alone machines that run Linux and use the same software firewall tools. In this chapter I discuss how to use Ubuntu’s tools to set up a secure software firewall on your server.
Traditionally, firewall rules under Linux required that one delve deeply into the dark arts of the iptables program. iptables (and before that, ipchains) is a program included with Linux distributions that works with the Linux kernel to evaluate and potentially block packets based on rules that you define. In addition to standard port and host blocking, iptables supports stateful packet matching, which means it monitors and can identify traffic as belonging to a particular pair of hosts and can keep track of the state of that connection. You can then define rules that activate based on these states.
You can (and I have done so) create very complicated iptables rule sets, but honestly, the way that the iptables syntax is constructed, even the most basic rules can seem complex at times. This complexity fights against the “Keep it simple” security principle so that the beginner administrator either ends up disabling the firewall altogether or relies on some long set of iptables rules found on the Internet—rules the admin doesn’t understand and therefore can’t easily debug. The advanced administrator soldiers on and either develops a minimal set of rules or learns the dark art of stateful packet matching and develops a long rule set that is bound to have mistakes.
At some point the initial pride of writing heavy-duty, complicated iptables rules wears off and you just wish it were simpler. Eventually your common iptables administrator discovers the OpenBSD pf firewall tool and fills with envy. In pf you have simple, easy-to-understand syntax that is still secure. Luckily for Ubuntu administrators, there is now a similar tool, ufw, that aims to simplify firewall administration by providing a front end to iptables commands.
The ufw program will be installed by default on your server but will be disabled. Since a default Ubuntu install has no external network services enabled, there really isn’t anything for a firewall to protect. As you start to add services, however, you will want to enable the firewall and add rules.
The basic set of ufw commands is pretty straightforward. If you run ufw -h, you will get a help page that describes the main ufw commands, but if you want full syntax information, you should type man ufw to read the full manual page. First I identify the main commands and then provide some examples.
enable and disable
These commands enable and disable ufw, respectively. By default ufw is disabled, so if you wanted to enable it, you would type sudo ufw enable.
status
If you aren’t sure whether or not your firewall is enabled, type sudo ufw status to check. If ufw is enabled and you have any rules defined with ufw, the status command will also output all of your rules.
default
A very important command to consider is the default command. This command defines the default policy of your firewall, as in whether by default all packets are allowed or denied. The general consensus is that a firewall is more secure if you deny all packets by default, and then enable services as you need to. That way, if you start a new service (or worse, a user starts a service) and you forget to set firewall rules for it, by default it will be blocked. So to deny by default, you would type sudo ufw default deny. To allow by default, type sudo ufw default allow. Note that ufw will deny by default unless you change it.
logging
This command toggles whether or not you want your firewall to dump logs of anything it blocks along with anything against your default policy. To enable logging, type sudo ufw logging on. To disable it, type sudo ufw logging off.
allow and deny
These are the commands that you will run more often than not, as they define your firewall rules. The arguments they accept are more involved because they can define complex firewall rules, so I discuss their syntax in the rule syntax section that follows.
delete allow and delete deny
These commands will undo a particular firewall rule you have created. Whenever you want to remove a rule, you copy the same command you used to create the rule and then add delete to the very beginning.
Note: Deny by Default
For all of my examples, I’m going to assume your firewall denies by default. When you deny by default (sudo ufw default deny), you construct your rules so that they are focused on what access to allow. If instead you decide to allow by default, your set of rules has to be focused on what access to deny. Generally speaking, you will find that denying by default results in a shorter list of rules for a secure firewall.
A basic ufw rule takes a port or service as an argument. To open port 53 (used for DNS servers), you would type
$ sudo ufw allow 53
ufw also accepts service names that are defined in the /etc/services file instead of specific ports. If you look in the /etc/services file, you can see that port 53 TCP and UDP is set to the domain service. So another way to state this rule is
$ sudo ufw allow domain
This name-based access makes it really simple to define rules because you don’t need to concern yourself as much with ports as with service names. For instance, to open access to a mail server, you could type
$ sudo ufw allow smtp
The SMTP service operates only over TCP, not UDP, and ufw will see this in the /etc/services file and allow only port 25 TCP traffic through. You can specify TCP or UDP on the command line as well, so the equivalent to the preceding command would be
$ sudo ufw allow 25/tcp
Once you have created some rules, you can view them with the status command:
$ sudo ufw status
Status: active
To Action From
__ ______ ____
53 ALLOW Anywhere
25/tcp ALLOW Anywhere
When you view the ufw status after you have set some basic rules, you will notice that by default these ports are open to any IP address. This may be exactly what you want. You may, however, want to lock down specific services further so that only certain hosts can access them. To do this, you need to extend the basic ufw rules.
A good example of why you might want to limit based on IP address is an intranet site, like an internal wiki. If you use a wiki for your internal documentation, you probably don’t want the entire world to read it. You might even host a wiki just for a particular group of users and want to restrict access so that only their network can see it. When you define more advanced ufw rules, you must use the extended ufw syntax. This syntax requires you to specify the protocol and port number explicitly, so you can’t use the same shortcuts as in the simpler commands. Here’s an example command to limit Web access (port 80) to just the 10.1.1.0 network:
$ sudo ufw allow proto tcp from 10.1.1.0/24 to any port 80
Let’s break this command down. The proto tcp section defines whether this rule applies to TCP or UDP. I limit what networks this rule applies to with from 10.1.1.0/24. The to any port 80 section of the command says that this rule applies to port 80 on any destination address on the machine.
Even if you deny by default, there might be circumstances when you also add a deny rule. For instance, let’s say that you are running an external SMTP (mail) server and you notice that a host inside your network at 10.1.1.75 appears to be infected with a virus and is flooding your mail server with invalid messages. To block just that IP address, you would type
$ sudo ufw deny proto tcp from 10.1.1.75 to any port 25
If you wanted to block all packets from that host, not just SMTP, you wouldn’t need to define the proto or to arguments:
$ sudo ufw deny from 10.1.1.75
Later on, once the virus has been removed and the host is back to normal, you can remove the rule with
$ sudo ufw delete deny proto tcp from 10.1.1.75 to any port 25
or, if you dropped all packets from the host, this command would remove it:
$ sudo ufw delete deny from 10.1.1.75
Note: How to Undo a Rule
Notice that when I wanted to undo a rule, I used the delete command. At first you might logically conclude that you just need to write an allow command so the host can connect again. This works, but when you do it, ufw just changes your existing rule, so you will have an unnecessary rule in your list. When you want to undo a rule, just delete it.
The following is a list of start-to-finish ufw commands to set up a firewall for a particular service. I assume you are starting from the default state, that is, ufw is disabled. Also, I assume you will probably want to manage your server remotely with SSH even if it’s a Web, mail, or DNS server, so I add rules to enable SSH in each example.
Note: About Remote Firewall Management
If this is the first time you have enabled ufw, and you plan to set this up remotely over the network, you should be careful about your steps. It’s very easy to make a mistake and lock yourself out. Specifically, when ufw is enabled, it flushes all connection data, so if you manage the server over SSH, your connection will be closed, and if you deny by default and haven’t set up an SSH rule yet, when ufw is enabled, you will be locked out. One simple safeguard you can put in place is a cron job that disables ufw every 15 minutes or so. That way, if you make a mistake and lock yourself out, you just have to wait at most 15 minutes for the firewall to be reset. To do this, add the following line to your /etc/crontab file:
*/15 * * * * root ufw disable
Of course, the downside to this is that every 15 minutes while you are tweaking your firewall, ufw will be disabled and you will have to remember to enable it. Still, it’s better than being locked out of the system completely. Just remember to delete the crontab rule once you are finished tweaking. On Ubuntu 8.10 and later, ufw actually warns you if you enable ufw while using SSH and prompts you before it enables.
I’m assuming you will probably want to have SSH enabled on just about any server you manage. Note the order in which I run the commands here, as I enable ufw at the very end. That way I don’t risk locking myself out because the SSH rule is defined before ufw is enabled:
$ sudo ufw allow ssh
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
DNS is an interesting case because DNS clients use UDP ports for queries, but DNS servers use TCP for actions like zone transfers. Fortunately, with AppArmor you don’t have to remember such details:
$ sudo ufw allow ssh
$ sudo ufw allow domain
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
53 ALLOW Anywhere
Here I open ports for both HTTP (80) and HTTPS (443), but if you don’t use HTTPS, you can remove that particular rule from the list:
$ sudo ufw allow ssh
$ sudo ufw allow www
$ sudo ufw allow https
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
80 ALLOW Anywhere
443 ALLOW Anywhere
Here I open port 25 to allow incoming SMTP connections to a mail server:
$ sudo ufw allow ssh
$ sudo ufw allow smtp
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
25/tcp ALLOW Anywhere
To simplify things, I list rules to enable POP2, POP3, and POP3 with SSL, IMAP2, IMAP3, and IMAP with SSL, since many administrators end up supporting all of them on the same server.
$ sudo ufw allow ssh
$ sudo ufw allow pop2
$ sudo ufw allow pop3
$ sudo ufw allow pop3s
$ sudo ufw allow imap2
$ sudo ufw allow imap3
$ sudo ufw allow imaps
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
109 ALLOW Anywhere
110 ALLOW Anywhere
995 ALLOW Anywhere
143 ALLOW Anywhere
220 ALLOW Anywhere
993 ALLOW Anywhere
This example uses the default MySQL ports. Of course, if you have moved MySQL to listen on a different port, you will have to manually specify the port to open.
$ sudo ufw allow ssh
$ sudo ufw allow mysql
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
3306 ALLOW Anywhere
This example uses the default PostgreSQL ports. Of course, if you have moved PostgreSQL to listen on a different port, you will have to manually specify the port to open.
$ sudo ufw allow ssh
$ sudo ufw allow postgresql
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
5432 ALLOW Anywhere
Samba is a little trickier to open because it listens on a set of ports and none of them are labeled in /etc/services with “Samba.”
$ sudo ufw allow ssh
$ sudo ufw allow netbios-ns
$ sudo ufw allow netbios-dgm
$ sudo ufw allow netbios-ssn
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
137 ALLOW Anywhere
138 ALLOW Anywhere
139 ALLOW Anywhere
NFS is a little trickier to firewall off than most other services because the connections don’t necessarily use a defined set of ports. As a result, it can be difficult to open a range of ports for NFS that will work long-term. The simplest solution, if you want to enable a firewall on an NFS server, is to deny by default and then allow access to all ports from specific NFS clients. If you don’t want to add a firewall rule for each individual host because there are many, you might consider putting all NFS clients on their own subnet and then allowing that subnet. I show two examples. The first allows all access from the 10.1.1.7, 10.1.1.8, and 10.1.1.9 hosts. The second example opens up access for the entire 10.1.2.0/24 subnet:
$ sudo ufw allow ssh
$ sudo ufw allow from 10.1.1.7
$ sudo ufw allow from 10.1.1.8
$ sudo ufw allow from 10.1.1.9
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
Anywhere ALLOW 10.1.1.7
Anywhere ALLOW 10.1.1.8
Anywhere ALLOW 10.1.1.9
Here are the steps to allow all of 10.1.2.0/24 access to NFS:
$ sudo ufw allow ssh
$ sudo ufw allow from 10.1.2.0/24
$ sudo ufw default deny
$ sudo ufw enable
$ sudo ufw status
Status: active
To Action From
__ ______ ____
22 ALLOW Anywhere
Anywhere ALLOW 10.1.2.0/24
For the most part you should be able to set up a firewall using ufw without worrying about configuration files. The ufw program will update its configuration files for you, so that if you set it to be enabled from the command line, you won’t have to tweak anything else. That being said, you might be interested in how ufw works behind the scenes, so I list ufw’s file conventions here:
/etc/ufw/
This directory contains all of the configuration files for ufw, including /etc/ufw/ufw.conf, the main configuration file. The only default setting in that file, though, defines whether the firewall is enabled at boot—something you can set with ufw itself.
/etc/ufw/before.rules and /etc/ufw/before6.rules
These files contain a set of IPv4 and IPv6 iptables rules, respectively, that ufw will set before any ufw rules are enabled. Advanced iptables users can refer to these files if they are curious about ufw’s behavior. If you really know what you are doing and want to modify this behavior with your own iptables rules, you can add them to the ufw-before-input, ufw-before-output, or ufw-before-forward chains, depending on which chain they belong to. If you don’t know what I’m talking about, then don’t worry; the average user shouldn’t need to edit these files.
/etc/ufw/after.rules and /etc/ufw/after6.rules
These files are like the before.rules and before6.rules files, except they are loaded after ufw rules are enabled. Again, if you really know iptables, you might want to set up specific iptables rules in this file to be started after ufw.
/etc/init.d/ufw
This is the init script for ufw. Generally speaking, you shouldn’t need to touch this script because you can add and delete rules and enable and disable ufw from the command line.
/etc/defaults/ufw
Like other init scripts, ufw has a file under /etc/defaults that defines the environment variables it uses when it starts. There are some settings in this file that are of interest to more advanced administrators; for example, you would come here to enable IPv6 support or add or delete extra connection tracking modules for iptables.
/lib/ufw/user.rules and /lib/ufw/user6.rules
When you define your own ufw rules, they end up in the user.rules file, or user6.rules file for IPv6 rules. If you want to know exactly what iptables command a particular ufw rule creates, you can look here. Generally speaking, you do not want to edit these files directly.
If you have enabled logging, ufw will dump its logs to the standard /var/log/syslog file. If logging is enabled, each connection attempt that is against your ufw policy will be logged here, so you can see the source host and port, the protocol, and the destination host and port for each connection attempt.
Once you have set up a firewall and locked down your system, how can you tell whether the system has been compromised? One way is to set up an intrusion detection system (IDS). If you think of your computer system like a house, your firewall and file permissions could be thought of as locks on the windows and doors. Think of an intrusion detection system as a burglar alarm—its job is not to prevent someone from breaking in, but instead to alert you when it happens.
There are a number of different intrusion detection systems out there, and most of the time you hear about network intrusion detection systems that sniff network traffic and look for suspicious activity. In this case I’m not talking about that sort of IDS, but instead a system to detect that an attacker has intruded into a particular server. One of the oldest and most common of these types of systems is Tripwire. Tripwire maintains a database of information about core files on the system. Once the database has been created, Tripwire scans the system once per day and e-mails you a report. If any of the files in the database has been altered, Tripwire alerts you in its report. Most of the files in Tripwire’s database are files that are common to replace with Trojan horse programs. Others, such as the /etc/passwd file, are files that only root can change. This means that if any of the files change, and you know you and your staff didn’t change it, you can be pretty confident of some sort of system breach.
Because Tripwire detects a breach based on files being different from the version in its database, Tripwire can’t detect if your system was attacked before Tripwire was installed. In addition, the effectiveness of Tripwire is based on the integrity of its database. If attackers can write to that database, they can update it with signatures for hacked versions of system files and continue undetected. Because of all of this, it’s important to install Tripwire as soon as possible on the system, preferably at install time. Also, if possible, you might consider switching to single-user mode beforehand (type sudo init 1 in a console). Note that single-user mode works only if you are physically logged in to the machine (no SSH). In single-user mode you can be sure that no other users can interfere with the initial Tripwire install.
Tripwire is packaged by Ubuntu; however, it is not ready to use out of the box. You must tweak its policy database and think about how you will store the signature database securely before the install is complete. First, install the Tripwire package:
$ sudo apt-get install tripwire
If you don’t have an MTA (Mail Transport Agent) installed, the Tripwire package will add one as a dependency. As the Tripwire package installs, you will be asked a number of questions about keys. Tripwire uses two different keys to sign files and ensure that they have not been altered. The first key is a site-wide key that you might use for all of the servers on a particular network. The second key is a local key that is unique for this particular machine. If you do not yet have a site key or you aren’t sure what this means, answer Yes to create a site key. Next you get a similar prompt about a local key. Unless you already have a local Tripwire key, answer Yes here as well so the installer can create one. At the next prompt, answer Yes to rebuild the Tripwire configuration file, and Yes one final time to re-create the policy file. Finally, enter passphrases to use for the site and local keys. Be sure to note what passphrases you selected, because there’s no method to retrieve or reset them if you forget.
The Tripwire policy file is located at /etc/tripwire/twpol.txt and defines all of the files and directories Tripwire will monitor along with what information to monitor and how to respond to changes in each file. The default twpol.txt file is a good starting place, but it contains a few files and directories that you will want to remove to avoid a lot of false positives every time you run a Tripwire check. Open the file in a text editor and remove the lines as indicated for each of the following files:
/etc/rc.boot
Ubuntu doesn’t have this file, so remove the line that says
/etc/rc.boot -> $(SEC_BIN) ;
/proc
Files in the /proc directory change constantly, so you will get tons of false alarms if this is scanned. Delete the following line from the file:
/proc -> $(Device) ;
/root
There is a large section of the twpol.txt file that lists files under the /root directory. The problem is that by default these files don’t exist, so they will generate false positives. Locate the section of the file that looks like this:
# These files change the behavior of the root account
(
rulename = "Root config files",
severity = 100
)
{
/root -> $(SEC_CRIT) ; # Catch all additions to /root
Delete the references to all of the files under the /root directory except for /root/.bashrc. The resulting segment of the twpol.txt file looks like this:
# These files change the behavior of the root account
(
rulename = "Root config files",
severity = 100
)
{
/root -> $(SEC_CRIT) ; # Catch all additions to /root
/root/.bashrc -> $(SEC_CONFIG) ;
}
Save all of your changes, and then use the twadmin tool to update the policy file:
$ sudo twadmin -m P /etc/tripwire/twpol.txt
Once the policy file has been changed, you are ready to initialize the database. Note that the system’s state at this point in time is what Tripwire considers the gold standard, so you want to initialize the database as soon after the system has been created as possible. Here is the command to initialize Tripwire’s database:
$ sudo tripwire -m i
Don’t worry about any errors you see about the /var/lib/tripwire/foo.twd file not existing—this is the database you are creating now. Once the database has been initialized, you should store it somewhere safe, because if attackers get root privileges they could update the database and hide their tracks. There are a few ways to secure the database. One, you could store the database on a floppy disk with the physical write-protect bit set and change the /etc/tripwire/twcfg.txt file to point to that new location. Alternatively, you could set up an NFS server that you consider secure to host all of the Tripwire database files and set all of the databases as read-only.
Since floppies are getting hard to come by, I describe how to set this up with NFS. Let’s assume that your Tripwire database is at /var/lib/tripwire/ host1.twd. You have an NFS server at 10.1.1.7 and have copied the host1 .twd file at /mnt/tripwire/host1/. Then you would use sudo chmod 400 /mnt/tripwire/host1/host1.twd to ensure that the file could not be changed. Since Tripwire checks are run as the root user, you will probably have to disable root squashing on the NFS server, at least whenever you need to update the database. If you wanted additional security, you could also modify the /etc/exports file so that this share could be mounted only as read-only. NFS server administration is out of the scope of this chapter, but for more information on NFS server configuration, check the NFS section of Chapter 5. With NFS configured, create a directory called /mnt/ tripwire on your local machine and mount the NFS share:
$ sudo mkdir /mnt/tripwire
$ sudo mount -o ro 10.1.1.7:/mnt/tripwire/host1/ /mnt/tripwire
If you get an error that this is an unknown file system type, be sure that you have the nfs-common package installed. Once the share mounts successfully, add a new line to /etc/fstab to make sure that it mounts automatically each time the system boots:
10.1.1.7:/mnt/tripwire/host1 /mnt/tripwire nfs defaults,ro 0 0
Now if you run sudo ls -l /mnt/tripwire/, you should be able to see the /mnt/tripwire/host1.twd file (the filename will be named after your host’s name, of course). Edit the /etc/tripwire/twcfg.txt file as root and change the part of the file that reads
DBFILE =/var/lib/tripwire/$(HOSTNAME).twd
to
DBFILE =/mnt/tripwire/$(HOSTNAME).twd
Finally, you need to re-create the encrypted Tripwire configuration file:
$ sudo twadmin -m F -S /etc/tripwire/site.key /etc/tripwire/
twcfg.txt
Once the Tripwire database is in a safe location, your base Tripwire configuration is complete. Now any time you want to check the system, you can run
$ sudo tripwire --check
and Tripwire will output a report that lists any files that have changed. In addition to the output on the screen, Tripwire stores all of its reports in date-stamped files under /var/lib/tripwire/reports/. By default, Tripwire runs a system check every night and e-mails you a report.
The very first time that you run a sudo tripwire --check you will probably notice that it complains because you modified /etc/tripwire/ twcfg.txt and a few other Tripwire files. Now and then, such as when you update major packages on the system, you will end up legitimately updating files that Tripwire scans, and unless you update the Tripwire database to note these changes, you will get false positives at every nightly scan. In fact, you might want to run a Tripwire check after you do major package updates to make sure that you catch any changes and update them immediately.
To update the Tripwire database, first you must set it so that you can write to the .twd file. If you set this up on an NFS share as in the example, you will need to go to the NFS server and run sudo chmod a+w /mnt/tripwire/host1/host1.twd (replace that path with the path to the file you are exporting). In addition, you need to disable root squashing and set the share to rw during the short time the database is updated. You will also need to remount the NFS share as rw. Once the NFS server is ready, go to the host itself and locate the report that contains the errors you want to override. This will probably be the newest file in /var/lib/tripwire/report. Then run the following Tripwire command to update the database:
$ sudo tripwire -m u -r /var/lib/tripwire/report/
hostname-20090107-190736.twr
Replace the filename under here with the name of your report. The update tool outputs the report to the screen in the root user’s default text editor (probably vi, unless you set the EDITOR environment variable to something else). If this is the default vi editor, you should be able to use the arrow keys (or the standard H, J, K, and L keys) to move throughout the report. Each item slated to be updated in the database will be on a line with [x] at the beginning. As long as that x is inside the box, that file will be updated. If you do not want to update a particular file, move the cursor over that x and type r and then the spacebar to remove it. When you are finished editing the file, hit :wq. You will be prompted for the local key, and then the database will be updated.
Once this command is finished and the database is updated, you must go back to the NFS server and reinstate all of the permissions and other protections you put in place to make this file read-only.
The following list details the common directories and files Tripwire uses, including where it stores configuration files and where it logs.
/etc/tripwire/
This directory contains all of the main configuration files used by Tripwire.
/etc/tripwire/*-local.key and /etc/tripwire/site.key
These files are your local and site keys, respectively. The local key will be named after the localhost.
/etc/tripwire/tw.cfg and /etc/tripwire/twcfg.txt
The tw.cfg is the encrypted database of Tripwire settings, and the twcfg.txt is the plain-text version. Whenever you want to change Tripwire settings, such as the location of the database file, you will make the changes in twcfg.txt and then use twadmin to update the encrypted database.
/etc/tripwire/tw.pol and /etc/tripwire/twpol.txt
These files define the policy that Tripwire uses when it scans the file system. This policy includes which files to scan, what attributes of those files to pay attention to, and with what severity to rate any changes to those files. The tw.pol file is the encrypted database, and the twpol.txt file is the plain-text file you edit to make changes. Once you have made changes to twpol.txt, you must run twadmin to update the tw.pol file.
/var/lib/tripwire/
This directory is the default location for the Tripwire database. Generally, you will end up moving the database file to another, more secure, read-only location.
/var/lib/tripwire/reports
When you run sudo tripwire --check (or when it runs automatically every night), a new report file is generated and stored in this directory. This provides a good history of how core files have changed on the system.
Tripwire logs to the standard system log. To see only Tripwire logs, you could run sudo grep tripwire /var/log/syslog.
Most of this chapter focuses on how to protect your systems so that they can’t be breached by an attacker, but what do you do when an attacker succeeds? Here I provide an overview of how to prepare for and respond to a successful attack.
Preparation before an attack occurs is just as important as the actions you take when it occurs. Even if you are naturally cool and calm during a crisis, there’s a good chance other members of your team won’t be, so a plan you have thought through when you are calm will be better than a plan you have thought up at the last minute with upper management breathing down your neck.
Before you develop any other responses, the first thing you should decide is under what circumstances you will wish to prosecute an attacker. If you are running a home office, that answer might be never. If you are part of a large organization, your company’s legal department might have to answer the question for you. In either case it’s important to have an idea of what circumstances will prompt prosecution, because it will define the rest of the steps you can take. Generally, investigators want to collect untainted evidence, and if you and your team have been touching a bunch of files on the system, their job will be that much harder. How you respond (and how you set up a system) so you can prosecute effectively will vary depending on your location, so if at all possible, consult an attorney.
Another question you should answer before an attack occurs is what you do the moment you have confirmed that a host has been attacked. There are different schools of thought on this, but I believe that the moment you detect an attack, you should immediately pull the power from the server. If the host is a virtual machine that supports snapshots, take a snapshot, then power off the VM. The reason I advocate this approach is that while there can be valuable data in RAM on the system, every command you run and every file you touch on the system potentially erases forensic clues you could use later. Plus, if the attacker has installed a root kit, you can’t even trust any output from the running machine—you could have Trojan versions of ps, bash, and lsmod, for instance, that mask the attacker’s existence.
Once the power has been pulled, do whatever you can to ensure that the machine doesn’t boot back up until you have been able to image all of the partitions on the system from a rescue disc. That way you can then perform forensic analysis on the image without overwriting the original evidence. Plus, once you have an image to work from, you can consider redeploying the server. If the host is a VM and you were able to take a snapshot, you have even more data to work from. Create a copy of the entire VM, snapshot and all. Then you can potentially replay the time you discovered the attack over and over and run tools on the running snapshot image without fear of corrupting data. If you have the space, consider creating two images. One is a gold image that you put away and don’t touch, and the other is an image that you use for any forensic analysis you might perform. When you have multiple images, if you make a mistake and accidentally write to one during your analysis, you will at least have the gold image to copy from.
Another thing to consider before a crisis occurs is whether and when you should rebuild a server. The best practice is to rebuild a server whenever there has been a breach. It can be easy, at least if the attacker was sloppy, to prove he or she did install a root kit if you see the software out in the open, but unless you are skilled at forensic analysis it can be difficult to prove an attacker didn’t install a root kit or some sort of Trojan horse on the system. A root kit can hide all sorts of things from the administrator, so unless you are absolutely sure there is no root kit, rebuild the machine.
How you go about rebuilding the server might be decided on a case-by-case basis. Some servers (particularly those in a cluster) often can be rebuilt from scratch without a thought. Other servers, such as large database or e-mail servers that aren’t in a cluster, can be more difficult because they hold data you need to transfer to the new host. These types of machines might have to go into quarantine until you can make sure that the data can be trusted. To be safe, you might even have to try to track down when the attack occurred and roll back the files on the system from a previous backup. Also, you might need to keep the machine in quarantine until you can track down how the attacker got in and patch the hole before risking another intrusion.
Once you have a valid image of the system’s partitions, you might want to perform some sort of forensic analysis on it. Computer forensics is a vast topic and it can take years of work for you to become proficient. That having been said, even if you aren’t a skilled forensics expert, you might want to try your hand at identifying how the attacker got in.
One basic method of forensic analysis is simply to take the image of your attacked server to another host, mount it loopback and read-only, and then look around the mounted system for clues. For instance, if I had an image of a partition on an external USB drive mounted at /media/disk1/ and the image itself was at /media/disk1/web1-sda1.img, I could use the following command to mount the disk at /mnt/temp:
$ sudo mkdir /mnt/temp
$ sudo mount -o loop,ro /media/disk1/web1-sda1.img /mnt/temp
If you are ready for more advanced forensic analysis, I recommend you check out Sleuth Kit (http://sleuthkit.org). Sleuth Kit is a complete set of forensics tools including a Web-based front end called Autopsy. These tools are packaged in Ubuntu as sleuthkit and autopsy, so you can install them on an Ubuntu desktop with your preferred package manager. Once both tools are installed, type autopsy in a terminal to start the program and then follow the instructions on the screen to see where Autopsy stores its files (/var/lib/autopsy by default) and what URL to open on a Web browser to use Autopsy (http://localhost:9999/autopsy by default). That URL will display the default Autopsy page as shown in Figure 6-1, and from there you can navigate through the tool, start a new investigation, and add images to scan. For more information on how to use Autopsy, read the official Autopsy user’s guide at http://wiki.sleuthkit.org/index.php?title=Autopsy_User’s_Guide.
Another useful forensics tool is chkrootkit. This program can check a file system for common root kits and then output a report. This tool is also packaged for Ubuntu with the package name chkrootkit. Note that you generally don’t want to run this on a live system because you will potentially overwrite evidence. Instead, mount an image somewhere on your system (for instance, in this example I mount the root file system image under /mnt/temp), and then point chkrootkit to it:
$ sudo chkrootkit -r /mnt/temp
Ultimately, complete forensics on a host could take days, weeks, or even months to complete, depending on your experience, the nature of the attack, and how thorough you want to be. Even if you decide to just rebuild the host and already know how the attacker got in, it’s worth experimenting with these forensics tools as they will provide you with greater insight into how your system works long term.
