So, you have gone and made your computer available to the Internet. In fact, it is a lot more available than you ever expected. With all the hackers and malware floating around in cyberspace, it is a wonder nothing happened sooner. Maybe it has, but you never knew it. It has sure happened to me more than once.
Imagine a scenario in which the latest and greatest Microsoft Windows vulnerability goes public. Apparently there are services running on your computer, available to the world, which have a bug allowing anyone or anything with the know-how to take control of your computer. Not even hours after (or maybe weeks before) this news is made public, exploits circulate the Internet to demonstrate how to best take advantage of this new vulnerability. Unfortunately, you were a little late to patch your server and you have a real problem on your hands. First, your server has inexplicably rebooted several times during the course of the night. Then you received a wake-up call at 8 A.M. from your friendly ISP informing you that your monthly bandwidth has been grossly exceeded because of a sudden increase in FTP traffic. Apparently, your tiny corner of the Internet has become the latest distribution center for stolen source code. You have been hacked: pull the plug!
That scenario was just a bit extreme. A far more likely situation is that a worm, taking advantage of the same new vulnerability, has turned your poor workstation into a spam relay or mindless drone in a massive IRC botnet. Either way, something with malicious intent is running on your system. What can you do to find out more about what is happening? This is where the field of computer forensics comes into play.
There are many tools out there designed to give you more insight into what exactly is happening behind the scenes on your server. This chapter covers some freely available power tools that allow you to dig a bit deeper into your system to discover what that worm or hacker has been doing with all the time they have had free reign.
By the way, after familiarizing yourself with some of these tools, it is a good idea to save a collection of your favorite forensic tools onto a CD. If you are ever called upon to analyze an incident, it is very likely that the system you are examining is offline, as pulling the plug is the only real way to make sure that the machine is not being attacked, monitored, or used while you are checking it out. If the system or situation is particularly sensitive, it may be worth taking the extra step of imaging the tainted drive and examining it on a purpose-built forensics platform. Several of these are available as bootable, CD-based Linux distributions, designed to examine, in a forensically safe manner, images of many different operating systems ranging from Microsoft Windows to Solaris. I recommend FIRE (http://fire.dmzs.com/), FCCU (http://www.d-fence.be/), and Snarl (http://snarl.eecue.com/).
For any Internet-based intrusion into your system, some sort of communication has to happen over the network. Unless a rootkit of some sort was installed that is designed specifically to hide network connections, there are utilities that allow you to have a sneak peek at exactly what your system is doing on the network. Be particularly interested in what ports it is listening on and who it may already be talking to. Most Unix-based operating systems and Microsoft Windows versions come equipped with a tool called Netstat. If you think your system has been backdoored and you are stuck for a place to start, this is it; Netstat is probably already installed on your machine. Since usage for the more common Linux Netstat and Windows Netstat differ, I will address them separately.
Usage of the Netstat on different Unixes often varies, but they should be similar. Consult the Netstat manpages to find out the appropriate flags for your variant.
On Linux, I call netstat with three different flags; none of them require any arguments:
-aTells netstat to show all sockets. This is important because you are primarily concerned with detecting listeners that you did not know about.
-nTells netstat that it does not need to resolve hostnames and service names. This speeds up the queries and provides raw port numbers, so you do not need to go through the extra step of looking up the service details in /etc/services. If you really want to know what the hosts resolve to, then leave this flag out.
-pThis is the money flag. It tells you exactly what process is using the socket in question. If you notice a strange listening socket that also happens to be opened by a process that you are not familiar with, then further investigation is required. To use this flag, you must be root.
Example 21-1 shows the command netstat -anp run as root on a Debian server (output is truncated).
Example 21-1. netstat –anp run as root on a Debian server
# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 2085/nc
tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN 5983/cyrmaster
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 21733/cupsd
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 6120/master
tcp6 0 0 :::80 :::* LISTEN 17702/apache2
tcp6 0 0 :::22 :::* LISTEN 6236/sshd
tcp6 0 48 ::ffff:10.157.6.131:22 ::ffff:207.17.136.44:3493
ESTABLISHED2052/sshd: br [priv
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 5 [ ] DGRAM 1059604 21815/syslogd /dev/log
...The output is separated into two different sections. One, labeled Active Internet Connections, is what you are primarily concerned with. The other section, labeled Active UNIX domain sockets, contains sockets used for local communication between processes on the system. Primary indicators for a backdoor are not usually seen here.
Active Internet Connections begins with a line labeling each field, and then proceeds to list all active sockets for the system. You do not care about all of the fields, but most of them are important:
ProtoThe Layer 4 protocol used for the socket; e.g., TCP or UDP.
Local AddressThe listening address, formatted in a way appropriate to the protocol. In TCP or UDP, for example, it is formatted ip:port when using the -n flag. Without the -n flag, it is formatted as hostname:service if they both resolve. Some special cases for the format of this field are worth noting:
An IP of 0.0.0.0 means that the socket is listening on all interfaces. This is often the case.
An IP of 127.0.0.1 means that the socket is listening on the Loopback interface. In most cases, this socket is not available to the Internet.
Foreign AddressThe address of the remote host, formatted in the same manner as the Local Address field.
It is worth noting that the port portion of the Foreign Address field may contain a *. In this case, the socket will accept connections from any originating port.
StateThe status of the socket in question. Be most concerned with sockets in the LISTEN state. These are unattached and waiting for a connection. It is also worth noting the sockets labeled ESTABLISHED. These may indicate a backdoor that is actually being used.
PID/Program NameThe process identifier and program name of the process that has opened the socket. This is important because it allows you to see exactly what is running on the port. If you see something other than a mail daemon running on port 25, it might be worth checking out.
Now that the fields have been explained, look again at Example 21-1. Do you see anything out of the ordinary there? I do. Check out line number 4. What on earth is Netcat (nc) doing listening on port 9090? Using the process ID that you gain by using the -p flag, now dig a bit further.
$ ps 2085
PID TTY STAT TIME COMMAND
2085 pts/0 S 0:00 nc -l -p 9090 -e /bin/bashThis is definitely a backdoor of some sort (check out the next section if this does not look familiar to you.) If the ps command reveals nothing significant about the process and you are still unsure of what it does, you can always try to interact with the socket directly. For example:
$nc localhost 9090iduid=0(root) gid=0(root) groups=0(root)uname -aLinux american 2.6.12-10-386 #1 Fri Jul 7 01:12:29 UTC 2006 i686 GNU/Linux^C$
No good can come of this. Time to pull the plug!
As with Netstat for Linux, I usually call Netstat for Windows with three flags. These might look familiar:
-aTells netstat to show all sockets. This is important because we are primarily concerned with detecting listeners that we did not know about.
-nTells netstat that it does not need to resolve hostnames and service names. This speeds up the queries and gives us raw port numbers, so we do not need to go through the extra step of looking up the service details in /etc/services ourselves. If you really want to know what the hosts resolve to, then leave this flag out.
-bIs like the Linux Netstat's -p; this is the money flag. It tells us exactly what process is using the socket in question. If we notice a strange listening socket that also happens to be opened by a process that we are not familiar with, then further investigation is required. To use this flag, you must be Administrator.
You can use the -v flag in conjunction with the -b flag to get more information about the libraries involved in each process that creates a socket, but I have yet to find a case where this would be useful.
-oDisplays only the process identifier (PID) of the process that created the socket. This information is displayed when using the -b flag, also. Use this flag only if you would rather look up the process yourself than wait for the output from -b.
To get a feel for the format of the Windows Netstat output, first run it without the -b flag:
C:\>netstat -an
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 1196
TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1692
TCP 127.0.0.1:1025 0.0.0.0:0 LISTENING 164
TCP 192.168.0.100:3278 207.17.137.201:443 CLOSE_WAIT 2996
UDP 0.0.0.0:9090 *:* 784
UDP 127.0.0.1:123 *:* 1232
UDP 192.168.0.100:138 *:* 4The output of this command is very similar to Linux netcat. For an explanation of each field, see Chapter 11. The key difference here is that there is no longer an Active Unix domain sockets section. Now dig a little bit further and see what these sockets are about:
C:\>netstat -anb
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 1196
c:\windows\system32\WS2_32.dll
C:\WINDOWS\system32\RPCRT4.dll
c:\windows\system32\rpcss.dll
C:\WINDOWS\system32\svchost.exe
C:\WINDOWS\system32\ADVAPI32.dll
[svchost.exe]
TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1692
[winvnc.exe]
TCP 127.0.0.1:1025 0.0.0.0:0 LISTENING 164
[alg.exe]
UDP 0.0.0.0:9090 *:* 784
[BOSERVER.EXE]
UDP 127.0.0.1:123 *:* 1232
c:\windows\system32\WS2_32.dll
c:\windows\system32\w32time.dll
ntdll.dll
C:\WINDOWS\system32\kernel32.dll
[svchost.exe]
... and so on.With the -b flag, netstat gets a good bit noisier. The format is actually the same as without the flag, but in square brackets [] underneath each socket is a list containing the name of the process that created the socket, and above that, the libraries involved. With Windows, there tends to be a lot more ports open to the world in a default installation. It certainly helps to familiarize yourself with these before scrutinizing each and every service. There is one here that should look just the least bit suspicious. My naming convention from Chapter 11 does not really help hide the fact that there is a BO2k Server running on UDP port 9090. If you did not already know what this was, it would be time to break out the Sysinternals Process Explorer to learn a bit more about what this is doing (see Sysinternals).