Rootkit detection or prevention is not an easily achievable task. Unlike most viruses, rootkits hijack or hook the system internals to improve its invisibility. Usually the race to detection is won by the one who was there first: rootkit or detector. Currently, different techniques for detecting rootkits do exist. Each of them has advantages and inconveniences, so using multiple techniques together and in the right circumstances is usually the best thing to do. Each technology provides different results, so learn to use them for their specific advantage. The most frequently used techniques are described in this chapter—including scanning signatures, call differentiation, and call table address. A basic knowledge of the limitations of these techniques greatly improves their application.
A signature scanner is similar to most anti-virus solutions; it looks at each file for a bit pattern, also known as a signature, which reveals the nature of the software. Table 12-2 lists the pros and cons of signature scanners.
Table 12-2. Advantages and disadvantages of signature scanners
For better results against kernel rootkits, the trick is to always use signature scanning on a disk image instead of locally. Some rootkits may hide their content from user applications due to a kernel compromise. So, running the application from another OS and scanning a dead or not-running partition greatly improves your chance of identifying malware. This inconvenience is due to the fact that in a running system compromise with a kernel rootkit, the rootkit will be able to hide itself on the disk in location where "user-space" applications such as backup solutions and anti-virus solutions don't have read access so won't be able to detect the presence of such malware.
Another method of detecting rootkits is by inspecting calls. An excessively simplified example would be to look every time an application tries to load a kernel module or writes to the syscall address table. These behaviors are pretty uncommon for normal applications but are common during kernel rootkit installation, so detection could be easy. Table 12-3 lists the general pros and cons of inspecting calls.
Table 12-3. Advantages and disadvantages of inspecting calls
Advantages | Disadvantages |
|---|---|
Block the rootkit before installation. | Identifies the faulty application, but not what rootkit it is. |
Catch even unknown rootkits (inspecting calls is not signature-based). | The number of possible entry points for rootkits constantly increases, making this technology less reliable. |
Low impact on system performance, if well designed. | Might require user interaction to choose the right solution; can you trust the user? The decision might be hard to make even for technical people. |
Could be homemade using a kernel debugger such as Softice (watch the performance impact and needed user interaction). | Could be evaded by any new entry point. |
Inspecting calls looks promising on paper, but the technical realization has proven very difficult because the number of possible entry points for a rootkit is growing as hackers try to circumvent this mode of protection. As the number of entry points grows, so does the probability of false negatives and false positives.
Another way to identify rootkits is by their behavior. Rootkits hide files, registries, and open ports. By statically re-implementing some basic functions in your rootkit detector and differentiating the results with the current system output, it is possible to catch rootkit behavior.
For example, parsing the registry file from a raw format to extract all registries (instead of using the Windows API function to do it) allows the user to identify hidden registries by comparing both outputs. This is based on the principle that even if a rootkit hijacks basic APIs, they try to stay as high-level as possible to minimize the need to re-implement a lot of code. Placing a filter on an API that list files is simple (removing an entry), but placing the same filter on a API that reads from a raw device is much harder, since the data post filtering needs to stay valid, and removing a single block may need multiple changes in the data. In Unix, that is equivalent to running a compromised version of ls versus a valid (statically compiled) one and differentiating the results. Table 12-4 lists the general pros and cons of identifying rootkits by their different calls.
Table 12-4. Advantages and disadvantages of call differentiation
Advantages | Disadvantages |
|---|---|
Works on new/unknown rootkits (call differentiation is not signature-based). | Easy to evade (rootkit could hide in plain sight). |
Works only once infected. | Works only once infected. |
Identifies most hidden files/registry/ports. | Could be false positive prone. |
Can be run as needed (low-performance impact). |
The first versions of this type of software were prone to the hide in plain sight problem. Rootkits tend to hide from all processes except themselves (it is hard to read your configuration file or password dump when they are hidden). By adding the detector process name to the exclusion list (what you do not hide from), the rootkit is kind of hiding itself from the differentiating detector by hiding nothing. Since this feature usually depends on the detector executable name, it is a good practice to rename the detector executable with a custom name to bypass this rootkit feature.
On Unix systems, the old way of writing rootkits was to rewrite versions of a basic application such as, ls, netstat, and login. On the Windows platform, this approach was replaced by hooking function calls to add filtering before returning the result. One way to catch this behavior is by looking at the call table for out-of-bound addresses. For example, is the address you are calling for function X out of bounds with the DLL in which function X is supposed to reside? If it does, this might be a bad sign. Table 12-5 lists the pros and cons of looking for hooks.
Finally, one of the oldest ways to check a system is to use filesystem integrity. It's a simple way of detection by getting a hash of all files and comparing the value with the preinfected one. Tripwire is a well-known example of this type of software. Table 12-6 lists the pros and cons of system integrity checking.
Table 12-6. Advantages and disadvantages of system integrity checking
Advantages | Disadvantages |
|---|---|
Fast at uncovering which files changed. | Needs a disk image to work at its best. |
Could be evaded by some kernel rootkits. | Needs to be done generating its base before infection occurs. |
Could uncover most abnormal file changes, rootkit or not. | Needs to be redone after each update or when files change. |
Works best when the system is down and analyzed from a different machine. |