The underlying technology used in a fuzzer greatly affects the results and the time necessary to set up and run it. Before starting to attack a specific application, you must understand what you want to fuzz and under what circumstances. You can then pick the right fuzzer for the job. Let's look at some different types of fuzzers, as well as their strengths and limitations, so you can better choose the right one.
A good example of a block-based fuzzer is Spike (see Writing a Fuzzer with Spike). This kind of fuzzer allows the user to create a framework representing the protocol or file format in a block-based manner, so it can fuzz each block depending on his value's type (the protocol's layout). A block-based structure allows the fuzzer to treat subparts of the protocol like small blocks that we can apply transformation on; size of block, encrypt block, and MD5 of the block. This approach works well for complex binary protocols where multiple consistency checks are done over the data before processing.
Block-Based Fuzzers shows the pros and cons of block-based fuzzers.
Table 22-1. Block-based fuzzers
Block-based fuzzer pros | Block-based fuzzer cons |
|---|---|
Easy to control what you want to attack | Time required for developing a layout for a new protocol |
Can target attack type by value type (e.g., Int attack on size only) | Hard to cover all the cases of a protocol |
Works well with checksum and size values | Reversing protocol of file format is very time-consuming |
Can implement almost any protocol with enough time |
Riot was one of the first fuzzers. While simple in structure, it offers an easy and fast way to attack multiple protocols, often in a matter of minutes. However, since the Riot model does not support blocks, it works well only on the simplest protocols. The Riot model was designed to fuzz many ASCII-based protocols or files. Riot works by detecting natural delimiters in ASCII streams (natural delimiters are characters such as CR, @, ., -, &, white space, Tab, /, and \). Next it replaces the string, between two natural delimiters, with the fuzz string.
For example, given the original string:
GET index.html HTTP/1.0
and a fuzz string of A, Riot generates these requests:
A index.html HTTP/1.0 GET A.html HTTP/1.0 GET index.A HTTP/1.0 GET index.html A/1.0 GET index.html HTTP/A.0 GET index.html HTTP/1.A
Table 22-2 shows the advantages and disadvantages of using Riot.
Table 22-2. Riot pros and cons
Riot pros | Riot cons |
|---|---|
Very quick to adapt to new protocols | Works well only on ASCII protocol |
Offers decent coverage | Becomes useless if one checksum or size is present in the payload |
Very easy to implement | |
Can create a fuzzer automatically from packet capture | |
Can be placed inline as an invisible proxy |
Flipper is a short name for a bit flipper. The bit-flipper fuzzer is mostly used to cover binaries protocols or files. This type of fuzzer technology is based on the assumption that the file header or protocol contains most of the information that affects the program behavior. In most cases of binary protocols, this is true, since the header contains the information for function call number, size, number of arguments, size of the payload, and flags. To keep the file consistency to a maximum, the bit flipper replays the base communication while just flipping one bit at a time. While this process can seem long (in the case of a 50-byte header, there are 400 tests to cover by flipping 1-bit at a time). However, it is important to use multiple base files to be able to reach decent code coverage. Table 22-3 shows the advantages and disadvantages of using a flipper.
Although not a real fuzzer, fault injection can be used to discover exploitable bugs. The main difference between a fuzzer and fault injection is that the fuzzer limits itself to the program entry point (file, network, or environment variable) while fault injection just tries to generate random errors to uncover bugs or test the program's error handling/recovery. Since most fault injection software works in a kind of sandbox, the possible fault injection that can be done varies from network data corruption (random bit flipper) to syscall output corruption. Even if the fault injection finds bugs, there is no insurance that a valid test case will confirm that exploitable conditions exist (without altering the underlying operating system), whereas a normal fuzzer provides an input data set that triggers an exception or a crash. Considering these circumstances, fault injection might be more appropriate when used in conjunction with other tracers as a reliability or faults-resilience testing tool, instead of as a security vulnerability discovery tool.
Known fault-injection applications include Security Innovation's Holodeck (http://www.securityinnovation.com/holodeck/). For more information, check out the book How to Break Software: A Practical Guide to Testing, by James A. Whittaker (Addison-Wesley).
Table 22-4 lists the pros and cons of fault injection.
Table 22-4. Pros and cons of fault injection
Fault injection pros | Fault injection cons |
|---|---|
If well-implemented, can offer more coverage than other fuzzers | May find bugs with no real dataflow |
Fast testing since it usually hooks DLL and syscall to inject errors; no need to pass through the whole transport layer every time | Not security-oriented, but faults-tolerance-oriented |
Integrated application exists | Some fault injectors are only based on random bit flipping |
Provides no warranties of code resistance |
Setting up a test bed is crucial to the success of your fuzzer. The normal setup I use includes three distinct computers, or in the worst case, three virtual machines on one server or laptop. These three systems are identified as the client, the server/target, and the fuzzer. I usually place the fuzzer between the client and the server in a proxy or bridge mode. You always want the fuzzer to be able to gather all the communication in transit during a session and spoof itself as a valid client. Usually it is better to use systems with two interfaces, one for the fuzzer network, and one for management and log gathering. Since fuzzing can generate a lot of noise, I prefer to use a separate Ethernet card or even a serial bus for log output and debug.
The client system is a normal box with one network card plugged into the fuzzer box. The client should have all the applications with access to the server so valid communications can be established.
The fuzzer is the test bed's main player. It has to have all the hardware to be able to pass traffic coming from the client through to the server, so a two-network interface is a must. I usually use another network card or serial port to connect to the server/target for logging and debugging purposes. Also the fuzzer should have tools like a network analyzer (e.g., tcpdump, Wireshark, tcpreplay, tcpflow, Netcat) installed, allowing it to capture, analyze, and replay network communications. It is also beneficial to have a crypto suite such as OpenSSL or stunnel for attacking encrypted protocols and the fuzzer framework. Possessing more than one framework can be useful to cover multiple cases such as block-based fuzzing, bit flipping, and on-the-fly modification.
The server/target is the second most important piece of your test bed. It must have the appropriate tools installed to be able to analyze, trace, and quantify system behavior. These tools must be used in a manner in which they report their log or results directly to the fuzzer system, so all the appropriate data is centralized on the fuzzer. Depending on the OS, the target is going to run multiple tools.
For example, the basic tools you absolutely need to have are a tracer (see http://www.bindview.com/services/razor/utilities/windows/) (ktrace or ptrace) and a fault monitor. These allow you to gather a large quantity of process information without stopping at every serious bug, like a debugger would. However, a good debugger is also needed. Something like SoftICE, GDB, OllyDbg, or WinDbg is beneficial to have around at least for the second phase of analysis, when the interesting cases are reviewed manually. To improve the usability of these debuggers, always load the debug symbol with them.
Some other tools to place on high-end test beds include memory profiler and coverage tools. Tools such as Purify (www-306.ibm.com/software/awdtools/purify/), Valgrind (www.valgrind.org), TrueCoverage, and BoundsChecker can be useful when the test bed tries to perform tests as completely as possible.
Tip
Even if it is possible to add multiple tools to trace the server/target system, always remember that the first goal of a fuzzer is to catch the easy-to-automate bugs, not everything. By that I mean the bugs that fall into the simple categories of buffer overflow, format string, and integer overflow. Complex logical bugs are very hard to find with a fuzzer. Do not overload your system with too many tracers, debuggers, and memory profilers or else the time required to pass through the results will be longer than doing a static analysis, and the main purpose of fuzzing will be lost. These higher-end tools should be used in a second pass to gather more information about a smaller subset of inputs. They should not be used in the first run.
Figure 22-1 shows how the test bed network should be built.
Getting the fuzzer to work is only the first part of the problem. Most modern applications generally have an error-handling feature that allows them to recover from small but exploitable crashes. To be able to gather the information needed to catch these handling cases, you must run the application in a debugger or at least be able to catch and log these cases.
However, most debuggers halt the process as soon as exceptions occur—a behavior that's not very useful when you want to test 10,000 test cases. Small applications such as FaultMonitor(research.eeye.com/html/tools/) or sTrace can be very useful in this situation. They allow you to grab the information you need to evaluate while allowing the application to keep running alone without continuous human supervision. FaultMonitor monitors the fault occurring in the software, while providing output about where the fault occurred and the state of the CPU at that time. Example 22-1 shows FaultMonitor and its output.
Example 22-1. FaultMonitor and its output
C:\Documents and Settings\phiber\Desktop>Faultmon.exe::=-==-===-=====-========-=====-===-==-=:: Faultmon v1.00 research@eeye.com (C) 2006 eEye Digital Security Last Updated: July 31, 2006 ::=-==-===-=====-========-=====-===-==-=:: Usage: FAULTMON {<pid> | <commandline>} [/C] [/A] [/P] [/R] [/S] Where: <pid> is the process ID of a process to debug, or <commandline> is the command line of a program to execute in debug mode /C debug child processes (only valid with <commandline>) /A wait for all processes to terminate (only valid with /C) /P pause whenever a severe first-chance exception occurs (use twice to pause for any first-chance exception) /R repeat intercepted debug output (useful with Dbgview) /S show selectors when an exception is logged C:\Documents and Settings\phiber\Desktop>Faultmon.exe notepad.exeFaultmon.exe: WARNING - could not disable process kill on debugger exit 16:40:00.870 pid=0CBC tid=0834 Created process "(unknown)" at 01000000 16:40:00.870 pid=0CBC tid=0834 Loaded module "ntdll.dll" at 7C900000 16:40:00.930 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\kernel32.dll" at 7C800000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\comdlg32.dll" at 763B0000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\SHLWAPI.dll" at 77F60000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\ADVAPI32.dll" at 77DD0000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\RPCRT4.dll" at 77E70000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\GDI32.dll" at 77F10000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\USER32.dll" at 77D40000 16:40:00.940 pid=0CBC tid=0834 Loaded module "C:\WINDOWS\system32\msvcrt.dll" at 77C10000 16:44:06.593 pid=00D8 tid=0234 EXCEPTION (first-chance) ---------------------------------------------------------------- Exception 000006F7 (unknown) ---------------------------------------------------------------- EAX=0126FA7C: F7 06 00 00 01 00 00 00-00 00 00 00 E1 A4 4E 7C EBX=0126FD38: 8C FD 26 01 F2 79 D4 77-A0 6C D3 77 00 00 00 00 ECX=0126FD38: 8C FD 26 01 F2 79 D4 77-A0 6C D3 77 00 00 00 00 EDX=00000000: ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ESP=0126FA74: A0 6C D3 77 01 00 00 00-F7 06 00 00 01 00 00 00 EBP=0126FACC: 48 FD 26 01 74 04 D4 77-F7 06 00 00 01 00 00 00 ESI=00000000: ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? EDI=77D36CA0: FF FF FF FF 00 00 00 00-86 37 D9 77 00 00 00 00 EIP=7C4EA4E1: E9 66 17 03 00 FF 74 24-08 FF 74 24 08 E8 62 FB --> JMP 7C51BC4C ----------------------------------------------------------------
Notice that FaultMonitor provides the user with the list of exceptions that occur during the test, documenting whether any of the registers were overwritten by an input string or what kind of exception occurred during the test.
sTrace monitors calls and the values passed by these calls. It also includes a simple filtering of calls by category or individual names and some system-call argument decoding. Example 22-2 shows sTrace's output.
Example 22-2. sTrace output
c:\strace -p 3820
...
2689 3820 1584 NtUserValidateHandleSecure (66246, ... ) == 0x1
2690 3820 1584 NtUserValidateHandleSecure (197298, ... ) == 0x1
2691 3820 1584 NtUserQueryWindow (197298, 0, ... ) == 0x978
2692 3820 1584 NtUserQueryWindow (197298, 1, ... ) == 0x980
2693 3820 1584 NtUserValidateHandleSecure (197298, ... ) == 0x1
2694 3820 1584 NtUserValidateHandleSecure (262476, ... ) == 0x1
2695 3820 1584 NtUserQueryWindow (262476, 0, ... ) == 0x600
2696 3820 1584 NtUserQueryWindow (262476, 1, ... ) == 0x604
2697 3820 1584 NtUserValidateHandleSecure (262476, ... ) == 0x1
2698 3820 1584 NtUserValidateHandleSecure (197192, ... ) == 0x1
2699 3820 1584 NtUserQueryWindow (197192, 0, ... ) == 0xb3c
2700 3820 1584 NtUserQueryWindow (197192, 1, ... ) == 0xf74
2701 3820 1584 NtUserValidateHandleSecure (197192, ... ) == 0x1
2702 3820 1584 NtUserValidateHandleSecure (66060, ... ) == 0x1
2703 3820 1584 NtUserQueryWindow (66060, 0, ... ) == 0xd44
2704 3820 1584 NtUserQueryWindow (66060, 1, ... ) == 0xd48
2705 3820 1584 NtUserValidateHandleSecure (66060, ... ) == 0x1
2706 3820 1584 NtUserBuildHwndList (0, 66060, 1, 0, 64, ...
(0x1020e, 0x10210, 0x10212, 0x10214, 0x10216, 0x10218, 0x1021a,
0x1021c, 0x1021e, 0x10220, 0x10222, 0x10224, 0x10226, 0x10228,
0x1022a, 0x1022c, 0x1022e, 0x10230, 0x10232, 0x10234, 0x10236,
0x10238, 0x1023a, 0x1023c, 0x10240, 0x1024c, 0x1024e, 0x10250,
0x10252, 0x10254, 0x10256, 0x10258, 0x1025a, 0x1025c, 0x1025e,
0x10260, 0x10262, 0x10264, 0x10266, 0x10268, 0x1026a, 0x1026c,
0x1026e, 0x10270, 0x1024a, 0x1, ), 46, ) == 0x0
2707 3820 1584 NtUserValidateHandleSecure (66062, ... ) == 0x1
2708 3820 1584 NtUserQueryWindow (66062, 0, ... ) == 0xd44
2709 3820 1584 NtUserQueryWindow (66062, 1, ... ) == 0xd48
2710 3820 1584 NtUserValidateHandleSecure (66064, ... ) == 0x1
2711 3820 1584 NtUserQueryWindow (66064, 0, ... ) == 0xd44
2712 3820 1584 NtUserQueryWindow (66064, 1, ... ) == 0xd48
2713 3820 1584 NtUserValidateHandleSecure (66066, ... ) == 0x1
...Notice again that sTrace provides all the calls and a list of arguments to those calls. This kind of tracing can be very useful when you are looking for specific calls that can trigger underlying Windows bugs.