Recall that debuggers can be used to set breakpoints or to single-step through a process in order to aid the malware analyst in reverse-engineering. However, when these operations are performed in a debugger, they modify the code in the process. Several anti-debugging techniques are used by malware to detect this sort of debugger behavior: INT scanning, checksum checks, and timing checks.
INT 3 is the software interrupt used by debuggers to
temporarily replace an instruction in a running program and to call the debug exception
handler—a basic mechanism to set a breakpoint. The opcode for INT
3 is 0xCC. Whenever you use a debugger to set a
breakpoint, it modifies the code by inserting a 0xCC.
In addition to the specific INT 3 instruction, an INT
immediate can set any interrupt, including 3
(immediate can be a register, such as EAX).
The INT
immediate instruction uses two opcodes:
0xCD
value. This 2-byte opcode is less commonly
used by debuggers.
One common anti-debugging technique has a process scan its own code for an INT 3 modification by searching the code for the 0xCC opcode, as shown in Example 16-6.
Example 16-6. Scanning code for breakpoints
call $+5 pop edi sub edi, 5 mov ecx, 400h mov eax, 0CCh repne scasb jz DebuggerDetected
This code begins with a call, followed by a pop that puts EIP into EDI. EDI is then adjusted
to the start of the code. The code is then scanned for 0xCC
bytes. If a 0xCC byte is found, it knows that a debugger is
present. This technique can be overcome by using hardware breakpoints instead of software
breakpoints.
Malware can calculate a checksum on a section of its code to accomplish the same goal as
scanning for interrupts. Instead of scanning for 0xCC, this check
simply performs a cyclic redundancy check (CRC) or a MD5 checksum of the opcodes in the
malware.
This technique is less common than scanning, but it’s equally effective. Look for the malware to be iterating over its internal instructions followed by a comparison to an expected value.
This technique can be overcome by using hardware breakpoints or by manually modifying the execution path with the debugger at runtime.
Timing checks are one of the most popular ways for malware to detect debuggers because processes run more slowly when being debugged. For example, single-stepping through a program substantially slows execution speed.
There are a couple of ways to use timing checks to detect a debugger:
Record a timestamp, perform a couple of operations, take another timestamp, and then compare the two timestamps. If there is a lag, you can assume the presence of a debugger.
Take a timestamp before and after raising an exception. If a process is not being debugged, the exception will be handled really quickly; a debugger will handle the exception much more slowly. By default, most debuggers require human intervention in order to handle exceptions, which causes enormous delay. While many debuggers allow you to ignore exceptions and pass them to the program, there will still be a sizable delay in such cases.
The most common timing check method uses the rdtsc
instruction (opcode 0x0F31), which returns the count of the
number of ticks since the last system reboot as a 64-bit value placed into EDX:EAX. Malware will
simply execute this instruction twice and compare the difference between the two readings.
Example 16-7 shows a real malware sample using the rdtsc technique.
Example 16-7. The rdtsc timing technique
rdtsc xor ecx, ecx add ecx, eax rdtsc sub eax, ecx cmp eax, 0xFFF ❶ jb NoDebuggerDetected rdtsc push eax ❷ ret
The malware checks to see if the difference between the two calls to rdtsc is greater than 0xFFF at ❶, and if too much time has elapsed, the conditional jump will not
be taken. If the jump is not taken, rdtsc is called again, and
the result is pushed onto the stack at ❷, which will
cause the return to take the execution to a random location.
Two Windows API functions are used like rdtsc in order to
perform an anti-debugging timing check. This method relies on the fact that processors have
high-resolution performance counters—registers that store counts of activities performed in
the processor. QueryPerformanceCounter can be called to query
this counter twice in order to get a time difference for use in a comparison. If too much time has
passed between the two calls, the assumption is that a debugger is being used.
The function GetTickCount returns the number of
milliseconds that have elapsed since the last system reboot. (Due to the size allocated for this
counter, it rolls over after 49.7 days.) An example of GetTickCount in practice is shown in Example 16-8.
Example 16-8. GetTickCount timing technique
a =GetTickCount(); MaliciousActivityFunction(); b =GetTickCount(); delta = b-a; if ((delta) > 0x1A) { //Debugger Detected } else { //Debugger Not Found }
All of the timing attacks we’ve discussed can be found during debugging or static analysis by identifying two successive calls to these functions followed by a comparison. These checks should catch a debugger only if you are single-stepping or setting breakpoints between the two calls used to capture the time delta. Therefore, the easiest way to avoid detection by timing is to run through these checks and set a breakpoint just after them, and then start your single-stepping again. If that is not an option, simply modify the result of the comparison to force the jump that you want to be taken.