Malware can use several techniques to interfere with normal debugger operation: thread local storage (TLS) callbacks, exceptions, and interrupt insertion. These techniques try to disrupt the program’s execution only if it is under the control of a debugger.
You might think that when you load a program into a debugger, it will pause at the first instruction the program executes, but this is not always the case. Most debuggers start at the program’s entry point as defined by the PE header. A TLS callback can be used to execute code before the entry point and therefore execute secretly in a debugger. If you rely only on the use of a debugger, you could miss certain malware functionality, as the TLS callback can run as soon as it is loaded into the debugger.
TLS is a Windows storage class in which a data object is not an automatic stack variable, yet
is local to each thread that runs the code. Basically, TLS allows each thread to maintain a
different value for a variable declared using TLS. When TLS is implemented by an executable, the code will typically contain a
.tls section in the PE header, as shown in Figure 16-1. TLS supports callback functions for
initialization and termination of TLS data objects. Windows executes these functions before running
code at the normal start of a program.
TLS callbacks can be discovered by viewing the .tls section
using PEview. You should immediately suspect anti-debugging if you see a .tls section, as normal programs typically do not use this section.
Analysis of TLS callbacks is easy with IDA Pro. Once IDA Pro has finished its analysis, you
can view the entry points for a binary by pressing CTRL-E to
display all entry points to the program, including TLS callbacks, as shown in Figure 16-2. All TLS callback functions have their labels
prepended with TlsCallback. You can browse to the callback
function in IDA Pro by double-clicking the function name.
TLS callbacks can be handled within a debugger, though sometimes debuggers will run the TLS callback before breaking at the initial entry point. To avoid this problem, change the debugger’s settings. For example, if you’re using OllyDbg, you can have it pause before the TLS callback by selecting Options ▶ Debugging Options ▶ Events and setting System breakpoint as the place for the first pause, as shown in Figure 16-3.
Note
OllyDbg 2.0 has more breaking capabilities than version 1.1; for example, it can pause at the start of a TLS callback. Also, WinDbg always breaks at the system breakpoint before the TLS callbacks.
Because TLS callbacks are well known, malware uses them less frequently than in the past. Not
many legitimate applications use TLS callbacks, so a .tls section
in an executable can stand out.
As discussed earlier, interrupts generate exceptions that are used by the debugger to perform operations like breakpoints. In Chapter 15, you learned how to set up an SEH to achieve an unconventional jump. The modification of the SEH chain applies to both anti-disassembly and anti-debugging. In this section, we will skip the SEH specifics (since they were addressed in Chapter 15) and focus on other ways that exceptions can be used to hamper the malware analyst.
Exceptions can be used to disrupt or detect a debugger. Most exception-based detection relies on the fact that debuggers will trap the exception and not immediately pass it to the process being debugged for handling. The default setting on most debuggers is to trap exceptions and not pass them to the program. If the debugger doesn’t pass the exception to the process properly, that failure can be detected within the process exception-handling mechanism.
Figure 16-4 shows OllyDbg’s default settings; all exceptions will be trapped unless the box is checked. These options are accessed via Options ▶ Debugging Options ▶ Exceptions.
A classic form of anti-debugging is to use exceptions to annoy the analyst and disrupt normal program execution by inserting interrupts in the middle of a valid instruction sequence. Depending on the debugger settings, these insertions could cause the debugger to stop, since it is the same mechanism the debugger itself uses to set software breakpoints.
Because INT 3 is used by debuggers to set software
breakpoints, one anti-debugging technique consists of inserting 0xCC opcodes into valid sections of code in order to trick the debugger into thinking
that the opcodes are its breakpoints. Some debuggers track where they set software breakpoints in
order to avoid falling for this trick.
The 2-byte opcode sequence 0xCD03 can also be used to
generate an INT 3, and this is often a valid way for malware to
interfere with WinDbg. Outside a debugger, 0xCD03 generates a
STATUS_BREAKPOINT exception. However, inside WinDbg, it catches
the breakpoint and then silently advances EIP by exactly 1 byte, since a breakpoint is normally the
0xCC opcode. This can cause the program to execute a different
set of instructions when being debugged by WinDbg versus running normally. (OllyDbg is not
vulnerable to interference using this 2-byte INT 3
attack.)
Example 16-9 shows assembly code that implements this technique.
This example sets a new SEH and then calls INT 3 to force the
code to continue.
The INT 2D anti-debugging technique functions like INT 3—the INT 0x2D instruction is
used to access the kernel debugger. Because INT 0x2D is the way
that kernel debuggers set breakpoints, the method shown in Listing 16-9 applies.
One of Intel’s undocumented instructions is the In-Circuit Emulator (ICE) breakpoint,
icebp (opcode 0xF1). This
instruction is designed to make it easier to debug using an ICE, because it is difficult to set an
arbitrary breakpoint with an ICE.
Executing this instruction generates a single-step exception. If the program is being traced via single-stepping, the debugger will think it is the normal exception generated by the single-step and not execute a previously set exception handler. Malware can take advantage of this by using the exception handler for its normal execution flow, which would be disrupted in this case.
In order to bypass this technique, do not single-step over an icebp instruction.