In Chapter 3, Dynamic Analysis, you learned the techniques to execute a DLL to perform dynamic analysis. In this section, you will use some of the concepts that you learned in Chapter 3, Dynamic Analysis, to debug a DLL using IDA. If you not familiar with dynamic analysis of a DLL, it is highly recommended to read Section 6, Dynamic-Link Library (DLL) Analysis, from Chapter 3, Dynamic Analysis, before proceeding further.
To debug a DLL using the IDA debugger, you first need to designate the executable (such as rundll32.exe) that will be used to load the DLL. To debug a DLL, first, load the DLL into IDA, which will likely display the disassembly of the DLLMain function. Set a breakpoint (F2) at the first instruction in the DLLMain function, as shown in the following screenshot. This ensures that when you run the DLL, the execution will pause at the first instruction in the DLLMain function. You can also set breakpoints on any function exported by the DLL by navigating to it from IDA's Exports window:
After you have set the breakpoint on the desired address (where you want the execution to pause), select the debugger via Debugger | Select debugger | Local Win32 debugger (or Debugger | Select debugger | Local Windows debugger) and click on OK. Next, select Debugger | Process options, which will bring up the dialog shown in the following screenshot. In the Application field, enter the full path to the executable that is used to load the DLL (rundll32.exe). In the Input file field, enter the full path to the DLL that you wish to debug, and in the Parameters field, enter the command-line arguments to pass to rundll32.exe, and click on OK. Now, you can run the program to reach the breakpoint, after which you can debug it, as you would debug any other program. The arguments that you pass to rundll32.exe should have the correct syntax to successfully debug the DLL (refer to the Working of rundll32.exe section in Chapter 3, Dynamic Analysis). A point to note is that rundll32.exe can also be used to execute a 64-bit DLL, in the same manner:
