CtxHideEx32 tool[34] allows a substring search for a window title or class, for example:
CtxHideEx32.exe HIDE "*error" "" OK
We have discovered that it allows to automatically dump any process displaying a message box with an error message in its window title. Here is an example using TestDefaultDebugger64 (Volume 1, page 641) to simulate an application fault message where the following instance of CtxHideEx32 was setup to dump a process showing WER dialog on Vista:
CtxHideEx32 NONE "*Microsoft Windows" "" "C:kktoolsuserdump8.1x64userdump.exe %d"
We click on a big lightning button:
And then WER dialog appears:
Immediately CtxHideEx32 kicks in and starts dumping the owner process incessantly so we better to dismiss that dialog by choosing some option:
We see it was WerFault.exe.
The main problem of memory dump analysis is the lack of consistent kernel virtual memory dumps saved on demand without system halt. At the time of this writing LiveKd[35] and Win32DD[36] tools are physical memory dumpers only and do not save kernel memory dump files. These dumps are known to be inconsistent (Volume 1, page 269). We propose a different scheme to save memory consistently, for example, 1) physical memory partition into two parts from OS boot time, 2) when memory snapshot is needed we raise IRQL on all processors, 3) pump memory contents from one part to another (with compression if necessary, in such partition the reserved part of physical memory could be smaller), 4) lower IRQL on all processors to resume normal OS functions and 5) save consistent memory snapshot from reserved part of physical memory to a dump file in the background. The crucial feature of osmosis[37] is its bipartite division and membrane. Hence the name of the project:
OSMOSIS
Optimally Saved Memory of System Internal State
Optimally Saved Memory (of) Operating System Internal State
This is, of course, for OS running on physical machines, virtual machine case is much simpler in theory because we can freeze the whole VM or save its snapshot and later run an external tool or file converter on it.
Sometimes there is a need to preserve a crashing application or a service from termination and keep it in memory without showing any GUI dialogs or message boxes. Here Crash2Hang tool comes handy. It is free and can be downloaded from here:
http://www.dumpanalysis.org/downloads/Crash2Hang.zip
The source code is simple as possible:
// Crash2Hang
// Copyright (c) 2009 Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt
#include <windows.h>
int main(int argc, WCHAR* argv[])
{
if (argc > 1)
MessageBox(NULL, L"One of processes has called a postmortem debugger!",
L"Crash2Hang", MB_OK | MB_ICONSTOP | MB_SETFOREGROUND);
else
Sleep(INFINITE);
return 0;
}The tool can be used as a postmortem debugger specified in AeDebug registry key, for example, instead of CDB (Volume 1, page 618). Any argument specified to Crash2Hang.exe causes it to display a message box when launched and exit itself upon its dismissal. If several threads in a problem process experience an unhandled exception then Crash2Hang process is launched several times which may result in several such message boxes. Without arguments Crash2Hang process hangs infinitely causing the problem thread with an unhandled exception to hang indefinitely too (see Volume 1, page 113 for the explanation).
To test various postmortem debuggers and WER to their fullest potential and especially Crash2Hang tool (page 356) we wrote a small program that models multiple exceptions in several threads. It is free and can be downloaded with full PDB and source code from here:
http://www.dumpanalysis.org/downloads/MTCrash.zip
The source code is simple as possible:
// MTCrash (Multithreaded crash)
// Copyright (c) 2009 Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt
#include <windows.h>
#include <process.h>
#include <iostream>
bool twice = false;
void thread_one(void *)
{
Sleep(1000);
std::cout << "Thread 1 is about to experience an AV exception..." <<
std::endl;
*(int *)NULL = 0;
}
void thread_two(void *)
{
Sleep(2000);
if (twice)
{
std::cout << "Thread 2 is about to experience an AV exception..." <<
std::endl;
*(int *)NULL = 0;
}while (true)
{
std::cout << "Thread 2 is still running..." << std::endl;
Sleep(1000);
}
}
int main(int argc, WCHAR* argv[])
{
if (argc > 1) twice = true;
_beginthread(thread_two, 0, NULL);
_beginthread(thread_one, 0, NULL);
while(true)
{
std::cout << "Main Thread is still running..." << std::endl;
Sleep(1000);
}
return 0;
}It creates 2 additional threads and the first of them tries to access a NULL pointer:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(d3c.e94): Access violation - code c0000005 (first/second chance not
available)
eax=00000000 ebx=005a4660 ecx=0041948c edx=00419ef0 esi=00419488
edi=00000000
eip=004013bd esp=007eff7c ebp=007effb4 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
MTCrash!thread_one+0×6d:
004013bd c7050000000000000000 mov dword ptr
ds:[0],0 ds:002b:00000000=????????
0:002> ~*kL
0 Id: d3c.eb4 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
002dfee4 7d4d0ec5 ntdll!ZwDelayExecution+0x15
002dff4c 7d4d14ef kernel32!SleepEx+0x68
002dff5c 0040157a kernel32!Sleep+0xf
002dff70 004046ac MTCrash!main+0xaa
002dffc0 7d4e7d2a MTCrash!__tmainCRTStartup+0x15f
002dfff0 00000000 kernel32!BaseProcessStart+0x281 Id: d3c.ebc Suspend: 1 Teb: 7efda000 Unfrozen ChildEBP RetAddr 006afee4 7d4d0ec5 ntdll!ZwDelayExecution+0x15 006aff4c 7d4d14ef kernel32!SleepEx+0x68 006aff5c 004014c5 kernel32!Sleep+0xf 006aff7c 00404352 MTCrash!thread_two+0xf5 006affb4 004043eb MTCrash!_callthreadstart+0x1b 006affb8 7d4dfe21 MTCrash!_threadstart+0x73 006affec 00000000 kernel32!BaseThreadStart+0x34 # 2 Id: d3c.e94 Suspend: 1 Teb: 7efd7000 Unfrozen ChildEBP RetAddr 007effb8 7d4dfe21 MTCrash!thread_one+0×6d 007effe4 00000000 kernel32!BaseThreadStart+0×34 3 Id: d3c.f0c Suspend: 1 Teb: 7efaf000 Unfrozen ChildEBP RetAddr 0083ffc8 7d665081 ntdll!DbgBreakPoint+0x1 0083fff4 00000000 ntdll!DbgUiRemoteBreakin+0x2d
The second thread and main thread continue to run:
C:Crash2Hang>MTCrash.exe Main Thread is still running... Thread 1 is about to experience an AV exception... Main Thread is still running... Thread 2 is still running... Main Thread is still running... Thread 2 is still running... Main Thread is still running... Thread 2 is still running... [...]
If launched with any parameter the second thread also experiences an unhandled exception (in bold) while the first one is suspended by an unhandled exception filter (in bold italics):
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(ca4.cb0): Access violation - code c0000005 (first/second chance not
available)
eax=00000000 ebx=005a4668 ecx=0041948c edx=00419ef0 esi=00419488
edi=00000000
eip=004013bd esp=007eff7c ebp=007effb4 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
MTCrash!thread_one+0×6d:0:002> ~*kL
0 Id: ca4.ca0 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
002dfee4 7d4d0ec5 ntdll!ZwDelayExecution+0x15
002dff4c 7d4d14ef kernel32!SleepEx+0x68
002dff5c 0040157a kernel32!Sleep+0xf
002dff70 004046ac MTCrash!main+0xaa
002dffc0 7d4e7d2a MTCrash!__tmainCRTStartup+0x15f
002dfff0 00000000 kernel32!BaseProcessStart+0x28
1 Id: ca4.ca8 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
006af8cc 7d5357f3 ntdll!ZwRaiseHardError+0×12
006afb38 7d508f4e kernel32!UnhandledExceptionFilter+0×519
006afb40 7d4d8a25 kernel32!BaseThreadStart+0×4a (FPO: [SEH])
006afb68 7d61ec2a kernel32!_except_handler3+0×61
006afb8c 7d61ebfb ntdll!ExecuteHandler2+0×26
006afc34 7d61ea36 ntdll!ExecuteHandler+0×24
006afc34 0040144f ntdll!KiUserExceptionDispatcher+0xe (CONTEXT @ 006afc9c)
006aff7c 00404352 MTCrash!thread_two+0×7f
006affb4 004043eb MTCrash!_callthreadstart+0×1b
006affb8 7d4dfe21 MTCrash!_threadstart+0×73
006affec 00000000 kernel32!BaseThreadStart+0×34
# 2 Id: ca4.cb0 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
007effb8 7d4dfe21 MTCrash!thread_one+0×6d
007effe4 00000000 kernel32!BaseThreadStart+0×34
0:002> .cxr 006afc9c
eax=00000000 ebx=005a4448 ecx=0041948c edx=00419ef0 esi=00419488
edi=00000000
eip=0040144f esp=006aff68 ebp=7d4d14e0 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
MTCrash!thread_two+0×7f:
0040144f c7050000000000000000 mov dword ptr
ds:[0],0 ds:002b:00000000=????????However, as soon as we dismiss the first error message box or if Auto is set to 1 in AeDebug registry key MTCrash terminates. If Crash2Hang (page 356) is set as a default postmortem debugger then we get two instances of it running and MTCrash hangs even if we dismiss the first message. The main thread continues to run:
C:Crash2Hang>MTCrash.exe 1 Main Thread is still running... Thread 1 is about to experience an AV exception... Main Thread is still running... Main Thread is still running... Thread 2 is about to experience an AV exception... Main Thread is still running... Main Thread is still running... Main Thread is still running... Main Thread is still running... [...]
In Volume 1 (page 616) we focused on using a debugger to extract a computer name from memory dumps. Here is a very simple approach for user dumps using built-in command line tools:
C:UserDumps>findstr "COMPUTERNAME=" new_0200_2008-04-28_14-11-54- 937_0cb0.dmp
Most of the time the last portion of output contains something like this:
If we don't see the variable we can redirect the output into a text file and look in it or simply open a memory dump file in any hex editor and search for a UNICODE string.
If we look at thread raw stacks in many process memory dumps from Windows Server 2008 applications we find ntdll!FinalExceptionHandler symbol:
Loading Dump File [App.dmp]
User Mini Dump File with Full Memory: Only application data is available
Windows Server 2008/Windows Vista Version 6001 (Service Pack 1) MP (4
procs) Free x86 compatible
0:000> !teb
TEB at 7ffde000
ExceptionList: 0022fdd8
StackBase: 00230000
StackLimit: 00225000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffde000
EnvironmentPointer: 00000000
ClientId: 00002458 . 00002478
RpcHandle: 00000000
Tls Storage: 7ffde02c
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: c0000035
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 00225000 00230000
[...]
0022ff7c 00000000
0022ff80 00000000
0022ff84 00000000
0022ff88 0022ff94
0022ff8c 76744911 kernel32!BaseThreadInitThunk+0xe
0022ff90 7ffdf000
0022ff94 0022ffd4
0022ff98 77b5e4b6 ntdll!__RtlUserThreadStart+0x23
0022ff9c 7ffdf000
0022ffa0 2497b80a
0022ffa4 00000000
0022ffa8 00000000
0022ffac 7ffdf000
0022ffb0 00000000
0022ffb4 00000000
0022ffb8 00000000
0022ffbc 0022ffa0
0022ffc0 00000000
0022ffc4 0022ffe4
0022ffc8 77b29834 ntdll!_except_handler40022ffcc 530d7826
0022ffd0 00000000
0022ffd4 0022ffec
0022ffd8 77b5e489 ntdll!_RtlUserThreadStart+0x1b
0022ffdc 00401110 App+0x1110
0022ffe0 7ffdf000
0022ffe4 ffffffff
0022ffe8 77bc75de ntdll!FinalExceptionHandler
0022ffec 00000000
0022fff0 00000000
0022fff4 00401110 App+0×1110
0022fff8 7ffdf000
0022fffc 00000000
00230000 78746341The following very interesting review article about Windows memory protection mechanisms explains the new "Final" exception mechanism in W2K8:
http://taossa.com/archive/bh08sotirovdowd.pdf