It was reported that one important Windows service stops responding from time to time. The customer was proactive in gathering memory dumps and we got several early crash dumps (Volume 1, page 465). Most of them were false positive showing normal error handling via throwing an exception:
0:042> kL
ChildEBP RetAddr
0f7bec6c 77c31e37 kernel32!RaiseException+0×53
0f7bec84 77c32042 rpcrt4!RpcpRaiseException+0×24
0f7bec94 77cb30e4 rpcrt4!NdrGetBuffer+0×46
0f7bf080 09a554a6 rpcrt4!NdrClientCall2+0×197
[...]However, one such dump also had a clearly blocked thread (Volume 2, page 184) that was blocking 10 other different threads:
0:042> !locks CritSec MyService!MainCriticalSection+0 at 0041b9a0 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread ad0 EntryCount 0 ContentionCount 0 *** Locked CritSec +339fb8 at 00339fb8 WaiterWoken NoLockCount 10RecursionCount 1OwningThread ad0EntryCount 0 ContentionCount 31 *** Locked
0:042> ~~[ad0]kL ChildEBP RetAddr 008dc1e0 7c94734b ntdll!KiFastSystemCallRet 008dc1e4 77d96c61ntdll!NtOpenKey+0xc008dc244 77d8e15f advapi32!LocalBaseRegOpenKey+0xd0 008dc278 6064fe47advapi32!RegOpenKeyExA+0×11cWARNING: Stack unwind information not available. Following frames may be wrong. 008dc8b8 6064fa00NotMyDLL!getvar+0×4e7[...]
By checking NotMyDLL module time stamp we identified Not My Version pattern (Volume 2, page 299) because we expected much later version of that module:
0:042> lmt m NotMyDLL
start end module name
60600000 60686000 NotMyDLL Mon Oct 30 10:14:07 1999We know this component often had problems in the past and, although being stuck in registry access could be a coincidence, registry corruption or a system-wide problem, we immediately advised to upgrade the component to the latest stable version. We also got a manual dump of the service when the customer tried to restart it and it showed the signs of Lost Opportunity pattern (Volume 1, page 332):
0:000> kv
ChildEBP RetAddr Args to Child
1744fd44 7c947d0b 7c821d1e 00001b58 00000000 ntdll!KiFastSystemCallRet
1744fd48 7c821d1e 00001b58 00000000 00000000
ntdll!NtWaitForSingleObject+0xc
1744fdb8 7c821c8d 00001b58 ffffffff 00000000
kernel32!WaitForSingleObjectEx+0xac
1744fdcc 67e223dd 00001b58 ffffffff 1744fdf4
kernel32!WaitForSingleObject+0x12
1744fde0 7c93a352 67e20000 00000000 00000001 MyDLL!DllInitialize+0xed
1744fe00 7c950e70 67e222f0 67e20000 00000000
ntdll!LdrpCallInitRoutine+0x14
1744feb8 7c8268a3 00000000 00000000 00000000
ntdll!LdrShutdownProcess+0x182
1744ffa4 7c826905 c0000005 77e8f3b0 ffffffff kernel32!_ExitProcess+0x43
1744ffb8 7c8392c1 c0000005 00000000 00000000 kernel32!ExitProcess+0×14
1744ffec 00000000 77c4b0f5 0b644720 00000000 kernel32!BaseThreadStart+0×5f0:000> !teb
TEB at 7ff4b000
ExceptionList: 1744fda8
StackBase: 17450000
StackLimit: 17449000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ff4b000
EnvironmentPointer: 00000000
ClientId: 00001e90 . 00001168
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdd000
LastErrorValue: 0
LastStatusValue: 103
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 17449000 17450000
[...]
1744f4b0 7c94775b ntdll!NtRaiseHardError+0xc
1744f4b4 7c842610 kernel32!UnhandledExceptionFilter+0×51a
1744f4b8 d0000144
1744f4bc 00000000
[...]
0:000> !error d0000144
Error code: (NTSTATUS) 0xd0000144 (3489661252) - {Application Error} The
exception %s (0x%08lx) occurred in the application at location 0x%08lx.Therefore, we additionally advised to dump the process manually using userdump.exe when an error message box appears on the console session. We thought that getting right dump files at the right time via the right method would prove or disprove our hypothesis about NotMyDLL component.
Lateral Damage pattern (Volume 1, page 264) can be found in memory dump files that are between the normal healthy dump files and corrupt dumps (Volume 2, page 151). For example, the following 8 GB complete memory dump that fits perfectly into 16 GB page file had the problem of missing processor control region making it impossible to get meaningful information from certain WinDbg commands:
0: kd> !analyze -v
[...]
UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind that
the kernel isn't allowed to have/catch (bound trap) or that is always
instant death (double fault). The first number in the bugcheck params is
the number of the trap (8 = double fault, etc) Consult an Intel x86 family
manual to learn more about what these traps are. Here is a *portion* of
those codes:
If kv shows a taskGate
use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
use .trap on that value
Else
.trap on the appropriate frame will show where the trap was taken
(on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000008, EXCEPTION_DOUBLE_FAULT
Arg2: f7727fe0
Arg3: 00000000
Arg4: 00000000
Debugging Details:
------------------
Unable to read selector for PCR for processor 1
Unable to read selector for PCR for processor 3
Unable to read selector for PCR for processor 1
Unable to read selector for PCR for processor 3
[...]
STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be
wrong.
8089a600 8088ddf2 00000000 0000000e 00000000 processr+0x2886
8089a604 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0xa0: kd> ~1
Unable to read selector for PCR for processor 1
WARNING: Unable to reset page directories
1: kd> !pcr
Unable to read selector for PCR for processor 1
Cannot get PRCB address
1: kd> kv
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 0×0The bugcheck argument 1 shows that we have a double fault that most often results from kernel stack overflow (Volume 1, page 314). If we go back to the processor 0 to inspect its TSS we don't get meaningful results too (we expect the value of Backlink to be 0×28):
0: kd> !pcr
KPCR for Processor 0 at ffdff000:
Major 1 Minor 1
NtTib.ExceptionList: ffffffff
NtTib.StackBase: 00000000
NtTib.StackLimit: 00000000
NtTib.SubSystemTib: 80042000
NtTib.Version: 2a1b0b08
NtTib.UserPointer: 00000001
NtTib.SelfTib: 00000000
SelfPcr: ffdff000
Prcb: ffdff120
Irql: 0000001f
IRR: 00000000
IDR: ffffffff
InterruptMode: 00000000
IDT: 8003f400
GDT: 8003f000
TSS: 80042000
CurrentThread: 8089d8c0
NextThread: 00000000
IdleThread: 8089d8c0
DpcQueue:0: kd> dt _KTSS80042000nt!_KTSS+0×000 Backlink:0xc45+0×002 Reserved0 : 0×4d8a +0×004 Esp0 : 0×8089a6a0 +0×008 Ss0 : 0×10 +0×00a Reserved1 : 0xb70f +0×00c NotUsed1 : [4] 0×5031ff00 +0×01c CR3 : 0×8b55ff8b +0×020 Eip : 0xc75ffec +0×024 EFlags : 0xe80875ff +0×028 Eax : 0xfffffbdd +0×02c Ecx : 0×1b75c084 +0×030 Edx : 0×8b184d8b +0×034 Ebx : 0×7d8b57d1 +0×038 Esp : 0×2e9c110 +0×03c Ebp : 0xf3ffc883 +0×040 Esi : 0×83ca8bab +0×044 Edi : 0xaaf303e1 +0×048 Es : 0xeb5f +0×04a Reserved2 : 0×6819 +0×04c Cs : 0×24fc +0×04e Reserved3 : 0×44 +0×050 Ss : 0×75ff +0×052 Reserved4 : 0xff18 +0×054 Ds : 0×1475 +0×056 Reserved5 : 0×75ff +0×058 Fs : 0xff10 +0×05a Reserved6 : 0xc75 +0×05c Gs : 0×75ff +0×05e Reserved7 : 0xe808 +0×060 LDT : 0 +0×062 Reserved8 : 0xffff +0×064 Flags : 0 +0×066 IoMapBase : 0×20ac +0×068 IoMaps : [1] _KiIoAccessMap +0×208c IntDirectionMap : [32] "???"
However, if we try to list all thread stacks we see one thread running on the processor 1:
0: kd> !process 0 ff [...]
THREAD 8a241db0 Cid 1218.4420 Teb: 00000000 Win32Thread: 00000000RUNNING on processor 1IRP List: 8b200008: (0006,0244) Flags: 00000884 Mdl: 00000000 89beedb8: (0006,0244) Flags: 00000884 Mdl: 00000000 Not impersonating DeviceMap e1002060 Owning Process 8bc63d88 Image: svchost.exe Wait Start TickCount 10242012 Ticks: 0 Context Switch Count 1832 UserTime 00:00:00.000 KernelTime 00:00:00.046 Start Address termdd (0xf75cc218) Stack Init 9c849000 Current 9c846938 Base 9c849000Limit 9c846000Call 0 Priority 11 BasePriority 10 PriorityDecrement 0 Unable to read selector for PCR for processor 1 [...]
Now we can look at its raw stack to see execution residue (Volume 2, page 239) and try to reconstruct partial stack traces:
0: kd> dds9c8460009c849000 9c846000 94040001 9c846004 00000014 9c846008 8d147848 9c84600c 8d0bfd08 9c846010 8d0bfd00 9c846014 00000001 9c846018 8d0bfd08 9c84601c 8d0bfd00 9c846020 8d0bfd009c846024 9c8460349c846028 80a5c456 hal!KfLowerIrql+0×62 9c84602c 8d0bfdd8 9c846030 8d0bfd009c846034 9c8460609c846038 80a5a56d hal!KeReleaseQueuedSpinLock+0×2d 9c84603c 00000011 9c846040 00000001 9c846044 8a241db0 9c846048 0000000e 9c84604c 00000000 9c846050 8d0bfdc0 9c846054 05000000 9c846058 00007400 9c84605c 000000019c846060 9c8460849c846064 808b6138 driverA!MapData+0×4a 9c846068 8d0bfd08 9c84606c 00007400 9c846070 00000000 9c846074 00000018
9c846078 00000028 9c84607c 00001000 9c846080 000000189c846084 9c84609c9c846088 f7b8f2e5 driverB!CheckData+0×7a 9c84608c 01b47538 9c846090 00000028 9c846094 0000001c [...] 0: kd> k L=9c846024 9c846024 9c846024ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 9c846024 80a5c456 0×9c846024 9c846034 80a5a56d hal!KfLowerIrql+0×62 9c8460f0 8080d164 hal!KeReleaseQueuedSpinLock+0×2d 9c846060 808b6138 driverA!RemapData+0×3e 9c846084 f7b8f2e5 driverA!MapData+0×4a 9c84609c f7b8f340 driverB!CheckData+0×7a 9c8460e4 808b4000 driverB!CheckAttributes+0×36f 9c84610c f7b8e503 driverB!AddToRecord+0×2a 9c846174 f7b90df0 driverB!ReadRecord+0×1d0 f7b8e508 90909090 driverB!ReadAllRecords+0×7a [...]
Using the current stack pointer we get another partial stack trace:
0: kd> k L=9c846034 9c846938 9c846938
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
9c846954 8081df65 0×9c846938
9c846968 808f5437 nt!IofCallDriver+0×45
9c84697c 808ef963 nt!IopSynchronousServiceTail+0×10b
9c8469a0 8088978c nt!NtQueryDirectoryFile+0×5d
9c8469a0 8082f1c1 nt!KiFastCallEntry+0xfc
9c846a44 f5296f4b nt!ZwQueryDirectoryFile+0×11
9c846a90 f5297451 DriverC+0×2f4b
9c846adc f52a54cb DriverC+0×3451
9c846af8 f52a44e6 DriverC+0×114cb
9c846b1c f52b2941 DriverC+0×104e6
9c846b4c f52b2626 DriverC+0×1e941
9c846b88 f52a34a7 DriverC+0×1e626
9c846be8 f52a487c DriverC+0xf4a7
[...]Using different base pointers for k command we can reconstruct different partial stack traces. We can also analyze the longest ones for any stack usage using variant of knf command that shows stack frame size in bytes and find drivers that consume the most of kernel stack. Because we see execution residue on top of the kernel stack (Limit) we can suspect this thread caused the actual stack overflow that resulted in the double fault bugcheck.
Let's look at a manual complete memory dump from a hang workstation where the default analysis pointed to a possible Spiking Thread pattern (Volume 1, page 305):
0: kd> !analyze -v
[...]
MANUALLY_INITIATED_CRASH (e2)
The user manually initiated this crash dump.
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000
[...]
GetContextState failed, 0x80070026
Unable to read selector for PCR for processor 1
GetContextState failed, 0x80070026
Unable to read selector for PCR for processor 1
PROCESS_NAME: AppA.exe
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from 808b73a4 to 808b72cb
STACK_TEXT:
f46f5b44 808b73a4 e1a36008 00000004 00000018
nt!HvpFindFreeCellInThisViewWindow+0xd7
f46f5b6c 808bd07b e1a36008 00000002 00000018 nt!HvpFindFreeCell+0x98
f46f5b98 808bd588 e1a36008 e1a362fc 00000000 nt!HvpDoAllocateCell+0x69
f46f5bbc 808d0b22 e1a36008 009214a0 c94e24a4 nt!HvReallocateCell+0x9a
f46f5bdc 808c1028 e1a36008 051fa3e8 00000003 nt!CmpAddValueToList+0x46
f46f5c28 808c406a e1a36008 cddb7ccc f46f5d0c nt!CmpSetValueKeyNew+0xfa
f46f5cb4 808b7e2f e45872e0 f46f5d0c 00000004 nt!CmSetValueKey+0x426
f46f5d44 8088978c 00000438 0012fad0 00000000 nt!NtSetValueKey+0x241
f46f5d44 7c8285ec 00000438 0012fad0 00000000 nt!KiFastCallEntry+0xfc
0012fa58 7c827b7b 77f77703 00000438 0012fad0 ntdll!KiFastSystemCallRet
0012fa5c 77f77703 00000438 0012fad0 00000000 ntdll!ZwSetValueKey+0xc
0012faa0 77f5ec90 00000438 0012fad0 00000004
ADVAPI32!LocalBaseRegSetValue+0x12c
0012fb04 60072e40 00000438 6290c0c4 00000000 ADVAPI32!RegSetValueExA+0x160
WARNING: Stack unwind information not available. Following frames may be
wrong.
0012fbf4 628e2d57 60062e70 60062b40 80000001 DLLA!GetObjectId+0×9600
[...]FOLLOWUP_IP: nt!HvpFindFreeCellInThisViewWindow+d7 808b72cb 034508 add eax,dword ptr [ebp+8]
According to this thread information it is RUNNING (this is also evident from its call stack):
0: kd> !thread THREAD 8a0b2890 Cid 0814.10e8 Teb: 7ffdf000 Win32Thread: bc217c68RUNNING on processor 0Not impersonating DeviceMap e440acc0 Owning Process 8a0d85f8 Image:AppA.exeWait Start TickCount 153974 Ticks: 0 Context Switch Count 16905 LargeStack UserTime 00:00:03.109KernelTime 00:00:17.500[...]
We also see that the thread accumulated 17 seconds as time spent in kernel. Switching to AppA process context and looking at its Image version we find that it is 5.80.x version:
0: kd> lmv m AppA
start end module name
00400000 0049c000 AppA (deferred)
Image path: C:PROGRA~1AppAAppA.exe
Image name: AppA.exe
Timestamp: Thu Jun 05 14:51:52 2008 (4847EF78)
CheckSum: 0009D068
ImageSize: 0009C000
File version: 5.80.5.1764
Product version: 5.80.0.0However, from Google search (Volume 1, page 395) we can find that there is a newer version available (variant of Not My Version pattern, Volume 1, page 299) and even some indication on various forums that the older versions had problems with CPU resource utilization. After scanning all threads for any additional suspicious signs we see another running thread on the second CPU:
THREAD 8a2ed5d0 Cid 11b4.1100 Teb: 7ffdf000 Win32Thread: bc342b80RUNNING on processor 1Not impersonating DeviceMap e44fc100 Owning Process 8a1efcb0Wait Start TickCount 153973 Ticks: 1 (0:00:00:00.015) Context Switch Count 50736 LargeStackImage: calc.exeKernelTime 00:00:00.015 Win32 Start Address calc (0×0101e23a) Start Address kernel32!BaseProcessStartThunk (0×77e617f8) Stack Init f4cd6000 Current f4cd5d00 Base f4cd6000 Limit f4cd1000 Call 0 Priority 6 BasePriority 6 PriorityDecrement 0 Unable to get context for thread running on processor 1, Win32 error 0n38UserTime 00:01:04.515
We also see that this thread spent more than a minute in user mode. Unfortunately, we cannot see its thread stack because the dump file shows signs of Truncated Dump pattern (Volume 1, page 340):
Loading Dump File [MEMORY.DMP] Kernel Complete Dump File: Full address space is available ************************************************************ WARNING: Dump file has been truncated. Data may be missing. ************************************************************ [...] 0: kd> ~1 GetContextState failed, 0×80070026 Unable to read selector for PCR for processor 1 WARNING: Unable to reset page directories GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 Unable to get program counter GetContextState failed, 0×80070026 Unable to get current machine context, Win32 error 0n38 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026
The dump file size 4,177,920 KB is less than the amount of physical memory 4,192,948 KB:
1: kd> !vm
GetContextState failed, 0x80070026
GetContextState failed, 0x80070026
GetContextState failed, 0x80070026
Unable to get program counter
*** Virtual Memory Usage ***
Physical Memory: 1048237 ( 4192948 Kb)
Page File: ??R:pagefile.sys
Current: 4177920 Kb Free Space: 4154440 Kb
Minimum: 4177920 Kb Maximum: 4194304 KbWe can stop here and still recommend upgrading AppA product seen from the thread running on the first processor but the fact that the second thread belongs to the innocent calc.exe demands some attention. Was it calculating incessantly some financial figures following button clicks from a financial genius? Taking advantage of a complete memory dump and the fact that this process spent most of the time in user space we can check for Hooked Functions pattern (Volume 1, page 468):
1: kd> .process /r /p 8a1efcb0 Implicit process is now 8a1efcb0 Loading User Symbols .......................... GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026 GetContextState failed, 0x80070026
1: kd> !chkimg -lo 50 -d !user32 -v
GetContextState failed, 0x80070026
GetContextState failed, 0x80070026
GetContextState failed, 0x80070026
Unable to get program counter
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path:
c:symdownstreamUSER32.dll45D70AC791000USER32.dll
No range specified
Scanning section: .text
Size: 392891
Range to scan: 77381000-773e0ebb
7738c341-7738c345 5 bytes - USER32!CreateWindowExA
[ 8b ff 55 8b ec:e9 ba 3c 00 c0 ]
[...]
Total bytes compared: 73728(18%)
Number of errors: 75
75 errors : !user32 (7738c341-773a154d)
GetContextState failed, 0×80070026
GetContextState failed, 0×80070026
GetContextState failed, 0×80070026
GetContextState failed, 0×80070026
GetContextState failed, 0×80070026
GetContextState failed, 0×80070026
1: kd> u 7738c341
USER32!CreateWindowExA:
7738c341 e9ba3c00c0 jmp 37390000
7738c346 6801000040 push 40000001h
7738c34b ff7534 push dword ptr [ebp+34h]
7738c34e ff7530 push dword ptr [ebp+30h]
7738c351 ff752c push dword ptr [ebp+2Ch]
7738c354 ff7528 push dword ptr [ebp+28h]
7738c357 ff7524 push dword ptr [ebp+24h]
7738c35a ff7520 push dword ptr [ebp+20h]
GetContextState failed, 0×80070026
GetContextState failed, 0×80070026
GetContextState failed, 0×800700261: kd> u3739000037390000 e96b91562e jmpHookA!CreateWindowExA(658f9170) 37390005 8bff mov edi,edi 37390007 55 push ebp 37390008 8bec mov ebp,esp 3739000a e937c3ff3f jmp USER32!CreateWindowExA+0×5 (7738c346) 3739000f 0000 add byte ptr [eax],al 37390011 0000 add byte ptr [eax],al 37390013 0000 add byte ptr [eax],al GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026 GetContextState failed, 0×80070026
Indeed we see that HookA module is involved and we recommend to test the stability of the system without the product that uses it or by upgrading or disabling that component.
Stack Trace Collection, Hidden Exception and NULL Code Pointer
This is common pattern interaction in user dumps coming from x64 environments. Its characteristic feature is stack trace collection (Volume 1, page 409) that appears to be damaged or corrupt with lots of zeroed call sites and sometimes having UNICODE values in saved RSP register:
0:001> ~*kL
0 Id: 668.684 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP RetAddr Call Site
00000000`000afcb8 000007ff`7f1ee614 USER32!NtUserWaitMessage+0xa
00000000`000afcc0 000007ff`7f1adf7a
SHELL32!CDesktopBrowser::_MessageLoop+0x287
00000000`000afd50 00000001`0001a46c SHELL32!SHDesktopMessageLoop+0x5c
00000000`000afd80 00000001`000231ba Explorer!ExplorerWinMain+0x6f1
00000000`000afec0 00000000`77d5964c Explorer!ModuleEntry+0x1c6
00000000`000aff80 00000000`00000000 kernel32!BaseProcessStart+0x29
1 Id: 668.20c Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP RetAddr Call Site
00000000`00000000 00000000`00000000 0×0
2 Id: 668.298 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
Child-SP RetAddr Call Site
00000000`00000000 00000000`00000000 0×0
3 Id: 668.f34 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
Child-SP RetAddr Call Site
00000000`00000000 00000000`00000000 0×0
4 Id: 668.824 Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
Child-SP RetAddr Call Site
00000000`00000000 00000000`00000000 0×0
5 Id: 668.820 Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
Child-SP RetAddr Call Site
00000000`00000000 00000000`00000000 0×06 Id: 668.914 Suspend: 1 Teb: 000007ff`fffac000 Unfrozen Child-SP RetAddr Call Site 00000000`0342fb98 000007ff`7fd6ff61 ntdll!ZwReplyWaitReceivePortEx+0xa 00000000`0342fba0 000007ff`7fd45369 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x2a5 00000000`0342feb0 000007ff`7fd65996 RPCRT4!RecvLotsaCallsWrapper+0x9 00000000`0342fee0 000007ff`7fd65d51 RPCRT4!BaseCachedThreadRoutine+0xde 00000000`0342ff50 00000000`77d6b6da RPCRT4!ThreadStartRoutine+0x21 00000000`0342ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a 7 Id: 668.8cc Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen Child-SP RetAddr Call Site 00000000`00000000 00000000`00000000 0x7 8 Id: 668.1078 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen Child-SP RetAddr Call Site00000000`00000000 00000000`00000000 0×0# 9 Id: 668.1118 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen Child-SP RetAddr Call Site00000000`00000000 00000000`00000000 0×010 Id: 668.574 Suspend: 1 Teb: 000007ff`fffa2000 Unfrozen Child-SP RetAddr Call Site00000000`00000020 00000000`00000000 0×011 Id: 668.a54 Suspend: 1 Teb: 000007ff`fffa0000 Unfrozen Child-SP RetAddr Call Site 90000000`0000679f 00000000`00000000 0x72505c74`6e656d65 12 Id: 668.163c Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen Child-SP RetAddr Call Site 0000001e`00057000 00000000`00000000 0x575c3a43`00000075 13 Id: 668.5b0 Suspend: 1 Teb: 000007ff`fff9e000 Unfrozen Child-SP RetAddr Call Site 49575c3a`43000000 00000000`00000000 0x4e49575c`3a430000 14 Id: 668.4c0 Suspend: 1 Teb: 000007ff`fff9c000 Unfrozen Child-SP RetAddr Call Site 3a430000`00200004 00000000`00000000 0x43000000`1f0001c0 15 Id: 668.774 Suspend: 1 Teb: 000007ff`fff9a000 Unfrozen Child-SP RetAddr Call Site 00410044`00500050 00000000`00000000 0×6e006f`00690074 16 Id: 668.17c0 Suspend: 1 Teb: 000007ff`fff98000 Unfrozen Child-SP RetAddr Call Site 005c0029`00360038 00000000`00000000 0×500048`005c0029
We notice '#' in front of the thread 9:
# 9This marks the thread that caught the exception. We can either switch to it by ~9s or using ~#s command:
0:010> ~#s 00000000`00000000 ?? ???
Now we look at the thread raw stack data to search for any hidden exceptions (Volume 1, page 271) and we find one indeed:
0:009> !teb
TEB at 000007fffffa4000
ExceptionList: 0000000000000000
StackBase: 0000000003000000
StackLimit: 0000000002ff2000
SubSystemTib: 0000000000000000
FiberData: 0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self: 000007fffffa4000
EnvironmentPointer: 0000000000000000
ClientId: 0000000000000668 . 0000000000001118
RpcHandle: 0000000000000000
Tls Storage: 0000000000000000
PEB Address: 000007fffffdb000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 0
0:010> dqs 0000000003232000 0000000003240000
00000000`03232000 00000000`00000000
00000000`03232008 00000000`00000000
00000000`03232010 00000000`00000000
[...]
00000000`02ffc8c8 00000000`77ef3202 ntdll!KiUserExceptionDispatcher+0×5200000000`02ffc8d00:009> .cxrrax=0000000000000000 rbx=0000000000000000 rcx=00000000671b4610 rdx=ffffffff9be48728 rsi=0000000000000000 rdi=0000000000000000 rip=0000000000000000 rsp=0000000002ffce68 rbp=00000000671b4610 r8=0000000002ffccc0 r9=0000000000000000 r10=000068aa62010001 r11=00000000671b4610 r12=0000000000000000 r13=00000000000006a5 r14=00000000671b45d0 r15=7ffffffffffffffd iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010 246 00000000`00000000 ?? ??? 0:009> kL *** Stack trace for last set context - .thread/.cxr resets it Child-SP RetAddr Call Site00000000`02ffc8d000000000`02ffd230 00000000`67194f93 DllA!DllUnregisterServer+0×1d48 00000000`02ffd350 000007ff`7ca9cae4 DllA!DllUnregisterServer+0×3b53 00000000`02ffd3f0 000007ff`7ca9d867 NETSHELL!GetPrimaryIPv6AddressForAdapter+0×64 00000000`02ffd720 000007ff`7ca1eb4f NETSHELL!CConnectionFolder::GetDetailsOf+0×62a 00000000`02ffdd20 000007ff`7f27f57f NETSHELL!CConnectionFolder::GetDetailsEx+0×21d 00000000`02ffe870 000007ff`7f27eea6 SHELL32!CNameSpaceItemUIProperty::GetPropertyDisplayValue+0xaf 00000000`02ffe910 000007ff`7f1e7bd4 SHELL32!CDetailsSectionInfoTask::RunInitRT+0×213 00000000`02fffc60 000007ff`7cec9c26 SHELL32!CRunnableTask::Run+0×52 00000000`02fffc90 000007ff`7ef773d8 BROWSEUI!CShellTaskScheduler_ThreadProc+0×1be 00000000`02fffd60 00000000`77eea78a SHLWAPI!ExecuteWorkItem+0×28 00000000`02fffd90 00000000`77eea99f ntdll!RtlpWorkerCallout+0×183 00000000`02fffec0 00000000`77eeac75 ntdll!RtlpExecuteWorkerRequest+0×63 00000000`02ffff00 00000000`77d6b6da ntdll!RtlpWorkerThread+0×71 00000000`02ffff80 00000000`00000000 kernel32!BaseThreadStart+0×3a00000000`02ffce68 00000000`67199841 0×0 00000000`02ffce70 00000000`67193188 DllA!DllUnregisterServer+0×8401
Checking disassembly we see that DllA module code dereferenced a NULL code pointer (Volume 2, page 237):
0:009> ubDllA!DllUnregisterServer+0×8401DllA!DllUnregisterServer+0×83e4: 00000000`67199824 0100 add dword ptr [rax],eax 00000000`67199826 00488b add byte ptr [rax-75h],cl 00000000`67199829 cdff int 0FFh 00000000`6719982b 1568ad0100 adc eax,1AD68h 00000000`67199830 85c0 test eax,eax 00000000`67199832 0f85d8000000 jne DllA!DllUnregisterServer+0×84d0 (00000000`67199910) 00000000`67199838 488bcd mov rcx,rbp 00000000`6719983b ff1547ad0100call qword ptr [DllA!DllUnregisterServer+0×23148()]00000000`671b4588
0:009> dq 00000000`671b4588Memory dump analysis always starts when a user complains. In this case it was a hanging application from a document processing suit. The manual dump (Volume 1, page 486) was saved:
Loading Dump File [processA.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Userdump generated complete user-mode minidump with Standalone function on SERVER1'
Main thread stack trace shows a virtualized process (WOW64, Volume 1, page 400):
0:000> kL Child-SP RetAddr Call Site 00000000`0016e7b8 00000000`6b006a5awow64cpu!WaitForMultipleObjects32+0×3a 00000000`0016e860 00000000`6b0097f4wow64!RunCpuSimulation+0xa 00000000`0016e890 00000000`6b2936a2 wow64!Wow64KiUserCallbackDispatcher+0×114 00000000`0016ebd0 00000000`77ef317f wow64win!whcbfnDWORD+0xc2 00000000`0016ed80 00000000`78b842d9 ntdll!KiUserCallbackDispatcherContinue 00000000`0016ee08 00000000`78b8428e wow64cpu!CpupSyscallStub+0×9 00000000`0016ee10 00000000`00000000 wow64cpu!Thunk0Arg+0×5
We switch to x86 32-bit mode and get the right thread stack:
0:000> .load wow64exts 0:000> .effmach x86 Effective machine: x86 compatible (x86) 0:000:x86> kv ChildEBP RetAddr Args to Child 0012dcac 7d948836 009db2c0 00000000 0000004a user32!NtUserMessageCall+0x15 0012dcd0 30059282000b02960000004a 00000000user32!SendMessageW+0×82WARNING: Stack unwind information not available. Following frames may be wrong. 0012fef8 3000268e 02110024 30000000 b90fcc31 ApplicationA+0×59282 0012ff30 3000260b 30000000 00000000 0022245d ApplicationA+0×268e 0012ffc0 7d4e7d2a 00000000 00000000 7efde000 ApplicationA+0×260b 0012fff0 00000000 30001d28 00000000 00000000 kernel32!BaseProcessStart+0×28
We see that the main threads is blocked (Volume 2, page 184) by sending a synchronous message via SendMessage Win32 API function call. The first argument to it is a window handle. In our case it is 000b0296. It is also known that ApplicationA launches another ApplicationB (Coupled Process pattern, Volume 1, page 419) and its manual memory dump was saved too. It is also a virtualized process and its main GUI thread is blocked:
0:000:x86> kv 100 ChildEBP RetAddr Args to Child 0012ce80 7d4e286c 00000003 0012cecc 00000000 ntdll_7d600000!NtWaitForMultipleObjects+0x15 0012cf28 7d4e3e8e 00000003 0012cf6c 00000001 kernel32!WaitForMultipleObjectsEx+0x11a 0012cf44 0cc7c897 00000003 0012cf6c 00000001kernel32!WaitForMultipleObjects+0×18WARNING: Stack unwind information not available. Following frames may be wrong. 0012cf74 0cc7c990 ffffffff 0cc74b23 000000013rdPartyDLL+0xc897[...] 0012d814 7d947568 3a0b28d7 000b0296 00000002 user32!InternalCallWinProc+0×28 0012d88c 7d947d93 00000000 3a0b28d7 000b0296 user32!UserCallWinProcCheckWow+0×114 0012d8e8 7d947e46 009db2c0 00000000 00000002 user32!DispatchClientMessage+0xdf 0012d924 7d61ea0e 0012d93c 00000000 0012d9b8 user32!__fnDWORD+0×2b 0012d958 3a0baf6a 000b0296 02114600 0012d98c ntdll_7d600000!KiUserCallbackDispatcher+0×2e [...] 0012db28 7d947568 3a0b28d7 000b0296 00000010 user32!InternalCallWinProc+0×28 0012dba0 7d94778d 00000000 3a0b28d7 000b0296 user32!UserCallWinProcCheckWow+0×114 0012dc18 7d9477d0 0012dc88 00000000 0012dc4c user32!DispatchMessageWorker+0×37b 0012dc28 3a0b89ec0012dc8800000000 0219401cuser32!DispatchMessageW+0xf[...] 0012ffc0 7d4e7d2a 00000000 00000000 7efde000 ApplicationB+0×260b 0012fff0 00000000 30001d28 00000000 00000000 kernel32!BaseProcessStart+0×28
We see that it is blocked waiting for synchronization objects after receiving a message to the same window handle 000b0296 sent from ApplicationA:
0:000:x86> dd 0012dc88 l1
00000000`0012dc88 000b0296DispatchMessage function has its first argument as a pointer to an MSG structure with the first field as a window handle (HWND).
Looking at arguments to WaitForMultipleObjects we see that it is waiting for all three objects to be signalled simultaneously:
0012cf44 0cc7c89700000003 0012cf6c00000001kernel32!WaitForMultipleObjects+0×18 0:000:x86> dd 0012cf6c l3 00000000`0012cf6c 00001490 0000149c 00001494 0:000:x86> !handleHandle 0000000000001490 Type Mutant 0:000:x86> !handle00001490Handle 000000000000149c Type Event 0:000:x86> !handle0000149cHandle 0000000000001494 Type Mutant00001494
Because the waiting call was originated from 3rdPartyDLL module we can recommend contacting its vendor after determining the origin from the output of lmv command.
Here we can look at one process dump with many patterns seen inside. Default WinDbg analysis command !analyze -v points to an invalid handle (Volume 2, page 269) exception perhaps at DLL initialization during thread attach to DllA module:
STACK_TEXT: 0296fa68 7c90eb93 ntdll!KiRaiseUserExceptionDispatcher+0x37 0296fa7c 10001252 ntdll!KiFastSystemCallRet+0x4 WARNING: Stack unwind information not available. Following frames may be wrong.0296faa8 771215f8 DllA!DllMain+0×2020296fbec 100014b0 OLEAUT32!DllMain+0×2c0296fc0c 7c9011a7 DllA!DllMain+0×4600296fc2c 7c918f65 ntdll!LdrpCallInitRoutine+0×14 0296fca0 7c918dde ntdll!LdrpInitializeThread+0xc0 0296fd18 7c90eac7 ntdll!_LdrpInitialize+0×219 00000000 00000000 ntdll!KiUserApcDispatcher+0×7 EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 7c90eb74 (ntdll!KiRaiseUserExceptionDispatcher+0x00000037)ExceptionCode: c0000008 (Invalid handle)ExceptionFlags: 00000000 NumberParameters: 0Thread tried to close a handle that was invalid or illegal to close
We may stop here after applying lmv command to DllA and recommending upgrading or removing that component. But, let's look a bit deeper inside that crash dump. If we list all thread stacks (Stack Trace Collection pattern, Volume 1, page 409) we would see another thread with unhandled exception processing stack:
0:000> ~*kL
. 0 Id: a1c.e78 Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr
0012da34 7c90e9ab ntdll!KiFastSystemCallRet
0012da38 7c86372c ntdll!ZwWaitForMultipleObjects+0xc
0012e1a8 77c32f0f kernel32!UnhandledExceptionFilter+0×8e4
0012e1c4 0041808b msvcrt!_XcptFilter+0×161
0012ffc0 7c816fd7 Application!WinMainCRTStartup+0×14f
0012fff0 00000000 kernel32!BaseProcessStart+0×231 Id: a1c.2ec Suspend: 1 Teb: 7ffdc000 Unfrozen ChildEBP RetAddr 02faff84 7c90e9ab ntdll!KiFastSystemCallRet 02faff88 5b890f8c ntdll!ZwWaitForMultipleObjects+0xc 02faffb4 7c80b683 NETAPI32!NetbiosWaiter+0x73 02faffec 00000000 kernel32!BaseThreadStart+0x37 2 Id: a1c.c14 Suspend: 1 Teb: 7ffdb000 Unfrozen ChildEBP RetAddr 036afe1c 7c90e9ab ntdll!KiFastSystemCallRet 036afe20 7c8094e2 ntdll!ZwWaitForMultipleObjects+0xc 036afebc 7e4195f9 kernel32!WaitForMultipleObjectsEx+0x12c 036aff18 7e4196a8 USER32!RealMsgWaitForMultipleObjectsEx+0x13e 036aff34 00450d91 USER32!MsgWaitForMultipleObjects+0x1f 036aff80 77c3a3b0 Application!ThreadProc+0x61 036affb4 7c80b683 msvcrt!_endthreadex+0xa9 036affec 00000000 kernel32!BaseThreadStart+0x37 3 Id: a1c.15c Suspend: 1 Teb: 7ffda000 Unfrozen ChildEBP RetAddr 0417ff78 7c90e31b ntdll!KiFastSystemCallRet 0417ff7c 71a5d320 ntdll!ZwRemoveIoCompletion+0xc 0417ffb4 7c80b683 mswsock!SockAsyncThread+0x5a 0417ffec 00000000 kernel32!BaseThreadStart+0x37 # 4 Id: a1c.96c Suspend: 1 Teb: 7ffde000 Unfrozen ChildEBP RetAddr 0296fa68 7c90eb93 ntdll!KiRaiseUserExceptionDispatcher+0x37 0296fa7c 10001252 ntdll!KiFastSystemCallRet+0x4 WARNING: Stack unwind information not available. Following frames may be wrong. 0296faa8 771215f8 DllA!DllMain+0x202 0296fbec 100014b0 OLEAUT32!DllMain+0x2c 0296fc0c 7c9011a7 DllA!DllMain+0x460 0296fc2c 7c918f65 ntdll!LdrpCallInitRoutine+0x14 0296fca0 7c918dde ntdll!LdrpInitializeThread+0xc0 0296fd18 7c90eac7 ntdll!_LdrpInitialize+0x219 00000000 00000000 ntdll!KiUserApcDispatcher+0x7
Seems we have multiple exceptions (Volume 1, page 255) here. Let's extract thread 0 exception:
0:000> kv ChildEBP RetAddr Args to Child 0012da34 7c90e9ab 7c86372c 00000002 0012dbac ntdll!KiFastSystemCallRet 0012da38 7c86372c 00000002 0012dbac 00000001 ntdll!ZwWaitForMultipleObjects+0xc 0012e1a8 77c32f0f0012e1f000000000 00000000 kernel32!UnhandledExceptionFilter+0×8e4 0012e1c4 0041808b 00000000 0012e1f0 77c35cf5 msvcrt!_XcptFilter+0×161 0012ffc0 7c816fd7 00160000 001ae3c6 7ffdd000 Application!WinMainCRTStartup+0×14f 0012fff0 00000000 00417f3c 00000000 78746341 kernel32!BaseProcessStart+0×23 0:000> .exptr0012e1f0----- Exception record at 0012e2e4: ExceptionAddress: 77c47fd4 (msvcrt!wcslen+0x00000008)ExceptionCode: c0000005 (Access violation)ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: 04649000 Attempt to read from address 04649000 ----- Context record at 0012e300: eax=04649000 ebx=00000000 ecx=0464006c edx=04648fb4 esi=04648fd0 edi=00000000 eip=77c47fd4 esp=0012e5cc ebp=0012e5cc iopl=0 nv up ei pl nz na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010206 msvcrt!wcslen+0x8: 77c47fd4 668b08 mov cx,word ptr [eax] ds:0023:04649000=???? 0:000> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child0012e5cc 7301561a0012e5f0 73016c32 04648fd0 04afefe8 00000000 DllB!UnicodeToAnsiString+0×105 [...]04648fd000000030 00000018 msvcrt!wcslen+0×8
We see invalid pointer (Volume 1, page 267) access violation while calculating string length. If we look at invalid address we see that UNICODE string crosses page boundary into a reserved page:
0:000> dd 04648fd0 04648fd0 0060004d 00620066 00680072 0020006f 04648fe0 00200034 00630022 007100ea 00710060 04648ff0 00200073 0060006e 0076006f 006d006604649000 ???????? ???????? ???????? ????????04649010 ???????? ???????? ???????? ???????? 04649020 ???????? ???????? ???????? ???????? 04649030 ???????? ???????? ???????? ???????? 04649040 ???????? ???????? ???????? ???????? 0:000> !address 04648fd0 04648000 : 04648000 - 00001000 Type 00020000 MEM_PRIVATE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageIsVAD 0:000> !address04649000045e0000 : 04649000 - 00001000 Type 00040000 MEM_MAPPEDState 00002000 MEM_RESERVEUsage RegionUsageIsVAD
And we also notice full page heap enabled to catch possible heap corruption (Dynamic Memory Corruption pattern, Volume 1, page 257):
0:000> !gflag
Current NtGlobalFlag contents: 0x02000000
hpa - Place heap allocations at ends of pagesThis explains why we see invalid handle exception which is normally ignored by a runtime unless we enable Application Verifier. Looking at DllB version data we see that it is the old component (Not My Version pattern, Volume 2, page 299) that needs to be upgraded.
Here is the simplified example of an executive resource wait chain (Volume 2, page 147):
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...
Resource @ 0x88094118 Exclusively owned
Contention Count = 1461106
NumberOfExclusiveWaiters = 172
Threads: 87571600-01<*>
Threads Waiting On Exclusive Access:
87a0cd70 86e478b0 86d73270 87463908
86ed5020 872d3a08 87a0b228 87985020
870e4430 870adb00 88197500 86e06db0
87030db0 86d86db0 88a22288 86db07a0
86815c50 87524628 899d2020 86da03f0
86fc8db0 86e43b40 86d86ac8 87320690
86da2020 879c0108 86d8f7a8 86876370
87565150 87142020 879ddd30 86ff8990
86e5c770 867a7200 87a97c50 86e21020
86dac6e8 876d6db0 876fadb0 86e36408
86e621c8 8770adb0 86fd7c50 86db6ba8
86b87020 867ea2f8 870b60e8 889dc6d8
877ebae0 86e267a8 88a8a9f0 8737e5e8
86fc0780 87993c98 88aead28 872bedb0
899e5628 87523770 870aaaf0 8717b3b0
86e19db0 86e11db0 86e5a7a0 87038448
8743adb0 8816b9a0 880955f8 867f3db0
875c3430 8714a4f8 879b6020 87642598
86ec2b40 884a7c50 87200020 86880db0
86e2f988 866fb020 86ddfdb0 867c1bd8
86645020 868c0db0 87613db0 872b0020
88a56898 8770d9e0 8680b418 87014db0
865e0720 868c7af0 8733aaf0 86929508
8798f928 879cd378 8822ec50 8721adb0
876b25a0 87b5b598 8684baf0 86e48db0
86eb5b90 86d969a8 87039db0 87486020
86d8f3c0 8680edb0 86fddb88 885c2cb0
870ba890 86e2e4f8 8695b948 86e6fa28
88a42b88 86e58af0 86ddd2e0 8695b540
86817520 86975800 86817020 88b40b50
87271620 8695b2d0 867b44c8 880b6af8
870e1898 87c711e0 87a77210 8676bdb0
86734630 86878db0 86fd0c50 872a81f8
86e09020 880cf4f8 87178970 868a1508
870a9db0 8692c020 867a4020 868c9c50
890c74e0 8687c9a8 8692c4f8 880cf238
8708cac0 86ef5db0 86fa9db0 87158330
87979868 87a4f510 879a3510 87a1cdb0 87094020 87095db0 8705d2a8 87b0d5b0
870c0020 879eb660 8737e2e0 86ea7918
86e46a28 87a49198 87d61db0 87067db0
8730e598 86f97db0 8668d020 89d671b8
8732a5c8 89a00bb8 867fa020 86e2a020
KD: Scanning for held locks..
Resource @ 0x88aaabe8 Exclusively owned
Contention Count = 97373
Threads: 87178598-01<*>
KD: Scanning for held locks.
Resource @ 0x87712650 Exclusively owned
Contention Count = 41716
NumberOfExclusiveWaiters = 2
Threads: 87178598 -01<*>
Threads Waiting On Exclusive Access:
87571600 879f5648
KD: Scanning for held locks...
Resource @ 0x87736048 Exclusively owned
Contention Count = 29109
NumberOfExclusiveWaiters = 1
Threads: 87ab30d0This is a straightforward single wait chain (172 threads -> 87571600 -> 87178598 -> 87ab30d0) culminating in the thread 87ab30d0 which loops in kernel mode (Spiking Thread pattern, Volume 1, page 305):
0: kd> !threadTHREAD 87ab30d0 Cid 3814.322c Teb: 00000000 Win32Thread: 0000000087ab30d0RUNNING on processor 3Not impersonating DeviceMap e1006e10 Owning Process 889d6d88 Image: Application.exe Wait Start TickCount 2518917 Ticks: 0 Context Switch Count 4057707 UserTime 00:00:00.000KernelTime 01:26:13.906*** WARNING: Unable to verify timestamp for driverA.sys *** ERROR: Module load completed but symbols could not be loaded for driverA.dllStart Address driverA (0xbfa1c930)Stack Init ae8ec000 Current ae8eae7c Base ae8ec000 Limit ae8e9000 Call 0 Priority 13 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr Args to Child WARNING: Stack unwind information not available. Following frames may be wrong. f773d3b0 ae8eaf40 00000010 00000000 00000000driverA+0×25880
In one system even Task Manager process was hanging and a forced complete memory dump reveals a blocked (Volume 2, page 184) GUI dialog thread waiting for an LPC reply (page 97):
THREAD fffffade69e36310 Cid 4b30.4f34 Teb: 000007fffffde000 Win32Thread:
fffff97ff3f97620 WAIT: (Unknown) UserMode Non-Alertable
fffffade69e36678 Semaphore Limit 0x1
Waiting for reply to LPC MessageId a7531ec4:
Current LPC port fffffa8007118190
Not impersonating
DeviceMap fffffa80021c6240
Owning Process fffffadee0332040 Image: taskmgr.exe
Wait Start TickCount 80209499 Ticks: 5596 (0:00:01:27.437)
Context Switch Count 173 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.031
Win32 Start Address taskmgr!wWinMainCRTStartup (0×000000010001bfe0)
Start Address kernel32!BaseProcessStart (0×0000000077d59620)
Stack Init fffffad7c6e5ae00 Current fffffad7c6e5a8e0
Base fffffad7c6e5b000 Limit fffffad7c6e52000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Child-SP RetAddr Call Site
fffffad7`c6e5a920 fffff800`01027752 nt!KiSwapContext+0×85
fffffad7`c6e5aaa0 fffff800`0102835e nt!KiSwapThread+0×3c9
fffffad7`c6e5ab00 fffff800`01283ebb nt!KeWaitForSingleObject+0×5a6
fffffad7`c6e5ab80 fffff800`0102e3fd nt!NtRequestWaitReplyPort+0xa60
fffffad7`c6e5ac70 00000000`77ef0c0a nt!KiSystemServiceCopyEnd+0×3 (TrapFrame @
fffffad7`c6e5ac70)
00000000`000adef8 000007ff`7fd6df46 ntdll!ZwRequestWaitReplyPort+0xa
00000000`000adf00 000007ff`7fd6e17f RPCRT4!LRPC_CCALL::SendReceive+0×447
00000000`000adfe0 000007ff`7fe974c8 RPCRT4!NdrSendReceive+0xf6
00000000`000ae010 000007ff`7fe975bd RPCRT4!NdrpClientCall2+0×7e2
00000000`000ae5f0 000007ff`7d4d8a0d RPCRT4!NdrClientCall2+0×1d
00000000`000ae620 00000001`00011029 WINSTA!WinStationGetProcessSid+0xfd
00000000`000ae6a0 00000001`00010e3b taskmgr!CProcInfo::SetProcessUsername+0×189
00000000`000ae930 00000001`000118e9 taskmgr!CProcInfo::SetData+0×53b
00000000`000aebb0 00000001`000122f4 taskmgr!CProcPage::UpdateProcInfoArray+0×449
00000000`000aee70 00000001`00013eab taskmgr!CProcPage::TimerEvent+0×14
00000000`000aeea0 00000001`00004e8f taskmgr!CProcPage::Initialize+0×29b
00000000`000af320 00000001`000070e8 taskmgr!MainWnd_OnInitDialog+0×2ff
00000000`000af620 00000000`77c34582 taskmgr!MainWindowProc+0×3f8
00000000`000af680 00000000`77c34bf2 USER32!UserCallDlgProcCheckWow+0×210
00000000`000af760 00000000`77c2ff5f USER32!DefDlgProcWorker+0xf6
00000000`000af7f0 00000000`77c29beb USER32!InternalCreateDialog+0×16a7
00000000`000af980 00000001`00007c97 USER32!CreateDialogParamW+0xab
00000000`000afa00 00000001`0001c24b taskmgr!wWinMain+0×5f7
00000000`000afed0 00000000`77d5964c taskmgr!wWinMainCRTStartup+0×26b
00000000`000aff80 00000000`00000000 kernel32!BaseProcessStart+0×29We search for that LPC message:
6: kd> !lpc message a7531ec4
Searching message a7531ec4 in threads ...
Server thread fffffade6b8cf7a0 is working on message a7531ec4
Client thread fffffade69e36310 waiting a reply from
a7531ec4
[...]Then we examine the corresponding server thread (for visual clarity we show only one argument for each function call and use smaller font):
6: kd> !thread fffffade6b8cf7a0 16
THREAD fffffade6b8cf7a0 Cid 0238.8ff4 Teb: 000007fffff48000 Win32Thread: 0000000000000000 WAIT:
(Unknown) UserMode Non-Alertable
fffffade6f2e93c0 SynchronizationEvent
Impersonation token: fffffa8013a17700 (Level Impersonation)
DeviceMap fffffa80021c6240
Owning Process fffffade6f337040 Image: svchost.exe
Wait Start TickCount 80209499 Ticks: 5596 (0:00:01:27.437)
Context Switch Count 40138
UserTime 00:00:00.140
KernelTime 00:00:00.312
Win32 Start Address 0×00000000a7531ec4
LPC Server thread working on message Id a7531ec4
Start Address kernel32!BaseThreadStart (0×0000000077d6b6a0)
Stack Init fffffad867f5ae00 Current fffffad867f5a950
Base fffffad867f5b000 Limit fffffad867f55000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Child-SP RetAddr : Args to Child : Call Site
fffffad8`67f5a990 fffff800`01027752 : fffffa80`071181f0 : nt!KiSwapContext+0×85
fffffad8`67f5ab10 fffff800`0102835e : 00000000`00000000 : nt!KiSwapThread+0×3c9
fffffad8`67f5ab70 fffff800`0127f03f : 00000000`00000000 : nt!KeWaitForSingleObject+0×5a6
fffffad8`67f5abf0 fffff800`0102e3fd : fffffade`6b8cf7a0 : nt!NtWaitForSingleObject+0xc1
fffffad8`67f5ac70 00000000`77ef0a2a : 00000000`77efb9a6 : nt!KiSystemServiceCopyEnd+0×3 (TrapFrame
@ fffffad8`67f5ac70)
00000000`04d0f098 00000000`77efb9a6 : 00000000`00000000 : ntdll!NtWaitForSingleObject+0xa
00000000`04d0f0a0 00000000`77efba10 : 000007ff`7b8c3500 : ntdll!RtlpWaitOnCriticalSection+0×240
00000000`04d0f120 000007ff`7b887042 : 00000000`04d0f300 : ntdll!RtlEnterCriticalSection+0xa9
[...]
00000000`04d0f190 000007ff`7fe96cc9 : 00000000`00000048 : RPCRT4!Invoke+0×65
00000000`04d0f210 000007ff`7fe9758d : 00000000`04d0f8f0 : RPCRT4!NdrStubCall2+0×54d
00000000`04d0f7d0 000007ff`7fd697b4 : 00000000`0450a8fd : RPCRT4!NdrServerCall2+0×1d
00000000`04d0f800 000007ff`7fd6990d : 00000000`00377aa0 : RPCRT4!DispatchToStubInCNoAvrf+0×14
00000000`04d0f830 000007ff`7fd69766 : 00000000`00000001 :
RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×50d
00000000`04d0f9a0 000007ff`7fd6b214 : 00000000`06c18ba0 :
RPCRT4!RPC_INTERFACE::DispatchToStub+0×2ec
00000000`04d0fa20 000007ff`7fd6b9e3 : 00000000`06c18ba0 :
RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×63b
00000000`04d0fae0 000007ff`7fd7007c : 00000000`048348a0 :
RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×3bf
00000000`04d0fba0 000007ff`7fd45369 : 00000000`0525fd30 :
RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×710
00000000`04d0feb0 000007ff`7fd65996 : 00000000`00000000 : RPCRT4!RecvLotsaCallsWrapper+0×9
00000000`04d0fee0 000007ff`7fd65d51 : 00000000`0492de90 : RPCRT4!BaseCachedThreadRoutine+0xde
00000000`04d0ff50 00000000`77d6b6da : 00000000`77d6b6a0 : RPCRT4!ThreadStartRoutine+0×21
00000000`04d0ff80 00000000`00000000 : 00000000`00000000 : kernel32!BaseThreadStart+0×3aWe see the thread above is waiting for a critical section and we examine it using !cs WinDbg extension command after switching to the process context of that thread:
6: kd> .process /r /pfffffade6f337040Implicit process is now fffffade`6f337040 Loading User Symbols 6: kd> !csCritical section = 0×000007ff7b8c3500 (DllA!Lock+0×0) DebugInfo = 0×00000000000ffea0 LOCKED LockCount = 0×8 WaiterWoken = No000007ff`7b8c3500OwningThread = 0×0000000000004100RecursionCount = 0×1 LockSemaphore = 0×8E0 SpinCount = 0×0000000000000000
From the output we know the critical section owner thread and examine it too (for visual clarity function call arguments are not shown here):
6: kd> !thread -t410016 Looking for thread Cid = 4100 ... THREAD fffffade6b9df4f0 Cid 0238.4100 Teb: 000007fffff58000 Win32Thread: 0000000000000000 WAIT: (Unknown) UserMode Non-Alertablefffffadff5edadc0 Mutant - owning thread fffffadfe2486bf0Impersonation token: fffffa8010197250 (Level Impersonation) Owning Process fffffade6f337040 Image: svchost.exe Wait Start TickCount 79672315 Ticks: 542780 (0:02:21:20.937) Context Switch Count 174 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address 0×00000000a75007ac LPC Server thread working on message Id a75007ac Start Address kernel32!BaseThreadStart (0×0000000077d6b6a0) Stack Init fffffad83a94ee00 Current fffffad83a94e950 Base fffffad83a94f000 Limit fffffad83a949000 Call 0 Priority 8 BasePriority 8 PriorityDecrement 0 Kernel stack not resident. Child-SP RetAddr : Call Site fffffad8`3a94e990 fffff800`01027752 : nt!KiSwapContext+0×85 fffffad8`3a94eb10 fffff800`0102835e : nt!KiSwapThread+0×3c9 fffffad8`3a94eb70 fffff800`0127f03f : nt!KeWaitForSingleObject+0×5a6 fffffad8`3a94ebf0 fffff800`0102e3fd : nt!NtWaitForSingleObject+0xc1 fffffad8`3a94ec70 00000000`77ef0a2a : nt!KiSystemServiceCopyEnd+0×3 (TrapFrame @ fffffad8`3a94ec70) 00000000`03c8efa8 000007ff`7b880427 : ntdll!NtWaitForSingleObject+0xa [...] 00000000`03c8f1a0 000007ff`7fe96cc9 : RPCRT4!Invoke+0×65 00000000`03c8f210 000007ff`7fe9758d : RPCRT4!NdrStubCall2+0×54d 00000000`03c8f7d0 000007ff`7fd697b4 : RPCRT4!NdrServerCall2+0×1d 00000000`03c8f800 000007ff`7fd6990d : RPCRT4!DispatchToStubInCNoAvrf+0×14 00000000`03c8f830 000007ff`7fd69766 : RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×50d 00000000`03c8f9a0 000007ff`7fd6b214 : RPCRT4!RPC_INTERFACE::DispatchToStub+0×2ec 00000000`03c8fa20 000007ff`7fd6b9e3 : RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×63b 00000000`03c8fae0 000007ff`7fd7007c : RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×3bf 00000000`03c8fba0 000007ff`7fd45369 : RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×710 00000000`03c8feb0 000007ff`7fd65996 : RPCRT4!RecvLotsaCallsWrapper+0×9 00000000`03c8fee0 000007ff`7fd65d51 : RPCRT4!BaseCachedThreadRoutine+0xde 00000000`03c8ff50 00000000`77d6b6da : RPCRT4!ThreadStartRoutine+0×21 00000000`03c8ff80 00000000`00000000 : kernel32!BaseThreadStart+0×3a
We see that it is waiting for a mutant and examine its owner thread:
6: kd> !thread fffffadfe2486bf0 16
THREAD fffffadfe2486bf0 Cid 0238.7010 Teb: 000007fffff5e000 Win32Thread:
0000000000000000 WAIT: (Unknown) UserMode Non-Alertable
fffffadfe2486f58 Semaphore Limit 0x1
Waiting for reply to LPC MessageId a7507485:
Current LPC port fffffa8006822c80
Not impersonating
DeviceMap fffffa8000003790
Owning
Process fffffade6f337040 Image: svchost.exe
Wait Start TickCount 79744596 Ticks: 470499 (0:02:02:31.546)
Context Switch Count 29636
UserTime 00:00:00.062
KernelTime 00:00:00.156
Win32 Start Address 0×00000000a74ff6e4
LPC Server thread working on message Id a74ff6e4
Start Address kernel32!BaseThreadStart (0×0000000077d6b6a0)
Stack Init fffffad7d98a0e00 Current fffffad7d98a08e0
Base fffffad7d98a1000 Limit fffffad7d989b000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
Child-SP RetAddr : Call Site
fffffad7`d98a0920 fffff800`01027752 : nt!KiSwapContext+0×85
fffffad7`d98a0aa0 fffff800`0102835e : nt!KiSwapThread+0×3c9
fffffad7`d98a0b00 fffff800`01283ebb : nt!KeWaitForSingleObject+0×5a6
fffffad7`d98a0b80 fffff800`0102e3fd : nt!NtRequestWaitReplyPort+0xa60
fffffad7`d98a0c70 00000000`77ef0c0a : nt!KiSystemServiceCopyEnd+0×3
(TrapFrame @ fffffad7`d98a0c70)
00000000`0430dc18 000007ff`7fd6df46 : ntdll!ZwRequestWaitReplyPort+0xa
00000000`0430dc20 000007ff`7fd6e17f : RPCRT4!LRPC_CCALL::SendReceive+0×447
00000000`0430dd10 000007ff`7fe974c8 : RPCRT4!NdrSendReceive+0xf6
00000000`0430dd40 000007ff`7fe975bd : RPCRT4!NdrpClientCall2+0×7e2
00000000`0430e320 00000000`676b1dd7 : RPCRT4!NdrClientCall2+0×1d
[...]The thread above is waiting for an LPC message and we examine its corresponding server thread:
6: kd> !lpc message a7507485
Searching message a7507485 in threads ...
Client thread fffffadfe2486bf0 waiting a reply from
a7507485
Server thread fffffade6f3c9720 is working on message
a7507485
[...]6: kd> !thread fffffade6f3c9720 16
THREAD fffffade6f3c9720 Cid 6308.a1c8 Teb: 000000007ef24000 Win32Thread:
0000000000000000 WAIT: (Unknown) UserMode Non-Alertable
fffffade68d31a40 SynchronizationEvent
Not impersonating
DeviceMap fffffa80016ce1b0
Owning Process fffffade69c29430 Image: AppA.exe
Wait Start TickCount 79744596 Ticks: 470499 (0:02:02:31.546)
Context Switch Count 7230
UserTime 00:00:00.218
KernelTime 00:00:00.109
Win32 Start Address 0×00000000a7507485
LPC Server thread working on message Id a7507485
Start Address kernel32!BaseThreadStartThunk (0×000000007d4d1504)
Stack Init fffffad83f7e4e00 Current fffffad83f7e4950
Base fffffad83f7e5000 Limit fffffad83f7df000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
Child-SP RetAddr : Call Site
fffffad8`3f7e4990 fffff800`01027752 : nt!KiSwapContext+0×85
fffffad8`3f7e4b10 fffff800`0102835e : nt!KiSwapThread+0×3c9
fffffad8`3f7e4b70 fffff800`0127f03f : nt!KeWaitForSingleObject+0×5a6
fffffad8`3f7e4bf0 fffff800`0102e3fd : nt!NtWaitForSingleObject+0xc1
fffffad8`3f7e4c70 00000000`78b842d9 : nt!KiSystemServiceCopyEnd+0×3
(TrapFrame @ fffffad8`3f7e4c70)
00000000`0257f108 00000000`78b840e5 : wow64cpu!CpupSyscallStub+0×9
00000000`0257f110 00000000`6b006a5a : wow64cpu!Thunk0ArgReloadState+0×1a
00000000`0257f180 00000000`6b005e0d : wow64!RunCpuSimulation+0xa
00000000`0257f1b0 00000000`77f109f0 : wow64!Wow64LdrpInitialize+0×2ed
00000000`0257f6f0 00000000`77ef30a5 : ntdll!LdrpInitialize+0×2aa
00000000`0257f7d0 00000000`7d4d1504 : ntdll!KiUserApcDispatcher+0×15
(TrapFrame @ 00000000`0257fb38)
00000000`0257fcc8 00000000`00000000 : kernel32!BaseThreadStartThunkWe see the thread belongs to a virtualized WOW64 process (Volume 1, page 400).
In one complete memory dump taken from a system refusing user connections but not hung completely we can see the signs of past pool allocation failures (see Insufficient Memory pattern, Volume 1, page 440):
0: kd> !vm
[...]
PagedPool Usage: 47391 ( 189564 Kb)
PagedPool Maximum: 67584 ( 270336 Kb)
********** 981 pool allocations have failed **********
[...]We check paged pool usage but the output is inconsistent (Volume 1, page 269) as shown in bold italic font:
0: kd> !poolused 4
Sorting by Paged Pool Consumed
Pool Used:
NonPaged Paged
Tag Allocs Used Allocs Used
LSmi 0 0 -1 4294967240 BlockTypeMisc
PpEE 0 0 -1 4294967040 PNP_DEVICE_EVENT_ENTRY_TAG ,
Binary: nt!pnp
CM 58 2320 -1 4294967000 Configuration Manager
(registry) , Binary: nt!cm
SeSc 0 0 -65 4294966112 Captured Security Descriptor
, Binary: nt!se
RxMs 1 1096 -99 4294947312 misc.
CM38 0 0 -2 4294942720 Internal Configuration
manager allocations , Binary: nt!cm
RxFc 0 0 -8 4294879664 FCB
Lfs 0 0 -907 4294872976 Lfs allocations
xSMB 0 0 -179 4293500928 IFSKIT sample SMB mini-
redirector , Binary: smbmrx.sys
AAAA 4 224 581 51639048 UNKNOWN pooltag 'AAAA' , please
update pooltag.txt
BBBB 2 65664 2582 16362984 UNKNOWN pooltag 'BBBB' , please
update pooltag.txt
MmSt 0 0 10718 14944776 Mm section object prototype
ptes , Binary: nt!mm
[...]However, we see that drivers using AAAA and BBBB pool tags consumed almost 65 MB and we can search for them as described in Double Free pattern (Volume 1, page 387). Dumping processes we notice signs of possible a handle leak (Volume 1, page 327):
0: kd> !process 0 0
[...]
PROCESS 89ac09c0 SessionId: 0 Cid: 04b0 Peb: 7ffd5000 ParentCid:
0480
DirBase: cc210000 ObjectTable: e13991a0 HandleCount: 3329.
Image: csrss.exe
PROCESS 89ae4508 SessionId: 0 Cid: 07a8 Peb: 7ffdf000 ParentCid:
04f4
DirBase: cb330000 ObjectTable: e115c220 HandleCount: 4476.
Image: svchost.exe
PROCESS 868d1d88 SessionId: 0 Cid: 4120 Peb: 7ffd8000 ParentCid:
04f4
DirBase: 95558000 ObjectTable: e1135428 HandleCount: 2255.
Image: AppC.exe
[...]We see lots of threads in the process 89ae4508 waiting for an LPC reply:
0: kd> !thread 86b64388 1f
THREAD 86b64388 Cid 07a8.0fbc Teb: 7ff73000 Win32Thread: bc173ac0 WAIT:
(Unknown) UserMode Non-Alertable
86b64574 Semaphore Limit 0x1
Waiting for reply to LPC MessageId 06345018:
Current LPC port e169dd90
Impersonation token: e492a6c0 (Level Impersonation)
DeviceMap e1603ce8
Owning Process 89ae4508 Image: svchost.exe
Wait Start TickCount 148053822 Ticks: 23982406 (4:08:05:25.093)
Context Switch Count 11 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address DllA!ThreadEntry (0×752e27fe)
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 901e5000 Current 901e4c08 Base 901e5000 Limit 901e2000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.Now we follow the LPC wait chain (page 97):
0: kd> !lpc message 06345018
Searching message 6345018 in threads ...
Client thread 86b64388 waiting a reply from
6345018
Server thread 87f0d790 is working on message 6345018
[...]
0: kd> !thread 87f0d790 1f
THREAD 87f0d790 Cid 0de4.5b2c Teb: 7ff8f000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
87f0d97c Semaphore Limit 0x1
Waiting for reply to LPC MessageId 0634501d:
Current LPC port eb2e6450
Impersonation token: e93ff870 (Level Impersonation)
DeviceMap e1603ce8
Owning Process 89c36690 Image: AppA.exe
Wait Start TickCount 113650910 Ticks: 58385318 (10:13:24:30.593)
Context Switch Count 373
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0×06345018
LPC Server thread working on message Id 6345018
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8b0e6000 Current 8b0e5c08 Base 8b0e6000 Limit 8b0e3000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
0: kd> !lpc message 0634501d
Searching message 634501d in threads ...
Client thread 87f0d790 waiting a reply from
634501d
Server thread 89137780 is working on message 634501d
[...]0: kd> !thread 89137780 1f
THREAD 89137780 Cid 1884.41f8 Teb: 7ff90000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
8913796c Semaphore Limit 0x1
Waiting for reply to LPC MessageId 064aa11b:
Current LPC port ea3fc860
Impersonation token: e93ff870 (Level Impersonation)
DeviceMap e1603ce8
Owning Process 8a608020 Image: AppB.exe
Wait Start TickCount 148002015 Ticks: 24034213 (4:08:18:54.578)
Context Switch Count 700
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0×0634501d
LPC Server thread working on message Id 634501d
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8b749000 Current 8b748c08 Base 8b749000 Limit 8b746000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
0: kd> !lpc message 064aa11b
Searching message 64aa11b in threads ...
Client thread 89137780 waiting a reply from
64aa11b
Server thread 87acb728 is working on message 64aa11b
[...]
0: kd> !thread 87acb728 1f
THREAD 87acb728 Cid 4120.4078 Teb: 7ff3f000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
87acb914 Semaphore Limit 0x1
Waiting for reply to LPC MessageId 064aa127:
Current LPC port e7ec63f0
Not impersonating
DeviceMap e1003910
Owning Process 868d1d88 Image: AppC.exe
Wait Start TickCount 147996856 Ticks: 24039372 (4:08:20:15.187)
Context Switch Count 440
UserTime 00:00:00.812
KernelTime 00:00:00.015
Win32 Start Address 0×064aa11b
LPC Server thread working on message Id 64aa11b
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8ba35000 Current 8ba34c08 Base 8ba35000 Limit 8ba32000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
0: kd> !lpc message 064aa127
Searching message 64aa127 in threads ...
Server thread 899e1750 is working on message 64aa127
Client thread 87acb728 waiting a reply from
64aa127
[...]0: kd> !thread 899e1750 1f
THREAD 899e1750 Cid 0a0c.6cfc Teb: 7ff16000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
89293390 Mutant - owning thread 89a9dc38
Not impersonating
DeviceMap e1003910
Owning Process 892b8a38 Image: svchost.exe
Wait Start TickCount 148115996 Ticks: 23920232 (4:07:49:13.625)
Context Switch Count 166
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0×064aa127
LPC Server thread working on message Id 64aa127
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8c7b1000 Current 8c7b0c60 Base 8c7b1000 Limit 8c7ae000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.We finally come to a thread waiting for a mutant and we inspect its owner:
0: kd> !thread 89a9dc38 1f
THREAD 89a9dc38 Cid 0a0c.185c Teb: 7ff8d000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
89a9de24 Semaphore Limit 0x1
Waiting for reply to LPC MessageId 064a90f1:
Current LPC port e15992a8
Not impersonating
DeviceMap e1003910
Owning Process 892b8a38 Image: svchost.exe
Wait Start TickCount 148115996 Ticks: 23920232 (4:07:49:13.625)
Context Switch Count 29043
UserTime 00:00:01.046
KernelTime 00:00:00.968
Win32 Start Address 0×064a8fb6
LPC Server thread working on message Id 64a8fb6
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 92201000 Current 92200c08 Base 92201000 Limit 921fe000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.We see it is waiting for an LPC reply from the thread waiting for the mutant we have already seen:
0: kd> !lpc message 064a90f1
Searching message 64a90f1 in threads ...
Client thread 89a9dc38 waiting a reply from 64a90f1
Server thread 88806a28 is working on message 64a90f1
[...]0: kd> !thread 88806a28 1f
THREAD 88806a28 Cid 0a0c.10b8 Teb: 7ff82000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
89293390 Mutant - owning thread 89a9dc38
Not impersonating
DeviceMap e1003910
Owning Process 892b8a38 Image: svchost.exe
Wait Start TickCount 148115996 Ticks: 23920232 (4:07:49:13.625)
Context Switch Count 532
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address 0×064a90f1
LPC Server thread working on message Id 64a90f1
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 94ef3000 Current 94ef2c60 Base 94ef3000 Limit 94ef0000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.We have this dependency:
... -> Thread (899e1750) -> Mutant (89293390) Thread (89a9dc38) -> LPC (064a90f1) Thread (88806a28) -> Mutant (89293390) Thread (89a9dc38)
This looks like a deadlock (page 85) although we cannot examine stack traces which are not resident (stack data resides in a page file). We also notice the system uptime which might suggest that all these abnormalities had been gradually accumulated (see Overaged System pattern, Volume 2, page 273):
0: kd> version [...] System Uptime: 61 days 3:40:01.122 [...]
One Vista computer suddenly became slower than usual and we brought Task Manager, sorted processes by CPU usage and discovered an instance of IE7 with 50% - 60% of CPU consumption. Dumping processes in Vista is easier than ever, so we did the right click on that process and selected Create Dump File menu option. The dump was saved and we killed the process. The size of the memory dump file was 1.2 GB and that definitely indicated a memory leak. Examining process heap revealed large heap segments amounting to 800 MB and therefore pointing to the possible heap leak (Volume 1, page 356):
0:000> !heap 0 0
Index Address Name Debugging options enabled
1: 00370000
Segment at 00370000 to 00470000 (00100000 bytes committed)
Segment at 04990000 to 04a90000 (00100000 bytes committed)
Segment at 063e0000 to 065e0000 (00200000 bytes committed)
Segment at 08440000 to 08840000 (00400000 bytes committed)
Segment at 0ce80000 to 0d680000 (00800000 bytes committed)
Segment at 160b0000 to 17080000 (00fd0000 bytes committed)
Segment at 19b00000 to 1aad0000 (00fd0000 bytes committed)
Segment at 1c8c0000 to 1d890000 (00fd0000 bytes committed)
Segment at 27870000 to 28840000 (00fd0000 bytes committed)
Segment at 29870000 to 2a840000 (00fd0000 bytes committed)
Segment at 2d1f0000 to 2e1c0000 (00fd0000 bytes committed)
Segment at 31fb0000 to 32f80000 (00fd0000 bytes committed)
Segment at 384c0000 to 39490000 (00fd0000 bytes committed)
Segment at 3c040000 to 3d010000 (00fd0000 bytes committed)
Segment at 41cf0000 to 42cc0000 (00fd0000 bytes committed)
Segment at 43c90000 to 44c60000 (00fd0000 bytes committed)
Segment at 44c60000 to 45c30000 (00fd0000 bytes committed)
Segment at 473f0000 to 483c0000 (00fd0000 bytes committed)
Segment at 4a390000 to 4b360000 (00fd0000 bytes committed)
Segment at 4b360000 to 4c330000 (00fd0000 bytes committed)
Segment at 4d300000 to 4e2d0000 (00fd0000 bytes committed)
Segment at 4e2d0000 to 4f2a0000 (00fd0000 bytes committed)
Segment at 50480000 to 51450000 (00fd0000 bytes committed)
Segment at 51450000 to 52420000 (00fd0000 bytes committed)
Segment at 533f0000 to 543c0000 (00fd0000 bytes committed)
Segment at 54810000 to 557e0000 (00fd0000 bytes committed)
Segment at 567b0000 to 57780000 (00fd0000 bytes committed)
Segment at 57c80000 to 58c50000 (00fc1000 bytes committed)
Segment at 59c20000 to 5abf0000 (00fc6000 bytes committed)
Segment at 5b0f0000 to 5c0c0000 (00fc1000 bytes committed)
Segment at 5c0c0000 to 5d090000 (00fc1000 bytes committed)
Segment at 5d090000 to 5e060000 (00fc1000 bytes committed)
Segment at 5f030000 to 60000000 (00fc1000 bytes committed)
Segment at 60000000 to 60fd0000 (00fc1000 bytes committed) Segment at 60fd0000 to 61fa0000 (00fd0000 bytes committed)
Segment at 61fa0000 to 62f70000 (00e26000 bytes committed)
2: 00010000
Segment at 00010000 to 00020000 (00003000 bytes committed)
3: 00d80000
Segment at 00d80000 to 00d90000 (00010000 bytes committed)
Segment at 00050000 to 00150000 (00014000 bytes committed)
4: 00190000
Segment at 00190000 to 001a0000 (00010000 bytes committed)
Segment at 00d90000 to 00e90000 (00100000 bytes committed)
Segment at 0a430000 to 0a630000 (00200000 bytes committed)
Segment at 0d8d0000 to 0dcd0000 (00400000 bytes committed)
Segment at 0ecc0000 to 0f4c0000 (00800000 bytes committed)
Segment at 18690000 to 19660000 (00fd0000 bytes committed)
Segment at 24fe0000 to 25fb0000 (00fd0000 bytes committed)
Segment at 2bf40000 to 2cf10000 (00fd0000 bytes committed)
Segment at 303b0000 to 31380000 (00fd0000 bytes committed)
Segment at 33370000 to 34340000 (00fd0000 bytes committed)
Segment at 39490000 to 3a460000 (00fd0000 bytes committed)
Segment at 40d20000 to 41cf0000 (00fd0000 bytes committed)
Segment at 483c0000 to 49390000 (00fd0000 bytes committed)
Segment at 557e0000 to 567b0000 (00452000 bytes committed)
5: 00330000
Segment at 00330000 to 00340000 (00010000 bytes committed)
Segment at 00c10000 to 00d10000 (00100000 bytes committed)
Segment at 0c910000 to 0cb10000 (00200000 bytes committed)
Segment at 18280000 to 18680000 (00400000 bytes committed)
Segment at 2ec20000 to 2f420000 (00800000 bytes committed)
Segment at 42cc0000 to 43c90000 (00fc7000 bytes committed)
Segment at 4c330000 to 4d300000 (00d45000 bytes committed)
Segment at 52420000 to 533f0000 (00d39000 bytes committed)
Segment at 58c50000 to 59c20000 (00ddc000 bytes committed)
Segment at 5e060000 to 5f030000 (00dd1000 bytes committed)
6: 00e90000
Segment at 00e90000 to 00ea0000 (00010000 bytes committed)
Segment at 06780000 to 06880000 (00026000 bytes committed)
7: 00170000
Segment at 00170000 to 00180000 (00010000 bytes committed)
Segment at 06880000 to 06980000 (00026000 bytes committed)
8: 01bf0000
Segment at 01bf0000 to 01c00000 (00010000 bytes committed)
Segment at 03bb0000 to 03cb0000 (00100000 bytes committed)
Segment at 0e610000 to 0e810000 (00200000 bytes committed)
9: 00bf0000
Segment at 00bf0000 to 00c00000 (00001000 bytes committed)
10: 00b70000
Segment at 00b70000 to 00b80000 (00003000 bytes committed)
11: 01b60000
Segment at 01b60000 to 01ba0000 (00040000 bytes committed)
12: 03650000
Segment at 03650000 to 03690000 (00009000 bytes committed)
13: 039c0000
Segment at 039c0000 to 039d0000 (00008000 bytes committed)
Segment at 07e30000 to 07f30000 (00012000 bytes committed) 14: 00b20000
Segment at 00b20000 to 00b30000 (00003000 bytes committed)
15: 01b00000
Segment at 01b00000 to 01b40000 (00040000 bytes committed)
Segment at 22b80000 to 22c80000 (00032000 bytes committed)
16: 00b30000
Segment at 00b30000 to 00b70000 (00040000 bytes committed)
Segment at 08f00000 to 09000000 (00100000 bytes committed)
Segment at 376f0000 to 378f0000 (000e3000 bytes committed)
17: 03700000
Segment at 03700000 to 03740000 (00040000 bytes committed)
18: 03a70000
Segment at 03a70000 to 03ab0000 (00040000 bytes committed)
19: 00be0000
Segment at 00be0000 to 00bf0000 (00010000 bytes committed)
Segment at 0a630000 to 0a730000 (000a8000 bytes committed)
20: 04df0000
Segment at 04df0000 to 04ef0000 (00100000 bytes committed)
21: 044d0000
Segment at 044d0000 to 044e0000 (00010000 bytes committed)
Segment at 04390000 to 04490000 (00028000 bytes committed)
22: 04730000
Segment at 04730000 to 04740000 (00010000 bytes committed)
Segment at 04620000 to 04720000 (00100000 bytes committed)
Segment at 23fb0000 to 241b0000 (001f6000 bytes committed)
23: 055e0000
Segment at 055e0000 to 056e0000 (00100000 bytes committed)
24: 05ce0000
Segment at 05ce0000 to 05cf0000 (00010000 bytes committed)
Segment at 06bb0000 to 06cb0000 (00012000 bytes committed)
25: 05e20000
Segment at 05e20000 to 05e60000 (00020000 bytes committed)
26: 04860000
Segment at 04860000 to 04870000 (00010000 bytes committed)
Segment at 0df60000 to 0e060000 (00024000 bytes committed)
27: 04dc0000
Segment at 04dc0000 to 04dd0000 (00010000 bytes committed)
Segment at 062e0000 to 063e0000 (00100000 bytes committed)
Segment at 26d70000 to 26f70000 (001eb000 bytes committed)
28: 06aa0000
Segment at 06aa0000 to 06ab0000 (00010000 bytes committed)
Segment at 06980000 to 06a80000 (00100000 bytes committed)
Segment at 1ede0000 to 1efe0000 (00200000 bytes committed)
Segment at 1efe0000 to 1f3e0000 (00322000 bytes committed)
Segment at 1f3e0000 to 1fbe0000 (00800000 bytes committed)
Segment at 205e0000 to 215b0000 (001c7000 bytes committed)
29: 05420000
Segment at 05420000 to 05430000 (00010000 bytes committed)
Segment at 06ab0000 to 06bb0000 (00053000 bytes committed)
30: 05980000
Segment at 05980000 to 05990000 (00010000 bytes committed)
Segment at 17d90000 to 17e90000 (00012000 bytes committed)
31: 07c20000 Segment at 07c20000 to 07c60000 (00040000 bytes committed)
Segment at 08cc0000 to 08dc0000 (00100000 bytes committed)
Segment at 1fbe0000 to 1fde0000 (001fd000 bytes committed)
Segment at 241b0000 to 245b0000 (003fa000 bytes committed)
Segment at 2a840000 to 2b040000 (0007c000 bytes committed)
32: 07be0000
Segment at 07be0000 to 07c20000 (0003a000 bytes committed)
Segment at 17900000 to 17a00000 (000fd000 bytes committed)
Segment at 3b2b0000 to 3b4b0000 (001fe000 bytes committed)
Segment at 45c30000 to 46030000 (00289000 bytes committed)
33: 07df0000
Segment at 07df0000 to 07e30000 (0003a000 bytes committed)
Segment at 22810000 to 22910000 (0001c000 bytes committed)
34: 08000000
Segment at 08000000 to 08040000 (00001000 bytes committed)
35: 07da0000
Segment at 07da0000 to 07de0000 (00001000 bytes committed)
36: 04b60000
Segment at 04b60000 to 04b70000 (00002000 bytes committed)
37: 08990000
Segment at 08990000 to 089a0000 (00010000 bytes committed)
Segment at 06cb0000 to 06db0000 (00024000 bytes committed)
38: 051f0000
Segment at 051f0000 to 05200000 (00010000 bytes committed)
Segment at 050c0000 to 051c0000 (00100000 bytes committed)
Segment at 0c110000 to 0c310000 (00200000 bytes committed)
Segment at 0c310000 to 0c710000 (003f6000 bytes committed)
Segment at 1bd00000 to 1c500000 (00529000 bytes committed)
Segment at 216c0000 to 22690000 (00376000 bytes committed)
39: 0ac10000
Segment at 0ac10000 to 0ac20000 (00010000 bytes committed)
Segment at 0aa80000 to 0ab80000 (000c4000 bytes committed)
40: 12ed0000
Segment at 12ed0000 to 12ee0000 (00010000 bytes committed)
Segment at 199e0000 to 19ae0000 (00022000 bytes committed)
41: 15450000
Segment at 15450000 to 15490000 (00001000 bytes committed)
42: 17ad0000
Segment at 17ad0000 to 17b10000 (00001000 bytes committed)
43: 1b2f0000
Segment at 1b2f0000 to 1b300000 (00010000 bytes committed)
Segment at 1ad30000 to 1ae30000 (0002c000 bytes committed)
44: 232b0000
Segment at 232b0000 to 232f0000 (00015000 bytes committed)
45: 21680000
Segment at 21680000 to 216c0000 (00001000 bytes committed)
46: 23490000
Segment at 23490000 to 234d0000 (00001000 bytes committed)
47: 23670000
Segment at 23670000 to 236b0000 (00001000 bytes committed)
48: 17ed0000
Segment at 17ed0000 to 17f10000 (00001000 bytes committed)
49: 247f0000
Segment at 247f0000 to 24830000 (00040000 bytes committed) 50: 28c40000
Segment at 28c40000 to 28c80000 (00040000 bytes committed)
51: 2ffd0000
Segment at 2ffd0000 to 2ffe0000 (00006000 bytes committed)
52: 376b0000
Segment at 376b0000 to 376f0000 (00040000 bytes committed)
53: 2ff90000
Segment at 2ff90000 to 2ffd0000 (00040000 bytes committed)
54: 26260000
Segment at 26260000 to 262a0000 (00040000 bytes committed)
55: 3a530000
Segment at 3a530000 to 3a570000 (00040000 bytes committed)However, we concentrated on CPU spike (Volume 1, page 305) and !runaway WinDbg command showed the following distribution of thread user mode times:
0:000> !runaway
User Mode Time
Thread Time
117:10a0 0 days 3:09:13.643
13:ca4 0 days 2:18:41.311
61:16c4 0 days 0:25:46.515
33:1690 0 days 0:25:25.954
4:fb0 0 days 0:22:20.797
29:840 0 days 0:21:25.385
23:1614 0 days 0:21:08.194
77:3e0 0 days 0:18:57.434
45:11f4 0 days 0:17:13.647
71:1314 0 days 0:17:10.667
31:1198 0 days 0:16:48.374
39:156c 0 days 0:16:40.980
59:d1c 0 days 0:16:37.610
115:3e8 0 days 0:16:32.384
57:170c 0 days 0:16:30.746
47:1364 0 days 0:16:18.360
84:12a8 0 days 0:15:56.145
112:a10 0 days 0:15:52.089
106:1374 0 days 0:15:51.652
89:b58 0 days 0:15:47.768
125:115c 0 days 0:15:41.122
101:1100 0 days 0:15:30.748
104:1294 0 days 0:15:16.147
99:d00 0 days 0:15:15.008
96:9b4 0 days 0:15:13.604
123:1624 0 days 0:15:12.247
86:1444 0 days 0:15:11.654
131:1728 0 days 0:14:35.914
135:100c 0 days 0:14:16.414
133:1530 0 days 0:14:04.963
137:a30 0 days 0:13:41.360
139:dd8 0 days 0:13:40.674
142:1098 0 days 0:12:51.2840:efc 0 days 0:02:43.005 1:f44 0 days 0:01:34.536 19:8d0 0 days 0:00:42.557 98:54c 0 days 0:00:28.282 114:138c 0 days 0:00:26.598 83:1060 0 days 0:00:22.354 88:17ec 0 days 0:00:22.027 103:da8 0 days 0:00:20.404 141:15c8 0 days 0:00:19.843 10:b14 0 days 0:00:12.526 8:5b8 0 days 0:00:02.246 21:cfc 0 days 0:00:00.795 12:10c 0 days 0:00:00.561 11:8d4 0 days 0:00:00.312 65:b0c 0 days 0:00:00.202 22:ae8 0 days 0:00:00.187 17:744 0 days 0:00:00.124 28:168c 0 days 0:00:00.093 6:5a8 0 days 0:00:00.046 2:f90 0 days 0:00:00.031 130:fa4 0 days 0:00:00.015 113:17c4 0 days 0:00:00.015 76:1a4 0 days 0:00:00.015 70:10a8 0 days 0:00:00.015 32:df0 0 days 0:00:00.015 18:ee0 0 days 0:00:00.015 7:3f4 0 days 0:00:00.015 148:11cc 0 days 0:00:00.000 147:132c 0 days 0:00:00.000 146:1458 0 days 0:00:00.000 145:133c 0 days 0:00:00.000 144:1268 0 days 0:00:00.000 143:838 0 days 0:00:00.000 140:1168 0 days 0:00:00.000 138:f48 0 days 0:00:00.000 136:1f0 0 days 0:00:00.000 134:17ac 0 days 0:00:00.000 132:119c 0 days 0:00:00.000 129:fc4 0 days 0:00:00.000 128:bd8 0 days 0:00:00.000 127:1528 0 days 0:00:00.000 126:1058 0 days 0:00:00.000 124:16a4 0 days 0:00:00.000 122:1518 0 days 0:00:00.000 121:7c 0 days 0:00:00.000 120:103c 0 days 0:00:00.000 119:a2c 0 days 0:00:00.000 118:1524 0 days 0:00:00.000 116:1240 0 days 0:00:00.000 111:1248 0 days 0:00:00.000 110:de8 0 days 0:00:00.000 109:dc8 0 days 0:00:00.000 108:17e8 0 days 0:00:00.000 107:994 0 days 0:00:00.000
105:162c 0 days 0:00:00.000 102:112c 0 days 0:00:00.000 100:1764 0 days 0:00:00.000 97:1548 0 days 0:00:00.000 95:1334 0 days 0:00:00.000 94:1024 0 days 0:00:00.000 93:1170 0 days 0:00:00.000 92:12f0 0 days 0:00:00.000 91:12d4 0 days 0:00:00.000 90:1264 0 days 0:00:00.000 87:12d8 0 days 0:00:00.000 85:153c 0 days 0:00:00.000 82:14c4 0 days 0:00:00.000 81:834 0 days 0:00:00.000 80:17f4 0 days 0:00:00.000 79:1784 0 days 0:00:00.000 78:530 0 days 0:00:00.000 75:1320 0 days 0:00:00.000 74:15fc 0 days 0:00:00.000 73:16e4 0 days 0:00:00.000 72:17b0 0 days 0:00:00.000 69:af0 0 days 0:00:00.000 68:83c 0 days 0:00:00.000 67:b78 0 days 0:00:00.000 66:cc4 0 days 0:00:00.000 64:14fc 0 days 0:00:00.000 63:14dc 0 days 0:00:00.000 62:16b0 0 days 0:00:00.000 60:1130 0 days 0:00:00.000 58:1504 0 days 0:00:00.000 56:1160 0 days 0:00:00.000 55:16c0 0 days 0:00:00.000 54:bfc 0 days 0:00:00.000 53:f70 0 days 0:00:00.000 52:1178 0 days 0:00:00.000 51:1448 0 days 0:00:00.000 50:15e8 0 days 0:00:00.000 49:1410 0 days 0:00:00.000 48:10c0 0 days 0:00:00.000 46:14e4 0 days 0:00:00.000 44:1150 0 days 0:00:00.000 43:1454 0 days 0:00:00.000 42:131c 0 days 0:00:00.000 41:8cc 0 days 0:00:00.000 40:17bc 0 days 0:00:00.000 38:17c0 0 days 0:00:00.000 37:15a4 0 days 0:00:00.000 36:1048 0 days 0:00:00.000 35:143c 0 days 0:00:00.000 34:1384 0 days 0:00:00.000 30:fa0 0 days 0:00:00.000 27:1688 0 days 0:00:00.000 26:1684 0 days 0:00:00.000
25:1680 0 days 0:00:00.000 24:161c 0 days 0:00:00.000 20:500 0 days 0:00:00.000 16:1a0 0 days 0:00:00.000 15:a18 0 days 0:00:00.000 14:c44 0 days 0:00:00.000 9:6c4 0 days 0:00:00.000 5:ec8 0 days 0:00:00.000 3:fa8 0 days 0:00:00.000
Threads 117 and 13 were waiting for the critical section 6e1876c4 (Volume 1, page 490):
0:000> ~117kv ChildEBP RetAddr Args to Child 35f0e468 77009254 76ff33b4 00000520 00000000 ntdll!KiFastSystemCallRet 35f0e46c 76ff33b4 00000520 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc 35f0e4d0 76ff323c 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x155 35f0e4f8 6e16ac32 6e1876c4 00071370 35f0e59c ntdll!RtlEnterCriticalSection+0x15235f0e510 6e16b4cc 6e16e2f1 00000000 35f0e59c AcRedir!NS_RedirectRegistry::RedirectorRegistry::LookupKOECache+0×22 35f0e524 6e16bb90 00071370 00000000 00000000 AcRedir!NS_RedirectRegistry::RedirectorRegistry::PreChecks+0xd3 35f0e544 6e16bbce 00071370 00000000 00000008 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeMergeW+0×1a 35f0e574 6e16e327 00071370 00000002 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×26*** ERROR: Symbol file could not be found. Defaulted to export symbols for PDFCreator_Toolbar.dll -35f0e620 05729772 00071370 00000002 35f0e690 AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36WARNING: Stack unwind information not available. Following frames may be wrong.35f0e6dc 73207be1 000319f8 00000128 00030001 user32!DefWindowProcW+0×86 76b60528 90909090 fffffffe 00000000 ffffffd0 comctl32!ToolbarWndProc+0×14f7 76b60528 00000000 fffffffe 00000000 ffffffd0 0×90909090 0:000> ~13kv ChildEBP RetAddr Args to Child 0c90e5ec 77009254 76ff33b4 00000520 00000000 ntdll!KiFastSystemCallRet 0c90e5f0 76ff33b4 00000520 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc 0c90e654 76ff323c 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x155 0c90e67c 6e16ac32 6e1876c4 00071348 0c90e720 ntdll!RtlEnterCriticalSection+0x15235f0e6a4 76b60528 c02193db 00000128 00000000 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce0c90e694 6e16b4cc 6e16e2f1 00000000 0c90e720 AcRedir!NS_RedirectRegistry::RedirectorRegistry::LookupKOECache+0×22 0c90e6a8 6e16bb90 00071348 00000000 00000000 AcRedir!NS_RedirectRegistry::RedirectorRegistry::PreChecks+0xd3 0c90e6c8 6e16bbce 00071348 00000000 00000008 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeMergeW+0×1a 0c90e6f8 6e16e327 00071348 0000000c 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×26 0c90e7a4 05729772 00071348 0000000c 0c90e814 AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36WARNING: Stack unwind information not available. Following frames may be wrong.0c90e8d4 76b5f8d2 626f6441 44502065 00200046 user32!DefWindowProcW+0×86 0c90e978 76b60817 0041fecc 73207ae0 000205e2 user32!InternalCallWinProc+0×23 00030ad4 0031002e 00300038 00350036 006e005f user32!DispatchClientMessage+0xda 00030ad4 00000000 00300038 00350036 006e005f 0×31002e0c90e858 76b60528 73207be1 000205e2 00000128 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
Examining critical section locks showed this section to be the only one locked with high contention (Volume 2, page 341):
0:000> !locks CritSec AcRedir!NS_RedirectRegistry::RedirectorRegistry::ClassLock+0 at 6e1876c4 WaiterWoken NoLockCount 32RecursionCount 1OwningThread d1cEntryCount 0ContentionCount c74ad4*** Locked Scanned 22054 critical sections
There were 32 threads waiting for it. Examining its owning thread d1c revealed similar stack trace pattern:
0:000> ~~[d1c]kv
ChildEBP RetAddr Args to Child
269ae72c 6e16f1da 269ae808 31f4a7e8 269ae75c
AcRedir!NS_RedirectRegistry::OwnedRegistryKeyPair::Match+0×14
269ae73c 6e16f40c 269ae7ec 269ae808 269ae808
AcRedir!NS_RedirectRegistry::MergedRegistryKey::Match+0×22
269ae75c 6e16bc11 269ae7ec 269ae808 269ae784
AcRedir!NS_RedirectRegistry::MergedRegistryKeyList::FindItem+0×25
269ae790 6e16e327 00c211b0 00000008 00000002
AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×69
269ae83c 05729772 000714a4 00000008 269ae8ac
AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
269ae8f0 76b60528 73207be1 00050cf8 00000128 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ceTwo components immediately came to suspicion, AcRedir.dll and PDFCreator_Toolbar.dll:
0:000> lmv m AcRedir
start end module name
6e150000 6e18e000 AcRedir (pdb
symbols) c:mssAcRedir.pdb923AF38F594246C99580DC1CFB4B3AE02AcR
edir.pdb
Loaded symbol image file: AcRedir.dll
Image path: C:WindowsAppPatchAcRedir.dll
Image name: AcRedir.dll
Timestamp: Sat Jan 19 07:26:39 2008 (4791A62F)
CheckSum: 0003F278
ImageSize: 0003E000
File version: 6.0.6001.18000
Product version: 6.0.6001.18000
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: Microsoft® Windows® Operating System
OriginalFilename: Microsoft® Windows® Operating System
ProductVersion: 6.0.6001.18000
FileVersion: 6.0.6001.18000 (longhorn_rtm.080118-1840)
FileDescription: Windows Compatibility DLL
LegalCopyright: © Microsoft Corporation. All rights reserved.
0:000> lmv m PDFCreator_Toolbar
start end module name
056e0000 057bb000 PDFCreator_Toolbar (export
symbols) PDFCreator_Toolbar.dll
Loaded symbol image file: PDFCreator_Toolbar.dll
Image path: C:Program FilesPDFCreator
Toolbarv3.3.0.1PDFCreator_Toolbar.dll
Image name: PDFCreator_Toolbar.dll
Timestamp: Sat Aug 09 08:53:38 2008 (489D4D02)
CheckSum: 000AA334
ImageSize: 000DB000
File version: 3.3.0.1
Product version: 3.3.0.1
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
ProductName: PDFCreator Toolbar
InternalName: PDFCreator Toolbar
OriginalFilename: Toolbar.dll
ProductVersion: 3,3,0,1
FileVersion: 3,3,0,1
FileDescription: PDFCreator Toolbar
LegalCopyright: Copyright 2006Then we decided to examine some heap blocks from leaked segments and found the prolifiration of UNICODE string fragments containing "PDFCreator Toolbar":
0:000> dc 567b0000 l3000 [...] 567b21a0 00000001 00000008 00000040 00650054 ............T.e. 567b21b0 0070006d 00610044 00610074 00730000 m.p.D.a.t.a...s. 567b21c0 00740069 006f0069 0000006e 00000072 i.t.i.o.n...r... 567b21d0 00000068 005c0067 00440050 00430046h...g .P.D.F.C. 567b21e0 00650072 00740061 0072006f 00540020 r.e.a.t.o.r. .T. 567b21f0 006f006f 0062006c 00720061 002d0000 o.o.l.b.a.r -.567b2200 00300031 00300030 00300000 00420025 1.0.0.0...0.% 567b2210 00250030 00310044 00380025 00250031 0.%.D.1.%.8.1.%. 567b2220 00310044 00380025 00000031 00000000 D.1.%.8.1 567b2230 52332e04 88000000 00000001 00000013 ..3R............ 567b2240 00000040 00690044 00450064 0061006e @...D.i.d.E.n 567b2250 006c0062 00410065 00740075 0053006f b.l.e.A.u.t.o.S. 567b2260 00610065 00630072 00000068 005c0067 e.a.r.c.h...g . 567b2270 00440050 00430046 00650072 00740061P.D.F.C.r.e.a.t. 567b2280 0072006f 00540020 006f006f 0062006c o.r. .T.o.o.l.b. 567b2290 00720061 002d0000 00300031 00300030 a.r... -.1.0.0.0.567b22a0 00300000 00420025 00250030 00310044 ..0.%.B.0.%.D.1. 567b22b0 00380025 00250031 00310044 00380025 %.8.1.%.D.1.%.8. 567b22c0 00000031 00000000 52332e1b 88000000 1.........3R.... 567b22d0 00000001 00000005 00000040 004c0053 ............S.L. 567b22e0 00730069 00000074 00450052 0070005c i.s.t...R.E. .p. 567b22f0 00660064 006f0066 00670072 002e0065 d.f.f.o.r.g 567b2300 0072006f 005c0067 00440050 00430046o.r.g..P.D.F.C. 567b2310 00650072 00740061 0072006f 00540020 r.e.a.t.o.r. .T. 567b2320 006f006f 0062006c 00720061 00300000 o.o.l.b.a.r...0.567b2330 00420025 00250042 00300044 00420025 %.B.B.%.D.0.%.B. 567b2340 00250030 00310044 00380025 00250031 0.%.D.1.%.8.1.%. 567b2350 00310044 00380025 00000031 00000000 D.1.%.8.1 567b2360 52332e2e 88000000 00000001 00000005 ..3R............
Because AcRedir code was called from PDFCreator Toolbar component the final decision was to uninstall PDFCreator Toolbar. Before we quit the debugger we dumped the list of modules and were astonished at the module variety (Volume 1, page 310):
0:000> lm start end module name 00850000 008eb000 iexplore (pdb symbols) 05430000 054ab000 ssv (deferred) 056e0000 057bb000 PDFCreator_Toolbar (export symbols) 0a7f0000 0aa73000 igdumd32 (deferred) 10000000 100a4000 swg (deferred) 16080000 160a5000 mdnsNSP (deferred) 28f90000 28f9a000 icalogon (deferred) 29330000 29337000 PScript (deferred) 29440000 29462000 ctxmui (deferred)
29470000 29476000 TcpPServ (deferred) 29480000 29492000 CgpCore (deferred) 295a0000 295b0000 confmgr (deferred) 295b0000 295b7000 logging (deferred) 296c0000 296c7000 icafile (deferred) 296d0000 296d6000 cgpcfg (deferred) 296e0000 296e5000 ctxmuiUI (deferred) 2bd20000 2bd8e000 Wfica (deferred) 30000000 303ae000 Flash9e (export symbols) 63f00000 63f0c000 mscorie (deferred) 655e0000 65639000 rpbrowserrecordplugin (deferred) 692a0000 69a66000 wmploc (deferred) 69a70000 6a4a0000 wmp (deferred) 6b220000 6b2e3000 VGX (deferred) 6b2f0000 6b3bc000 d3dim700 (deferred) 6b450000 6b4d3000 AdobeUpdater (deferred) 6b4e0000 6b7de000 agcore (deferred) 6b7e0000 6ba63000 fastsearch_219B3E1547538286 (deferred) 6ba70000 6be17000 GoogleToolbarDynamic_F423308312A7B033 (export symbols) 6be20000 6be89000 vbscript (deferred) 6bf90000 6c302000 mshtml (export symbols) 6c320000 6c36a000 ntshrui (deferred) 6c3d0000 6c447000 mshtmled (deferred) 6c4d0000 6c527000 dxtmsft (deferred) 6c590000 6c60d000 jscript (pdb symbols) 6c610000 6c649000 dxtrans (pdb symbols) 6c770000 6c7e0000 dsound (deferred) 6c810000 6c839000 msls31 (deferred) 6ca50000 6ca6b000 cryptnet (deferred) 6ca90000 6cada000 rasapi32 (deferred) 6cca0000 6ccb4000 rasman (deferred) 6cd40000 6cd71000 tapi32 (deferred) 6d0b0000 6d0fc000 Wpc (deferred) 6d350000 6d410000 npctrl (deferred) 6d450000 6d482000 iepeers (deferred) 6d4d0000 6d530000 ieapfltr (deferred) 6d5b0000 6d603000 AcroIEFavClient (deferred) 6d6b0000 6d795000 ddraw (deferred) 6d7a0000 6d818000 AcSpecfc (deferred) 6d820000 6d82e000 pngfilt (deferred) 6d830000 6d892000 mscms (deferred) 6dbc0000 6dc5b000 msvcr80 (deferred) 6dc60000 6dce7000 msvcp80 (deferred) 6dd70000 6ddf8000 AcLayers (deferred) 6de00000 6de0a000 ddrawex (deferred) 6de60000 6de83000 msvfw32 (deferred) 6dfb0000 6dfc1000 AcroIEHelperShim (deferred) 6dff0000 6e036000 GoogleToolbar (deferred) 6e060000 6e086000 dssenh (deferred) 6e090000 6e0f0000 tiptsf (deferred) 6e0f0000 6e11f000 ieui (pdb symbols) 6e130000 6e140000 AcroIEHelper (deferred) 6e150000 6e18e000 AcRedir (pdb symbols)
6e570000 6e57b000 msimtf (deferred) 6e580000 6e58f000 davclnt (deferred) 6e590000 6e5a3000 ntlanman (deferred) 6e610000 6e618000 drprov (deferred) 6e620000 6e630000 iebrshim (deferred) 6e650000 6e680000 mlang (deferred) 6f7b0000 6f7b8000 dispex (deferred) 6f8a0000 6f8ab000 cscapi (deferred) 6fa70000 6fb4c000 dbghelp (deferred) 6fe40000 6fe73000 msrating (deferred) 6ff00000 6ff3a000 sqlite (deferred) 70530000 70afe000 ieframe (pdb symbols) 71260000 71462000 msi (deferred) 717c0000 717d2000 pnrpnsp (deferred) 71870000 71877000 wsock32 (deferred) 718a0000 718a7000 msiltcfg (pdb symbols) 71920000 71973000 actxprxy (deferred) 71980000 7198c000 wshbth (deferred) 71990000 71998000 winrnr (deferred) 719e0000 71b06000 msxml3 (deferred) 71b10000 71b1f000 NapiNSP (deferred) 71b20000 71b29000 linkinfo (deferred) 71c70000 71c76000 SensApi (deferred) 71d10000 71e56000 browseui (deferred) 71ee0000 71fe7000 shdocvw (deferred) 72100000 72109000 snmpapi (deferred) 72580000 725c2000 winspool (deferred) 725d0000 725d6000 rasadhlp (deferred) 72610000 72615000 sfc (deferred) 72620000 7262c000 dwmapi (deferred) 72640000 72676000 mfplat (deferred) 72850000 72857000 midimap (deferred) 72860000 72874000 msacm32_72860000 (deferred) 72880000 72933000 WindowsCodecs (deferred) 72940000 729a6000 AudioEng (deferred) 729b0000 729d1000 AudioSes (deferred) 729e0000 72a0f000 wdmaud (pdb symbols) 72a50000 72a59000 msacm32 (deferred) 72a60000 72a64000 ksuser (deferred) 72a70000 72aa2000 winmm (pdb symbols) 72b60000 72b6c000 imgutil (deferred) 72b80000 72b8d000 sfc_os (deferred) 72b90000 72bae000 shimeng (deferred) 72bb0000 72bb6000 dciman32 (deferred) 72c60000 72c8f000 xmllite (deferred) 72c90000 72c9c000 rtutils (deferred) 72ed0000 72f66000 FWPUCLNT (deferred) 73080000 731ca000 msxml6 (deferred) 731d0000 731d5000 msimg32 (deferred) 73200000 73285000 comctl32 (pdb symbols) 73290000 732ef000 winhttp (deferred) 73380000 733b9000 oleacc (deferred) 733c0000 733ff000 uxtheme (deferred)
73400000 73430000 duser (deferred) 73430000 735db000 GdiPlus (deferred) 738d0000 7398b000 propsys (deferred) 74460000 74474000 atl (deferred) 74580000 7471e000 comctl32_74580000 (pdb symbols) 74890000 748b7000 MMDevAPI (deferred) 74960000 74975000 cabinet (deferred) 74980000 749ad000 wintrust (deferred) 74a40000 74a4f000 nlaapi (deferred) 74a50000 74a5a000 wtsapi32 (deferred) 74b10000 74b15000 WSHTCPIP (deferred) 74b20000 74b27000 avrt (deferred) 74b30000 74b4a000 powrprof (deferred) 74b50000 74b71000 ntmarta (deferred) 74bb0000 74beb000 rsaenh (deferred) 74c20000 74c64000 schannel (deferred) 74dd0000 74de5000 gpapi (deferred) 74ed0000 74f0b000 mswsock (pdb symbols) 74f10000 74f55000 bcrypt (deferred) 74f60000 74f95000 ncrypt (deferred) 74fb0000 74fd1000 dhcpcsvc6 (deferred) 74fe0000 74fe7000 winnsi (deferred) 74ff0000 75025000 dhcpcsvc (deferred) 75030000 75049000 IPHLPAPI (deferred) 75050000 75090000 wevtapi (deferred) 75090000 750ca000 SLC (deferred) 750d0000 751c1000 crypt32 (deferred) 75200000 75214000 mpr (deferred) 75260000 75265000 wship6 (deferred) 75270000 75278000 version (deferred) 75280000 75287000 credssp (deferred) 752c0000 752d2000 msasn1 (deferred) 752e0000 752f1000 samlib (deferred) 75300000 7532c000 dnsapi (deferred) 75360000 753d5000 netapi32 (deferred) 755a0000 755ff000 sxs (deferred) 75660000 7568c000 apphelp (deferred) 756c0000 756d4000 secur32 (deferred) 756e0000 756fe000 userenv (deferred) 75820000 75865000 iertutil (deferred) 75870000 76380000 shell32 (deferred) 76380000 763ca000 Wldap32 (deferred) 763d0000 76428000 shlwapi (deferred) 76430000 764b4000 clbcatq (deferred) 764c0000 76588000 msctf (deferred) 76590000 765b9000 imagehlp (deferred) 765c0000 76682000 rpcrt4 (pdb symbols) 76690000 76760000 wininet (pdb symbols) 76760000 767ab000 gdi32 (deferred) 767b0000 767dd000 ws2_32 (pdb symbols) 767e0000 76924000 ole32 (pdb symbols) 76930000 76aba000 setupapi (deferred) 76ac0000 76b33000 comdlg32 (deferred) 76b40000 76bdd000 user32 (pdb symbols)
76be0000 76cbb000 kernel32 (pdb symbols) 76cc0000 76d3d000 usp10 (deferred) 76d40000 76dea000 msvcrt (pdb symbols) 76df0000 76f19000 urlmon (deferred) 76f20000 76fad000 oleaut32 (deferred) 76fb0000 770d7000 ntdll (pdb symbols) 770e0000 770e7000 psapi (deferred) 770f0000 770f9000 lpk (deferred) 77100000 7711e000 imm32 (deferred) 77120000 77126000 nsi (deferred) 77130000 77133000 normaliz (deferred) 77140000 77206000 advapi32 (deferred) 79000000 79046000 mscoree (deferred) 7c340000 7c396000 msvcr71 (deferred) 7c3a0000 7c41b000 msvcp71 (deferred)
Actually, before we quit the debugger, we saved a secured stripped version of the dump file using this command:
0:000> .dump /mrRFt c:UserDumpsie7_pattern_cooperation.dmp
The dump file is available on ftp:
ftp://dumpanalysis.org/pub/ie7_pattern_cooperation.zip
Thread times and stack traces are available in it together with module information. However, heap data and critical section list were not included.
After an upgrade to the new version of productivity software package one unrelated application started to crash frequently. A crash dump file was collected and the following stack trace pointed to a NULL Code Pointer pattern (Volume 2, page 237):
0:000> r eax=09680104 ebx=0013aefc ecx=0968a710 edx=0cdc0c0c esi=16a19058 edi=00000001eip=00000000esp=0013aea8 ebp=0013aeb8 iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200202 00000000 ?? ??? 0:000> k 100 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong.0013aea4 096e28a0 0×00013aeb8 32e688bd dllC!Abort+0×10 0013aec8 32c82395 dllB+0×589e 0013aed8 32865718 dllB+0×18f1 [...] 0013b0c0 314de1ff dllB+0×4c6 0013b154 31293494 dllA!DllGetLCID+0×46d2d 0013b178 312af217 dllA!DllGetClassObject+0×4e896 [...] 0013f3d0 300e8721 dllA!DllGetClassObject+0×69e42 0013f578 300e7f5a application+0xcff5 [...] 0013ffc0 7c816ff7 application+0×51d5 0013fff0 00000000 kernel32!BaseProcessStart+0×23
To see if the changed environment (Volume 1, page 283) somehow affected this application we checked the presence of any DLL hooks. Indeed, the following hooked functions (Volume 1, page 468) were found in user32.dll:
0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:mssUSER32.dll45F02D7C90000USER32.dll
No range specified
Scanning section: .text
Size: 389095
Range to scan: 7e411000-7e46ffe7
7e4188a6-7e4188aa 5 bytes - USER32!GetWindowLongW
[ 6a 08 68 e0 88:e9 55 77 a4 01 ]
7e418f9c-7e418fa0 5 bytes - USER32!GetSystemMetrics (+0×6f6)
[ 6a 10 68 00 90:e9 5f 70 b5 01 ]
7e41945d-7e419461 5 bytes - USER32!GetWindowLongA (+0×4c1)
[ 6a 10 68 78 9f:e9 f5 60 a2 01 ]
7e41b6ae-7e41b6b2 5 bytes - USER32!GetClientRect (+0×17a8)
[ 8b ff 55 8b ec:e9 4d 49 9f 01 ]
7e41b6d4-7e41b6d8 5 bytes - USER32!GetWindowRect (+0×26)
[ b8 74 11 00 00:e9 98 30 af 01 ]
7e41d60d-7e41d611 5 bytes - USER32!SetWindowLongA (+0×6aa)
[ 8b ff 55 8b ec:e9 ee 29 a5 01 ]
7e41d62b-7e41d62f 5 bytes - USER32!SetWindowLongW (+0×1e)
[ 6a 08 68 28 f5:e9 0e 0b b2 01 ]
7e41fc25-7e41fc29 5 bytes - USER32!CreateWindowExW (+0×738)
[ 8b ff 55 8b ec:e9 d6 03 b6 01 ]
7e41ff33-7e41ff37 5 bytes - USER32!CreateWindowExA (+0×30e)
Total bytes compared: 389095(100%)
Number of errors: 52
52 errors : !user32 (7e4188a6-7e42e8d5)The hooking DLL was found from that upgraded package:
0:000> u7e4188a6USER32!GetWindowLongW: 7e4188a6 e95577a401 jmp7fe600007e4188ab 41 inc ecx 7e4188ac 7ee8 jle USER32!_GetWindowLong+0xda (7e418896) 7e4188ae 0e push cs 7e4188af fd std 7e4188b0 ff ??? 7e4188b1 ff8b4d08e816 dec dword ptr [ebx+16E8084Dh] 7e4188b7 fc cld
0:000> u7fe600007fe60000 e9bb62b080 jmphookA+0×62c0(009662c0) 7fe60005 6a08 push 8 7fe60007 68e088417e push offset USER32!'string'+0×34 7fe6000c e99c885bfe jmp USER32!GetWindowLongW+0×7 (7e4188ad) 7fe60011 0000 add byte ptr [eax],al 7fe60013 0000 add byte ptr [eax],al 7fe60015 0000 add byte ptr [eax],al 7fe60017 0000 add byte ptr [eax],al 0:000> lmv mhookAstart end module name 00960000 00976000 hookA (no symbols) Loaded symbol image file: hookA.dll Image path: C:Program FilesCompanyAhookA.dll Image name: hookA.dll [...]ProductName: ProductA[...]
Execution residue (Volume 2, page 239) from hookA module was also found on the problem thread raw stack and it looks like the real code (not coincidental symbolic information, Volume 1, page 390):
0:000> !teb
TEB at 7ffdf000
ExceptionList: 0013f02c
StackBase: 00140000
StackLimit: 0010c000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffdf000
EnvironmentPointer: 00000000
ClientId: 00000c38 . 00000840
RpcHandle: 00000000
Tls Storage: 00163268
PEB Address: 7ffdb000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 00:000> dds 0010c000 00140000 0010c000 00000000 0010c004 00000000 0010c008 00000000 [...] 00121f50 0096d7d1*** ERROR: Module load completed but symbols could not be loaded for hookA.dll hookA+0xd7d1 00121f54 00009924 00121f58 00121fb4 00121f5c 00000000 00121f60 00121f9c 00121f64 0096d895 hookA+0xd895 00121f68 00121f78 00121f6c 00000000 00121f70 00122008 00121f74 00000000 00121f78 00000000 00121f7c 00000000 00121f80 00000000 00121f84 00121f6c 00121f88 000001fe 00121f8c 001220fc 00121f90 0096ec98 hookA+0xec98 00121f94 00970e48 hookA+0×10e48 00121f98 00121fb4 00121f9c 7e41b6a3 USER32!_GetClientRect+0×6e 00121fa0 00122008 00121fa4 fffffa01 [...] 0:000> uhookA+0xd895: 0096d895 8bc6 mov eax,esi 0096d897 8b4df0 mov ecx,dword ptr [ebp-10h] 0096d89a 64890d00000000 mov dword ptr fs:[0],ecx 0096d8a1 5f pop edi 0096d8a2 5e pop esi 0096d8a3 5b pop ebx 0096d8a4 8be5 mov esp,ebp 0096d8a6 5d pop ebp 0:000> ub0096d895hookA+0xd876: 0096d876 8b65e8 mov esp,dword ptr [ebp-18h] 0096d879 be0d000000 mov esi,0Dh 0096d87e c745fcffffffff mov dword ptr [ebp-4],0FFFFFFFFh 0096d885 eb05 jmp hookA+0xd88c (0096d88c) 0096d887 be72000000 mov esi,72h 0096d88c 8d55dc lea edx,[ebp-24h] 0096d88f 52 push edx 0096d890 e8fbfeffff call hookA+0xd790 (0096d790)0096d895
As was found later, the upgraded application had special DLL hooks to improve productivity and ease of use of GUI applications. Fortunately it was possible to disable that hook on a per-application basis and application crashes disappeared.
Here is our further analysis of a memory dump used to illustrate Swarm of Shared Locks pattern (page 107). In that dump there were also exclusively held locks with many blocked threads (Volume 2, page 184):
Resource @ 0x8a04c408 Exclusively owned
Contention Count = 344875
NumberOfExclusiveWaiters = 6
Threads: 87eb3db0 -01<*>
Threads Waiting On Exclusive Access:
88573db0 87b90378 86a49db0 891f4610
8662b020 87127db0
Resource @ 0x89678e80 Exclusively owned
Contention Count = 10261
NumberOfExclusiveWaiters = 2
Threads: 87eb3db0-01<*>
Threads Waiting On Exclusive Access:
89131bf0 88d12db0
Resource @ 0x88d099d8 Exclusively owned
Contention Count = 562811
NumberOfExclusiveWaiters = 4
Threads: 873611d8-01<*>
Threads Waiting On Exclusive Access:
88b8bb88 88a72c50 89359af0 88d865e8
Resource @ 0x86db9248 Exclusively owned
Contention Count = 1382269
NumberOfSharedWaiters = 2
NumberOfExclusiveWaiters = 11
Threads: 86ab2020-01<*> 8769cdb0-01 880c77b8-01
Threads Waiting On Exclusive Access:
87bf4020 890dc020 874c01c0 884ef020
86913af8 875bab10 88e8a0d8 8923cdb0
894eca18 86aa6830 86f293a8
Resource @ 0x873a88d0 Exclusively owned
Contention Count = 719758
NumberOfExclusiveWaiters = 8
Threads: 88d5f990-01<*>
Threads Waiting On Exclusive Access:
8759ea88 871b6db0 88117710 87cb4718
883eb638 87239020 881ad020 891b9188Resource @ 0x88c379a0 Exclusively owned
Contention Count = 126686
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 8
Threads: 882b8020-01<*> 88951520-01
Threads Waiting On Exclusive Access:
877d34a8 8939fdb0 87fc5668 8851fdb0
86fad850 87f1f450 8a1749f0 876a78d0
Resource @ 0x88ca9250 Exclusively owned
Contention Count = 319721
NumberOfExclusiveWaiters = 4
Threads: 88607908-01<*>
Threads Waiting On Exclusive Access:
86829370 892ae8e8 87205208 87b6d7e0
Resource @ 0x86a90ef8 Exclusively owned
Contention Count = 852830
NumberOfExclusiveWaiters = 12
Threads: 87571640 -01<*>
Threads Waiting On Exclusive Access:
88a9c9b0 88a50db0 87117928 890e4c50
874ffb30 88b540f8 8705d020 8687edb0
87143188 8703e430 885b6aa0 8842bc50
Resource @ 0x88954538 Exclusively owned
Contention Count = 40708
NumberOfExclusiveWaiters = 1
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
878ee980
Resource @ 0x88617eb8 Exclusively owned
Contention Count = 43531
NumberOfExclusiveWaiters = 2
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
88851db0 87382c50
Resource @ 0x87288bc8 Exclusively owned
Contention Count = 644675
NumberOfExclusiveWaiters = 2
Threads: 874e4508-01<*>
Threads Waiting On Exclusive Access:
88863b08 89479650Resource @ 0x87c3d8b0 Exclusively owned
Contention Count = 335064
NumberOfExclusiveWaiters = 8
Threads: 87f44520-01<*>
Threads Waiting On Exclusive Access:
88277190 88eceb48 87f0d308 8694d460
88461db0 876734a8 871721b0 88c2adb0All threads owning various locks exclusively are stuck in processing page fault code, for example:
0: kd> !thread 87eb3db0 1f
THREAD 87eb3db0 Cid 47ac.57c8 Teb: 7ffd7000 Win32Thread: bc151230 WAIT:
(Unknown) KernelMode Non-Alertable
8743e4e0 NotificationEvent
IRP List:
8660c900: (0006,0094) Flags: 00000900 Mdl: 00000000
Not impersonating
DeviceMap e1003890
Owning Process 88e49918 Image: csrss.exe
Wait Start TickCount 15420972 Ticks: 2527 (0:00:00:39.484)
Context Switch Count 1430991 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:02.734
Start Address 0×75a8e96c
Stack Init a3cf7000 Current a3cf6430 Base a3cf7000 Limit a3cf4000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr
a3cf6448 8083d5b1 nt!KiSwapContext+0×26
a3cf6474 8083df9e nt!KiSwapThread+0×2e5
a3cf64bc 8082629e nt!KeWaitForSingleObject+0×346
a3cf64e4 80826480 nt!MiWaitForInPageComplete+0×1f
a3cf656c 8084790e nt!MiDispatchFault+0xda3
a3cf65c8 80836c2a nt!MmAccessFault+0×64a
a3cf65c8 bfa38de0 nt!KiTrap0E+0xdc (TrapFrame @ a3cf65e0)
a3cf6a24 bf854a72 win32k!vSetPointer+0×36f
a3cf6a50 bf8b1b74 win32k!GreSetPointer+0×66
a3cf6a7c bf883183 win32k!zzzUpdateCursorImage+0×1cc
a3cf6a8c bf884b06 win32k!zzzSetFMouseMoved+0xd5
a3cf6ad4 bf81530a win32k!ProcessQueuedMouseEvents+0×1c4
a3cf6d30 bf86fd25 win32k!RawInputThread+0×5b4
a3cf6d40 bf898a52 win32k!xxxCreateSystemThreads+0×60
a3cf6d54 80833bef win32k!NtUserCallOneParam+0×23
a3cf6d54 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ a3cf6d64)0: kd> !thread 87571640 1f
THREAD 87571640 Cid 49f4.65b4 Teb: 7ffdf000 Win32Thread: bc011680 WAIT:
(Unknown) KernelMode Non-Alertable
8870db90 NotificationEvent
Not impersonating
DeviceMap e24f6570
Owning Process 87be4a00 Image: ApplicationC.EXE
Wait Start TickCount 15420974 Ticks: 2525 (0:00:00:39.453)
Context Switch Count 25640 LargeStack
UserTime 00:00:00.921
KernelTime 00:00:03.859
Win32 Start Address 0×30002658
Start Address 0×77e617f8
Stack Init 9a318600 Current 9a317b70 Base 9a319000 Limit 9a314000 Call
9a31860c
Priority 14 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
9a317b88 8083d5b1 nt!KiSwapContext+0×26
9a317bb4 8083df9e nt!KiSwapThread+0×2e5
9a317bfc 8082629e nt!KeWaitForSingleObject+0×346
9a317c24 80826480 nt!MiWaitForInPageComplete+0×1f
9a317cac 8084790e nt!MiDispatchFault+0xda3
9a317d08 80836c2a nt!MmAccessFault+0×64a
9a317d08 bf8b5485 nt!KiTrap0E+0xdc (TrapFrame @ 9a317d20)
9a317db4 bf8b526c win32k!vSolidFillRect1+0xb0
9a317f58 bf8ad7d2 win32k!vDIBSolidBlt+0×102
9a317fc4 bfa285d1 win32k!EngBitBlt+0xe1
9a3180c8 bf899b57 win32k!GrePatBltLockedDC+0×1ea
9a318160 bf8b32bb win32k!GrePolyPatBltInternal+0×17c
9a31819c bf8bd71c win32k!GrePolyPatBlt+0×45
9a31822c bf85e3d5 win32k!DrawEdge+0×23a
9a318274 bf8ae338 win32k!xxxDrawWindowFrame+0×170
9a3182d4 bf8847d1 win32k!xxxRealDefWindowProc+0×7a7
9a3182ec bf884801 win32k!xxxWrapRealDefWindowProc+0×16
9a318308 bf8c1769 win32k!NtUserfnDWORD+0×27
9a318340 80833bef win32k!NtUserMessageCall+0xc0
9a318340 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ 9a318364)We also see that their waiting time is almost the same, 39 seconds. This means that the problem with paging probably started at that time before the crash dump was forced.
This is a continuation of the story of a process hang simulation (page 377) where we made all threads in IE7 instance frozen. We left that process frozen after experiments and later tried to reply to one e-mail using a PHP-based browser client running in another IE7 process. And we found that the mouse click on a "Reply" button didn't bring out any GUI response. We tried to close IE7 instance: all tabs closed except the one that was hanging. So we dumped the process and found a blocking thread inside it waiting for an RPC call. We made all threads in the first IE7 process unfrozen and the second hanging IE7 process immediately exited. Instead of digging into the dump further we decided to repeat the problem. First, we launched the fresh instance of IE7 and opened the e-mail client. After clicking on the "Reply" button with success we dumped the process using Vista Task Manager and renamed the resulted memory dump as iexplore2.dmp. Then we launched another IE instance and made all threads frozen. Then we came back to the first instance of IE7 and tried to push the "Reply" button again. After waiting for about 10 minutes for any response we dumped the process again and renamed the dump file as iexplore3.dmp. Comparing thread stack traces (Volume 1, page 409) from both memory dump files showed one difference: the blocked (Volume 2, page 184) OLE/RPC thread obviously processing some JavaScript code:
[Before hang]
0:000> ~[0n36]k 100 ChildEBP RetAddr 161bfc00 76b60dde ntdll!KiFastSystemCallRet 161bfc04 705e41a1 user32!NtUserWaitMessage+0xc 161bfc68 76c24911 ieframe!CTabWindow::_TabWindowThreadProc+0x2d0 161bfc74 76fee4b6 kernel32!BaseThreadInitThunk+0xe 161bfcb4 76fee489 ntdll!__RtlUserThreadStart+0x23 161bfccc 00000000 ntdll!_RtlUserThreadStart+0x1b
[After hang]
0:000> ~[0n36]k 100
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be
wrong.
161bc27c 76b60208 ntdll!KiFastSystemCallRet
161bc2d0 767fab28 user32!RealMsgWaitForMultipleObjectsEx+0x13c
161bc2f8 767fac88 ole32!CCliModalLoop::BlockFn+0×97
161bc320 76907b73 ole32!ModalLoop+0×5b
161bc33c 76908b68 ole32!ThreadSendReceive+0×12c
161bc364 769089d4 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0×194
161bc444 767fad2e ole32!CRpcChannelBuffer::SendReceive2+0xef161bc460 767face0 ole32!CCliModalLoop::SendReceive+0×1e 161bc4d8 7681e688 ole32!CAptRpcChnl::SendReceive+0×73 161bc52c 7667264e ole32!CCtxComChnl::SendReceive+0×1c5 161bc544 766726af rpcrt4!NdrProxySendReceive+0×43 161bc9c8 76f3ad86 rpcrt4!NdrProxySendReceive+0xa4161bc9e0 76f3ad58 oleaut32!IEnumVARIANT_RemoteNext_Proxy+0×19 161bc9fc 6c1f2a7b oleaut32!IEnumVARIANT_Next_Proxy+0×1c 161bca5c 6c1f2b9c mshtml!SearchBrowsersForWindow+0×1bd 161bca84 6c1a2932 mshtml!GetTargetWindow+0×53 161bcabc 6c1b1300 mshtml!CWindow::FindWindowByName+0xe1 161bcad4 706498d4 mshtml!CWindow::FindWindowByName+0×17 161bcaf4 70649e5a ieframe+0×1198d4 161bcb48 70649ff6 ieframe+0×119e5a 161bcbac 70649b82 ieframe+0×119ff6 161bcbe0 6c189f9b ieframe+0×119b82 161bcc10 6c119cba mshtml!COmWindowProxy::FindFrame+0×5c 161bcc44 6c18be8e mshtml!COmWindowProxy::AccessAllowedToFrame+0×7f 161bccb4 6c1c4a2e mshtml!COmWindowProxy::open+0×15b 161bcd1c 6c0371b6 mshtml!Method_IDispatchpp_oDoBSTR_oDoBSTR_oDoBSTR_oDoVARIANTBOOL+0xeb 161bcdb4 6c037493 mshtml!CBase::ContextInvokeEx+0×4ef 161bcde0 6c037607 mshtml!CBase::InvokeEx+0×25 161bce48 6c0374c2 mshtml!COmWindowProxy::InvokeEx+0×297 161bce70 6c5b348e mshtml!COmWindowProxy::subInvokeEx+0×26161bd404 6c1b1931 mshtml!CScriptCollection::ParseScriptText+0×240 161bf48c 6c12adae mshtml!CWindow::ExecuteScriptUri+0×197 161bf4cc 6c1b2f77 mshtml!CWindow::NavigateEx+0×50 161bf530 6c1b3372 mshtml!CDoc::ExecuteScriptUri+0×1f7 161bf560 6c27b8ac mshtml!CDoc::ExecuteScriptURL+0×4b 161bf5a8 6c27a54c mshtml!CHyperlink::ClickAction+0×1a9 161bf5b8 6c121847 mshtml!CAnchorElement::ClickAction+0×10 161bf5e4 6c07a7ef mshtml!CElement::DoClick+0×121 161bf610 6c07a5bd mshtml!CAnchorElement::DoClick+0×4d 161bf69c 6c07f680 mshtml!CDoc::PumpMessage+0xcbd 161bf7e8 6c12a7e0 mshtml!CDoc::OnMouseMessage+0×3d7 161bf90c 6c039a11 mshtml!CDoc::OnWindowMessage+0×8f7 161bf938 76b5f8d2 mshtml!CServer::WndProc+0×78 161bf964 76b5f794 user32!InternalCallWinProc+0×23161bcea8 6c5b33fe jscript!IDispatchExInvokeEx2+0xac 161bcee0 6c5b3e09 jscript!IDispatchExInvokeEx+0×56 161bcf50 6c5b30eb jscript!InvokeDispatchEx+0×78 161bcf98 6c5b18ab jscript!VAR::InvokeByName+0xba 161bcfd8 6c5b2109 jscript!VAR::InvokeDispName+0×43 161bcffc 6c5b28d8 jscript!VAR::InvokeByDispID+0xb9 161bd0b4 6c5b1019 jscript!CScriptRuntime::Run+0×167f 161bd0cc 6c5b2aa8 jscript!ScrFncObj::Call+0×8d 161bd158 6c5b00f2 jscript!NameTbl::InvokeInternal+0xe0 161bd184 6c5b28d8 jscript!VAR::InvokeByDispID+0xfd 161bd23c 6c5b1019 jscript!CScriptRuntime::Run+0×167f 161bd254 6c5b1b7f jscript!ScrFncObj::Call+0×8d 161bd2c4 6c59f9d2 jscript!CSession::Execute+0xa7 161bd314 6c59fdf7 jscript!COleScript::ExecutePendingScripts+0×147 161bd378 6c59fc46 jscript!COleScript::ParseScriptTextCore+0×243 161bd3a4 6bfcca36 jscript!COleScript::ParseScriptText+0×2b
161bf9dc 76b606f6 user32!UserCallWinProcCheckWow+0×14b 161bfa0c 76b6069c user32!CallWindowProcAorW+0×97 161bfa2c 6baad980 user32!CallWindowProcW+0×1b 161bfa98 6baa104a GoogleToolbarDynamic_F423308312A7B033+0×3d980 161bfabc 6bb67e57 GoogleToolbarDynamic_F423308312A7B033+0×3104a 161bfae8 76b5f8d2 GoogleToolbarDynamic_F423308312A7B033+0xf7e57 161bfb14 76b5f794 user32!InternalCallWinProc+0×23 161bfb8c 76b60008 user32!UserCallWinProcCheckWow+0×14b 161bfbf0 76b60060 user32!DispatchMessageWorker+0×322 161bfc00 705e42c1 user32!DispatchMessageW+0xf 161bfc68 76c24911 ieframe+0xb42c1 161bfc74 76fee4b6 kernel32!BaseThreadInitThunk+0xe
We see SendReceive2 on the latter stack trace. It is possible to know the target process PID as shown in the Volume 2, page 136 (In Search of Lost CID). The same procedure applied here reveals PID = 0xdec (stack trace is shown in smaller font for visual clarity):
0:000> ~[0n36]kv 9
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be
wrong.
161bc27c 76b60208 161bc230 161bc2a4 00000000 ntdll!KiFastSystemCallRet
161bc2d0 767fab28 00000000 161bc318 00000000
user32!RealMsgWaitForMultipleObjectsEx+0x13c
161bc2f8 767fac88 161bc318 00000000 161bc328
ole32!CCliModalLoop::BlockFn+0x97
161bc320 76907b73 00000000 00000000 161bc42c ole32!ModalLoop+0x5b
161bc33c 76908b68 00000000 161bc440 00000000 ole32!ThreadSendReceive+0x12c
161bc364 769089d4 161bc42c 00000000 161bc488
ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x194
161bc444 767fad2e 14f75040 161bc56c 161bc550
ole32!CRpcChannelBuffer::SendReceive2+0xef
161bc460 767face0 161bc56c 161bc550 00000000
ole32!CCliModalLoop::SendReceive+0×1e
161bc4d8 7681e688 14f75040 161bc56c 161bc550
ole32!CAptRpcChnl::SendReceive+0×73Note: 14f75040 is 00000000 in iexplore3.dmp from ftp because the memory dumps were stripped from almost all process data and contain only values necessary to reconstruct stack traces. Therefore, it is not possible to extract correct raw stack data from them.
0:000> ddp 14f75040 14f75040 76828438 76907c77 ole32!CRpcChannelBuffer::QueryInterface 14f75044 7681c7e4 7689b57c ole32!CRpcChannelBuffer::QueryInterface 14f75048 00000003 14f7504c 00000002 14f75050 00000000 14f75054 00000000 14f750580046ccd00046ce50 14f7505c 0e8de858 00000000 14f75060 1acb7310 00000044 14f75064 1acb3130 76828510 14f75068 7682b098 767f8066 ole32!CDestObject::QueryInterface 14f7506c 00070005 ee0100ed 14f75070 00000000 14f75074 00000000 14f75078 00000d78 14f7507c 00000000 14f75080 76828438 76907c77 ole32!CRpcChannelBuffer::QueryInterface 14f75084 7681c7e4 7689b57c ole32!CRpcChannelBuffer::QueryInterface 14f75088 00000001 14f7508c 00000024 14f75090 00000000 14f75094 00000000 14f75098 07abd4a8 07aacab0 14f7509c 00000000 14f750a0 00000000 14f750a4 1ae12b10 76828510 14f750a8 7682b098 767f8066 ole32!CDestObject::QueryInterface 14f750ac 00070005 ee0100ed 14f750b0 ffffffff 14f750b4 00001134 14f750b8 00001134 14f750bc 00000000 0:000> dd 0046ccd0 l4 0046ccd0 0046ce50 0046cc5000000dec00000000
In Task Manager we found this to be ieuser.exe process and suspected a high degree of process coupling (Volume 1, page 419) between all launched IE7 processes and ieuser.exe including COM/OLE runtime.
The stripped versions of dumps are available for practice on ftp:
ftp://dumpanalysis.org/pub/ie7_pattern_cooperation2.zip
It was reported that one system became slower and slower and eventually a complete memory dump was generated for analysis. Usual !vm 4 command shows pool allocation errors and particularly insufficient session pool memory (Volume 1, page 440) for user session 1:
1: kd> !vm 4 *** Virtual Memory Usage *** Physical Memory: 1048503 ( 4194012 Kb) Page File: ??C:pagefile.sys Current: 5109760 Kb Free Space: 3668764 Kb Minimum: 5109760 Kb Maximum: 5109760 Kb Available Pages: 409409 ( 1637636 Kb) ResAvail Pages: 708135 ( 2832540 Kb) Locked IO Pages: 108 ( 432 Kb) Free System PTEs: 137566 ( 550264 Kb) Free NP PTEs: 27288 ( 109152 Kb) Free Special NP: 0 ( 0 Kb) Modified Pages: 780 ( 3120 Kb) Modified PF Pages: 771 ( 3084 Kb)NonPagedPool Usage: 37587 ( 150348 Kb) NonPagedPool Max: 65214 ( 260856 Kb)PagedPool 0 Usage: 21583 ( 86332 Kb) PagedPool 1 Usage: 3054 ( 12216 Kb) PagedPool 2 Usage: 3076 ( 12304 Kb) PagedPool 3 Usage: 3050 ( 12200 Kb) PagedPool 4 Usage: 3040 ( 12160 Kb)********** 185454 pool allocations have failed ********** Shared Commit: 123289 ( 493156 Kb) Special Pool: 0 ( 0 Kb) Shared Process: 101937 ( 407748 Kb) PagedPool Commit: 33850 ( 135400 Kb) Driver Commit: 1697 ( 6788 Kb) Committed pages: 1159581 ( 4638324 Kb) Commit limit: 2283111 ( 9132444 Kb) [...]PagedPool Usage: 33803 ( 135212 Kb) PagedPool Maximum: 66560 ( 266240 Kb)
Session ID 1 @ af925000: Paged Pool Usage: 16868K *** 371034 Pool Allocation Failures *** Commit Usage: 17960K Session ID 2 @ af927000: Paged Pool Usage: 2856K Commit Usage: 3860K [...]
The exploration of session 1 processes shows Process Factory (page 112) pattern (5,000 launched processes) with explorer.exe ran amok:
1: kd> !sprocess 1
Dumping Session 1
_MM_SESSION_SPACE af925000
_MMSESSION af925d80
PROCESS 87db34a0 SessionId: 1 Cid: 4a68 Peb: 7ffd4000 ParentCid:
01b8
DirBase: cffb6d20 ObjectTable: e6084268 HandleCount: 11677.
Image: csrss.exe
PROCESS 86083020 SessionId: 1 Cid: 4560 Peb: 7ffde000 ParentCid:
2978
DirBase: cffb6f40 ObjectTable: e6c41410 HandleCount: 79.
Image: AnotherApp.exe
PROCESS 8607c020 SessionId: 1 Cid: 44c8 Peb: 7ffdc000 ParentCid:
4cf8
DirBase: cffb7080 ObjectTable: e3c9fd38 HandleCount: 25407.
Image: explorer.exe
[...]
PROCESS 85e94738 SessionId: 1 Cid: 4868 Peb: 7ffde000 ParentCid:
44c8
DirBase: cffb7c00 ObjectTable: e7038840 HandleCount: 39.
Image: application.exe
PROCESS 85e5c020 SessionId: 1 Cid: 4668 Peb: 7ffd4000 ParentCid:
4a50
DirBase: cffb7de0 ObjectTable: e11891d0 HandleCount: 172.
Image: application.exePROCESS 85e51578 SessionId: 1 Cid: 4c9c Peb: 7ffde000 ParentCid: 44c8 DirBase: cffb7e40 ObjectTable: e2ee0070 HandleCount: 39. Image: application.exe [...] PROCESS 85c81020 SessionId: 1 Cid: 53a4 Peb: 7ffd7000 ParentCid: 44c8 DirBase: cffb89c0 ObjectTable: e6d2f600 HandleCount: 39. Image: application.exe PROCESS 85c6fb18 SessionId: 1 Cid: 53a8 Peb: 7ffd7000 ParentCid: 44c8 DirBase: cffb89e0 ObjectTable: e54df078 HandleCount: 39. Image: application.exe PROCESS 85c60020 SessionId: 1 Cid: 53bc Peb: 7ffdf000 ParentCid: 44c8 DirBase: cffb8a40 ObjectTable: e1214e90 HandleCount: 39. Image: application.exe [...]
We can also see the very big number of handles in both csrss.exe and explorer.exe. If we look at one of 5,000 application.exe processes we would see that it is a GUI application and this explains session pool exhaustion:
1: kd> !process 85c60020 1f
PROCESS 85c60020 SessionId: 1 Cid: 53bc Peb: 7ffdf000 ParentCid:
44c8
DirBase: cffb8a40 ObjectTable: e1214e90 HandleCount: 39.
Image: application.exe
VadRoot 85c97450 Vads 80 Clone 0 Private 173. Modified 159. Locked 0.
DeviceMap e3743340
Token e2ead5e0
ElapsedTime 00:54:17.218
UserTime 00:00:00.015
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 32460
QuotaPoolUsage[NonPagedPool] 3200
Working Set Sizes (now,min,max) (30, 50, 345) (120KB, 200KB, 1380KB)
PeakWorkingSetSize 778
VirtualSize 16 Mb
PeakVirtualSize 17 Mb
PageFaultCount 829
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 212THREAD 85c25db0 Cid 53bc.5470 Teb: 7ffde000 Win32Thread: bd1a18a8 WAIT:
(Unknown) UserMode Non-Alertable
85bbf2a8 SynchronizationEvent
Not impersonating
DeviceMap e3743340
Owning Process 85c60020 Image: application.exe
Wait Start TickCount 1121133 Ticks: 206877 (0:00:53:52.453)
Context Switch Count 19355 NoStackSwap LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address application (0x00402a38)
Start Address kernel32!BaseProcessStartThunk (0x7c8217f8)
Stack Init ad2b8000 Current ad2b7b1c Base ad2b8000 Limit ad2b4000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
ad2b7b34 80833ec5 nt!KiSwapContext+0x26
ad2b7b60 80829bc0 nt!KiSwapThread+0x2e5
ad2b7ba8 bf89aacc nt!KeWaitForSingleObject+0x346
ad2b7c04 bf8c4816 win32k!xxxSleepThread+0×1be
ad2b7ca0 bf8a134f win32k!xxxInterSendMsgEx+0×798
ad2b7cec bf85ae3c win32k!xxxSendMessageTimeout+0×1f3
ad2b7d10 bf8c13dc win32k!xxxWrapSendMessage+0×1b
ad2b7d40 8088ad3c win32k!NtUserMessageCall+0×9d
ad2b7d40 7c9485ec nt!KiFastCallEntry+0xfc (TrapFrame @ ad2b7d64)
0012f814 00000000 ntdll!KiFastSystemCallRetWe also see extremely high executive resource contention (Volume 1, page 421) and various signs of a busy system (Volume 1, page 448) like the fact that all processors are busy and the very high number of ready threads at normal priority (about 1,000):
1: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
Resource @ DriverA (0xbae016a0) Exclusively owned
Contention Count = 277088870
NumberOfExclusiveWaiters = 3591
Threads: 83b46db0-01<*> << Too many exclusive waiters to list>>1: kd> !running
System Processors ff (affinity mask)
Idle Processors 0
Prcb Current Next
0 ffdff120 86300b40 ................
1 f772f120 f77320a0 852fadb0 ................
2 f7737120 f773a0a0 861528d0 ................
3 f773f120 f77420a0 877a4020 ................
4 f7747120 f774a0a0 865909a8 ................
5 f774f120 f77520a0 83d62cd0 ................
6 f7757120 f775a0a0 841aa020 ................
7 f775f120 f77620a0 ffb7b5e0 ................
1: kd> !ready
Processor 0: No threads in READY state
Processor 1: Ready Threads at priority 10
THREAD 861b1898 Cid 4860.4894 Teb: 7ffd8000 Win32Thread: bc1669a8
READY
Processor 1: Ready Threads at priority 9
THREAD 83b46db0 Cid 44c8.a11c Teb: 7f8f8000 Win32Thread: bcbabb38
READY
Processor 1: Ready Threads at priority 8
THREAD 851408d0 Cid 6338.6398 Teb: 7ffdf000 Win32Thread: bc9bfbb8
READY
THREAD 83eb1730 Cid 8f30.9568 Teb: 7ffdf000 Win32Thread: 00000000
READY
THREAD fd310af8 Cid d53c.b4d8 Teb: 7ffdf000 Win32Thread: 00000000
READY
THREAD 83bce8d0 Cid abec.6450 Teb: 7ffdd000 Win32Thread: 00000000
READY
THREAD 83de2388 Cid a618.d79c Teb: 7ffde000 Win32Thread: 00000000
READY
THREAD 84197b40 Cid 44c8.6974 Teb: 7fe6a000 Win32Thread: bc662d48
READY
[...]
Processor 1: Ready Threads at priority 7
THREAD 86d0c890 Cid 3524.365c Teb: 7ff8e000 Win32Thread: bc403bb0
READY
Processor 2: No threads in READY state
Processor 3: No threads in READY state
Processor 4: No threads in READY state
Processor 5: No threads in READY state
Processor 6: No threads in READY state
Processor 7: No threads in READY stateThere is AnotherApp.exe in the session process list that executes a VBS script:
1: kd> .process /r /p 86083020
Implicit process is now 86083020
Loading User Symbols
1: kd> !process 86083020
[...]
THREAD 860b0db0 Cid 4560.4e4c Teb: 7ffdd000 Win32Thread:
bc1bf4d0 WAIT: (Unknown) KernelMode Non-Alertable
860b0e28 NotificationTimer
1: kd> .thread 860b0db0
Implicit thread is now 860b0db0
1: kd> kv 100
ChildEBP RetAddr Args to Child
ae7ec76c 80833ec5 860b0db0 860b0ea0 00000000 nt!KiSwapContext+0x26
ae7ec798 80829069 0000001e 00000000 bae016a0 nt!KiSwapThread+0x2e5
ae7ec7e0 badff394 00000000 00000000 ffdff5f0
nt!KeDelayExecutionThread+0x2ab
[...]
0012f6ac 7c8024d5 00000000 00000000 00a43280
kernel32!CreateProcessInternalA+0x28b
0012f6e4 00401b31 00000000 00a43280 00000000 kernel32!CreateProcessA+0×2c
[...]
1: kd> da /c 100 00a43280
00a43280 "wscript.exe Scripts C: script.vbs parameters"The examination of that script showed that it contained a coding error resulted in an infinite loop.
In one kernel memory dump we can see signs of a busy system (Volume 1, page 448) where all processors are executing non-idle threads:
0: kd> !running
System Processors ff (affinity mask)
Idle Processors 0
Prcb Current Next
0 ffdff120 8b223928 ................
1 f772f120 8a765380 ................
2 f7737120 89365db0 ................
3 f773f120 8833adb0 ................
4 f7747120 889bbdb0 ................
5 f774f120 8c085db0 ................
6 f7757120 8aa79698 ................
7 f775f120 896c0668 ................When inspecting them we see that some are kernel worker threads without a process context, for example:
0: kd> !thread 8aa79698 1f THREAD 8aa79698 Cid 0004.6edc Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 6 Not impersonating DeviceMap d66008d0 Owning Process 8d15d648Image: SystemWait Start TickCount 2548878 Ticks: 3 (0:00:00:00.046) Context Switch Count 248713UserTime 00:00:00.000 KernelTime 00:00:00.906Start Address nt!ExpWorkerThread (0×80881860) Stack Init acfbc000 Current acfbbcec Base acfbc000 Limit acfb9000 Call 0 Priority 13 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr WARNING: Stack unwind information not available. Following frames may be wrong. acfbbaec b19483f3 driver+0×3f65 acfbbb6c 8081e095 driver+0×23f3 acfbbb80 af36044a nt!IofCallDriver+0×45 [...]acfbbdac 8094bea4 nt!ExpWorkerThread+0xebacfbbddc 8088f57e nt!PspSystemThreadStartup+0×2e 00000000 00000000 nt!KiThreadStartup+0×16
Some threads with an associated process context are running in kernel space:
0: kd> !thread 889bbdb0 1f THREAD 889bbdb0 Cid 6c58.6f98 Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 4 Not impersonating DeviceMap d66008d0 Owning Process 89ad8b18Image: csrss.exeWait Start TickCount 2548880 Ticks: 1 (0:00:00:00.015) Context Switch Count 129536UserTime 00:00:00.000 KernelTime 00:00:00.312Start Address displaydriver!Thread (0xbfad4a60) Stack Init a439d000 Current a439cc70 Base a439d000 Limit a439a000 Call 0 Priority 13 BasePriority 10 PriorityDecrement 3 ChildEBP RetAddr a439c004 bfad707f displaydriver!CalcRegion+0×30 [...]a439cddc 8088f57e nt!PspSystemThreadStartup+0×2e00000000 00000000 nt!KiThreadStartup+0×16
And some threads with an associated process context are running in user space:
0: kd> !thread 8c085db0 1f THREAD 8c085db0 Cid 2318.231c Teb: 7ffdd000 Win32Thread: b4b5ebe8 RUNNING on processor 5 Not impersonating DeviceMap dc1a71f0 Owning Process 8b02e458Image: Application.EXEWait Start TickCount 2548881 Ticks: 0 Context Switch Count 725122 LargeStackUserTime 00:00:01.625 KernelTime 00:00:03.062Win32 Start Address 0×30001084 Start Address 0×7c8217f8 Stack Init ad648000 Current ad647c50 Base ad648000 Limit ad642000 Call 0 Priority 12 BasePriority 10 PriorityDecrement 0 ChildEBP RetAddr0013fb7c 00000000 0×7c81b910
Because none of them consumed much CPU the pattern of Spiking Thread (Volume 1, page 305) is ruled out and the CPU load can be explained by the number of active user sessions that appears to be normal:
0: kd> !session Sessions on machine: 50
However looking at ERESOURCE locks we see many blocked threads (Volume 2, page 184) and signs of possible wait chains (Volume 2, page 147):
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
Resource @ 0x8cbfaa68 Exclusively owned
Contention Count = 22969
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 109
Threads: 8a961db0-01<*> 8bf532b0-01
Threads Waiting On Exclusive Access:
8b4532f0 884fc648 88c58a00 8a751360
88ed64f8 89aa6738 89870db0 881dedb0
8a6d7b40 8b4a4db0 89818ad0 8afcedb0
8a2ca020 88684db0 8b411020 89d595c0
8d1573f0 88d06020 8aed8b38 8a8c9020
8a5a0a50 8a1f63b0 89b66688 89bf1db0
880dab18 882e6730 895d8020 88e6d3f0
896e6748 89802100 8a604508 8907c5e8
8890a020 885e2300 8a061bd8 88445340
88113db0 8a680db0 89b53370 88c3f2a0
88a774f8 8834ddb0 89d78888 88386020
897ca8d8 8b3532d0 882341d0 8a4a9b80
87e7c4f8 895e5db0 8846f4e8 89df3db0
889b8b40 89d82db0 89e4b720 8aadadb0
8aa63020 88852020 8a249ba8 891b8c20
8b3f95f0 8aace760 8b470020 897ad388
8c07dba8 8a331628 896c74d0 8997cb40
88e133c8 886eddb0 8864e518 89ab5698
88d8bdb0 89996db0 8ac54d28 87f42020
882b1020 8857fdb0 895f3db0 88b0ab40
8a1aadb0 8b819020 8b3bf388 88315660
8a45db18 883fbdb0 88f53db0 87f209a0
8978ddb0 8840c868 8823c1c8 88277db0
89c0a8c8 88322940 8a475db0 8a6ad460
8a35a4c8 88e3da40 886b1b40 8886a2a0
8897d750 8b30bdb0 8a123020 8b0ad7f8
8a256930 885cedb0 88ec8db0 887d7ba8
88175b90
Resource @ 0x8b8f09a8 Shared 1 owning threads
Contention Count = 123597
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 6
Threads: 88200840-01 8a92ddb0-01<*>
Threads Waiting On Exclusive Access:
8a317db0 8d151840 899acdb0 8a961db0
891ac940 89ee5db0Resource @ 0x8ac79f08 Exclusively owned
Contention Count = 717691
NumberOfExclusiveWaiters = 12
Threads: 8a5193f0-01<*>
Threads Waiting On Exclusive Access:
880e7b40 8a60adb0 8a543108 8a4be020
8a77c360 8a470730 87f12db0 8a4618d0
895c5600 8a942b98 8a453b40 8a3bf020
Resource @ 0x8a73ed28 Exclusively owned
Contention Count = 4
NumberOfExclusiveWaiters = 2
Threads: 8a45db18-01<*>
Threads Waiting On Exclusive Access:
8a412db0 8a542268
Resource @ 0x8a621bf8 Exclusively owned
Contention Count = 8532
NumberOfExclusiveWaiters = 3
Threads: 8a412db0-01<*>
Threads Waiting On Exclusive Access:
8a5193f0 8a60cdb0 8a595c78
Resource @ 0x8a4c8b20 Exclusively owned
Contention Count = 1
NumberOfExclusiveWaiters = 1
Threads: 8a92ddb0-01<*>
Threads Waiting On Exclusive Access:
89524a70Resource @ 0x89f82f28 Exclusively owned
Contention Count = 26674
NumberOfExclusiveWaiters = 2
Threads: 89fc3020-01<*>
Threads Waiting On Exclusive Access:
8aaa3020 8a02db40
Resource @ 0x89315320 Exclusively owned
Contention Count = 509247
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 19
Threads: 89261428-01<*> 89313a08-01
Threads Waiting On Exclusive Access:
89cc7db0 8ad26528 8970db68 88ef64d0
8a629020 89450798 8825c9a8 89206378
8a7c7b90 89162890 8ae7c020 883318e0
88bd6358 89367db0 8952aaa0 8a817b40
881d65b8 8ab74db0 889202c0
Resource @ 0x893872d8 Exclusively owned
Contention Count = 5079
NumberOfExclusiveWaiters = 3
Threads: 896e6748-01<*>
Threads Waiting On Exclusive Access:
89261428 893bc3c8 892a88a8
Resource @ 0x8924adf8 Exclusively owned
Contention Count = 1
NumberOfExclusiveWaiters = 1
Threads: 88ed64f8-01<*>
Threads Waiting On Exclusive Access:
89146660
Resource @ 0x890281b0 Exclusively owned
Contention Count = 4
NumberOfExclusiveWaiters = 4
Threads: 88d06020-01<*>
Threads Waiting On Exclusive Access:
88b5c528 88c5aa98 87ef77b8 88c48b40
Resource @ 0x88d40440 Exclusively owned
Contention Count = 13
NumberOfExclusiveWaiters = 1
Threads: 899acdb0-01<*>
Threads Waiting On Exclusive Access:
895e6db0Resource @ 0x88ed0c20 Exclusively owned
Contention Count = 2
NumberOfExclusiveWaiters = 2
Threads: 895e6db0-01<*>
Threads Waiting On Exclusive Access:
88ad7540 88b5f620
Resource @ 0x894e7990 Exclusively owned
Contention Count = 3852647
NumberOfExclusiveWaiters = 12
Threads: 881b14b8-01<*>
Threads Waiting On Exclusive Access:
88a13db0 87f12020 8aad7a20 8820a020
8824bdb0 88213db0 88eefdb0 88ab7550
889fe808 89df17a0 8aa83430 8a8f73c8
Resource @ 0x88559288 Exclusively owned
Contention Count = 7422
NumberOfExclusiveWaiters = 3
Threads: 880dab18-01<*>
Threads Waiting On Exclusive Access:
881b14b8 88311020 882ab660
Resource @ 0x8aff12b0 Exclusively owned
Contention Count = 6
NumberOfExclusiveWaiters = 1
Threads: 89524a70Starting with the thread 8a961db0 that blocks 109 other threads we can unravel the following deadlock:
109 threads -> 8a961db0 -> 8a92ddb0 -> 89524a70 -> 8a92ddb0 -> 89524a70 -> ...
Looking at threads involved in the deadlock (Volume 1, page 323) we see that they belong to the same process and deadlocked in kernel space when running through driverA.sys code:
0: kd> !thread 89524a70 1f
THREAD 89524a70 Cid 1fdc.26cc Teb: 7ffdd000 Win32Thread: b4d0fea8 WAIT:
(Unknown) KernelMode Non-Alertable
89170648 SynchronizationEvent
89524ae8 NotificationTimer
IRP List:
88e7a008: (0006,0268) Flags: 00000000 Mdl: 00000000
8a7cc228: (0006,0268) Flags: 00000000 Mdl: 00000000
89e67b90: (0006,0268) Flags: 00000000 Mdl: 00000000 Not impersonating
DeviceMap e2e671d0
Owning Process 88c37020 Image: ApplicationA.exe
Wait Start TickCount 2548760 Ticks: 121 (0:00:00:01.890)
Context Switch Count 4850 LargeStack
UserTime 00:00:00.734
KernelTime 00:00:01.718
Win32 Start Address 0×00404054
Start Address 0×7c8217f8
Stack Init 91971000 Current 91970278 Base 91971000 Limit 9196a000 Call 0
Priority 14 BasePriority 10 PriorityDecrement 4
ChildEBP RetAddr
91970290 80833ec5 nt!KiSwapContext+0×26
919702bc 80829bc0 nt!KiSwapThread+0×2e5
91970304 8087e0db nt!KeWaitForSingleObject+0×346
91970340 8087e2f5 nt!ExpWaitForResource+0xd5
91970360 b0a1cf6d nt!ExAcquireResourceExclusiveLite+0×8d
91970374 b0a08cef driverA+0×2ef6d
919703cc b0a089cc driverA+0×1acef
919703f4 b0a209d9 driverA+0×1a9cc
9197056c b0a20386 driverA+0×329d9
91970630 b0a1dc32 driverA+0×32386
919706e8 b0a20508 driverA+0×2fc32
919707ac b0a1eec1 driverA+0×32508
919708a0 b0a21e90 driverA+0×30ec1
91970930 b0a171c9 driverA+0×33e90
919709c4 b0a16c9d driverA+0×291c9
91970a38 b0a600b3 driverA+0×28c9d
91970a84 b0a45dda driverA+0×720b3
91970afc b0a4657a driverA+0×57dda
91970b48 8081e095 driverA+0×5857a
91970b5c f7876d28 nt!IofCallDriver+0×45
91970b88 8081e095 fltmgr!FltpDispatch+0×152
91970b9c f74fc6ca nt!IofCallDriver+0×45
91970bb4 f7876d28 driverB+0×56ca
91970be0 8081e095 fltmgr!FltpDispatch+0×152
91970bf4 b195a4e1 nt!IofCallDriver+0×45
91970c18 b195a5d0 driverC!PassThrough+0xd1
91970c28 8081e095 driverC!Dispatch+0×80
91970c3c f7876d28 nt!IofCallDriver+0×45
91970c68 8081e095 fltmgr!FltpDispatch+0×152
91970c7c 808f7601 nt!IofCallDriver+0×45
91970c90 808f5339 nt!IopSynchronousServiceTail+0×10b
91970d38 8088ac9c nt!NtWriteFile+0×663
91970d38 7c9485ec nt!KiFastCallEntry+0xfc0: kd> !thread 8a92ddb0 1f
THREAD 8a92ddb0 Cid 1fdc.7b98 Teb: 7ffa9000 Win32Thread: b4deeae8 WAIT:
(Unknown) KernelMode Non-Alertable
8b422388 SynchronizationEvent
8a92de28 NotificationTimer
Not impersonating
DeviceMap e2e671d0
Owning Process 88c37020 Image: ApplicationA.exe
Wait Start TickCount 2548760 Ticks: 121 (0:00:00:01.890)
Context Switch Count 956 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address 0×01381fa0
Start Address 0×7c8217ec
Stack Init 917c1000 Current 917c034c Base 917c1000 Limit 917bd000 Call 0
Priority 14 BasePriority 10 PriorityDecrement 4
ChildEBP RetAddr
917c0364 80833ec5 nt!KiSwapContext+0×26
917c0390 80829bc0 nt!KiSwapThread+0×2e5
917c03d8 8087e0db nt!KeWaitForSingleObject+0×346
917c0414 8087e2f5 nt!ExpWaitForResource+0xd5
917c0434 b0a1cf6d nt!ExAcquireResourceExclusiveLite+0×8d
917c0448 b0a08cef driverA+0×2ef6d
917c04a0 b0a089cc driverA+0×1acef
917c04c8 b0a13787 driverA+0×1a9cc
917c053c b0a0bfaa driverA+0×25787
917c057c b0a0c3b3 driverA+0×1dfaa
917c0858 b0a0ccaf driverA+0×1e3b3
917c0934 b0a6074c driverA+0×1ecaf
917c097c b0a4f9d2 driverA+0×7274c
917c0a18 b0a501f6 driverA+0×619d2
917c0a40 b0a5020c driverA+0×621f6
917c0a4c b0a50442 driverA+0×6220c
917c0a6c b0a50687 driverA+0×62442
917c0ac4 b0a50cb0 driverA+0×62687
917c0b08 b0a50ddd driverA+0×62cb0
917c0b18 8081e095 driverA+0×62ddd
917c0b2c f7876d28 nt!IofCallDriver+0×45
917c0b58 8081e095 fltmgr!FltpDispatch+0×152
917c0b6c f74fc6ca nt!IofCallDriver+0×45
917c0b84 f7876d28 driverB+0×56ca
917c0bb0 8081e095 fltmgr!FltpDispatch+0×152
917c0bc4 b195a4e1 nt!IofCallDriver+0×45
917c0be8 b195a5d0 driverC!PassThrough+0xd1
917c0bf8 8081e095 driverC!Dispatch+0×80
917c0c0c f7876d28 nt!IofCallDriver+0×45
917c0c38 8081e095 fltmgr!FltpDispatch+0×152
917c0c4c 808f7601 nt!IofCallDriver+0×45
917c0c60 808f1b45 nt!IopSynchronousServiceTail+0×10b
917c0c84 afdfebd5 nt!NtQueryDirectoryFile+0×5d
917c0cf8 afdff95d driverD+0×8bd5
917c0d30 8088ac9c driverD+0×995d
917c0d30 7c9485ec nt!KiFastCallEntry+0xfcOther wait chains seem to be subordinate to the main deadlock chain.
The following memory dump of a hanging process was manually (Volume 1, page 486) generated:
Loading Dump File [Process.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER'
Applying default analysis command shows the following stack traces:
0:000> .kframes 100 Default stack trace depth is 0n256 frames 0:000> !analyze -v [...] STACK_TEXT: 009ef258 7c827d0b ntdll!KiFastSystemCallRet 009ef25c 7c83d236 ntdll!NtWaitForSingleObject+0xc009ef298 7c83d281 ntdll!RtlpWaitOnCriticalSection+0×1a3 009ef2b8 7c82dabf ntdll!RtlEnterCriticalSection+0xa8009ef358 7c82dab1 ntdll!LdrpGetProcedureAddress+0×128 009ef374 77e764ea ntdll!LdrGetProcedureAddress+0×18009efa00 7c82a319 ntdll!RtlpFindEntry+0×68 009efc2c 7c3416b3 ntdll!RtlAllocateHeap+0×606 009efc6c 7c3416db msvcr71!_heap_alloc+0xe0 009efc74 7c360947 msvcr71!_nh_malloc+0×10 009efc80 0285f893 msvcr71!operator new+0xb [...] 009effb8 77e64829 msvcr71!_endthreadex+0xa0 009effec 00000000 kernel32!BaseThreadStart+0×34 [...] BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT_STATUS_BREAKPOINT009ef5d8 7c34c456 kernel32!UnhandledExceptionFilter+0×46f 009ef5f4 7c34957c msvcr71!_XcptFilter+0×15f 009ef600 7c34246e msvcr71!_endthreadex+0xb7 009ef628 7c828752 msvcr71!_except_handler3+0×61 009ef64c 7c828723 ntdll!ExecuteHandler2+0×26 009ef6f4 7c82855e ntdll!ExecuteHandler+0×24 009ef6f4 7c82be3e ntdll!KiUserExceptionDispatcher+0xe
We see the exeption was dispatched because of heap corruption (Volume 1, page 257) and the unhandled exception filter is blocked (Volume 2, page 184) waiting for a critical section. We can immediately recommend enabling full page heap. However, let's explore the memory dump file further. By listing all threads (Volume 1, page 409) we find that there were two exceptions (Volume 1, page 255) with the second one having the following stack trace (shown in smaller font for visual clarity):
0:000> ~*kb
[...]
98 Id: be4.2ca4 Suspend: 1 Teb: 7ff68000 Unfrozen
ChildEBP RetAddr Args to Child
0f83c80c 7c827d0b 7c83d236 00000154 00000000 ntdll!KiFastSystemCallRet
0f83c810 7c83d236 00000154 00000000 00000000
ntdll!NtWaitForSingleObject+0xc
0f83c84c 7c83d281 00000154 00000004 0f83c8b0
ntdll!RtlpWaitOnCriticalSection+0×1a3
0f83c86c 7c82dabf 7c8877a0 00000000 00000000
ntdll!RtlEnterCriticalSection+0xa8
0f83c90c 7c82dab1 00000000 77e767cc 00000000
ntdll!LdrpGetProcedureAddress+0×128
0f83c928 77e764ea 00000000 77e767cc 00000000
ntdll!LdrGetProcedureAddress+0×18
0f83cb8c 77e792a3 0f83cbb4 77e61ac1 0f83cbbc
kernel32!UnhandledExceptionFilter+0×46f
0f83cb94 77e61ac1 0f83cbbc 00000000 0f83cbbc kernel32!BaseThreadStart+0×4a
0f83cbbc 7c828752 0f83cca0 0f83ffdc 0f83ccbc
kernel32!_except_handler3+0×61
0f83cbe0 7c828723 0f83cca0 0f83ffdc 0f83ccbc ntdll!ExecuteHandler2+0×26
0f83cc88 7c82855e 0f83c000 0f83ccbc 0f83cca0 ntdll!ExecuteHandler+0×24
0f83cc88 7c82be3e 0f83c000 0f83ccbc 0f83cca0
ntdll!KiUserExceptionDispatcher+0xe0f83fdfc 77c812f0 001c85c0 0bf0f510 00189d08
rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×41e
0f83fe20 77c88678 0bf0f548 0f83fe38 001c85c0
rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0f83ff84 77c88792 0f83ffac 77c8872d 0bf0f510
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0f83ff8c 77c8872d 0bf0f510 00000000 00000000
rpcrt4!RecvLotsaCallsWrapper+0xd
0f83ffac 77c7b110 00145520 0f83ffec 77e64829
rpcrt4!BaseCachedThreadRoutine+0×9d
0f83ffb8 77e64829 0be74ce0 00000000 00000000
rpcrt4!ThreadStartRoutine+0×1b
0f83ffec 00000000 77c7b0f5 0be74ce0 00000000 kernel32!BaseThreadStart+0×34
[...]
102 Id: be4.2ac0 Suspend: 1 Teb: 7ff4f000 Unfrozen
ChildEBP RetAddr Args to Child
134df91c 7c827d0b 7c83d236 000004e4 00000000 ntdll!KiFastSystemCallRet
134df920 7c83d236 000004e4 00000000 00000000
ntdll!NtWaitForSingleObject+0xc
134df95c 7c83d281 000004e4 00000004 00340000
ntdll!RtlpWaitOnCriticalSection+0×1a3
134df97c 7c82a264 00340608 00000000 0000008c
ntdll!RtlEnterCriticalSection+0xa8
134dfba4 7c3423aa 00340000 00000008 0000008c ntdll!RtlAllocateHeap+0×313
134dfbe4 7c3422cb 00000001 0000008c 00000000 msvcr71!calloc+0xe6
134dfbfc 7c81a352 7c340000 00000002 00000000 msvcr71!_CRTDLL_INIT+0×138
134dfc1c 7c82ed97 7c34229f 7c340000 00000002
ntdll!LdrpCallInitRoutine+0×14
134dfcb8 7c82ec9f 134dfd28 134dfd28 00000000
ntdll!LdrpInitializeThread+0×10d
134dfd14 7c8284c5 134dfd28 7c800000 00000000 ntdll!_LdrpInitialize+0×16f
00000000 00000000 00000000 00000000 00000000
ntdll!KiUserApcDispatcher+0×25If we inspect the list of locked critical sections and compare stack trace parameters for threads 2ac0 and 2ca4 we would see them deadlocked (Volume 1, page 276):
0:000> !locks CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0 WaiterWoken No LockCount 6 RecursionCount 1 OwningThread 2ac0 EntryCount 0 ContentionCount 36 *** Locked
CritSec +340608 at 00340608 WaiterWoken No LockCount 32 RecursionCount 1 OwningThread 2ca4 EntryCount 0 ContentionCount 6f *** Locked [...]
Looking at other locks (omitted here) we can find additional wait chains (Volume 1, page 490) but we employ the default hang analysis command to find one chain for us:
0:000> !analyze -v -hang [...] LOADERLOCK_BLOCKED_API: UnhandledExceptionFilter:LdrGetProcedureAddress: LdrpGetProcedureAddress: LOADERLOCK_OWNER_API: _LdrpInitialize:LdrpInitializeThread:LdrpCallInitRo utine: DERIVED_WAIT_CHAIN: Dl Eid Cid WaitType -- --- ------- -------------------------- 2 be4.c20 Critical Section --> x 98 be4.2ca4 Critical Section --> x 102 be4.2ac0 Critical Section --^ WAIT_CHAIN_COMMAND: ~2s;k;;~98s;k;;~102s;k;; BLOCKING_THREAD: 00002ca4 DEFAULT_BUCKET_ID: APPLICATION_HANG_DEADLOCK_HeapCorruption PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_DEADLOCK_HeapCorruption [...]
Let's look at one complete memory dump from a hanging system. It was reported that an application was freezing when trying to create a printer. In its process we can identify the blocked thread that was processing an LPC request and waiting for an LPC reply from a coupled process (Volume 1, page 419), a spooler service:
0: kd> !thread 88db89a8 1f
THREAD 88db89a8 Cid 0764.31b8 Teb: 7ffdc000 Win32Thread: bc2dc980 WAIT:
(Unknown) UserMode Non-Alertable
88db8b94 Semaphore Limit 0x1
Waiting for reply to LPC MessageId 0016346d:
Current LPC port e179d780
Not impersonating
DeviceMap e157b068
Owning Process 8a95f690 Image: ApplicationA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2750643 Ticks: 153453 (0:00:39:57.703)00e6e5c8 73082e15 WINSPOOL!RpcAddPrinter+0×1c
00e6ea50 0040dd93 WINSPOOL!AddPrinterW+0×102
00e6ef40 0040f535 ApplicationA!CreatePrinter+0×8d9
[...]
00e6f8b4 77ce33e1 RPCRT4!Invoke+0×30
00e6fcb4 77ce35c4 RPCRT4!NdrStubCall2+0×299
00e6fcd0 77c7ff7a RPCRT4!NdrServerCall2+0×19
00e6fd04 77c8042d RPCRT4!DispatchToStubInCNoAvrf+0×38
00e6fd58 77c80353 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×11f00e6fd7c 77c7e0d4 RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3 00e6fdbc 77c7e080 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xc0 00e6fdfc 77c812f0 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×41e 00e6fe20 77c88678 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127 00e6ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430 00e6ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd 00e6ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d 00e6ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
We follow an LPC wait chain (page 97) and see a thread blocked by a message box (Volume 2, page 177):
0: kd> !lpc message0016346dSearching message 16346d in threads ...Server thread 891e9db0 is working on message 16346dClient thread 88db89a8 waiting a reply from 16346d Searching thread 88db89a8 in port rundown 0: kd> !thread891e9db01f THREAD 891e9db0 Cid 045c.3b3c Teb: 7ffa7000 Win32Thread: bc2d4ea8 WAIT: (Unknown) UserMode Non-Alertable 88f5c850 SynchronizationEvent Impersonation token: e76f5030 (Level Impersonation) DeviceMap e157b068 Owning Process 8ab10460 Image: spoolsv.exe Attached Process N/A Image: N/A Wait Start TickCount 2414962Context Switch Count 6797 LargeStack UserTime 00:00:00.359 KernelTime 00:00:01.437 Win32 Start Address 0×0016346d LPC Server thread working on message Id 16346d Start Address kernel32!BaseThreadStartThunk (0×77e617ec) Stack Init b6aaa000 Current b6aa9c50 Base b6aaa000 Limit b6aa6000 Call 0 Priority 11 BasePriority 8 PriorityDecrement 0 Kernel stack not resident. ChildEBP RetAddr b6aa9c68 80833ec5 nt!KiSwapContext+0×26 b6aa9c94 80829bc0 nt!KiSwapThread+0×2e5 b6aa9cdc bf89aacc nt!KeWaitForSingleObject+0×346 b6aa9d38 bf89d952 win32k!xxxSleepThread+0×1be b6aa9d4c bf89e310 win32k!xxxRealWaitMessageEx+0×12 b6aa9d5c 8088ad3c win32k!NtUserWaitMessage+0×14 b6aa9d5c 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ b6aa9d64) 021267bc 7739bf53 ntdll!KiFastSystemCallRet 021267f4 7738965e USER32!NtUserWaitMessage+0xc 0212681c 7739f762 USER32!InternalDialogBox+0xd0 02126adc 7739f047 USER32!SoftModalMessageBox+0×94bTicks: 489134 (0:02:07:22.718)
02126c2c 7739eec9 USER32!MessageBoxWorker+0×2ba
02126c84 773d7d0d USER32!MessageBoxTimeoutW+0×7a
02126cb8 773c42c8 USER32!MessageBoxTimeoutA+0×9c
02126cd8 773c42a4 USER32!MessageBoxExA+0×1b
02126cf4 6dfcf8c2 USER32!MessageBoxA+0×45
WARNING: Frame IP not in any known module. Following frames may be wrong.
02126d98 530a2935 0×6dfcf8c2
02126d9c 20656d6f 0×530a2935
[...]Looking at that thread waiting time (Volume 1, page 343) we see the problem happened about 1 hour 20 minutes before our hanging application attempted to print. We want to see other threads in the spooler but most of them are paged out (page 118):
0: kd> !process 8ab10460 1f
[...]
THREAD 8a6aedb0 Cid 045c.1090 Teb: 7ffab000 Win32Thread: bc33bbe8 WAIT:
(Unknown) UserMode Non-Alertable
8abb2f40 SynchronizationEvent
IRP List:
8a0a2ed0: (0006,0094) Flags: 00000800 Mdl: 00000000
Impersonation token: e7c94760 (Level Impersonation)
DeviceMap e9194e68
Owning Process 8ab10460 Image: spoolsv.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1897118 Ticks: 1006978 (0:04:22:14.031)
Context Switch Count 9711 LargeStack
UserTime 00:00:00.343
KernelTime 00:00:02.812
Win32 Start Address 0x00165a40
LPC Server thread working on message Id 165a40
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init b7c28000 Current b7c27c60 Base b7c28000 Limit b7c24000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
[...]Coming back to our message box thread we inspect MessageBox parameters to see the message:
0: kd> .process /r /p 8ab10460 Implicit process is now 8ab10460 Loading User Symbols
0: kd> !thread 891e9db0 [...] ChildEBP RetAddr Args to Child [...] 02126cf4 6dfcf8c2 0000000002126d4c02126d2c USER32!MessageBoxA+0×45 [...] 0: kd> da /c 100 02126d4c 02126d4c "Function address0×77e63ce4caused a protection fault. (exception code 0xc0000005). 0: kd> ln 0x77e63ce4 (77e63c87)kernel32!GetModuleFileNameW+0×83| (77e63d7a) kernel32!G etProcAddress
We see that the thread experienced an access violation in GetModuleFileName API call. We also see that the stack trace is incorrect (Volume 1, page 288) and we try to reconstruct the point of exception by looking at the thread raw stack and searching for any hidden exception (Volume 1, page 271):
0: kd> .thread 891e9db0
Implicit thread is now 891e9db0
0: kd> !teb
TEB at 7ffa7000
ExceptionList: 0212757c
StackBase: 02130000
StackLimit: 02123000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffa7000
EnvironmentPointer: 00000000
ClientId: 0000045c . 00003b3c
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffde000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 00: kd> dds 02123000 02130000 02123000 00000000 02123004 00000000 02123008 00000000 0212300c 00000000 [...] 021275a8 02127648 021275ac 0212aa0c021275b0 7c8315c2 ntdll!RtlDispatchException+0×91 021275b4 02127648 021275b8 0212aa0c 021275bc 02127664021275c0 02127624 021275c4 6dfd0a54 021275c8 063b0000 021275cc 02127648 021275d0 7c8877ec ntdll!PebLdr+0xc 021275d4 77e61a60 kernel32!_except_handler3 021275d8 77e69cf8 kernel32!'string'+0xc 021275dc ffffffff 021275e0 00080000 021275e4 00089878 021275e8 77b92e40 VERSION!`string' 021275ec 00000000 021275f0 00000000 021275f4 00017608 021275f8 02127540 021275fc 77b92e40 VERSION!`string' 02127600 02127648 02127604 7c828270 ntdll!_except_handler3 02127608 7c829f60 ntdll!CheckHeapFillPattern+0×64 0212760c ffffffff 02127610 7c829f59 ntdll!RtlFreeHeap+0×70f 02127614 77e62444 kernel32!LocalFree+0×27 02127618 00080000 0212761c 00000000 02127620 77e62457 kernel32!LocalFree+0×108 02127624 5dff1510 02127628 02130000 0212762c 00000000 02127630 0212797c02127634 7c82855e ntdll!KiUserExceptionDispatcher+0xe 02127638 02123000 0212763c 0212766402127640 02127648 02127644 02127664 02127648 c0000005 0212764c 00000000 02127650 00000000 02127654 77e63ce4 kernel32!GetModuleFileNameW+0×83 02127658 00000002 0212765c 00000000 02127660 00000018 02127664 0001003f
02127668 00000000 0212766c 00000000 02127670 00000000 [...] 0: kd> .cxr 02127664 eax=00000000 ebx=00000000 ecx=7c82d279 edx=7c8877a0 esi=7c8877ec edi=063b0000 eip=77e63ce4 esp=02127930 ebp=0212797c iopl=0 nv up ei ng nz ac pe cy cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010297 kernel32!GetModuleFileNameW+0x83: 001b:77e63ce4 3b7818 cmp edi,dword ptr [eax+18h] ds:0023:00000018=???????? 0: kd> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child 0212797c 77390f3d 063b0000 021279d0 00000104 kernel32!GetModuleFileNameW+0x83 02127990 77390f13 063b0000 021279d0 00000104 USER32!WowGetModuleFileName+0x14 02127bdc 7738bcbd 063b0000 00000097 00000003 USER32!LoadIcoCur+0x76 02127bfc 06425db5063b000000000097 02127d28 USER32!LoadIconW+0×1b02127c38 7c82cefe 00000704 00001f10 02127c58 0×6425db5 02127c4c 7c82ceb0 00001f10 00002614 00000704 ntdll!ARRAY_FITS+0×29 02127c8c c0150008 00000000 03923f68 0000001d ntdll!RtlpLocateActivationContextSection+0×1da 02127d20 7c82e0ef 00000000 00000000 00071f10 0xc0150008WARNING: Frame IP not in any known module. Following frames may be wrong.
What we see is the fact that the thread tried to load an icon from 063b0000 module not listed in lmv command output list perhaps because it was not fully loaded yet. We see that the thread also holds a loader critical section:
0: kd> !cs -l -o -s ----------------------------------------- DebugInfo = 0x7c8877c0Critical section = 0×7c8877a0 (ntdll!LdrpLoaderLock+0×0)LOCKED LockCount = 0xC WaiterWoken = No OwningThread = 0×00003b3c RecursionCount = 0×1 LockSemaphore = 0×108 SpinCount = 0×00000000OwningThread = .thread 891e9db0ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
However we can inspect the unknown module (Volume 1, page 367) PE headers left in memory:
0: kd> !dh 063b0000
File Type: DLL
FILE HEADER VALUES
14C machine (i386)
4 number of sections
42B9172C time date stamp Wed Jun 22 08:45:48 2005
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
210E characteristics
Executable
Line numbers stripped
Symbols stripped
32 bit word machine
DLL
[...]
Debug Directories(1)
Type Size Address Pointer
cv 16 10198 f598 Format: NB10, 42b9172c, 1,
c:printerdriver123UI.pdb
[...]We see that the module name is printerdriver123UI. Therefore, we should recommend checking the printer driver package containing this component. lmv command lists many printer drivers with the similar name prefix and we look at the thread execution residue (Volume 2, page 239) to get the better stack trace and find the corresponding printer. We know that the application called AddPrinter API and we try to find the corresponding call on the raw stack that we examined for hidden exceptions before (the stack trace below is shown in smaller font for visual clarity):
[...] 0212ef2c 01f203900212ef300212f3f8 0212ef34 76144799 localspl!SplAddPrinter+0×5f3 0212ef38 01efb730 0212ef3c 00000003 [...] 0212f3f8 0212f424 0212f3fc 76144ab2 localspl!LocalAddPrinterEx+0×2e 0212f400 00000000
[...] 0212f424 0212f874 0212f428 7407018b SPOOLSS!AddPrinterExW+0×151 0212f42c 00000000 [...] 0212f874 0212f890 0212f878 74070254 SPOOLSS!AddPrinterW+0×17 0212f87c 00000000 [...] 0: kd> kv L=0212ef30 ChildEBP RetAddr Args to Child 0212ef30 76144799 01efb730 00000003 00000000 kernel32!GetModuleFileNameW+0x83 0212f3f8 76144ab2 00000000 00000002 011116e0 localspl!SplAddPrinter+0x5f3 0212f424 7407018b 00000000 00000002 011116e0 localspl!LocalAddPrinterEx+0x2e 0212f874 74070254 00000000 00000002 011116e0 SPOOLSS!AddPrinterExW+0x151 0212f890 0100792d 00000000 00000002011116e0SPOOLSS!AddPrinterW+0×17 0212f8ac 01006762 00000000 0118f968 00159180 spoolsv!YAddPrinter+0×75 0212f8d0 77c80193 00000000 0118f968 00159180 spoolsv!RpcAddPrinter+0×37 0212f8f8 77ce33e1 0100672b 0212fae0 00000005 RPCRT4!Invoke+0×30 0212fcf8 77ce35c4 00000000 00000000 000cefbc RPCRT4!NdrStubCall2+0×299 0212fd14 77c7ff7a 000cefbc 000bbd20 000cefbc RPCRT4!NdrServerCall2+0×19 0212fd48 77c8042d 01002c47 000cefbc 0212fdec RPCRT4!DispatchToStubInCNoAvrf+0×38 0212fd9c 77c80353 00000005 00000000 0100d228 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×11f 0212fdc0 77c811dc 000cefbc 00000000 0100d228 RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3 0212fdfc 77c812f0 000c9e80 000a7af0 0014ce70 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×42c 0212fe20 77c88678 000a7b28 0212fe38 000c9e80 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127 0212ff84 77c88792 0212ffac 77c8872d 000a7af0 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430 0212ff8c 77c8872d 000a7af0 00000000 00000000 RPCRT4!RecvLotsaCallsWrapper+0xd 0212ffac 77c7b110 000885d8 0212ffec 77e64829 RPCRT4!BaseCachedThreadRoutine+0×9d 0212ffb8 77e64829 000a3750 00000000 00000000 RPCRT4!ThreadStartRoutine+0×1b 0212ffec 00000000 77c7b0f5 000a3750 00000000 kernel32!BaseThreadStart+0×34
Examination of the server-side AddPrinter arguments reveals the printer name:
0: kd> dpu 011116e0 l5
011116e0 00000000
011116e4 00158f48
011116e8 00000000
011116ec 00158fe4
011116f0 00159040 "PRINTER NAME ABCDEF"For one manual user dump (Volume 1, page 486) of a hanging service neither !analyze -v nor !analyze -v -hang command found the problem:
Loading Dump File [ServiceA.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER' 0:000> !analyze -v -hang DERIVED_WAIT_CHAIN: Dl Eid Cid WaitType -- --- ------- -------------------------- 0 678.67c File IO WAIT_CHAIN_COMMAND: ~0s;k;; BLOCKING_THREAD: 0000067c DEFAULT_BUCKET_ID: APPLICATION_HANG_BlockedOn_FileIO PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_BlockedOn_FileIO LAST_CONTROL_TRANSFER: from 7c82776b to 7c8285ec STACK_TEXT: 0012fbc8 7c82776b ntdll!KiFastSystemCallRet 0012fbcc 77e218b2 ntdll!NtReadFile+0xc 0012fc34 77d85edb kernel32!ReadFile+0x16c 0012fc60 77d85f82 advapi32!ScGetPipeInput+0x2a 0012fcd4 77d71ed9 advapi32!ScDispatcherLoop+0x51 0012ff38 004025e0 advapi32!StartServiceCtrlDispatcherW+0xe3 0012ff60 0042c8fd ServiceA!main+0x110 0012ffc0 77e4f23b ServiceA!mainCRTStartup+0x143 0012fff0 00000000 kernel32!BaseProcessStart+0x23
As usual, the critical section list was checked for any locked sections and also to find wait chains (Volume 1, page 490):
0:000> !cs -l -o -s ----------------------------------------- DebugInfo = 0x7c8877c0Critical section = 0×7c8877a0 (ntdll!LdrpLoaderLock+0×0) LOCKED LockCount = 0xA2WaiterWoken = No OwningThread = 0×00011f64 RecursionCount = 0×1 LockSemaphore = 0×188 SpinCount = 0×00000000OwningThread DbgId = ~20sOwningThread Stack = ChildEBP RetAddr Args to Child 0327f91c 7c827d0b 7c83d236 00000754 00000000 ntdll!KiFastSystemCallRet 0327f920 7c83d236 00000754 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 0327f95c 7c83d281 00000754 00000004 00370000 ntdll!RtlpWaitOnCriticalSection+0×1a3 0327f97c 7c82a2640037060800000000 0000008c ntdll!RtlEnterCriticalSection+0xa8 0327fba4 7c3423aa 00370000 00000008 0000008c ntdll!RtlAllocateHeap+0×313 0327fbe4 7c3422cb 00000001 0000008c 00000000 msvcr71!calloc+0xe6 0327fbfc 7c81a352 7c340000 00000002 00000000 msvcr71!_CRTDLL_INIT+0×138 0327fc1c 7c82ed97 7c34229f 7c340000 00000002 ntdll!LdrpCallInitRoutine+0×14 0327fcb8 7c82ec9f 0327fd28 0327fd28 00000000 ntdll!LdrpInitializeThread+0×10d 0327fd14 7c8284c5 0327fd28 7c800000 00000000 ntdll!_LdrpInitialize+0×16f 00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0×25 ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled. [...] DebugInfo = 0×7c887e20Critical section = 0×00370608 (+0×370608) LOCKED LockCount = 0×7WaiterWoken = No OwningThread = 0×000027ac RecursionCount = 0×1 LockSemaphore = 0×754 SpinCount = 0×00000fa0OwningThread DbgId = ~17sOwningThread Stack = ChildEBP RetAddr Args to Child 0170ebe0 7c82775b 77e62610 d0000144 00000004 ntdll!KiFastSystemCallRet0170ebe4 77e62610 d0000144 00000004 00000000 ntdll!NtRaiseHardError+0xc 0170ee50 77e592a3 0170ee78 77e41ac1 0170ee80 kernel32!UnhandledExceptionFilter+0×51a0170ee58 77e41ac1 0170ee80 00000000 0170ee80 kernel32!BaseThreadStart+0×4a 0170ee80 7c828752 0170ef64 0170ffdc 0170ef80 kernel32!_except_handler3+0×61 0170eea4 7c828723 0170ef64 0170ffdc 0170ef80 ntdll!ExecuteHandler2+0×26 0170ef4c 7c82855e 0170d000 0170ef80 0170ef64 ntdll!ExecuteHandler+0×24 0170ef4c 7c82a754 0170d000 0170ef80 0170ef64 ntdll!KiUserExceptionDispatcher+0xe (CONTEXT @ 0170ef80) 0170f258 7c82a82b 0185d6a8 00690043 0170f320 ntdll!RtlpCoalesceFreeBlocks+0×36e 0170f340 7c34218a 00370000 00000000 0185bad8 ntdll!RtlFreeHeap+0×38e 0170f388 004288df 0185bad8 05afd9c4 00000000 msvcr71!free+0xc3 0170f3b4 00428a16 05afd9c4 00001bc4 0185bad8 ServiceA!SetData+0×4df ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled. [...]
We see that 162 threads (0xA2) are waiting for 0×7c8877a0 locked critical section whose owner thread #20 is waiting for the critical section 00370608 owned by the thread #17. The latter thread is blocked (Volume 2, page 184) in an unhandled exception filter raising a hard error message box. Applying the new exception context we confirm heap corruption (Volume 1, page 257) problem:
0:000> .cxr 0170ef80
eax=0185d6a8 ebx=00370000 ecx=00690043 edx=00000001 esi=0185d6a0
edi=01858030
eip=7c82a754 esp=0170f24c ebp=0170f258 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
ntdll!RtlpCoalesceFreeBlocks+0×36e:
7c82a754 8b09 mov ecx,dword ptr
[ecx] ds:0023:00690043=????????
0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0170f258 7c82a82b ntdll!RtlpCoalesceFreeBlocks+0×36e
0170f340 7c34218a ntdll!RtlFreeHeap+0×38e
0170f388 004288df msvcr71!free+0xc3
0170f3b4 00428a16 ServiceA!SetData+0×4df
[...]
0170ffec 00000000 kernel32!BaseThreadStart+0×34Therefore, we suggest enabling full page heap either by using gflags or Application Verifier. Looking further at other locked critical sections omitted above we notice that one is inaccessible and from an unloaded module:
Cannot read structure field value at 0x5531953c, error 0
Cannot determine if the critical section is locked or not.
-----------------------------------------
Critical section = 0x55319538 (<Unloaded_DetouredA.DLL>+0×29538)
DebugInfo = 0×00147108
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not
enabled.We see that execution residue (Volume 2, page 239) in the list of unloaded modules too:
0:000> lmt
[...]
Unloaded modules:
[...]
697b0000 697c7000 faultrep.dll
Timestamp: Sat Feb 17 15:12:57 2007 (45D71B79)
Checksum: 0001D340
[...]
552f0000 55320000 DetouredA.DLL
Timestamp: Wed Dec 19 09:38:46 2007 (4768E6A6)
Checksum: 00030564Recalling that this hooksware (Volume 2, page 63) had problems before, we also suggest removing the software package that injects DetouredA module to test the stability of the system.
One service was hanging and its user dump was taken. Inspection of its critical sections reveals the one thread blocking 32 other threads. The owner stack trace points to Message Box pattern (Volume 2, page 177):
0:000> !cs -l -o -s [...] DebugInfo = 0x00151e20 Critical section = 0x00455bd0 (ServiceA!g_CritSec+0x0) LOCKEDLockCount = 0×20WaiterWoken = No OwningThread = 0×00005774 RecursionCount = 0×1 LockSemaphore = 0×18 SpinCount = 0×00000000 OwningThread DbgId = ~24s OwningThread Stack = ChildEBP RetAddr Args to Child 00d0d854 77e3bf53 77e3610a 00000000 00000000 ntdll!KiFastSystemCallRet 00d0d88c 77e2969d 07a1006e 00000000 00000001 user32!NtUserWaitMessage+0xc 00d0d8b4 77e3f762 77e20000 00181620 00000000 user32!InternalDialogBox+0xd0 00d0db74 77e3f047 00d0dcd0 00d0e810 00d0f3dc user32!SoftModalMessageBox+0×94b 00d0dcc4 77e3eec9 00d0dcd0 00000028 00000000 user32!MessageBoxWorker+0×2ba 00d0dd1c 77e3ee65 00000000 00d0de94 00d0dd84 user32!MessageBoxTimeoutW+0×7a 00d0dd3c 77e3ee41 00000000 00d0de94 00d0dd84 user32!MessageBoxExW+0×1b 00d0dd58 25010b67 0000000000d0dd8400d0de94user32!MessageBoxW+0×45 WARNING: Stack unwind information not available. Following frames may be wrong. 00d0de84 25010372 00d0de94 00010010 00690044moduleA!DllMain+0×8b27 00d0e298 25018e3d 00002754 25067c58 25066a74moduleA!DllMain+0×8332 [...] 00d0f974 00403e83 00000008 00d0f98c 00d0fae0 ServiceA!Create+0×39 [...]
Inspecting message box parameters we see a self-diagnostic message (Volume 2, page 318):
0:000> du /c 10000d0de94 "The installation of Software because00d0de94moduleAcould not be installed."
The system was hanging and a manual kernel dump (Volume 1, page 479) file was generated:
0: kd> !analyze -v [...] NMI_HARDWARE_FAILURE (80) This is typically due to a hardware malfunction. The hardware supplier should be called. Arguments: Arg1: 004f4454 Arg2: 00000000 Arg3: 00000000 Arg4: 00000000 [...]
Initially it looked like an earlier premature crash dump taken after the reboot:
0: kd> version
[...]
System Uptime: 0 days 0:00:54.203
[...]However, the incident description stated that the system was hanging during its startup so we went further to look at a stack trace collection (Volume 1, page 409) of all its threads and found that the main thread (Volume 1, page 436) of spooler was hanging for about 23 seconds after 4 seconds since the process creation:
0: kd> !process 0 ff
[...]
PROCESS 8cf06020 SessionId: 0 Cid: 04a4 Peb: 7ffde000 ParentCid:
0244
DirBase: cff8c1a0 ObjectTable: d70086d0 HandleCount: 24.
Image: spoolsv.exe
VadRoot 8bb19130 Vads 32 Clone 0 Private 60. Modified 0. Locked 0.
DeviceMap d66028f0
Token d6e88988
ElapsedTime 00:00:27.921
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 27844
QuotaPoolUsage[NonPagedPool] 1280
Working Set Sizes (now,min,max) (312, 50, 345) (1248KB, 200KB,
1380KB)
PeakWorkingSetSize 312
VirtualSize 12 Mb
PeakVirtualSize 12 Mb
PageFaultCount 322
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 80
THREAD 8cf22db0 Cid 04a4.04a8 Teb: 7ffdd000 Win32Thread: bc155948 WAIT:
(Unknown) KernelMode Non-Alertable
8beeca28 SynchronizationEvent
IRP List:
8beecd08: (0006,0094) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap d66028f0
Owning Process 8cf06020 Image: spoolsv.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1981 Ticks: 1488 (0:00:00:23.250)
Context Switch Count 9 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0×0100468c
Start Address 0×77e617f8
Stack Init b6a63b30 Current b6a630cc Base b6a64000 Limit b6a60000 Call
b6a63b34
Priority 9 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
b6a630e4 80833465 nt!KiSwapContext+0×26
b6a63110 80829a62 nt!KiSwapThread+0×2e5
b6a63158 b5dbc727 nt!KeWaitForSingleObject+0×346
b6a6318c b5dbd020 mrxdav!RxConstructSrvCall+0×187
b6a63200 b5dbd144 mrxdav!RxFindOrCreateConnections+0×58a
b6a6324c b5dbd552 mrxdav!RxConstructVirtualNetRoot+0×38
b6a632b8 b5db27fc mrxdav!RxFindOrConstructVirtualNetRoot+0×220
b6a63304 b5da7ff1 mrxdav!RxPrefixClaim+0×160b6a6331c b5d93f72 mrxdav!RxCommonDevFCBIoCtl+0×71 b6a633ac b5da780c mrxdav!RxFsdCommonDispatch+0×320 b6a633cc b5da0fc4 mrxdav!RxFsdDispatch+0xd4 b6a63440 8081df65 mrxdav!MRxDAVFsdDispatch+0×1f0 b6a63454 bace232e nt!IofCallDriver+0×45 b6a63460 bace24f6 Mup!MupiIssueQueryRequest+0×18 b6a63488 bace2414 Mup!MupiResolvePrefix+0×11b b6a634d4 bacdf4ed Mup!CreateRedirectedFile+0×385 b6a63530 8081df65 Mup!MupCreate+0×1cb b6a63544 808f8f71 nt!IofCallDriver+0×45 b6a6362c 80937942 nt!IopParseDevice+0xa35 b6a636ac 80933a76 nt!ObpLookupObjectName+0×5b0 b6a63700 808ec76b nt!ObOpenObjectByName+0xea b6a63884 8088978c nt!NtQueryAttributesFile+0×11d b6a63884 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ b6a63894)
Looking at IRP we see that I/O request was redirected to the following file share:
0: kd> !irp 8beecd08
Irp is active with 1 stacks 1 is current (= 0x8beecd78)
No Mdl: No System Buffer: Thread 8cf22db0: Irp stack trace.
cmd flg cl Device File Completion-Context
>[ 0, 0] 0 0 8cfdcee8 b6a637f0 00000000-00000000
FileSystemMup
Args: b6a63570 01200000 00070000 00000000
0: kd> !fileobj b6a637f0
SERVER-Af$AppFilesComponent.dll
Device Object: 0x8cfdcee8 FileSystemMup
Vpb is NULL
Flags: 0x0
CurrentByteOffset: 0When looking at other processes we see 2 threads blocked (Volume 1, page 184) in svchost.exe in the same redirector component:
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[8bb2a728 svchost.exe]
3d0.0003d4 8bb2a290 0000701 Blocked nt!NtReadFile+0x5d5
3d0.0003d8 8bb2aba0 00006ff Blocked nt!ObpWaitForMultipleObjects+0x202
3d0.0003dc 8bb338c0 00005eb Blocked nt!ObpWaitForMultipleObjects+0x202
3d0.0003e8 8bb2cdb0 0000892 Blocked nt!ObpWaitForMultipleObjects+0x202
3d0.0003ec 8bb2c838 0000892 Blocked nt!KiFastCallEntry+0xfc
3d0.0003f0 8bb26cd0 0000892 Blocked nt!KiFastCallEntry+0xfc
3d0.000488 8ceffb88 00006fb Blocked mrxdav!UMRxAssignWork+0xdf
3d0.00048c 8cec3db0 00005d0 Blocked mrxdav!UMRxAssignWork+0xdf
3d0.000490 8cec3b40 00006fb Blocked nt!KiFastCallEntry+0xfc
3d0.0004d4 8bb02918 00005c7 Blocked nt!KiFastCallEntry+0xfc
3d0.0004d8 8bafdb40 00005c4 Blocked nt!KiFastCallEntry+0xfc
[...]In these two threads we see pass through functions (page 129) in driverA and driverB:
0: kd> !thread 8ceffb88
THREAD 8ceffb88 Cid 03d0.0488 Teb: 7ffd8000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
8cef1f28 QueueObject
Not impersonating
DeviceMap d6e2c298
Owning Process 8bb2a728 Image: svchost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1682 Ticks: 1787 (0:00:00:27.921)
Context Switch Count 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x5a005e80
Start Address 0x77e617ec
Stack Init b5ed8000 Current b5ed7a64 Base b5ed8000 Limit b5ed5000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
b5ed7a7c 80833465 nt!KiSwapContext+0x26
b5ed7aa8 8082b60f nt!KiSwapThread+0x2e5
b5ed7af0 b5da714d nt!KeRemoveQueue+0x417
b5ed7b10 b5d9d7db mrxdav!UMRxAssignWork+0xdf
b5ed7b70 bae71bc5 mrxdav!MRxDAVFastIoDeviceControl+0x13d
b5ed7ba8 bae7f42d fltmgr!FltpPerformFastIoCall+0x153
b5ed7bf4 b5f57300 fltmgr!FltpFastIoDeviceControl+0xb1
b5ed7c20 b6e20f8f driverB!FS_FastIoDeviceControl+0×60
b5ed7c5c 808f5e2f driverA!FastIoDeviceControl+0xcf
b5ed7d00 808eed08 nt!IopXxxControlFile+0×255
b5ed7d34 8088978c nt!NtDeviceIoControlFile+0×2a
b5ed7d34 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ b5ed7d64)One application was hanging after printing and its user dump was taken. Inspection of threads reveals only one blocked thread (Volume 2, page 184):
0:000> ~*kL . 0 Id: 486c.5704 Suspend: 1 Teb: 7ffa9000 Unfrozen ChildEBP RetAddr 01c0fb68 7c827cfb ntdll!KiFastSystemCallRet 01c0fb6c 77e6202c ntdll!NtWaitForMultipleObjects+0xc 01c0fc14 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a01c0fdc0 77bcaf82 msvcrt!_cinit+0xad01c0fc30 00439f16 kernel32!WaitForMultipleObjects+0×1801c0fd94 77bcaed6 App!~MyPrinter+0xe01c0fdd0 77baf582 msvcrt!_cexit+0xb01c0fde0 7c81a352 msvcrt!__CRTDLL_INIT+0×15c 01c0fe00 7c830e70 ntdll!LdrpCallInitRoutine+0×1401c0feb8 77e668a3 ntdll!LdrShutdownProcess+0×182
We see that the process was being terminated and the last thread was blocked in a destructor of MyPrinter class. However, we see some historical information (Volume 1, page 457), faultrep.dll unloaded (Volume 1, page 113), and this means that some exception happened prior to the process shutdown:
0:000> lm
[...]
Unloaded modules:
69500000 69517000 faultrep.dll
[...]Then we look at the thread raw stack date to see any signs of execution residue (Volume 2, page 239) and hidden exceptions (Volume 1, page 271) in particular:
0:000> !teb
TEB at 7ffa9000
ExceptionList: 01c0fc04
StackBase: 01c10000
StackLimit: 01c08000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffa9000
EnvironmentPointer: 00000000
ClientId: 0000486c . 00005704
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffd9000
LastErrorValue: 0
LastStatusValue: c0000010
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 01c08000 01c10000
01c08000 00000000
01c08004 00000000
01c08008 00000000
[...]
01c0e704 00001000
01c0e708 01c0ea14
01c0e70c 7c82855e ntdll!KiUserExceptionDispatcher+0xe
01c0e710 01c0a000
01c0e714 01c0e73c
01c0e718 01c0e720
01c0e71c 01c0e73c
01c0e720 c0000005
01c0e724 00000000
01c0e728 00000000
01c0e72c 7c82a756 ntdll!RtlpCoalesceFreeBlocks+0×370
01c0e730 00000002
01c0e734 00000000
[...]After applying the exception context we see Heap Corruption patern (Volume 1, page 257):
0:000> .cxr01c0e73ceax=01998db0 ebx=00370000 ecx=01998db0 edx=4d445443 esi=01998da8 edi=01998010 eip=7c82a756 esp=01c0ea08 ebp=01c0ea14 iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246ntdll!RtlpCoalesceFreeBlocks+0×370:7c82a756 3b4a04 cmp ecx,dword ptr [edx+4] ds:0023:4d445447=????????
However, stack trace is a bit incorrect (Volume 1, page 288) at the end (shown in bold italics) but this can be explained as a residual stack trace that happened in the past with some data overwritten by exception processing:
0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
01c0ea14 7c82a82b ntdll!RtlpCoalesceFreeBlocks+0x370
01c0eafc 77bbcef6 ntdll!RtlFreeHeap+0x38e
01c0eb44 00421fdc msvcrt!free+0xc3
01c0eb6c 00422395 App!SetData+0x3a5
[...]
01c0f81c 7c829f60 App!CreatePrinter+0x6a
01c0f824 7c829f59 ntdll!CheckHeapFillPattern+0×64
01c0f948 7c82770b ntdll!RtlFreeHeap+0×70f
01c0f96c 77c27631 ntdll!NtQueryValueKey+0xc
01c0fab8 00000000 gdi32!ReadDisableMetaFilesRegKey+0xfcBecause the hang happens after printing we detect old printer drivers in the module list (certainly not my version of the environment used for application testing, Volume 2, page 299):
0:000> lmt start end module name [...] 6b900000 6b925000 PrinterDriverA [...]20026c900000 6c947000 PrinterDriverB [...]2002[...]
One application was constantly crashing in a highly dynamic multi-user environment with various value-added GUI-enhancing hooking and patching 3rd-party products. The dump was analyzed and it shows the data NULL pointer access violation (page 56):
0:000> r eax=05503ff0 ebx=05503f10 ecx=6bb7da25 edx=00000000 esi=02203c10edi=00000000eip=6b542ba3 esp=04f6f860 ebp=04f6f87c iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246 gui_plugin!entry+0xe63: 6b542ba3 8b0f mov ecx,dword ptr [edi] ds:0023:00000000=????????
Unfortunately, due to the lack of gui_plugin symbols the stack trace is incorrect (Volume 1, page 288):
0:000> kv ChildEBP RetAddr Args to Child WARNING: Stack unwind information not available. Following frames may be wrong. 04f6f87c 00000000 000003b9 00000690 00000d20 gui_plugin!entry+0xe63
We do not attempt stack trace reconstruction here (Volume 1, page 157) but show one of troubleshooting methods to eliminate possible effects of the changed environment (Volume 1, page 283). We look for any code patching modules by examining hooked functions (Volume 1, page 468):
0:000> !chkimg -lo 50 -d !kernel32 -v Searching for module with expression: !kernel32 Will apply relocation fixups to file used for comparison Will ignore NOP/LOCK errors Will ignore patched instructions Image specific ignores will be applied Comparison image path: c:symdownstreamkernel32.dll4791A76Ddb000kernel32.dll No range specified
Scanning section: .text
Size: 835001
Range to scan: 76701000-767ccdb9
7670a672-7670a676 5 bytes - kernel32!MoveFileW
[ 8b ff 55 8b ec:e9 89 59 80 09 ]
7671c5c8-7671c5cc 5 bytes - kernel32!DeleteFileW (+0×11f56)
[ 8b ff 55 8b ec:e9 33 3a 80 09 ]
76721070-76721074 5 bytes - kernel32!MoveFileExW (+0×4aa8)
[ 8b ff 55 8b ec:e9 8b ef 83 09 ]
7678ff3f-7678ff43 5 bytes - kernel32!ReplaceFileW (+0×6eecf)
[ 68 b4 03 00 00:e9 bc 00 7a 09 ]
Total bytes compared: 835001(100%)
Number of errors: 20
20 errors : !kernel32 (7670a672-7678ff43)
0:000> u 7670a672
kernel32!MoveFileW:
7670a672 e989598009 jmp 7ff10000
7670a677 6a02 push 2
7670a679 6a00 push 0
7670a67b 6a00 push 0
7670a67d ff750c push dword ptr [ebp+0Ch]
7670a680 ff7508 push dword ptr [ebp+8]
7670a683 e8c4690100 call kernel32!MoveFileWithProgressW (7672104c)
7670a688 5d pop ebp
0:000> u 7ff10000
7ff10000 e99b1e94f5 jmp 3rdPartyHook!MoveFileW (75851ea0)
7ff10005 8bff mov edi,edi
7ff10007 55 push ebp
7ff10008 8bec mov ebp,esp
7ff1000a e968a67ff6 jmp kernel32!MoveFileW+0×5 (7670a677)
7ff1000f 0000 add byte ptr [eax],al
7ff10011 0000 add byte ptr [eax],al
7ff10013 0000 add byte ptr [eax],alWe see lots of references to 3rdPartyHook on the thread raw stack:
0:000> !teb
TEB at 7ffd5000
ExceptionList: 04f6ff70
StackBase: 04f70000
StackLimit: 04f6c000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffd5000
EnvironmentPointer: 00000000
ClientId: 00001790 . 000017d0
RpcHandle: 00000000
Tls Storage: 7ffd502c
PEB Address: 7ffd8000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 04f6c000 04f70000
[...]
04f6cf28 00440042
04f6cf2c 04f6d2d4
04f6cf30 77179834 ntdll!_except_handler4
04f6cf34 003ce48e
04f6cf38 fffffffe
04f6cf3c 7719d584 ntdll!LdrpResSearchResourceInsideDirectory+0x80f
04f6cf40 7719cf9d ntdll!LdrpResSearchResourceMappedFile+0x521
04f6cf44 6b920002
04f6cf48 00000000
04f6cf4c 000bf000
04f6cf50 6b9d9000
04f6cf54 6b9200f8
04f6cf58 00000000
04f6cf5c 01f6d1d4 <Unloaded_D3D9.DLL>+0x12d1d4
04f6cf60 00000003
04f6cf64 04f6cf88
04f6cf68 04f6d370
04f6cf6c 04f6d340
04f6cf70 771d8fb2 ntdll!_SEH_epilog4_GS+0xa
04f6cf74 7719d018 ntdll!LdrpResSearchResourceMappedFile+0x781
04f6cf78 73d77612
04f6cf7c 6b920002
04f6cf80 000bf000
04f6cf84 00000000
04f6cf88 00000001
04f6cf8c 00000000
04f6cf90 00000000
04f6cf94 00000000
04f6cf98 00000000
04f6cf9c 04f6d44c
04f6cfa0 75851c2f 3rdPartyHook+0×1c2f
04f6cfa4 04f6d1e8
04f6cfa8 0000000004f6cfac 00000000 04f6cfb0 00000000 [...]
The symbolic references point to valid and sound code:
0:000> ub 75851c2f 3rdPartyHook!GetData+0x22d: 75851c1d c20c00 ret 0Ch 75851c20 8b4dfc mov ecx,dword ptr [ebp-4] 75851c23 5f pop edi 75851c24 5e pop esi 75851c25 33cd xor ecx,ebp 75851c27 33c0 xor eax,eax 75851c29 5b pop ebx 75851c2a e8590f0000 call 3rdPartyHook!__security_check_cookie (75852b88) 0:000> u 75851c2f 3rdPartyHook!GetData+0x23f: 75851c2f 8be5 mov esp,ebp 75851c31 5d pop ebp 75851c32 c20c00 ret 0Ch 75851c35 cc int 3 75851c36 cc int 3 75851c37 cc int 3 75851c38 cc int 3 75851c39 cc int 3
We also see the other 4th-party module that we know as patching from our past experience:
[...]
04f6f850 00000000
04f6f854 00000000
04f6f858 00000000
04f6f85c 00000000
04f6f860 1000c200 4thPartyHook!Shared+0×80
04f6f864 00000000
04f6f868 04f6f87c
04f6f86c 00000001
04f6f870 00000000
04f6f874 00000000
04f6f878 00000000
04f6f87c 0000006c
[...]The address 1000c200 looks suspicious and coincidental (Volume 1, page 390) as the set of flags. However, when we disassemble the address, we see the valid and sound code and the module seems to be loaded at 0×10000000 address:
0:000> ub 1000c200 4thPartyHook!Shared+0x80: 1000c1e8 e8234effff call 10001010 1000c1ed a160020710 mov eax,dword ptr [10070260] 1000c1f2 83c418 add esp,18h 1000c1f5 8b750c mov esi,dword ptr [ebp+0Ch] 1000c1f8 85f6 test esi,esi 1000c1fa 7530 jne 1000c22c 1000c1fc 85c0 test eax,eax 1000c1fe 742c je 1000c22c
0:000> u 1000c200
4thPartyHook!Shared+0x80:
1000c200 8b08 mov ecx,dword ptr [eax]
1000c202 8b5004 mov edx,dword ptr [eax+4]
1000c205 53 push ebx
1000c206 8bd9 mov ebx,ecx
1000c208 0bda or ebx,edx
1000c20a 5b pop ebx
1000c20b 741f je 1000c22c
1000c20d f6400801 test byte ptr [eax+8],1
0:000> !dh 10000000
[...]
File Type: DLL
FILE HEADER VALUES
14C machine (i386)
5 number of sections
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL
[...]
OPTIONAL HEADER VALUES
10B magic #
8.00 linker version
62000 size of code
15000 size of initialized data
0 size of uninitialized data
3E1B6 address of entry point
1000 base of code
----- new -----
10000000 image base
1000 section alignment
1000 file alignment
2 subsystem (Windows GUI)
4.00 operating system version
0.00 image version
4.00 subsystem version
7A000 size of image
1000 size of headers
[...]
0:000> lm
start end module name
00400000 00425000 App
[...]
10000000 1007a000 4thPartyHook
[...]The recommendation we usually give in such cases it to remove or disable 3rd- and 4th-party products that do code patching to see if this resolves the problem. This makes the module list internally clean and if the problem persists then we should look for other causes and restore 3rd- and 4th-party products.
This is a synthesized dump analysis of many similar print spooler crashes in multi-user terminal service environments where old printer drivers are used that were tested only in single-user environments or insufficiently tested in multi-threaded environments. Many such crashes result from dynamic memory corruption of a process heap (Volume 1, page 257):
(40dc.4278): Access violation - code c0000005 (!!! second chance !!!) eax=00000079 ebx=00000001 ecx=0008bff8 edx=00107898 esi=07d7522a edi=00107890 eip=7c8199b2 esp=0155fc14 ebp=0155fc44 iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202ntdll!RtlpLowFragHeapFree+0×30:7c8199b2 8b4604 mov eax,dword ptr [esi+4] ds:0023:07d7522e=???????? 0:017> kL ChildEBP RetAddr0155fc44 7c819770 ntdll!RtlpLowFragHeapFree+0×30 0155fd1c 77c87a2b ntdll!RtlFreeHeap+0×5c0155fd30 77c87a02 RPCRT4!FreeWrapper+0×1e 0155fd3c 77c821c2 RPCRT4!operator delete+0xd 0155fd50 77c8047b RPCRT4!LRPC_SCALL::FreeBuffer+0×77 0155fd9c 77c80353 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×192 0155fdc0 77c811dc RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3 0155fdfc 77c812f0 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×42c 0155fe20 77c88678 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127 0155ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430 0155ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd 0155ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d 0155ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b 0155ffec 00000000 kernel32!BaseThreadStart+0×34
Although any module could corrupt the heap and either Gflags or Application Verifier is recommended to enable full page heap, sometimes we need to point to some print drivers to eliminate or upgrade them in the meantime. When there are many of them (Module Variety pattern, Volume 1, page 310) we can point to the oldest one:
0:017> lmt
start end module name
[...]
010d0000 010d8000 PrintDriver1 2007
01260000 01272000 PrintDriver2 1999
01290000 012da000 PrintDriver3 2009
012f0000 01302000 PrintDriver4 2003
01310000 01320000 PrintDriver5 2004
01320000 01332000 PrintDriver6 2004
01340000 01353000 PrintDriver7 2005
01360000 0139e000 PrintDriver8 2007
013b0000 013c3000 PrintDriver9 2004
013d0000 013e0000 PrintDriver10 2005
013e0000 013f3000 PrintDriver11 2005
01400000 01413000 PrintDriver12 2006
01420000 0146b000 PrintDriver13 2007
01480000 01488000 PrintDriver14 2003
017f0000 0181e000 PrintDriver15 2004
01920000 0192d000 PrintDriver16 2008
01930000 01936000 PrintDriver17 2008
01950000 01959000 PrintDriver18 2008
01960000 01969000 PrintDriver19 2008
01f80000 021e8000 PrintDriver20 2004
032f0000 03514000 PrintDriver21 2003
03cd0000 03cd6000 PrintDriver22 2008
32100000 32148000 PrintDriver23 2008
3ea40000 3ea46000 PrintDriver24 2007
3f000000 3f03d000 PrintDriver25 2009
3f100000 3f133000 PrintDriver26 2009
[...]The age distribution among 121 modules can be visualized on a CAD diagram (Volume 2, page 126):
Alternatively we can look at the execution residue (Volume 2, page 239) on a raw thread stack data:
0:017> !teb
TEB at 7ffa9000
ExceptionList: 0155fd0c
StackBase: 01560000
StackLimit: 01550000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffa9000
EnvironmentPointer: 00000000
ClientId: 000040dc . 00004278
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffd8000
LastErrorValue: 0
LastStatusValue: 8000001a
Count Owned Locks: 0
HardErrorMode: 0
0:017> dds 01550000 01560000
01550000 00000000
01550004 00000000
[...]
01554e78 00000000
01554e7c 00000000
01554e80 01554ecc
01554e84 7c82d1bb ntdll!RtlFindActivationContextSectionString+0xe1
01554e88 01554ea4
01554e8c 01554efc
01554e90 00000000
01554e94 020a0018 PrintDriver20!Callback+0×5c88
01554e98 7ffa9c00
01554e9c 00000000
01554ea0 01554ed4
01554ea4 7c82dd6c ntdll!RtlEncodeSystemPointer+0×45b
01554ea8 00020000
01554eac 01554ec8
01554eb0 01554ec8
01554eb4 01554ec8
[...]
01555eb4 0040003e
01555eb8 01556854
01555ebc 00000000
01555ec0 0000003e
01555ec4 0208003e PrintDriver20!GetValue+0xb37fe
01555ec8 00000000
01555ecc 43000000
01555ed0 0000003e
01555ed4 01555ffa
01555ed8 01555fbc01555edc 01555fa8 01555ee0 001190d8 01555ee4 7c81990d ntdll!RtlpLowFragHeapAlloc+0×210 01555ee8 7c819962 ntdll!RtlpLowFragHeapAlloc+0xc6a 01555eec 0008bff8 01555ef0 00000000 01555ef4 00080000 [...]
The first address 020a0018 seems to be coincidental (Volume 1, page 390) because its disassembled code is not good:
0:017> ub 020a0018
^ Unable to find valid previous instruction for 'ub
020a0018'
0:017> u 020a0018
PrintDriver20!Callback+0x5c88:
020a0018 048b add al,8Bh
020a001a c7 ???
020a001b ebe8 jmp PrintDriver20!Callback+0×5c75 (020a0005)
020a001d 8d4e28 lea ecx,[esi+28h]
020a0020 e8d9960100 call PrintDriver20!DlgProc+0×86ee (020b96fe)
020a0025 8d4e28 lea ecx,[esi+28h]
020a0028 e87b930100 call PrintDriver20!DlgProc+0×8398 (020b93a8)
020a002d 8b4618 mov eax,dword ptr [esi+18h]However, the second address code 0208003e seems sound: cmp is followed by jne:
0:017> ub 0208003e
PrintDriver20!GetValue+0xb37e5:
02080025 8d442414 lea eax,[esp+14h]
02080029 8b4b2c mov ecx,dword ptr [ebx+2Ch]
0208002c 50 push eax
0208002d e8ce3a0000 call PrintDriver20!GetValue+0xb72c0 (02083b00)
02080032 8b38 mov edi,dword ptr [eax]
02080034 8b4c2410 mov ecx,dword ptr [esp+10h]
02080038 8b41fc mov eax,dword ptr [ecx-4]
0208003b 3947fc cmp dword ptr [edi-4],eax0:017> u 0208003e
PrintDriver20!GetValue+0xb37fe:
0208003e 751b jne PrintDriver20!GetValue+0xb381b (0208005b)But this is not a function call that resulted in saved return address so we can still consider it as a coincidence. However, in the raw stack data we also see a large chunk of ASCII data pointing to the same driver in a textual form:
[...] 0155d360 6f742064 0155d364 696e6920 0155d368 6c616974 0155d36c 20657a69 0155d370 61636562 0155d374 20657375 0155d378 75732061 0155d37c 62617469 [...] 0:017> da 0155d360 0155d734 0155d360"d to initialize because a suitab"0155d380"lePrinterDriver20inf file was[...]"
This reinforces our belief in PrinterDriver20. Finally, when looking at critical section list (Volume 2, page 321), we see corruption signs pointing to the same driver addresses:
0:017> !cs -l -o -s DebugInfo = 0x0014bc60 Critical section = 0x020f7140 (PrintDriver20!DlgProc+0×46130) LOCKED LockCount = 0xFF85EA7F WaiterWoken = Yes OwningThread = 0×8b0c244c RecursionCount = 0×8BFFFBB4 LockSemaphore = 0×83182444 SpinCount = 0×088908c4 WARNING: critical section DebugInfo = 0x00000008 doesn't point back to the DebugInfo found in the active critical sections list = 0x0014bc60. The critical section was probably reused without calling DeleteCriticalSection. Cannot read structure field value at 0x0000000a, error 0 ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled. ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled. DebugInfo = 0x0014bc88 Critical section = 0x020f7110 (PrintDriver20!DlgProc+0×46100) LOCKED LockCount = 0×1E7245FF WaiterWoken = No OwningThread = 0xccccc304 RecursionCount = 0xC483FFFD LockSemaphore = 0×158638B9 SpinCount = 0xff96e902 WARNING: critical section DebugInfo = 0x0f712068 doesn't point back to the DebugInfo found in the active critical sections list = 0x0014bc88. The critical section was probably reused without calling DeleteCriticalSection. Cannot read structure field value at 0x0f71206a, error 0 [...]
A server was hanging and a complete memory dump was generated. Usual checks for resource contention or CPU spikes like !locks and !running WinDbg commands didn't reveal anything. Therefore we decided to list all threads in the system using !stacks and !process 0 ff commands (Volume 1, page 409). The former command gives a bird eye overview of threads and it was found suspicious that many threads in different processes had 0 ticks waiting time and had this top level call:
0: kd> !stacks Proc.Thread .Thread Ticks ThreadState Blocker [...] [88d9cd88 explorer.exe] Proc.Thread .Thread Ticks ThreadState Blocker 1ecc.001eb4 88d99db00000000Blocked3rdPartyAVDriver+0×1394[...] [862bad88 userinit.exe] 4394.005570 878f07900000000Blocked3rdPartyAVDriver+0×1394
Inspection of the latter command detailed output of thread stacks revealed that all these suspicious threads were in fact blocked (Volume 2, page 184) when loading a DLL:
0: kd> .process /r /p 862bad88 Implicit process is now 862bad88 Loading User Symbols 0: kd> .thread 878f0790 Implicit thread is now 878f0790 0: kd> kL 100 ChildEBP RetAddr 9aeb529c 80833e95 nt!KiSwapContext+0x26 9aeb52c8 80829027 nt!KiSwapThread+0x2e5 9aeb5310 b8dff394 nt!KeDelayExecutionThread+0x2ab WARNING: Stack unwind information not available. Following frames may be wrong.9aeb5338 b8dffe35 3rdPartyAVDriver+0×1394 9aeb5354 b8e00208 3rdPartyAVDriver+0×1e35 9aeb5398 b8e0e45a 3rdPartyAVDriver+0×2208 9aeb53d0 8081e087 3rdPartyAVDriver+0×1045a9aeb53e4 b7d6e41b nt!IofCallDriver+0×459aeb53f4 b7d6a3ee DriverA!Dispatch+0xfe 9aeb5404 8081e087 DriverA!KUI_dispatch+0×6d
9aeb5418 b8bdb4e1 nt!IofCallDriver+0×45
9aeb543c b8bdb755 DriverB!PassThrough+0xd3
9aeb545c 8081e087 DriverB!Dispatch+0×153The functions shown in bold italics are known from past issues to be pass through (page 129) forwarding IRP to lower drivers in a device driver stack. Actually the thread stack above belongs to the main thread (Volume 1, page 436) blocked while loading application DLLs. The conclusion was to check 3rdPartyAVDriver.sys vendor.
When looking at a stack trace of one crashed process we notice an invalid code pointer (Volume 1, page 267). It is not a NULL code pointer (Volume 1, page 237) but has the same stack trace pattern (Volume 1, page 395):
0:000> kL
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0013dfb4 00401791 0×5e388583
0013fdf4 0040189f Application!RequestData+0×3e1
0013fee4 00401d0a Application!main+0×3f
0013ffc0 77e4f23b Application!mainCRTStartup+0×16c
0013fff0 00000000 kernel32!BaseProcessStart+0×23When we look at raw stack data and examine the backward disassembly of the return address we see that invalid code was called from RequestData function and WinDbg stack trace reconstruction is not suspicious (it is structurally and semantically correct, Volume 1, page 288):
0:000> dds esp l10 0013dfb8 00401791 Application!RequestData+0x3e1 0013dfbc 00000140 0013dfc0 0013ee50 0013dfc4 00000fa4 0013dfc8 00000000 0013dfcc 00000000 0013dfd0 00000ece 0013dfd4 0013ffc0 0013dfd8 7ffdc000 0013dfdc 00000140 0013dfe0 0000054c 0013dfe4 50000002 0013dfe8 4b4919ac 0013dfec 00000000 0013dff0 00000000 0013dff4 003f003c 0:000> .asm no_code_bytes Assembly options: no_code_bytes
0:000> ub 00401791
Application!RequestData+0x3c8:
00401778 Application!RequestData+0x3d0 (00401780)
0040177a lea ebx,[ebx]
00401780 push 0
00401782 push eax
00401783 lea ecx,[esp+esi+0E30h]
0040178a push ecx
0040178b push edi
0040178c call Application!recv (0040e382)When seeing recv call we might suspect that the crash happened just inside that function because the raw stack data upwards (lower addresses) doesn't have any execution residue (Volume 2, page 239) left from nested function calls:
0:000> dds esp-100 esp 0013deb8 00000000 0013debc 00000000 0013dec0 00000000 0013dec4 00000000 0013dec8 00000000 0013decc 00000000 0013ded0 00000000 0013ded4 00000000 0013ded8 00000000 0013dedc 00000000 0013dee0 00000000 0013dee4 00000000 0013dee8 00000000 0013deec 00000000 0013def0 00000000 0013def4 00000000 0013def8 00000000 0013defc 00000000 0013df00 00000000 0013df04 00000000 0013df08 00000000 0013df0c 00000000 0013df10 00000000 0013df14 00000000 0013df18 00000000 0013df1c 00000000 0013df20 00000000 0013df24 00000000 0013df28 00000000 0013df2c 00000000 0013df30 00000000 0013df34 00000000 0013df38 00000000 0013df3c 00000000 0013df40 00000000 0013df44 00000000 0013df48 00000000
0013df4c 00000000 0013df50 00000000 0013df54 00000000 0013df58 00000000 0013df5c 00000000 0013df60 00000000 0013df64 00000000 0013df68 00000000 0013df6c 00000000 0013df70 00000000 0013df74 00000000 0013df78 00000000 0013df7c 00000000 0013df80 00000000 0013df84 00000000 0013df88 00000000 0013df8c 00000000 0013df90 00000000 0013df94 00000000 0013df98 00000000 0013df9c 00000000 0013dfa0 00000000 0013dfa4 00000000 0013dfa8 00000000 0013dfac 00000000 0013dfb0 00000000 0013dfb4 00000000 0013dfb8 00401791 Application!RequestData+0x3e1
So we follow recv call forward disassembly (and notice that the first jump is indirect):
0:000> u 0040e382
Application!recv:
0040e382 jmp dword ptr[Application!_imp__recv (00410180)]
Application!closesocket:
0040e388 jmp dword ptr [Application!_imp__closesocket (00410170)]
Application!WSAGetLastError:
0040e38e jmp dword ptr [Application!_imp__WSAGetLastError (00410174)]
Application!send:
0040e394 jmp dword ptr[Application!_imp__send (00410178)]
Application!connect:
0040e39a jmp dword ptr [Application!_imp__connect (0041017c)]
Application!htons:
0040e3a0 jmp dword ptr [Application!_imp__htons (00410198)]
Application!setsockopt:
0040e3a6 jmp dword ptr [Application!_imp__setsockopt (00410184)]
Application!socket:
0040e3ac jmp dword ptr [Application!_imp__socket (00410188)]0:000> dps00410180l1000410180 71ad2f7f ws2_32!recv00410184 71ad2d47 ws2_32!setsockopt 00410188 71ad410c ws2_32!socket 0041018c 71ad7ca1 ws2_32!gethostbyname 00410190 71ad4f3b ws2_32!WSAStartup 00410194 71ad7b5b ws2_32!gethostname 00410198 71ad28bc ws2_32!htons 0041019c 71ad3da8 ws2_32!WSACleanup 004101a0 00000000 004101a4 00000000 004101a8 00000000 004101ac 00000000 004101b0 00000000 004101b4 45cd184e 004101b8 00000000 004101bc 00000002 0:000> u 71ad2f7fws2_32!recv: 71ad2f7f jmp 7fd6000071ad2f84 sub esp,10h 71ad2f87 push ebx 71ad2f88 xor ebx,ebx 71ad2f8a cmp dword ptr [ws2_32!PrologPointer (71ae4044)],offset ws2_32!Prolog_v2 (71ad6067) 71ad2f94 push esi 71ad2f95 je ws2_32!recv+0×18 (71ad6207) 71ad2f9b lea eax,[ebp-8] 0:000> u 7fd600007fd60000 jmp7fd60005 mov edi,edi 7fd60007 push ebp 7fd60008 mov ebp,esp 7fd6000a jmp ws2_32!recv+0×5 (71ad2f84) 7fd6000f add byte ptr [eax],al 7fd60011 add byte ptr [eax],al 7fd60013 add byte ptr [eax],alDllA!recv_patch (612101b6)
Finally we see that sockets library functions were patched (Volume 1, page 468) by a 3rd-party module DllA and we need to contact its vendor.
The following WER dialog appeared after an RTF text was copied from an e-mail client to IE while editing another blog post:
Although the dialog points to html.iec, it is good to show the basic techniques for component identification using WinDbg. We took a manual user dump (Volume 1, page 486) of that unresponsive process while it was showing the dialog above using Task Manager (Volume1, page 39):
The saved process memory dump shows that IE was a virtualized 32-bit process running on x64 Windows (Volume 1, page 400):
STACK_TEXT: 00000000`000dee18 00000000`75bbab46 :wow64cpu!TurboDispatchJumpAddressEnd+0xef00000000`000deec0 00000000`75bba14c :wow64!Wow64SystemServiceEx+0×27a00000000`000deef0 00000000`778f52d3 :wow64!Wow64LdrpInitialize+0×4b400000000`000df450 00000000`778f5363 : ntdll!CsrClientConnectToServer+0×493 00000000`000df700 00000000`778e85ce : ntdll!CsrClientConnectToServer+0×523 00000000`000df7b0 00000000`00000000 : ntdll!LdrInitializeThunk+0xe
However, even after switching to x86 mode, !analyze -v doesn't report any signs of exception violation, c0000005, seen from WER dialog above:
0:000> .load wow64exts; .effmach x86 Effective machine: x86 compatible (x86)
0:000:x86> !analyze -v
[...]
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint
has been reached.
STACK_TEXT:
0027e428 7679e91a ntdll_77a60000!NtWaitForMultipleObjects+0x15
0027e4c4 76a58f76 kernel32!WaitForMultipleObjectsEx+0x11d
0027e518 75ad6071 user32!RealMsgWaitForMultipleObjectsEx+0x14d
WARNING: Frame IP not in any known module. Following frames may be wrong.
0027e538 75ad61f0 ieui+0x6071
0027e560 75ad6196 ieui+0x61f0
0027e59c 7291ffc6 ieui+0x6196
0027e5a8 7290f579 ieframe!BrowserThreadProc+0x3f
0027e5cc 7290f4c7 ieframe!BrowserNewThreadProc+0x7b
0027f63c 728fd1ba ieframe!SHOpenFolderWindow+0x188
0027f86c 013133c3 ieframe!IEWinMain+0x2d9
0027fcb0 0131325a iexplore!wWinMain+0x27b
0027fd44 7680e4a5 iexplore!_initterm_e+0x1b1
0027fd50 77adcfed kernel32!BaseThreadInitThunk+0xe
0027fd90 77add1ff ntdll_77a60000!__RtlUserThreadStart+0x23
0027fda8 00000000 ntdll_77a60000!_RtlUserThreadStart+0x1bWe list all threads (Volume 1, page 409) for any exception processing signs (Volume 1, page 255) and we find one such thread indeed:
0:002:x86> ~*kc 100
[...]
2 Id: 404.ba4 Suspend: 0 Teb: 7efaa000 Unfrozen
ntdll_77a60000!NtWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!UnhandledExceptionFilter
ntdll_77a60000!__RtlUserThreadStart
ntdll_77a60000!_EH4_CallFilterFunc
ntdll_77a60000!_except_handler4
ntdll_77a60000!ExecuteHandler2
ntdll_77a60000!ExecuteHandler
ntdll_77a60000!KiUserExceptionDispatcherWARNING: Frame IP not in any known module. Following frames may be wrong. 0×0 html!LwMultDivRU html!FMarkListCallback html!JcCalcFromXaExtents html!EmitNonBreakingSpace html!FEmitHtmlFnOtag html!ConvertRtfToForeign html!FceRtfToForeign html!RtfToForeign32 mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml mshtmled!CHTMLEditor::ConvertRTFToHTML mshtmled!CPasteCommand::PasteFromClipboard mshtmled!CPasteCommand::PrivateExec mshtmled!CCommand::Exec mshtmled!CMshtmlEd::Exec mshtml!CEditRouter::ExecEditCommand mshtml!CDoc::ExecHelper mshtml!CFrameSite::Exec mshtml!CDoc::RouteCTElement mshtml!CDoc::ExecHelper mshtml!CDoc::Exec mshtml!CDoc::OnCommand mshtml!CDoc::OnWindowMessage mshtml!CServer::WndProc user32!InternalCallWinProc user32!UserCallWinProcCheckWow user32!SendMessageWorker user32!SendMessageW mshtml!CElement::PerformTA mshtml!CDoc::PerformTA mshtml!CDoc::PumpMessage mshtml!CDoc::DoTranslateAccelerator mshtml!CServer::TranslateAcceleratorW mshtml!CDoc::TranslateAcceleratorW ieframe!CProxyActiveObject::TranslateAcceleratorW ieframe!CDocObjectView::TranslateAcceleratorW ieframe!CCommonBrowser::v_MayTranslateAccelerator ieframe!CShellBrowser2::_MayTranslateAccelerator ieframe!CShellBrowser2::v_MayTranslateAccelerator ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll_77a60000!__RtlUserThreadStart ntdll_77a60000!_RtlUserThreadStart [...]
The same stack fragment with function parameters has a zero for UnhandledExceptionFilter first parameter, perhaps because of optimized code (Volume 1, page 265) reusing local variable slots, and we cannot apply .exptr meta-command (Volume 2, page 75):
0:002:x86> ~2kv ChildEBP RetAddr Args to Child 0361d938 7679e91a 00000002 0361d988 00000001 ntdll_77a60000!NtWaitForMultipleObjects+0x15 0361d9d4 767949d9 0361d988 0361da24 00000000 kernel32!WaitForMultipleObjectsEx+0x11d 0361d9f0 7684573d 00000002 0361da24 00000000 kernel32!WaitForMultipleObjects+0x18 0361da5c 76845969 0361db2c 00000001 00000001 kernel32!WerpReportFaultInternal+0x16d 0361da70 7681c66f 0361db2c 00000001 35137a6c kernel32!WerpReportFault+0x70 0361dafc 77add03e0000000077aaf2d0 00000000 kernel32!UnhandledExceptionFilter+0×1b5 0361db04 77aaf2d0 00000000 0361fe04 77a8da38 ntdll_77a60000!__RtlUserThreadStart+0×6f 0361db18 77b129b3 00000000 00000000 00000000 ntdll_77a60000!_EH4_CallFilterFunc+0×12 0361db40 77a83099 fffffffe 0361fdf4 0361dc7c ntdll_77a60000!_except_handler4+0×8e 0361db64 77a8306b 0361dc2c 0361fdf4 0361dc7c ntdll_77a60000!ExecuteHandler2+0×26 0361dc14 77a82eff 0161dc2c 0361dc7c 0361dc2c ntdll_77a60000!ExecuteHandler+0×24 0361dc18 0161dc2c 0361dc7c 0361dc2c 0361dc7c ntdll_77a60000!KiUserExceptionDispatcher+0xf WARNING: Frame IP not in any known module. Following frames may be wrong.0361df5c 72333b0e 07feaee4 0000001f 00000000 0×161dc2c0361df74 72332ba9 00000000 0361dffc 00000000 html!LwMultDivRU+0×4b6 0361dfb8 7233518a 0361dffc 0000000c 0361e074 html!FMarkListCallback+0×56c 0361dfc8 723368b7 0000001f 0361dffc 0000000f html!JcCalcFromXaExtents+0×91 0361e074 723370c5 0000000f 00000000 000007d0 html!EmitNonBreakingSpace+0×445 0361e1a0 723407ff 00000000 0000000f 00000104 html!FEmitHtmlFnOtag+0×17d 0361e1c4 7231c6a8 00000001 00000000 00000000 html!ConvertRtfToForeign+0×105 0361e64c 7231c745 00ed03bc 00000000 00ed0374 html!FceRtfToForeign+0×266
Also the transition from html.iec module to exception processing code is through an invalid address 0×161dc2c so we might guess that this was an instance of a wild code pointer (Volume 2, page 219) or the case of an incorrect stack trace (Volume 1, page 288). However, by using techniques to get exception context from hidden exceptions (Volume 1, page 271) we get the following stack trace:
0:002:x86> .cxr 0361dc7c
eax=0000004e ebx=09d5755f ecx=07feaee4 edx=07feaf08 esi=07feaee4
edi=00000000
eip=7233d50b esp=0361df48 ebp=0361df5c iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
html!FPseudoStyleBis+0×26:
7233d50b 0fb70f movzx ecx,word ptr [edi] ds:002b:00000000=????0:002:x86> kL 100 ChildEBP RetAddr 0361df50 7233d597 html!FPseudoStyleBis+0x26 0361df5c 72333b0e html!BisFromLpxszStyle+0x1c 0361df74 72332ba9 html!LwMultDivRU+0x4b6 0361dfb8 7233518a html!FMarkListCallback+0x56c 0361dfc8 723368b7 html!JcCalcFromXaExtents+0x91 0361e074 723370c5 html!EmitNonBreakingSpace+0x445 0361e1a0 723407ff html!FEmitHtmlFnOtag+0x17d 0361e1c4 7231c6a8 html!ConvertRtfToForeign+0x105 0361e64c 7231c745 html!FceRtfToForeign+0x266 0361e674 744e5ad4 html!RtfToForeign32+0x51 0361eabc 744e5c83 mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml+0x163 0361ef10 7449cc15 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x11a 0361ef2c 7449cd81 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x38 0361ef40 744cdcea mshtmled!CHTMLEditor::ConvertRTFToHTML+0x12 0361efac 744ce392 mshtmled!CPasteCommand::PasteFromClipboard+0x2c0 0361f01c 7448d218 mshtmled!CPasteCommand::PrivateExec+0x47a 0361f040 7448d1ad mshtmled!CCommand::Exec+0x4b 0361f064 74000d14 mshtmled!CMshtmlEd::Exec+0xf9 0361f094 73ff88a8 mshtml!CEditRouter::ExecEditCommand+0xd6 0361f43c 7415eccf mshtml!CDoc::ExecHelper+0x338d 0361f488 73ff8a2f mshtml!CFrameSite::Exec+0x264 0361f4bc 73ff87af mshtml!CDoc::RouteCTElement+0xf1 0361f854 73ff8586 mshtml!CDoc::ExecHelper+0x325e 0361f874 740a0e7b mshtml!CDoc::Exec+0x1e 0361f8ac 7401a708 mshtml!CDoc::OnCommand+0x9c 0361f9c0 73f297e1 mshtml!CDoc::OnWindowMessage+0x841 0361f9ec 76a58807 mshtml!CServer::WndProc+0x78 0361fa18 76a58962 user32!InternalCallWinProc+0x23 0361fa90 76a5c4b6 user32!UserCallWinProcCheckWow+0x109 0361fad4 76a5c517 user32!SendMessageWorker+0x55b 0361faf8 7400fb9b user32!SendMessageW+0x7f 0361fb24 73f68e5a mshtml!CElement::PerformTA+0x71 0361fb44 73f68db9 mshtml!CDoc::PerformTA+0xd8 0361fbc0 73ff381c mshtml!CDoc::PumpMessage+0x8e0 0361fc74 73ff3684 mshtml!CDoc::DoTranslateAccelerator+0x33f 0361fc90 73ff34cc mshtml!CServer::TranslateAcceleratorW+0x56 0361fcb0 7296f550 mshtml!CDoc::TranslateAcceleratorW+0x83 0361fccc 7296f600 ieframe!CProxyActiveObject::TranslateAcceleratorW+0x30 0361fcf0 7296fca1 ieframe!CDocObjectView::TranslateAcceleratorW+0xb1 0361fd10 7296faf4 ieframe!CCommonBrowser::v_MayTranslateAccelerator+0xda 0361fd3c 7296f7b0 ieframe!CShellBrowser2::_MayTranslateAccelerator+0x68 0361fd4c 7296f7f5 ieframe!CShellBrowser2::v_MayTranslateAccelerator+0x15 0361fdb8 7680e4a5 ieframe!CTabWindow::_TabWindowThreadProc+0x264 0361fdc4 77adcfed kernel32!BaseThreadInitThunk+0xe 0361fe04 77add1ff ntdll_77a60000!__RtlUserThreadStart+0x23 0361fe1c 00000000 ntdll_77a60000!_RtlUserThreadStart+0x1b
Then we can use lmv command to see the component version:
0:002:x86> lmv m html
start end module name
722f0000 7236f000 html # (pdb
symbols) c:msshtml.pdb34A9E6645AEA4E6A83EA51D2849C731C1html.pdb
Loaded symbol image file: html.iec
Image path: C:WindowsSystem32html.iec
Image name: html.iec
Timestamp: Tue Mar 03 03:26:33 2009 (49ACA369)
CheckSum: 00086241
ImageSize: 0007F000
File version: 2017.0.0.18226
Product version: 6.0.6001.18226
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: HTML.IEC
OriginalFilename: HTML.IEC
ProductVersion: 6.0.6001.18226
FileVersion: 2017.0.0.18226 (vistasp1_gdr.090302-1506)
FileDescription: Microsoft HTML Converter
LegalCopyright: © Microsoft Corporation. All rights reserved.An application failed to start and exception monitor saved the process memory dump with the following stack trace of the main thread (Volume 1, page 436) blocked (Volume 2, page 184) by raising a status:
0:000> !analyze -v [...] ERROR_CODE: (NTSTATUS)0xc0150004 - The referenced assembly is not installed on your system.[...] STACK_TEXT: 0012fca4 7c93f14ec01500040012fd30 7c90d950 ntdll!RtlRaiseStatus+0×260012fd1c 7c90e437 0012fd30 7c900000 00000000 ntdll!_LdrpInitialize+0×241 00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0×7 [...]
The error code suggests that some DLL was missing (Volume 2, page 233). We see that the process just started and only a few DLL were loaded:
0:000> version
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
[...]
System Uptime: 0 days 18:00:08.328
Process Uptime: 0 days 0:00:05.000
Kernel time: 0 days 0:00:00.000
User time: 0 days 0:00:00.000
[...]0:000> !dlls -l
0x00261ec0: C:Program FilesCompanyAApplication.exe
Base 0x00400000 EntryPoint 0x00410a13 Size 0x00057000
Flags 0x00005000 LoadCount 0x0000ffff TlsIndex 0x00000000
LDRP_LOAD_IN_PROGRESS
LDRP_ENTRY_PROCESSED
0x00261f18: C:WINDOWSsystem32
tdll.dll
Base 0x7c900000 EntryPoint 0x7c912c28 Size 0x000af000
Flags 0x00085004 LoadCount 0x0000ffff TlsIndex 0x00000000
LDRP_IMAGE_DLL
LDRP_LOAD_IN_PROGRESS
LDRP_ENTRY_PROCESSED
LDRP_PROCESS_ATTACH_CALLED
0x00261fc0: C:WINDOWSsystem32kernel32.dll
Base 0x7c800000 EntryPoint 0x7c80b63e Size 0x000f6000
Flags 0x00085004 LoadCount 0x0000ffff TlsIndex 0x00000000
LDRP_IMAGE_DLL
LDRP_LOAD_IN_PROGRESS
LDRP_ENTRY_PROCESSED
LDRP_PROCESS_ATTACH_CALLEDWe try to find any execution residue (Volume 2, page 239) left on the raw stack that suggests the name of the missing DLL. We dump all raw stack range and try to interpret every double word as a pointer to UNICODE and ASCII strings. We find these entries:
0:000> !teb
TEB at 7ffde000
ExceptionList: 0012fd0c
StackBase: 00130000
StackLimit: 0012e000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffde000
EnvironmentPointer: 00000000
ClientId: 000001dc . 00000dc8
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 00:000> dpa0012e000 001300000012e000 00000000 0012e004 00000000 0012e008 00000000 0012e00c 00000000 [...] 0012f28c 0012f2a0 "" 0012f290 7c945afc"SXS: RtlGetAssemblyStorageRoot() unable to resolve stor"0012f294 00000000 0012f298 7c91eaf4 "" 0012f29c 7c91eaf4 "" 0012f2a0 00000000 0012f2a4 00000001 0012f2a8 35313063 0012f2ac 34303030 0012f2b0 00000001 0012f2b4 0012f34c"SXS: RtlGetAssemblyStorageRoot().Z.|..."0012f2b8 000074da 0012f2bc 0012f2f8 "" 0012f2c0 7c92fb96 ".......t..M...^.." 0012f2c4 0012f2d8 0012f2c8 7c945afc"SXS: RtlGetAssemblyStorageRoot() unable to resolve stor"0012f2cc 00000000 [...] 0:000> dpu0012e000 001300000012e000 00000000 0012e004 00000000 0012e008 00000000 0012e00c 00000000 0012e010 00000000 [...] 0012f164 020800ac0012f168 0012f194 WINDOWSWinSxS "C: x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e"0012f16c 00000000 0012f170 00000000 0012f174 00860086 0012f178 001505d0"x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.o.50727.762_x"0012f17c 00000000 0012f180 00000000 0012f184 019101bb 0012f188 ffffffff [...]
We also find the following path by visual inspection of the raw stack data located (Volume 2, page 300) near the exception code c0150004:
0:000> dc0012e000 001300000012e000 00000000 00000000 00000000 00000000 ................ 0012e010 00000000 00000000 00000000 00000000 ................ [...]0012f190 0012f164 003a0043 0057005c 004e0049 d...C.: .W.I.N. 0012f1a0 004f0044 00530057 0057005c 006e0069 D.O.W.S..W.i.n. 0012f1b0 00780053 005c0053 00380078 005f0036 S.x.S..x.8.6._. 0012f1c0 0069004d 00720063 0073006f 0066006f M.i.c.r.o.s.o.f. 0012f1d0 002e0074 00430056 00300038 0043002e t...V.C.8.0...C. 0012f1e0 00540052 0031005f 00630066 00620038 R.T._.1.f.c.8.b. 0012f1f0 00620033 00610039 00650031 00380031 3.b.9.a.1.e.1.8. 0012f200 00330065 005f0062 002e0038 002e0030 e.3.b._.8...0... 0012f210 00300035 00320037 002e0037 00360037 5.0.7.2.7...7.6 0012f220 005f0032 002d0078 00770077 0036005f 2._.x.-.w.w._.6. 0012f230 00310062 00380032 00300037 005c0030 b.1.2.8.7.0.0..0012f240 00000000 002a056c 000006e2 001601c8 ....l.*......... 0012f250 00000001 00000fa0 7c911066 7c9101bb ........f..|...| 0012f260 00000000 00271f10 00000004 7c910ee4 ......'........| 0012f270 35313063 34303030 00000314 0000000ac0150004........ 0012f280 000074da 0012f2c0 7c92fb96 0012f2a0 .t.........|.... 0012f290 7c945afc 00000000 7c91eaf4 7c91eaf4 .Z.|.......|...| 0012f2a0 00000000 00000001 35313063 34303030 ........c01500040012f2b0 00000001 0012f34c 000074da 0012f2f8 ....L....t...... 0012f2c0 7c92fb96 0012f2d8 7c945afc 00000000 ...|.....Z.|.... 0012f2d0 7c91eaf4 7c91eaf4 00000000 00000001 ...|...|........ 0012f2e0 0012f2fc 7c92fcbb 00000001 0012f384 .......|........ 0012f2f0 0000005b 00000035 00000000 0012f5a0 [...5........... 0012f300 7c92fc78 0012f37c 00000035 7c91eab5 x..||...5......| 0012f310 00000000 c0150004 00000003 00000000 ................ 0012f320 00150020 00000000 0012f57c 7c935996 .......|....Y.| 0012f330 00162f90 7c945afc 0012f598 00000000 ./...Z.|........ 0012f340 0000005b 7c91005b 0012f34c 3a535853 [...[..|L...SXS: 0012f350 6c745220 41746547 6d657373 53796c62 RtlGetAssemblyS 0012f360 61726f74 6f526567 2928746f 7c945afc torageRoot().Z.| [...] 0:000> du /c 90 012f190+4 0012f194"C: WINDOWSWinSxSx86_Microsoft. VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.762_x-ww_6b128700"
Doing Google search we find that other users experienced similar problems and they were resolved by (re-)installing VC++ 2005 redistribution package and the latest service pack.
The busier a system is the more inconsistent (Volume 1, page 269) are complete memory dumps produced by external physical memory dumpers. On the contrary, quiet systems, with idle existent users and when no one can connect, are more consistent than usual. Here is one example:
Loading Dump File [complete_dump.dmp]
Kernel Complete Dump File: Full address space is available
[...]
WARNING: Process directory table base 00039000 doesn't match CR3 CB759000
[...]
BugCheck 0, {0, 0, 0, 0}
Probably caused by : LiveKdD.SYS ( LiveKdD+12d1 )In such cases we often dump all processes and threads to look for anomalies and find plenty of blocked threads (Volume 2, page 184) in several services. For example, ServiceA.exe has blocked threads waiting for critical sections and LPC requests:
0: kd> !process 0 ff **** NT ACTIVE PROCESS DUMP **** [...]
THREAD 86f21118 Cid 07d4.2914 Teb: 7ffa5000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
86f21304 Semaphore Limit 0x1
Waiting for reply to LPC MessageId 00f2b546:
Current LPC port d79c6e00
Impersonation token: db621028 (Level Impersonation)
DeviceMap dca24d50
Owning Process 88948d88 Image: ServiceA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 14788708 Ticks: 3643121 (0:15:48:43.765)
Context Switch Count 52
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ServiceA!EventWorkerThread (0×0042e2f8)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b2246000 Current b2245c08 Base b2246000 Limit b2243000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
b2245c10 00000000 0×149c3
[...]
THREAD 8736fdb0 Cid 07d4.3cc4 Teb: 7ff9c000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
8736ff9c Semaphore Limit 0x1
Waiting for reply to LPC MessageId 00f2b518:
Current LPC port d79c6e00
Impersonation token: da1b51a8 (Level Impersonation)
DeviceMap d9f9e368
Owning Process 88948d88 Image: ServiceA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 14788707 Ticks: 3643122 (0:15:48:43.781)
Context Switch Count 44
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ServiceA!EventWorkerThread (0×0042e2f8)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init ae5d3000 Current ae5d2c08 Base ae5d3000 Limit ae5d0000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
ae5d2c10 3fe63bfa 0×73e40763
[...]THREAD 85e89c80 Cid 07d4.5648 Teb: 7ff6a000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
871d6e80 SynchronizationEvent
Impersonation token: d7a1c028 (Level Impersonation)
DeviceMap d66ef860
Owning Process 88948d88 Image: ServiceA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 14320472 Ticks: 4111357 (0:17:50:39.953)
Context Switch Count 28
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00ebebcc
LPC Server thread working on message Id ebebcc
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init a536b000 Current a536ac60 Base a536b000 Limit a5368000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
a536ac78 8083d5b1 nt!KiSwapContext+0x26
a536aca4 8083df9e nt!KiSwapThread+0x2e5
a536acec 8092ae57 nt!KeWaitForSingleObject+0x346
a536ad50 80833bdf nt!NtWaitForSingleObject+0x9a
a536ad50 7c8285ec nt!KiFastCallEntry+0xfc
05b6f7d0 7c827d0b ntdll!KiFastSystemCallRet
05b6f7d4 7c83d236 ntdll!NtWaitForSingleObject+0xc
05b6f810 7c83d281 ntdll!RtlpWaitOnCriticalSection+0×1a3
05b6f830 0040bb5b ntdll!RtlEnterCriticalSection+0xa8
05b6f874 00415889 ServiceA!OnRequest+0×23
[...]
05b6f8b4 77ce33e1 RPCRT4!Invoke+0×30
05b6fcb4 77ce35c4 RPCRT4!NdrStubCall2+0×299
05b6fcd0 77c7ff7a RPCRT4!NdrServerCall2+0×19
05b6fd04 77c8042d RPCRT4!DispatchToStubInCNoAvrf+0×38
05b6fd58 77c80353 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×11f
05b6fd7c 77c7e0d4 RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3
05b6fdbc 77c7e080 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xc0
05b6fdfc 77c812f0 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×41e
05b6fe20 77c88678 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
05b6ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
05b6ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
05b6ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d
05b6ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
05b6ffec 00000000 kernel32!BaseThreadStart+0×34Knowing that ServiceA is crucial for our system functions we examine LPC wait chains (page 97). All "Waiting for reply to LPC MessageId <...>" threads depend on another service:
0: kd> !lpc message 00f2b546
Searching message f2b546 in threads ...
Client thread 86f21118 waiting a reply from
f2b546
Server thread 860cc9e0 is working on message f2b546
Server process : 87d21698 (svchost.exe)We find this thread in svchost.exe hosting ServiceB:
0: kd> !thread 860cc9e0 1f
THREAD 860cc9e0 Cid 1310.4ad8 Teb: 7ff88000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
871d47d0 Mutant - owning thread 867322e0
Not impersonating
DeviceMap d64008f8
Owning Process 87d21698 Image: svchost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 14788708 Ticks: 3643121 (0:15:48:43.765)
Context Switch Count 1791
UserTime 00:00:00.031
KernelTime 00:00:00.046
Win32 Start Address 0×00f2b546
LPC Server thread working on message Id f2b546
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b055e000 Current b055dc60 Base b055e000 Limit b055b000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
b055dc78 8083d5b1 nt!KiSwapContext+0×26
b055dca4 8083df9e nt!KiSwapThread+0×2e5
b055dcec 8092ae57 nt!KeWaitForSingleObject+0×346
b055dd50 80833bdf nt!NtWaitForSingleObject+0×9a
b055dd50 7c8285ec nt!KiFastCallEntry+0xfc
0400f718 7c827d0b ntdll!KiFastSystemCallRet
0400f71c 7654bf59 ntdll!NtWaitForSingleObject+0xc
0400f73c 7654c694 ServiceB!Lock+0×42
0400f754 76558b17 ServiceB!FindObjectById+0×39
0400f8ac 7654fa50 ServiceB!QueryInformation+0×3e
0400f8c8 77c80193 ServiceB!RpcQueryInformation+0×49
0400f8f8 77ce33e1 RPCRT4!Invoke+0×30
0400fcf8 77ce35c4 RPCRT4!NdrStubCall2+0×299
0400fd14 77c7ff7a RPCRT4!NdrServerCall2+0×19
0400fd48 77c8042d RPCRT4!DispatchToStubInCNoAvrf+0×38
0400fd9c 77c80353 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×11f
0400fdc0 77c811dc RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3
0400fdfc 77c812f0 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×42c
0400fe20 77c88678 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0400ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0400ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
0400ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d
0400ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
0400ffec 00000000 kernel32!BaseThreadStart+0×34We notice that about 45 threads are waiting for the mutant 871d47d0 owned by thread 867322e0, like this one with incorrect stack trace (Volume 1, page 288) because of the inconsistent dump file data:
0: kd> !thread 86cad210 1f
THREAD 86cad210 Cid 1310.31a4 Teb: 7ff15000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
871d47d0 Mutant - owning thread 867322e0
Not impersonating
DeviceMap d64008f8
Owning Process 87d21698 Image: svchost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 18020896 Ticks: 410933 (0:01:47:00.828)
Context Switch Count 43
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address RPCRT4!ThreadStartRoutine (0×77c7b0f5)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init abb1b000 Current abb1ac60 Base abb1b000 Limit abb18000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
abb1ac68 60203020 0×84
[...]We inspect the thread 867322e0:
0: kd> !thread 867322e0 1f
THREAD 867322e0 Cid 1310.26dc Teb: 7ff56000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
869d4020 ProcessObject
86732358 NotificationTimer
Not impersonating
DeviceMap d64008f8
Owning Process 87d21698 Image: svchost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 18454684
Context Switch Count 5172
UserTime 00:00:00.000
KernelTime 00:00:00.031
Win32 Start Address 0×0083fd5d
LPC Server thread working on message Id 83fd5d
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init ad350000 Current ad34fc60 Base ad350000 Limit ad34d000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
ad34fc78 8083d5b1 nt!KiSwapContext+0×26
ad34fca4 8083df9e nt!KiSwapThread+0×2e5
ad34fcec 8092ae57 nt!KeWaitForSingleObject+0×346
ad34fd50 80833bdf nt!NtWaitForSingleObject+0×9a
ad34fd50 7c8285ec nt!KiFastCallEntry+0xfc
04f5fbec 7c827d0b ntdll!KiFastSystemCallRet
04f5fbf0 76548721 ntdll!NtWaitForSingleObject+0xc
04f5fc0c 7654c596 ServiceB!WaitForProcess+0×4a
04f5ffb8 77e64829 ServiceB!LpcThread+0×14e
04f5ffec 00000000 kernel32!BaseThreadStart+0×34We see that it is waiting for the process 869d4020 and we find the latter one with a single thread blocked by DriverA:
0: kd> !process 869d4020 ff
PROCESS 869d4020 SessionId: 18 Cid: 56b8 Peb: 7ffde000 ParentCid:
3608
DirBase: 758b3000 ObjectTable: dd7083c0 HandleCount: 79.
Image: rundll32.exe
VadRoot 869d5228 Vads 107 Clone 0 Private 330. Modified 1706. Locked
0.
DeviceMap d8d260a8
Token dfa8f028
ElapsedTime 1 Day 23:52:06.918
UserTime 00:00:00.312
KernelTime 00:00:01.484
QuotaPoolUsage[PagedPool] 34784
QuotaPoolUsage[NonPagedPool] 5016
Working Set Sizes (now,min,max) (810, 50, 345) (3240KB, 200KB,
1380KB)
PeakWorkingSetSize 2088
VirtualSize 37 Mb
PeakVirtualSize 47 Mb
PageFaultCount 34473
MemoryPriority BACKGROUND
BasePriority 10
CommitCharge 707
[..]
CommandLine: 'rundll32 WINDOWSsystem32DllA.dll,CheckDataC:
/name"PropertyA"'
[...]
THREAD 86bac020 Cid 56b8.4904 Teb: 7ffdb000 Win32Thread: 00000000 WAIT:
(Unknown) KernelMode Non-Alertable
8774a930 NotificationEvent
Not impersonating
DeviceMap d8d260a8
Owning Process 869d4020 Image: rundll32.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10113815 Ticks: 8318014 (1:12:06:08.968)
Context Switch Count 188
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x2b169ecb
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init af12b000 Current af12ac34 Base af12b000 Limit af128000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
af12ac4c 8083d5b1 nt!KiSwapContext+0x26
af12ac78 8083df9e nt!KiSwapThread+0x2e5
af12acc0 b5075a82 nt!KeWaitForSingleObject+0x346
WARNING: Stack unwind information not available. Following frames may be
wrong.
af12acf4 b507f989 DriverA+0×3a82
[...]We also check virtual memory consumption and instantly see hundreds of rundll32.exe processes like in a Process Factory pattern (page 112):
0: kd> !vm [...] 57f4 rundll32.exe 586 ( 2344 Kb) 57d0 rundll32.exe 586 ( 2344 Kb) 57b4 rundll32.exe 586 ( 2344 Kb) 578c rundll32.exe 586 ( 2344 Kb) 5780 rundll32.exe 586 ( 2344 Kb) 577c rundll32.exe 586 ( 2344 Kb) 5744 rundll32.exe 586 ( 2344 Kb) 572c rundll32.exe 586 ( 2344 Kb) 5724 rundll32.exe 586 ( 2344 Kb) 5708 rundll32.exe 586 ( 2344 Kb) 56d4 rundll32.exe 586 ( 2344 Kb) 56a4 rundll32.exe 586 ( 2344 Kb) 564c rundll32.exe 586 ( 2344 Kb) 5630 rundll32.exe 586 ( 2344 Kb) 55e0 rundll32.exe 586 ( 2344 Kb) 55b8 rundll32.exe 586 ( 2344 Kb) 55ac rundll32.exe 586 ( 2344 Kb) 558c rundll32.exe 586 ( 2344 Kb) 550c rundll32.exe 586 ( 2344 Kb) 546c rundll32.exe 586 ( 2344 Kb) 5428 rundll32.exe 586 ( 2344 Kb) 5418 rundll32.exe 586 ( 2344 Kb) 53d0 rundll32.exe 586 ( 2344 Kb) 53bc rundll32.exe 586 ( 2344 Kb) 53b0 rundll32.exe 586 ( 2344 Kb) 53a8 rundll32.exe 586 ( 2344 Kb) 530c rundll32.exe 586 ( 2344 Kb) 5300 rundll32.exe 586 ( 2344 Kb) 52e8 rundll32.exe 586 ( 2344 Kb) 52a4 rundll32.exe 586 ( 2344 Kb) 5294 rundll32.exe 586 ( 2344 Kb) 5228 rundll32.exe 586 ( 2344 Kb) 5118 rundll32.exe 586 ( 2344 Kb) 50bc rundll32.exe 586 ( 2344 Kb) 50a8 rundll32.exe 586 ( 2344 Kb) 5080 rundll32.exe 586 ( 2344 Kb) 5030 rundll32.exe 586 ( 2344 Kb) 4fc8 rundll32.exe 586 ( 2344 Kb) 4f88 rundll32.exe 586 ( 2344 Kb) 4ef0 rundll32.exe 586 ( 2344 Kb) 4e84 rundll32.exe 586 ( 2344 Kb) 4e70 rundll32.exe 586 ( 2344 Kb) 4e68 rundll32.exe 586 ( 2344 Kb) 4e4c rundll32.exe 586 ( 2344 Kb) 4e38 rundll32.exe 586 ( 2344 Kb) 4df8 rundll32.exe 586 ( 2344 Kb) 4df4 rundll32.exe 586 ( 2344 Kb) 4d2c rundll32.exe 586 ( 2344 Kb)
4d00 rundll32.exe 586 ( 2344 Kb) 4cd4 rundll32.exe 586 ( 2344 Kb) 4cbc rundll32.exe 586 ( 2344 Kb) 4c90 rundll32.exe 586 ( 2344 Kb) 4c80 rundll32.exe 586 ( 2344 Kb) 4c68 rundll32.exe 586 ( 2344 Kb) 4c40 rundll32.exe 586 ( 2344 Kb) [...]
By switching the current process context to 869d4020 (.process /r /p command) and using lmv command we check vendors for DllA and DriverA. It seems that they are different. We also see that DllA module is no longer loaded (not seen in the module list) by the time DriverA is waiting for a notification event. All rundll32.exe processes are waiting for that DriverA kernel module. We therefore decide to contact its vendor.
The memory dump shows an invalid pointer access while copying a buffer:
Loading Dump File [MEMORY.DMP] Kernel Summary Dump File:Only kernel address space is available6: kd> !analyze -v [...] PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced. This cannot be protected by try- except, it must be protected by a Probe. Typically the address is just plain bad or it is pointing at freed memory. Arguments: Arg1: fe2cc128, memory referenced. Arg2: 00000001, value 0 = read operation, 1 = write operation. Arg3: b574414c, If non-zero, the instruction address which referenced the bad memory address. Arg4: 00000000, (reserved) [...] TRAP_FRAME: aefed9d8 -- (.trap 0xffffffffaefed9d8) ErrCode = 00000002 eax=fe2fb09c ebx=00000006 ecx=00000001 edx=00000006 esi=f5312bcdedi=fe2cc128eip=b574414c esp=aefeda4c ebp=aefeda58 iopl=0 nv up ei pl nz na po cy cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010203 Driver+0×1614c: b574414c f3a5 rep movs dword ptr es:[edi],dword ptr [esi] Resetting default scope STACK_TEXT: aefed948 8085ed19 00000050 fe2cc128 00000001 nt!KeBugCheckEx+0x1b aefed9c0 8088c7c8 00000001 fe2cc128 00000000 nt!MmAccessFault+0xb25 aefed9c0b574414c00000001 fe2cc128 00000000 nt!KiTrap0E+0xdcWARNING: Stack unwind information not available. Following frames may be wrong.aefeda58 b574472d f5312bcd 00000006 fe2fb09c Driver+0×1614c [...] aefedb70 8081df85 fe2fb000 aefedbcc 88d0e010 Driver+0×2758a aefedc50 808f5437 8c556c88 8a1fea78 8c556c18 nt!IofCallDriver+0×45 aefedc64 808f61bf 90510070 8c556c18 8a1fea78 nt!IopSynchronousServiceTail+0×10b aefedd00 808eed08 00001250 00000000 00000000 nt!IopXxxControlFile+0×5e5 aefedd34 808897bc 00001250 00000000 00000000 nt!NtDeviceIoControlFile+0×2a aefedd34 7c82860c 00001250 00000000 00000000 nt!KiFastCallEntry+0xfc 0620f428 00000000 00000000 00000000 00000000 0×7c82860c
6: kd> !pteediVA fe2cc000 PDE at 00000000C0603F88 PTE at 00000000C07F1660 contains 0000000000ABE863 contains0000000000000000pfn abe - DA-KWEV
The warning about frames suggests that stack trace may be incorrect (Volume 1, page 288) but backward and forward disassembly and preceding frames show that it is correct indeed and the warning results from the lack of symbol files:
6: kd> ub b574414c
*** ERROR: Module load completed but symbols could not be loaded for
Driver.SYS
Driver+0×16139:
b5744139 8d341a lea esi,[edx+ebx]
b574413c 03fa add edi,edx
b574413e 3bf1 cmp esi,ecx
b5744140 8b7508 mov esi,dword ptr [ebp+8]
b5744143 771f ja Driver+0×16164 (b5744164)
b5744145 8bcb mov ecx,ebx
b5744147 8bd1 mov edx,ecx
b5744149 c1e902 shr ecx,2
6: kd> u b574414c
Driver+0×1614c:
b574414c f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
b574414e 8bca mov ecx,edx
b5744150 83e103 and ecx,3
b5744153 f3a4 rep movs byte ptr es:[edi],byte ptr [esi]
b5744155 8b4810 mov ecx,dword ptr [eax+10h]
b5744158 5f pop edi
b5744159 03cb add ecx,ebx
b574415b 5e pop esiAt the same time we see a second fault (page 78) on another processor but it seems to be a demand zero page access:
6: kd> ~7s;kv ChildEBP RetAddr Args to Child ae331854 8095dddf badb0d00 0000000f 808a623cnt!KiTrap0E+0xbc (TrapFrame @ae331854) ae3318cc 809731b2 de019000 00000001 3b9aca07 nt!RtlCreateSecurityDescriptorRelative+0×13 ae331934 808d3229 ae3319e8 de019000 ae3319ec nt!SeQuerySecurityDescriptorInfo+0×198 ae33194c 808d443c dc0d9330 ae3319e8 de019000 nt!CmpQuerySecurityDescriptorInfo+0×23 ae3319b0 809383d3 e09e33f0 00000001 ae3319e8 nt!CmpSecurityMethod+0×212 ae3319f0 8093866e e09e33f0 ae331a1c e09e33ec nt!ObpGetObjectSecurity+0×99 ae331a20 808d8ecc e09e33f0 8ae88740 00000001 nt!ObCheckObjectAccess+0×2c ae331a7c 808da58b dbef6008 009fd5e0 00000000 nt!CmpDoOpen+0×3a0 ae331b90 809374b1 f0d9d348 00000000 8ae88740 nt!CmpParseKey+0×547 ae331c10 80933a76 0000004c ae331c50 00000040 nt!ObpLookupObjectName+0×11f ae331c64 808bb471 00000000 908fb868 80934601 nt!ObOpenObjectByName+0xea ae331d50 808897bc 0013f7bc 00020019 0013f720 nt!NtOpenKey+0×1ad ae331d50 7c82860c 0013f7bc 00020019 0013f720 nt!KiFastCallEntry+0xfc (TrapFrame @ ae331d64) 7: kd> .trapae331854ErrCode = 00000002 eax=00000000 ebx=ae3319e8 ecx=de019000 edx=0000000f esi=de019000 edi=de019000 eip=8095dddf esp=ae3318c8 ebp=ae3318cc iopl=0 nv up ei pl zr na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246 nt!RtlCreateSecurityDescriptorRelative+0×13: 8095dddf ab stos dword ptr es:[edi]es:0023:de019000=???????? 7: kd> !pteediVA de019000 PDE at 00000000C0603780 PTE at 00000000C06F00C8 contains 00000003D2FF9863 contains 00000000000000C0 pfn 3d2ff9 - DA-KWEV not validProtect: 6 - ReadWriteExecuteDemandZero
We suspect that users cannot connect because of the lack of kernel pool resources and see the shortage of paged pool (Volume 1, page 440) along with session pool shortage for one session:
7: kd> !vm 4 *** Virtual Memory Usage *** Physical Memory: 4193705 ( 16774820 Kb) Page File: ??C:pagefile.sys Current: 25160704 Kb Free Space: 18350628 Kb Minimum: 25160704 Kb Maximum: 25160704 Kb Available Pages: 2556764 ( 10227056 Kb) ResAvail Pages: 3996322 ( 15985288 Kb) Locked IO Pages: 292 ( 1168 Kb) Free System PTEs: 22827 ( 91308 Kb)******* 4 system PTE allocations have failed ******Free NP PTEs: 32534 ( 130136 Kb) Free Special NP: 0 ( 0 Kb) Modified Pages: 86 ( 344 Kb) Modified PF Pages: 85 ( 340 Kb) NonPagedPool Usage: 32376 ( 129504 Kb) NonPagedPool Max: 65215 ( 260860 Kb) PagedPool 0 Usage: 84341 ( 337364 Kb) PagedPool 1 Usage: 7945 ( 31780 Kb) PagedPool 2 Usage: 8073 ( 32292 Kb) PagedPool 3 Usage: 7979 ( 31916 Kb) PagedPool 4 Usage: 8048 ( 32192 Kb)PagedPool Usage: 116386 ( 465544 Kb) PagedPool Maximum: 134144 ( 536576 Kb) ********** 841 pool allocations have failed **********Shared Commit: 95929 ( 383716 Kb) Special Pool: 0 ( 0 Kb) Shared Process: 126106 ( 504424 Kb) PagedPool Commit: 116435 ( 465740 Kb) Driver Commit: 2605 ( 10420 Kb) Committed pages: 3454252 ( 13817008 Kb) Commit limit: 10419631 ( 41678524 Kb) Total Private: 2961488 ( 11845952 Kb)721c Application.exe 334248 ( 1336992 Kb)4b4c iexplore.exe 36048 ( 144192 Kb) [...] Terminal Server Memory Usage By Session: Session Paged Pool Maximum is 32768K Session View Space Maximum is 20480K
Session ID 0 @ ba1c5000: Paged Pool Usage: 2288K Commit Usage: 3344K [...]Session ID 1b@ ba1e3000: Paged Pool Usage: 12020K*** 622 Pool Allocation Failures ***Commit Usage: 13176K [...]
We also see that the process with PID 721c (Application.exe) consumed 1.3Gb of memory and it belongs to the problem session 0×1b (0n27):
7: kd> !sprocess 1b
Dumping Session 1b
_MM_SESSION_SPACE ba1e3000
_MMSESSION ba1e3d80
PROCESS 8a4fd020 SessionId: 27 Cid: 1f24 Peb: 7ffde000 ParentCid:
01e4
DirBase: cfe3f820 ObjectTable: ec731d08 HandleCount: 281.
Image: csrss.exe
PROCESS 8a4d6b18 SessionId: 27 Cid: 2160 Peb: 7ffde000 ParentCid:
01e4
DirBase: cfe3f800 ObjectTable: ee932bf0 HandleCount: 436.
Image: winlogon.exe
[...]
PROCESS 8a467020 SessionId: 27 Cid: 721c Peb: 7ffdf000 ParentCid:
5728
DirBase: cfe3f8a0 ObjectTable: f0397350 HandleCount: 380.
Image: Application.exe
[...]Unfortunately the dump file is only a kernel dump and we cannot inspect the user space of that process. We can only say that it was not a handle leak (Volume 1, page 327) because the number of handles is low (380). We also inspect paged pool stats and find a 3rd-party driver with AAAA tag that used almost 80Mb of paged pool:
7: kd> !poolused 4
Sorting by Paged Pool Consumed
Pool Used:
NonPaged Paged
Tag Allocs Used Allocs Used
MmSt 0 0 57216 91753456 Mm section object prototype
ptes , Binary: nt!mm
AAAA 0 0 613 79465472 UNKNOWN pooltag 'AAAA' , please
update pooltag.txt
CM35 0 0 529 40398848 Internal Configuration manager
allocations , Binary: nt!cm
Ntff 5 1040 35955 29339280 FCB_DATA , Binary: ntfs.sys
Toke 0 0 3334 12631056 Token objects , Binary: nt!se
IoNm 0 0 43494 9611392 Io parsing names , Binary:
nt!io
CMAl 0 0 1981 8114176 internal registry memory
allocator pool tag , Binary: nt!cm
Obtb 0 0 2569 6687344 object tables via EX handle.c ,
Binary: nt!ob
CM16 0 0 1429 6299648 Internal Configuration manager
allocations , Binary: nt!cm
[...]We can find this driver like explained in Volume 1, page 206. We recommend contacting the vendor of that driver and also monitoring instances of Application.exe and getting 2-3 subsequent process memory dumps, when it grows, to inspect its virtual memory (Volume 2, page 210) using differential memory analysis techniques.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.