Connect with us on Facebook and Linkedin

Case Study

This case study illustrates the pattern-driven approach we use to analyze customer memory dumps saved after crashes, hangs, memory leaks, CPU spikes and any other kind of abnormal software behavior.

One application was crashing on selected computers. A customer sent a process memory dump. Upon opening it we recognized it as a minidump (Abridged Dump pattern):

Loading Dump File [C:\MemoryDumps\CustomerA\IncidentB\DrWatson.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Comment: 'Dr. Watson generated MiniDump'

Instead of suggesting to configure Dr. Watson to save full user dumps by running its GUI (drwtsn32.exe) and selecting the checkbox Crash Dump Type: Full we gave it a try.

The default analysis gave us an indication of a C++ Exception pattern:

0:023> .symfix c:\mss

0:023> .reload

0:023> !analyze -v

[...]

EXCEPTION_RECORD:  1022fa1c -- (.exr 0x1022fa1c)
ExceptionAddress: 7c812afb (kernel32!RaiseException+0x00000053)
   ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
   Parameter[0]: 19930520
   Parameter[1]: 1022fda4
   Parameter[2]: 0b985074
Unable to load image C:\WINNT\system32\MSVCR71.DLL, Win32 error 0n2
*** WARNING: Unable to verify timestamp for MSVCR71.DLL
  pExceptionObject: 1022fda4
  _s_ThrowInfo    : 0b985074

[...]

CONTEXT:  1022fa3c -- (.cxr 0x1022fa3c)
eax=1022fd0c ebx=00000001 ecx=00000000 edx=1022fda4 esi=1022fd94 edi=77606068
eip=7c812afb esp=1022fd08 ebp=1022fd5c iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000206
kernel32!RaiseException+0x53:
7c812afb 5e              pop     esi
Resetting default scope

[...]

STACK_TEXT:
00000000 00000000 ApplicationA.exe+0x0

It didn't give us a good stack trace (Incorrect Stack Trace pattern). Also, it wasn't the current thread (#23) that caused the exception:

0:023> k
ChildEBP RetAddr
00fcffc8 7c951e40 ntdll!DbgBreakPoint
00fcfff4 00000000 ntdll!DbgUiRemoteBreakin+0x2d

So we dumped all stack traces (Stack Trace Collection pattern) and found a one that has exception processing functions (Exception Stack Trace pattern):

   9  Id: 1df4.a08 Suspend: -1 Teb: 7fff4000 Unfrozen
ChildEBP RetAddr
1022f5a8 7c90df4a ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 kernel32!UnhandledExceptionFilter+0x8b9
1022f908 7c839b39 kernel32!BaseThreadStart+0x4d
1022f930 7c9032a8 kernel32!_except_handler3+0x61
1022f954 7c90327a ntdll!ExecuteHandler2+0x26
1022fa04 7c90e48a ntdll!ExecuteHandler+0x24
1022fa04 7c812afb ntdll!KiUserExceptionDispatcher+0xe
1022fd5c 0b82e680 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
1022fd94 0b82d2f2 DllA+0x31e680
1022fde8 7753004f DllA+0x31d2f2
1022fdfc 7753032f ole32!CClassCache::CDllPathEntry::CanUnload_rl+0x3b
1022ff3c 7753028b ole32!CClassCache::FreeUnused+0x70
1022ff4c 775300b5 ole32!CoFreeUnusedLibrariesEx+0x36
1022ff58 77596af5 ole32!CoFreeUnusedLibraries+0x9
1022ff6c 77566ff9 ole32!CDllHost::MTAWorkerLoop+0x25
1022ff8c 7752687c ole32!CDllHost::WorkerThread+0xc1
1022ff94 774fe3ee ole32!DLLHostThreadEntry+0xd
1022ffa8 774fe456 ole32!CRpcThread::WorkerLoop+0x1e
1022ffb4 7c80b729 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1b
1022ffec 00000000 kernel32!BaseThreadStart+0x37

We also revisited the previous output and exercised this command after switching to the thread #9:

0:023> ~9s
eax=000000c0 ebx=1022f5c8 ecx=00001000 edx=7c90e514 esi=000015ac edi=00000000
eip=7c90e514 esp=1022f5ac ebp=1022f900 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000206
ntdll!KiFastSystemCallRet:
7c90e514 c3              ret

0:009> .cxr 0x1022fa3c
eax=1022fd0c ebx=00000001 ecx=00000000 edx=1022fda4 esi=1022fd94 edi=77606068
eip=7c812afb esp=1022fd08 ebp=1022fd5c iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000206
kernel32!RaiseException+0x53:
7c812afb 5e              pop     esi

We then got our stack trace:

0:009> k
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
1022fd5c 0b82e680 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
1022fd94 0b82d2f2 DllA+0x31e680
1022fde8 7753004f DllA+0x31d2f2
1022fdfc 7753032f ole32!CClassCache::CDllPathEntry::CanUnload_rl+0x3b
1022ff3c 7753028b ole32!CClassCache::FreeUnused+0x70
1022ff4c 775300b5 ole32!CoFreeUnusedLibrariesEx+0x36
1022ff58 77596af5 ole32!CoFreeUnusedLibraries+0x9
1022ff6c 77566ff9 ole32!CDllHost::MTAWorkerLoop+0x25
1022ff8c 7752687c ole32!CDllHost::WorkerThread+0xc1
1022ff94 774fe3ee ole32!DLLHostThreadEntry+0xd
1022ffa8 774fe456 ole32!CRpcThread::WorkerLoop+0x1e
1022ffb4 7c80b729 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1b
1022ffec 00000000 kernel32!BaseThreadStart+0x37

Although we got the warning about possible wrong frames after RaiseException the overall stack trace sequence looked normal.

We see the exception was caused by DllA written by this vendor, version and timestamp (Not My Version pattern):

0:009> lmv m DllA
start    end        module name
0b510000 0b9ea000   DllA T (no symbols)          
    Loaded symbol image file: DllA.dll
    Image path: C:\Program Files\VendorB\DllA.dll
    Image name: DllA.dll
    Timestamp:     [...]
    File version:  [...]
    Product version:
    File flags:       1 (Mask 3F) Debug
    File OS:          4 Unknown Win32
    File type:        2.0 Dll

We sent the detailed analysis report and advised the customer to contact VendorB.