Friday, May 29, 2009

How to find the thread which triggered a GC

How to find the thread which triggered a GC

1. Take a memory dump of the application at the correct time ( The time its using high CPU)
2. Run ~* kb 2000 to get all native stacks
3. Search for mscorwks!SVR::GCHeap::GarbageCollectGeneration, its the thread that triggered the GC ()

In addition you can see if a thread is blocked as a result of GC call by searching GCHeap::WaitUntilGCComplete

OS Thread Id: 0xcf4 (37)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr Caller,Callee
1a2ee3e4 7c90e9c0 ntdll!ZwWaitForSingleObject+0xc
1a2ee3e8 7c8025db kernel32!WaitForSingleObjectEx+0xa8, calling ntdll!NtWaitForSingleObject
1a2ee43c 7c8399f3 kernel32!_except_handler3
1a2ee44c 79e77fd1 mscorwks!PEImage::LoadImage+0x199
1a2ee490 79e77f9a mscorwks!CLREvent::WaitEx+0x117, calling mscorwks!PEImage::LoadImage+0x16a
1a2ee4b0 79f8eb00 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x165, calling ntdll!RtlSetLastWin32Error
1a2ee4b4 79f8eb08 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x16d, calling mscorwks!_SEH_epilog4
1a2ee4e0 79e77f50 mscorwks!CLREvent::Wait+0x17, calling mscorwks!CLREvent::WaitEx
1a2ee4f0 79f762f7 mscorwks!WKS::GCHeap::WaitUntilGCComplete+0x32, calling mscorwks!CLREvent::Wait
1a2ee500 79f75d6a mscorwks!Thread::RareDisablePreemptiveGC+0x1a1

Error: Failed to load data access DLL, 0x80004005

Problem: Ever got the following error while executing a command in WinDbg?

Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

Resolution:

Make sure you have correct version of mscordacwks.dll in your .Net Framework path (normally C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727). To know which version the WinDbg is looking for run the following command.
.symfix nosiy
The above command will enable the verbose mode of all module loads, including any erorr found by WinDbg.
Run .reload command to know which module is failing to load.
If the above command passes without any error, and you are still getting the same error while exeucting any command, check the output of WinDbg after you ran the command. It will be something like

CLRDLL: Unable to get version info for 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll', Win32 error 0n87
CLRDLL: Unable to find mscordacwks_x86_x86_2.0.50727.42.dll by mscorwks search
CLRDLL: Unable to find 'mscordacwks_x86_x86_2.0.50727.42.dll' on the path
CLRDLL: Unable to find mscorwks.dll by search
CLRDLL: ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.42.dll, Win32 error 0n2

Now you need to find the correct version and rename the dll to the above bold convention mscordacwks_x86_x86_.dll . You can get this file from the system the dump was taken on.
Place this dll in your framework path and everything should work now.

Tuesday, May 19, 2009

Scripting WinDbg: .foreach command

The .foreach token parses the output of one or more debugger commands and uses each value in this output as the input to one or more additional commands.

Syntax
.foreach [Options] ( Variable { InCommands } ) { OutCommands }

Options
Can be any combination of the following options:
/pS InitialSkipNumber
Causes some initial tokens to be skipped. InitialSkipNumber specifies the number of output tokens that will not be passed to the specified OutCommands.
/ps SkipNumber
Causes tokens to be skipped repeatedly each time a command is processed. After each time a token is passed to the specified OutCommands, a number of tokens equal to the value of SkipNumber will be ignored.

Sample Usage
Consider a example where we need to dump all GCRoot references of System.Drawing.Icon objects. The MT for System.Drawing.Icon in our application is 7ae7613
To find all objects of MT 7ae76134, we will run the command !DumpHeap -mt 7ae76134
The output of the above command looks like

0:000> !DumpHeap -mt 7ae76134
Address MT Size
0989d6a4 7ae76134 40
09ab24b8 7ae76134 40
09b6a044 7ae76134 40
09b6cc70 7ae76134 40
09b6cdb4 7ae76134 40
09d444e4 7ae76134 40
0a0e480c 7ae76134 40
total 7 objects
Statistics:
MT Count TotalSize Class Name
7ae76134 7 280 System.Drawing.Icon
Total 7 objects

Now in normal scenario we have to run !GCRoot on each of the object address, that is seven times to get the desired result. .foreach comes to rescue here. We can run it for each object address for a specific command. As you have noted the above output have three columns with Address as the first which will be needed to passed on to the GCRoot command.

Executing !DumpHeap -mt 7ae76134 -short gives us the following, which only gives us the required object instance Address
0:000> !DumpHeap -mt 7ae76134 -short
0989d6a4
09ab24b8
09b6a044
09b6cc70
09b6cdb4
09d444e4
0a0e480c

We can now get the desired output (that is find all references to all System.Drawing.Icon objects) by .foreach using the following two variants

  1. .foreach ( ico {!DumpHeap -mt 7ae76134 -short}) {!GCRoot ico }
  2. .foreach /ps2 ( ico {!DumpHeap -mt 7ae76134 }) {!GCRoot ico }

Both of above commands prints the exact same output, but they differ in the way its parsed.

Example 1 uses the -short option that outputs only the address of the objects, which are directly passed on to the OutCommands
Example 2 is more versatile as it does not depend on the -short option. It uses the /ps option which causes the token to be skipped. In our case the first token (Which is Address) of the Output is read and passed to GCRoot, and next two subsequent tokens ( which are MT and Size are skipped for every line)