Saturday, 18 February 2012

Use ADPlus to troubleshoot hangs and crashes in our Delphi applications

It's again debugging time and in this post I'm going to put forward how to use ADPlus to troubleshoot hangs or crashes of our Delphi applications. As I'm sure you know, applications are getting more and more complicated using different libraries and third party tools and sometimes it is quite difficult to find out where the current problem is located. It has happened to me quite often that a tool or app blows app without reason, without a proper exception (even though all exception handling mechanisms are there) so the final user never gets any indication about what's wrong. With few simple steps from this article you will be able to set up ADPlus correctly, attach it to your running process and then create the crash dump for its further analysis. Let's start with debugging time!.

Installing Debugging tools for windows:
This is the Swiss knife for any good developer. Bring it always with you as it is really helpful. You can download the library from here. Just install it and save it to and USB to be portable. Then spend time using it, not only using ADPlus but also with Windbg, as it is crucial that you know about how to analyse the crash dump.

Setting up our Delphi Applications:
To be sure our Delphi app is correctly identified, we need to generate the map file and then use an external tool to convert all that information to symbols.
Edit in the project options to generate debug information and a detailed map file. Then using map2dbg we will transform the map file into debug symbols (.dbg files).
Once your project is correctly set up, build it and you will get the map file. Download the latest version of map2dbg v1.3 and copy map2dbg.exe where your project is located and run the following command line:

C:\testAdPlus>map2dbg.exe Project1.exe
Converted 6882 symbols.

You will now see a .dbg file with all necessary information for ADPlus.

Setting up the Symbols:
Once again, if we run ADPlus without setting up the symbols, we will only see address of memory without descriptions:
Call stack below ---
*** ERROR: Module load completed but symbols could not be loaded for C:\testAdPlus\Project1.exe
 # ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0012f550 004832bd 01aba8c0 0045f20b 0012f700 Project1+0xb3295
01 0012f698 00487615 0012f91c 01aba8c0 0012f9a8 Project1+0x832bd
02 0012f6e4 0045eed1 01aba8c0 0012f91c 01aba8c0 Project1+0x87615
03 0012f710 00487768 00050980 0012f9a8 01acb650 Project1+0x5eed1
04 0012f86c 00487615 00000111 0012f91c 09010401 Project1+0x87768
05 0012f8b8 004a7c25 0012f9a8 00000111 01acb650 Project1+0x87615
06 0012f8e4 00486cb3 0012f8f8 00486ccb 0012f914 Project1+0xa7c25
07 0012f914 0043fd4a 00000111 00000980 00050980 Project1+0x86cb3
08 0012f92c 762cfd72 000209ae 00000111 00000980 Project1+0x3fd4a

Setting up _NT_SYMBOL_PATH environment variable:
Create a new general environment variable with the following name: _NT_SYMBOL_PATH and the following value:
Where C:\Symbols is the path to your symbols. This is the same parameter set in WinDbg which I explained in my previous post (monitoring atom table part I).

Setting up ADPlus:
We can either run ADPlus using a simple configuration or use the configuration file which is much more complete. In any case, the fastest way is using the simple configuration which will help us to get the expected outcome.
C:\Program Files\Debugging Tools for Windows (x86)>adplus -crash -pmn Project1.exe -o C:\Adplus -mss c:\symbols

This configuration will run ADPlus looking for crashes, waiting for Project1.exe and it will output the results in C:\ADPlus and it will use the symbols from C:\Symbols.

Sample test:
This small piece of code will help me to generate a small access violation and then ADPlus will catch the crashing and it will generate the crash dump.

procedure TForm1.Button1Click(Sender: TObject);

procedure TForm1.SimulateSystemException;
  p: PChar;
  p := PChar(5);
  p^ := #9; //Access Violation here

Once ADPlus is running, run your project, in my case Project1.exe and wait until it crashes (in my case it is just simulated so, the crash dump is instantly generated).

You will get a new line in the cmd window telling you:
Attaching to 4812 - Project1 in crash mode 02/18/2012 00:16:00

Once it crashes, go to C:\ADPlus folder and you will get a new folder with current date and the crash dump in it.

Analysing crash dump with Windbg:
Just open Windbg, load the first crash dump (FirstChance_Process_Shut_Down) and use the following commands:

0:000> !sym noisy
noisy mode - symbol prompts on
0:000> .reload Project1.exe
0:000> !analyze -v
ERROR: FindPlugIns 8007007b
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *

DBGHELP: kernel32 - public symbols  c:\symbols\kernel32.pdb\FCCF6FAC09804D49A4BB256A77F519572\kernel32.pdb
DBGHELP: Project1.exe is stripped.  Searching for dbg file
SYMSRV:  c:\symbols\Project1.dbg\4F3D9543457000\Project1.dbg not found
SYMSRV: not found
DBGHELP: .\Project1.dbg - file not found
DBGHELP: .\exe\Project1.dbg - path not found
DBGHELP: .\symbols\exe\Project1.dbg - path not found
DBGHELP: Project1.exe missing debug info.  Searching for pdb anyway
DBGHELP: Can't use symbol server for Project1.pdb - no header information available
DBGHELP: Project1.pdb - file not found
*** WARNING: Unable to verify checksum for Project1.exe
*** ERROR: Module load completed but symbols could not be loaded for Project1.exe
DBGHELP: Project1 - no symbols loaded
DBGHELP: user32 - public symbols c:\symbols\user32.pdb\CFD2C4C8EB9C406D8B6DC29512EB176A2\user32.pdb
DBGHELP: ole32 - public symbols c:\symbols\ole32.pdb\EDE30219D57144FAAC83675A6573D1982\ole32.pdb

Once processed, we can actually spot that the symbols are missing. This is because we need to place the symbols at the location defined by Windbg -> c:\symbols\Project1.dbg\4F3D9543457000\Project1.dbg. Just copy your dbg file into defined location and try again reloading Project1.exe from Windbg command line.

Now we can analyse again the crash dump and get all the information needed about the crash. This time with the correct symbols:

Call stack below ---
*** WARNING: Unable to verify checksum for C:\Users\jordi coll\Desktop\testAdPlus\Project1.exe
 # ChildEBP RetAddr  Args to Child              
00 0012f550 00473c9d 008fa8c0 00455747 0012f700 Project1!Unit1.TForm1.SimulateSystemException+0x5
01 0012f698 00477ff5 0012f91c 008fa8c0 0012f9a8 Project1!Controls.TControl.Click+0x75
02 0012f6e4 0045540d 008fa8c0 0012f91c 008fa8c0 Project1!Controls.TWinControl.WndProc+0x56d
03 0012f710 00478148 00010d2e 0012f9a8 0090b650 Project1!StdCtrls.TButtonControl.WndProc+0x71
04 0012f86c 00477ff5 00000111 0012f91c 09010401 Project1!Controls.DoControlMsg+0x28
05 0012f8b8 004984b1 0012f9a8 00000111 0090b650 Project1!Controls.TWinControl.WndProc+0x56d
06 0012f8e4 00477693 0012f8f8 004776ab 0012f914 Project1!Forms.TCustomForm.WndProc+0x599
07 0012f914 0043c146 00000111 00000d2e 00010d2e Project1!Controls.TWinControl.MainWndProc+0x2f
08 0012f92c 762cfd72 00010d26 00000111 00000d2e Project1!Classes.StdWndProc+0x16
09 0012f958 762cfe4a 00380fc8 00010d26 00000111 USER32!InternalCallWinProc+0x23
0a 0012f9d0 762d0943 00000000 00380fc8 00010d26 USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
0b 0012fa10 762d0b36 00fd2450 00fd23e8 00000d2e USER32!SendMessageWorker+0x4b7 (FPO: [Non-Fpo])
0c 0012fa30 74d1b4ba 00010d26 00000111 00000d2e USER32!SendMessageW+0x7c (FPO: [Non-Fpo])
0d 0012fa50 74d1b51c 00a54518 00000000 00080032 comctl32!Button_NotifyParent+0x3d (FPO: [Non-Fpo])
0e 0012fa6c 74d1b627 54010001 00000001 0012fb48 comctl32!Button_ReleaseCapture+0x112 (FPO: [Non-Fpo])
0f 0012facc 762cfd72 00010d2e 00000202 00000000 comctl32!Button_WndProc+0xa98 (FPO: [Non-Fpo])
10 0012faf8 762cfe4a 74cb70f8 00010d2e 00000202 USER32!InternalCallWinProc+0x23
11 0012fb70 762d09d3 00000000 74cb70f8 00010d2e USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
12 0012fba0 762d0979 74cb70f8 00010d2e 00000202 USER32!CallWindowProcAorW+0x97 (FPO: [Non-Fpo])
13 0012fbc0 004780f5 74cb70f8 00010d2e 00000202 USER32!CallWindowProcW+0x1b (FPO: [Non-Fpo])
14 0012fd44 00477ff5 00000202 008fa8c0 0012fd80 Project1!Controls.TWinControl.DefaultHandler+0xdd
15 0012fd90 0045540d 0012fe64 00000202 008fa8c0 Project1!Controls.TWinControl.WndProc+0x56d
16 0012fdd0 0043c146 00000202 00000000 00080032 Project1!StdCtrls.TButtonControl.WndProc+0x71
17 0012fde8 762cfd72 00010d2e 00000202 00000000 Project1!Classes.StdWndProc+0x16
18 0012fe14 762cfe4a 00380fbb 00010d2e 00000202 USER32!InternalCallWinProc+0x23
19 0012fe8c 762d018d 00000000 00380fbb 00010d2e USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
1a 0012fef0 762d022b 00380fbb 00000000 00010d2e USER32!DispatchMessageWorker+0x322 (FPO: [Non-Fpo])
1b 0012ff00 004a15d2 0012ff24 00010001 0012ff70 USER32!DispatchMessageW+0xf (FPO: [Non-Fpo])
1c 0012ff1c 004a1617 00010d2e 00000202 00000000 Project1!Forms.TApplication.ProcessMessage+0x122
1d 0012ff40 004a1942 0012ff54 004a194c 0012ff70 Project1!Forms.TApplication.HandleMessage+0xf
1e 0012ff70 004a9c96 0012ffc4 00405a14 0012ff88 Project1!Forms.TApplication.Run+0xce
1f 0012ff88 7610d0e9 7ffdd000 0012ffd4 77711603 Project1!Project1.Project1+0x4e
20 0012ff94 77711603 7ffdd000 7736d39e 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
21 0012ffd4 777115d6 004a9c48 7ffdd000 00000000 ntdll!__RtlUserThreadStart+0x23 (FPO: [Non-Fpo])
22 0012ffec 00000000 004a9c48 7ffdd000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

0:000> kvn
 # ChildEBP RetAddr  Args to Child              
00 0012ff30 774a5370 7747b148 ffffffff 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
01 0012ff34 7747b148 ffffffff 00000000 0012ff58 ntdll!ZwTerminateProcess+0xc (FPO: [2,0,0])
02 0012ff44 76f241ec 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x7a (FPO: [Non-Fpo])
03 0012ff58 00405f73 00000000 0012ff88 00000000 kernel32!ExitProcess+0x12 (FPO: [1,0,0])
04 0012ff70 004a9c9b 0012ffc4 00405a14 0012ff88 Project1!System.Halt0+0xf3
05 0012ff88 76f2d0e9 7ffd3000 0012ffd4 77481603 Project1!Project1.Project1+0x53
06 0012ff94 77481603 7ffd3000 77aa9471 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
07 0012ffd4 774815d6 004a9c48 7ffd3000 00000000 ntdll!__RtlUserThreadStart+0x23 (FPO: [Non-Fpo])
08 0012ffec 00000000 004a9c48 7ffd3000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

The only remaining thing is to analyse the crash dump trying to identify which are the regions involved in the crash and tackle them!.

Happy ninja debugging!.

Related links:


Post a Comment