I have an application, that I want to profile using Windows Performance Analyzer. It all works, but I don't get any reasonable stack traces from my application.
The application in question is a demo application. This is to give me a good feeling if all checks out. Then I want to profile another application. Since I have full control over my demo application, I included some marker functions, that should show up in the stack trace.
When running the application on Windwos 71, Process Explorer shows the correct stack trace for the part, that I want to profile. Here is the stack trace with the marker functions in lines 7 - 9:
Since I installed all performance analytics tools insinde a Windows 10 VM2, I started profiling there. The first thing to notice: Process Explorer does not show the correct stack trace. The marker functions that I implemented are nowhere to be found.
Nevertheless, I recorded performance traces using UIforETW and Windows Performance Recorder. When opening them in WPA and focussing on the target application, this is the stack trace:
All the information, I'm interested in is missing. The stack shows up as <Application>.exe!<Missing ImageId event>
What did I do wrong?
If this gives you a hint, here is the relevant software, that is installed:
1: The Windows 7 computer has Visual Studio (C#) installed.
2: The Windows 10 VM dowsn't have Visual Studio, but has WinDBG (Preview) and Windows Performance Toolkit installed.
I tagged delphi, because the target application is written in Delphi.
The Windows 10 WPA (as well as Windows 8.1, to a lesser extent) dropped support for older debug symbol formats; it now only supports the "RSDS" format that has been standard since MSVC 7. PE files using older symbol file formats (for example, VB6 generates NB10 PDB files) will result in that "Missing ImageId event" error.
(The message itself is technically incorrect; there likely is an ImageId even in the trace file but it is looking for an ImageId/DbgID_RSDS event, which can't be generated for non-RSDS PDBs)
<Missing ImageId event> will also be reported when the session is not merged with the "NT Kernel Logger" which provides some information necessary to resolve the symbols.
The "proper" way to stop the session is:
xperf.exe -stop my_trace -stop -d merged_trace.etl
Note that second -stop in necessary to stop another session (implicitly "NT Kernel Logger"), and -d to merge both into the merged_trace.etl.
Related
My scenario is I'm supporting a VB6 app at the place I work and in the last few weeks it has started crashing more often than it ever used to. It uses both a local Access MDB database and a remote SQL Server DB for different types of storage. The good news is we are writing a replacement app, the band news I need to support this one in the meantime and the vendor is long gone from this world.
What are some ways I could try and diagnose what is causing the crash? For example so far I've tried ODBC tracing (For the MDB component), SQL Profiler tracing and ProcMon on a client PC.
Is there anything else I could try to discover what the app was trying to do at the time of the crash?
You can also start in a debugger.
windbg or ntsd (ntsd is a console program and maybe installed). Both are also from Debugging Tools For Windows.
Download and install Debugging Tools for Windows
http://msdn.microsoft.com/en-us/windows/hardware/hh852363
Install the Windows SDK but just choose the debugging tools.
Create a folder called Symbols in C:\
Start Windbg. File menu - Symbol File Path and enter
srv*C:\symbols*http://msdl.microsoft.com/download/symbols
then
windbg -o -g -G c:\windows\system32\cmd.exe /k batfile.bat
You can press F12 to stop it and kb will show the call stack (g continues the program). If there's errors it will also stop and show them.
Type lm to list loaded modules, x *!* to list the symbols and bp symbolname to set a breakpoint
Use db address (as in db 01244 to see what's at that memory.
If programming in VB6 then this environmental variable link=/pdb:none stores the symbols in the dll rather than seperate files. Make sure you compile the program with No Optimisations and tick the box for Create Symbolic Debug Info. Both on the Compile tab in the Project's Properties.
I have a completely random error popping up on a particular piece of software out in the field. The application is a game written in VB6 and is running on Windows 7 64-bit. Every once in a while, the app crashes, with a generic "program.exe has stopped responding" message box. This game can run fine for days on end until this message appears, or within a matter of hours. No exception is being thrown.
We run this app in Windows 2000 compatibility mode (this was its original OS), with visual themes disabled, and as an administrator. The app itself is purposely simple in terms of using external components and API calls.
References:
Visual Basic for Applications
Visual Basic runtime objects and procedures
Visual Basic objects and procedures
OLE Automation
Microsoft DAO 3.51 Object Library
Microsoft Data Formatting Object Library
Components:
Microsoft Comm Control 6.0
Microsoft Windows Common Controls 6.0 (SP6)
Resizer XT
As you can see, these are pretty straightforward, Microsoft-standard tools, for the most part. The database components exist to interact with an Access database used for bookkeeping, and the Resizer XT was inserted to move this game more easily from its original 800x600 resolution to 1920x1080.
There is no networking enabled on the kiosks; no network drivers, and hence no connections to remote databases. Everything is encapsulated in a single box.
In the Windows Application event log, when this happens, there is an Event ID 1000 faulting a seemingly random module -- so far, either ntdll.dll or lpk.dll. In terms of API calls, I don't see any from ntdll.dll. We are using kernel32, user32, and winmm, for various file system and sound functions. I can't reproduce as it is completely random, so I don't even know where to start troubleshooting. Any ideas?
EDIT: A little more info. I've tried several different versions of Dependency Walker, at the suggestion of some other developers, and the latest version shows that I am missing IESHIMS.dll and GRPSVC.dll (these two seems to be well-known bugs in Depends.exe), and that I have missing symbols in COMCTRL32.dll and IEFRAME.dll. Any clues there?
The message from the application event log isn't that useful - what you need is a post mortem process dump from your process - so you can see where in your code things started going wrong.
Every time I've seen one of these problems it generally comes down to a bad API parameter rather than something more exotic, this may be caused by bad data coming in, but usually it's a good ol fashioned bug that causes the problem.
As you've probably figured already this isn't going to be easy to debug; ideally you'd have a repeatable failure case to debug, instead of relying on capturing dump files from a remote machine, but until you can make it repeatable remote dumps are the only way forwards.
Dr Watson used to do this, but is no longer shipped, so the alternatives are:
How to use the Userdump.exe tool to create a dump file
Sysinternals ProcDump
Collecting User-Mode dumps
What you need to get is a minidump, these contain the important parts of the process space, excluding standard modules (e.g. Kernel32.dll) - and replacing the dump with a version number.
There are instructions for Automatically Capturing a Dump When a Process Crashes - which uses cdb.exe shipped with the debugging tools, however the crucial item is the registry key \\HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug
You can change your code to add better error handling - especially useful if you can narrow down the cause to a few procedures and use the techniques described in Using symbolic debug information to locate a program crash. to directly process the map files.
Once you've got a minidump and the symbol files WinDbg is the tool of choice for digging into these dumps - however it can be a bit of a voyage to discover what the cause is.
The only other thing I'd consider, and this depends on your application structure, is to attempt to capture all input events for replay.
Another option is to find a copy of VMWare 7.1 which has replay debugging and use that as the first step in capturing a reproducible set of steps.
Right click your executable object and let it be WINXP compatible pending
when you discover source of the problem to finally solve it
I'm writing a Windows C++ program in Visual Studio 2010 and have heap corruption. The !heap and !avrf debugger extensions from Application Verifier have several tools that sound quite useful for tracking this down, however I am unclear as to how to effectively use Application Verifier. For instance, Application Verifier documentation states:
You can use !avrf -hp -a Address to display all heap space operations that affected the specified Address.
Where exactly does one type this command? (Yes, I know Address needs to be replaced by an actual address.) Does it need to be prefixed by anything?
It's a Windows debugger command. You can see the help for the command here.
My winform application crashes as soon as its been launched. This problem is only happening in one of the client computer. Works fine for all other clients. I was thinking of using ADPlus to get a crash dump but problem is in order to configure ADPlus in crash mode, debugger need to be attached to running application first. That means application must be running when I configure ADPlus, however, as I said my application crashes as soon as I start it. It does not give me a chance to run ADPlus. Any idea if ADPlus can be used in this scenario? Are there any other tools that I can use generate a process dump in this case?
Since you can't start the program and then attach a debugger in time, you can try one of the following ideas:
Start the program under the debugger on the machine where it's crashing. As already mentioned, ProcDump is enough for that purpose, or you can use Visual Studio or WinDBG.
If you can't easily start the program under the debugger (for example, if it is actually a Windows service), use gflags to make Windows start the program under the debugger. This will create a subkey for your program filename under the Image File Execution Options registry key.
Set your debugger as the postmortem debugger, so it always launches when any program crashes.
If and when the crash happens, the debugger will automatically break in and either create a dump automatically or let you create one manually.
If you can't run a debugger, not even ProcDump, you can use Windows' built-in crash dump facility to create a dump automatically:
On Windows XP and Windows Server 2003, set Dr. Watson as the postmortem debugger by running this command:
drwtsn32 -i
Here is info on what this command does and how to use Dr. Watson: Capturing Application Crash Dumps
On Windows Vista SP1, Windows Server 2008, and later, set Windows Error Reporting to save dumps locally by creating this registry key:
HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
You don't need to create or set any values or subkeys under this key. If you want to change settings, here is more info: Collecting User-Mode Dumps
Note that "applications that do their own custom crash reporting, including .NET applications, are not supported by" WER. This is a problem for you because your app uses WinForms - although there is some indication that the feature does work with .NET 4 apps on Windows 7.
You can use ProcDump from Sysinternals to capture a dump.
I've got a whole bunch of minidumps which were recorded during the runtime of an application through MiniDumpWriteDump. The minidumps were created on a machine with a different OS version than my development machine.
Now I'm trying to write a program to extract stack traces from the minidumps, using dbghelp.dll. I'm walking the MINIDUMP_MODULE_LIST and call SymLoadModule64, but this fails to download the pdbs (kernel32 etc.) from the public symbol server. If I add "C:\Windows\System32" to the symbol path it finds the dlls and downloads the symbols, but of course they don't match the dlls from the minidump, so the results are useless.
So how do I tell dbghelp.dll to download and use the proper pdbs?
[edit]
I forgot to state that SymLoadModule64 only takes a filename and no version/checksum information, so obviously with SymLoadModule64 alone it's impossible for dbghelp to figure out which pdb to download.
The information is actually available in the MINIDUMP_MODULE_LIST but I don't know how to pass it back to the dbghelp API.
There is SymLoadModuleEx which takes additional parameters, but I have no idea if that's what I need or what I should pass for the additional parameters.
[edit]
No luck so far, though I've noticed there's also dbgeng.dll distributed together with dbghelp.dll in the debugging SDK. MSDN looks quite well documented and says it's the same engine as windbg uses. Maybe I can use that to extract the stack traces.
If anyone can point me to some introduction to using dbgeng.dll to process minidumps that would probably help too, as the MSDN documents only the individual components but not how they work together.
Just in case anyone else wants to automate extracting stack traces from dumps, here's what I ended up doing:
Like I mentioned in the update it's possible to use dbgeng.dll instead of dbghelp.dll, which seems to be the same engine WinDbg uses. After some trial and error here's how to get a good stack trace with the same symbol loading mechanism as WinDbg.
call DebugCreate to get an instance of the debug engine
query for IDebugClient4, IDebugControl4, IDebugSymbols3
use IDebugSymbols3.SetSymbolOptions to configure how symbols are loaded (see MSDN for the options WinDbg uses)
use IDebugSymbols3.SetSymbolPath to set the symbol path like you would do in WinDbg
use IDebugClient4.OpenDumpFileWide to open the dump
use IDebugControl4.WaitForEvent to wait until the dump is loaded
use IDebugSymbols3.SetScopeFromStoredEvent to select the exception stored in the dump
use IDebugControl4.GetStackTrace to fetch the last few stack frames
use IDebugClient4.SetOutputCallbacks to register a listener receiving the decoded stack trace
use IDebugControl4.OutputStackTrace to process the stack frames
use IDebugClient4.SetOutputCallbacks to unregister the callback
release the interfaces
The call to WaitForEvent seems to be important because without it the following calls fail to extract the stack trace.
Also there still seems to be some memory leak in there, can't tell if it's me not cleaning up properly or something internal to dbgeng.dll, but I can just restart the process every 20 dumps or so, so I didn't investigate more.
An easy way to automate the analysis of multiple minidump files is to use the scripts written by John Robbins in his article "Automating Analyzing Tons Of Minidump Files With WinDBG And PowerShell" (you can grab the code on GitHub).
This is easy to tweak to have it perform whatever WinDbg commands you'd like it to, if the default setup is not sufficient.