I'm using windbg to examine some crash dumps sent in by an app. There seems to be some correlation between a crash I'm seeing and having a certain 3rd party DLL loaded into the process (a flaky Winsock LSP, I suspect). To make this sort of analysis easier in the future, is there a windbg script that would just show me a list of modules that are non-Microsoft? This would make patterns between crashes more obvious to me. I'm using "lm D sm", but going through the list manually right now is a pain.
Thanks!
Try using "lm e" with your symbol path set to Microsoft's symbol server (and with only MS symbols loaded). That will cause WinDbg to show a list of all modules with any sort of symbol "problem" including modules that have not been loaded.
The keys to making this work are:
The sympath is only set to use the MS symbol store (use ".symfix" to achieve this)
Symbols have been loaded using the above sympath
From there you can add on the other options for "lm" to get info like full path, etc.
You can use cdb to script the debugger, and it just prints to stdout - open the crash dump, have it print the loaded module list then exit, then you can use your favorite text manipulation tool (hint: its name is Perl ;) ) to search the list.
EDIT: Just to add some extra info, cdb is the command-line version of WinDbg; they both use the same engine, it's just a different frontend.
I'm not sure I see why you want to do this, but you can output from WinDbg to a log and correlate with a list of DLLs. That's pretty easy to do in any scripting language such as Perl, Python etc.
The way that I do this now is I run sos.dll from the CLR10 directory from the Debugging Tools for Windows installation.
.load clr10\sos
!sam c:\temp\modules
I open the directory c:\temp\modules in Windows Explorer. I right click in the Header column and I add the column for Company. Then I sort on company and move DLLs with Company of "Microsoft Corporation" into a separate subdirectory called "Microsoft"
Whatever DLLs are left in the Directory are usually 3rd Party or custom developed code.
Thanks,
Aaron
I wrote a small command-line app to solve exactly this problem a while back.
http://www.sleep1000.com/software/dumpmod
Related
I am modifying some old VB6 code and there are a number of calls to a subroutine called SDKPress(Index as integer, PressStatus as Integer, PressX as double, PressY as double, PressDataX as double, PressDataY as double).
(This routine returns the position of the mouse click and Index returns which graph on a form the mouse is in.)
Since there is no code for this in the project, I assume it is from some Windows software development kit. However, there is no trace of it on the development PC. just an SDK directory in the VisualBasic directory containing winsdk_web.exe.
I cannot call that subroutine from another project. Where does the subroutine live? Where can I find documentation on it? How can I get it in a form where I can use it on another project?
To find out where it lives. Put the cursor in one of the routine calls in the IDE, right-click on it, and choose go to definition. See where it takes you.
It may be an ordinary routine in a code file in your project.
It may be a routine in a DLL which is imported into your project via a Declare statement.
It may be a routine in a COM/ActiveX DLL or OCX control or even ActiveX EXE. In which case the IDE will show you the "object browser" window and you should be able to figure out where the routine lives, and go from there. I.e. figure out how to add a reference to it into another VB project, and how to install/register the relevant DLL/OCX/EXE on another PC.
OR, and I'm sure you tried this already, but just in case, search the source code for the routine name. Use Ctr-F or menu item Edit-Find in the VB IDE.
DLL files contains the API functions, vb6 code will load the DLL (Declare Function SDKPress Lib "DLLName" ....) and call the functions, use a tool like DllExpView: https://www.nirsoft.net/utils/dll_export_viewer.html and list DLLs and their functions. Or it also might be an internal subroutine/function.
If the original program which uses the SDK runs, you can gather some information using Process Monitor and Process Explorer. These can help you locate dependencies of many types.
I'd start with the second - when you run your program, ProcExp will show you all the files, including DLLs, that are loaded into the process. This alone might give you some insight.
If that is insufficient then use ProcMon to actively monitor the activities of your program. While ProcExp is basically a snapshot in time, ProcMon will dynamically record many of the actions of your program, including how it searches for and ultimately loads DLLs.
https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
I have found these tools to be invaluable on many occasions. They help to break down the "black box" around a running process which appears in many programmers minds.
Exactly much from the original build do I need to save in order to properly debug a crash dump file sent to me by a customer? Obviously I need the PDBs Do I need something else?
(This would be for a crash dump file written by the MiniDumpWriteDump function from dbghelp.dll.)
Until now I've always saved off the entire build folder. Code, PDBs, .OBJ files, output binaries, everything. Just to be safe. I'd like to minimimize what I save. But I can't afford to find out the hard way that I missed something.
The actual source code will be in source control and tagged with the build label so I can pull by label and get exactly what I used to build. Would I even need to bother pulling source before debugging the crash dump or is it enough to have just the PDBs?
From your company, PDBs should be all you need. Source code is helpful, too, because PDBs provide line numbers and you can have a direct look into the source code.
Typically you would not copy the PDBs, but set up a symbol server instead. You would also not copy the source code, but let a source server manage that. Team Foundation Server (TFS) provides both. But there are alternatives that do not need TFS.
In case of a .NET program, you also need the SOS.dll and MSCORDACWKS.DLL from the machine of the client where the crash occurred. having the correct version number (note that there may be several .NET frameworks installed).
SOS is the debugging extension for WinDbg that enables .NET debugging. MSCORDACWKS is the Data Access Component (DAC) that knows how objects in memory look like. (MS = Microsoft, COR = .NET, WKS = Workstation). There's a dedicated WinDbg command for that, .cordll.
I'm in the process of building a program that has several external libraries and extensions outside of the main program files. My projects total size is 134.2 MB. I would like to make a portable version of it with Turbo Studio, but I face one glaring problem; After capturing the files and building the project I'm left with a 138.9MB executable. The program does run, but I don't want it to be so bloated.
I would like to bundle the bulk of the data as a separate .dat payload (or equivalent) but I can't seem to find any means of doing this with Turbo Studio. VMware ThinApp and Cameyo both do this automatically after the project exceeds a certain size, but it would appear that Turbo Studio doesn't.
Any help with is greatly appreciated.
There was a hack I had discovered some time ago that did exactly what you were asking.
It appears that the site that described it is not online anymore so read on.
Install your application while monitoring with TurboStudio.
If there are multiple entry points (e.g., a suite like office may have
shortcuts for excel, word etc) then make sure there are shortcuts for those entry points in the Start Menu. If there are not, then click Start, right click on All Programs, then Open All Users. Make sure you create in that folder shortcuts for all your desired entry points (e.g., one shortcut for word, one for excel etc.)
Capture and Diff with TurboStudio. Set your virtualization settings as desired.
Click Output File-->Browse, and select "All files(.)" in the Save as type list. Then enter a filename with the extension .dat
Build your app. You will now get a .dat file instead of an .exe
In the next step you must use ExeBuilder.exe. This file was originally hosted in csgotwisted.com. Do a google search for "2 utilities for spoon studio exebuilder" and it will most likely be the first result. Unfortunately, the link is dead now when I click it. So I uploaded the file to NitroFlare. You can find it here. Put it in the same folder as the .dat file and run it. It will create a shortcut with your executable. Sometimes it misses the icon, but it gives you the option to locate it manually.
I use TurboStudio often and I have found this way to be the most quick and reliable in allowing me to generate small executables and storing the virtual filesystem and registry in a .dat file. In addition, it has the advantage that it can get you multiple entry points and not only one, just like Thinapp does.
EDIT: This is due to stupidity. It is a multiple monitor issue. It's just that from cmd.exe we always opened in the primary monitor, whilst from explorer, we always opened in the secondary. Thanks all for the help!
We hit a weird bug recently. We have a Qt + osg app that behaves differently if we run it from explorer than if we run it from a command line. Running from explorer is unusable, while running from command line (or by running from the explorer a simple batch file that calls the .exe) works as expected.
We suspect environment variables, because that's all we can think of. But the fact that it runs fine with a one line batch file seems to refute this. I'm not familiar enough with windows to know of any subtle differences in how it loads executables, nor where to look to find out.
Are there any other differences that could explain this? Does windows load different sets of user environment variables in each case? OS is Windows XP Service Pack 3.
The behavior experienced when running from explorer (double click program.exe) is consistent with a driver issue or improper OSG scene setup: image artifacts, flashing, and weird colors.
The behavior experienced when running the same executable from cmd.exe (or by double clicking a .bat file next to the .exe containing only a line to run the .exe) is the correct, expected behavior: the scene is correct, no flashing, etc.
To rule out potential library load path issues, try using dot-local DLL redirection.
Towards that end, create an (empty) file in the same directory as your executable and give it the same name as your binary, except with .local appended. I.e., if your binary is named yourbinary.exe, name that file yourbinary.exe.local. That will force the PE loader to first look in that directory to resolve LoadLibrary calls (and that includes DLLs loaded indirectly via system DLLs or via COM, no matter how many indirection levels are involved.) Place as many supporting DLLs (including Qt DLLs) in that directory. If you're using Qt plugins, also place the plugins directory there (or use a custom trolltech.conf.)
More details on dot-local redirection here, for example.
This thread looks like it might have the answer to your question:
http://forum.soft32.com/windows/Start-Run-Command-Prompt-ftopict353085.html
In short, I think it might be looking for your executable in different places depending on which method you attempt to use to run it. Perhaps you have 2 different versions hiding somewhere that explorer uses instead of the one you want?
You have not given enough details so I will give you a general answer. In order to use QT and its tools you need 2 environment variables. *QTDIR, and PATH * Make sure you have these variables set instructions are below. I have taken them from this site. See also this link for deployment on windows.
Setup the QTDIR environmental
variable.
1) Create a new System variable
called: QTDIR
a. Right click on My Computer -> Properties -> Advanced Tab ->
Environment Variables button
b. Find System variables -> New -> Type in "QTDIR" 2) Set the value to: C:\your\Qt\directory (NOTICE: No
trailing '\' character!!!)
Now, add the QTDIR on to your PATH
variable.
1) Edit your PATH variable, add onto
the end of it a ';' if one isn't
already on the end. 2) Now add on:
%QTDIR%\bin;
Example:
Before
PATH=%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;
After,
PATH=%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%QTDIR%\bin;
That will make sure that our Qt
application(s) will be able to find
the Qt DLL files when you go to run
it.
I hope this helps.
Perhaps there is a difference caused by the way Explorer launches an executable vs directly running it from the console. I think that Explorer uses ShellExecute or ShellExecuteEx and I doubt that executing an application from a console or batch file does the same.
I would create a test app that tries some of the ShellExecute variants and use different parameters to see if the Explorer behavior can be reproduced in order to try to diagnose what parameters passed to ShellExecute might be causing the problem.
There is an interesting community note on the ShellExecuteEx page that may or may not be applicable:
ShellExecuteEx ignores the current input desktop. It always uses winsta0\default. Instead use ShellExecute or CreateProcess.
I would also investigate whether or not AppCompatFlags affect console executed applications (or see if any AppCompatFlags have been set for your application).
Are there any VC++ settings I should know about to generate better PDB files that contain more information?
I have a crash dump analysis system in place based on the project crashrpt.
Also, my production build server has the source code installed on the D:\, but my development machine has the source code on the C:\. I entered the source path in the VC++ settings, but when looking through the call stack of a crash, it doesn't automatically jump to my source code. I believe if I had my dev machine's source code on the D:\ it would work.
"Are there any VC++ settings I should know about"
Make sure you turn off Frame pointer ommision. Larry osterman's blog has the historical details about fpo and the issues it causes with debugging.
Symbols are loaded successfully. It shows the callstack, but double clicking on an entry doesn't bring me to the source code.
What version of VS are you using? (Or are you using Windbg?) ... in VS it should defintely prompt for source the first time if it doesn't find the location. However it also keeps a list of source that was 'not found' so it doesn't ask you for it every time. Sometimes the don't look list is a pain ... to get the prompt back up you need to go to solution explorer/solution node/properties/debug properties and edit the file list in the lower pane.
Finally you might be using 'stripped symbols'. These are pdb files generated to provide debug info for walking the callstack past FPO, but with source locations stripped out (along with other data). The public symbols for windows OS components are stripped pdbs. For your own code these simply cause pain and are not worth it unless you are providing your pdbs to externals. How would you have one of these horrible stripped pdbs? You might have them if you use "binplace" with the -a command.
Good luck! A proper mini dump story is a godsend for production debugging.
If your build directly from your sourcecode management system, you should annotate your pdb files with the file origins. This allows you to automatically fetch the exact source files while debugging. (This is the same proces as used for retrieving the .Net framework sourcecode).
See http://msdn.microsoft.com/en-us/magazine/cc163563.aspx for more information. If you use subversion as your SCM you can check out the SourceServerSharp project.
You could trying using the MS-DOS subst command to assign your source code directory to the D: drive.
This is the procedure I used after some trouble similar to yours:
a) Copied to the production server all the EXE & DLL files that were built, each with its corresponding PDB to the same directory, started the system, and waited for the crash to happen.
b) Copied back all the EXE, DLL & PDB files to the development machine (to a temporary folder) along with the minidump (in the same folder). Used Visual Studio to load the minidump from that folder.
Since VS found the source files where they were originally compiled, it was always able to identify them and load them correctly. As with you, in the production machine the drive used was not C:, but in the development machine it was.
Two more tips:
One thing I did often was to copy an EXE/DLL rebuilt and forget to copy the new PDB. This ruined the debug cycle, VS would not be able to show me the call stack.
Sometimes, I got a call stack that didn't make sense in VS. After some headache, I discovered that windbg would always show me the correct stack, but VS often wouldn't. Don't know why.
In case anyone is interested, a co-worker replied to this question to me via email:
Artem wrote:
There is a flag to MiniDumpWriteDump()
that can do better crash dumps that
will allow seeing full program state,
with all global variables, etc. As for
call stacks, I doubt they can be
better because of optimizations...
unless you turn (maybe some)
optimizations off.
Also, I think disabling inline
functions and whole program
optimization will help quite a lot.
In fact, there are many dump types,
maybe you could choose one small
enough but still having more info
http://msdn.microsoft.com/en-us/library/ms680519(VS.85).aspx
Those types won't help with call stack
though, they only affect the amount of
variables you'll be able to see.
I noticed some of those dump types
aren't supported in dbghelp.dll
version 5.1 that we use. We could
update it to the newest, 6.9 version
though, I've just checked the EULA for
MS Debugging Tools -- the newest
dbghelp.dll is still ok to
redistribute.
Is Visual Studio prompting you for the path to the source file? If it isn't then it doesn't think it has symbols for the callstack. Setting the source path should work without having to map the exact original location.
You can tell if symbols are loaded by looking at the 'modules' window in Visual Studio.
Assuming you are building a PDB then I don't think there are any options that control the amount of information in the PDB directly. You can change the type of optimizations performed by the compiler to improve debuggabilty, but this will cost performance -- as your co-worker points out, disabling inline will help make things more obvious in the crash file, but will cost at runtime.
Depending on the nature of your application I would recommend working with full dump files if you can, they are bigger, but give you all the information about the process ... and how often does it crash anyway :)
Is Visual Studio prompting you for the
path to the source file?
No.
If it isn't then it doesn't think it has symbols
for the callstack. Setting the source
path should work without having to map
the exact original location.
Symbols are loaded successfully. It shows the callstack, but double clicking on an entry doesn't bring me to the source code. I can of course search in files for the line in question, but this is hard work :)