This is really driving me crazy. I am using WinDbg as my primary debugger.
It is used to debug local service (WinDbg running locally, debugging service on the same machine).
The PDB files are stored on local hard drive.
Source code is accessed via SMB share.
Debugging works in bursts, sometimes it flow well, most of the time I keep seeing the unbelievably annoying "*BUSY*" message, this happens almost every time when I perform a "step-over".
Any ideas what I could do to speed things up?
Thanks
This may happens if you have many unqualified breakpoints that track module load events (created via bu) saved in your workspace.
Also it is worth to check network connection and local symbols cache size
I was having the exact same problem, and was able to see a big improvement by adjusting the symbol options. Specifically, the SYMOPT_NO_PUBLICS option seemed to be the most important, but I adjusted a few other related options. I did the following...
.symopt-0x4
.symopt+0x100
.symopt+0x8000
.symopt-0x10000
...which is:
-SYMOPT_DEFERRED_LOADS
+SYMOPT_NO_UNQUALIFIED_LOADS
+SYMOPT_NO_PUBLICS
-SYMOPT_AUTO_PUBLICS
After all that, I had a symopt bit mask value of 0x80028333, which I now use as a WinDbg command line option, as in:
windbg.exe -sflags 0x80028333
I haven't yet discovered if there is any downside to this approach. Perhaps there are some cases where using SYMOPT_NO_PUBLICS results in missing information, but it's been working well for me thus far, and it's definitely way faster.
WinDbg Symbol Options MS Documentation
Usually this happened to me when multiple tool windows (locals, watches, call stack ) were opened. If those windows are opened, WinDbg will spend cycles to update all of them using slow symbols lookups on each 'step-over' command.
In this regard using just a command line ( ntsd ) debugger is much faster.
If it's slowly pulling Windows symbol files from the internet, consider downloading all of them to your hard drive and pointing WinDbg to their location on it. It would be best to have your service's symbol files and source files on the local drive too.
A couple of things to try:
Use the !sym noisy command to show more detailed information on the images and pdbs that windbg is referencing. You may see that it's accessing the network more than you're expecting, or that a network server is unavailable and requests are timing out
Ensure that your symbol path has got a cache in it
Take a copy of the sources locally and add to the source path
Instead of using 'step-over', set a breakpoint on the next command, or even a hardware breakpoint (using ba)
When your symbol path contains a large file store, WinDbg can take a long time searching for symbols that don't exist. You'll can diagnose this behavior by, as user eran suggested, by running Process Monitor and observing the file operations being made by WinDbg.
This was an extreme pain point for our organization, but I found a workaround detailed in this question: "WinDbg takes extremely long time to loading symbols; is searching every directory in large network UNC symbol store"
Related
In the past, I have heard references to parsing .dmp files using WinDbg (I think - I might be wrong).
I have also done fairly extensive debugging with the help of .map files, and I have done extensive debugging using standard logical heuristics and the Visual Studio debugger.
However, occasionally, the program I am developing crashes and creates a .dmp file. I have never been able to interpret the .dmp file. A while ago, I posted a SO question regarding how to interpret .dmp files ( How to view .dmp file on Windows 7? ), but after somewhat significant effort I was unable to figure out how to interpret .dmp files using the answer to that question.
Today, I was viewing an unrelated SO question ( C++ try/throw/catch => machine code ), and a useful comment underneath the accepted answer has, once again, made reference to WinDbg.
If you really want to find this out though, it's easy - just trace
through it in WinDbg
I would like to follow this advice. However, for me, it's not easy to "just trace through it in WinDbg". I've tried in the past and can't figure out what exactly this means or what to do!
So, I'm trying again. "For once and for all", I would like to have plain-and-simple instructions regarding:
What is WinDbg
Assuming WinDbg is related to .dmp files, what exactly is a dump file and how does it relate to WinDbg (and correct me if my assumption is wrong)
How do you create .dmp files and, correspondingly, how do you use WinDbg to analyze them (again, correct me if I'm wrong about the relationship between WinDbg and .dmp files).
If you could please answer this question from the "starting point" of a programmer who ONLY has Visual Studio installed and running.
Thanks!
WinDbg is a multipurpose debugger. It can debug a live process by attaching, set breakpoints, etc like you would with any other debugger. It can also analyze crash dump files as well, which are .dmp files. It functions by you giving it commands.
A .dmp file is a memory dump of something. What that something is depends on what the memory dump is for. It could be for a process, for example. It could also be for the kernel. What is in the memory dump depends, too. In your case, it's probably what your process looked like at the time of it crashing. What the memory dump contains can vary depending on the dump type.
There are various ways. On Windows Vista+, Server 2008+ - you can do it right from the task manager. Right click the process, and click "Create Memory Dump". WinDbg can make a memory dump from a live process too by using the .dump command. Other tools like adplus can be used to automatically create a memory dump on certain conditions, like when a process exceeds a memory or CPU threshold, or when it crashes.
WinDbg can open a Crash Dump easily enough. What is important is that you get your symbols loaded correctly, first. Usually in the form of .pdb files or from a symbol server (though not necessary, or always possible, it is greatly helpful).
Once you have WinDbg running, take a look at the list of commands available to poke around in your crash dump.
WinDbg is a Gui version of the command line debugger cdb.exe, both are user-process and kernel mode debuggers, it uses DbgHelp.dll to issue commands to your application or NT kernel (you can also do the same as it has an api).
.Dmp files are memory dumps of varying detail, some can have minimal detail enough for call stacks of all threads, whilst others will put the entire user-mode memory, handle information, thread information, memory information etc.. see this for more info. So dump files have nothing to do with WinDbg, other than it can open them, incidentally you can open .dmp files in Visual Studio
Like #vcsjones has already stated you can do this using task manager (at least you can from Vista onwards), you can use procdump, you can do this once WinDbg is attached, I usually do a full mini dump like this: .dump /ma c:\mem.dmp, you can also set Windows to do this when a crash happens using Dr. Watson
However, you must have the symbols for Windows and your application in order to be able to generate sensible call stacks, note that for obvious reasons you cannot step through or set breakpoints in a a memory dump, you can only do this for a live process. You can also have WinDbg attach non invasively, so Visual Studio could be attached and you can attach WinDbg non invasively and use the toolset in WinDbg to assist debugging.
For me the main advantage of WinDbg is its free, it is a small download and install, it is fast, it has a very rich toolset for diagnosing problems that are either difficult or impossible to do using visual studio.
What is the need/use of 'symbols' in the Microsoft debugger?
I spent some time trying to figure out the debugger a while back and never was able to get it making any sense (I was trying to debug a server hang...). Part of my problem was not having the proper 'symbols'.
What are they? And why would I need them? Aren't I just looking for text?
Are there any better links out there to using it than How to solve Windows system crashes in minutes ?
You need symbols in order to translate addresses into meaningful names. For example, you have locations on your stack at every function call:
0x00003791
0x00004a42
Symbols allows the debugger to map these addresses to methods
0x00003791 myprog!methodnamea
0x00004a42 myprog!methodnameb
When you build a debug version of a program, the compiler emits symbols with the extension .PDB. It also contains line information so you can do source code debugging, etc..
You need to set your symbol search path correctly for the debugger to pick this up. IN the command window you can do
.sympath c:\symbols;c:\temp\symbols
in order to have it search for the .PDB in these directories. It will also look in the same directory that the executable is ran from.
It also might be helpful to use the Microsoft public symbols server so that you can resolve OS binaries such as NTDLL, GDI, etc.. with this path at the beginning:
.sympath SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols;c:\symbols
You will need to create c:\websymbols first.
On the Windows binary architecture, the information needed for debugging (function names, file and line numbers, etc.) aren't present in the binary itself. Rather, they're collected into a PDB file (Program DataBase, file extension .pdb), which the debugger uses to correlate binary instructions with the sorts of information you probably use while debugging.
So in order to debug a server hang, you need the PDB file both for the server application itself, and optionally for the Windows binaries that your server is calling into.
As a general note, my experience with WinDbg is that it was much, much harder to learn how to use compared to GDB, but that it had much greater power once you understood how to use it. (The opposite of the usual case with Windows/Linux tools, interestingly.)
If you just have the binary file, the only info you can typically get is the stack trace, and maybe the binary or IL(in .NET) instructions. Having the symbols lets you actually match that binary/IL instruction up with a corresponding line in the source code. If you have the source code, it also lets you hook up the debugger in Visual Studio and step through the source code.
Well, fortunately I haven't written many applications that cause a BSOD but I just wonder about the usefullness of the information on this screen. Does it contain any useful information that could help me to find the error in my code? If so, what do I need, exactly?
And then, the system restarts and probably has written some error log or other information to the system somewhere. Where is it, what does it contain and how do I use it to improve my code?
I did get a BSOD regularly in the past when I was interacting with a PBX system where the amount of documentation of it's drivers were just absent, so I had to do some trial-and-error coding. Fortunately, I now work for a different company and don't see any BSOD's as a result of my code.
If you want a fairly easy way to find out what caused an OS crash that will work ~90% of the time - assuming you have a crash dump available - then try the following:
Download WinDbg as part of the Debugging tools for Windows package. Note, you only need to install the component called Debugging Tools for Windows.
Run WinDbg
Select "Open Crash Dump" from the file menu
When the dump file has loaded type analyze -v and press enter
WinDbg will do an automated analysis of the crash and will provide a huge amount of information on the system state at the time of the crash. It will usually be able to tell you which module was at fault and what type of error caused the crash. You should also get a stack trace that may or may not be helpful to you.
Another useful command is kbwhich prints out a stack trace. In that list, look for a line contains .sys. This is normally the driver which caused the crash.
Note that you will have to configure symbols in WinDbg if you want the stack trace to give you function names. To do this:
Create a folder such as C:\symbols
In WinDbg, open File -> Symbol File Path
Add: SRV*C:\symbols*http://msdl.microsoft.com/download/symbols
This will cache symbol files from Microsoft's servers.
If the automated analysis is not sufficient then there are a variety of commands that WinDbg provides to enable you to work out exactly what was happening at the time of the crash. The help file is a good place to start in this scenario.
Generally speaking, you cannot cause a OS crash or bug check from within your application code. That said, if you are looking for general tips and stuff, I recommend the NTDebugging blog. Most of the stuff is way over my head.
What happens when the OS crashes is it will write a kernel dump file, depending on the current flags and so on, you get more or less info in it. You can load up the dump file in windbg or some other debugger. Windbg has the useful !analyze command, which will examine the dump file and give you hints on the bucket the crash fell into, and the possible culprits. Also check the windbg documentation on the general cause of the bug check, and what you can do to resolve it.
I have a VB6 program running on Windows 7. It is copying a large number of files and sometimes FileCopy fails with an access violation (between every 60 and 500 files).
I cannot reproduce it using a single file, only during such mass-copying operations this problem happens.
It makes no difference, if source/target are on hard disks, network shares or CD-ROMs.
What could trigger this problem?
EDIT: My question might be a little bit convoluted, so here's some more data:
Run 1:
Start copying 5.000 files
Access violation on file #983
Access violation on file #1437
Access violation on file #1499
Access violation on file #2132
Access violation on file #3456
Access violation on file #4320
Done
Run 2:
Start copying 5.000 files
Access violation on file #60
Access violation on file #3745
Done
Observations
The affected files are always different
The number of affected files tends to decrease if the same file batch is copied multiple times in succession.
Running as Administrator makes no difference
The application has read/write access to all necessary file system objects
This problem happens on Windows 7 workstations only!
Best guess: Is it possible that another user/application is using the specified file at the time the process is running? (anti-virus scanner, Win7 search indexing tool, windows defender, etc) You might try booting the machine in safe-mood to eliminate any of the background services/apps and try running the process to see.
Is there any consistency in the file types or size of the files causing the issue?
Is the machine low on resources? RAM/Disk Space
You said it occurs on Win7 – is it multiple Win7 machines or just one. (help to rule out system resources vs. software/OS)
Any hints from the event viewer (control panel > admin tools) – doubtful
Does the process take a long time to complete? If you can take the performance hit you might look at destroying and recreating the FSO object after every copy or every X files to make sure there isn’t some odd memory leak issue with Win7/VB6.
Not necessarily a recommended solution but if all else fails you could handle that error and save the files that trigger it in a dictionary/collection and reloop through the process with any those files when done. No guarantee it wouldn’t happen again.
Not enough information (as you probably know). Do you log the activity? If not, it's a good place to start. Knowing whether certain files are the problem, and if the issue is repeatable, can help narrow it down.
In your case I would also trap (and log) all errors and retry N times after waiting N seconds. You could be trying to copy in-use files locked by another process, and a retry may allow time for that lock to go away.
Really, more data is the key, and logging is the way to get it.
Is there any chance your antivirus program or some indexer is getting in the way?
Try creating a procmon trace while reproducing the error and see what is actually failing. With the trace you can see if there is another program causing the issue or if your app is trying to write somewhere it should't (incorrect permissions) or can't (a temp/scratch directory without enough space).
Check out the presentations linked to on the procmon page or Mark Russinovich's blog for some cool examples of using this tool to solve various Windows/application mysteries.
Is there a a hidden/system file in the directory that is potentially blocking it?
Does running the VB6 App with right-click "Run As Administrator" make a difference?
Is the point where it dies at the max # of files in the directory? e.g. Are you sure the upper limit on whatever loop structure you are using in VB6 is correct (Count vs count -1)?
I'm compiling a vc8 C++ project in a WinXp VmWare session. It's a hell of a lot slower than gcc3.2 in a RedHat VmWare session, so I'm looking at Task Manager. It's saying a very large percentage of my compile process is spent in the kernel. That doesn't sounds right to me.
Is there an equivalent of strace for Win32? At least something which will give me an overview of which kernel functions are being called. There might be something that stands out as being the culprit.
Windows Resource Kit contains a tool called kernrate. It's a sampling profiler. It can profile entire system or a particular process. By default, its resolution is on a module level, but can be tuned down to several bytes. You should be fine with default resolution as you'll see which modules/drivers are consuming most of the time.
Here is some info regarding its use.
Not exactly strace, but there is a way of getting visibility into the kernel call stack, and by sampling it at times of high CPU usage, you can usually estimate what's using up all the time.
Install Process Explorer and make sure you configure it with symbol server support. You can do this by:
Installing WinDebug to get an updated dbghelp.dll
Set Process Explorer to use this version of dbghelp.dll by setting the path in the Options | Configure Symbols menu of Process Explorer.
Also in the same dialog, set the symbols path such that it includes the MS symbol server and a local cache.
Here's an example value for the symbol path:
SRV*C:\symbolcache*http://msdl.microsoft.com/download/symbols
(You can set _NT_SYMBOL_PATH environment variable to the same value to have the debugging tools use the same symbol server and cache path.) This path will cause dbghelp.dll to download symbols to local disk when asked for symbols for a module that doesn't have symbols locally.
After having set up Process Explorer like this, you can then get a process's properties, go to the threads tab, and double-click on the busiest thread. This will cause Process Explorer to temporarily hook into the process and scan the thread's stack, and then go and look up the symbols for the various return addresses on the stack. The return addresses's symbols, and the module names (for non-MS third-party drivers) should give you a strong clue as to where your CPU time is being spent.
VmWare support should be address that question. It's probably somewhere in the VmWare implementation.
You can use for example IrpTracker that give you an idea what is going on in the kernel.
Another option is using kernel debugger i.e WinDbg. If the cpu load very high just randomly breaking in the debugger and looking on the call stack can give you an idea who is the driver behind the cpu load. But as i stated i will guess that it will be some VmWare component. It worth to check if the problem persist on same computer on WinXP without emulation.