When debugging with GDB during one debug session it becomes slower and slower over time. Even simplest operations like step over and step into can take dozens of seconds and sometimes even minutes.
I was debugging a rather big project (Chromium browser). The only reason I could think of was that gdb is getting slower over time because it loads more and more symbols and it takes longer to work with them. However Chromium compiles entire code into one huge executable, which contains all symbols which should be loaded in the very beginning. Thus symbol database won't grow during the debugging. Moreover why would one need to look up symbols just to perform step over or step into operation?
While testing I have tried using gdb with front-ends (Eclipse, QtCreator, Emacs) and from the command line to confirm that this is not an IDE problem. Both use cases demonstrate same problem, however it seems like it starts to appear sooner in IDEs (probably because IDE also loads symbols for the watch view, call stack, list of threads etc.).
Why is GDB getting slower? Is it a design flaw, a bug or some specific problem in my computer? Are there any free alternatives to GDB that work faster?
Why is GDB getting slower?
It's a bug. Try newer version of GDB (preferably current CVS snapshot). If the problem is still there, report it to GDB bugzilla with repro instructions.
all symbols which should be loaded in the very beginning.
GDB loads partial symbols (psymbols) on startup, and reads more "on-demand", so some growth is expected.
why would one need to look up symbols just to perform step over or step into
In order to step over or into, GDB would likely need line tables for the current translation unit (TU). If your "step into" operation takes you to a new TU, then new line tables would have to be loaded.
Still, it should not take GDB anywhere near minutes to step or next.
Related
Whenever I get a crash dump for an x64 release build of my application I find its rare I can view the locals, this makes it hard or impossible to fix some issues. In x86 I can usually view all locals without any issues.
Are there any compiler options in the release build that will allow me to view the locals in release build crash dumps? Obviously I don't want to turn off optimizations but perhaps there is some way to force it to save the locals with minor performance impact?
You've said a couple things that hint at why you can't see locals...
#1 - It's a release build.
With certain optimizations turned on, the compiler is free to do a few things that make looking at locals more difficult.
It can inline a function. When this happens, the locals of the function that aren't optimized away are mixed with the calling stack frame.
It can free up a register and save a couple clock cycles on the function call using a trick called Frame-Pointer Omission.
In order to save stack space, the compiler can reuse the location that held a variable earlier in the function body to hold a different variable later in the function. This means that where you are in the function determines which locals you are actually able to see.
#2 - It's an x64 build.
MSVC uses a new calling convention for 64-bit code aptly called the x64 Calling Convention. The first 4 function arguments are stored in registers instead of on the stack. This means that even though you are looking at a stack frame, you will not see some of the arguments and you may not even be able to recover them if the registers have been reused for something else by the time you look at them.
So, now what?
Now that we know why you are going to have such a difficult time, let's see what you can do to get around the issues above. None of those issues are really show stoppers, but they all work together to make things just that much more difficult for you.
Turn off some optimizations. You can try making with a release build with optimizations at a level that doesn't impede debugging quite so much. You would probably want to start with the optimizations mentioned above that play with stack frames (/Oy and /Ob). Then you would need to hope that you can still reproduce the issue with those optimizations turned off. Also, depending on your internal policy and the contract that you have with your customer, you may need to involve a lawyer before sending an unofficial build to the customer -- probably not the most fun thing in the world.
Build a better symbol file. VS2012 and higher has a new compiler switch, /d2Zi+ in VS2012 and /Zo in VS2013, that generates better debug info when optimizations are turned on. This puts debugging optimized code on par with GCC/Clang. Even though it is undocumented in VS2012, I'd still consider it pretty safe since I saw no difference in the generated code -- only in the symbol file. You may even be able to rebuild locally with this flag and force windbg to use your new symbol file via .symopt+ 0x40. This gives you a chance to get more out of the dumps you already have.
Use windbg extensions to do the heavy lifting. In other StackOverflow answers, I've mentioned a tool called CMKD that has saved my bacon a couple times. It, among other things, attempts to reconstruct the arguments in the x64 calling convention that were passed in registers. It's not a sure thing, but it is probably the best hope of getting them back.
Anyway, I hope my ramblings will prove helpful in your debugging.
after GDB 7.0, the Reverse Debugging is supported.
when a core dump is generated, can I use Reverse Debugging commands?
How can I do that?
You can't. A core file is a snapshot of a program's state at a point in time. To move backwards in that state, you'd need an earlier snapshot of the program state. GDB can do this by keeping this history when you're debugging a live program, but this info is not present in a core file.
Please see this question for a full description of how reverse debugging works in GDB.
See http://jakob.engbloms.se/archives/1547 for a more in-depth discussion on reverse execution. Fundamentally, you need to have seen the past to get back to it, you cannot semantically reverse a computer program. That said, given a more capable reverse system than gdb, you can indeed reverse execute your way out of a program crash or even system crash. It is just a matter of having the reversible debugger applied when you run the failing case.
I have to debug a very big program, which takes around 10 minutes until it reaches the most important debugging state. I just want to modify some values in this part of the program, but sometimes I would like to go back and modify them again, like travelling to the past. As far as I know it is called historical debugging. Reading some info it seems it has been implemented in Visual Studio 2010. But I only use Eclipse or Xcode or vi :)
I wonder if know some other software with these capabilites.
By the way, I ask about your opinion, do you think it will be possible, once I reach this state of my program, to modify some small part of the code, after the breakpoint, and compile it again (supposing it does not affect to the past execution and code) so I can test it without recompiling?
Thanks
Also discussed here
Mostly only available for interpreted languages like Java (ODB). And I am not sure if you can continue from some point of execution with changed data.
Have you tried to set watch point in Eclipse that would break as soon as a variable reach a specific value? This means your code executes normally until it breaks your data and execution stops so that you can see how you code got to this point.
I have a application which randomly freezes, including the IDE and it's driving me mad. That makes me wonder:
What's a good general strategy for finding the cause of random freezes?
If you are wanting to check from outside of a running app then I would potentially use the sysinternals.com toolset from Mark Russonivich, the perfmon tool allows you to trace file / registry access and check the trace for delays - and what is being accessed at that time. It will show the DLL call stack at that time with the right symbols can is useful for debugging problems external to an application that are causing delays. (I've used it to find out that an I/O filter associated to a security suite was the reason an application was piccking up a number of 1.5sec delays.)
If you're lucky, you can run your code in a debugger until it freezes, then stop the debugger to find the offending line of code. But if it were that easy, you probably wouldn't be asking for advice. :-)
Two strategies that can be used together are to "divide and conquer" and "leave bread crumbs."
Divide and conquer: Comment out increasingly larger portions of your code. If it still freezes, you've reduced the amount of code that might be responsible for causing the freeze. Caveat: eventually you'll comment out some code and the program will not freeze. This doesn't mean that last bit of code is necessarily responsible for the freeze; it's just somehow involved. Put it back and comment out something else.
Leave bread crumbs: Make your program tell you where it is and what it's doing as it executes. Display a message, add to a log file, make a sound, or send a packet over the network. Is the execution path as you expected? What was the last thing it was doing before it froze? Again, be aware that the last message may have come from a different thread than the one responsible for freezing the program, but as you get closer to the cause you'll adjust what and where the code logs.
You're probably doing things in the UI thread when you shouldn't be.
I would install the UserDump tool, and follow these instructions for generating a user dump of the application....
Once you have the user dump, you can use WinDbg, or cdb to inspect the threads, stacks, and locks, etc.
Often I find hangs are caused by locked mutexes or things like that.
The good general strategy is, run the program until it hangs. Then attach a debugger to it and see what's going on. In a GUI program, you're most interested in what the UI thread is doing.
You say the application hangs the IDE. This isn't supposed to happen, and I imagine it means the program is putting so much strain on the OS (perhaps CPU load or memory) that the whole system is struggling.
Try running it until it hangs, going back to the IDE, and clicking the Stop button. You may have to be really patient. If the IDE is really permanently stuck, then you'll have to give more details about your situation to get useful help.
Im sure this has happened to folks before, something works in debug mode, you compile in release, and something breaks.
This happened to me while working on a Embedded XP environment, the best way i found to do it really was to write a log file to determine where it would go wrong.
What are your experiences/ discoveries trying to tackle an annoying Release-mode bug?
Make sure you have good debug symbols available (you can do this even with a release build, even on embedded devices). You should be able to get a stack trace and hopefully the values of some variables. A good knowledge of assembly language is probably also useful at this point.
My experience is that generally the bug is related to code that is near the area of breakage. That is to say, if you are seeing an issue arising in the function "LoadConfigInfoFromFile" then probably you should start by closely analysing that for issues, rather than "DrawControlsOnScreen", if you know what I mean. "Spooky action at a distance" type bugs do not tend to arise often (although when they do, they tend to be a major bear).
Tracefile is always a good idea.
When it's about crashes, I'm using adplus, which is part of debugging tools for windows. basically what adplus does, is, it attaches windbg to the executable you're monitoring. When the application crashes, you get a crash dump and a log file. You can load the crash dump in your preferred debugger and find out, which instruction lead to the crash.
As release builds are heavily optimized compared to debug builds, the way you compile your code affects its behaviour. This is basically true when crashes in multithreaded code happen in the release version but not the debug version. adplus and windbg helped me, to find out, where this happened.
ADPlus is explained here:
httx://support.microsoft.com/?scid=kb%3Ben-us%3B286350&x=15&y=12
Basically what you have to do is:
1. Download and install WinDbg into C:\debuggers
httx://www.microsoft.com/whdc/devtools/debugging/default.mspx
Start your application
open a cmd and cd to c:\debuggers
start adplus like this:
"adplus.bat -crash your_exe.exe"
reproduce the crash
analyze the crashdump in vs2005 or in windbg
If it's only a small portion of the application that needs debugging then you can change those source files only to be built without optimisations. Presumably you generate debug info for all builds, and so this makes the application run mostly as it would in release, but allows you to debug the interesting parts properly.
How about using Trace statements. They are there for Release mode value checking.
Trace.WriteLine(myVar);
I agree on log file debugging to narrow it down.
I've used "Entering FunctionName" "Leaving FunctionName" until I can find what method it enters before the crash. Then I add more log messages re-compile and re-release.
Besides playing with turning off optimization and/or turning on debug information for your Release build as pauldoo said, a log file will good data can really help. I once wrote a "trace" app that would capture trace logs for the app if it was running when the release build started (otherwise the results would go to the debugger's output window if running under the debugger). I was able to have end-users email me log files from them reproducing the bugs they were seeing, and it was the only way I would have found the problem in at least one case.
Though it's probably not usable in an embedded environment, I've had good luck with WinDbg for debugging release-mode Windows applications. Even if the application is not compiled with symbol information, you can at least get a usable stack trace and plenty of other useful crash information.
You could also copy your debug symbols to the production environment even if it's compiled in relase mode
Here's an article with more information
If you problem is synchronization related dumping log in the file might be problematic.
In this case i usually will use some big array of string and dump this to screen/file after the problem was reproduces.
This is of course depend on your memory restriction, sometime i use just few symbols and numbers to store in the array if the memory on the platform is limited. Reading such logs is not a big pleasure, but sometimes this is the only choice.