How can I recognize that the callstack that is shown by the debugger when my program crashes may be wrong and misleading. For example when the callstack says the following frames may be missing or incorrect, what that actually means? Also what the + number after the function call in the callstack means :
kernel32!LoadLibrary + 0x100 bytes
Should this number be important to me, and is it true that if this number is big the callstack may be incorrect ?
Sorry if I am asking something trivial and obvious
Thank you all
Generally, you can trust your callstack to be correct.
However, if you re-throw exceptions explicitly instead of allowing them to bubble up the callstack naturally, the actual error can be hidden from the stack trace.
To start with the 2nd one: kernel32!LoadLibrary + 0x100 bytes means that the call was from the function LoadLibrary (offset: +100 bytes); appearantly there was no symbolic information exactly identifying the caller. This in itself is no reason for the callstack to be corrupted.
A call stack may be corrupted if functions overwrite values on the stack (i.e. by buffer overflow. This would likely show as '0x41445249' (if it were my name to overwrite it) as a call function. That is something outside your program memory ranges.
A way to diagnose the cause of your crash would be to set breakpoints on functions identified by the call stack. Or use your debugger to backtrace (depending on debugger & system). It is interesting to find out what arguments were included in the calls. Pointers are generally a good start (NULL pointers, uninitialized pointers). Good luck.
Related
In this answer: https://stackoverflow.com/a/8646611/192359 , it is explained that when debugging x86 code, symbols allow the debugger to display the callstack even when FPO (Frame Pointer Omission) is used.
The given explanation is:
On the x86 PDBs contain FPO information, which allows the debugger to reliably unwind a call stack.
My question is what's this information? As far as I understand, just knowing whether a function has FPO or not does not help you finding the original value of the stack pointer, since that depends on runtime information.
What am I missing here?
Fundamentally, it is always possible to walk the stack with enough information1, except in cases where the stack or execution context has been irrecoverably corrupted.
For example, even if rbp isn't used as the frame pointer, the return address is still on the stack somewhere, and you just need to know where. For a function that doesn't modify rsp (indirectly or directly) in the body of the function it would be at a simple fixed offset from rsp. For functions that modify rsp in the body of the function (i.e., that have a variable stack size), the offset from rsp might depend on the exact location in the function.
The PDB file simply contains this "side band" information which allows someone to determine the return address for any instruction in the function. Hans linked a relevant in-memory structure above - you can see that since it knows the size of the local variables and so on it can calculate the offset between rsp and the base of the frame, and hence get at the return address. It also knows how many instruction bytes are part of the "prolog" which is important because if the IP is still in that region, different rules apply (i.e., the stack hasn't been adjusted to reflect the locals in this function yet).
In 64-bit Windows, the exact function call ABI has been made a bit more concrete, and all functions generally have to provide unwind information: not in a .pdb but directly in a section included in the binary. So even without .pdb files you should be able to unwind a properly structured 64-bit Windows program. It allows any register to be used as the frame pointer, and still allows frame-pointer omission (with some restrictions). For details, start here.
1 If this weren't true, ask yourself how the currently running function could ever return? Now, technically you could design a program which clobbers or forgets the stack in a way that it cannot return, and either never exits or uses a method like exit() or abort() to terminate. This is highly unusual and not possibly outside of assembly.
I have written a function in C, which, when called, immediately results in a stack overflow.
Prototype:
void dumpOutput( Settings *, char **, FILE * );
Calling line:
dumpOutput( stSettings, sInput, fpOut );
At the time of calling it, stSettings is already a pointer to Settings structure, sInput is a dynamically allocated 2D array and fpOut is a FILE *. It reaches all the way to the calling line without any errors, no memory leaks etc.
The actual function is rather lengthy and i think its not worth sharing it here as the overflow occurs just as the code enters the function (called the prologue part, i think)
I have tried calling the same function directly from main() with dummy variables for checking if there are any problems with passed arguments but it still throws the stack overflow condition.
The error arises from the chkstk.asm when the function is called. This asm file (according to the comments present in it) tries to probe the stack to check / allocate the memory for the called function. It just keeps jumping to Find next lower page and probe part till the stack overflow occurs.
The local variables in dumpOutput are not memory beasts either, just 6 integers and 2 pointers.
The memory used by code at the point of entering this function is 60,936K, which increases to 61,940K at the point when the stack overflow occurs. Most of this memory goes into the sInput. Is this the cause of error? I don't think so, because only its pointer is being passed. Secondly, i fail to understand why dumpOutput is trying to allocate 1004K of memory on stack?
I am totally at a loss here. Any help will be highly appreciated.
Thanks in advance.
By design, it is _chkstk()'s job to generate a stack overflow exception. You can diagnose it by looking at the generated machine code. After you step into the function, right-click the edit window and click Go To Disassembly. You ought to see something similar to this:
003013B0 push ebp
003013B1 mov ebp,esp
003013B3 mov eax,1000D4h ; <== here
003013B8 call #ILT+70(__chkstk) (30104Bh)
The value passed through the EAX register is the important one, that's the amount of stack space your function needs. Chkstk then verifies it is actually available by probing the pages of stack. If you see it repeatedly looping then the value for EAX in your code is high. Like mine, it is guaranteed to consume all bytes of the stack. And more. Which is what it protects against, you normally get an access violation exception. But there's no guarantee, your code may accidentally write to a mapped page that belongs to, say, the heap. Which would produce an incredibly difficult to diagnose bug. Chkstk() helps you find these bugs before you blow your brains out in frustration.
I simply did it with this little test function:
void test()
{
char kaboom[1024*1024];
}
We can't see yours, but the exception says that you either have a large array as a local variable or you are passing a large value to _alloca(). Fix by allocating that array from the heap instead.
Most likely a stack corruption or recursion error but it's hard to answer without seeing any code
I was immediately suspicious of the crash. A Floating Point Exception in a method whose only arithmetic was a "divide by sizeof(short)".
I looked at the stack crawl & saw that the offset into the method was "+91". Then I examined a disassembly of that method & confirmed that the Program Counter was in fact foobar at the time of the crash. The disassembly showed instructions at +90 and +93 but not +91.
This is a method, 32-bit x86 instructions, that gets called very frequently in the life of the application. This crash has been reported 3 times.
How does this happen? How do I set a debugging trap for the situation?
Generally when you fault in the middle of an instruction, its due to bad flow control(ie: a broken jump, call, retn), an overflow, bad dereferencing or your debug symbols being out-of-sync making the stack trace show incorrect info. Your first step is to reliably reproduce the error everytime, else you'll have trouble trapping it, from there I'd just run it in a debugger, force the conditions to make it explode, then examine the (call) stack and registers to see if they are valid values etc.
I just spent some time chasing down a bug that boiled down to the following. Code was erroneously overwriting the stack, and I think it wrote over the return address of the function call. Following the return, the program would crash and stack would be corrupted. Running the program in valgrind would return an error such as:
vex x86->IR: unhandled instruction bytes: 0xEA 0x3 0x0 0x0
==9222== valgrind: Unrecognised instruction at address 0x4e925a8.
I figure this is because the return jumped to a random location, containing stuff that were not valid x86 opcodes. (Though I am somehow suspicious that this address 0x4e925a8 happened to be in an executable page. I imagine valgrind would throw a different error if this wasn't the case.)
I am certain that the problem was of the stack-overwriting type, and I've since fixed it. Now I am trying to think how I could catch errors like this more effectively. Obviously, valgrind can't warn me if I rewrite data on the stack, but maybe it can catch when someone writes over a return address on the stack. In principle, it can detect when something like 'push EIP' happens (so it can flag where the return addresses are on the stack).
I was wondering if anyone knows if Valgrind, or anything else can do that? If not, can you comment on other suggestions regarding debugging errors of this type efficiently.
If the problem happens deterministically enough that you can point out particular function that has it's stack smashed (in one repeatable test case), you could, in gdb:
Break at entry to that function
Find where the return address is stored (it's relative to %ebp (on x86) (which keeps the value of %esp at the function entry), I am not sure whether there is any offset).
Add watchpoint to that address. You have to issue the watch command with calculated number, not an expression, because with an expression gdb would try to re-evaluate it after each instruction instead of setting up a trap and that would be extremely slow.
Let the function run to completion.
I have not yet worked with the python support available in gdb7, but it should allow automating this.
In general, Valgrind detection of overflows in stack and global variables is weak to non-existant. Arguably, Valgrind is the wrong tool for that job.
If you are on one of supported platforms, building with -fmudflap and linking with -lmudflap will give you much better results for these kinds of errors. Additional docs here.
Udpdate:
Much has changed in the 6 years since this answer. On Linux, the tool to find stack (and heap) overflows is AddressSanitizer, supported by recent versions of GCC and Clang.
I am trying to figure out a crash in my application.
WinDbg tells me the following: (using dashes in place of underscores)
LAST-CONTROL-TRANSFER: from 005f5c7e to 6e697474
DEFAULT-BUCKET-ID: BAD_IP
BUGCHECK-STR: ACCESS-VIOLATION
It is obvious to me that 6e697474 is NOT a valid address.
I have three questions:
1) Does the "BAD_IP" bucket ID mean "Bad Instruction Pointer?"
2) This is a multi-threaded application so one consideration was that the object whose function I was attempting to call went out of scope. Does anyone know if that would lead to the same error message?
3) What else might cause an error like this? One of my co-workers suggested that it might be a stack overflow issue, but WinDBG in the past has proven rather reliable at detecting and pointing these out. (not that I'm sure about the voodoo it does in the background to diagnose that).
Bad-IP is Bad Instruction Pointer. From the description of your problem, I would assume it is a stack corruption instead of a stack overflow.
I can think of the following things that could cause a jump to invalid address, in decreasing order of likelyhood:
calling a member function on a deallocated object. (as you suspect)
calling a member function of a corrupted object.
calling a member function of an object with a corrupted vtable.
a rouge pointer overwriting code space.
I'd start debugging by finding the code at 005f5c7e and looking at what objects are being accessed around there.
It may be helpful to ask, what could have written the string 'ttie' to this location? Often when you have bytes in the 0x41-0x5A, 0x61-0x7A ([a-zA-Z]) range, it indicates a string buffer overflow.
As to what was actually overwritten, it could be the return address, some other function pointer you're using, or occasionally that a virtual function table pointer (vfptr) in an object got overwritten to point to the middle of a string.