I'm trying to use gdb in postmortem mode with the core dump of a crashed process. I can get a stack trace, but instead of showing me the actual location in the offending function, gdb shows me the line number of a two-line inlined function that the offending function calls.
The inlined function is called many, many places; how do I find which call triggered the crash? How do I find the code immediately around the inlined function?
Go to the stack frame in question, print the instruction point (e.g. p $rip),
then use it to look it up manually with e.g. "addr2line -e -i 0x84564756".
This doesn't scale, but at least it works.
I assume that the "many many calls to the inlined function" are all happening from within a single "offending function" (otherwise your question doesn't make sense to me).
Your best bet is to note the IP address of the crash point in GDB, then use "objdump -dS ./a.out" and find that IP in the output.
You can try setting OPTIMIZE to NO (eg. setenv OPTIMIZE NO) and rebuild the project: this tells compiler not optimize code, hence it may not inline function calls.
Related
I've generated an gfortran executable I call mtc08.exe that exhibits the following behavior:
1) If I run it in gdb it runs successfully to the end
2) If I run it normally, re routing the output with the windows command
mtc08.exe >out
It gives me "floating point exception - erroneous arithmetic operation" but does not say where that occurs. (The "backtrace" information is incomprehensible to me, and it seems it cannot contain much information because all is the letter "f".)
Then I'm trying to localize the problem by seeing where the program stops writing results, but having difficulties there also, because I get the impression the program may be multi tasking and proceeding with "future" arithmetic operations before completing output operations that in any case do not interfere.
Does such multi tasking occur? If so, can I turn it off with a compiler switch, so I'm really sure it is doing all operations sequentially?
The currently used compilation command in windows is:
rem debug compilation:
gfortran -static -fdefault-real-8 -fdefault-integer-8 -g -ffpe-trap=invalid,zero,overflow,underflow,denormal -Wall -fcheck=all #mtc08.fls -o mtc08.exe
where mtc08.fls is a file containing the names of all source files.
It may be that by removing some of the "-ffpe-trap" options it would run, but this rattles my confidence, and I'd like to get to the bottom of it, rather than just find a way around.
I can of course give more information, but seeing the error is not localized, that is not so practical.
I still don't know how to get better diagnostics, but still managed to find the cause of the problem by trial and error, which came from an uninitialized variable.
The problem appears to arise because gdb used zero for uninitialized variables, which should have been their values anyhow, and running the same executable without gdb did use values that lead to "floating point exception".
One way to get better diagnostics in gdb is to use -finit-real-nan in the compilation so that uninitialized variables will get more attention from the debugger.
Perhaps with better coding practice, such as intent declarations for all arguments, the debugger would also be better in picking up uninitialized variables without having to initialize them to nan.
While running an executable in gdb, I encountered the following error:
Program received signal SIGFPE, Arithmetic exception.
0x08158307 in radtra_ ()
How do I understand what line number and file does 0x08158307 without recompiling or otherwise modifying the source? if it helps, the source language was Fortran.
How do I understand what line number and file does 0x08158307 without recompiling or otherwise modifying the source?
That isn't easy. You could use GDB disassemble command, look for access to global variables and CALL instructions, and make a guess where inside radtra_ you are. This is harder the larger the routine is, the more optimizations compiler has applied to it, and the fewer calls and global variable accesses are performed.
If you can't guess, your only options are:
Rebuild the application adding -g flag, but leaving all other compile options unmodified, then use addr2line to translate the address to line number. (This is how you should build the application from the start.)
If you can't rebuild the entire application, rebuild just the source containing radtra_ (again with same flags, but add -g). You should be able to match the output from objdump -d radtra.o with the output from disassemble. Once you have a match, read output from readelf -wl radtra.o or objdump -g radtra.o to associate code offsets within radtra_ with source lines that code was generated from.
Hire an expert to guess for you. This wouldn't be cheap, as people skilled in this kind of reverse engineering are usually gainfully employed and value their time.
I have a function that I am trying to examine. I want to find all callers of this function, but there are a few issues:
I am doing this to understand the code because I did not write it, but I need to know exactly how it behaves
It runs through the STL beforehand, so I can't just use something like callgrind to get it's immediate callers
There is a stack trace of 10+ function calls until you get to actual code that is not in the STL that caused this function to be invoked. But those STL entry points vary, as it is a compare function and calls for is_equal go through a different sequence than those that go through not_equal, etc. I will need to do this for at least 10+ different functions, and I want to streamline this as much as possible.
I want a tool that can dump each unique, full backtrace each time the function is called. Does anybody know a tool that can do this?
I am using gdb and c++ on Ubuntu 14.04.
You can make gdb execute a series of commands each time a given breakpoint is executed, e.g.,
break someFunction
commands
bt
continue
end
The feature is mentioned in gdb scripting: execute commands at selected breakpoint, which has a link to the online documentation for gdb 5.1.7 Breakpoint Command Lists
I have a C program, which I compiled with the -g options, and then runned with:
valgrind --tool=cachegrind --branch-sim=yes ./myexecutable
This let me know which function contains a bottleneck. However this is a pretty long function and it is not clear to me from which part of the function I get most of the cache missings. I cannot (do not want to) divide it in two different parts.
Is there a way (maybe including a valgrind.h or with some magical #pragma stuff), to instruct Valgrind to make different statistics for different parts of a function?
To check function-by-function values, you have probably used cg_annotate like this:
cg_annotate cachegrind.out.1234
If you add the "--auto=yes" flag to that command, the values will be displayed for each line:
cg_annotate --auto=yes cachegrind.out.1234
You can print the result into a file so that you can search for your function. Note that only lines and functions that have a major performance impact will be displayed, so if you can't find certain lines they likely have a negligible impact on the execution.
This is homework. Tips only, no exact answers please.
I have a compiled program (no source code) that takes in command line arguments. There is a correct sequence of a given number of command line arguments that will make the program print out "Success." Given the wrong arguments it will print out "Failure."
One thing that is confusing me is that the instructions mention two system tools (doesn't name them) which will help in figuring out the correct arguments. The only tool I'm familiar with (unless I'm overlooking something) is GDB so I believe I am missing a critical component of this challenge.
The challenge is to figure out the correct arguments. So far I've run the program in GDB and set a breakpoint at main but I really don't know where to go from there. Any pro tips?
Are you sure you have to debug it? It would be easier to disassemble it. When you disassemble it look for cmp
There exists not only tools to decompile X86 binaries to Assembler code listings, but also some which attempt to show a more high level or readable listing. Try googling and see what you find. I'd be specific, but then, that would be counterproductive if your job is to learn some reverse engineering skills.
It is possible that the code is something like this: If Arg(1)='FOO' then print "Success". So you might not need to disassemble at all. Instead you only might need to find a tool which dumps out all strings in the executable that look like sequences of ASCII characters. If the sequence you are supposed to input is not in the set of characters easily input from the keyboard, there exist many utilities that will do this. If the program has been very carefully constructed, the author won't have left "FOO" if that was the "password" in plain sight, but will have tried to obscure it somewhat.
Personally I would start with an ltrace of the program with any arbitrary set of arguments. I'd then use the strings command and guess from that what some of the hidden argument literals might be. (Let's assume, for the moment, that the professor hasn't encrypted or obfuscated the strings and that they appear in the binary as literals). Then try again with one or two (or the requisite number, if number).
If you're lucky the program was compiled and provided to you without running strip. In that case you might have the symbol table to help. Then you could try single stepping through the program (read the gdb manuals). It might be tedious but there are ways to set a breakpoint and tell the debugger to run through some function call (such as any from the standard libraries) and stop upon return. Doing this repeatedly (identify where it's calling into standard or external libraries, set a breakpoint for the next instruction after the return, let gdb run the process through the call, and then inspect what the code is doing besides that.
Coupled with the ltrace it should be fairly easy to see the sequencing of the strcmp() (or similar) calls. As you see the string against which your input is being compared you can break out of the whole process and re-invoke the gdb and the program with that one argument, trace through 'til the next one and so on. Or you might learn some more advanced gdb tricks and actually modify your argument vector and restart main() from scratch.
It actually sounds like fun and I might have my wife whip up a simple binary for me to try this on. It might also create a little program to generate binaries of this sort. I'm thinking of a little #INCLUDE in the sources which provides the "passphrase" of arguments, and a make file that selects three to five words from /usr/dict/words, generates that #INCLUDE file from a template, then compiles the binary using that sequence.