I am using a syscall checker in combination with -fsanitize=address and when ASAN finds a bug, it calls some syscalls (ioctl(ISATTY), etc) when printing out the report. The syscall checker interrupts ASAN's ioctls and the error report is not collected properly.
What I would like is for ASAN to simply abort without printing the report, or failing that, a way to determine (using a libasan4 API call maybe) that ASAN found an error, so I can stop the syscall checker from intercepting syscalls.
Unfortunately __asan_error_report, __sanitizer_set_death_callback and __asan_set_error_report_callback from libasan4 all kick in after ASAN has collected the report:
0 __asan_error_report()
1 syscall_checker()
2 ioctl(ISATTY)
3 asan::PrintReport()
4 app_code_that_crashes()
And the syscall checker does not handle ASAN's ioctl() calls properly, so it exit()s normally, while I am hoping to keep ASAN's behavior of abort()ing.
You should be able to intercept before report is printed by overriding __asan_on_error (declared in asan_interface.h, empty by default):
// User may provide function that would be called right when ASan detects
// an error. This can be used to notice cases when ASan detects an error, but
// the program crashes before ASan report is printed.
void __asan_on_error();
Note that due to weird Asan callback interface you'd better implement this callback in main binary (definitions in shared libraries are likely to be unable to intercept default definition from libasan.a).
Related
My program is panicking so I followed its advice to run RUST_BACKTRACE=1 and I get this (just a little snippet).
1: 0x800c05b5 - std::sys::imp::backtrace::tracing::imp::write::hf33ae72d0baa11ed
at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42
2: 0x800c22ed - std::panicking::default_hook::{{closure}}::h59672b733cc6a455
at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:351
If the program panics it stops the whole program, so where can I figure out at which line it's panicking on?
Is this line telling me there is a problem at line 42 and line 351?
The whole backtrace is on this image, I felt it would be to messy to copy and paste it here.
I've never heard of a stack trace or a back trace. I'm compiling with warnings, but I don't know what debugging symbols are.
What is a stack trace?
If your program panics, you encountered a bug and would like to fix it; a stack trace wants to help you here. When the panic happens, you would like to know the cause of the panic (the function in which the panic was triggered). But the function directly triggering the panic is usually not enough to really see what's going on. Therefore we also print the function that called the previous function... and so on. We trace back all function calls leading to the panic up to main() which is (pretty much) the first function being called.
What are debug symbols?
When the compiler generates the machine code, it pretty much only needs to emit instructions for the CPU. The problem is that it's virtually impossible to quickly see from which Rust-function a set of instructions came. Therefore the compiler can insert additional information into the executable that is ignored by the CPU, but is used by debugging tools.
One important part are file locations: the compiler annotates which instruction came from which file at which line. This also means that we can later see where a specific function is defined. If we don't have debug symbols, we can't.
In your stack trace you can see a few file locations:
1: 0x800c05b5 - std::sys::imp::backtrace::tracing::imp::write::hf33ae72d0baa11ed
at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42
The Rust standard library is shipped with debug symbols. As such, we can see where the function is defined (gcc_s.rs line 42).
If you compile in debug mode (rustc or cargo build), debug symbols are activated by default. If you, however, compile in release mode (rustc -O or cargo build --release), debug symbols are disabled by default as they increase the executable size and... usually aren't important for the end user. You can tweak whether or not you want debug symbols in your Cargo.toml in a specific profile section with the debug key.
What are all these strange functions?!
When you first look at a stack trace you might be confused by all the strange function names you're seeing. Don't worry, this is normal! You are interested in what part of your code triggered the panic, but the stack trace shows all functions somehow involved. In your example, you can ignore the first 9 entries: those are just functions handling the panic and generating the exact message you are seeing.
Entry 10 is still not your code, but might be interesting as well: the panic was triggered in the index() function of Vec<T> which is called when you use the [] operator. And finally, entry 11 shows a function you defined. But you might have noticed that this entry is missing a file location... the above section describes how to fix that.
What do to with a stack trace? (tl;dr)
Activate debug symbols if you haven't already (e.g. just compile in debug mode).
Ignore any functions from std and core at the top of the stack trace.
Look at the first function you defined, find the corresponding location in your file and fix the bug.
If you haven't already, change all camelCase function and method names to snake_case to stick to the community wide style guide.
I'm having an issue where my application is failing a debug assertion (_CrtIsValidHeapPointer) before anything is even executed. I know this because I added a breakpoint on the first statement of my main function, and it fails the assertion before the breakpoint is reached.
Is there a way to somehow "step through" everything that happens before my main function is called? Things like static member initializations, etc.
I should note that my program is written in C++/CLI. I recently upgraded to VS2015 and am targeting the v140 toolset. The C++ libraries I'm using (ImageMagick, libsquish, and one of my own C++ libraries) have been tested individually, and I do not receive the assertion failure with these libraries, so it has to be my main application.
I haven't changed any of the code since I upgraded from VS2013, so I'm a little stumped on what is going on.
EDIT:
Here is the call stack. This is after I click "Retry" in the assertion failed window. I then get a multitude of other exceptions being thrown, but they are different each time I run the program.
> ucrtbased.dll!527a6853()
[Frames below may be incorrect and/or missing, no symbols loaded for ucrtbased.dll]
ucrtbased.dll!527a7130()
ucrtbased.dll!527a69cb()
ucrtbased.dll!527c8116()
ucrtbased.dll!527c7eb3()
ucrtbased.dll!527c7fb3()
ucrtbased.dll!527c84b0()
PathCreator.exe!_onexit(int (void)* const function) Line 268 + 0xe bytes C++
PathCreator.exe!atexit(void (void)* const function) Line 276 + 0x9 bytes C++
PathCreator.exe!std::`dynamic initializer for '_Fac_tidy_reg''() Line 65 + 0xd bytes C++
[External Code]
mscoreei.dll!7401cd87()
mscoree.dll!741fdd05()
kernel32.dll!76c33744()
ntdll.dll!7720a064()
ntdll.dll!7720a02f()
You have to debug the C runtime initialization code. Not intuitive to do because the debugger tries hard to avoid it and get you into the main() entrypoint instead. But still possible, use Debug > New Breakpoint > Function Breakpoint.
Enter _initterm for the function name, Language = C.
Press F5 and the breakpoint will hit. You should see the C runtime source code. You can now single-step through the initialization functions of your program one-by-one, every call to (**it)() executes one.
That's exactly what you asked for. But not very likely what you actually want. The odds that your code produces this error are very low. Much more likely is that one of these libraries causes this problem. They are likely to be built targeting another version of the C runtime library. And therefore have their own _initterm() function.
Having more than one copy of the C runtime library in a process is generally very unhealthy. And highly likely to generate heap corruption. If you can't locate it from the stack trace (be sure to change the Debugger Type from Auto to Mixed, always post the stack trace in an SO question) then the next thing you should strongly consider is rebuilding those libraries with the VS version you use.
In the Linux kernel, when a breakpoint I register with register_wide_hw_breakpoint is triggered, the callback handler endlessly runs until the breakpoint is unregistered.
Background: To test a driver for some hardware we are making, I am writing a second kernel module that emulates the hardware interface. My intent is to set a watchpoint on a memory location that in the hardware would be a control register, so that writing to this 'register' can trigger an operation by the emulator driver.
See here for a complete sample.
I set the breakpoint as follows:
hw_breakpoint_init(&attr);
attr.bp_addr = kallsyms_lookup_name("test_value");
attr.bp_len = HW_BREAKPOINT_LEN_4;
attr.bp_type = HW_BREAKPOINT_W;
test_hbp = register_wide_hw_breakpoint(&attr, test_hbp_handler, NULL);
but when test_value is written to, the callback (test_hbp_handler) is triggered continually without control ever returning to the code that was about to write to test_value.
1) What should I be doing differently for this to work as expected (return execution to code that triggered breakpoint)?
2) How do I capture the value that was being written to the memory location?
In case this matters:
$ uname -a
Linux socfpga-cyclone5 3.10.37-ltsi-rt37-05714-ge4ee387 #1 SMP PREEMPT RT Mon Jan 5 17:51:35 UTC 2015 armv7l GNU/Linux
This is by design. When an ARM hardware watchpoint is hit, it generates a Data Abort exception. On ARM, Data Abort exceptions trigger before the instruction that triggers them finishes1. This means that, in the exception handler, registers and memory locations affected by the instruction still hold their old values (or, in some cases, undefined values). As such, when the handler finishes, it must retry the aborted instruction so that the interrupted program runs as intended2. If the watchpoint is still set when the handler returns, the instruction will trigger it again. This causes the loop you're seeing.
To get around this, userspace debuggers like GDB single-step over any instruction that hits a watchpoint with that watchpoint disabled before resuming execution. The underlying kernel API, however, just exposes the hardware watchpoint behavior directly. Since you're using the kernel API, it's up to your event handler to ensure that the watchpoint doesn't trigger on the retried instruction.
[The ARM watchpoint code in the kernel actually does support automatic single-step, but only under very specific conditions. Namely, it requires 1) that no event handler is registered to the watchpoint, 2) that the watchpoint is in userspace, and 3) that the watchpoint is not associated with a particular CPU. Since your use case violates at least (1) and (2), you have to find another solution.]3
Unfortunately, on ARM, there's no foolproof way to keep the watchpoint enabled without causing a loop. The breakpoint mode that GDB uses to single-step programs, "instruction mismatch," generates UNPREDICTABLE behaviour when used in kernel mode4. The best you can do is disable the watchpoint in your handler and then set a standard breakpoint to re-enable it on an instruction that you know will execute soon after.
For your MMIO emulation driver, watchpoints are probably not the answer. In addition to the issues just mentioned, most ARM cores have very few watchpoint registers, so the solution would not scale. I'm afraid I'm not familiar enough with ARM's memory model to suggest an alternative approach. However, Linux's existing code for emulating memory-mapped IO for virtual machines might be a good place to start.
1There are two types of Data Abort exceptions, synchronous and asynchronous, and it's left to the implementation to decide which one a watchpoint generates. I'm describing the behavior of synchronous exceptions in this answer, because that's what would cause the problem you're having.
2ARMv7-A/R Architecture Reference Manual, B1.9.8, "Data Abort exception."
3Linux Kernel v4.6, arch/arm/kernel/hw_breakpoint.c, lines 634-661.
4ARMv7-A/R Architecture Reference Manual, C3.3.3, "UNPREDICTABLE cases when Monitor debug-mode is selected."
My program crashed when I added the option -fstack-check and -fstack-protector. __stack_chk_fail is called in the back trace.
So how could I know where the problem is ? What does -fstack-check really check ?
The information about gcc seems too huge to find out the answer.
After checked the assembly program.
I think -fstack-check, will add code write 0 to an offset of the stack pointer, so to test if the program visit a violation address, the program went crash if it does.
e.g. mov $0x0,-0x928(%esp)
-fstack-check: If two feature macros STACK_CHECK_BUILTIN and STACK_CHECK_STATIC_BUILTIN are left at the default 0, it just inserts a NULL byte every 4kb (page) when the stack grows.
By default only one, but when the stack can grow more than one page, which is the most dangerous case, every 4KB. linux >2.6 only has only one small page gap between the stack and the heap, which can lead to stack-gap attacks, known since 2005.
See What exception is raised in C by GCC -fstack-check option for assembly.
It is enabled in gcc at least since 2.95.3, in clang since 3.6.
__stack_chk_fail is the inserted -fstack-protector code which verifies an inserted stack canary value which might be overwritten by a simple stack overflow, e.g. by recursion.
"`-fstack-protector' emits extra code to check for buffer overflows, such as stack
smashing attacks. This is done by adding a guard variable to
functions with vulnerable objects. This includes functions that
call alloca, and functions with buffers larger than 8 bytes. The
guards are initialized when a function is entered and then checked
when the function exits. If a guard check fails, an error message
is printed and the program exits"
GCC Options That Control Optimization
GCC extension for protecting applications from stack-smashing attacks
Smashing The Stack For Fun And Profit
I Hope this will give some clue..
I am writing a memory tracking application that hooks all the calls to HeapAlloc using IAT patching mechanism. The idea is to capture all the calls to HeapAlloc and get a callstack.
However I am currently facing a problem with getting the callstack using DBGHELP Apis. I found that the dbghelp dll itself is linking to MSVCRT dll and this dependancy results in a recursive call. When I try to get a callstack for any of the calls from the target application, dbghelp internally calls some method from MSVCRT that again calls HeapAlloc. And since I have already patched MSVCRT it results in an infinite loop.
Has anyone faced this problem and solved it ? Is there any way out of this impasse?
This is a standard problem in function interception code. We had a similar issue with a logging library that used shared memory for storing log level information, while the shared memory library had to log information.
The way we fixed it could be applied to your situation, I believe.
In your intercept code, maintain a static flag that indicates whether or not you're in the middle of an intercept. When your intercept is called and the flag isn't set, set the flag then do what you currently do, including calling DbgHelp, then clear the flag.
If your intercept is called while the flag is set, only call the back-end HeapAlloc code without doing any of the other stuff (including calling DbgHelp which is what's causing your infinite recursion).
Something along the lines of (pseudo-code):
function MyHookCode:
static flag inInterceptMode = false
if inInterceptMode:
call HeapAlloc
return
inInterceptMode = true
call DbgHelp stuff
call HeapAlloc
inInterceptMode = false
return
function main:
hook HeapAlloc with MyHookCode
: : :
return
What about using some real memory tracking products like GlowCode?
You can use Deviare API Hook and get full stack-trace without using that API that has a big number of problems.