set watchpoint on pthread's thread-local storage in gdb - debugging

Question

Is it possible to set a watchpoint on pthread's thread-local storage using GDB? I have a program that runs:
struct stored_type *res = pthread_getspecific(tls_key);
...and after a few thousand calls it returns 0 instead of a valid pointer. I'd really love to figure out what's setting that value to 0. I've tried setting breakpoints on pthread_setspecific and pthread_delete_key (the only things I could think of that would reasonably cause the key to change value) and those breakpoints aren't getting hit, so I'm thinking there's some kind of overrun happening.
I'm using Linux x86_64 with glibc 2.23.

Answer 1

... the only things I could think of that would reasonably cause the key to change value
The most likely reasons for pthread_getspecific to return NULL:
you are in fact executing in a new thread, one in which pthread_setspecific hasn't been called,
you are calling pthread_getspecific while current thread is in the process of being destroyed (i.e. pthread_exit is somewhere on the stack),
you are calling pthread_getspecific in a signal handler (none of pthread_* functions are async-signal safe).
Assuming none of the above reasons are true in your case, on with the show.
First we need a test case to demonstrate on.
#include <assert.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
pthread_key_t key;
void *thrfn(void *p) {
int rc = pthread_setspecific(key, &p);
assert(rc == 0);
sleep(60);
rc = pthread_setspecific(key, (void*)0x112233);
assert(rc == 0);
return p;
}
int main()
{
pthread_t thr;
int rc = pthread_key_create(&key, NULL);
assert(rc == 0);
rc = pthread_create(&thr, NULL, thrfn, NULL);
assert(rc == 0);
sleep(90);
return 0;
}
gcc -g -pthread t.c
gdb -q ./a.out
(gdb) start
Now, it helps very much to have GLIBC that is compiled with debug info. Most distributions provide a libc-dbg or similar package, which supplies that. Looking at pthread_setspecific source, you can see that inside the thread descriptor (self) there is a specific_1stblock array, where the space for first PTHREAD_KEY_2NDLEVEL_SIZE == 32 key slots is pre-allocated (32 distinct keys is usually more than enough).
The value that we pass will be stored in self->specific_1stblock[key].data, and that's exactly the location you'll want to set the watchpoint on.
In our sample program, key == 0 (as this is the very first key). Putting it all together:
Starting program: /tmp/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Temporary breakpoint 1, main () at t.c:21
21 int rc = pthread_key_create(&key, NULL);
(gdb) b pthread_setspecific
Breakpoint 2 at 0x7ffff7bc9460: file pthread_setspecific.c, line 28.
(gdb) c
Continuing.
[New Thread 0x7ffff77f6700 (LWP 58683)]
[Switching to Thread 0x7ffff77f6700 (LWP 58683)]
Breakpoint 2, __GI___pthread_setspecific (key=0, value=0x7ffff77f5ef8) at pthread_setspecific.c:28
28 pthread_setspecific.c: No such file or directory.
(gdb) n
35 in pthread_setspecific.c
(gdb) n
28 in pthread_setspecific.c
(gdb) p self
$1 = (struct pthread *) 0x7ffff77f6700
(gdb) watch -l self.specific_1stblock[key].data
Hardware watchpoint 3: -location self.specific_1stblock[key].data
(gdb) c
Continuing.
Hardware watchpoint 3: -location self.specific_1stblock[key].data
Old value = (void *) 0x0
New value = (void *) 0x7ffff77f5ef8
__GI___pthread_setspecific (key=<optimized out>, value=0x7ffff77f5ef8) at pthread_setspecific.c:89
89 in pthread_setspecific.c
Note that the new value is exactly the value that we passed to pthread_setspecific.
(gdb) c
Continuing.
Breakpoint 2, __GI___pthread_setspecific (key=0, value=0x112233) at pthread_setspecific.c:28
28 in pthread_setspecific.c
(gdb) c
Continuing.
Hardware watchpoint 3: -location self.specific_1stblock[key].data
Old value = (void *) 0x7ffff77f5ef8
New value = (void *) 0x112233
__GI___pthread_setspecific (key=<optimized out>, value=0x112233) at pthread_setspecific.c:89
89 in pthread_setspecific.c
This is our second pthread_setspecific call
(gdb) c
Continuing.
Hardware watchpoint 3: -location self.specific_1stblock[key].data
Old value = (void *) 0x112233
New value = (void *) 0x0
__nptl_deallocate_tsd () at pthread_create.c:152
152 pthread_create.c: No such file or directory.
And this is thread destruction, which deallocates the thread descriptor itself.
(gdb) c
Continuing.
[Thread 0x7ffff77f6700 (LWP 58683) exited]
[Inferior 1 (process 58677) exited normally]