When I'm debugging multi thread process in gdb, I meet "[Switching to Thread ~~ (LWP ~~)]". When the thread is changing to other thread, and why? I guess switching can be occurred by interrupt such as breakpoints. Am I right?
Yes, this is gdb notifying you that gdb's notion of the selected thread has changed. This happens if there is some debugging event, like a signal or a breakpoint stop, in a thread other than the one you had last selected.
Related
I am writing a little tracer using the Win32 Debug Api (not dbgeng). The general flow is:
DEBUG_EVENT event;
WaitForDebugEvent(&event);
...
ContinueDebugEvent(event.pid, event.tid,...)
My question is: How can I switch the debugger to a different thread than the one that reported the current debugging event (event.tid)?
IE Thread A reports an event in WaitForDebugEvent, but I want to continue stepping Thread B.
ContinueDebugEvent documentation on the second parameter says:
The thread identifier of the thread to continue. The combination of process identifier and thread identifier must identify a thread that has previously reported a debugging event.
So I cant pass the id of a different thread.
I know this is possible because you can do it in Visual Studio GUI as well as windbg:
https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/-s--set-current-thread-
We develop a user-space process running on Linux 3.4.11 in an embedded MIPS system. The process creates multiple (>10) threads using pthreads. The process has a SIGSEGV signal handler which, among other things, generates a log message which goes to our log file. As part of this flow, it acquires a semaphore (bad, I know...).
During our testing the process appeared to hang. We're currently unable to build gdb for the target platform, so I wrote a CLI tool that uses ptrace to extract the register values and USER data using PTRACE_PEEKUSR.
What surprised me to see is that all of our threads were inside our crash handler, trying to acquire the semaphore. This (obviously?) indicates a deadlock on the semaphore, which means that a thread died while holding it. When I dug up the stack, it seemed that almost all of the threads (except one) were in a blocking call (recv, poll, sleep) when the signal handler started running. Manual stack reconstruction on MIPS is a pain so we have not fully done it yet. One thread appeared to be in the middle of a malloc call, which to me indicates that it crashed due to a heap corruption.
A couple of things are still unclear:
1) Assuming one thread crashed in malloc, why would all other threads be running the SIGSEGV handler? As I understand it, a SIGSEGV signal is delivered to the faulting thread, no? Does it mean that each and every one of our threads crashed?
2) Looking at the sigcontext struct for MIPS, it seems it does not contain the memory address which was accessed (badaddr). Is there another place that has it? I couldn't find it anywhere, but it seemed odd to me that it would not be available.
And of course, if anyone can suggest ways to continue the analysis, it would be appreciated!
Yes, it is likely that all of your threads crashed in turn, assuming that you have captured the thread state correctly.
siginfo_t has a si_addr member, which should give you the address of the fault. Whether your kernel fills that in is a different matter.
In-process crash handlers will always be unreliable. You should use an out-of-process handler, and set kernel.core_pattern to invoke it. In current kernels, it is not necessary to write the core file to disk; you can either read the core file from standard input, or just map the process memory of the zombie process (which is still available when the kernel invokes the crash handler).
From what I understand, a thread that executes in user mode can eventually enter code that switches to kernel mode (using sysenter). But how can a thread that's been emanating from user code execute kernel code?
Eg: I'm calling CreateFile(), it then delegates to NtCreateFile(), which in turn calls ZwCreateFile(), than ZiFastSystemCall()... than sysenter... profit, kernel access?
Edit
This question:
How does Windows protect transition into kernel mode has an answer that helped me understand, see quote:
"The user mode thread is causing an exception that's caught by the Ring 0 code. The user mode thread is halted and the CPU switches to a kernel/ring 0 thread, which can then inspect the context (e.g., call stack & registers) of the user mode thread to figure out what to do." Also see this blog post, very informative: http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection
The short answer is that it can't.
What happens is that when you create a user-mode thread, the kernel creates a matching kernel mode thread. When "your" thread needs to execute some code in kernel mode, it's actually executed in the matching kernel mode thread.
Disclaimer: The last time I really looked closely at this was probably with Win2K or maybe even NT4 -- but I doubt much has changed in this respect.
I'm debugging a deadlock. I would like to wake up a thread that's waiting on a condition variable, so that I can trace as it re checks the sleep conditions. Sleeping on a condition variable is subject to spurious wake ups, so how can I force such a wake up?
If you're using windbg you can call WakeAllConditionVariable in the debugee using windbg's .call command.
To debug multithreaded programs to in case of conditions such as deadlock or livelock, what are the useful utilities? I was wondering if gcore gives the stack dump of all running threds in the process or just the main thread. Also, does gcore suspend/kill the running process? Any information on debugging multithreaded programs will be very useful.
gdb supports switching between threads to investigate the state of everything going on. Here is some more information.
gdb has some nice features for working with threads. One of my favorites is thread apply. This allows you to run the same command for multiple threads.
For example, if you wanted to get a backtrace of all threads, you can use this:
thread apply all where
To break this down, the command starts with thread apply.
Next is the list of threads. Here I used the keyword all to apply this to every thread in the process. You can also use a space separated list of the gdb thread ids (thread apply 1 2 3 command).
And finally comes the command to perform. I used where which shows you the call stack but you can use any command you want.
As Carl stated, gdb supports threads. Using a UI (such as one provided by Eclipse) for GDB makes this easier, but you can get thread information when running via the command line by typing "info threads". This will list the threads and allow you to switch by typing "thread 3" etc. Once you switch to a thread, you can do backtraces in order to see the current threads stack and other commands that you're used to using with a single threaded process.