I have a legacy application that I am trying to debug which schedules a background task like this:
err = QueueUserWorkItem(
Foo,
NULL,
WT_EXECUTEINIOTHREAD);
I can see that my function Foo is never executed (i put a breakpoint on the function and it is never hit). So i thought lets see if we have any IO thread avaialble for the function ever or not. But sadly i dont know how to list such threads in windbg. Can anyone help me here please. I am doing this debugging in user mode.
Load sos: .loadby sos clr or .loadby sos mscorwks
Dump the threads: !Threads
Pick an interesting thread: ~<thread number>s (eg: ~13s) Then show the stack: !DumpStack
You could also try !EEStack to show all the threads (can be information overload).
Related
C Code vs. Debugger Scripts
So, I'm trying to use some essentially obfuscated C code (that I cannot easily replicate in a debugger script) to "decode" an object in memory during a WinDBG live debugging session.
My intention is to invoke the function like:
.call MyDLL!decode_obj(0x86206922)
The catch is that the object pointer I have may be bad (i.e. corrupt, the object may be in some transitional state, etc.) and so I would prefer to create a new thread in which to run this function invocation. If I repurpose one of the existing threads, then I risk hosing my ability to do further investigation on it later.
It seems like it should be possible to create a new thread in the debugging session.
I thought perhaps CreateThread would do the trick. However, the debugger complains when I attempt to invoke it saying that it's not a function.
> .call KERNELBASE!CreateThread(0x0,0x0,0x6329c410,0x0,0x0)
^ Symbol not a function in '.call KERNELBASE!CreateThread(0x0,0x0,0x6329c410,0x0,0x0)'
When I try to examine the symbol for it, I learn that the debugger doesn't know its prototype.
> x KERNELBASE!CreateThread
76a58930 KERNELBASE!CreateThread (<no parameter info>)
The debugger has (public, I presume) symbols loaded for that module:
> lm v m KERNELBASE*
76900000 76aa2000 KERNELBASE (pdb symbols)
c:\data\src\symcache\wkernelbase.pdb\50E684CA8883458E868AF15C58DF1C2E1\wkernelbase.pdb
The .call documentation mentions the /s parameter, in which you can specify a prototype, but I could not find examples of how to specify it and my few attempts were unsuccessful.
Can I create a new thread from WinDBG?
This seems like it ought to be possible. WinDBG does it when I attach to a running process, and I just need to invoke the correct API, I would think.
How do I create a new thread?
Is CreateThread the right API to use, or is there a better one?
.call /s Example(s)?
Are there any examples of how to specify a prototype? If it were a C++ function, would I have to do name mangling? Do I have to resolve all the typedefs and CPP macro definitions?
Problem
I need a kernel thread that is able to work for prolonged periods of time without yielding, basically fully dedicating a CPU core to it on demand:
int my_kthread(void *arg)
{
while(!kthread_should_stop()) {
do_some_work();
if(sleeping_enabled) msleep(1000);
else {
// What to do here to avoid lockup warnings
// and ensure system stability?
}
}
return 0;
}
Background
The thread is created like this when the module that I am working on is loaded:
my_task = kthread_run(&my_kthread, (void *)some_data, "My KThread")
set_cpus_allowed(my_task, *cpumask_of(10)); // Pin thread to core #10
and stopped like this when the module is unloaded:
kthread_stop(my_task);
Everything works just fine when sleeping_enabled is true.
Otherwise, soon after the thread is started, the kernel complains of the apparent lockup.
At first, I merely aimed to avoid the various warnings such as
BUG: soft lockup - CPU#10 stuck for 22s!
and
INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 15, t=30567 jiffies)
since they tend to flood my console with dumps for all >20 cores, and the "lockup" is desired behavior.
I tried poking the watchdog like this:
if(sleeping_enabled) msleep(1000);
else touch_softlockup_watchdog();
in combination with (echo 1 > /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress)
and pretty much got what I want (a never-sleeping thread that successfully does what I want and no spam in the console).
However, not only does this "solution" feel like cheating, it seems I am completely breaking something by hogging that one core: when unloading the module via rmmod, the whole system freezes. The console starts periodically dumping soft lockups on all cores, with this call trace:
[<ffffffff810c96b0>] ? queue_stop_cpus_work+0xd0/0xd0
[<ffffffff810c9903>] cpu_stopper_thread+0xe3/0x1b0
[<ffffffff8108639a>] ? finish_task_switch+0x4a/0xf0
[<ffffffff8169e654>] ? __schedule+0x3c4/0x700
[<ffffffff81080e98>] ? __wake_up_common+0x58/0x90
[<ffffffff810c9820>] ? __stop_cpus+0x80/0x80
[<ffffffff81077e93>] kthread+0x93/0xa0
[<ffffffff816a9724>] kernel_thread_helper+0x4/0x10
[<ffffffff81077e00>] ? flush_kthread_worker+0xb0/0xb0
[<ffffffff816a9720>] ? gs_change+0x13/0x13
Meanwhile, my kernel thread continues running (as evidenced by some console messages that it prints out every now and then), so it never saw kthread_should_stop() return true.
Unloading did work correctly and stopped the thread before I switched to not sleeping at all. Now, I am unable to make iterative modifications without having to reboot.
Note that I have simplified the description here a lot. I am trying to add such a thread (to poll some hardware registers and log their changes) to a GPU driver, so there may be module-dependent reasons for the freeze on unload. However, this does not change my general question about how to best implement a thread that never sleeps.
I think this question is similar to your question "whole one core dedicated to single process" , please check the replies there.
Does anyone know how I can list all threads in WinDbg while kernel debugging. I have found older references that say '~' but that does not work.
Specifically I am looking to the find the ID of a thread that caused an event, namely a breakpoint.
Thanks.
~ only works in user mode. To list all threads on the system, it's !process 0 1 as I recall (it's been awhile).
"Specifically I am looking to the find the ID of a thread that caused an event, namely a breakpoint."
This statement doesn't make much sense to do from kernel mode. Can you descrive more about what your scenario is?
Edit: Ah, now I get it. You want to know which thread you're currently in right now. Give !thread a go.
You can always use the #$thread pseudo register to reference the current thread object:
0: kd> r #$thread
$thread=fffff80002c02cc0
If you want the ID of the thread, you'll need to dig it out of the ETHREAD. Luckily, the #$thread is typed as a pointer to an ETHREAD if you're using the C++ evaluator:
0: kd> ?? #$thread->Cid
struct _CLIENT_ID
+0x000 UniqueProcess : 0x00000000`00001408 Void
+0x008 UniqueThread : 0x00000000`0000144c Void
-scott
is there any way in WinDbg to determine since what date/time a Windows thread is blocked by functions like WaitForSingleObjects or WaitForMultipleObjects?
I know how to do this in kernel debugging (using !thread), but I have no idea how to do this in user-mode debugging.
In WinDbg, you can use !runaway to get thread timings:
!runaway
!runaway 1
(user time)
!runaway 2
(kernel time)
!runaway 4
(elapsed time)
(You'll find these documented as 0, 1 and 2 some places, but in my experience those don't work. Perhaps it depends on the WinDbg version or something...)
You can compute the time spent suspended by subtracting a thread's user and kernel time from it's elapsed time, but unfortunately I don't know of any way (short of writing a WinDbg plugin) to get WinDbg to do that for you.
If you're not set on WinDbg, you can use Process Explorer to get the same information. When you right-click a process and select the threads tab in the properties dialog, you get a list of all the threads in the process. Selecting a particular thread will show the same timing information, among other things.
I have a process suspended at breakpoint under visual studio debugger.
I can attach as many as cdb (Microsoft's console debugger) in non-invasive mode as
cdb -p pid -pvr
How to achieve the same using my own program which uses Debug Engine API.
IDebugClient* debugClient = 0;
(DebugCreate( __uuidof(IDebugClient), (void **)&debugClient );
debugClient->AttachProcess(0,id,DEBUG_ATTACH_NONINVASIVE
|DEBUG_ATTACH_NONINVASIVE_NO_SUSPEND);
This code causes E_INVALIDARG. Is this combination is not allowed ? The one below works, but when it calls GetStackTrace, it returns E_UNEXPECTED.
debugClient->AttachProcess(0,id,DEBUG_ATTACH_NONINVASIVE);
debugControl->GetStackTrace(0, 0, 0, pStackFrames, maxFrames, &framesFilled);
I am interested to attach to a process already at debug break noninvasive way , and get a few local variable from its current stack & some global variable value.
Secondly, can someone point me the function used to dump the content of memory for a symbol iteratively like !stl does. I need to write a plugin to dump one of my vector like structure.
Thanks
I believe there's nothing wrong with
DEBUG_ATTACH_NONINVASIVE|DEBUG_ATTACH_NONINVASIVE_NO_SUSPEND
combination - it is perfectly permissible and is even featured in assert sample.
Otherwise, as far as documentation goes - it is not that detailed. I would suggest debugging your extension with the help of wt (trace and watch data) - it is particularly useful when you need to locate the exact subroutine that is returning an error which might provide you with better insight on the problem.
As for remotely accessing typed data in your apps from an extension, I've found ExtRemoteTyped class (available in engextcpp.hpp in the sdk subfolder) to be very helpful and intuitive to use.