I understand how powerful windbg can be at debugging, but when is an appropriate or best time to use it to debug an issue? Is it more issue specific, or just experience, intuition, and knowing that using it can just get the job done best?
It's a little bit of all those things, and a lot of personal perference. Many WinDbg people only use WinDbg so that's what they are best at debugging with.
WinDbg also has some good extensions out there like SOS. So a particular extension might provide you with the specific piece of information that another debugger does not.
One reason to use a different debugger in certain circumstances is if you believe the debugger is incorrect. This is rare of course. For things like stack walking for instance, the debuggers use different methods, so you can confirm the stack is what you expect by using the other.
So sum up, for most issues it doesn't matter. It's whatever you are best at using. For some particular issues it's what you say, knowing which tool is the best for that particular issue.
While Windbg is also a fine tool for user-mode debugging, if you end up doing kernel-mode debugging it is really the only serious choice.
The kernel-mode debugging scenario often involves two machines, a debugger and a debuggee. You will be running Windbg on a debugger machine which is connected to the debuggee over a serial connection, Firewire or USB. In this scenario you can "freeze" the target machine and have full control over everything running on it. Often your debuggee (the target) will be a virtual machine running under VMWare or similar -- in this case the connection also typically uses virtual serial ports.
Here are instructions from VMWare on how to set up kernel debugging of a virtual machine:
http://www.vmware.com/support/ws5/doc/ws_devices_serial_advanced_example_debugging.html
You can also use VirtualKD which makes the setup easier and the connection much faster:
http://virtualkd.sysprogs.org/
You can also use Windbg for local kernel debugging. In this case, you only have a single machine where you connect Windbg to the running kernel. You cannot "freeze" the machine, as it would also freeze Windbg running on the same machine, but you can analyze the contents of memory and so on.
Good point. Another good solution for virtual kernel debugging is LiveKd from sysinternals.
http://technet.microsoft.com/en-us/sysinternals/bb897415.aspx
Related
From what I understand, on a high level, user mode debugging provides you with access to the private virtual address for a process. A debug session is limited to that process and it cannot overwrite or tamper w/ other process' virtual address space/data.
Kernel mode debug, I understand, provides access to other drivers and kernel processes that need full access to multiple resources, in addition to the original process address space.
From this, I get to thinking that kernel mode debugging seems more robust than user mode debugging. This raises the question for me: is there a time, when both options of debug mode are available, that it makes sense to choose user mode over a more robust kernel mode?
I'm still fairly new to the concept, so perhaps I am thinking of the two modes incorrectly. I'd appreciate any insight there, as well, to better understand anything I may be missing. I just seem to notice that a lot of people seem to try to avoid kernel debugging. I'm not entirely sure why, as it seems more robust.
The following is mainly from a Windows background, but I guess it should be fine for Linux too. The concepts are not so different.
Some inline answers first
From what I understand, on a high level, user mode debugging provides you with access to the private virtual address for a process.
Correct.
A debug session is limited to that process
No. You can attach to several processes at the same time, e.g. with WinDbg's .tlist/.attach command.
and it cannot overwrite or tamper w/ other process' virtual address space/data.
No. You can modify the memory, e.g. with WinDbg's ed command.
Kernel mode debug, I understand, provides access to other drivers and kernel processes that need full access to multiple resources,
Correct.
in addition to the original process address space.
As far as I know, you have access to physical RAM only. Some of the virtual address space may be swapped, so not the full address space is available.
From this, I get to thinking that kernel mode debugging seems more robust than user mode debugging.
I think the opposite. If you write incorrect values somewhere in kernel mode, the PC crashes with a blue screen. If you do that in user mode, it's only the application that crashes.
This raises the question for me: is there a time, when both options of debug mode are available, that it makes sense to choose user mode over a more robust kernel mode?
If you debug an application only and no drivers are involved, I prefer user mode debugging.
IMHO, kernel mode debugging is not more robust, it's more fragile - you can really break everything at the lowest level. User mode debugging provides the typical protection against crashes of the OS.
I just seem to notice that a lot of people seem to try to avoid kernel debugging
I observe the same. And usually it's not so difficult once they try it. In my debugging workshops, I explain processes and threads from kernel point of view and do it live in the kernel. And once people try kernel debugging, it's not such a mystery any more.
I'm not entirely sure why, as it seems more robust.
Well, you really can blow up everything in kernel mode.
User mode debugging
User mode debugging is the default that any IDE will do. The integration is usually good, in some IDEs it feels quite native.
During user mode debugging, things are easy. If you access memory that is paged out to disk, the OS is still running and will simply page it in, so you can read and write it.
You have access to everything that you know from application development. There are threads and you can suspend or resume them. The knowledge you have from application development will be sufficient to operate the debugger.
You can set breakpoints and inspect variables (as long as you have correct symbols).
Some kinds of debugging is only available in user mode. E.g. the SOS extension for WinDbg to debug .NET application only works in user mode.
Kernel debugging
Kernel debugging is quite complex. Typically, you can't simply do local kernel debugging - if you stop somewhere in the kernel, how do you control the debugger? The system will just freeze. So, for kernel debugging, you need 2 PCs (or virtual PCs).
During kernel mode debugging, things are complex. While you are just inside an application, a millisecond later, some interrupt occurs and does something completely different. You don't only have threads, you also need to deal with call stacks that are outside your application, you'll see CPU register content, instruction pointers etc. That's all stuff a "normal" app developer does not want to care about.
You don't only have access to everything that you implemented. You also have access to everything that Microsoft, Intel, NVidia and lots of other companies developed.
You cannot simply access all memory, because some memory that is paged out to the swap file will first generate a page fault, then involve some disk driver to fetch the data, potentially page out some other data, etc.
There is so much giong on in kernel mode and in order to not break it, you need to have really professional comprehension of all those topics.
Conclusion
Most developers just want to care about their source code. So if they are writing programs (aka. applications, scripts, tools, games), they just want user mode debugging. If "their code" is driver code, of course they want kernel debugging.
And of course Security Specialists and Crackers want kernel mode debugging because they want privileges.
I have a malicious .scr file which i heard steals your gaming account items. Is there a way to debug this .scr file in safe environment without executing it. I opened it in IDA PRO but did not find it much useful.
Any pointers regarding the debugging process will be highly appreciable.
Analysis of malware is risky. If you are committed you at least need a Linux host with Windows guest on a sacrificial computer that doesn't have a wireless card and is unplugged from internet. Malware often has a mechanism to detect the jail. Common VMs use standard drivers and checking if running in jail is easy. If in jail then do nothing is a one line of code. To make matters worse, decent malware often has a mechanism to jailbreak from a VM and infect host Windows. Don't even mention Sandboxie. Last, but not least - some malware starts revenging if you try to decompile it. So, instead of just a stolen game password you might have a big mess.
So, if you have a sacrificial computer and free time, then go ahead. But do it carefully.
First: DEBUG a program means that you execute it, so if you know that the program is a malware you should do it in a safe enviroment like a virtual machine.
Malware usually use to protect themselves from debuggers, so this kind of job needs so many skills.
A kind of analysis that you can do is to execute the malware in a virtual machine and take a snapshot of the machine, then analyze the dump with Volatily and see what connections have been made by the system, the state of process etc..
I'm just cross posting the same question I did on virtualbox.org. http://forums.virtualbox.org/viewtopic.php?f=9&t=26702&p=119139#p119139
If not breaking any rule, I'd appreciate to kwon more about it since stackoverflow promisses to be more dynamic!
"Hi,
I did some search and could not find any tool to debug a guest system from the early boot in virtual box. Although, I came across JCP, a x86 emulator in java that is not so powerful and beautyful but has a debug mode where one can view the Physical Memory, the CPU registers along other things. It also makes it possible to execute CPU instructions step by step and set break points, watchpoints and conditional ones. There's such thing in Virtual Box?
I think would be amazing to have it and be able to inspect the system while its running. For learn about PC architecture or as a tool to develop a kernel as well.
In the case you think its good idea (I think it is) how can it be achieved? I'm interested in develop such sort o things and would like to know if it is feasible if not already implemented somewhere."
EDT: Are modern x86 able to interrupt its execution just after a cpu cycle and pass execution addres to another code to just do this? Yes, the trap flag can be set to put the processor in step by step execution mode. x86 will execute one instruction and call INT 3.
Contrary to what is stated above, VirtualBox now contains a (limited) debugger. Add --dbg to the command line when starting the VM. For more information consult:
12.1.3. The built-in VM debugger
The OSDev wiki has some useful information on debugging a guest operating system, though according to this page VirtualBox doesn't have a debugger at present. I've been using QEmu with the GDB stub and it works quite nicely, so you might like to give that a go instead.
I have compiled my own Kernel module and now I would like to be able to load it
into the GNU Debugger GDB. I did this once, a year ago or so to have a look
at the memory layout. It worked fine then, but of course I was too silly to
write down the single steps I took to accomplish this... Can anyone enlighten
me or point me to a good tutorial?
Thank you so much
For kernels > 2.6.26 (i.e. after May 2008), the preferred way is probably to use "kgdb light" (not to be confused with its ancestor kgdb, available as a set of kernel patches).
"kgdb light" is now part of the kernel (in by default in current Ubuntu kernels, for instance), and it's capabilities are improving fast (Jason Wessel is working on it - possible google key).
Drawback: You need two machines, the one you're debugging and the development machine (host) where gdb runs. Currently, those two machines can only be linked through a serial link.
kgdb runs in the target machine where it handles the breakpoints, stepping, etc. and the remote debugging protocol use to talk with the development machine.
gdb runs in the development machine where it handles the user interface.
An USB-to-serial adapter works OK on the development machine, but currently, you need a real UART on the target machine - and that's not so frequent anymore on recent hardware.
The (terse) kgdb documentation is in the kernel sources, in
Documentation/DocBook
I suggest you google around for "kgdb light" for the complete story.
Again, don't confuse kgdb and kgdb light, they come together in google searches but are mostly different animals. In particular, info from linsyssoft.com relate to the "ancestor" kgdb, so try queries like:
kgdb module debugging -"linsyssoft.com" -site:linsyssoft.com
and discard articles prior to May 2008 / 2.6.26 kernel.
Finally, for module debugging, you need to manually load the module symbols in the dev machine for all the code and sections you are interested in. That's a bit too long to address here, but some clues there, there and there.
Bottom line is, kgdb is a very welcome improvement but don't expect this trip to be as easy as running gdb in user mode. Yet. :)
It has been a while since I was actively developing drivers for Linux, so maybe my answer is a bit out of date. I would say you cannot use GDB. If at all, only to debug post mortem on dump files. To debug you should rather use a kernel debugger. Build the kernel with a kernel debugger enabled (there is one out-of-the box debugger for 2.6, which was lacking at the time I was active). I used the kernel patches for KDB from Sun ftp://oss.sgi.com/www/projects/kdb/download/, which I was quite happy with. A user space tool won't be of much use unless new gdb communicate somehow with the internal kernel debugger (which anyway you would have to activate)
I hope this gives you at least some hints, while not being a detailled answer. Better than no answer at all. Regards.
I suspect what you did was
gdb /boot/vmlinux /proc/kcore
Of course you can't actually do any debugging, but it's certainly good enough to have a poke around the kernel.
Just curious here: is it possible to invoke a Windows Blue Screen of Death using .net managed code under Windows XP/Vista? And if it is possible, what could the example code be?
Just for the record, this is not for any malicious purpose, I am just wondering what kind of code it would take to actually kill the operating system as specified.
The keyboard thing is probably a good option, but if you need to do it by code, continue reading...
You don't really need anything to barf, per se, all you need to do is find the KeBugCheck(Ex) function and invoke that.
http://msdn.microsoft.com/en-us/library/ms801640.aspx
http://msdn.microsoft.com/en-us/library/ms801645.aspx
For manually initiated crashes, you want to used 0xE2 (MANUALLY_INITIATED_CRASH) or 0xDEADDEAD (MANUALLY_INITIATED_CRASH1) as the bug check code. They are reserved explicitly for that use.
However, finding the function may prove to be a bit tricky. The Windows DDK may help (check Ntddk.h) - I don't have it available at the moment, and I can't seem to find decisive info right now - I think it's in ntoskrnl.exe or ntkrnlpa.exe, but I'm not sure, and don't currently have the tools to verify it.
You might find it easier to just write a simple C++ app or something that calls the function, and then just running that.
Mind you, I'm assuming that Windows doesn't block you from accessing the function from user-space (.NET might have some special provisions). I have not tested it myself.
I do not know if it really works and I am sure you need Admin rights, but you could set the CrashOnCtrlScroll Registry Key and then use a SendKeys to send CTRL+Scroll Lock+Scroll Lock.
But I believe that this HAS to come from the Keyboard Driver, so I guess a simple SendKeys is not good enough and you would either need to somehow hook into the Keyboard Driver (sounds really messy) or check of that CrashDump has an API that can be called with P/Invoke.
http://support.microsoft.com/kb/244139
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters
Name: CrashOnCtrlScroll
Data Type: REG_DWORD
Value: 1
Restart
I would have to say no. You'd have to p/invoke and interact with a driver or other code that lives in kernel space. .NET code lives far removed from this area, although there has been some talk about managed drivers in future versions of Windows. Just wait a few more years and you can crash away just like our unmanaged friends.
As far as I know a real BSOD requires failure in kernel mode code. Vista still has BSOD's but they're less frequent because the new driver model has less drivers in kernel mode. Any user-mode failures will just result in your application being killed.
You can't run managed code in kernel mode. So if you want to BSOD you need to use PInvoke. But even this is quite difficult. You need to do some really fancy PInvokes to get something in kernel mode to barf.
But among the thousands of SO users there is probably someone who has done this :-)
You could use OSR Online's tool that triggers a kernel crash. I've never tried it myself but I imagine you could just run it via the standard .net Process class:
http://www.osronline.com/article.cfm?article=153
I once managed to generate a BSOD on Windows XP using System.Net.Sockets in .NET 1.1 irresponsibly. I could repeat it fairly regularly, but unfortunately that was a couple of years ago and I don't remember exactly how I triggered it, or have the source code around anymore.
Try live videoinput using directshow in directx8 or directx9, most of the calls go to kernel mode video drivers. I succeded in lots of blue screens when running a callback procedure from live videocaptureing source, particulary if your callback takes a long time, can halt the entire Kernel driver.
It's possible for managed code to cause a bugcheck when it has access to faulty kernel drivers. However, it would be the kernel driver that directly causes the BSOD (for example, uffe's DirectShow BSODs, Terence Lewis's socket BSODs, or BSODs seen when using BitTorrent with certain network adapters).
Direct user-mode access to privileged low-level resources may cause a bugcheck (for example, scribbling on Device\PhysicalMemory, if it doesn't corrupt your hard disk first; Vista doesn't allow user-mode access to physical memory).
If you just want a dump file, Mendelt's suggestion of using WinDbg is a much better idea than exploiting a bug in a kernel driver. Unfortunately, the .dump command is not supported for local kernel debugging, so you would need a second PC connected over serial or 1394, or a VM connected over a virtual serial port. LiveKd may be a single-PC option, if you don't need the state of the memory dump to be completely self-consistent.
This one doesn't need any kernel-mode drivers, just a SeDebugPrivilege. You can set your process critical by NtSetInformationProcess, or RtlSetProcessIsCritical and just kill your process. You will see same bugcheck code as you kill csrss.exe, because you set same "critical" flag on your process.
Unfortunately, I know how to do this as a .NET service on our server was causing a blue screen. (Note: Windows Server 2008 R2, not XP/Vista).
I could hardly believe a .NET program was the culprit, but it was. Furthermore, I've just replicated the BSOD in a virtual machine.
The offending code, causes a 0x00000f4:
string name = string.Empty; // This is the cause of the problem, should check for IsNullOrWhiteSpace
foreach (Process process in Process.GetProcesses().Where(p => p.ProcessName.StartsWith(name, StringComparison.OrdinalIgnoreCase)))
{
Check.Logging.Write("FindAndKillProcess THIS SHOULD BLUE SCREEN " + process.ProcessName);
process.Kill();
r = true;
}
If anyone's wondering why I'd want to replicate the blue screen, it's nothing malicious. I've modified our logging class to take an argument telling it to write direct to disk as the actions prior to the BSOD weren't appearing in the log despite .Flush() being called. I replicated the server crash to test the logging change. The VM duly crashed but the logging worked.
EDIT: Killing csrss.exe appears to be what causes the blue screen. As per comments, this is likely happening in kernel code.
I found that if you run taskkill /F /IM svchost.exe as an Administrator, it tries to kill just about every service host at once.