Does the operating system assumes anything about callee-saved registers when control returns to it?
I've wondered whether the OS, say Windows, assumes anything about the callee-saved registers like ebp, esi, edi?
In other words, does the OS require the value in any of these registers preserved, when control transfers back to it (ret in main)?
I cannot find anything specified, but I guess the answer is no (having looked at compiler generated code). Is there any documentation on the topic?
Windows 32 is designed to have process isolation.
Nothing* that a process does can cause another process (including) the operating system itself to fail.
For this reason it does not matter what you do with the registers upon exit.
The only exception is esp. If the stack pointer is messed up your application will terminate with a stack fault or access violation.
This will still not affect the OS however, it will merely terminate your app slightly early.
*Obviously this does not include the effects on the system by legitimate system calls, or the exploitation of bugs.
Note that the ret in main does not return control to the OS. Almost all Win32 c applications have a runtime library included. If so the ret in main returns to some initialization code that look like this:
//pseudo-init
do set up (setup command line params for main to read).
call main;
call Windows.ExitProcess();
Having a 'clean' exit to Windows is important to an application so it can clean up its own resources (close files etc). The OS does not really care. If an application does not clean up after itself, the OS will do the job for it.
Much worse than having a crashing up is a 'hung' one. If an application is stuck in an endless loop, or worse an endless loop that keeps claiming more and more resources then the system can be brought to its knees quite easily.
I would like to set a memory breakpoints on access in windbg in the kernel mode debugger
I want the debugger breaks everytime a specific module in usermode is hit with the kernel debugger.
but I've read somewhere its impossible to set it, in order to make a memory breakpoints I have to write a plugin to make it
I tried to use SDbgExt plugin with the !vprotect command, but it fails to set memory bp
If I have to write a plugin to allow memory bp in kernel mode It has to be a driver?
I've read some chapters in windows internals book, but it doesn't help me at all.
I couldn't find too much info how to start deal with it
You can set breakpoints on user mode addresses from kernel mode. The only thing you should take care is to switch to the right process with ".process /i " command
If it is a one-off breakpoint -- that is, you are content with process being destroyed by debugging -- zero out the entire module using e command (edit memory). Set the whole thing to cc (which is int 3 as far as I remember)... zeros will do as well. You will break as soon as you touch any of the module's code.
Next step, remember where you were (relative to the module) and set a proper breakpoint.
Hope that helps.
(editing) Do you have full symbols? If you do, did you try bm module!*
Sounds like you want to set a "breakpoint on access" but instead of specifying an address you want to specify a range? I have never seen it done in windbg. The BA breakpoints uses HW debug registers instead of inserting INTs like SW breakpoints so this is definitely HW platform specific.
I have done this on an ARM chipset once using a HW debugger. ETM on ARM allows you to set triggers on address ranges.
I know that software breakpoints in an executable file can work through replacing some assembler instruction at the desired place with another one, which cause interrupt. So debugger can stop execution exactly at this place and replace this instruction with original one and ask user about what to do the next or call some commands and etc.
But code of such executable file is not used by another programs and has only one copy in memory. How can software breakpoints work with a shared libraries? For instance, how software breakpoints work if I set one at some internal function of C-library (as I understand it has only one copy for all the applications, so we cannot just replace some instruction in it)? Are there any "software breakpoints" techniques for that purpose?
The answer for Linux is that the Linux kernel implements COW (Copy-on-Write): If the code of a shared library is written to, the kernel makes a private duplicate copy of the shared page first, remaps internally virtual memory just for that process to the copy, and allows the application to continue. This is completely invisible to userland applications and done entirely in the kernel.
Thus, until the first time a software breakpoint is put into the shared library, its code is indeed shared; But afterwards, not. The process thereafter operates with a dirty but private copy.
This kernel magic is what allows the debugger to not cause every other application to suddenly stop.
On OSes such as VxWorks, however, this is not possible. From personal experience, when I was implementing a GDB remote debug server for VxWorks, I had to forbid my users from ever single-stepping within semTake() and semGive() (the OS semaphore functions), since a) GDB uses software breakpoints in its source-level single-step implementation and b) VxWorks uses a semaphore to protect its breakpoints list...
The unpleasant consequence was an interrupt storm in which a breakpoint would cause an interrupt, and within this interrupt there would be another interrupt, and another and another in an unescapable chain resistant even to Ctrl-Z. The only way out was to power off the machine.
I was debugging a program in which I hit
int 0x80
I know this means a system call and then the kernel executed it. However, GDB does not allow me to look at the instructions run by the kernel while executing this system call. It just executes the system call and takes me to the next instruction.
Is there anyway I can look into the kernel mode code while debugging a user mode program? If not, then what are the over best alternatives available to me?
Is there anyway I can look into the kernel mode code while debugging a user mode program?
No.
(Actually, you could do that if you use UML, but that is likely too complicated for you to set up.)
I keep wondering how does a debugger work? Particulary the one that can be 'attached' to already running executable. I understand that compiler translates code to machine language, but then how does debugger 'know' what it is being attached to?
The details of how a debugger works will depend on what you are debugging, and what the OS is. For native debugging on Windows you can find some details on MSDN: Win32 Debugging API.
The user tells the debugger which process to attach to, either by name or by process ID. If it is a name then the debugger will look up the process ID, and initiate the debug session via a system call; under Windows this would be DebugActiveProcess.
Once attached, the debugger will enter an event loop much like for any UI, but instead of events coming from the windowing system, the OS will generate events based on what happens in the process being debugged – for example an exception occurring. See WaitForDebugEvent.
The debugger is able to read and write the target process' virtual memory, and even adjust its register values through APIs provided by the OS. See the list of debugging functions for Windows.
The debugger is able to use information from symbol files to translate from addresses to variable names and locations in the source code. The symbol file information is a separate set of APIs and isn't a core part of the OS as such. On Windows this is through the Debug Interface Access SDK.
If you are debugging a managed environment (.NET, Java, etc.) the process will typically look similar, but the details are different, as the virtual machine environment provides the debug API rather than the underlying OS.
As I understand it:
For software breakpoints on x86, the debugger replaces the first byte of the instruction with CC (int3). This is done with WriteProcessMemory on Windows. When the CPU gets to that instruction, and executes the int3, this causes the CPU to generate a debug exception. The OS receives this interrupt, realizes the process is being debugged, and notifies the debugger process that the breakpoint was hit.
After the breakpoint is hit and the process is stopped, the debugger looks in its list of breakpoints, and replaces the CC with the byte that was there originally. The debugger sets TF, the Trap Flag in EFLAGS (by modifying the CONTEXT), and continues the process. The Trap Flag causes the CPU to automatically generate a single-step exception (INT 1) on the next instruction.
When the process being debugged stops the next time, the debugger again replaces the first byte of the breakpoint instruction with CC, and the process continues.
I'm not sure if this is exactly how it's implemented by all debuggers, but I've written a Win32 program that manages to debug itself using this mechanism. Completely useless, but educational.
In Linux, debugging a process begins with the ptrace(2) system call. This article has a great tutorial on how to use ptrace to implement some simple debugging constructs.
If you're on a Windows OS, a great resource for this would be "Debugging Applications for Microsoft .NET and Microsoft Windows" by John Robbins:
http://www.amazon.com/dp/0735615365
(or even the older edition: "Debugging Applications")
The book has has a chapter on how a debugger works that includes code for a couple of simple (but working) debuggers.
Since I'm not familiar with details of Unix/Linux debugging, this stuff may not apply at all to other OS's. But I'd guess that as an introduction to a very complex subject the concepts - if not the details and APIs - should 'port' to most any OS.
I think there are two main questions to answer here:
1. How the debugger knows that an exception occurred?
When an exception occurs in a process that’s being debugged, the debugger gets notified by the OS before any user exception handlers defined in the target process are given a chance to respond to the exception. If the debugger chooses not to handle this (first-chance) exception notification, the exception dispatching sequence proceeds further and the target thread is then given a chance to handle the exception if it wants to do so. If the SEH exception is not handled by the target process, the debugger is then sent another debug event, called a second-chance notification, to inform it that an unhandled exception occurred in the target process. Source
2. How the debugger knows how to stop on a breakpoint?
The simplified answer is: When you put a break-point into the program, the debugger replaces your code at that point with a int3 instruction which is a software interrupt. As an effect the program is suspended and the debugger is called.
Another valuable source to understand debugging is Intel CPU manual (Intel® 64 and IA-32 Architectures
Software Developer’s Manual). In the volume 3A, chapter 16, it introduced the hardware support of debugging, such as special exceptions and hardware debugging registers. Following is from that chapter:
T (trap) flag, TSS — Generates a debug exception (#DB) when an attempt is
made to switch to a task with the T flag set in its TSS.
I am not sure whether Window or Linux use this flag or not, but it is very interesting to read that chapter.
Hope this helps someone.
My understanding is that when you compile an application or DLL file, whatever it compiles to contains symbols representing the functions and the variables.
When you have a debug build, these symbols are far more detailed than when it's a release build, thus allowing the debugger to give you more information. When you attach the debugger to a process, it looks at which functions are currently being accessed and resolves all the available debugging symbols from here (since it knows what the internals of the compiled file looks like, it can acertain what might be in the memory, with contents of ints, floats, strings, etc.). Like the first poster said, this information and how these symbols work greatly depends on the environment and the language.