How to debug hacked Linux Kernel code - linux-kernel

I am newbie in this filed.
I don't have no good way to debug it. Just build it, reboot and run.
Any hint is very appreciated!

Depending on what you are hacking, it might be better to use UML (User Mode Linux). If you're messing with non-hardware dependent code, then I think it will help a great deal.
UML allows you to compile the kernel as an ordinary user mode program, and run it as any other application on your system. Because it runs like a regular application, you can very easily debug it with gdb, or any other debugger of choice.
Here's a good start for UML

I'd recommend you to start from reading 4th chapter ("Debugging Techniques") of classical Linux Device Drivers book.

Related

debugging a simulated processor

I have an embedded project which runs on a 68332 processor target (68k family). There is no OS on the target. We have a custom simulator that will allow our code to execute within Windows. The simulator is completely without our control to modify. Basically the simulator is executing the machine code which isn't very good when you need to debug. What I would really like to do is interface a debugger to allow us to debug at the source level rather than at the machine/assembly level. Has anyone ever done such a thing? Is there a spec that debuggers support? Perhaps would something like gdb work for this? Any advice is appreciated.
This is not necessarily an answer to your question - I'm not familiar with hooking up an existing 3rd-party debugger to a program executing inside a VM so I can't advise about that.
However, you control the source of your simulator so you can try implementing an interface (maybe a local socket, etc.) where your simulator keeps reporting status information about the code that's executing and links it up with source files by reading debug information from some generated debugging database. You'd likely have to support reading the debugging format of the compiler that compiles your 68k code and then use that information to link back assembly instructions to source code lines.
This way you're effectively implementing a debugger, but since you already have the simulator (a VM really), that's probably not too much of extra work - the simulator already has all state information about the executing 68k code, you just need a way to temporarily pause execution and extract state information during pause. Stepping through code after that is probably a trivial repeat of these steps.

Starting point for coding a virtual device

I want to write something like DaemonTools: a software that presents itself to the system as a real device (a DVD-ROM in the previous example) but it reads the data from a file instead. My requirement is not limited to DVD-ROM. The first goal is a joystick/gamepad for Windows.
I'm a web developer, so I don't know from where I could start such a project. I believe it will have to be written in C/C++, but other than that, I have no clue where to start.
Did anyone tried something like this and can give me some starting tips ?
Most drivers are written in either C or C++, so if you don't know those languages reasonably well, you'll want to get familiar with them before you start. Windows programming uses a lot of interesting shortcuts that might be confusing to a beginner - for example PVOIDs (typedef void* PVOID) and LPVOIDs (typedef void* far LPVOD;). You'll need to be happy with pointers as concepts as well as structures because you'll be using a lot of them. I'd suggest writing a really straightforward win32 app as an exercise in getting to grips with the Windows style of doing C/C++.
Your next port of call then is to navigate the Windows Driver Kit - specifically, you'll need it to build drivers for Windows. At this stage my ability to advise really depends on what you're doing and the hardware you have available etc, or whether or not you're really using hardware. You'll need to know how to drive your hardware and from there you'll need to choose an appropriate way of writing a driver - there are several different types of driver depending on what you need to achieve and it might be you can plug into one of these.
The windows driver kit contains quite a large number of samples, including a driver that implements a virtual toaster. These should provide you with starting points.
I strongly suggest you do the testing of this in a virtual machine. If your driver successfully builds, but causes a runtime error, the result could well crash windows entirely if you're in kernel-mode. You will therefore save yourself some pain by being able to revert the virtual machine if you damage it, as well as not having to wait on your system restarting. It'll also make debugging easier as virtual serial cables can be used.
This is quite a big undertaking, so before you start, I'd research Windows development more thoroughly - check you can't do it using the Windows APIs first, then have a look at the user-mode driver framework, then finally and only if you need to, look at the kernel level stuff.

How are operating systems debugged?

How are operating systems typically debugged? They cannot be stepped through with a debugger like simple console programs, and the build times are too large to repeatedly make small changes and recompile the whole thing.
They aren't debugged as a multi-gigabyte programs! :)
If you mean the individual user-mode components, they can mainly be debugged just like normal programs and libraries (because they are normal programs/libraries!).
For kernel-mode components, though, each OS has its own mechanism; here is some information regarding the way that we do kernel debugging in Windows. It can be done using the help of another machine connected to the machine you're debugging, via a serial port or something. I'm not familiar with the process itself, but that's the gist of how they work. (You need to set some boot loader options so that the system is ready for the debugger to be connected as early as possible.)
It depends on which part of the operating system you're talking about. When I worked at MSFT, I worked on the IE team. We debugged IE and the shell (Windows Explorer) in Visual Studio and stepped through them line by line all day long. Though, sometimes, it's easier to debug using a command line tool such as NTSD.
If, however, you want to debug anything in Kernel land such as the OS kernel or device drivers, which I suspect is really what you're asking, then you must use the Kernel debugger. For Windows that is a command line tool called kd, and generally you run the debugger on one machine and remotely debug the target.
There are a whole set of techniques throughout history from flashing lights on the console, to the use of hardware devices like an ICE, to more modern techniques utilizing fairly standard debuggers. One technique that is more common among OS developers then application developers is the analysis of a core dump. Look at something like mdb on solaris for ideas about how Solaris kernel developers do some of their debugging. Also tracing technologies are used. Anywhere from fairly straightforward logging packages to more modern techniques like dtrace.
Also note that the techniques used depend on the layer of software. Initial boot tends to be a fairly hard place to get your fingers into. But after that the environment of modern operation systems looks more and more like the application setting you are use to. In the end, it is all code :)

Step-by-step execution for Intel AT&T assembler?

I'm writing a compiler that converts source code (written in a small imperative programming language) to Intel AT&T 32-bit assembler.
I tend to spend a lot of time debugging, because of nasty offset-mistakes etc. in the generated code, and I would like to know if anyone knows of a tool to "walk through" the generated assembler code step-by-step, visualizing what's on the stack etc.
I use Ubuntu Linux as my development platform, and I'm comfortable with the terminal -- a GUI-program would be nice though. Does it exist? Or is there a good reason it doesn't (maybe the problem isn't so straightforward..?)
If you have good ideas for approaching debugging tasks in assembly code, I'll be glad to hear from you!
I like EDB (Evan's Debugger) on Linux. It has a nice, easy-to-use, QT4-based GUI. Its developer's goal is to make it similar to OllyDbg. And it's being actively maintained:
EDB on FreshMeat
I'm pretty sure it's installable through Synaptic on Ubuntu as well. Enjoy!
Is the end result of the compile process something that you can actually execute, and therefore examine in a debugger? If so the Data Display Debugger (ddd) might be useful.
My experience with debuggers such as Olly and EDB is quite sparse, so I wasn't able to solve my problem with those. I ended up
scattering calls around to a Debug function in the source code, nailing down bad register values
letting the compiler output HTML-formatted code with useful metadata for different iterations in the liveness analysis etc.

Which is easier to write drivers for? Windows or LInux?

I am trying to learn a little about driver development and my question is, which OS has an easier API to develop drivers with? Windows or Linux?
Remember that I am new to this.
Thanks
If you are doing this for learning (I guess its your case), then go for Linux, you will have a more general ideal on how an OS works, you will interact with low level implementation straight to the core.
If you are just writing a driver for a device, I recommend using Visual Studio under Windows, it has a lot of features that will really help on your devel.
I am pretty sure you'll get lots of Windows vs Linux war here, unfortunately.
On Windows, you'll get pretty good development tools, with some awkward stuff, but that you can get along with easily. API is documented provided you have access to a Windows Developer account. Remote debugging is supported and it helps a lot for these kernel stuff that are triggering BSOD so quickly.
On Linux, you'll need to get your hands more dirty. Rubini's "Linux Device Drivers" is a very good starting point. Openness will bring you more flexibility, and more understanding of what is happening.
This tutorial might be helpful.
It is about writing a simple device driver for linux.
My personal idea is Linux. You can see the entire code other than an API.
I think Windows is good.
Because it has a rich set of API's which is easier to use

Resources