How are operating systems typically debugged? They cannot be stepped through with a debugger like simple console programs, and the build times are too large to repeatedly make small changes and recompile the whole thing.
They aren't debugged as a multi-gigabyte programs! :)
If you mean the individual user-mode components, they can mainly be debugged just like normal programs and libraries (because they are normal programs/libraries!).
For kernel-mode components, though, each OS has its own mechanism; here is some information regarding the way that we do kernel debugging in Windows. It can be done using the help of another machine connected to the machine you're debugging, via a serial port or something. I'm not familiar with the process itself, but that's the gist of how they work. (You need to set some boot loader options so that the system is ready for the debugger to be connected as early as possible.)
It depends on which part of the operating system you're talking about. When I worked at MSFT, I worked on the IE team. We debugged IE and the shell (Windows Explorer) in Visual Studio and stepped through them line by line all day long. Though, sometimes, it's easier to debug using a command line tool such as NTSD.
If, however, you want to debug anything in Kernel land such as the OS kernel or device drivers, which I suspect is really what you're asking, then you must use the Kernel debugger. For Windows that is a command line tool called kd, and generally you run the debugger on one machine and remotely debug the target.
There are a whole set of techniques throughout history from flashing lights on the console, to the use of hardware devices like an ICE, to more modern techniques utilizing fairly standard debuggers. One technique that is more common among OS developers then application developers is the analysis of a core dump. Look at something like mdb on solaris for ideas about how Solaris kernel developers do some of their debugging. Also tracing technologies are used. Anywhere from fairly straightforward logging packages to more modern techniques like dtrace.
Also note that the techniques used depend on the layer of software. Initial boot tends to be a fairly hard place to get your fingers into. But after that the environment of modern operation systems looks more and more like the application setting you are use to. In the end, it is all code :)
Related
How can I compile a C++ program with a command-line interface and use it under Android 2.3.5 on my phone?
No - the model is completely different. Simple C++ programs are single threaded - they do whatever they have to do as quickly as they can in a single thread of execution and if they have to wait or block on something like retrieving data from the network then they just have to wait. They are given timeslices by a multitasking operating system and when they're finished they're finished.
In Android there is always one thread running which handles GUI interactions and passes the results into 'hooks' in your Activity instance. Anything that might block the GUI thread has to be farmed out to another thread, and call back on another method in your Activity. It's event-driven, and you have remarkably little control or certainty about things like object lifetime. So you need to program in a completely different way.
An emulator of some kind running as an Android app could - in principle - run C++ binaries compiled for a specific VM. But as far as I'm aware such an app doesn't exist and neither does the toolchain to produce such binaries. Google have discouraged such an approach too AFAIK. There are fully-fledged computer emulators but for obvious reasons they're mainly old 8-bit nostalgia fests :)
I'm a C++ programmer who recently got involved in Android programming and I'd recommend it. You'll think about programs in a different way from the single-threaded IFTT way you may be used to.
I want to write something like DaemonTools: a software that presents itself to the system as a real device (a DVD-ROM in the previous example) but it reads the data from a file instead. My requirement is not limited to DVD-ROM. The first goal is a joystick/gamepad for Windows.
I'm a web developer, so I don't know from where I could start such a project. I believe it will have to be written in C/C++, but other than that, I have no clue where to start.
Did anyone tried something like this and can give me some starting tips ?
Most drivers are written in either C or C++, so if you don't know those languages reasonably well, you'll want to get familiar with them before you start. Windows programming uses a lot of interesting shortcuts that might be confusing to a beginner - for example PVOIDs (typedef void* PVOID) and LPVOIDs (typedef void* far LPVOD;). You'll need to be happy with pointers as concepts as well as structures because you'll be using a lot of them. I'd suggest writing a really straightforward win32 app as an exercise in getting to grips with the Windows style of doing C/C++.
Your next port of call then is to navigate the Windows Driver Kit - specifically, you'll need it to build drivers for Windows. At this stage my ability to advise really depends on what you're doing and the hardware you have available etc, or whether or not you're really using hardware. You'll need to know how to drive your hardware and from there you'll need to choose an appropriate way of writing a driver - there are several different types of driver depending on what you need to achieve and it might be you can plug into one of these.
The windows driver kit contains quite a large number of samples, including a driver that implements a virtual toaster. These should provide you with starting points.
I strongly suggest you do the testing of this in a virtual machine. If your driver successfully builds, but causes a runtime error, the result could well crash windows entirely if you're in kernel-mode. You will therefore save yourself some pain by being able to revert the virtual machine if you damage it, as well as not having to wait on your system restarting. It'll also make debugging easier as virtual serial cables can be used.
This is quite a big undertaking, so before you start, I'd research Windows development more thoroughly - check you can't do it using the Windows APIs first, then have a look at the user-mode driver framework, then finally and only if you need to, look at the kernel level stuff.
I have a program written in CA-Clipper 5.2 and linked with Blinker 7. I recently learned how to compile it into protected mode in place of real mode. Now the real mode debugger won't work with the program. So now I need a way to debug my code. The documentation for Blinker says to use "NuMega SoftICE" or "Periscope". I'm not family with those debuggers, and can't find much on them from Google. It sounds like SofeICE was turn into some type of hacking tool. Any suggestion on a way to debug my program?
NuMega was bought out, and SoftICE was killed (something like five years ago, if memory serves). It was a kernel debugger, which is a kind of tool some hackers (in either sense of the word) find useful, but wasn't really a hacking tool as such. (Silly trivia of the day: people who beta-tested the original version of SoftICE for Windows NT got a T-shirt that read: "...and they said it couldn't be done!").
Periscope is (was) an in-circuit emulator. It was a board with a plug to fit into your CPU socket, and a socket where you put the original CPU. It would then monitor all the traffic over the CPU bus, providing a lot of debugging capability that most software debuggers can't even hope to match. As CPU buses got faster, however, it got extremely expensive, and eventually got to the point that there was no market left. There was definitely a version for the 486 (I've used it), but I don't think there was ever a version for the Pentium or newer.
As to what you would use: the HX DOS Extender is probably the only DOS Extender still maintained. Their page lists debuggers that can be used with it. I certainly can't guarantee compatibility with the DOS extender you're using, but there's at least a chance one of them might work.
try watcom debugger with commandline startup: wd /tr=rsi
trap for rational systems dos extender
be sure to get latest version: open-watcom-c-dos-1.9.7z
it has problems doing search
but earlier versions do not work well
I am into new project, which should use microcontroller. The easiest way to program it is using parallel port. But, there are few things I hope you can help me with. Oh, and the preferred language is C and platform Windows.
So, I studied LPT ports and Windows a bit, and from what I learned the most important is: Since Windows NT based systems, you cannot use instructions for direct port manipulation. This should be, because now programs are run in different privilege mode, which doesn't support the kind of instructions that are used by outport() function.
But at this point, I don´t understand a few things. First, I thought that Windows actually used privilege levels since first protected mode version, but that's the wrong assumption.
But more importantly, I thought that Windows has included functions for just about any hardware communication. I mean, anything you do in Windows these days, you just call windows functions which further call kernel services. I assumed that outport() doesn´t use any Windows function, and just makes the communication itself, which is prohibited now. But I am literally shocked that there is no system function to control parallel ports in modern Windows systems. At least that's what I read.
But even if I could get the control of parallel port, there comes my second problem.
For programming the controller, I need to follow special protocol, especially timing. But since Windows is multitasked, I worry about what if Scheduler switches to another app, and therefore when is the right time to switch signals on LPT, my program just will not be able to run.
Oh, by the way, I know I could use any 3rd-party apps, but I just like to be able to do it myself, or at least before I use some 3rd-party app, I want to know how it works. And yes, you can program some microcontrollers just by parallel port with some resistors, I know this for sure.
Thanks.
For windows you need to install a DLL which contains a driver to run at elevated privileges to get access to the HW ports.
You can find such a library at :
http://logix4u.net/Legacy_Ports/Parallel_Port/Inpout32.dll_for_Windows_98/2000/NT/XP.html
There are also some links to sample code.
I do not know which uController you are using, but I programmed in the past a variety of them and never had issues with timings, well for programming at least. The programming protocols are usually robust enough to deal with the jitter caused by multitasking. Just keep your clock edges and signa edges well separated and it should go fine.
I was going through some general stuff about operating systems and struck on a question. How will a developer debug when developing an operating system i.e. debug the OS itself? What tools are available to debug for the OS developer?
Debugging a kernel is hard, because you probably can't rely on the crashing machine to communicate what's going on. Furthermore, the codes which are wrong are probably in scary places like interrupt handlers.
There are four primary methods of debugging an operating system of which I'm aware:
Sanity checks, together with output to the screen.
Kernel panics on Linux (known as "Oops"es) are a great example of this. The Linux folks wrote a function that would print out what they could find out (including a stack trace) and then stop everything.
Even warnings are useful. Linux has guards set up for situations where you might accidentally go to sleep in an interrupt handler. The mutex_lock function, for instance, will check (in might_sleep) whether you're in an unsafe context and print a stack trace if you are.
Debuggers
Traditionally, under debugging, everything a computer does is output over a serial line to a stable test machine. With the advent of virtual machines, you can now wire one VM's execution serial line to another program on the same physical machine, which is super convenient. Naturally, however, this requires that your operating system publish what it is doing and wait for a debugger connection. KGDB (Linux) and WinDBG (Windows) are some such in-OS debuggers. VMWare supports this story explicitly.
More recently the VM developers out there have figured out how to debug a kernel without either a serial line or kernel extensions. VMWare has implemented this in their recent stuff.
The problem with debugging in an operating system is (in my mind) related to the Uncertainty principle. Interrupts (where most of your hard errors are sure to be) are asynchronous, frequent and nondeterministic. If your bug relates to the overlapping of two interrupts in a particular way, you will not expose it with a debugger; the bug probably won't even happen. That said, it might, and then a debugger might be useful.
Deterministic Replay
When you get a bug that only seems to appear in production, you wish you could record what happened and replay it, like a security camera. Thanks to a professor I knew at Illinois, you can now do this in a VMWare virtual machine. VMWare and related folks describe it all better than I can, and they provide what looks like good documentation.
Deterministic replay is brand new on the scene, so thus far I'm unaware of any particularly idiomatic uses. They say it should be particularly useful for security bugs, too.
Moving everything to User Space.
In the end, things are still more brittle in the kernel, so there's a tremendous development advantage to following the Nucleus (or Microkernel) design, where you shave the kernel-mode components to their bare minimum. For everything else, you can use the myriad of user-space dev tools out there, and you'll be much happier. FUSE, a user-space filesystem extension, is the canonical example of this.
I like this last idea, because it's like you wrote the program to be writeable. Cyclic, no?
In a bootstrap scenario (OS from scratch), you'd probably have to introduce remote debugging capabilities (memory dumping, logging, etc.) in the OS kernel early on, and use a separate machine. Or you could use a virtual machine/hypervisor.
Windows CE has a component called KITL - Kernel Independent Transport Layer. I guess the title speaks for itslf.
You can use a VM: eg. debug ring0 code with bochs/gdb
or Debugging NetBSD kernel with qemu
or a serial line with something like KDB.
printf logging
attach to process
serious unit tests
etc..
Remote debugging with kernel debuggers, which can also be done via virtualization.
Debugging an operating system is not for the faint of heart. Because the kernel is being debugged, your options would be quite limited. Copious amount of printf statements is one trick, and furthermore, it depends on really what 'operating system' is being debugged, we could be talking about
Filesystem
Drivers
Memory management
Raw Disk input/output
Screen input/output
Kernel
Again, it is a widely varying exercise as in the above, they all interact with one another. Even more complicated is the fact, supposing you were to debug the kernel, how would you do it if the runtime environment is not properly set (by that, I am talking about the kernel's responsibility for loading binary executables).
Some kernels may (not all of them have them) incorporate a simple debug monitor, in fact, if I rightly recall, in the book titled 'Developing your own 32bit Operating System' by Richard A Burgess, Sams publishing, he incorporated a debug monitor which displays various states of the CPU, registers and so on.
Again, take into account of the fact that the binary executables require a certain loading mechanism, for example a gdb equivalent, if the environment for loading binaries are not set up, then your options are quite limited.
By using copious amount of printf statements to display errors, logs etc to a separate terminal or to a file is the best line of debugging, it does sound a nightmare but it would be worth the effort to do so.
Hope this helps,
Best regards,
Tom.