I am very curious in messing up with HW. But my top level "messing" so far was linked or inline assembler in C program. If my understanding of CPU and ring mode is right, I cannot directly from user mode app access some low level CPU features, like disabling interrupts, or changing protected mode segments, so I must use system calls to do everything I want.
But, if I am right, drivers can run in ring mode 0. I actually don´t know much about drivers, but this is what I ask for. I just want to know, is learning how to write your own drivers and than call them the way I should go, to do what I wrote?
I know I could write whole new OS (at least to some point), but what I exactly want to do is acessing some low level features of HW from standart windows application. So, is driver the way to go?
Short answer: yes.
Long answer: Managing access to low-level hardware features is exactly the job of the OS kernel and if you only want access to a single feature there's no need to start your own OS from scratch. Most modern OSes, such as WIndows, Linux, or the BSDs, allow you to add code to the kernel through kernel modules.
When writing a kernel module (or device driver), you write code that is going to be executed inside the OS kernel and will thus be running in CPU ring 0. Great power comes with great responsibility, which in this case means that you should really know what you're doing as there will be no pre-configured OS interface to prevent you from doing the wrong things. You should therefore study the manuals of your hardware (e.g., Intel's x86 software developer's manuals, device specs, ...) as well as standard operating systems development literature (where you're also going to find plenty on the web -- OSDev, OSDever, OSR, Linux Device Drivers).
If you want to play with HW write some programs for 16-bit real-mode (or even with your own transition to protected-mode). There you have to deal with ASM, BIOS interrupts, segments, video memory and a lot of other low-level stuff.
Related
NVIDIA has recently announced they are open-sourcing (a variant of) their GPU Linux kernel driver. They are not, however, open-sourcing the user-mode driver libraries (e.g. libcuda.so).
It's a gradual process and not all GPUs are supported initially, but regardless of these details: Is there some way that developers of user-space code can leverage this open-sourcing? Or is it only interesting/useful for kernel developers?
What I would personally love to be able to do is avoid having to make libcuda calls to get the current context. If that piece of information were somehow readable now from userspace, that could be neat. Of course that's just wishful thinking on my part - I don't know how to check what the driver directly "exposes" - if anything.
I'm now a bit experienced with using OpenGL, which I started using because it's said that it is the only way to invoke video card functions. (besides DirectX - which I like less than OpenGL)
For programming (e.g. in C/C++) the OS gives many APIs, like functions for printing. But these can also be bypassed, by coding in Assembly-language - and call much lower level APIs (which gain speed) or direct CPU calls.
So I started wondering why this wouldn't be possible on the video card. Why should an API like OpenGL or DirectX be needed? The process going on with those is:
API-call >
OS calls video card (with complex opcodes, I think) >
video card responses (in complex binary format) >
OS decodes this format and responses to user (in expected API format)
I believe this should decrease the speed of the rendering process.
So my question is:
Is there any possibility to bypass any graphical API (under Windows) and make direct calls to the video card?
Thanks,
Dennis
Using assembly or bypassing an api doesnt automatically make something faster, often slower as you dont know what the folks that wrote the library know.
it is absolutely possible yes, those libraries are just processor instructions that poke and peek at registers and ram, and you could just as easily poke and peek at registers and ram. The first problem is can you get that information, sure, you can look at the linux drivers or other open source resources. Second, much of the heavy lifting today is done in the graphics chip by logic or graphics processors, so the host is just a go between and not necessarily the bottleneck if there is a bottleneck. And yes you can program the gpus depending on your video card/chip, etc.
You need to determine where the bottleneck really is, if there really is one, maybe the bus is your problem, maybe the operating system is your problem, or the compiler, or the hard disk or the system memory, the processor and architecture itself, caches, etc. At the same time how will you ever learn how to find these things unless you try.
I recommend getting rid of windows completely, no operating system, go bare metal. Take the linux and other open source resources plus anything you can get from the vendor and get closer to the metal. You will also need a lot of info about the pci/pcie bus and bridges, dma controllers, everything in the path. If you dont want to go that low then use linux or bsd or some other command line environment where it is well known how to take over the video system, and take over the video system while retaining an operating system and a development environment (vi/emacs, gcc).
if that is all way too advanced, then I recommend, dabbling in simple gpu routines to get a feel for how the video card works at least at some level and tackle this learning exercise one step at a time.
i want to access my system resources such as CPU without the use of OS system calls.
is there any way to make this possible?
The only way to access the hardware directly on most modern operating systems, Linux and Windows included, is via kernel code. Linux Device Drivers is an excellent starting point for writing such code on Linux, even if it is a bit dated.
Otherwise, the OS provides various I/O facilities and controls the allocation of resources to the user applications, using the system call interface. The system call interface is omnipresent in its basic concept among all operating systems that actually have some sort of separation between kernel and user code. The use of software interrupts is the standard way to implement system calls on current hardware.
You need a system call to allocate the slightest amount of memory and even to read or write a single character. Not to mention that even a program that does absolutely nothing generally needs a few system calls just to be loaded.
You could gain more direct access to the hardware if you used DOS or an exokernel design.
But why would you want to do that anyway? Modern hardware is far from trivial to work with directly.
I'm creating an application that needs to use some kernel level modules, for which I've divided the app into 2: one user-level program and one kernel level program.
After reading about device drivers and walking through some tutorials, I'm a little confused.
Can there be a device driver without any specific device associated with it? Is there anything other than the device driver (kernel code or something) which works in kernel mode?
How do anti-virus programs and other such applications work in kernel mode? Is device driver the correct way or am I missing something?
Yes, device drivers can work without an actual piece of hardware (i.e. the device) attached to the machine. Just think of the different programs that emulate a connected SCSI drive (CD-ROM, whatever) for mounting ISO images. Or think about TrueCrypt, which emulates (removable) drives using containers, which are nothing more than encrypted files on your hard drive.
A word of warning, though: Driver development requires much more thought and has to be done more carefully, no shortcuts, good testing and in general expects you to know quite a good deal about the Windows driver model. Remember that faulty and poor drivers put the whole system's stability in jeopardy.
Honestly, I don't think reading a tutorial is sufficient here. You might want to at least invest in a decent book on that subject. Just my 2 cents, though.
Sorry, but the Windows Internals book is more of a general reading for the curious. I cannot recommend it if you want to engage in driver development - or at most as prerequisite reading to understand the architecture. There are plenty of other books around, although most of them are a bit older.
Depending on your goal, you may get away with one of the simpler driver models. That is not to say that driver development is trivial - in fact I second all aspects of the warning above and would even go further - but it means that you can save some of the more tedious work, if instead of writing a legacy file system filter you'd write one based on the filter manager. However, Windows XP before SP2 did not have it installed by default and Windows 2000 would require SP4+SRP+patch if I remember correctly. WDF (Windows Driver Foundation) makes writing drivers even easier, but it is not suitable for all needs.
The term device is somewhat of bad choice here. Device has a meaning in drivers as well, and it does not necessarily refer to the hardware device (as pointed out). Roughly there is a distinction between PDOs (physical device objects) and CDOs (control device objects). The latter are usually what you get to see in user mode and what can be accessed by means of CreateFile, ReadFile, WriteFile, DeviceIoControl and friends. CDOs are usually made visible to the Win32 realm by means of symbolic links (not to be confused with the file system entities of the same name). Drive letter assignments like C: are actually symbolic links to an underlying device. It depends on the driver whether that'd be a CDO or PDO. The distinction is more of a conceptual one taught as such in classes.
And that's what I would actually recommend. Take a class about Windows driver development. Having attended two seminars from OSR myself, I can highly recommend it. Those folks know what they're talking about. Oh, and sign up to their mailing lists over at OSR Online.
Use Sysinternals' WinObj to find out more about the device and driver objects and symlinks.
As for the question about AVs, yes they use file system filter drivers (briefly mentioned above). The only alternative to a full-fledged legacy FSFD is a mini-filter.
It is possible to load a special kind of DLL in kernel mode, too. But in general a driver is the way into the kernel mode and well documented as such.
Books you may want to consider (by ISBN): Most importantly "Programming the Windows Driver Model" (0735618038), "Windows NT Device Driver Development" (1578700582), "Windows NT File System Internals" (0976717514 (OSR's new edition)), "Undocumented Windows NT" (0764545698) and "Undocumented Windows 2000 Secrets" (0201721872) - and of course "Windows NT/2000 Native API Reference" (9781578701995) (classic). Although the last three more or less give you a better insight and are not strictly needed as reading for driver developers.
Anti-virus (and system recovery) software generally make use of file-system filter drivers. A device can have multiple filter drivers arranged like a stack, and any event/operation on this device has to pass through all the stacked up drivers. For example, anti-viruses install a filter driver for disk device so that they can intercept and scan all file system (read/write) operation.
As mentioned in above post, going through a good book would be a nice way to start. Also, install DDK/WDK and refer the bundled examples.
I was going through some general stuff about operating systems and struck on a question. How will a developer debug when developing an operating system i.e. debug the OS itself? What tools are available to debug for the OS developer?
Debugging a kernel is hard, because you probably can't rely on the crashing machine to communicate what's going on. Furthermore, the codes which are wrong are probably in scary places like interrupt handlers.
There are four primary methods of debugging an operating system of which I'm aware:
Sanity checks, together with output to the screen.
Kernel panics on Linux (known as "Oops"es) are a great example of this. The Linux folks wrote a function that would print out what they could find out (including a stack trace) and then stop everything.
Even warnings are useful. Linux has guards set up for situations where you might accidentally go to sleep in an interrupt handler. The mutex_lock function, for instance, will check (in might_sleep) whether you're in an unsafe context and print a stack trace if you are.
Debuggers
Traditionally, under debugging, everything a computer does is output over a serial line to a stable test machine. With the advent of virtual machines, you can now wire one VM's execution serial line to another program on the same physical machine, which is super convenient. Naturally, however, this requires that your operating system publish what it is doing and wait for a debugger connection. KGDB (Linux) and WinDBG (Windows) are some such in-OS debuggers. VMWare supports this story explicitly.
More recently the VM developers out there have figured out how to debug a kernel without either a serial line or kernel extensions. VMWare has implemented this in their recent stuff.
The problem with debugging in an operating system is (in my mind) related to the Uncertainty principle. Interrupts (where most of your hard errors are sure to be) are asynchronous, frequent and nondeterministic. If your bug relates to the overlapping of two interrupts in a particular way, you will not expose it with a debugger; the bug probably won't even happen. That said, it might, and then a debugger might be useful.
Deterministic Replay
When you get a bug that only seems to appear in production, you wish you could record what happened and replay it, like a security camera. Thanks to a professor I knew at Illinois, you can now do this in a VMWare virtual machine. VMWare and related folks describe it all better than I can, and they provide what looks like good documentation.
Deterministic replay is brand new on the scene, so thus far I'm unaware of any particularly idiomatic uses. They say it should be particularly useful for security bugs, too.
Moving everything to User Space.
In the end, things are still more brittle in the kernel, so there's a tremendous development advantage to following the Nucleus (or Microkernel) design, where you shave the kernel-mode components to their bare minimum. For everything else, you can use the myriad of user-space dev tools out there, and you'll be much happier. FUSE, a user-space filesystem extension, is the canonical example of this.
I like this last idea, because it's like you wrote the program to be writeable. Cyclic, no?
In a bootstrap scenario (OS from scratch), you'd probably have to introduce remote debugging capabilities (memory dumping, logging, etc.) in the OS kernel early on, and use a separate machine. Or you could use a virtual machine/hypervisor.
Windows CE has a component called KITL - Kernel Independent Transport Layer. I guess the title speaks for itslf.
You can use a VM: eg. debug ring0 code with bochs/gdb
or Debugging NetBSD kernel with qemu
or a serial line with something like KDB.
printf logging
attach to process
serious unit tests
etc..
Remote debugging with kernel debuggers, which can also be done via virtualization.
Debugging an operating system is not for the faint of heart. Because the kernel is being debugged, your options would be quite limited. Copious amount of printf statements is one trick, and furthermore, it depends on really what 'operating system' is being debugged, we could be talking about
Filesystem
Drivers
Memory management
Raw Disk input/output
Screen input/output
Kernel
Again, it is a widely varying exercise as in the above, they all interact with one another. Even more complicated is the fact, supposing you were to debug the kernel, how would you do it if the runtime environment is not properly set (by that, I am talking about the kernel's responsibility for loading binary executables).
Some kernels may (not all of them have them) incorporate a simple debug monitor, in fact, if I rightly recall, in the book titled 'Developing your own 32bit Operating System' by Richard A Burgess, Sams publishing, he incorporated a debug monitor which displays various states of the CPU, registers and so on.
Again, take into account of the fact that the binary executables require a certain loading mechanism, for example a gdb equivalent, if the environment for loading binaries are not set up, then your options are quite limited.
By using copious amount of printf statements to display errors, logs etc to a separate terminal or to a file is the best line of debugging, it does sound a nightmare but it would be worth the effort to do so.
Hope this helps,
Best regards,
Tom.