I was working as a QA engineer for a proprietary embedded operating system. They built their own ATN stack and stepping though it with a debugger was the most eye opening experience I have had with networking. Watching each layer of the stack build their part of the packet was amazing. Then finally being able to see the built packet on the wire had more meaning.
As an educator I would like share this experience with others. Does anyone know of a straight forward method stepping though a TCP/IP stack? Ideally I would like something easier than debugging a *BSD or Linux kernel, although if this is the only option then some tips and tricks for this process would be nice. A reference stack written in C/C++ that could be run in user mode with Visual Studio or Eclipse would be ideal.
This all depends on what you want to focus on. From your question, the thing you are most interested in is the data flow throughout the different layers (user-space stream -> voltage on the cable).
For this, I propose you use http://www.csse.uwa.edu.au/cnet/, which is a full network simulator. It allows you to step through all levels of the stack.
Real systems will always have a clear distinction between Layer3, Layer2 and Layer1 (Ethernet and CRC-checking firmware on chip, hardware MAC). You will have trouble getting into the OS and some implementation details will be messy and confusing for students. For Linux, you'll have to explain kernel infrastructure to make sense of the TCP/IP stack design.
If you are only interested in the TCP/IP part, I recommend you use an embedded TCP/IP stack like http://www.sics.se/~adam/lwip/ . You can incorporate this into a simple user-space program and fully construct the TCP/IP packet.
Please note that there are a lot of network communication aspects that you cannot address while stepping through the TCP/IP stack. There is still a MAC chip in between which regulates medium access, collisions etc. Below that, there is a PHY chip which translates everything into electric/optical signals, and there is even a protocol which handles communication between MAC and PHY. Also, you are not seeing all aspects related to queueing, concurrency, OS resource allocation ea. A full picture should include all of these aspects, which can only be seen in a network simulator.
I would run Minix in a virtual machine and debug that. It is perfect for this.
Minix is a full OS with TCP/IP stack so you have the code you need. However, unlike Linux/BSD its roots and design goal are to be a teaching tool, so it eschews a certain level of complexity in favor of being clear. In fact, this is the OS Linus Torvalds started hacking on when he started out with Linux :-)
You can run minix in an VM such as VirtualBox or VMware and debug it. There are instruction on the web site: http://www.minix3.org/
I personally learned TCP/IP stack using DOS and SoftICE (oops, leaked that I'm an old guy). Using DOS on a virtual machine and debug through a TCP/IP driver will be much simpler since your goal is to educate how TCP/IP works. Modern OS does a lot of optimization on network I/O and it's not easy to debug through.
http://www.crynwr.com/ has a bunch of open source packet drivers. Debugging with source code shall be a bit easier.
This not exactly what you are looking for but I hope this helps
1995 - TCP/IP Illustrated, Volume 2: The Implementation (with Gary R. Wright) - ISBN 0-201-63354-X
Just walk through the code side by side. Near stepping through experience. Mr Steven's explains key variables too. Just awesome. Note: Code may have changed since the book but still awesome.
Probably lwIP project is what you are looking for because it can be run without an operating system.
As for debugging Linux kernel, there is not very simple, but well-known way to do it. Use KGDB. Install debugging version of Linux kernel on virtual machine or on separate box. And remotely connect GDB to this machine. Probably you would like to use some GDB frontend instead of text-only interface. If you need more details on kernel debugging from more competent people, just add "linux" tag to the question.
I actually wrote a small subset of a TCP/IP stack in a 8051 once, it was a very enlightening experience.
I believe that the best way to learn something is by doing it. Once you finished your task, go and get feedback with other developers and compare your implementation with other existing ones.
My opinion might be biased here, but I think that doing this in a embedded platform is the best way to go. What you are trying to do is very low level, and a PC will just add more complexity into the problem. A embedded chip has no operational system to get in your way. Besides that, it is very satisfying to see a simple 8051 respond to ping requests and telnet calls.
They key is to start small, don't try to create a full TCP/IP stack all at once. Write the code to handle the MAC first, then IP, Ping, UDP and finally TCP.
I don't think that studying an existing implementation is a good ideia. TCP/IP implementations tend to be bloated with code that is unrelated with your goal.
I work in the TCP/IP industry. In BSD and variants, the function tcp_input() is an ideal starting point to explore the innards of TCP. Setting a breakpoint on this function and stepping through it on a live system can give a lot of enlightenment. If that is hard, you can simply browse through the source to get a broad feel of it:
http://fxr.watson.org/fxr/source/netinet/tcp_input.c
It will take time, many weeks at least, to understand the big picture. Quite exhilarating. :-)
You can run the NetBSD IP stack in userspace in Linux or other OS, with gdb or whatever see http://www.netbsd.org/docs/rump/ and https://github.com/anttikantee/buildrump.sh and then eg feed it to a tun/tap device so you can see whats on the wire.
Related
My boss has given a code for testing PCI express on an Altera board. The code consist of several c code files having instructions such as reading Bios, setting some registers, writing to buffers etc.
My job at present is to see the functionality of the code by running it.
I am new to FPGA and I am unable to understand what tools, compilers etc will I use for compiling it for the FPGA.
Since it is a C code so I am sure I cannot use the same environment as that of Verilog/VHDL. Can I get some hints as to what compilers are available for compiling C code for testing various interfaces of an FPGA?
Thanks and regards
H
If the FPGA board is just connected through a standard PCIexpress interface, it isn't that hard to create a Linux driver to simply access a couple of registers. This might even be easier to do than getting some old DOS-based drivers to work.
I did some work on this in a pre-project to my masters thesis some years ago - if you're interested, it's available here: http://loejer.dk/files/FORK,%20pdf.zip
It sounds like the intent is that you connect the PCI-E card to a normal computer with a PCI-E slot, then run the test software on the host computer, so it will talk to the board via PCI-E, and exercise the board from the host, collect data on the host, and so on.
Such code will almost certainly be quite non-portable. You'll probably need to ask what system it's for (or examine the code to find hints -- e.g., if it starts with #include <windows.h> that's a pretty fair indication that it's for Windows).
The OS it's written for will give at least an 80% (or so) clue about what compiler to use -- if it's for Windows, chances are pretty good that it's intended for Microsoft's compiler. If it's for Linux, there's an even better chance that it's for gcc/g++. If it's for MacOS, it's probably for g++, but if it's really new, might target Clang.
I'm working with various (mostly Bluetooth) development boards (ConnectBlue, Ubertooth, USRPs etc.) in order to research about Bluetooth communication behaviour at PHY level. In order to get some more insights I'm looking for a way to debug the Bluetooth stack on a Windows 7 Desktop computer. My use-case is relatively simple: I have custom baseband implementations, which establish connections with the Windows computer. I'd like to see everything the Bluetooth hardware/driver does.
I'm not sure how to approach this: I'd like to see when the Bluetooth Chip/Windows driver receives a Signal, and how it (the message) gets interpreted/formatted/passed through the various APIs concerned. Mostly this relates to kernel debugging.
Is there a way to display the state of the attached hardware in Windows in WinDBG? Maybe to perform (Kernel) API logging on the Bluetooth kernel service?
I hope somebody more familiar with device driver debugging and Windows Kernel services can give me some pointers here.
Since you don't appear to have gotten any hits on this, I'll post what I can.
I don't have any definite answers, but on the NTDebugging blog they often do hardware level debugging in windbg.
I.e.
http://blogs.msdn.com/b/ntdebugging/archive/2007/06/22/where-the-rubber-meets-the-road-or-in-this-case-the-hardware-meets-the-probe.aspx
To be honest this is going to require extensive knowledge not only of your hardware, but also of the deep internals of windows, and how the bluetooth stack is written, but the WDK would probably be a good place to start for understanding the bluetooth stack. I would also check out the blog for tips and tricks.
The other place to check and ask questions is http://osronline.com/ It's one of the better communities about device drivers, so they should have some reasonable tips on doing what you're trying to do.
I want to write something like DaemonTools: a software that presents itself to the system as a real device (a DVD-ROM in the previous example) but it reads the data from a file instead. My requirement is not limited to DVD-ROM. The first goal is a joystick/gamepad for Windows.
I'm a web developer, so I don't know from where I could start such a project. I believe it will have to be written in C/C++, but other than that, I have no clue where to start.
Did anyone tried something like this and can give me some starting tips ?
Most drivers are written in either C or C++, so if you don't know those languages reasonably well, you'll want to get familiar with them before you start. Windows programming uses a lot of interesting shortcuts that might be confusing to a beginner - for example PVOIDs (typedef void* PVOID) and LPVOIDs (typedef void* far LPVOD;). You'll need to be happy with pointers as concepts as well as structures because you'll be using a lot of them. I'd suggest writing a really straightforward win32 app as an exercise in getting to grips with the Windows style of doing C/C++.
Your next port of call then is to navigate the Windows Driver Kit - specifically, you'll need it to build drivers for Windows. At this stage my ability to advise really depends on what you're doing and the hardware you have available etc, or whether or not you're really using hardware. You'll need to know how to drive your hardware and from there you'll need to choose an appropriate way of writing a driver - there are several different types of driver depending on what you need to achieve and it might be you can plug into one of these.
The windows driver kit contains quite a large number of samples, including a driver that implements a virtual toaster. These should provide you with starting points.
I strongly suggest you do the testing of this in a virtual machine. If your driver successfully builds, but causes a runtime error, the result could well crash windows entirely if you're in kernel-mode. You will therefore save yourself some pain by being able to revert the virtual machine if you damage it, as well as not having to wait on your system restarting. It'll also make debugging easier as virtual serial cables can be used.
This is quite a big undertaking, so before you start, I'd research Windows development more thoroughly - check you can't do it using the Windows APIs first, then have a look at the user-mode driver framework, then finally and only if you need to, look at the kernel level stuff.
I am into new project, which should use microcontroller. The easiest way to program it is using parallel port. But, there are few things I hope you can help me with. Oh, and the preferred language is C and platform Windows.
So, I studied LPT ports and Windows a bit, and from what I learned the most important is: Since Windows NT based systems, you cannot use instructions for direct port manipulation. This should be, because now programs are run in different privilege mode, which doesn't support the kind of instructions that are used by outport() function.
But at this point, I don´t understand a few things. First, I thought that Windows actually used privilege levels since first protected mode version, but that's the wrong assumption.
But more importantly, I thought that Windows has included functions for just about any hardware communication. I mean, anything you do in Windows these days, you just call windows functions which further call kernel services. I assumed that outport() doesn´t use any Windows function, and just makes the communication itself, which is prohibited now. But I am literally shocked that there is no system function to control parallel ports in modern Windows systems. At least that's what I read.
But even if I could get the control of parallel port, there comes my second problem.
For programming the controller, I need to follow special protocol, especially timing. But since Windows is multitasked, I worry about what if Scheduler switches to another app, and therefore when is the right time to switch signals on LPT, my program just will not be able to run.
Oh, by the way, I know I could use any 3rd-party apps, but I just like to be able to do it myself, or at least before I use some 3rd-party app, I want to know how it works. And yes, you can program some microcontrollers just by parallel port with some resistors, I know this for sure.
Thanks.
For windows you need to install a DLL which contains a driver to run at elevated privileges to get access to the HW ports.
You can find such a library at :
http://logix4u.net/Legacy_Ports/Parallel_Port/Inpout32.dll_for_Windows_98/2000/NT/XP.html
There are also some links to sample code.
I do not know which uController you are using, but I programmed in the past a variety of them and never had issues with timings, well for programming at least. The programming protocols are usually robust enough to deal with the jitter caused by multitasking. Just keep your clock edges and signa edges well separated and it should go fine.
I was going through some general stuff about operating systems and struck on a question. How will a developer debug when developing an operating system i.e. debug the OS itself? What tools are available to debug for the OS developer?
Debugging a kernel is hard, because you probably can't rely on the crashing machine to communicate what's going on. Furthermore, the codes which are wrong are probably in scary places like interrupt handlers.
There are four primary methods of debugging an operating system of which I'm aware:
Sanity checks, together with output to the screen.
Kernel panics on Linux (known as "Oops"es) are a great example of this. The Linux folks wrote a function that would print out what they could find out (including a stack trace) and then stop everything.
Even warnings are useful. Linux has guards set up for situations where you might accidentally go to sleep in an interrupt handler. The mutex_lock function, for instance, will check (in might_sleep) whether you're in an unsafe context and print a stack trace if you are.
Debuggers
Traditionally, under debugging, everything a computer does is output over a serial line to a stable test machine. With the advent of virtual machines, you can now wire one VM's execution serial line to another program on the same physical machine, which is super convenient. Naturally, however, this requires that your operating system publish what it is doing and wait for a debugger connection. KGDB (Linux) and WinDBG (Windows) are some such in-OS debuggers. VMWare supports this story explicitly.
More recently the VM developers out there have figured out how to debug a kernel without either a serial line or kernel extensions. VMWare has implemented this in their recent stuff.
The problem with debugging in an operating system is (in my mind) related to the Uncertainty principle. Interrupts (where most of your hard errors are sure to be) are asynchronous, frequent and nondeterministic. If your bug relates to the overlapping of two interrupts in a particular way, you will not expose it with a debugger; the bug probably won't even happen. That said, it might, and then a debugger might be useful.
Deterministic Replay
When you get a bug that only seems to appear in production, you wish you could record what happened and replay it, like a security camera. Thanks to a professor I knew at Illinois, you can now do this in a VMWare virtual machine. VMWare and related folks describe it all better than I can, and they provide what looks like good documentation.
Deterministic replay is brand new on the scene, so thus far I'm unaware of any particularly idiomatic uses. They say it should be particularly useful for security bugs, too.
Moving everything to User Space.
In the end, things are still more brittle in the kernel, so there's a tremendous development advantage to following the Nucleus (or Microkernel) design, where you shave the kernel-mode components to their bare minimum. For everything else, you can use the myriad of user-space dev tools out there, and you'll be much happier. FUSE, a user-space filesystem extension, is the canonical example of this.
I like this last idea, because it's like you wrote the program to be writeable. Cyclic, no?
In a bootstrap scenario (OS from scratch), you'd probably have to introduce remote debugging capabilities (memory dumping, logging, etc.) in the OS kernel early on, and use a separate machine. Or you could use a virtual machine/hypervisor.
Windows CE has a component called KITL - Kernel Independent Transport Layer. I guess the title speaks for itslf.
You can use a VM: eg. debug ring0 code with bochs/gdb
or Debugging NetBSD kernel with qemu
or a serial line with something like KDB.
printf logging
attach to process
serious unit tests
etc..
Remote debugging with kernel debuggers, which can also be done via virtualization.
Debugging an operating system is not for the faint of heart. Because the kernel is being debugged, your options would be quite limited. Copious amount of printf statements is one trick, and furthermore, it depends on really what 'operating system' is being debugged, we could be talking about
Filesystem
Drivers
Memory management
Raw Disk input/output
Screen input/output
Kernel
Again, it is a widely varying exercise as in the above, they all interact with one another. Even more complicated is the fact, supposing you were to debug the kernel, how would you do it if the runtime environment is not properly set (by that, I am talking about the kernel's responsibility for loading binary executables).
Some kernels may (not all of them have them) incorporate a simple debug monitor, in fact, if I rightly recall, in the book titled 'Developing your own 32bit Operating System' by Richard A Burgess, Sams publishing, he incorporated a debug monitor which displays various states of the CPU, registers and so on.
Again, take into account of the fact that the binary executables require a certain loading mechanism, for example a gdb equivalent, if the environment for loading binaries are not set up, then your options are quite limited.
By using copious amount of printf statements to display errors, logs etc to a separate terminal or to a file is the best line of debugging, it does sound a nightmare but it would be worth the effort to do so.
Hope this helps,
Best regards,
Tom.