I have a device driver that is freezing the OS. The mouse wont even move. I am trying to debug this issue and I believe one good approach is to use gdb with qemu, two things I have never used before. Is there a better approach?
So first I need to compile the kernel with debug symbols which I have done already.
Now, there is a new file that is generated called vmlinux that is located in that same folder as the source. It seems that I also need a bzImage file according to this so I can run the newly compiled kernel using:
qemu-system-i386 -kernel bzImage
or in debug mode
qemu-system-i386 -s -S -kernel bzImage
I cannot locate the bzImage file. Where do I find it or what is missing here? Is the bzImage referring to the OS Image I created using qemu-img create?
Also, what I do not understand is that now the kernel is compiled (vmlinux) how does I run it with qemu? So my question is when I run it with qemu or the debugger is the kernel running as an app in my main OS?
also how can I install my device driver? My understanding the kernel is not Ubuntu so there is no UI?
Also, I installed qemu and when I type qemu I get command not found. I am guessing I have to pick a specific processor emulator as in qemu-system-i386, qemu-system-x86_64, or qemu-x86_64?
How is qemu different or similar to the kvm command?
Thanks.
So, if I understand the problem correctly, you have a kernel module that needs no specific hardware. When you are working with the module, the system freezes but the kernel log contains nothing special.
The following may be helpful.
Getting the log
The symptoms you described may still be a result of a kernel oops or panic. The logging facilities sometimes die before they can output the information about the error to the log file. You may try to output the log via a serial port, this should be more reliable.
As your kernel module does not need any specific hardware, the easiest way is probably to install the same Linux distro as you use to a virtual machine and connect the virtual serial port (COM) of that machine to a pipe on your host system.
This is usually quite easy to do. For example, this blog post contains the detailed instructions in case the host OS and the guest OS are Ubuntu 11.10.
VirtualBox is used there to manage the virtual machines. If you prefer QEMU, this should be possible as well. I suppose it is a bit easier to go with VirtualBox though but it is a matter of personal preference.
Basically, you need to perform the following steps.
Create a virtual machine and install the Linux distro you need as a guest OS there.
Enable a serial port (COM1, ...) in the configuration of the virtual machine and configure it to connect to a special file on the host ("host pipe"), say /tmp/vbox_serial.
Start the guest OS and adjust its boot options: at least, add console=ttyS0,115200 or something like that to the kernel options in the boot loader menu.
On the host, start minicom, socat or whatever else to read from /tmp/vbox_serial.
That is it. Now you should get the kernel log of the guest OS pouring to your host system via /tmp/vbox_serial. If the guest system crashes then, you will get the log even if it is not saved into a file on the guest itself.
To make things easier, you may use socat on your host system rather than minicom that the author of that blog post suggests. The power of minicom is probably not needed here.
This way, you can use socat and tee to save the log to guest.log file while still outputting it to the console:
socat /tmp/vbox_serial - | tee guest.log
If there was a kernel oops or panic, the backtrace in the log usually helps to find out what
has gone wrong.
Detecting Deadlocks
If you have obtained the full log via a serial connection or some other means and still there is nothing suspicious there and you suspect there has been a deadlock in the kernel,
lockdep tool may help. It is included into the kernel (but you may need to rebuild the kernel with CONFIG_LOCKDEP_SUPPORT=y).
Lockdep detects the potential deadlocks and outputs the results to the kernel log. This presentation may help you analyse its output.
Tracing Facilities
If you need tracing of some events in the kernel to debug your system, there are some tools that could be handy.
Kprobes - a kind of breakpoints you can set in almost arbitrary place in the kernel. Can be used to trace function calls among other things, with a moderate performance impact.
SystemTap - a powerful system to analyze what is going on in the kernel. Part of it is based on Kprobes.
Ftrace - a tracing system included into the kernel, incurs less overhead than Kprobes if that matters.
Related
I have a custom i.MX6Q-based board with working U-Boot and Linux (Ubuntu) setups. The micro and board have support for USB-OTG and one serial port; currently, the serial port serves the console for both U-Boot and Linux. However, we may need to use the serial port for another purpose, but we don't want to lose the console for U-Boot and Linux. Is it possible to use the USB-OTG port for the system console for both U-Boot and Linux?
I've done some research and found a couple of promising articles here and here, though the second article says this tidbit:
Unfortunately it won't work as system console as the gadget driver is loaded as a module, but we can use it for serial console.
I'm not sure I understand this, but it sounds like the method won't meet my needs, which is to use USB-OTG for both U-Boot and Linux system consoles. I did try these methods, but without luck, which may mean that U-Boot and Linux aren't built properly for the desired functionality.
So here are my questions:
Can this work for U-Boot?
Can this work for Linux?
Am I insane for contemplating this path?
For either, any guidance (e.g. tutorials, examples, etc.) would be greatly appreciated.
Thanks!
Can this work for U-Boot?
Yes, at least since U-Boot version 2008.10, the README file has stated:
Define the below if you wish to use the USB console.
CONFIG_USB_DEVICE
Define this to build a UDC device
CONFIG_USB_TTY
Define this to have a tty type of device available to
talk to the UDC device
CFG_CONSOLE_IS_IN_ENV
Define this if you want stdin, stdout &/or stderr to
be set to usbtty.
Note that these configuration symbols are not accessible using the menuconfig, and must be enabled in a configuration file.
Currently at least five boards use this U-Boot capability, based on the occurrence of CONFIG_USB_TTY in files in include/configs/, for example include/configs/ti_omap4_common.h.
This USB configuration requires non-default definitions for the stdin and stdout environment variables. Refer to the README documentation for the details.
Can this work for Linux?
Yes, Linux (at least since version 4.5) can have a serial console on a USB connection, either a USB-to-serial adapter on a host port or a USB serial gadget on a device port (using CDC/ACM).
For instance, in drivers/usb/gadget/Kconfig there's the selection:
config U_SERIAL_CONSOLE
bool "Serial gadget console support"
depends on USB_G_SERIAL
help
It supports the serial gadget can be used as a console.
In the Linux 5.7.8 kernel only two boards have default configurations that use this capability, for example see arch/arm/configs/aspeed_g4_defconfig.
Besides a proper configuration to build the necessary drivers, a serial-gadget console requires (1) the kernel parameter specification (e.g. console=ttyGS0,...), and (2) a login session initiated by a getty command (e.g. in the inittab file).
Am I insane for contemplating this path?
No comment.
Beware that should you encounter a kernel boot issue, the Linux serial-gadget console does not support earlycon nor earlyprintk capability.
Personally I prefer to use a serial link that is persistent regardless of the target board's state. That ensures the terminal emulator program does not complain about lost connections.
Addendum
Unfortunately this Linux console on a USB serial gadget does not display boot messages generated by the kernel (before the login prompt), even if all drivers are statically linked in to the kernel image.
Although the syslog has messages like
console [ttyGS0] enabled
g_serial gadget: g_serial ready
...
gs_open: ttyGS0 ((ptrval),(ptrval))
before the salient Freeing unused kernel memory message, the host side does not receive any console messages until userspace is active.
This shortcoming is also reported in this guide: https://linux-sunxi.org/USB_Gadget/Serial
I am trying to read the LDD book by Jonathan Corbet, Greg Kroah-Hartman, Alessandro Rubini and implement the sample modules. So to begin with, I tried setting up a development system. Installed Ubuntu 16.04 Xenial. Now, I just created a directory and wrote the hello_world module with a Makefile. Got it built and run it, verified the dmesg logs.
Is that all the development setup? I searched online and found articles where they are asking to download and compile the kernel, use a VM to boot the kernel. What is the reason? Or what am I missing?
Is there any better article which clarifies this?
Thanks
hago
You can try one more way:
If you have native windows, install virtual machine software such as
Virtual box. Get your favourite Linux distribution (no bias, just
an example - Ubuntu) and install it through Virtual box.
Get the latest kernel (or of your choice) from kernel.org.
Choose the platform you want to build this kernel for. E.g arm64 or x86.
In case you do not have real boards (e.g RPi for arm variant), you can use qemu-arm64 or qemu-x86 to run your compiled kernel. This is also a good option when users do not have the boards.
Another good use case for using qemu for the newbie kernel developers is even they write some modules which crashes, then the qemu instance is crashed so no harm.
I think using qemu is a good option for people who starts to learn kernel programming and also want to try writing some of their modules and do not intend to purchase hardware at this point of time.
It depends on your target. For your case, you have made a kernel driver for your computer (it run Linux kernel).
But if you want to develop a Kernel driver for another target like Rasberry Pi, ARM board, X86-X64 board, ... you must learn to compile, edit Kernel config, boot Kernel image, ... because each target has different kernel versions.
You can refer to this training for more detail: https://bootlin.com/training/embedded-linux/
I know I can get Linux booting log using dmesg command but don't know how can get booting log of U-boot when my system just started.I Tried Go-ogling also but all I got is dmesg reference everywhere.
Also I am not sure what is CRM in U-boot?
I am on ARM based target board.
The console output from U-Boot is not saved anywhere (by default).
Hence, there is no "log" is display.
You have to capture U-Boot's console messages as they are output (e.g. using a terminal emulation program's logging feature), or they are gone forever.
Also I am not sure what is CRTM in U-boot?
"CRTM" is not standard terminology for U-Boot AFAIK.
It probably refers to a secure bootstrap program that loads U-Boot.
I've to test some low level code on an ARM architecture. Typically experimentation is quite complicated on the real board, so I was thinking about QEMU.
What I'd like to get is some kind of debugging information like printfs or gdb. I know that this is simple with linux since it implements both the device driver for the QEMU Integrator and the gdb feature, but I'm not working with Linux. Also I suspect that extracting this kind of functionality from the Linux kernel source code would be complicated.
I'm searching from some simple operating system that already implements one of those features. Do you have some advice?
You don't need a target OS to debug code that's running inside QEMU -- QEMU already does that for you.
Specifically, QEMU supports remote debugging from GDB -- you can run QEMU with the appropriate command-line options and it will export an interface that a copy of GDB (running on the host machine) can connect to. At that point, you can debug the program in GDB pretty much just as if you were running it on the host machine.
http://wiki.osdev.org/GDB appears to have a bit more basic information; possibly not enough to completely get you started, but at least give you the basic idea and some terms to look for in the QEMU and GDB documentation. Skip over the bit about "Implementing GDB Stubs", which doesn't apply here since QEMU has one already, and start at the section on "Using Emulator Stubs". The short form is simply that you start QEMU with the -s option (export a GDB connection on localhost:1234) and the -S option (wait for a GDB "continue" command before starting execution), and then in GDB on your host you say target remote :1234 instead of run. Also, of course, you need to be using an ARM version of GDB rather than a native-x86 one.
(In addition, if you're willing to pay for a commercial solution, CodeSourcery's ARM toolchain has the IDE integration to set all of this up automatically, including support for "printf" to print into the debugger console. That works on a physical board, too, if you've got a hardware debugger. Usual disclaimer about me being a CodeSourcery employee applies -- but I do find it very easy to use.)
Update, 2012: CodeSourcery's toolchain is now called Mentor Graphics Sourcery CodeBench, but all the above still applies.
I realise that I am addressing your original problem here rather than your proposed solution (perhaps that's better?), but to use GDB (or Insight/GDB) directly on the target, use a low-cost JTAG tool and OpenOCD. An example of such a set-up and how to implement it can be found here.
If you have a larger budget, a more fully featured JTAG debugger may be useful, such as the Abatron BDI3000 with bdiGDB firmware which allows remote debugging and device programming over Ethernet with GDB and no special drivers or target debug agent.
Maybe a microkernel like OKL4 would suit your needs?
I'm currently setting up vmware Server 2.0 for kernel debugging with gdb ( see this setup guide ) and someone asked me why not use kvm?
So I ask: kvm vs. vmware for kernel debugging / USB driver development
what are the pros and cons of each?
Driver development? are you working on a driver for a particular piece of hardware? if so, then you probably won't be able to use virtualization, because the virtualized instance won't have access to the new hardware.
For this you will need two machines, one running a remote debugger on the other.
*Edit: * Apparently you're developing a driver for a USB Device? this is one area in particular that a VM actually Can help. These days most VM's have the ability to delegate specific USB devices to a guest OS.
That said, this situation doesn't really offer any benefits over the remote debugger option, because you still need a way to inspect the state of the running or crashed OS, and VM's offer very little assistance in this regard. You might be able to replay saved states from just before a crash.
You might be able to get a bit of traction using UML, which would allow you to do local debugging as on a regular user process, which is a little bit less trouble.
Instead of answering the direct question I'll add another option... Depending on if the kernel in question is a Linux kernel, and what part(s) of it you are working on, you might find that UserModeLinux (included in the 2.6.x source, and available as patch sets for 2.4 and 2.2) may trump both of those options.
As it runs the kernel as a userland process under the host kernel it is easier to attach common debugging tools to. I believe it is very commonly used in the early stages of updates/additions to file-system related code. If you are developing/debugging modules that interact directly with hardware it may be much less use to you though.
Reference links: home,
other
I recently started building GNU Mach/HURD and found the combination of QEmu/KVM to work really quite well.. for the following reasons:
QEmu presents quite a clean environment
Networking has alot of options
I can easily mount the filesystem using a raw device file / loopback
Bottom line is, for kernel work I just want the minimum of functionality to boot and see the result. VMWare is much more for usable virtualization rather than down-and-dirty.
There is however no comparison to booting on a real machine with real hardware. The VM environment can seem like a safety blanket somtimes ... because even my toaster would know what a Realtek RTL8139C was.
If it is a "real hardware" device, of course, vmware will not emulate it, so you won't be able to debug the driver under it (nor will any other virtualisation software, unless you extend one to do so).
Device driver debugging can be done to some extent with a real hardware machine with a normal kernel - although there are obviously things you can't do - like set breakpoints.
It is still possible to attach a debugger to the kernel and inspect stuff. Moreover, traditional printf() debugging is quite possible (printk, anyone), and there are various features in the kernel which make debugging easier. It's possible to build the kernel with various debug options to try to detect pointer problems, memory leaks etc.
By default, the kernel even gives a nice-ish stack trace on the log when it encounters an OOPS or BUG condition (obviously this does not necessarily get written anywhere if the system hangs or crashes). Of course a pointer-out-of-range condition happening inside an interrupt is a recipe for disaster, but you could still get a stack trace on the screen immediately before the panic :)