I have tried to run qemu-arm with a complied linux kernel (Version 4.9)
and with an initfs that i have created with a sample program.
This is was based on an excellent post from here.
This is the command that i have executed:
qemu-system-arm -M vexpress-a9 -kernel linux-4.9/arch/arm/boot/zImage -initrd initramfs -append "console=tty1"
then, qemu shows me these errors and its graphical window is getting stuck:
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
Even when I run it without the -initrd parameter, for just loading the kernel - nothing happens.
When I tried run it with a vmlinuz-3.2.0-4-vexpress image in this example, it worked for me.
Does someone have clue what may be the problem? Something with the fact that it is a zImage? Is there a way to debug it?
Thanks!
"QEMU sits there and prints nothing" is quite a common symptom, and it almost always means "the guest kernel crashed before being able to print anything, because it wasn't configured correctly". This is pretty much the same effect you get if you try to boot a wrongly configured kernel on real hardware, and the process for debugging it is about the same:
check the obvious kernel config options are set correctly: in particular, that you have built it to support the ARM board and CPU that you're trying to run it on, and that you've enabled support for whatever devices you're trying to use for console output
give yourself the maximum chance of being able to see something, by configuring QEMU to output serial port information, and configuring the guest to send its console output to serial, and enabling any earlycon/earlyprintk options you can (serial output happens much earlier than graphics output, and the Linux kernel earlycon/earlyprintk options mean the kernel will start printing output earlier than it defaults to)
if you have a kernel that works, and one that doesn't, look at the differences between the kernel configs to see if one is missing something
if all else fails, you have to break out the debugger to find out what's going on
Nothing about this is particularly QEMU specific -- it's the same sort of pain you have to go through if you're trying to do kernel bringup on hardware.
PS: my first guess is that the kernel is crashing because it doesn't have enough memory -- you haven't passed QEMU a '-m' option, so it is defaulting to 128MB; the vexpress-a9 board can handle up to 1GB. earlycon would probably be sufficient debug output to identify this issue. You also aren't passing a device tree blob via -dtb, which may be an issue for newer kernels (older kernels would happily boot without one).
Related
I am trying to write a simple OS, I already wrote a bootloader but now I want to debug it, so I switched from using VirtualBox to QEMU because I saw it had better debugging.
The problem is that after I added the -s parameter to QEMU command and successfully connected via GDB, it says that the symbol table isn't loaded and that I should use the "file" command.
The only difference from what I did to what I saw people on the Internet do, is that they started GDB with gdb vmlinux, but I can't do that because I am not debugging a Linux kernel... so I figured that the issue is that I didn't start GDB with an executable, but using the "file" command on my OS image, and the compiled and linked .out file, tells me it's a "DOS/MBR boot sector", so I can't start GDB with either of them (I tried to do that anyways, but GDB failed).
Help would be appreciated.
EDIT: also, I did assemble the bootloader with the -g and --gstabs+ options.
gdb would like a file so that it can give you symbolic debugging information. For that you will need to give it a file in a format with debug info which corresponds to where your OS ends up in RAM. The "DOS/MBR boot sector" file is a disk image (the BIOS will load part of this into RAM for you, and it will then presumably finish loading code itself).
But gdb will also entirely happily let you do assembly-level debugging; you can just ignore the warning about not having a symbol table, and use the single step instruction, disassemble-from-pc and similar commands:
"disas $pc,+32" disassembles 32 bytes from the current PC
the display command prints after execution stops, so "disp /3i $pc" will print the next 3 instructions every time gdb gets control
"stepi" and "nexti" do single-instruction step/next ("step" and "next" are source-line stepping and require debug info)
I've just compiled BPF examples from kernel tools/testing/selftests/bpf and tried to load as explained in http://cilium.readthedocs.io/en/v0.10/bpf/:
% tc filter add dev enp0s1 ingress bpf \
object-file ./net-next.git/tools/testing/selftests/bpf/sockmap_parse_prog.o \
section sk_skb1 verbose
Program section 'sk_skb1' not found in ELF file!
Error fetching program/map!
This happens on Ubuntu 16.04.3 LTS with kernel 4.4.0-98, llvm and clang of version 3.8 installed from packages, iproute2 is the latest from github.
I suspect I'm running into some toolchain/kernel version/features mismatch.
What am I doing wrong?
I do not know why tc complains. On my setup, with a similar command, the program loads. Still, here are some hints:
I think the problem might come, as you suggest, from some incompatibility between kernel headers version and iproute2, and that some relocation fails to occur, although on a quick investigation I did not find exactly why it refuses to load the section. On my side I'm using clang-3.8, latest iproute2, but also the latest kernel (some commit close to 4.14).
If you manage to load the section somehow, I believe you would still encounter problems when trying to attach the program in the kernel. The feature called “direct packet access” is only present on kernels 4.7 and higher. This is what makes you able to use skb->data and skb->data_end in your programs.
Then as a side note, this program sockmap_parse_prog.c is not meant to be used with tc. It is supposed to be attached directly to a socket (search for SOCKMAP_PARSE_PROG in file test_maps.c in the same directory to see how it is loaded there). Technically this does not prevent one to attach the program as a tc filter, but it will probably not work as expected. In particular, the value returned from the program will probably not have a meaning that tc classifier hook will understand.
So I would advise to try with a recent kernel, and to see if you have more success. Alternatively, try compiling and running the examples that you can find in your own kernel sources. Good luck!
There are lots of pages that explain it but I can't find it. Many of the articles I find only work on El Capitan and older systems.
I cannot use the fwkpfv right now as I don't have the right dongles. My client is getting me a used MacBook that will support firewire.
My kernel extension panics my box. Quite oddly if my coworker builds my extension, it works just fine. I remain flummoxed.
You can get "live" local kernel logs using the command
log stream --process 0
For looking at past logs, use log show instead, e.g.:
log show --predicate 'processID == 0' --last 1h | less
None of that will help you much with kernel panics, however, as the logging happens asynchronously in user space, so you won't get the very last messages before the panic.
A few more options for debugging KPs without firewire, which you're probably already aware of but I'll mention them for completeness' sake:
Ethernet-based kernel debugging (as opposed to firewire). Only the test device needs wired/thunderbolt ethernet, the Mac running the debugger can be on wifi.
You can often extract quite a lot of info from the panic log itself: in addition to symbolicating the stack (use keepsyms=1 boot-arg so you don't have to do it retroactively), looking at the register contents and disassembly can often tell you the values of variables.
If you're missing parts of Apple's code the stack trace, run a debug or development kernel instead of the release one. Those are built with fewer optimisations enabled, so functions are less likely to be inlined, etc.
There are a bunch of memory debugging and other diagnostic options you can turn on in the kernel, e.g. -zp, -zc and so on.
If you can repro the crash in a VM (VMWare Fusion, Parallels, VirtualBox, KVM/Qemu, whatever), you can use the VM's simulated serial port to log kprintf output. The virtual ethernet ports also tend to support kernel debugging if you set them up right.
I am trying to remotely debug a linux kernel running on an arm cortex-a9 target using jtag probe and gdb.
I can connect to the kernel and halt it with gdb. I am able to set breakpoints in the kernel code and gdb confirms there placement as well but the problem is that once I start the execution and issue a continue command, the breakpoints are never hit and the kernel continues to run....
Please help me out in this regard.
Thanks.
As pointed in this thread, you should set the breakpoints as hardware breakpoints, namely - using the hbreak command. Only then the breakpoints will be hit.
For anyone reading this, the debugger will not break with software breakpoints by default, see the relevant doc:
"If the architecture that you are using supports the kernel option CONFIG_STRICT_KERNEL_RWX, you should consider turning it off. This option will prevent the use of software breakpoints because it marks certain regions of the kernel’s memory space as read-only. If kgdb supports it for the architecture you are using, you can use hardware breakpoints if you desire to run with the CONFIG_STRICT_KERNEL_RWX option turned on, else you need to turn off this option."
at https://www.kernel.org/doc/html/v4.14/dev-tools/kgdb.html
Disable RWX and recompile, then software breakpoints should work (they started working normally here after this)
Is some cases, KASLR (Kernel Address Space Layout Randomization) can be the culprit.
Even though you setup hbreak, the actual code location can be different from the address seen from the .elf file when using KASLR, so either pass --append "nokaslr" to the kernel boot argument, or configure the kernel with RANDOMIZE_BASE=n. This applies for arm64 and x86_64. (maybe other architectures too).
I am programming a pci device with verilog and also writing its driver,
I have probably inserted some bug in the hardware design and when i load the driver with insmod the kernel just gets stuck and doesnt respond. Now Im trying to figure out what's the last driver code line that makes my computer stuck. I have inserted printk in all relevant functions like probe and init but non of them get printed.
What other code is running when i use insmod before it gets to my init function? (I guess the kernel gets stuck over there)
printks are often not useful debugging such a problem. They are buffered sufficiently that you won't see them in time if the system hangs shortly after printk is called.
It is far more productive to selectively comment out sections of your driver and by process of elimination determine which line is the (first) problem.
Begin by commenting out the entire module's init section leaving only return 0;. Build it and load it. Does it hang? Reboot system, reenable the next few lines (class_create()?) and repeat.
From what you are telling, it is looks like that Linux scheduler is deadlocking by your driver. That's mean that interrupts from the system timer doesn't arrive or have a chance to be handled by kernel. There are two possible reasons:
You hang somewhere in your driver interrupt handler (handler starts its work but never finish it).
Your device creates interrupts storm (Device generates interrupts too frequently as a result your system do the only job -- handling of your device interrupts).
You explicitly disable all interrupts in your driver but doesn't reenable them.
In all other cases system will either crash, either oops or panic with all appropriate outputs or tolerate potential misbehavior of your device.
I guess that printk won't work for such extreme scenario as hang in kernel mode. It is quite heavy weight and due to this unreliable diagnostic tool for scenarios like your.
This trick works only in simpler environments like bootloaders or more simple kernels where system runs in default low-end video mode and there is no need to sync access to the video memory. In such systems tracing via debugging output to the display via direct writing to the video memory can be great and in many times the only tool that can be used for debugging purposes. Linux is not the case.
What techniques can be recommended from the software debugging point of view:
Try to review you driver code devoting special attention to interrupt handler and places where you disable/enable interrupts for synchronization.
Commenting out of all driver logic with gradual uncommenting can help a lot with localization of the issue.
You can try to use remote kernel debugging of your driver. I advice to try to use virtual machine for that purposes, but I'm not aware about do they allow to pass the PCI device in the virtual machine.
You can try the trick with in-memory tracing. The idea is to preallocate the memory chunk with well known virtual and physical addresses and zeroes it. Then modify your driver to write the trace data in this chunk using its virtual address. (For example, assign an unique integer value to each event that you want to trace and write '1' into the appropriate index of bytes array in the preallocated memory cell). Then when your system will hang you can simply force full memory dump generation and then analyze the memory layout packed in the dump using physical address of the memory chunk with traces. I had used this technique with VmWare Workstation VM on Windows. When the system had hanged I just pause a VM instance and looked to the appropriate .vmem file that contains raw memory latout of the physical memory of the VM instance. Not sure that this trick will work easy or even will work at all on Linux, but I would try it.
Finally, you can try to trace the messages on the PCI bus, but I'm not an expert in this field and not sure do it can help in your case or not.
In general kernel debugging is a quite tricky task, where a lot of tricks in use and all they works only for a specific set of cases. :(
I would put a logic analyzer on the bus lines (on FPGA you could use chipscope or similar). You'll then be able to tell which access is in cause (and fix the hardware). It will be useful anyway in order to debug or analyze future issues.
Another way would be to use the kernel crash dump utility which saved me some headaches in the past. But depending your Linux distribution requires installing (available by default in RH). See http://people.redhat.com/anderson/crash_whitepaper/
There isn't really anything that is run before your init. Bus enumeration is done at boot, if that goes by without a hitch the earliest cause for freezing should be something in your driver init AFAIK.
You should be able to see printks as they are printed, they aren't buffered and should not get lost. That's applicable only in situations where you can directly see kernel output, such as on the text console or over a serial line. If there is some other application in the way, like displaying the kernel logs in a terminal in X11 or over ssh, it may not have a chance to read and display the logs before the computer freezes.
If for some other reasons the printks still do not work for you, you can instead have your init function return early. Just test and move the return to later in the init until you find the point where it crashes.
It's hard to say what is causing your freezes, but interrupts is one of those things I would look at first. Make sure the device really doesn't signal interrupts until the driver enables them (that includes clearing interrupt enables on system reset) and enable them in the driver only after all handlers are registered (also, clear interrupt status before enabling interrupts).
Second thing to look at would be bus master transfers, same thing applies: Make sure the device doesn't do anything until it's asked to and let the driver make sure that no busmaster transfers are active before enabling busmastering at the device level.
The fact that the kernel gets stuck as soon as you install your driver module makes me wonder if any other driver (built in to kernel?) is already driving the device. I made this mistake once which is why i am asking. I'd look for the string "kernel driver in use" in the output of 'lspci' before installing the module. In any case, your printk's should be visible in dmesg output.
in addition to Claudio's suggestion, couple more debug ideas:
1. try kgdb (https://www.kernel.org/doc/htmldocs/kgdb/EnableKGDB.html)
2. use JTAG interfaces to connect to debug tools (these i think vary between devices, vendors so you'll have to figure out which debug tools you need to the particular hardware)