I'm currently trying to debug my kernel. My goal is to put a breakpoint in a new syscall that I am implementing. The kernel runs on a remote Imx6q board. I've setup the JTAG debugger and I can connect GDB to it and pause the execution.
My issue is whith debug symbols.
I've added those properties to my defconfig :
CONFIG_GDB_SCRIPTS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_RANDOMIZE_BASE=n
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_DEBUG_INFO=y
When I start a session :
(gdb) tar ext :3333
Remote debugging using :3333
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0xa7780ef0 in ?? ()
(gdb) c
Continuing.
I can attach a symbol file by hand, but it is required to provide an address to attach it.
(gdb) add-symbol-file /home/tlavocat/development/android/out/target/product/wandboard_qca/kernel-imx/kernel/sys.o
The address where /home/tlavocat/development/android/out/target/product/wandboard_qca/kernel-imx/kernel/sys.o has been loaded is missing
The function I want to stop in is this one :
wandboard_qca:/ # cat /proc/kallsyms | grep sys_keeper_get_state
c003e0ac T sys_keeper_get_state
And it is implemented in kernel/sys.c.
My question is, how can I attach my symbols correctly to the right address ?
Thank's for your answers
I simply needed to load le kernel binary.
file .out/target/product/wandboard_qca/kernel-imx/vmlinux
And then connect to the remote target.
My desktop is Intel x86_64 processor with Ubuntu operating system.
I know there is perf tool to get a list of statistics of a program.
But what I am trying to do is read performance counter directly without using the perf tool.
First Question
First Questions is I downloaded this code from Github: Github Code Reference.
It compiled successfully with linux-headers-5.3.0-40-generic kernel without any errors. Once I use "insmod" the .ko file, the system hangs. The .ko file is not inserted when I checked the dmesg, so I have to cease it after I do "insmod" the .ko file. Does it happen because I attempted unauthorized access? If there are suggestions that I can try, I am glad to hear that.
The corresponding code is below.
static void set_pce(void *arg)
{
int to_val = (arg != 0);
u_int64_t cr4_val;
cr4_val = __read_cr4();
if (to_val) {
cr4_val |= X86_CR4_PCE;
} else {
cr4_val &= ~X86_CR4_PCE;
}
__write_cr4(cr4_val);
}
static int __init user_rdpmc_init(void){
int cpu;
num_cpus = num_online_cpus();
printk(KERN_INFO "Enabling RDPMC from ring 3 for %d CPUs\n", num_cpus);
for (cpu = 0; cpu < num_cpus; cpu++) {
smp_call_function_single(cpu, set_pce, (void *) 1, 1);
}
return 0;
}
Second Question
Second question is I am using linux-headers-5.3.0-40-generic kernel version in my Ubuntu desktop. I downloaded kernel code version 5.5.3 from kernel.org. I followed the perf code given in the 5.5.3 kernel code thoroughly and discovered that core.c file under linux-5.5.3/arch/x86/events/intel directory actually does setting and reading the performance counters. I used the core.c file contents to make it as a module to read the performance counter. When I compile it, it creates a bunch of errors because I use linux-headers-5.3.0-40-generic to build the module but my ubuntu kernel doesn't have all header files linked to the core.c file from kernel code from kernel.org.
How can I make my Ubuntu kernel use all the files linked to core.c from kernel.org and build the .ko file?
Or Is there any module source code that has x86 performance counter reading that I can use as a reference?
Thank you for your help in advance.
I know there is perf tool to get a list of statistics of a program. But what I am trying to do is read performance counter directly without using the perf tool.
If you do not want to use perf tool, you can try to use oprofile tool or intel vtune or https://github.com/RRZE-HPC/likwid or https://github.com/opcm/pcm. Or you can use perf_event_open syscall which is how perf tool works (you can study or modify perf tool sources from https://mirrors.edge.kernel.org/pub/linux/kernel/tools/perf/ - and perf tool version may not be equal to kernel version).
If you want to access msr registers as root, use modprobe msr (this is standard kernel module, already compiled for your kernel in ubuntu) and wrmsr and rdmsr tools (msr-tools deb/ubuntu package, by intel), like in slide 27 of Performance Monitoring Chris Dahnken Intel SSG EMEA HPCTC presentation.
I don't understand why do you want to work with performance counters without perf tool. If you want to get counter readings from inside of your program, for example before and after some loops, you can use perf_event_open syscall (with specific ioctls) directly. (Or try to use perf stat + same ioctls PERF_EVENT_IOC_* or try to learn perf + JIT integration)
Or you can use existing kernel module which will export msr register access to root user - the msr.ko. And msr tools - https://01.org/msr-tools. Or with this msr+pmc example https://technicalandstuff.wordpress.com/2015/05/15/using-intels-pcm-in-linux-and-inside-c/ + https://software.intel.com/en-us/articles/intel-performance-counter-monitor (https://github.com/opcm/pcm)
There are also some examples of perf counters usage in https://github.com/RRZE-HPC/likwid.
You can also use PAPI library to access counters from your code, it will handle most of perf_event_open stuff for you. http://icl.cs.utk.edu/projects/papi/wiki/PAPITopics:Getting_Started
First Questions is I downloaded this code https://github.com/softdevteam/user_rdpmc ... "insmod" the .ko file, the system hangs.
There are too low "Stars" rating and the code is too old (2016) to really doing any investigations on the hang. Direct access of PMC may interfere with NMI watchdog (do echo 0 > /proc/sys/kernel/nmi_watchdog as root) or other perf session. It is safer to use perf_event_open syscall.
Second question ... discovered that core.c file under linux-5.5.3/arch/x86/events/intel directory actually does setting and reading the performance counters
This file is part of perf_event_open syscall implementation (perf_events subsystem of the kernel, https://github.com/torvalds/linux/tree/master/kernel/events + https://github.com/torvalds/linux/tree/master/arch/x86/events).
To use this code you can use the perf tool or perf_event_open syscall.
You should not compile the perf_events subsystem of the kernel as separate module because it is already compiled into your kernel (intel/amd specific part can be partially ko) and the Subsystem itself does not support compilation as module:
https://github.com/torvalds/linux/tree/master/kernel/events
Makefile: obj-y := core.o ring_buffer.o callchain.o
How can I make my Ubuntu kernel use all the files linked to core.c from kernel.org and build the .ko file?
Your ubuntu kernel already have all perf_events subsystem files compiled, some are linked into the kernel image and other are .ko files already installed like intel-rapl-perf.ko
$ grep _PERF_ /boot/config-`uname -r`
$ ls -l /lib/modules/`uname -r`/kernel/arch/x86/events/intel
Is there any Compatibility Test Suite for linux kernel like we have for Android Operating system.
e.g. like when we make changes to the kernel , adding static modules, adding our own programs,configuring kernel and then building.
is there any tool to check that can check compatibility of our kernel.
like is the kernel services like IPC,Memory Memory management,Device Manager will work correct ?
So that could pass/fail/rate the generated kernel.To predict the behavior in the target system.
You can at least check compatibility of API/ABI of your changed kernel via abi-dumper and abi-compliance-checker tools: http://upstream.rosalinux.ru/kernel/
To do this you should add these config options when building your kernel:
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=n
And then prepare compatibility report:
abi-dumper vmlinux.old -o vmlinux-ABI-0.dump
abi-dumper vmlinux.new -o vmlinux-ABI-1.dump
abi-compliance-checker -l vmlinux -old vmlinux-ABI-0.dump -new vmlinux-ABI-1.dump
EDIT: the project has been moved to http://abi-laboratory.pro/tracker/timeline/linux/
I'm trying to start kernel debugging with this sytem:
Amontec JTAGkey2, openocd, gdb, eclipse.
At the end I would like to debug kernel and application that is running within.
I have few problems, and it seems that I need to solve them sequently.
Now I have CPU suspend/resume, read/write RAM
What is missing: Step into, Step over, C/C++ Level debugging.
I do following:
- Connect JTAG, Power up board, start uImage with Debug messages via Uboot
- start openocd:
# openocd -f /usr/share/openocd/scripts/interface/jtagkey2.cfg -f /usr/share/openocd/scripts/board/at91sam9g20-ek.cfg
Output:
jtag_nsrst_delay: 200
jtag_ntrst_delay: 200
RCLK - adaptive
TapName | Enabled | IdCode Expected IrLen IrCap IrMask Instr
---|--------------------|---------|------------|------------|------|------|------|---------
0 | at91sam9g20.cpu | Y | 0x00000000 | 0x0792603f | 0x04 | 0x01 | 0x0f | 0x0f
Info : max TCK change to: 30000 kHz
Info : RCLK (adaptive clock speed)
Info : JTAG tap: at91sam9g20.cpu tap/device found: 0x0792603f (mfg: 0x01f, part: 0x7926, ver: 0x0)
Info : Embedded ICE version 6
And problems starts here:
openocd:
Warn : acknowledgment received, but no packet pending
undefined debug reason 6 - target needs reset
Warn : target not halted
eclipse:
symbol-file /opt/Tixi_Repos/KiwiG6v2/buildroot-2011.05/package_tixi/linux-2.6.39/arch/arm/boot/compressed/vmlinux
target remote localhost:3333
start () at arch/arm/boot/compressed/head.S:108
108 kphex r5, 8 /* end of kernel */
It seems also that JTAG is trying to load the code into 0x0, what is incorrect I suppose:
Update 1:
After analyzing some online tutorials for ARM:
Eclipse Reset and Halt commands doesn't work perfect. It is better to uncheck them and write into command window. Also load address can be add:
monitor halt
load arch/arm/boot/compressed/vmlinux 0x22000000
I don't use
monitor reset
I let Uboot start and initialize RAM and other peripherals. Then I stop Uboot by getting into shell. Then I let eclipse write linux into RAM, and start it. It takes very long, but works bit better. Kernel starts and stopps on RPC initialization without giving console back.
would it be possible to load kernel into RAM within Uboot console, and start JTAG session afterwards ?
What is the difference between [load ...] and [monitor load...] commands
Why do I need to load /compressed/vmlinux instead of uImage ?
in eclipse window I have two load fields: load image i load symbol. I disable both options but write only load arch/arm/boot/compressed/vmlinux 0x22000000. Is it maybe the reason for next problems ?
Update 2:
Ok. Thank you for hints.
I've made some progress. Could you give me some advices, maybe I'm still doing something wrong.
Now my kernel runs under JTAG control, but I still can't debug on source code level.
I do as follows:
Power up the board, go into uboot shell.
start openOCD session
Set Uboot breakpoint in bootm.c on theKernel call:
cleanup_before_linux ();
theKernel (0, machid, bd->bi_boot_params);
start eclipse debug session :
monitor halt
load uboot-a without offset
load u-boot-2010.06/u-boot
Loading section .text, size 0x349ec lma 0x26f00000
start uboot and let it run
uboot stopps on "theKernel" call
I know that kernel is located on address 0x20008000.
restart openOCD session
start ecipse debugger once more with kernel configuration:
monitor halt
load kernel on address 0x20008000
load arch/arm/boot/compressed/vmlinux 0x20008000
Loading section .text, size 0x8bdc7c lma 0x20008000
start debugg session
Everything works fine now, and kernel starts, but I still can't debug on source code level.
"symbol is not available"
DEBUG and DEBUG_INFO are on for kernel.
vmlinux screenshot
What seems starnge for me that there are around 50 function symbols in this file.
I'm compiling a custom kernel under Ubuntu and I'm running into the problem that my kernel doesn't seem to know where to look for firmware. Under Ubuntu 8.04, firmware is tied to kernel version the same way driver modules are. For example, kernel 2.6.24-24-generic stores its kernel modules in:
/lib/modules/2.6.24-24-generic
and its firmware in:
/lib/firmware/2.6.24-24-generic
When I compile the 2.6.24-24-generic Ubuntu kernel according the "Alternate Build Method: The Old-Fashioned Debian Way" I get the appropriate modules directory and all my devices work except those requiring firmware such as my Intel wireless card (ipw2200 module).
The kernel log shows for example that when ipw2200 tries to load the firmware the kernel subsystem controlling the loading of firmware is unable to locate it:
ipw2200: Detected Intel PRO/Wireless 2200BG Network Connection
ipw2200: ipw2200-bss.fw request_firmware failed: Reason -2
errno-base.h defines this as:
#define ENOENT 2 /* No such file or directory */
(The function returning ENOENT puts a minus in front of it.)
I tried creating a symlink in /lib/firmware where my kernel's name pointed to the 2.6.24-24-generic directory, however this resulted in the same error. This firmware is non-GPL, provided by Intel and packed by Ubuntu. I don't believe it has any actual tie to a particular kernel version. cmp shows that the versions in the various directories are identical.
So how does the kernel know where to look for firmware?
Update
I found this solution to the exact problem I'm having, however it no longer works as Ubuntu has eliminated /etc/hotplug.d and no longer stores its firmware in /usr/lib/hotplug/firmware.
Update2
Some more research turned up some more answers. Up until version 92 of udev, the program firmware_helper was the way firmware got loaded. Starting with udev 93 this program was replaced with a script named firmware.sh providing identical functionality as far as I can tell. Both of these hardcode the firmware path to /lib/firmware. Ubuntu still seems to be using the /lib/udev/firmware_helper binary.
The name of the firmware file is passed to firmware_helper in the environment variable $FIRMWARE which is concatenated to the path /lib/firmware and used to load the firmware.
The actual request to load the firmware is made by the driver (ipw2200 in my case) via the system call:
request_firmware(..., "ipw2200-bss.fw", ...);
Now somewhere in between the driver calling request_firmware and firmware_helper looking at the $FIRMWARE environment variable, the kernel package name is getting prepended to the firmware name.
So who's doing it?
From the kernel's perspective, see /usr/src/linux/Documentation/firmware_class/README:
kernel(driver): calls request_firmware(&fw_entry, $FIRMWARE, device)
userspace:
- /sys/class/firmware/xxx/{loading,data} appear.
- hotplug gets called with a firmware identifier in $FIRMWARE
and the usual hotplug environment.
- hotplug: echo 1 > /sys/class/firmware/xxx/loading
kernel: Discard any previous partial load.
userspace:
- hotplug: cat appropriate_firmware_image > \
/sys/class/firmware/xxx/data
kernel: grows a buffer in PAGE_SIZE increments to hold the image as it
comes in.
userspace:
- hotplug: echo 0 > /sys/class/firmware/xxx/loading
kernel: request_firmware() returns and the driver has the firmware
image in fw_entry->{data,size}. If something went wrong
request_firmware() returns non-zero and fw_entry is set to
NULL.
kernel(driver): Driver code calls release_firmware(fw_entry) releasing
the firmware image and any related resource.
The kernel doesn't actually load any firmware at all. It simply informs userspace, "I want a firmware by the name of xxx", and waits for userspace to pipe the firmware image back to the kernel.
Now, on Ubuntu 8.04,
$ grep firmware /etc/udev/rules.d/80-program.rules
# Load firmware on demand
SUBSYSTEM=="firmware", ACTION=="add", RUN+="firmware_helper"
so as you've discovered, udev is configured to run firmware_helper when the kernel asks for firmware.
$ apt-get source udev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Need to get 312kB of source archives.
Get:1 http://us.archive.ubuntu.com hardy-security/main udev 117-8ubuntu0.2 (dsc) [716B]
Get:2 http://us.archive.ubuntu.com hardy-security/main udev 117-8ubuntu0.2 (tar) [245kB]
Get:3 http://us.archive.ubuntu.com hardy-security/main udev 117-8ubuntu0.2 (diff) [65.7kB]
Fetched 312kB in 1s (223kB/s)
gpg: Signature made Tue 14 Apr 2009 05:31:34 PM EDT using DSA key ID 17063E6D
gpg: Can't check signature: public key not found
dpkg-source: extracting udev in udev-117
dpkg-source: unpacking udev_117.orig.tar.gz
dpkg-source: applying ./udev_117-8ubuntu0.2.diff.gz
$ cd udev-117/
$ cat debian/patches/80-extras-firmware.patch
If you read the source, you'll find that Ubuntu wrote a firmware_helper which is hard-coded to first look for /lib/modules/$(uname -r)/$FIRMWARE, then /lib/modules/$FIRMWARE, and no other locations. Translating it to sh, it does approximately this:
echo -n 1 > /sys/$DEVPATH/loading
cat /lib/firmware/$(uname -r)/$FIRMWARE > /sys/$DEVPATH/data \
|| cat /lib/firmware/$FIRMWARE > /sys/$DEVPATH/data
if [ $? = 0 ]; then
echo -n 1 > /sys/$DEVPATH/loading
echo -n -1 > /sys/$DEVPATH/loading
fi
which is exactly the format the kernel expects.
To make a long story short: Ubuntu's udev package has customizations that always look in /lib/firmware/$(uname -r) first. This policy is being handled in userspace.
Wow this is very useful information and it led me to the solution for my problem when making a custom USB kernel module for a device requiring firmware.
Basically, every Ubuntu brings a new rehash of hal,sysfs,devfs,udev,and so on...and things just change. In fact I read they stopped using hal.
So let's reverse engineer this yet again so it's pertinent to the latest [Ubuntu] systems.
On Ubuntu Lucid (the latest at time of writing), /lib/udev/rules.d/50-firmware.rules is used. This file calls the binary /lib/udev/firmware, where magic happens.
Listing: /lib/udev/rules.d/50-firmware.rules
# firmware-class requests, copies files into the kernel
SUBSYSTEM=="firmware", ACTION=="add", RUN+="firmware --firmware=$env{FIRMWARE} --devpath=$env{DEVPATH}"
The magic should be something along these lines (source: Linux Device Drivers, 3rd Ed., Ch. 14: The Linux Device Model):
echo 1 to loading
copy firmware to data
on failure, echo -1 to loading and halt firmware loading process
echo 0 to loading (signal the kernel)
then, a specific kernel module receives the data and pushes it to the device
If you look at Lucid's source page for udev, in udev-151/extras/firmware/firmware.c, the source for that firmware /lib/udev/firmware binary, that's exactly what goes on.
Excerpt: Lucid source, udev-151/extras/firmware/firmware.c
util_strscpyl(datapath, sizeof(datapath), udev_get_sys_path(udev), devpath, "/data", NULL);
if (!copy_firmware(udev, fwpath, datapath, statbuf.st_size)) {
err(udev, "error sending firmware '%s' to device\n", firmware);
set_loading(udev, loadpath, "-1");
rc = 4;
goto exit;
};
set_loading(udev, loadpath, "0");
Additionally, many devices use an Intel HEX format (textish files containing checksum and other stuff) (wiki it i have no reputation and no ability to link). The kernel program ihex2fw (called from Makefile in kernel_source/lib/firmware on .HEX files) converts these HEX files to an arbitrary-designed binary format that the Linux kernel then picks up with request_ihex_firmware, because they thought reading text files in the kernel was silly (it would slow things down).
On current Linux systems, this is handled via udev and the firmware.agent.
Linux 3.5.7 Gentoo, I have the same issue.
SOLVED:
emerge ipw2200-firmware
Then go to /usr/src/linux
make menucofig
on device driver, remove all wirless drivers don't needed, set Intell 2200 as module and recompile.
make
make modules_install
cp arch/x86/boot/bzImage /boot/kernel-yourdefault