I want to read dmesg logs that are present in the kernel core file. The usual way is to make use of a utility like crash to open the core file with a corresponding vmlinux.
If I am not wrong, reading the dmesg logs from a core file has no symbol dependency and hence no vmlinux dependency.
Secondly, the running system may not have any utilities available to open core files. So I want to implement my own script/utility that can open the core file as a normal file and parse it to dump the dmesg logs.
Is it possible? If yes, what's the format of the core file, so that I can locate where the dmesg starts and end?
The crash tool has in his interactive prompt the command log to display the kernel message buffer (Analyzing a core dump). You'll find the source under crash-utility/crash and may proceed further with searching for log_buf.
What's the format of the core file, so that I can locate where the dmesg starts and end?
It seems that "the core dump file format is using the ELF format but is not described in the ELF standard."
You may also find further information for "How to analyze Linux memory or core dumps" within the Volatility Framework and which is written in Python. In example search for dmesg.
I don't believe the coredump file has kernel logs in in, if it had strings should've printed that out. What I think crash does is to print out the content in the kernel ring buffer to the user, if any. Usually when a fault occurs in the kernel, the kernel puts some useful information in the kernel ring buffer but it's not for user-space applications.
A coredump is simply just an ELF file with some additional data in it, if you want to parse the ELF format you'll have to look at the specs or maybe use a tool/library.
Related
I'm currently working on an embedded CPU (ARM cortex-M0) on board. I'm experiencing crashes (HardFault) and thus, I'm trying to debug them.
Currently I can get a dump of the memory (I send everything in hex over the printf console, my only access to the memory).
My idea is to load that dump as coredump to gdb and thus being able to debug the program.
How is it possible to create a "real" coredump file from the raw memory dump, so that I can give it directly to gdb ?
I also tried to run my program in the gdb simulator to use the restore command with my raw memory but I don't understand how to run it. Do you have an idea ?
The main question here is in fact: How to create a coredump file that is compatible with gdb based on the program elf and the raw memory content ?
There are a pair of libraries that will do this: CrashCatcher and CrashDebug. This first prints a dump (which you are already doing but this prints in a specific format) and the second interfaces into gdb to allow loading the dump. For some reason CrashDebug doesn't create a core file but rather interfaces into gdb using gdb's target remote command. Same effect though. It supports Cortex-M0, M3, and M4.
I am testing a new driver on FreeBSD kernel.
This might be trivial for experienced developers, but I can't figure out the solution to this problem.
I have a kernel panic and when it panics, I get the backtrace of the panic.
The backtrace says that the panic occurred at say foo_bar() + 0x94. How can I extract the line no corresponding to foo_bar() + 0x94?
The kernel is built with debugging symbols. I have tried grepping nm kernel but it only contains debugging symbols.
What can I do to find the exact line no?
I suggest to read the FreeBSD Handbook on Kernel debugging
https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html
It has detailed explanation of how to create a core file and how to invoke the gdb.
Configure crashdumps - it's usually just adding 'dumpdev="AUTO"' to /etc/rc.conf and rebooting - and, after crash and subsequent reboot, analyze the dump with debugger, like this: "kgdb /boot/kernel/kernel /var/crash/vmcore.latest". The "kgdb" thing is basically GDB hacked up to support kernel debugging; the "where" command should show you the backtrace.
I understand that the .exit.text section will be filled by the linker with functions defined with __exit macro. I can see the section on dissassembly of the vmlinux file. But I did not understand at what point are the contents discarded once the file is loaded in memory? Does it happen in the kernel code? Can anyone explain this more?
Thanks in advance!
The files vmlinux.lds.S and module.c handle this. The handling depends on your kernel version and configuration; neither have been given. My assumption is you mean the .exit.text section in the kernel and not some user space task. Generally, the kernel doesn't exit, so they are discarded by the linker script unless you have some debugging configurations.
Edit: The android kernel was made with a .config file which enables Linux kernel features at compile time. The .config for CONFIG_HOTPLUG_CPU is a CPU; some big servers can keep running even when the CPU is exchanged. It is hard to answer your question. I don't know where your Android kernel source is or what .config options it was built with.
The sections are kept with GENERIC_BUG; so it is possible your kernel was built with this. From the stock vmlinux.lds.S, the .exit.data section is put before __init_end, so it is freed and returned to memory after the init portion runs. Ie, present only during boot. Under these circumstances, you will see it in the vmlinux ELF, but not in the runtime kcore or however you are dumping it.
Specifically, main.c's start_kernel(), rest_init(), kernel_init(), and free_initmem() are the place where this stuff is discarded.
Trying to get my head around symbolicate and dSYM file and how can they be useful to debugging.
Been following this start guide. http://aplus.rs/ios-dev/guide-to-symbolicating-iphone-app-crash-logs-with-xcode-4-2/
So in what way can a dSYM file be useful? I guess I'm confused with the terminology "symbolicate". Is it just useful to generate a .crash log file or can it actually do more than that?
Thank you,
Tee
A dSYM contains a dwarf file for mapping of memory addresses and methods and lines of code in your source code. When an app crashes it writes a .crash log file which only contains memory addresses of the code being executed at the time a crash occurred. "Symbolicate" means transforming these memory addresses to class, method and line of code that. So it helps understanding what an app was doing when it crashed.
The debugger uses the same file to provide stack traces and more information while debugging a running application.
So it is useful and needed in both cases for understanding a crash report and also while debugging.
Is it possible to get backtrace of kext without attaching with gdb as described
at
http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KEXTConcept/KEXTConceptDebugger/debug_tutorial.html
if I have the panic log?
Somehow like this:
Get the address of kext caused panic from panic log
Generate dSYM file with kextutil
Paste the method's names from dSYM file into panic log to get backtrace?
Apple's tech note tn2063 describes analysing panics in detail. http://developer.apple.com/library/mac/ipad/#technotes/tn2063/_index.html
In addition, tn2118 describes analyzing kernel core dumps:
http://developer.apple.com/library/mac/#technotes/tn2004/tn2118.html
You can get the kernel to dump on panic, then take that core dump and analyze it against the symbolicated kernel. You add your own kext's symbols to the kernel's with gdb's add-symbol-file command.