Is it possible to get backtrace of kext without attaching with gdb as described
at
http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KEXTConcept/KEXTConceptDebugger/debug_tutorial.html
if I have the panic log?
Somehow like this:
Get the address of kext caused panic from panic log
Generate dSYM file with kextutil
Paste the method's names from dSYM file into panic log to get backtrace?
Apple's tech note tn2063 describes analysing panics in detail. http://developer.apple.com/library/mac/ipad/#technotes/tn2063/_index.html
In addition, tn2118 describes analyzing kernel core dumps:
http://developer.apple.com/library/mac/#technotes/tn2004/tn2118.html
You can get the kernel to dump on panic, then take that core dump and analyze it against the symbolicated kernel. You add your own kext's symbols to the kernel's with gdb's add-symbol-file command.
Related
I want to read dmesg logs that are present in the kernel core file. The usual way is to make use of a utility like crash to open the core file with a corresponding vmlinux.
If I am not wrong, reading the dmesg logs from a core file has no symbol dependency and hence no vmlinux dependency.
Secondly, the running system may not have any utilities available to open core files. So I want to implement my own script/utility that can open the core file as a normal file and parse it to dump the dmesg logs.
Is it possible? If yes, what's the format of the core file, so that I can locate where the dmesg starts and end?
The crash tool has in his interactive prompt the command log to display the kernel message buffer (Analyzing a core dump). You'll find the source under crash-utility/crash and may proceed further with searching for log_buf.
What's the format of the core file, so that I can locate where the dmesg starts and end?
It seems that "the core dump file format is using the ELF format but is not described in the ELF standard."
You may also find further information for "How to analyze Linux memory or core dumps" within the Volatility Framework and which is written in Python. In example search for dmesg.
I don't believe the coredump file has kernel logs in in, if it had strings should've printed that out. What I think crash does is to print out the content in the kernel ring buffer to the user, if any. Usually when a fault occurs in the kernel, the kernel puts some useful information in the kernel ring buffer but it's not for user-space applications.
A coredump is simply just an ELF file with some additional data in it, if you want to parse the ELF format you'll have to look at the specs or maybe use a tool/library.
I am trying to hunt down retain leaks in an open-source project to support I2C based trackpads (https://github.com/kprinssu/VoodooI2CHID).
The reason why I believe that there are retain leaks is because when I attempt to unload the kernel extension via the following commands:
sudo kextunload -verbose 6 VoodooI2CHID.kext
I get the following output:
Kext user-space log filter changed from 0xff2 to 0xfff.
Kext kernel-space log filter changed from 0xff2 to 0xfff.
Kext library architecture set to x86_64.
Requesting unload of com.alexandred.VoodooI2CHID (with termnation of IOServices).
(kernel) User-space log flags changed from 0x0 to 0xfff.
(kernel) Received 'Unload' request from user space.
(kernel) Rescheduling scan for unused kexts in 60 seconds.
(kernel) Can't unload kext com.alexandred.VoodooI2CHID; classes have instances:
(kernel) Kext com.alexandred.VoodooI2CHID class VoodooI2CPrecisionTouchpadHIDEventDriver has 1 instance.
(kernel) Kext com.alexandred.VoodooI2CHID class VoodooI2CMultitouchHIDEventDriver has 1 instance.
Kernel error handling kext request - (libkern/kext) kext is in use or retained (cannot unload).
Failed to unload com.alexandred.VoodooI2CHID - (libkern/kext) kext is in use or retained (cannot unload).
I came across pmdj's excellent answer on tracking down retain leaks (Can't Unload Kernel Extension; Classes Have Instances). I verfied that my situation is the second case via ioreg (classes are being terminated but are not properly freed). Additionally, I used pmdj's hint by overiding taggedRelease and taggedRetain (https://stackoverflow.com/a/13471512/48660) to print the stack trace of the function calls.
Here's where I run into problems, I cannot use atos to convert the hex addresses back into human readable symbols. I use the follow command to generate the symbols:
atos -arch x86_x64 -o VoodooI2C.kext/Contents/MacOS/VoodooI2C -l 0xffffff7f8432b000 0xffffff804588dfa0
The load address parameter is retrieved from kextstat and I expect the -l argument should handle the slide arithmetic.
atos should return a valid symbol but all I get is the hex address back. In the above example, I get 0xffffff804588dfa0 as the output. Can anybody point out what I exactly I am missing?
Both kextstat and OSReportWithBacktrace report unslid addresses, so KASLR is not your problem.
Notice that your kext is apparently loaded at 0xffffff7f8432b000, whereas your backtrace frame address is 0xffffff804588dfa0. This is quite far apart, and indeed kexts are always loaded in the 0xffffff7f8??????? (unslid) range, so 0xffffff804588dfa0 can't be anywhere near kext code. (the offset is about 3GB) It's almost certainly a function in the kernel proper. If you use atos with the appropriate running kernel's binary, it should be able to locate which one. For example:
atos -o /Library/Developer/KDKs/KDK_10.14.5_18F132.kdk/System/Library/Kernels/kernel 0xffffff804588dfa0
(I don't know what kernel version you are using, and this address doesn't seem to be meaningful in the 18F132 kernel, but you get the idea.)
I would like to use it to debug kernel drivers but I would try to avoid to add logging to all functions. OSReportWithBacktrace seems to work but I need symbols.
I'm not aware of a way to print symbolicated stack traces directly from a kext. You can get symbolicated panic logs by adding keepsyms=1 to the boot-args nvram variable. I suspect the data structures for this have private linkage so you probably can't replicate the symbolicated panic code in your own kext. (It's in osfmk/i386/AT386/model_dep.c of the xnu source though if you want to try.)
Your other option is to send the output from OSReportWithBacktrace through the atos command-line tool. For kext symbols, you'll need to find the kext's load address from kextstat and pass that to the -l command line argument.
Finally, you can of course use lldb kernel debugging to get a stack trace. If you need to set a breakpoint during early kext load, before you get a chance to do it from the lldb command line, you can insert __asm__("int $3") (IIRC) at the point in the code where you want to break into the debugger.
I am doing some OSX kernel debugging with lldb and KDK.
When the kernel crash,I want to view the zones,and search the zones.
So I use:
(lldb) command script import lldb.macosx.heap
(lldb) cstr_refs CSTRING
This command is always working in Ring3 debugging, but when at kernel debugging, lldb give me an error:
error: error: use of undeclared identifier 'malloc_get_all_zones'
error: 1 errors parsing expression
The script heap.py is unusable in kernel?
How to search the kernel zones at this situation？
Someone more familiar with kernel issues can maybe tell you how to get the information you want out of the kernel. I can answer the part about "heap.py". It is only meant to be used when debugging userland programs. It relies on details of the userland malloc implementation, and it relies on being able to call functions in the debugee which is not currently possible when debugging the kernel.
Note, if you get the KDK so you have the dSYM for the mach kernel, it defines a bunch of commands that poke around in the kernel's data structures. It may be one of them will tell you what you want to know. Remember to run the lldb command:
(lldb) settings set target.load-script-from-symbol-file true
in order to allow lldb to read in the Python from the dSYM that defines all these macros. Then running the lldb help command will show you all the kernel-specific commands.
I am testing a new driver on FreeBSD kernel.
This might be trivial for experienced developers, but I can't figure out the solution to this problem.
I have a kernel panic and when it panics, I get the backtrace of the panic.
The backtrace says that the panic occurred at say foo_bar() + 0x94. How can I extract the line no corresponding to foo_bar() + 0x94?
The kernel is built with debugging symbols. I have tried grepping nm kernel but it only contains debugging symbols.
What can I do to find the exact line no?
I suggest to read the FreeBSD Handbook on Kernel debugging
https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html
It has detailed explanation of how to create a core file and how to invoke the gdb.
Configure crashdumps - it's usually just adding 'dumpdev="AUTO"' to /etc/rc.conf and rebooting - and, after crash and subsequent reboot, analyze the dump with debugger, like this: "kgdb /boot/kernel/kernel /var/crash/vmcore.latest". The "kgdb" thing is basically GDB hacked up to support kernel debugging; the "where" command should show you the backtrace.