I am doing some OSX kernel debugging with lldb and KDK.
When the kernel crash,I want to view the zones,and search the zones.
So I use:
(lldb) command script import lldb.macosx.heap
(lldb) cstr_refs CSTRING
This command is always working in Ring3 debugging, but when at kernel debugging, lldb give me an error:
error: error: use of undeclared identifier 'malloc_get_all_zones'
error: 1 errors parsing expression
The script heap.py is unusable in kernel?
How to search the kernel zones at this situation？
Someone more familiar with kernel issues can maybe tell you how to get the information you want out of the kernel. I can answer the part about "heap.py". It is only meant to be used when debugging userland programs. It relies on details of the userland malloc implementation, and it relies on being able to call functions in the debugee which is not currently possible when debugging the kernel.
Note, if you get the KDK so you have the dSYM for the mach kernel, it defines a bunch of commands that poke around in the kernel's data structures. It may be one of them will tell you what you want to know. Remember to run the lldb command:
(lldb) settings set target.load-script-from-symbol-file true
in order to allow lldb to read in the Python from the dSYM that defines all these macros. Then running the lldb help command will show you all the kernel-specific commands.
Related
I am trying to write a simple OS, I already wrote a bootloader but now I want to debug it, so I switched from using VirtualBox to QEMU because I saw it had better debugging.
The problem is that after I added the -s parameter to QEMU command and successfully connected via GDB, it says that the symbol table isn't loaded and that I should use the "file" command.
The only difference from what I did to what I saw people on the Internet do, is that they started GDB with gdb vmlinux, but I can't do that because I am not debugging a Linux kernel... so I figured that the issue is that I didn't start GDB with an executable, but using the "file" command on my OS image, and the compiled and linked .out file, tells me it's a "DOS/MBR boot sector", so I can't start GDB with either of them (I tried to do that anyways, but GDB failed).
Help would be appreciated.
EDIT: also, I did assemble the bootloader with the -g and --gstabs+ options.
gdb would like a file so that it can give you symbolic debugging information. For that you will need to give it a file in a format with debug info which corresponds to where your OS ends up in RAM. The "DOS/MBR boot sector" file is a disk image (the BIOS will load part of this into RAM for you, and it will then presumably finish loading code itself).
But gdb will also entirely happily let you do assembly-level debugging; you can just ignore the warning about not having a symbol table, and use the single step instruction, disassemble-from-pc and similar commands:
"disas $pc,+32" disassembles 32 bytes from the current PC
the display command prints after execution stops, so "disp /3i $pc" will print the next 3 instructions every time gdb gets control
"stepi" and "nexti" do single-instruction step/next ("step" and "next" are source-line stepping and require debug info)
I would like to use it to debug kernel drivers but I would try to avoid to add logging to all functions. OSReportWithBacktrace seems to work but I need symbols.
I'm not aware of a way to print symbolicated stack traces directly from a kext. You can get symbolicated panic logs by adding keepsyms=1 to the boot-args nvram variable. I suspect the data structures for this have private linkage so you probably can't replicate the symbolicated panic code in your own kext. (It's in osfmk/i386/AT386/model_dep.c of the xnu source though if you want to try.)
Your other option is to send the output from OSReportWithBacktrace through the atos command-line tool. For kext symbols, you'll need to find the kext's load address from kextstat and pass that to the -l command line argument.
Finally, you can of course use lldb kernel debugging to get a stack trace. If you need to set a breakpoint during early kext load, before you get a chance to do it from the lldb command line, you can insert __asm__("int $3") (IIRC) at the point in the code where you want to break into the debugger.
I am testing a new driver on FreeBSD kernel.
This might be trivial for experienced developers, but I can't figure out the solution to this problem.
I have a kernel panic and when it panics, I get the backtrace of the panic.
The backtrace says that the panic occurred at say foo_bar() + 0x94. How can I extract the line no corresponding to foo_bar() + 0x94?
The kernel is built with debugging symbols. I have tried grepping nm kernel but it only contains debugging symbols.
What can I do to find the exact line no?
I suggest to read the FreeBSD Handbook on Kernel debugging
https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html
It has detailed explanation of how to create a core file and how to invoke the gdb.
Configure crashdumps - it's usually just adding 'dumpdev="AUTO"' to /etc/rc.conf and rebooting - and, after crash and subsequent reboot, analyze the dump with debugger, like this: "kgdb /boot/kernel/kernel /var/crash/vmcore.latest". The "kgdb" thing is basically GDB hacked up to support kernel debugging; the "where" command should show you the backtrace.
So far, with gdb + qemu, I can step into/over linux kernel source code. Is it possible to debug the user space programs simultaneously? For example, single step a program from user space to kernel space so I can observe the changes of registers on the qemu monitor by issuing info registers?
Minimal step-by-setep setup
Mahouk is right, but here is a fully automated QEMU + Buildroot example which presuposes that you already know how to debug the kernel with QEMU + gdb and a more detailed exaplanation:
readelf -h myexecutable | grep Entry
Gives:
Entry point address: 0x4003a0
So inside GDB we need to do:
add-symbol-file myexecutable 0x4003a0
b main
And only then start the executable in QEMU:
myexecutable
A more reliable way to do that is to set myexecutable as the init process if you can do that.
add-symbol-file is also mentioned at: How to load multiple symbol files in gdb
Why would you ever want to do this instead of gdbserver?
I can only see one use case for this so far: debugging init: Debug init on Qemu using gdb
Otherwise, why not just use the following more reliable method, e.g. to step into a syscall:
start two remote GDBs:
one with qemu-system-* -s
the other gdbserver myexecutable as explained at: https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-mips-elf-with-qemu-toolchain/16214#16214
step in gdbserver's GDB as close as possible to the system call, which often mean stepping into the libc
on the QEMU's GDB, do e.g. b sys_read for the read syscall
back on gdbserver, do continue
I propose this because:
using the QEMU GDB for userland can lead to random jumps as the kernel context switches to another process that uses the same virtual addresses
I was not able to load shared libraries properly without gdbserver: attempting sharedlibrary directly gives:
(gdb) sharedlibrary ../../staging/lib/libc.so.0
No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.
As a consequence, since most kernel interactions go through the stdib, you would need to do a lot of smart assembly stepping to find the kernel entry, which could be impractical.
Until, that is, someone writes a smarter GDB scripts that steps every instruction until a context switch happens or until source become available. I wonder if such scripts would't be too slow, as the naive approach has the overhead of communication to-from GDB for every instruction.
This might get you started: Tell gdb to skip standard files
Parsing Linux kernel data structures
To do userland process debug properly, that's what we would have to do eventually: thread-aware gdb for the Linux kernel
I achieve it by using the gdb command add-symbol-file to add userspace programs debugging information. But you must know these programs loading addresses. so to be precise, you have to launch the kernel debugging by connecting gdb to gdbserver as usual; and then, you can add those program debugging information. You can also use .gdbinit script though. Read this
I am using gdb attached to a serial port of a virtual machine to debug linux kernel.
I am wondering, if there is any patches/plugins which can make the gdb understand some of linux kernel's data structure and make it "thread aware"?
By that I mean under gdb I can see how many kernel threads are there, their status, and for each thread, their stack information.
libvmi
https://github.com/libvmi/libvmi
This project does "LibVMI: Simplified Virtual Machine Introspection" which sounds really close.
This project in particular https://github.com/Wenzel/pyvmidbg uses libvmi and features a demo video of debugging a Windows userland application form inside it, without memory conflicts.
As of May 2019, there are two limitations however as of May 2019, both of which could be overcome with some work: https://github.com/Wenzel/pyvmidbg/issues/24
Linux memory parsing is not yet complete
requires Xen
The developer of that project also answered further at: https://stackoverflow.com/a/56369454/895245
Implementing it with those libraries would be in my opinion the best way to achieve this goal today.
Linaro lkd-python
First, this Linaro page claims to have a working setup: https://wiki.linaro.org/LandingTeams/ST/GDB that allows you to do usual thread operations such as thread, bt, etc., but it relies on a GDB fork. I will test it out later. In 2016, https://youtu.be/pqn5hIrz3A8 says that the implementation was in C, not as Python scripts unfortunately, which would be better and avoid forking. The sketch for lkd-python can be found at: https://git.linaro.org/people/lee.jones/kieran.bingham/binutils-gdb.git/log/?h=lkd-python
Linux kernel in-tree GDB scripts + my brain
I then tried to see what I could do with the kernel in-tree Python scripts at v4.17 + some manual intervention as a prototype, but didn't quite get there yet.
I have tested using this highly automated QEMU + Buildroot setup.
First follow the procedure I described at: How to debug the Linux kernel with GDB and QEMU? to get GDB working.
Then, as described at: How to debug Linux kernel modules with QEMU? run GDB with:
gdb -ex add-auto-load-safe-path /full/path/to/linux/kernel
This loads the in-tree GDB Python scripts from scripts/gdb.
One of those scripts provides:
lx-ps
which lists all threads with format:
0xffff88000ed08000 1 init
0xffff88000ed08ac0 2 kthreadd
The first field is the address of the task_struct struct, so we can see the entire struct with:
p (struct task_struct)*0xffff88000ed08000
which should in theory allow us to get any information we want about the process.
Now I wanted to find the PC. For ARM, I've seen: Find program counter of process in kernel and I tried:
task_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))->uregs[ARM_pc]
but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: How do I print a #defined constant in GDB? which are apparently not set?
I don't think GDB understands kernel data structures, that would make them version dependent. GDB uses ptrace for gathering information on any running process.
That's all I know :(
pyvmidbg developer here.
I will add some clarifications:
yes the goal of the project is indeed to have a cross-platform, guest-aware GDB stub.
Most of the implementation is already done for Windows, where we are aware of processes and their threads context.
It's possible to intercept a specific process (cmd.exe in the demo) and singlestep its execution (this is limited to 1 process with 1 thread for now), as well as attaching to a new process's entrypoint.
Regarding Linux, I looked at the internals and the resources that I could find, but I'm lacking the whole picture to figure out how I can:
- intercept a task when it's being scheduled (core/sched.c:switch_to() ?)
- read the task state (Windows's KTRAP_FRAME equivalent for Linux ?)
I asked a question on SO, but nobody answered :/
Linux context switch internals: how does a process goes back to userland after the switch?
If you can help with this, I can guide you through the implementation :)
Regarding the hypervisor support, only Xen is fully supported in the Libvmi interface at the moment.
I added a section in the README to describe where we are in terms of VMI APIs with other hypervisors.
Thanks !