I'm currently working on an embedded CPU (ARM cortex-M0) on board. I'm experiencing crashes (HardFault) and thus, I'm trying to debug them.
Currently I can get a dump of the memory (I send everything in hex over the printf console, my only access to the memory).
My idea is to load that dump as coredump to gdb and thus being able to debug the program.
How is it possible to create a "real" coredump file from the raw memory dump, so that I can give it directly to gdb ?
I also tried to run my program in the gdb simulator to use the restore command with my raw memory but I don't understand how to run it. Do you have an idea ?
The main question here is in fact: How to create a coredump file that is compatible with gdb based on the program elf and the raw memory content ?
There are a pair of libraries that will do this: CrashCatcher and CrashDebug. This first prints a dump (which you are already doing but this prints in a specific format) and the second interfaces into gdb to allow loading the dump. For some reason CrashDebug doesn't create a core file but rather interfaces into gdb using gdb's target remote command. Same effect though. It supports Cortex-M0, M3, and M4.
Related
I want to read dmesg logs that are present in the kernel core file. The usual way is to make use of a utility like crash to open the core file with a corresponding vmlinux.
If I am not wrong, reading the dmesg logs from a core file has no symbol dependency and hence no vmlinux dependency.
Secondly, the running system may not have any utilities available to open core files. So I want to implement my own script/utility that can open the core file as a normal file and parse it to dump the dmesg logs.
Is it possible? If yes, what's the format of the core file, so that I can locate where the dmesg starts and end?
The crash tool has in his interactive prompt the command log to display the kernel message buffer (Analyzing a core dump). You'll find the source under crash-utility/crash and may proceed further with searching for log_buf.
What's the format of the core file, so that I can locate where the dmesg starts and end?
It seems that "the core dump file format is using the ELF format but is not described in the ELF standard."
You may also find further information for "How to analyze Linux memory or core dumps" within the Volatility Framework and which is written in Python. In example search for dmesg.
I don't believe the coredump file has kernel logs in in, if it had strings should've printed that out. What I think crash does is to print out the content in the kernel ring buffer to the user, if any. Usually when a fault occurs in the kernel, the kernel puts some useful information in the kernel ring buffer but it's not for user-space applications.
A coredump is simply just an ELF file with some additional data in it, if you want to parse the ELF format you'll have to look at the specs or maybe use a tool/library.
I am trying to debug a driver in UEFI firmware (OVMF) via gdb as described here:
https://github.com/tianocore/tianocore.github.io/wiki/How-to-debug-OVMF-with-QEMU-using-GDB
It works well, but I discovered that just having debug symbols for my driver is not enough. I also need debug symbols for the whole OVMF image to properly see what's going on. I have a lot of .debug files after OVMF is built with edk2, but I don't understand which ones I need to load into gdb, and what addresses I should use.
I found some instructions involving DebugPkg, but I couldn't make gdb_uefi.py work no matter what. It always failed to locate EFI_SYSTEM_TABLE_POINTER.
In the end, I ended up writing my own script, which implements gdb command that does manage to successfully load all debug symbols. It is probably a worse solution, since it requires a setup: "debug.log" with driver addresses must be present when loading is performed, so you need to run QEMU at least once first. But, this is good enough for me.
My script can be found here:
https://github.com/artem-nefedov/uefi-gdb
I am trying to write a simple OS, I already wrote a bootloader but now I want to debug it, so I switched from using VirtualBox to QEMU because I saw it had better debugging.
The problem is that after I added the -s parameter to QEMU command and successfully connected via GDB, it says that the symbol table isn't loaded and that I should use the "file" command.
The only difference from what I did to what I saw people on the Internet do, is that they started GDB with gdb vmlinux, but I can't do that because I am not debugging a Linux kernel... so I figured that the issue is that I didn't start GDB with an executable, but using the "file" command on my OS image, and the compiled and linked .out file, tells me it's a "DOS/MBR boot sector", so I can't start GDB with either of them (I tried to do that anyways, but GDB failed).
Help would be appreciated.
EDIT: also, I did assemble the bootloader with the -g and --gstabs+ options.
gdb would like a file so that it can give you symbolic debugging information. For that you will need to give it a file in a format with debug info which corresponds to where your OS ends up in RAM. The "DOS/MBR boot sector" file is a disk image (the BIOS will load part of this into RAM for you, and it will then presumably finish loading code itself).
But gdb will also entirely happily let you do assembly-level debugging; you can just ignore the warning about not having a symbol table, and use the single step instruction, disassemble-from-pc and similar commands:
"disas $pc,+32" disassembles 32 bytes from the current PC
the display command prints after execution stops, so "disp /3i $pc" will print the next 3 instructions every time gdb gets control
"stepi" and "nexti" do single-instruction step/next ("step" and "next" are source-line stepping and require debug info)
I am running QEmu on Linux and the virtualized OS is Windows XP. I would like to have either the 1st or the 2nd possibility:
to apply a command in QEmu which creates a dump that can be opened with WinDbg
to apply a command in QEmu which creates a dump of a particular type and then use another utility to make it readable with WinDbg
I know about QEmu's "dump-guest-memory" (creates an ELF-dump), rekall's raw2dmp plugin and Volatility's raw2dmp plugin. The output can be scanned by WinDbg, but in fact you face the problem of a bad header - probably, the most important part of a dump, which is good for nothing when the dump was created using tools/plugins mentioned above. Many fields are lacking in correct details. Comparing to a normal dump created by Windows OS itself (for example, when crashing), which contains information which is full and correct.
I am a student and I got the task to resolve this problem. I think it is reasonable to ensure that it has not been completed yet by anybody so doing it does make sense.
if Qemu Monitor commands are acceptable set the crashonctrlscroll registry key in the guest os
set the dump creation parameters in SystemApplet
Mycomputer->properties->Advanced->Startup and Recovery->Settings->Complete memory dump)
path , no overwrite , uncheck automatically restart
and from Qemu monitor shell use sendkeys ctrl_r-scroll_lock-scroll_lock
that will generate a dump that will be windbg compatible
manually initiated_crash bugcheck code 0xe2
screen shot of a 700 mb ram xp64 guest os running inside qemu on a 32 bit xp32 os
So far, with gdb + qemu, I can step into/over linux kernel source code. Is it possible to debug the user space programs simultaneously? For example, single step a program from user space to kernel space so I can observe the changes of registers on the qemu monitor by issuing info registers?
Minimal step-by-setep setup
Mahouk is right, but here is a fully automated QEMU + Buildroot example which presuposes that you already know how to debug the kernel with QEMU + gdb and a more detailed exaplanation:
readelf -h myexecutable | grep Entry
Gives:
Entry point address: 0x4003a0
So inside GDB we need to do:
add-symbol-file myexecutable 0x4003a0
b main
And only then start the executable in QEMU:
myexecutable
A more reliable way to do that is to set myexecutable as the init process if you can do that.
add-symbol-file is also mentioned at: How to load multiple symbol files in gdb
Why would you ever want to do this instead of gdbserver?
I can only see one use case for this so far: debugging init: Debug init on Qemu using gdb
Otherwise, why not just use the following more reliable method, e.g. to step into a syscall:
start two remote GDBs:
one with qemu-system-* -s
the other gdbserver myexecutable as explained at: https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-mips-elf-with-qemu-toolchain/16214#16214
step in gdbserver's GDB as close as possible to the system call, which often mean stepping into the libc
on the QEMU's GDB, do e.g. b sys_read for the read syscall
back on gdbserver, do continue
I propose this because:
using the QEMU GDB for userland can lead to random jumps as the kernel context switches to another process that uses the same virtual addresses
I was not able to load shared libraries properly without gdbserver: attempting sharedlibrary directly gives:
(gdb) sharedlibrary ../../staging/lib/libc.so.0
No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.
As a consequence, since most kernel interactions go through the stdib, you would need to do a lot of smart assembly stepping to find the kernel entry, which could be impractical.
Until, that is, someone writes a smarter GDB scripts that steps every instruction until a context switch happens or until source become available. I wonder if such scripts would't be too slow, as the naive approach has the overhead of communication to-from GDB for every instruction.
This might get you started: Tell gdb to skip standard files
Parsing Linux kernel data structures
To do userland process debug properly, that's what we would have to do eventually: thread-aware gdb for the Linux kernel
I achieve it by using the gdb command add-symbol-file to add userspace programs debugging information. But you must know these programs loading addresses. so to be precise, you have to launch the kernel debugging by connecting gdb to gdbserver as usual; and then, you can add those program debugging information. You can also use .gdbinit script though. Read this