Background
I am working with an ARM device using a custom-built toolchain (based on Yocto with gcc 4.7 and gdb 7.5) and make use of remote gdb debugging with Eclipse CDT as debugger frontend. Recently I had the issue that I could not debug a particular executable on the remote target due to this error (reported by the gdbserver on the target) that occurred immediately when the host gdb connected to the target:
error while loading shared libraries: unexpected PLT reloc type 0xf0
I could finally track down the issue to mismatching binaries of the dynamic linker library /lib/ld-2.16.so on the target and on the host, where, by calling set sysroot in gdb, I used a locally stored root directory of the target that is generated along with the toolchain.
Keeping the local file in sync with the remote file works, but I could also just omit setting the sysroot in order to debug at least the executable itself. This leads me to the following
Question
How does the usage of a wrong ld.so binary on the debugging host influence the execution of the application within gdbserver on the target? I would rather expect that I get just wrong debug information on the host as the executable runs without problems on the target (within gdbserver) if I have no ld.so present at all on the host. But as the behavior is different, it seems that there is some feedback from the host to the target when the file is available.
How does the usage of a wrong ld.so binary on the debugging host influence the execution of the application within gdbserver on the target?
Good question.
One possible explanation: in order to properly keep track of e.g. loaded shared libraries on the target, GDB sets a number of internal breakpoints (these are visible in maintenance info breakpoints output -- they have negative breakpoint number).
When you don't supply a local file at all, GDB has no idea where to set these breakpoints, and so it doesn't (you can't e.g. debug library initializers without them).
When you supply an incorrect local file, GDB does set the breakpoint ... in the wrong place (by overwriting what GDB thinks is an instruction, but what in reality is a PLT relocation). When the loader then encounters this overwritten relocation, it complains.
Related
I have an C-application which runs on many machines and from time to time an instance makes issues and behaves weird. Unfortunately, this happens almost never. These prod-instances are compiled with heavy optimizations (-march=XXX -Ofast) and do not include any debug-symbols, so I cannot easily attach a debugger to analyze their state.
But I thought it should be possible to compile the application again with the same flags plus -g3 and then I can load it in gdb with symbol-file application_executable_with_debug_symbols. However, if I do that then breakpoints never trigger.
Is there another way to attach a debugger to a running application and loading debug-symbols? Or is there something (obvious) which I do wrong?
Thanks
The best practice is to build the application with debug symbols, keep the resulting binary for debugging, but run strip -g app.debug -o app.release and run the stripped binary in production.
When you find a misbehaving instance, you can copy the full-debug version to target machine, and run gdb -ex 'attach $PID' app.debug. Voila: you have full debug symbols.
The most likely reason that compiling the application again didn't work is that you got a new binary with different symbols (compare nm app.debug vs. nm app.release), and the most likely reason for that (if using GCC) is that you omitted some of the optimization flags used to build app.release, or you used slightly different sources -- you must use exactly the same flags (and add -g) and exactly the same sources for any hope of success with that approach.
I am trying to write a simple OS, I already wrote a bootloader but now I want to debug it, so I switched from using VirtualBox to QEMU because I saw it had better debugging.
The problem is that after I added the -s parameter to QEMU command and successfully connected via GDB, it says that the symbol table isn't loaded and that I should use the "file" command.
The only difference from what I did to what I saw people on the Internet do, is that they started GDB with gdb vmlinux, but I can't do that because I am not debugging a Linux kernel... so I figured that the issue is that I didn't start GDB with an executable, but using the "file" command on my OS image, and the compiled and linked .out file, tells me it's a "DOS/MBR boot sector", so I can't start GDB with either of them (I tried to do that anyways, but GDB failed).
Help would be appreciated.
EDIT: also, I did assemble the bootloader with the -g and --gstabs+ options.
gdb would like a file so that it can give you symbolic debugging information. For that you will need to give it a file in a format with debug info which corresponds to where your OS ends up in RAM. The "DOS/MBR boot sector" file is a disk image (the BIOS will load part of this into RAM for you, and it will then presumably finish loading code itself).
But gdb will also entirely happily let you do assembly-level debugging; you can just ignore the warning about not having a symbol table, and use the single step instruction, disassemble-from-pc and similar commands:
"disas $pc,+32" disassembles 32 bytes from the current PC
the display command prints after execution stops, so "disp /3i $pc" will print the next 3 instructions every time gdb gets control
"stepi" and "nexti" do single-instruction step/next ("step" and "next" are source-line stepping and require debug info)
I'm doing an exploit type challenge. The challenge's binary to exploit uses a modified version of libc.so.
So before running linux server or radare2 / ida, the export environment variable LD_PRELOAD=/path2/libc_modified.so and then I put a breakpoint at the first assembly statement of the main. With ida I cannot launch binary however I can attach to process, but the breakpoints inserted after a read are bypassed so ida cannot stop and I can't debug binary.
With radare2 I can launch binary but breakpoint inserted at first instruction of main are bypassed.
I can debug executables that doesn't use libraries to be loaded with the LD_PRELOAD and my breakpoints are catched.
Does anyone know the cause of this behavior?
I have recently started windows driver development. I am wondering how does it actually debug my driver. The setup I have is --> Win7 as host, XP as guest on VMware, and I am debugging through serial port.
The research I have done:
I found only this link saying very few things that I am talking about.
I already know how debugger works on single OS, in that case debugger is also on the same OS, so it knows which process is running. That is understandable. But here, debugger is on entirely different OS, an entirely different environment. I just say file->open source files and I AM able to put breakpoints!! Moreover when I load driver, it actually breaks there. I mean why../How? How does XP's kernel comes to know(drivers are extension to kernel, atleast WDM, don't know about WDK) that there is source code of this driver? and that also outside its control(environment)? I mean I can have 10 files open with breakpoint in them, but it works beautifully, I am not able to fail/fool it.
So what I am thinking is like, whenever we add source to windbg on Win7, it creates the binary from that source, and whenever XP is going to load any binary, it checks if this is the binary that windbg is waiting for. what is confusing in above link is, Vikrant is saying that debugger asks kernel(XP) that it is willing to debug a process --> Bus HELLO... process is running on XP, and windbg on Win7 and does not know name or id of process. It has source code, but consider a case where there is a driver which is build out of 300 files, and just one, probably simplest file is open in windbg, how it matched that this source code is of the driver being run?
#Kjelll answer is correct. Here is the full scenario, including explanation to your comment:
PDB files have line information. This is a mapping from each (file,line) location to address (RVA - relative virtual address).
When you set a break point on a source file, WinDBG checks whether this source file correspond to a current address. If it is - it sets the breakpoint. Otherwise, it becomes a "future breakpoint" (not sure whether Microsoft uses this terminology).
When a new binary is a loaded, the agent on the client communicates with the host, informs it about the binary. At this point - WinDBG will try to allocate a PDB file.
WinDBG will start at the PDB location embedded within the file. You can see this value by using this command line: windbg -dump -pdbpath xxx.sys. This should explain how WinDBG will find a symbol file even if not on the .sympathy path (that I believe answers your comment to Kjell).
WinDBG will then search at the .sympathy.
Once symbol is find, it will look at all future breakpoint, and if applicable will set an actual breakpoint.
I think your link explain your question pretty well, but you have probably not realized what the mechanism of the pdb do for the debugger. The windbg on your host OS uses the pdb file to translate line nubers in the source files to addresses in your guest OS (xp) . Then the the debugger agent uses this address to set break points (Int 3) in the guest OS.This is much in the same way as a local debugger do to a local process.
I am using Qemu to learn some linux kernel development/hacking and wanted to debug the boot process of Linux (2.6.34.3). I have compiled for the ARM versatile platform and is using Codesourcerys arm-none-eabi crosscompiler. I am using Eclipse as the environment to build and debug using gdbserver.
So I have manged to successfully build and run the kernel in qemu but the problem is that I dont see any source code in the debugger at the boot process(at address 0), I can only see the disassembly code. However, when it switches to virtual memory at init/main.c (address over 0xC0000000), the source code appears and I can see the source code and step through and over code. Why is that? I want that from the beginning.
Anyone have any tips on how to debug the boot process of Linux? All the guides in google shows how to debug the kernel, but they all show from start_kernel() (located in init/main.c) and not from the beginning of the boot process (in arch/arm/boot/compressed/head.S). Anyone with experience help please, thank you!
Looked into the System.map in the root folder and there is only symbols for stuff from c0004000 (where the virtual address start). I load vmlinux into gdbserver to get debug information, Maybe thats why theres no source?
The Linux kernel uses a 2-step booting processing (and this does not include any boot loader like u-Boot ...). You can better understand this especially by looking into 2 .lds files (detailed below) for linking:
arch/arm/boot/compressed/vmlinux.lds.in, which generates arch/arm/boot/compressed/vmlinux.lds.
Along with other .o files in arch/arm/boot/compressed, a vmlinux is generated inside this folder.
You can use arm-none-eabi-nm -a -n arch/arm/boot/compressed/vmlinux to see the symbols for this stage. All addresses are physical addresses.
These symbols are NOT included in System.map
The second vmlinux is generated by kernel .o files and arch/arm/kernel/vmlinux.lds (note: the path is different)
I hope this explains why you can not see the booting source code in Eclipse.
linux kernel is too complex to understand(for a beginner).
Why dont use use a smaller OS like xv6:
OS is small, sourcecode is about 8000 lines
used by many universities
based on V6(unix),
boot process is the same except that its less complicated than that of linux.
Appendix B of the xv6 book deals with boot process(its short and sweet).You can run gdb on qemu and see the boot process, the main files to check out for are bootasm.S(in assembler) and bootmain.c.
This is much simpler and easier to do and understand when compared to linux.(atleast for beginners).There are assingmennts on , setting up qemu , using gdb ,tracking the boot process , doing changes to the source code etc in the link given.Give it a shot :)
Cheers,
sharan
head.S is written in assembly, not C. That's what the .S suffix indicates.