I'm just cross posting the same question I did on virtualbox.org. http://forums.virtualbox.org/viewtopic.php?f=9&t=26702&p=119139#p119139
If not breaking any rule, I'd appreciate to kwon more about it since stackoverflow promisses to be more dynamic!
"Hi,
I did some search and could not find any tool to debug a guest system from the early boot in virtual box. Although, I came across JCP, a x86 emulator in java that is not so powerful and beautyful but has a debug mode where one can view the Physical Memory, the CPU registers along other things. It also makes it possible to execute CPU instructions step by step and set break points, watchpoints and conditional ones. There's such thing in Virtual Box?
I think would be amazing to have it and be able to inspect the system while its running. For learn about PC architecture or as a tool to develop a kernel as well.
In the case you think its good idea (I think it is) how can it be achieved? I'm interested in develop such sort o things and would like to know if it is feasible if not already implemented somewhere."
EDT: Are modern x86 able to interrupt its execution just after a cpu cycle and pass execution addres to another code to just do this? Yes, the trap flag can be set to put the processor in step by step execution mode. x86 will execute one instruction and call INT 3.
Contrary to what is stated above, VirtualBox now contains a (limited) debugger. Add --dbg to the command line when starting the VM. For more information consult:
12.1.3. The built-in VM debugger
The OSDev wiki has some useful information on debugging a guest operating system, though according to this page VirtualBox doesn't have a debugger at present. I've been using QEmu with the GDB stub and it works quite nicely, so you might like to give that a go instead.
Related
I was wondering if there was any way that I could debug a floppy disk emulation in Linux.
The main thing I want to be able to do is to see the values of registers and custom defined bytes and words as the emulation runs.
Another thing I would like to be able to do is to run the emulation one step at a time, and see what line of code my emulation is currently on.
I am currently just running my floppy disk image under qemu-system-x86_64 and letting it run without any feedback besides the emulation.
If anyone can supply me with something along the lines of command line statements that accomplish this sort of thing, and what programs I could either move to or install alongside to help me out, it would be much appreciated.
I don't have enough idea of qemu because I haven't used it much. But from your requirement perspective, I think you should try bochs emulator. It's quite easy to use and comes with a built-in debugger. Only thing is, you need to compile it from source (if on Linux) making sure the --enable-debug and --enable-disasm switches are enabled (alongwith any other options you might want to enable). (On Windows however, the debugger comes as a pre-built (separate) executable in the installation, but that's sort of irrelevant in your case I guess.)
I am using gdb attached to a serial port of a virtual machine to debug linux kernel.
I am wondering, if there is any patches/plugins which can make the gdb understand some of linux kernel's data structure and make it "thread aware"?
By that I mean under gdb I can see how many kernel threads are there, their status, and for each thread, their stack information.
libvmi
https://github.com/libvmi/libvmi
This project does "LibVMI: Simplified Virtual Machine Introspection" which sounds really close.
This project in particular https://github.com/Wenzel/pyvmidbg uses libvmi and features a demo video of debugging a Windows userland application form inside it, without memory conflicts.
As of May 2019, there are two limitations however as of May 2019, both of which could be overcome with some work: https://github.com/Wenzel/pyvmidbg/issues/24
Linux memory parsing is not yet complete
requires Xen
The developer of that project also answered further at: https://stackoverflow.com/a/56369454/895245
Implementing it with those libraries would be in my opinion the best way to achieve this goal today.
Linaro lkd-python
First, this Linaro page claims to have a working setup: https://wiki.linaro.org/LandingTeams/ST/GDB that allows you to do usual thread operations such as thread, bt, etc., but it relies on a GDB fork. I will test it out later. In 2016, https://youtu.be/pqn5hIrz3A8 says that the implementation was in C, not as Python scripts unfortunately, which would be better and avoid forking. The sketch for lkd-python can be found at: https://git.linaro.org/people/lee.jones/kieran.bingham/binutils-gdb.git/log/?h=lkd-python
Linux kernel in-tree GDB scripts + my brain
I then tried to see what I could do with the kernel in-tree Python scripts at v4.17 + some manual intervention as a prototype, but didn't quite get there yet.
I have tested using this highly automated QEMU + Buildroot setup.
First follow the procedure I described at: How to debug the Linux kernel with GDB and QEMU? to get GDB working.
Then, as described at: How to debug Linux kernel modules with QEMU? run GDB with:
gdb -ex add-auto-load-safe-path /full/path/to/linux/kernel
This loads the in-tree GDB Python scripts from scripts/gdb.
One of those scripts provides:
lx-ps
which lists all threads with format:
0xffff88000ed08000 1 init
0xffff88000ed08ac0 2 kthreadd
The first field is the address of the task_struct struct, so we can see the entire struct with:
p (struct task_struct)*0xffff88000ed08000
which should in theory allow us to get any information we want about the process.
Now I wanted to find the PC. For ARM, I've seen: Find program counter of process in kernel and I tried:
task_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))->uregs[ARM_pc]
but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: How do I print a #defined constant in GDB? which are apparently not set?
I don't think GDB understands kernel data structures, that would make them version dependent. GDB uses ptrace for gathering information on any running process.
That's all I know :(
pyvmidbg developer here.
I will add some clarifications:
yes the goal of the project is indeed to have a cross-platform, guest-aware GDB stub.
Most of the implementation is already done for Windows, where we are aware of processes and their threads context.
It's possible to intercept a specific process (cmd.exe in the demo) and singlestep its execution (this is limited to 1 process with 1 thread for now), as well as attaching to a new process's entrypoint.
Regarding Linux, I looked at the internals and the resources that I could find, but I'm lacking the whole picture to figure out how I can:
- intercept a task when it's being scheduled (core/sched.c:switch_to() ?)
- read the task state (Windows's KTRAP_FRAME equivalent for Linux ?)
I asked a question on SO, but nobody answered :/
Linux context switch internals: how does a process goes back to userland after the switch?
If you can help with this, I can guide you through the implementation :)
Regarding the hypervisor support, only Xen is fully supported in the Libvmi interface at the moment.
I added a section in the README to describe where we are in terms of VMI APIs with other hypervisors.
Thanks !
hi :
i was building kernel for my gentoo linux . when i start the kernel , i
got this message , and can't going on.
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
non-volatile memory driver v1.3
i don't know how to solve this problem . and i need help . thanks .
Why don't you try to disable pci hotplug support in kernel (if I recall correctly is in main config menu / PCI support)? You probably don't need this.
I'm going to have to disagree with those voting to close, because I think there really is a question here, and the question is "How to debug this?"
I'm going to propose two approaches:
1) Studious approach: Learn about mechanisms intended for handling boot problems. See if you can increase the kernel debug message level. Disable un-needed drivers as Quizzo suggested.
2) Cowboy approach: grep the kernel sources for strings seen in the final messages, and start shotgunning all possibly relevant bits of code with your own "still alive at" printk messages. Once you know where it's hanging, figure out why and either remove that mechanism or fix it.
At an extreme there's also a tool for debugging the kernel - kgdb - which you could set up if you have a second machine available.
If you already have linux running on this box, see if there's a config.gz in /proc or in a boot folder which you can extract and compare to the configuration you are trying to compile. It might not be a bad idea to first recompile and test exactly the same version and configuration as you have running, and then make desired changes one by one.
Also you might see if there's odd hardware in your system you could temporarily remove. For example, an older PC I have has a bios that hangs during drive enumeration if I have a large USB external drive plugged in during boot.
i have solved the problem by enable all pci hotplug flag in kernel config file.
thinks all.
I understand how powerful windbg can be at debugging, but when is an appropriate or best time to use it to debug an issue? Is it more issue specific, or just experience, intuition, and knowing that using it can just get the job done best?
It's a little bit of all those things, and a lot of personal perference. Many WinDbg people only use WinDbg so that's what they are best at debugging with.
WinDbg also has some good extensions out there like SOS. So a particular extension might provide you with the specific piece of information that another debugger does not.
One reason to use a different debugger in certain circumstances is if you believe the debugger is incorrect. This is rare of course. For things like stack walking for instance, the debuggers use different methods, so you can confirm the stack is what you expect by using the other.
So sum up, for most issues it doesn't matter. It's whatever you are best at using. For some particular issues it's what you say, knowing which tool is the best for that particular issue.
While Windbg is also a fine tool for user-mode debugging, if you end up doing kernel-mode debugging it is really the only serious choice.
The kernel-mode debugging scenario often involves two machines, a debugger and a debuggee. You will be running Windbg on a debugger machine which is connected to the debuggee over a serial connection, Firewire or USB. In this scenario you can "freeze" the target machine and have full control over everything running on it. Often your debuggee (the target) will be a virtual machine running under VMWare or similar -- in this case the connection also typically uses virtual serial ports.
Here are instructions from VMWare on how to set up kernel debugging of a virtual machine:
http://www.vmware.com/support/ws5/doc/ws_devices_serial_advanced_example_debugging.html
You can also use VirtualKD which makes the setup easier and the connection much faster:
http://virtualkd.sysprogs.org/
You can also use Windbg for local kernel debugging. In this case, you only have a single machine where you connect Windbg to the running kernel. You cannot "freeze" the machine, as it would also freeze Windbg running on the same machine, but you can analyze the contents of memory and so on.
Good point. Another good solution for virtual kernel debugging is LiveKd from sysinternals.
http://technet.microsoft.com/en-us/sysinternals/bb897415.aspx
I have compiled my own Kernel module and now I would like to be able to load it
into the GNU Debugger GDB. I did this once, a year ago or so to have a look
at the memory layout. It worked fine then, but of course I was too silly to
write down the single steps I took to accomplish this... Can anyone enlighten
me or point me to a good tutorial?
Thank you so much
For kernels > 2.6.26 (i.e. after May 2008), the preferred way is probably to use "kgdb light" (not to be confused with its ancestor kgdb, available as a set of kernel patches).
"kgdb light" is now part of the kernel (in by default in current Ubuntu kernels, for instance), and it's capabilities are improving fast (Jason Wessel is working on it - possible google key).
Drawback: You need two machines, the one you're debugging and the development machine (host) where gdb runs. Currently, those two machines can only be linked through a serial link.
kgdb runs in the target machine where it handles the breakpoints, stepping, etc. and the remote debugging protocol use to talk with the development machine.
gdb runs in the development machine where it handles the user interface.
An USB-to-serial adapter works OK on the development machine, but currently, you need a real UART on the target machine - and that's not so frequent anymore on recent hardware.
The (terse) kgdb documentation is in the kernel sources, in
Documentation/DocBook
I suggest you google around for "kgdb light" for the complete story.
Again, don't confuse kgdb and kgdb light, they come together in google searches but are mostly different animals. In particular, info from linsyssoft.com relate to the "ancestor" kgdb, so try queries like:
kgdb module debugging -"linsyssoft.com" -site:linsyssoft.com
and discard articles prior to May 2008 / 2.6.26 kernel.
Finally, for module debugging, you need to manually load the module symbols in the dev machine for all the code and sections you are interested in. That's a bit too long to address here, but some clues there, there and there.
Bottom line is, kgdb is a very welcome improvement but don't expect this trip to be as easy as running gdb in user mode. Yet. :)
It has been a while since I was actively developing drivers for Linux, so maybe my answer is a bit out of date. I would say you cannot use GDB. If at all, only to debug post mortem on dump files. To debug you should rather use a kernel debugger. Build the kernel with a kernel debugger enabled (there is one out-of-the box debugger for 2.6, which was lacking at the time I was active). I used the kernel patches for KDB from Sun ftp://oss.sgi.com/www/projects/kdb/download/, which I was quite happy with. A user space tool won't be of much use unless new gdb communicate somehow with the internal kernel debugger (which anyway you would have to activate)
I hope this gives you at least some hints, while not being a detailled answer. Better than no answer at all. Regards.
I suspect what you did was
gdb /boot/vmlinux /proc/kcore
Of course you can't actually do any debugging, but it's certainly good enough to have a poke around the kernel.