I would like to know if there is indeed a way to toggle between gdb and ddb while debugging a remote kernel.
I am already at the gdb (or rather kgdb) prompt. From here how do I switch to local ddb on the debugged machine?
My kernel configuration file already contains options BREAK_TO_DEBUGGER and I have BOTH GDB and DDB configured aka:
options GDB
options DDB
As per the developer's handbook, "Every time you type gdb, the mode will be toggled between remote GDB and local DDB. In order to force a next trap immediately, simply type s (step). Your hosting GDB will now gain control over the target kernel:"
So, I did try typing 'gdb' at the gdb prompt (funny me:D) and as expected, it was an unrecognized command. Obviously, this command is supposed to be typed at the ddb prompt. But my question is, how do I drop to ddb from within a running machine whose serial ports (albeit virtual ones) are remotely attached to another machine's KGDB? When remote GDB is listening and I force a panic using sysctl debug.kdb.enter=1, it does drop into remote KGDB. However, when it is NOT listening, the system just freezes.
What I want, is to enter ddb on the local machine. Do some debugging using it; drop to remote KGDB for things that are best done using KGDB, then switch back to local DDB when I'm done.
Is there a way to do that?
KDB contains DDB & KDB backends, but there are a lot of conditions when they are available. To check if backend is available, check sysctl debug.kdb.available and debug.kdb.current (sys/kern/subr_kdb.c). If both backends are available, debug.kdb.available should contains "ddb gdb".
The possible way to toggle between backends (gdb / ddb) is to enter ddb at first, then call gdb and make debugging. Then exit gdb, and return to ddb (actually new trap will happen, seems Ctrl+C is required). It means that before panic, debug.kdb.current should be set to "ddb".
I hope it will help.
Related
Working on virtual machine environment, I'm using DbgView to obtain my printout log messages using debug_wsting
However, Working on large number of processes that produces those messages seem to consume lot to cpu and eventually halt the VM, So I'm looking of a way to offload this activity to the machine that runs this VM.
Perhaps there's a way to output the logs to remote windbg ?
Is there anything to configure on Client/Server ?
thanks
I never tried it on my own, but it's described in Windows Sysinternals Administrator's reference (Amazon). I thought I knew the Sysinternals tools quite well, but it turned aout I learned a lot. Totally worth reading.
To perform remote monitoring, DebugView runs in agent mode on the remote system [...]
To do so, run DbgView /a. Add /k if you need kernel messages as well and /gfor session 0.
[...] sending debug output it captures to a central DebugView viewer that displays the output.
and
To begin remote monitoring, press Ctrl+R or choose Connect from the Computer menu [...]
Be careful not to connect multiple viewers to a single computer because the debug output will be split between those viewers.
So, the output is there only once and after whatever machine fetches it, it's gone. Sounds normal.
There are lots of pages that explain it but I can't find it. Many of the articles I find only work on El Capitan and older systems.
I cannot use the fwkpfv right now as I don't have the right dongles. My client is getting me a used MacBook that will support firewire.
My kernel extension panics my box. Quite oddly if my coworker builds my extension, it works just fine. I remain flummoxed.
You can get "live" local kernel logs using the command
log stream --process 0
For looking at past logs, use log show instead, e.g.:
log show --predicate 'processID == 0' --last 1h | less
None of that will help you much with kernel panics, however, as the logging happens asynchronously in user space, so you won't get the very last messages before the panic.
A few more options for debugging KPs without firewire, which you're probably already aware of but I'll mention them for completeness' sake:
Ethernet-based kernel debugging (as opposed to firewire). Only the test device needs wired/thunderbolt ethernet, the Mac running the debugger can be on wifi.
You can often extract quite a lot of info from the panic log itself: in addition to symbolicating the stack (use keepsyms=1 boot-arg so you don't have to do it retroactively), looking at the register contents and disassembly can often tell you the values of variables.
If you're missing parts of Apple's code the stack trace, run a debug or development kernel instead of the release one. Those are built with fewer optimisations enabled, so functions are less likely to be inlined, etc.
There are a bunch of memory debugging and other diagnostic options you can turn on in the kernel, e.g. -zp, -zc and so on.
If you can repro the crash in a VM (VMWare Fusion, Parallels, VirtualBox, KVM/Qemu, whatever), you can use the VM's simulated serial port to log kprintf output. The virtual ethernet ports also tend to support kernel debugging if you set them up right.
I'm using gdb to communicate with a LEON2-based ASIC through a home made gdb server (not sure though that "gdb server" is the correct phrase here). It works like this: the gdb client uses the ordinary gdb protocol to talk to the gdb server, which then translates the gdb requests to reads and writes from/to the HW and sends back the result to the client, if any. My gdb client is sparc-rtems-gdb 6.6 in RTEMS 4.8.0 on a Windows 7 PC.
When I start the gdb client I run the following command to attach to the gdb server:
target extended-remote localhost:5000
Then I want to change a word in RAM so I run this gdb command:
set *((unsigned int*) 0x40000000)=2
While debugging the gdb server I can see that it receives the following line, which is expected and correct according to the gdb protocol, i.e. writing 4 bytes, value 2 to address 0x40000000:
M40000000,4:00000002
Now the confusion: After the write request above, another request comes from the gdb client, read 4 bytes from address 0xABD37787:
mabd37787,4
Why is the gdb client trying to read from that address? As far as I know, I haven't done anything to request this read, I only wanted to perform the write. If gdb would have read back the address 0x40000000, for example to verify the write, it would be OK. But the out-of-nowhere-address 0xABD37787 does not exist on my HW, which causes problems for me.
Is there any way that I can debug the gdb client to determine exactly what (and why) it is sending and receiving? Or is there a setting in gdb that can explain this behaviour?
Best regards
Henrik
While debugging the gdb server I can see that it receives the following line
You don't need to be debugging the gdbserver. You can simply turn on set debug remote 1 in GDB, and have GDB print all sent and received packets.
Why is the gdb client trying to read from that address?
There are several possibilities:
GDB believes that program counter is currently at 0xABD37787
GDB believes that it needs to set a breakpoint there
GDB believes that there is some data that it needs to read
One possible way to figure out why GDB is trying to read that location is to set debug infrun 1. This will print a lot of info about what GDB itself is trying to do.
Another way is to debug GDB itself. Put a breakpoint on putpkt, and when the packet of interest is being sent, examine the stacktrace to see why it is being sent.
I ssh into a linux VM which is setup remotely. I use Vim to write my code. For debugging however, I use netbeans through X11 which can sometimes be painfully slow. I tried using gdb buts its an efficiency killer. I love to hover over my variable and get to now their value rather that doing p variable_name , plus I like see and navigate through the code. Is there something light simple gui based debugging tool I can use. I have tried to use clewn http://clewn.sourceforge.net/ , but that doesnt work because it has a missing netbeans_intg feature. Is there any other similar vim gui based debugging tool ?
You can try ddd
which is a gui for gdb, I think it's lighter than netbeans.
cgdb is an interface to gdb but it is not a graphical one. It does not offer the possibility of hovering over a variable, but it shows you a window with the source code.
Well, I was in sort of your situation sometime ago, and you can have a look at my question about using gdb with remote sources.
First of all, your problem with netbeans_intg feature is related to vim which has been compiled with no support for it. If you can rebuild vim yourself, you can then enable it. Otherwise, as you can see in the answer that I gave myself to my question, you can leverage clewn's remote-vim capabilities.
In a nutshell, you can have a "local" vim (i.e. on a desktop/laptop machine presumably), which must still be built with netbeans_intg support, but now it is a vim under your complete control (i.e. it's on "your" machine), while clewn will run on the linux host where gdb and your debuggee will run.
You can then keep the source files on your desktop/laptop and have the remote clewn sort of "drive" your local vim to the proper source files while debugging.
IOW: clewn will get information out of gdb to know exactly which file/line you're into and connect to remote vim and tell it: "hey, go grab this file and show it around this line", highlighting current line, breakpoints etc.
This is a great solution for when you have far-away deployed systems and you need to debug them with minimum impact on the host where they are running, and presumably no option to transfer there all of your source files.
I don't know if this fits in any way with what you're trying to do, but it did really change things for me.
Hth,
Andrea.
Check out GDB server. Theoretcially, you should be able to start gdb on your linux machine in server mode and connect via GUI of your choice. As long as that GUI supports remote gdb connections, which Netbeans does.
What is your favorite technique for launching a windbg user-mode remote debugging session?
Why is do you prefer this technique over other techniques? (pros/cons)
There are at least four different ways to do user-mode remote debug using windbg, as documented in the "Remote Debugging" section of the debugging tools for windows help file.
run app on target then attach to it from the host windbg
have the host windbg use remote.exe to launch the app on the target
have the "smart client" host windbg launch the app on the target via a process server that is running on the target
run a windbg instance on the target machine using the option "-server" to automatically start a server, then connect to the server from a 2nd machine.
Option 1 is my favourite because it is the simplest. I get to launch the app in the normal way without worry about getting WinDbg to set the right working directory, pass any command line arguments, etc.
Fortunately I've not run into any cases where this hasn't worked!
There is no "the best" solution. Each of the possibilities has advantages and disadvantages and it's good to understand all of them. It depends on several factors like:
where are the symbols located
which PC has access to the Internet to download the OS symbols
what amount of data may you copy to the server (clients often accept better if it's just a single Exe)
what's the bandwidth between client and server
do you need other commands that just CDB/WinDbg, e.g. access to CMD, then consider remote.exe
who's available on the server side, a debugging expert whom you can easily tell a lot of cryptic commands or a normal user who barely knows how to start a command prompt
are both sides in a private network, so you need a "man in the middle" server to be able to access each other (or port forwarding as an alternative, which IT guys don't want and it may take days to get it set up)
From those 4 options, don't forget that clients often want to see exactly what you do, so they require an RDP session, Teamviewer or similar. That's something they understand.
I tend to use option 4 (-server) because it is the only one that doesn't "pop" when you break into the kernel debugger long enough for the TCP connection to timeout. But this is more complex and not fully satisfying. So I'm looking for "best practices".