I want to trace a kernel module I've written using Intel PT but I can not get perf to recognize symbols from my kernel modules. For the sake of simplicity, I tried tracing a module that periodically prints a string to the log, using perf record -e intel_pt// -a --filter 'filter print_hello' sleep 1. This results in the following error:
Kernel symbol lookup: Symbol 'print_hello' not found.
Note that symbols must be functions.
Failed to parse address filter: 'filter print_hello'
Filter format is: filter|start|stop|tracestop <start symbol or address> [/ <end symbol or size>] [#<file name>]
Where multiple filters are separated by space or comma.
Recording without a filter using perf record -a -e intel_pt//k sleep 1 and then grep'ing the perf script output for print_hello does not return anything either.
However, perf kallsyms print_hello returns
print_hello: [hello_periodic] /lib/modules/5.4.161/extra/hello-periodic.ko 0xffffffffc07af07c-0xffffffffc07af0b6 (0x7c-0xb6)
so I assume perf can find the symbol after all.
Why could this happen?
A workaround is possible:
perf record -e intel_pt// -a "$(printf 'filter print_hello\t[hello_periodic]')" sleep 1
Or just use numbers:
perf record -e intel_pt// -a --filter 'filter 0xffffffffc07af07c/0x3a' sleep 1
However, at least kernel v5.18 is probably needed because it has commit c243cecb58e3 ("perf/x86/intel/pt: Relax address filter validation")
For perf help, also try the linux perf users mailing list:
linux-perf-users#vger.kernel.org
and mail archive at:
https://lore.kernel.org/linux-perf-users/
For general information, the kernel perf wiki:
https://perf.wiki.kernel.org
And for Intel PT, the Intel PT wiki page:
https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace
I have been trying to get a w3wp crash dump to see the crash callstack. I got two dumps but both of them have a single thread in them - seems almost like AppDomain has been recycled already and there is nothing useful left in the process when the dump was saved.
Command used: "procdump -mm -e -n 1 -l pt <PID>"
Also tried -ma for full dump but result is the same:
0:000> ~
. 0 Id: fa4.1dc8 Suspend: -1 Teb: 000000b1`77a78000 Unfrozen
I am not sure if I am missing something in the command, or IIS does not provide usable managed dumps when capturing them with procdump - any inputs are highly appreciated!
Additional detail: I was seeing STACK_OVERFLOW exception being logged by the procdump, which - apparently needs different method to capture useful dump. See my own answer below for details.
It only took few hours - hopefully this will save some time for others like me.
Found a way to do this:
procdump -mm -e 1 -l -f C00000FD.STACK_OVERFLOW -g <PID>
It works! Thanks to the unknown fellow member whose hint lead me to this. I was reading too many pages and missed saving the page link to post acknowledgement here.
I am writing a bpf filter to prevent certain netlink messages. I am trying to debug the bpf code. Is there any debug tool that could help me?
I was initially thinking of using nlmon to capture netlink messages:
From https://jvns.ca/blog/2017/09/03/debugging-netlink-requests/
# create the network interface
sudo ip link add nlmon0 type nlmon
sudo ip link set dev nlmon0 up
sudo tcpdump -i nlmon0 -w netlink.pcap # capture your packets
Then use ./bpf_dbg (
https://github.com/cloudflare/bpftools/blob/master/linux_tools/bpf_dbg.c)
1) ./bpf_dbg to enter the shell (shell cmds denoted with '>'):
2) > load bpf 6,40 0 0 12,21 0 3 20... (this is the bpf code I intend to debug)
3) > load pcap netlink.pcap
4) > run /disassemble/dump/quit (self-explanatory)
5) > breakpoint 2 (sets bp at loaded BPF insns 2, do run then;
multiple bps can be set, of course, a call to breakpoint
w/o args shows currently loaded bps, breakpoint reset for
resetting all breakpoints)
6) > select 3 (run etc will start from the 3rd packet in the pcap)
7) > step [-, +] (performs single stepping through the BPF)
Did anyone try this before?
Also, I was not able to make nlmon module to load on my linux kernel(Is there a doc for this?)
I am running kernel version Linux version 4.10.0-40-generic
The nlmon module seems to be present in the kernel source:
https://elixir.free-electrons.com/linux/v4.10/source/drivers/net/nlmon.c#L41
But, when I search inside, /lib/modules/ for nlmon.ko I dont find anything.
instance-1:/lib/modules$ find . | grep -i nlmon
instance-1:/lib/modules$
I have a virtual Windows 7 x64 machine on a Windows 10 host, and I kernel debug it with windbg 10.0.10586.567. I'm running my own application on it, which I have full source and private symbols for. Whenever I break in and ask for stack traces of the app's threads, the backtrace always stops when one of my application's binaries are "hit."
So for instance, if I break in, switch to the process, and request a stacktrace with !thread [thread address] 1f, I get something like this (note the "early" zero return address at the last line):
fffff880`0534e870 fffff800`026d6992 nt!KiSwapContext+0x7a
fffff880`0534e9b0 fffff800`026d81a2 nt!KiCommitThreadWait+0x1d2
fffff880`0534ea40 fffff800`029c7a2e nt!KeDelayExecutionThread+0x186
fffff880`0534eab0 fffff800`026d08d3 nt!NtDelayExecution+0x59
fffff880`0534eae0 00000000`76e7165a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame # fffff880`0534eae0)
00000000`00276708 000007fe`fcf91203 ntdll!NtDelayExecution+0xa
00000000`00276710 00000001`410e7dd9 KERNELBASE!SleepEx+0xab
00000000`002767b0 00000000`00000000 MyApp!MainMessageLoop+0x4b1 [d:\whatever\path\myapplication.cpp # 3024]
This looks very similar to when you you are missing a binary while debugging a user-mode dump (lack of unwind data) of an x64 process, except in that case the stack trace usually does not stop "this sudden", rather it goes astray at that point, and shows bogus values.
Some extra info/things I tried:
I have the correct symbol paths set up (both the Microsoft symbol server, and a local folder on the host with matching PDBs, even though the latter is not needed for just the stack trace)
I have a binary path set up (.exepath) containing matching binaries on the host (I've made absolutely sure of this; copied the binaries directly from the guest to the host machine)
If I put a breakpoint in one of the app's exported DLL functions, then when the debugger breaks in, I get a one-liner stack trace like this: 0000000000274b40 0000000000000000 MyAppDLL!SomeExportedFunction+0x32 [d:\whatever\path\myapplicationDLL.cpp # 232]
I've tried virtually every combination of commands to get a stacktrace (.process /i, .process /r /p, !process -1 7, .reloads, .reload /users, .reload /f MyApp.exe, !thread [address] 1f, etc.) with no success
Tried with an older version of windbg (6.11.0001.404) as well, same result
Also tried on Windows 8.1 as a guest with the very same binaries, same result
!sym noisy output (irrelevant lines omitted):
0: kd>.process /i [address]
0: kd>g
0: kd>.reload /user
0: kd> !process -1 2
0: kd> !thread [address] 1f
[...]
DBGHELP: d:\symbolcache\MyApp.pdb\76931C5A6C284779AD2F916CA324617E1\MyApp.pdb already cached
DBGHELP: MyApp - private symbols & lines
[...]
lmvm MyApp output:
[...]
Loaded symbol image file: MyApp.exe
Image path: C:\MyApp\MyApp.exe
[...]
Any ideas?
I accidentally stumbled into a linker switch that solves this problem: /DEBUGTYPE with the PDATA argument. If you link your binaries with this switch, unwind information will be copied into your PDBs.
I recompiled/relinked the application in question with /DEBUGTYPE:CV,PDATA (/DEBUGTYPE:CV is the default if /DEBUG is specified, see the documentation), now everything works like a charm, I always get full call stacks.
One strange aspect of this: windbg happily uses unwind data found in the PDBs, but ignores the very same data in the mapped binaries (both on the host machine).
This is not a perfect solution to the problem (or any solution at all, one might say), but I'm providing this provisional answer with a workaround.
You should be able to get the information you want, albeit not so well-formatted using something like dps #rsp L10.
In x86-64 you don't have a parallel of the x86 ebp-chain, but the return addresses are still on the stack. Those will give you the functions in the stack, and the values between them will be the arguments passed to the functions (and saved registers on the stack, etc.). A random example from Google (as I'm not on my Windows machine right now):
0:017> dps #rsp
00000000`1bb0fbb8 00000000`00000020
00000000`1bb0fbc0 00000000`00000000
00000000`1bb0fbc8 00000000`008bc6c6 Dolphin!ReadDataFromFifoOnCPU+0xb6 [d:\sources\comex\source\core\videocommon\fifo.cpp # 245]
00000000`1bb0fbd0 00000000`1ba0ffeb
00000000`1bb0fbd8 00000000`00000020
00000000`1bb0fbe0 00000000`00000020
00000000`1bb0fbe8 00000000`00000800
00000000`1bb0fbf0 00000000`1ba0ffeb
00000000`1bb0fbf8 00000000`008c2ff5 Dolphin!InterpretDisplayListPreprocess+0x45 [d:\sources\comex\source\core\videocommon\opcodedecoding.cpp # 87]
00000000`1bb0fc00 00000000`00000000
00000000`1bb0fc08 00000000`008bc041 Dolphin!RunGpu+0x81 [d:\sources\comex\source\core\videocommon\fifo.cpp # 389]
00000000`1bb0fc10 00000000`8064cbc0
00000000`1bb0fc18 00000000`1bb0fcc0
00000000`1bb0fc20 00000000`00000000
00000000`1bb0fc28 00000000`008c2dda Dolphin!OpcodeDecoder_Preprocess+0x14a [d:\sources\comex\source\core\videocommon\opcodedecoding.cpp # 326]
00000000`1bb0fc30 00000000`8064cbe0
Given that you have symbols, the return addresses are easily distinguishable.
The unwind data is lazy loaded for user mode modules, so it's not going to be mapped unless someone needs it. Unfortunately the kernel debugger doesn't force the information to be present for user images, so sometimes you get this behavior. You can see if the data is mapped or not by dumping the PE header (!dh) and checking the state of the Exception Directory (!pte imagename+offset).
Given that you own the app, try forcing the information to be resident by doing a stack walk NOP somewhere in your app:
PVOID stack[2];
(VOID)CaptureStackBackTrace(0, 2, (PVOID*)&stack, NULL);
That doesn't guarantee the entire directory will be present, but usually good enough.
I would like to debug a user defined function (called prepareTheOutputRecord implemented in C/C++ that is a part of user defined function in postgres. Here's how I achieve this with gdb:
The function prepareTheOutputRecord resides in libMyExtenstion.so file in the lib directory of postgresql server
I start the psql shell, retrieve the pid of the process
postgres=# SELECT pg_backend_pid();
pg_backend_pid
- - - - - - - - - - - - - -
4120
(1 row)
Run the gdb with the attached pid:
gdb -p 4120
Search now the .so file, how the function is exactly called:
nm -as libMyExtenstion.so | grep prepareTheOputRecord
00000000002633fe t _ZN6libafd6LIBAFD22prepareTheOutputRecordEP20FunctionCallInfoData
Set a breakpoint in gdb and run the program:
(gdb) b _ZN6libafd6LIBAFD22prepareTheOutputRecordEP20FunctionCallInfoData
Function "_ZN6libafd6LIBAFD22prepareTheOutputRecordEP20FunctionCallInfoData" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_ZN6libafd6LIBAFD22prepareTheOutputRecordEP20FunctionCallInfoData) pending.
(gdb) c
Execute the SQL in psql. At a certain point the breakpoint is hit in gdb:
Breakpoint 1, prepareTheOutputRecord (this=0x1116410, fcinfo=0x7fff3a41e150)
at ../Dir/file.cpp:1736
1736 funcctx = SRF_PERCALL_SETUP();
(gdb)
Continue debugging the code.
I want to do exactly the same in kdbg. For that I loaded the postgres executable, attached to the process, loaded the cpp file, set the breakpoint with the mouse at the function, continued the execution of the postgres process, but the breakpoint was never hit :( I repeated the same with .so file (instead of postgres executable) without any success. I even tried to set the breakpoint to _ZN6libafd6LIBAFD22prepareTheOutputRecordEP20FunctionCallInfoData (without the mouse clicks) but the program does not stop in kdbg :(
I believe the problem was that the kdb was not run as the root (or postgres). Due to wrong permissions the symbols were not loaded and therefore no breakpoint was shown (nor allowed to be placed at a function)