I have a script that initializes a driver on startup, which worked beautifully before I enabled kernel tracing and recompiled the kernel to try and debug an issue with a piece of software. If I try to initialize the driver in any way (modprobe, insmod, etc) this output prints to the screen:
[ 26.263308] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 26.263322] IP: [<c108664d>] trace_module_notify+0x16b/0x20a
[ 26.263325] *pde = 00000000
[ 26.263329] Oops: 0000 [#1] PREEMPT SMP
[ 26.263335] Modules linked in: phddrv(O+)
[ 26.263343] Pid: 704, comm: insmod Tainted: G O 3.6.3-rt9 #21 Advanced Digital Logic, Inc CB4053/ADLS15PC
[ 26.263346] EIP: 0060:[<c108664d>] EFLAGS: 00010213 CPU: 0
[ 26.263350] EIP is at trace_module_notify+0x16b/0x20a
[ 26.263353] EAX: ee6e9274 EBX: f082550c ECX: ee6e920c EDX: f082550c
[ 26.263356] ESI: 00000000 EDI: ee6e92dc EBP: ee6ebf4c ESP: ee6ebf24
[ 26.263359] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 26.263362] CR0: 8005003b CR2: 00000000 CR3: 2f2ea000 CR4: 000007d0
[ 26.263365] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 26.263367] DR6: ffff0ff0 DR7: 00000400
[ 26.263371] Process insmod (pid: 704, ti=ee6ea000 task=ef218000 task.ti=ee6ea000)
[ 26.263372] Stack:
[ 26.263381] ee6e9274 ee6e9344 ee6e92dc ee6e920c ee6e9274 ee6e9344 c2086424 c15a5d58
[ 26.263388] 00000000 00000001 ee6ebf68 c1046d33 f082550c c15a51bc c15a3778 00000000
[ 26.263396] c15a3790 ee6ebf8c c1046fa9 fffffffd 00000000 f082550c 00000001 f082550c
[ 26.263397] Call Trace:
[ 26.263407] [<c1046d33>] notifier_call_chain+0x2b/0x4d
[ 26.263413] [<c1046fa9>] __blocking_notifier_call_chain+0x3c/0x51
[ 26.263419] [<c1046fcf>] blocking_notifier_call_chain+0x11/0x13
[ 26.263426] [<c10671b7>] sys_init_module+0x57/0x190
[ 26.263434] [<c13a3d10>] sysenter_do_call+0x12/0x26
[ 26.263489] Code: 00 c7 42 04 64 5d 5a c1 89 15 64 5d 5a c1 89 45 ec 8d 42 74 83 c2 0c 89 45 e8 89 55 e4 eb 19 57 8b 4d e4 89 da ff 75 ec ff 75 e8 <8b> 06 83 c6 04 e8 c2 fb ff ff 83 c4 0c 3b 75 f0 72 e2 eb 77 b8
[ 26.263495] EIP: [<c108664d>] trace_module_notify+0x16b/0x20a SS:ESP 0068:ee6ebf24
[ 26.263497] CR2: 0000000000000000
[ 26.267381] ---[ end trace 0000000000000002 ]---
Any hint as to what is going on would be greatly appreciated!
I got similar issue as yours (almost the same stack trace of panic).
The root cause on my side is that after I changed the kernel config (enable trace point) I only rebuilt the kernel bzImage but forgot to rebuilt the ko modules! That may cause some execution mismatch between the new kernel and old ko modules.
After rebuild and update both kernel image and ko modules, the issue is gone.
Somewhere in the driver there is a NULL pointer. A pointer variabile has value NULL and the driver is trying to use it.
myPtr->value; /* if myPtr is NULL, this will raise the kernel oops */
You have to debug the driver to find where and why there is a NULL pointer
Related
I don't know about kernel, I don't know how to troubleshoot.
When the problem happened, my system kernel crashed, the following is the vmcore-dmesg.txt log
##dmesg logs
[ 378.442884] SPDMD-LUN:[ERROR]lun_del:2218 device nvfile-mgmtd-0 is busy :2
[ 424.511211] XFS (nvfile-storage-0): Mounting V5 Filesystem
[ 424.513045] XFS (nvfile-storage-2): Mounting V5 Filesystem
[ 424.538953] XFS (nvfile-storage-2): Starting recovery (logdev: internal)
[ 424.546536] XFS (nvfile-storage-2): Ending recovery (logdev: internal)
[ 425.512217] XFS (nvfile-storage-0): Starting recovery (logdev: internal)
[ 425.518929] XFS (nvfile-storage-1): Mounting V5 Filesystem
[ 425.520813] XFS (nvfile-storage-3): Mounting V5 Filesystem
[ 426.288987] XFS (nvfile-storage-3): Starting recovery (logdev: internal)
[ 426.475465] XFS (nvfile-storage-1): Starting recovery (logdev: internal)
[ 427.496551] XFS (nvfile-storage-0): Ending recovery (logdev: internal)
[ 428.320932] XFS (nvfile-storage-3): Ending recovery (logdev: internal)
[ 428.977479] XFS (nvfile-storage-1): Ending recovery (logdev: internal)
[ 445.927118] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[ 445.927685] PGD 0 P4D 0
[ 445.928216] Oops: 0000 [#1] SMP NOPTI
[ 445.928723] CPU: 4 PID: 591 Comm: kworker/u193:1 Kdump: loaded Tainted: G OE --------- - - 4.18.0-305.el8.x86_64 #1
[ 445.929735] Hardware name: Lenovo ThinkSystem SR860 V2/7Z60CTO1WW, BIOS M5E118K-1.52 08/06/2021
[ 445.930305] Workqueue: xfs-cil/nvfile-storage- xlog_cil_push_work [xfs]
[ 445.930910] RIP: 0010:blk_queue_split+0x1c6/0x660
[ 445.933048] Code: c0 45 31 f6 89 44 24 44 89 7c 24 40 31 ff 85 db 0f 84 56 04 00 00 8b 44 24 44 48 89 c1 48 89 44 24 30 48 c1 e1 04 49 03 4d 78 <8b> 41 08 8b 71 0c 48 8b 11 44 29 f8 39 d8 48 89 54 24 48 0f 47 c3
[ 445.936323] RSP: 0018:ffffbcbf9a4bfaf8 EFLAGS: 00010246
[ 445.936711] RAX: 0000000000000000 RBX: 0000000000003000 RCX: 0000000000000000
[ 445.937028] RDX: 0000000000000018 RSI: 0000000000000018 RDI: 0000000000000000
[ 445.937343] RBP: ffffbcbf9a4bfb90 R08: 0000000000000000 R09: ffff940c1965d700
[ 445.937659] R10: 0000000000000000 R11: 0000000000000000 R12: ffff94ca6f590000
[ 445.937979] R13: ffff940bda0ddc80 R14: 0000000000000000 R15: 0000000000000000
[ 445.938304] FS: 0000000000000000(0000) GS:ffff940effd00000(0000) knlGS:0000000000000000
Q1: admin-guide/bug-hunting.html says that:
If the kernel is compiled with CONFIG_DEBUG_INFO, you can enhance
the quality of the stack trace by using
file:scripts/decode_stacktrace.sh.
Is CONFIG_DEBUG_INFO a prerequisite for running the script decode_stacktrace.sh?"
Q2: this patch says:
./decode_stacktrace.sh vmlinux /home/sasha/linux/ < input.log > output.log
Where can I find the input.log? I know it's sort of stack info. Will it only be available when CONFIG_DEBUG_INFO is y or when there is a kernel panic or oops?
Yes, you will need CONFIG_DEBUG_INFO=y in order for decode_stacktrace.sh to extract useful debugging information (such as file names and line numbers) from the kernel image (vmlinux). In theory, the script will run fine even without debug info, but it will give you less information (no file names and line numbers).
NOTE: in recent kernels (>= v5.12) a multiple choice option for the DWARF version was added (CONFIG_DEBUG_INFO_DWARF{4,5,_TOOLCHAIN_DEFAULT}). Then, in v5.18 CONFIG_DEBUG_INFO was completely removed, and the multiple choice option is now the one responsible for enabling debug info. Any choice except CONFIG_DEBUG_INFO_NONE is fine as long as your toolchain supports it.
Where can I find the input.log?
The "log" they are referring to is simply the kernel log. This will be available regardless of CONFIG_DEBUG_INFO. Kernel panics and other OOPSes will usually write stack traces to the kernel log. You can take this directly from the console (if console logging is enabled), or using the dmesg command. See also this doc page for more info. Once you have the log, simply copy-paste it into a text file (input.log), and pass that file to the script's standard input with < input.log.
For an actual panic on the same system you are working on (and not simply inside QEMU or a VM), chances are that the logging can only be done to console, so you might want to enable that:
$ dmesg --console-level 7
$ dmesg --console-on
Take a look at Where are kernel panic logs? on Ask Ubuntu for more ways to capture the log in case of a panic.
As an example, here's the log for a crash generated running echo c > /proc/sysrq-trigger as root on a system running inside QEMU on my machine. I copy-pasted the kernel log into input.log, which contains the following:
[ 7.952685] sysrq: Trigger a crash
[ 7.952850] Kernel panic - not syncing: sysrq triggered crash
[ 7.953098] CPU: 0 PID: 71 Comm: linuxrc Not tainted 5.19.0-rc2 #1
[ 7.953259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[ 7.953655] Call Trace:
[ 7.954133] <TASK>
[ 7.954332] dump_stack_lvl+0x34/0x44
[ 7.954651] panic+0x102/0x27b
[ 7.954756] ? _printk+0x53/0x6a
[ 7.954847] sysrq_handle_crash+0x11/0x20
[ 7.954953] __handle_sysrq.cold+0x43/0x11b
[ 7.955065] write_sysrq_trigger+0x1f/0x30
[ 7.955167] proc_reg_write+0x4c/0x90
[ 7.955267] vfs_write+0xb4/0x290
[ 7.955362] ksys_write+0x5a/0xd0
[ 7.955453] do_syscall_64+0x3b/0x90
[ 7.955553] entry_SYSCALL_64_after_hwframe+0x46/0xb0
[ 7.955773] RIP: 0033:0x4a8531
[ 7.955999] Code: e0 ff ff ff f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 8b 05 d2 26 1e 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 8
[ 7.956427] RSP: 002b:00007ffde8168508 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 7.956625] RAX: ffffffffffffffda RBX: 000000000101a8a0 RCX: 00000000004a8531
[ 7.956787] RDX: 0000000000000002 RSI: 00000000010201e0 RDI: 0000000000000001
[ 7.956949] RBP: 0000000000000001 R08: fefefefefefefeff R09: fefefefefefeff62
[ 7.957113] R10: 00000000000001b6 R11: 0000000000000246 R12: 00000000010201e0
[ 7.957275] R13: 0000000000000002 R14: 00007ffde8168701 R15: 00007ffde8168578
[ 7.957467] </TASK>
[ 7.957806] Kernel Offset: 0x34a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 7.958215] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
The only section that you really need to extract from the above is the following:
[ 7.954332] dump_stack_lvl+0x34/0x44
[ 7.954651] panic+0x102/0x27b
[ 7.954756] ? _printk+0x53/0x6a
[ 7.954847] sysrq_handle_crash+0x11/0x20
[ 7.954953] __handle_sysrq.cold+0x43/0x11b
[ 7.955065] write_sysrq_trigger+0x1f/0x30
[ 7.955167] proc_reg_write+0x4c/0x90
[ 7.955267] vfs_write+0xb4/0x290
[ 7.955362] ksys_write+0x5a/0xd0
[ 7.955453] do_syscall_64+0x3b/0x90
[ 7.955553] entry_SYSCALL_64_after_hwframe+0x46/0xb0
[ 7.955773] RIP: 0033:0x4a8531
[ 7.955999] Code: e0 ff ff ff f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 8b 05 d2 26 1e 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 8
Running decode_stacktrace.sh poiting it to my vmlinux and the kernel source directory yields more info:
$ ./scripts/decode_stacktrace.sh /path/to/vmlinux /path/to/kernel-source-dir < input.log
[ 7.952685] sysrq: Trigger a crash
[ 7.952850] Kernel panic - not syncing: sysrq triggered crash
[ 7.953098] CPU: 0 PID: 71 Comm: linuxrc Not tainted 5.19.0-rc2 #1
[ 7.953259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[ 7.953655] Call Trace:
[ 7.954133] <TASK>
[ 7.954332] dump_stack_lvl (/path/to/kernel-source-dir/lib/dump_stack.c:107 (discriminator 1))
[ 7.954651] panic (/path/to/kernel-source-dir/kernel/panic.c:292)
[ 7.954756] ? _printk (/path/to/kernel-source-dir/kernel/printk/printk.c:2426)
[ 7.954847] sysrq_handle_crash (/path/to/kernel-source-dir/drivers/tty/sysrq.c:155)
[ 7.954953] __handle_sysrq.cold (/path/to/kernel-source-dir/drivers/tty/sysrq.c:626)
[ 7.955065] write_sysrq_trigger (/path/to/kernel-source-dir/drivers/tty/sysrq.c:1168)
[ 7.955167] proc_reg_write (/path/to/kernel-source-dir/fs/proc/inode.c:335 /path/to/kernel-source-dir/fs/proc/inode.c:347)
[ 7.955267] vfs_write (/path/to/kernel-source-dir/fs/read_write.c:589)
[ 7.955362] ksys_write (/path/to/kernel-source-dir/fs/read_write.c:644)
[ 7.955453] do_syscall_64 (/path/to/kernel-source-dir/arch/x86/entry/common.c:50 /path/to/kernel-source-dir/arch/x86/entry/common.c:80)
[ 7.955553] entry_SYSCALL_64_after_hwframe (/path/to/kernel-source-dir/arch/x86/entry/entry_64.S:115)
[ 7.955773] RIP: 0033:0x4a8531
[ 7.955999] Code: e0 ff ff ff f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 8b 05 d2 26 1e 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 8
All code
========
0: e0 ff loopne 0x1
2: ff (bad)
3: ff f7 push %rdi
5: d8 64 89 02 fsubs 0x2(%rcx,%rcx,4)
9: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax
10: eb b3 jmp 0xffffffffffffffc5
12: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
19: 8b 05 d2 26 1e 00 mov 0x1e26d2(%rip),%eax # 0x1e26f1
1f: 85 c0 test %eax,%eax
21: 75 16 jne 0x39
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 57 ja 0x89
32: c3 retq
33: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
39: 08 .byte 0x8
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 57 ja 0x5f
8: c3 retq
9: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
f: 08 .byte 0x8
[ 7.956427] RSP: 002b:00007ffde8168508 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 7.956625] RAX: ffffffffffffffda RBX: 000000000101a8a0 RCX: 00000000004a8531
[ 7.956787] RDX: 0000000000000002 RSI: 00000000010201e0 RDI: 0000000000000001
[ 7.956949] RBP: 0000000000000001 R08: fefefefefefefeff R09: fefefefefefeff62
[ 7.957113] R10: 00000000000001b6 R11: 0000000000000246 R12: 00000000010201e0
[ 7.957275] R13: 0000000000000002 R14: 00007ffde8168701 R15: 00007ffde8168578
[ 7.957467] </TASK>
[ 7.957806] Kernel Offset: 0x34a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 7.958215] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
Regarding Q2, I setup a serial console where I can get all the kernel messages, including the input.log.
Below I list the steps to setup a serial console.
Components: 1x Linux machine, 1 x PC with Win10 and Putty, 1x USB to Serial Converter
Your Win10 should automatically recognize the USB2Serial converter and install the driver. The converter appears in Device Manager, in my case COM6.
Setup Putty, Serial, baud-rate 115200, Databits 8, Stopbits 1, Parity none, Flowcontrol XON/XOFF.
Make sure that serial port is enabled in Linux machine.
$ dmesg | grep tty
[ 0.108687] printk: console [tty0] enabled
[ 2.752397] 00:03:ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 2.773666] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
It shows that both ttyS0(COM1) and ttyS1(COM2) work fine. In my case I choose ttyS1.
Use getty to manage ttyS1.
sudo /sbin/getty -L 115200 ttyS1 vt102
Then the Putty would prompt login, just input your username and password to get into the command line interface.
But still at this stage, the ttyS1 is another pseudo-terminal, but not a console. You need to configure it as a console.
Make sure that you've already compiled in the serail port in your Linux kernel image. make menuconfig : Device driver ‣ Character devices ‣ Serial drivers ‣ 8250/16550 and compatible serial support ‣ Console on 8250/16550 and compatible serial port: choose built-in. If it's alreay that, leave it. Otherwise, change the configuration, rebuilt the kernel, and reboot.
Check the current console.
$cat /sys/devices/virtual/tty/console/active
tty0
It means that current console is tty0. We need to add ttyS1 as a console.
Add console=ttyS1 in Linux kernel boot argument.
$sudo vim /etc/default/grub
Change the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" to GRUB_CMDLINE_LINUX_DEFAULT="quiet console=ttyS1,115200 splash"
Then
$sudo update-grub
$sudo reboot
Check the current console.
$ cat /sys/devices/virtual/tty/console/active
ttyS1
That's it.
Call getty again.
sudo /sbin/getty -L 115200 ttyS1 vt102
Then Putty would get the login prompt and after you login, it becomes a real console.
We have setup procdump as the (AeDebug) postmortem debugger to capture dumps of unhandled exceptions. The registry key is set to "c:\my\sysinternals\procdump.exe" -accepteula -ma -j "c:\dumps" %ld %ld %p
Currently I'm looking at a dump where the process that triggered a crash dump is still running hours after dumping the process has completed?!
It was my assumption that any process that triggers a crash dump is about to get terminated?
From WinDbg
Debug session time: Tue Dec 1 01:53:06.000 2015 (UTC + 1:00)
System Uptime: 18 days 18:09:24.556
Process Uptime: 1 days 0:09:31.000
0:000> ~
. 0 Id: 178c.eb4 Suspend: 0 Teb: fffdd000 Unfrozen
...
0:000> ? 178c
Evaluate expression: 6028 = 0000178c
From ProcExp
Looking at PID 6028 on Tue Dec 1 10:00:00, hours after the dump from 01:53:06.000
<3thParty>.exe:6028 Properties
Started: 1:43:35 30/11/2015
Edit
0:000> ~*kb1000
. 0 Id: 178c.eb4 Suspend: 0 Teb: fffdd000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0018de38 77cd8567 00000574 00000001 00000000 ntdll!ZwWaitForSingleObject+0x15
01 0018debc 77cd8695 0018e0c8 0018e118 00000001 ntdll!RtlReportExceptionEx+0x14b
02 0018df14 772f8dd8 0018e0c8 0018e118 00000001 ntdll!RtlReportException+0x86
03 0018df34 772f8edb 0018dfa4 0018dfa4 00000000 ole32!SilentlyReportExceptions+0x79
04 0018df48 772f94e5 0018dfa4 00000000 00000000 ole32!ServerExceptionFilter+0xbb
05 0018df60 77360827 0018dfa4 0148502c 01048240 ole32!AppInvokeExceptionFilterWithMethodAddress+0x11
06 0018df7c 77183f21 00000000 0018faf8 77278760 ole32!SyncStubInvoke+0x82
07 0018df90 77183eae 00000000 00000000 00000000 msvcrt!_EH4_CallFilterFunc+0x12
08 0018dfbc 77286771 7736662c 7726ea10 0018e0c8 msvcrt!_except_handler4_common+0x8e
09 0018dfdc 77c8b499 0018e0c8 0018fae8 0018e118 ole32!_except_handler4+0x20
0a 0018e000 77c8b46b 0018e0c8 0018fae8 0018e118 ntdll!ExecuteHandler2+0x26
0b 0018e024 77c8b40e 0018e0c8 0018fae8 0018e118 ntdll!ExecuteHandler+0x24
0c 0018e0b0 77c40133 0118e0c8 0018e118 0018e0c8 ntdll!RtlDispatchException+0x127
0d 0018e0b0 00404880 0118e0c8 0018e118 0018e0c8 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
0e 0018f61c 75a15932 01485018 01549420 00000000 <3thParty>+0x4880
0f 0018f640 75a905f1 00523859 0018f828 00000004 rpcrt4!Invoke+0x2a
10 0018fa44 7735aec1 010ac688 0101fb58 0104ea78 rpcrt4!NdrStubCall2+0x2ea
11 0018fa8c 76acffd3 010ac688 0104ea78 0101fb58 ole32!CStdStubBuffer_Invoke+0x3c
12 0018fab0 7735d876 010ac688 0104ea78 0101fb58 oleaut32!CUnivStubWrapper::Invoke+0xcb
13 0018faf8 7735ddd0 0104ea78 01048240 01095448 ole32!SyncStubInvoke+0x3c
14 0018fb44 77278a43 0104ea78 010be578 01097470 ole32!StubInvoke+0xb9
15 0018fc20 77278938 0101fb58 00000000 01097470 ole32!CCtxComChnl::ContextInvoke+0xfa
16 0018fc3c 7727950a 0104ea78 00000001 01097470 ole32!MTAInvoke+0x1a
17 0018fc68 7735dccd 0104ea78 00000001 01097470 ole32!STAInvoke+0x46
18 0018fc9c 7735db41 d0908070 0101fb58 01097470 ole32!AppInvoke+0xab
19 0018fd7c 7735e1fd 0104ea20 0101fef0 00000400 ole32!ComInvokeWithLockAndIPID+0x372
1a 0018fda4 77279367 0104ea20 00000400 00ff3548 ole32!ComInvoke+0xc5
1b 0018fdb8 77279326 0104ea20 00000000 77279286 ole32!ThreadDispatch+0x23
1c 0018fdfc 76eb62fa 00e2019e 00000400 0000babe ole32!ThreadWndProc+0x161
1d 0018fe28 76eb6d3a 77279286 00e2019e 00000400 user32!InternalCallWinProc+0x23
1e 0018fea0 76eb77c4 00000000 77279286 00e2019e user32!UserCallWinProcCheckWow+0x109
1f 0018ff00 76eb788a 77279286 00000000 00e2019e user32!DispatchMessageWorker+0x3bc
20 0018ff10 004aea6a 0018ff34 00010001 0018ff70 user32!DispatchMessageW+0xf
21 0018ff2c 004aeaaf 00e2019e 00000400 0000babe <3thParty>+0xaea6a
22 0018ff50 00a0c705 0018ff78 00a0c733 0018ff70 <3thParty>+0xaeaaf
23 0018ff70 00a3024b 0018ffc4 00404cac 0018ff88 <3thParty>!FrobWidget+0x4db279
24 0018ff88 7766338a fffde000 0018ffd4 77c69f72 <3thParty>!FrobWidget+0x4fedbf
25 0018ff94 77c69f72 fffde000 2ca36a95 00000000 kernel32!BaseThreadInitThunk+0xe
26 0018ffd4 77c69f45 00a30220 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
27 0018ffec 00000000 00a30220 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
1 Id: 178c.17a4 Suspend: 0 Teb: fffda000 Unfrozen
# ChildEBP RetAddr Args to Child
00 023cfed4 76b614ab 00000180 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
01 023cff40 77661194 00000180 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
02 023cff58 77661148 00000180 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
03 023cff6c 76bb511d 00000180 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
04 023cff88 7766338a 00234b88 023cffd4 77c69f72 ws2_32!SockAsyncThread+0x25
05 023cff94 77c69f72 00234b88 2e876a95 00000000 kernel32!BaseThreadInitThunk+0xe
06 023cffd4 77c69f45 76bb50f8 00234b88 ffffffff ntdll!__RtlUserThreadStart+0x70
07 023cffec 00000000 76bb50f8 00234b88 00000000 ntdll!_RtlUserThreadStart+0x1b
2 Id: 178c.1108 Suspend: 0 Teb: fffd7000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0273fdf4 77c82f91 00000003 010169d0 00000001 ntdll!NtWaitForMultipleObjects+0x15
01 0273ff88 7766338a 00000000 0273ffd4 77c69f72 ntdll!TppWaiterpThread+0x33d
02 0273ff94 77c69f72 010169a0 2ec86a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0273ffd4 77c69f45 77c82e65 010169a0 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0273ffec 00000000 77c82e65 010169a0 00000000 ntdll!_RtlUserThreadStart+0x1b
3 Id: 178c.1794 Suspend: 0 Teb: fffac000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0908fe28 77c83392 000001c0 0908fedc 25b36ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 0908ff88 7766338a 01015748 0908ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 0908ff94 77c69f72 01015748 25b36a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0908ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0908ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b
4 Id: 178c.5e4 Suspend: 0 Teb: fffaf000 Unfrozen
# ChildEBP RetAddr Args to Child
00 08b0fe28 77c83392 000001c4 08b0fedc 240b6ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 08b0ff88 7766338a 01015748 08b0ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 08b0ff94 77c69f72 01015748 240b6a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 08b0ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 08b0ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b
5 Id: 178c.c80 Suspend: 0 Teb: fffa9000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0b03fed8 76b63bd5 00000000 0b03ff1c 57962de4 ntdll!ZwDelayExecution+0x15
01 0b03ff40 76b644a5 0000ea60 00000000 0b03ff78 KERNELBASE!SleepEx+0x65
02 0b03ff50 7724d98d 0000ea60 010bd460 7724cd48 KERNELBASE!Sleep+0xf
03 0b03ff5c 7724cd48 00000000 00000000 010bd460 ole32!CROIDTable::WorkerThreadLoop+0x14
04 0b03ff78 7724d87a 00000000 00000000 0b03ff94 ole32!CRpcThread::WorkerLoop+0x26
05 0b03ff88 7766338a 010bd460 0b03ffd4 77c69f72 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x16
06 0b03ff94 77c69f72 010bd460 27b86a95 00000000 kernel32!BaseThreadInitThunk+0xe
07 0b03ffd4 77c69f45 7724d864 010bd460 ffffffff ntdll!__RtlUserThreadStart+0x70
08 0b03ffec 00000000 7724d864 010bd460 00000000 ntdll!_RtlUserThreadStart+0x1b
6 Id: 178c.14f4 Suspend: 0 Teb: fffa6000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0b13fe28 77c83392 000001bc 0b13fedc 27a86ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 0b13ff88 7766338a 01015748 0b13ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 0b13ff94 77c69f72 01015748 27a86a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0b13ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0b13ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b
7 Id: 178c.ee4 Suspend: 0 Teb: fffa3000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0b23fcbc 76b614ab 00000578 00000000 0b23fd04 ntdll!ZwWaitForSingleObject+0x15
01 0b23fd28 77661194 00000578 00002710 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
02 0b23fd40 77661148 00000578 00002710 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
03 0b23fd54 6e22b3d6 00000578 00002710 00000000 kernel32!WaitForSingleObject+0x12
04 0b23ff88 7766338a 01072188 0b23ffd4 77c69f72 comsvcs!PingThread+0x131
05 0b23ff94 77c69f72 01072188 27986a95 00000000 kernel32!BaseThreadInitThunk+0xe
06 0b23ffd4 77c69f45 6e22b320 01072188 ffffffff ntdll!__RtlUserThreadStart+0x70
07 0b23ffec 00000000 6e22b320 01072188 00000000 ntdll!_RtlUserThreadStart+0x1b
8 Id: 178c.914 Suspend: 0 Teb: fff9d000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0d26ff5c 72be635c 00000534 0d26ff90 0d26ff84 ntdll!ZwRemoveIoCompletion+0x15
01 0d26ff88 7766338a 72be64b3 0d26ffd4 77c69f72 mswsock!SockAsyncThread+0x83
02 0d26ff94 77c69f72 0101d3e8 219d6a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0d26ffd4 77c69f45 72be62ee 0101d3e8 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0d26ffec 00000000 72be62ee 0101d3e8 00000000 ntdll!_RtlUserThreadStart+0x1b
Edit 2
0:000> .exr -1
ExceptionAddress: 00404880 (<3thParty>+0x00004880)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000000
Attempt to read from address 00000000
0:000> ~* e s -d esp L0xffff 0x1003f
0018e118 0001003f 00000000 00000000 00000000 ?...............
0018e9a8 0001003f 00000000 00000000 00000000 ?...............
0018efa8 0001003f 00000000 00000000 00000000 ?...............
0:000> .cxr 0018e118
eax=00fc2c80 ebx=00000000 ecx=00000000 edx=00fc2c50 esi=00000000 edi=00000000
eip=00404880 esp=0018e400 ebp=0018f61c iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
<3thParty>+0x4880:
00404880 8b11 mov edx,dword ptr [ecx] ds:002b:00000000=????????
0:000> kbnf
*** Stack trace for last set context - .thread/.cxr resets it
# Memory ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0018f61c 75a15932 01485018 01549420 00000000 <3thParty>+0x4880
01 24 0018f640 75a905f1 00523859 0018f828 00000004 rpcrt4!Invoke+0x2a
02 404 0018fa44 7735aec1 010ac688 0101fb58 0104ea78 rpcrt4!NdrStubCall2+0x2ea
03 48 0018fa8c 76acffd3 010ac688 0104ea78 0101fb58 ole32!CStdStubBuffer_Invoke+0x3c
04 24 0018fab0 7735d876 010ac688 0104ea78 0101fb58 oleaut32!CUnivStubWrapper::Invoke+0xcb
05 48 0018faf8 7735ddd0 0104ea78 01048240 01095448 ole32!SyncStubInvoke+0x3c
06 4c 0018fb44 77278a43 0104ea78 010be578 01097470 ole32!StubInvoke+0xb9
07 dc 0018fc20 77278938 0101fb58 00000000 01097470 ole32!CCtxComChnl::ContextInvoke+0xfa
08 1c 0018fc3c 7727950a 0104ea78 00000001 01097470 ole32!MTAInvoke+0x1a
09 2c 0018fc68 7735dccd 0104ea78 00000001 01097470 ole32!STAInvoke+0x46
0a 34 0018fc9c 7735db41 d0908070 0101fb58 01097470 ole32!AppInvoke+0xab
0b e0 0018fd7c 7735e1fd 0104ea20 0101fef0 00000400 ole32!ComInvokeWithLockAndIPID+0x372
0c 28 0018fda4 77279367 0104ea20 00000400 00ff3548 ole32!ComInvoke+0xc5
0d 14 0018fdb8 77279326 0104ea20 00000000 77279286 ole32!ThreadDispatch+0x23
0e 44 0018fdfc 76eb62fa 00e2019e 00000400 0000babe ole32!ThreadWndProc+0x161
0f 2c 0018fe28 76eb6d3a 77279286 00e2019e 00000400 user32!InternalCallWinProc+0x23
10 78 0018fea0 76eb77c4 00000000 77279286 00e2019e user32!UserCallWinProcCheckWow+0x109
11 60 0018ff00 76eb788a 77279286 00000000 00e2019e user32!DispatchMessageWorker+0x3bc
12 10 0018ff10 004aea6a 0018ff34 00010001 0018ff70 user32!DispatchMessageW+0xf
13 1c 0018ff2c 004aeaaf 00e2019e 00000400 0000babe <3thParty>+0xaea6a
14 24 0018ff50 00a0c705 0018ff78 00a0c733 0018ff70 <3thParty>+0xaeaaf
15 20 0018ff70 00a3024b 0018ffc4 00404cac 0018ff88 <3thParty>!FrobWidget+0x4db279
16 18 0018ff88 7766338a fffde000 0018ffd4 77c69f72 <3thParty>!FrobWidget+0x4fedbf
17 c 0018ff94 77c69f72 fffde000 2ca36a95 00000000 kernel32!BaseThreadInitThunk+0xe
18 40 0018ffd4 77c69f45 00a30220 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
19 18 0018ffec 00000000 00a30220 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
Normally, an unhandled exception does cause the process to be terminated once the dump has been captured. What's happening here is that you are running afoul of a compatibility behavior: If a COM server crashes while handling an inbound request, COM historically "handles" the crash by abandoning the call, returning RPC_E_SERVERFAULT to the external caller, and then pretending that the crash never occurred.
That's why the process is still running after the crash dump is taken: The exception was handled. Mind you, it was "handled" by ignoring it, but technically it was handled.
I would recommend using IGlobalOptions to set the COMGLB_EXECPTION_HANDLING property to one of the COMGLB_EXCEPTION_DONOT_HANDLE... values.
I am getting following Oops message while testing on a device which is running on Linux Kernel 3.4.5 and ARM processor.I am unable to trace the issue.
If you look at call stack , you would see it starting from ret_fast_syscall() , now actually sysrq for Crash has been triggered inside the code but i am not getting from where.How can i find that. I have Lauterbach installed but no idea from where to find which part of kernel code has actually triggered this SysRq.
[50728.239318] C0 [ sh] SysRq : Trigger a crash
[50728.239501] C0 [ sh] **************** READ GIC status
[50728.239654] C0 [ sh] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[50728.239776] C0 [ sh] pgd = ebc9c000
[50728.239929] C0 [ sh] [00000000] *pgd=00000000
[50728.240081] C0 [ sh] Internal error: Oops: 805 [#1] PREEMPT SMP ARM
[50728.240234] C0 [ sh] kona_fb: die notifier invoked
[50728.240356] C0 [ sh] Modules linked in: bcmdhd dm_crypt(O) moc_crypto(PO) moc_platform_mod(O) texfat(PO)
[50728.241088] C0 [ sh] CPU: 0 Tainted: P W O (3.4.5-g4192471-dirty #116)
[50728.241271] C0 [ sh] PC is at sysrq_handle_crash+0x14/0x20
[50728.241424] C0 [ sh] LR is at __handle_sysrq+0xa0/0x14c
[50728.241516] C0 [ sh] pc : [<c026ad48>] lr : [<c026b210>] psr: 60000093
[50728.241516] C0 [ sh] sp : ebd0ff20 ip : 0000000c fp : 4003a404
[50728.241821] C0 [ sh] r10: 4070002c r9 : edbc2f0c r8 : 00000000
[50728.241912] C0 [ sh] r7 : 60000013 r6 : 00000063 r5 : 00000007 r4 : c0a26b90
[50728.242065] C0 [ sh] r3 : 00000001 r2 : 00000000 r1 : c07ab3a2 r0 : 00000063
[50728.242248] C0 [ sh] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[50728.242401] C0 [ sh] Control: 10c53c7d Table: adc9c06a DAC: 00000015
[50728.242492] C0 [ sh]
.................. (Register Contents).............
[50728.284851] C0 [ sh] Stack: (0xebd0ff20 to 0xebd10000)
[50728.284942] C0 [ sh] ff20: 00000002 c026b2bc ec2a4ef0 ebd0ff88 00000002 c026b2e0 edbc2ec0 c014c840
[50728.285125] C0 [ sh] ff40: 00000002 ec2a4ef0 4070002c ebd0ff88 00000002 ebd0e000 00000000 c0109d24
[50728.285308] C0 [ sh] ff60: ec2a4ef0 4070002c ec2a4ef0 4070002c 00000000 00000000 00000002 c0109f5c
[50728.285491] C0 [ sh] ff80: 00000001 00000000 00000000 00000000 00000003 00000002 00000001 00000004
[50728.285675] C0 [ sh] ffa0: c000e304 c000e140 00000003 00000002 00000001 4070002c 00000002 ffffffff
[50728.285858] C0 [ sh] ffc0: 00000003 00000002 00000001 00000004 4070002c 00000000 406ffc84 4003a404
[50728.286010] C0 [ sh] ffe0: 40035f38 bea5e710 40020257 40150e44 20000010 00000001 00000000 00000000
[50728.286224] C0 [ sh] [<c026ad48>] (sysrq_handle_crash+0x14/0x20) from [<c026b210>] (__handle_sysrq+0xa0/0x14c)
***[50728.286376] C0 [ sh] [<c026b210>] (__handle_sysrq+0xa0/0x14c) from [<c026b2e0>] (write_sysrq_trigger+0x24/0x34)
[50728.286560] C0 [ sh] [<c026b2e0>] (write_sysrq_trigger+0x24/0x34) from [<c014c840>] (proc_reg_write+0x80/0x94)
[50728.286682] C0 [ sh] [<c014c840>] (proc_reg_write+0x80/0x94) from [<c0109d24>] (vfs_write+0xb0/0x128)
[50728.286865] C0 [ sh] [<c0109d24>] (vfs_write+0xb0/0x128) from [<c0109f5c>] (sys_write+0x38/0x64)
[50728.287048] C0 [ sh] [<c0109f5c>] (sys_write+0x38/0x64) from [<c000e140>] (ret_fast_syscall+0x0/0x48)***
[50728.287200] C0 [ sh] Code: e3a03001 e5823000 f57ff04f e3a02000 (e5c23000)
[50728.287384] C0 [ sh] ---[ end trace 1b75b31a2719ed7e ]---
[50728.287475] C0 [ sh] Kernel panic - not syncing: Fatal exception
[50728.239318] C0 [ sh] SysRq : Trigger a crash
Someone at userland triggered a crash as you would do via echo c > /proc/sysrq-trigger.
I have already encountered some Oops in my developer's life and whereas I am familiar with some information that I can retrieve from these Oops, there are still pieces of information I can't understand and therefore, can't use to solve problems.
Below you will find an Oops example and I will describe what I can deduce from it. Then, I will ask what the remaining info can teach me about the problem.
[ 716.485951] BUG: unable to handle kernel paging request at fc132158
[ 716.485973] IP: [<fc1936e7>] ubi_change_vtbl_record+0x87/0x1c0 [ubi]
[ 716.485986] *pdpt = 00000000019e6001 *pde = 000000002c558067 *pte = 0000000000000000
[ 716.485997] Oops: 0002 [#1] SMP
[ 716.486004] Modules linked in: ubi(O) mtdchar nandsim nand mtd nand_ids nand_bch bch nand_ecc bnep rfcomm bluetooth parport_pc ppdev lp parport nfsd nfs_acl auth_rpcgss nfs fscache lockd sunrpc binfmt_misc dm_crypt snd_hda_codec_hdmi snd_hda_codec_analog kvm_intel snd_hda_intel snd_hda_codec snd_hwdep kvm snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event hid_generic snd_seq cdc_acm snd_timer snd_seq_device mei tpm_tis snd mac_hid serio_raw soundcore lpc_ich snd_page_alloc microcode coretemp usbhid hid nouveau usb_storage ttm drm_kms_helper drm floppy e1000e i2c_algo_bit mxm_wmi video wmi
[ 716.486128] Pid: 3994, comm: ubimkvol Tainted: G O 3.8.0-rc3+ #3 LENOVO 6239AS8/LENOVO
[ 716.486136] EIP: 0060:[<fc1936e7>] EFLAGS: 00010246 CPU: 0
[ 716.486144] EIP is at ubi_change_vtbl_record+0x87/0x1c0 [ubi]
[ 716.486151] EAX: 000000ac EBX: eb5ea000 ECX: 0000002b EDX: 00000000
[ 716.486157] ESI: eb4d1d74 EDI: fc132158 EBP: eb4d1d40 ESP: eb4d1d20
[ 716.486164] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 716.486170] CR0: 8005003b CR2: fc132158 CR3: 27542000 CR4: 000407f0
[ 716.486176] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 716.486183] DR6: ffff0ff0 DR7: 00000400
[ 716.486188] Process ubimkvol (pid: 3994, ti=eb4d0000 task=ec01d9b0 task.ti=eb4d0000)
[ 716.486195] Stack:
[ 716.486199] e755f000 eb4d1d2c c11cad11 eb4d1d34 eb543c00 eb5ea000 00000000 eb4d1e20
[ 716.486215] eb4d1e30 fc195412 e755f000 fc1adf01 eb5ea26c 00000002 0000009e eb5ea480
[ 716.486232] 00000002 e755f22c e755f2ac e755f000 eb4d1d74 2a000000 01000000 00000000
[ 716.486248] Call Trace:
[ 716.486257] [<c11cad11>] ? sysfs_create_file+0x21/0x30
[ 716.486266] [<fc195412>] ubi_create_volume+0x4b2/0x790 [ubi]
[ 716.486277] [<fc19967a>] ubi_cdev_ioctl+0x5da/0xac0 [ubi]
[ 716.486285] [<c117202a>] ? link_path_walk+0x5a/0x7d0
[ 716.486294] [<fc1990a0>] ? vol_cdev_ioctl+0x440/0x440 [ubi]
[ 716.486842] [<c1177e12>] do_vfs_ioctl+0x82/0x5b0
[ 716.487703] [<c1171ced>] ? final_putname+0x1d/0x40
[ 716.488564] [<c1171ced>] ? final_putname+0x1d/0x40
[ 716.489422] [<c1171ced>] ? final_putname+0x1d/0x40
[ 716.489891] [<c1171eb4>] ? putname+0x24/0x40
[ 716.489891] [<c1167239>] ? do_sys_open+0x169/0x1d0
[ 716.489891] [<c11783b0>] sys_ioctl+0x70/0x80
[ 716.489891] [<c16205cd>] sysenter_do_call+0x12/0x38
[ 716.489891] Code: ac 00 00 00 03 bb c8 04 00 00 f7 c7 01 00 00 00 0f 85 ee 00 00 00 f7 c7 02 00 00 00 0f 85 ca 00 00 00 89 c1 31 d2 c1 e9 02 a8 02 <f3> a5 74 0b 0f b7 16 66 89 17 ba 02 00 00 00 a8 01 74 07 0f b6
[ 716.489891] EIP: [<fc1936e7>] ubi_change_vtbl_record+0x87/0x1c0 [ubi] SS:ESP 0068:eb4d1d20
[ 716.489891] CR2: 00000000fc132158
[ 716.516453] ---[ end trace 473b15a7780e19ea ]---
It seems that the kernel wanted to access a wrong page. Now,
The Oops code 0002 tells me that it occurred while trying to read something in user-mode.
The Instruction Pointer is at ubi_change_vtbl_record, which means the offending instruction is located in this function.
I can deduce the path that lead to the faulting function from the
call trace (an ioctl launched from process ubimkvol)
From there, Is the "stack" a dump of the raw stack of the task ? I can see that some values mentioned are also function addresses found in the call trace. Then, I got fancy looking values like EAX, EBX ... DR7. I think they are CPU registers but still, I don't know what they really are.
Finally, the following line gets me lost :
[ 716.485986] *pdpt = 00000000019e6001 *pde = 000000002c558067 *pte = 0000000000000000
What are pdpt, pde and pte ? I feel they are information about the page fault but I could not retrieve further information after some googling around.
Yes, EAX, etc. are 32-bit x86 processor registers. pdpt (page directory pointer table), pde (page directory entry), and pte (page table entry) are all paging structures.
IP (also EIP for 32-bit or RIP for 64-bit processors) is the instruction pointer at the time of the Oops.
The stack is the raw stack for this processor. Each processor will have its own stack. Note that on this architecture the stack grows down (addresses start with 0xfxxxxxx).
Correct me if I am wrong but,
OOPS 0002 means no page found when writing in kernel mode:
bit 0 == 0 means no page found, 1 means a protection fault
bit 1 == 0 means read, 1 means write
bit 2 == 0 means kernel, 1 means user-mode