I had a kernel crash on a Redhat 6.
I want to retreive which file was openend
reading the dump I see
crash> bt
(...)
#8 [ffff8805050d7dc0] page_fault at ffffffff814ef845
[exception RIP: configfs_readdir+244]
RIP: ffffffffa0422444 RSP: ffff8805050d7e78 RFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff880814a738d0 RCX: ffff880814a738c8
RDX: 0000000000000006 RSI: ffff880814a73830 RDI: ffffffffa04486d0
RBP: ffff8805050d7ed8 R8: ffff880814a738d0 R9: 0000000000000004
R10: 00000000000000a8 R11: 0000000000000246 R12: ffff8804cf782b00
R13: ffffffffa04486d0 R14: ffff8802087039d0 R15: ffff8802087039d8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8805050d7ee0] vfs_readdir at ffffffff8118a670
(...)
crash > bt -f
(...)
#9 [ffff8805050d7ee0] vfs_readdir at ffffffff8118a670
ffff8805050d7ee8: ffff8805050d7f28 ffffffff81176652
ffff8805050d7ef8: 0000000000001000 00000000009ca848
ffff8805050d7f08: ffff8804cf782b00 0000000000008000
ffff8805050d7f18: 0000000000000000 00000000009ca820
ffff8805050d7f28: ffff8805050d7f78 ffffffff8118a7f9
I look at vfs_readdir
crash> whatis vfs_readdir
int vfs_readdir(struct file *, filldir_t, void *);
Then searching for the file (address ffff8805050d7f28 picked from top of stack at #9 )
crash> struct file ffff8805050d7f28
struct file {
f_u = {
fu_list = {
next = 0xffff8805050d7f78,
prev = 0xffffffff8118a7f9
},
fu_rcuhead = {
next = 0xffff8805050d7f78,
func = 0xffffffff8118a7f9 <sys_getdents+137>
}
},
f_path = {
mnt = 0x9ca878,
dentry = 0x9ca860
},
f_op = 0xffffffea00007fd0,
f_lock = {
raw_lock = {
slock = 578155200
}
},
At this point, I can go no further, how can I get string from f_path struct ?
I want to convert
f_path = {
mnt = 0x9ca878,
dentry = 0x9ca860
},
to strings
reading the struct, file owner is -1, effective uid didn't map to a known user.
from crash, files command show only 2 regular file
EDIT: as requested
crash> p ffff8805050d7f28
No symbol "ffff8805050d7f28" in current context.
p: gdb request failed: p ffff8805050d7f28
crash> p *(struct file) ffff8805050d7f28
No symbol "ffff8805050d7f28" in current context.
p: gdb request failed: p *(struct file) ffff8805050d7f28
Related
I updated my mac Big Sur to Monterey Beta.
After update, my application (installed before) not opens on iPhone Simulator (crashes immediately). And on xcode lots of errors appears, can't build.
Simulator crash report:
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000
VM Region Info: 0 is not in any region. Bytes before following region: 4341362688
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START
--->
mapped file 102c3f000-103ce7000 [ 16.7M] r-x/r-x SM=COW ...t_id=ca4c8d93
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: SIGNAL 11 Segmentation fault: 11
Terminating Process: exc handler [15816]
Triggered by Thread: 0
Application Specific Information:
Thread 0 Crashed:
0 ??? 0x7ff7ffd02bd8 ???
1 <translation info unavailable> 0x1067fbe5c ???
2 dyld 0x2068aa88f dyld4::prepareSim(dyld4::RuntimeState&, char const*) + 890
3 dyld 0x2068a96b5 dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 244
4 dyld 0x2068a94b4 start + 388
5 dyld 0x2068a4000 ???
Thread 1:: com.apple.rosetta.exceptionserver
0 ??? 0x7ff7ffcef320 ???
1 ??? 0x7ff7ffd081a0 ???
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x000000010e852af8 rcx: 0x0000000000000000 rdx: 0x0000000000000001
rdi: 0x0000000000000000 rsi: 0x0000000000000000 rbp: 0x0000000000000000 rsp: 0x000000010eee6000
r8: 0xbae6d4fe37da0005 r9: 0x0000000000000000 r10: 0x000000010eee6000 r11: 0x000000010ed12010
r12: 0x0000000000000000 r13: 0x000000010ed12060 r14: 0x0000000206918080 r15: 0x0000000000000000
rip: <unavailable> rfl: 0x0000000000000283
tmp0: 0x00000001067e9c64 tmp1: 0x00000001067e8f8c tmp2: 0x00000002068c2ca3
Binary Images:
0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???
0x2068a4000 - 0x20690bfff dyld (*) <4e207376-cc5d-3986-a0bd-4d09f4217e68> /usr/lib/dyld
I want to know how can I symbolicate a kernel panic report (not a regular app crash report). I have an OSX app that a user claims caused a kernel panic. It doesn't install any kernel extension but a launch daemon. I have followed this link https://developer.apple.com/library/mac/technotes/tn2063/_index.html but it only describes symbolication steps for the 3rd party kernel extension causing a kernel panic. How can I symbolicate the following panic report if I have not installed any kext? Any help will be appreciable.
Anonymous UUID: DF6F780A-AA27-6D40-3441-F26F828C7096
Tue Oct 27 12:02:41 2015
*** Panic Report ***
panic(cpu 1 caller 0xffffff80111d6a9a): Kernel trap at 0xffffff80113cec30, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000100000057, CR3: 0x000000025eade116, CR4: 0x00000000001627e0
RAX: 0xffffff8037f9bdb0, RBX: 0xffffff803e68cd08, RCX: 0xffffff8037f9bdb0, RDX: 0xffffff8011910910
RSP: 0xffffff9236923890, RBP: 0xffffff9236923900, RSI: 0x0000000000000001, RDI: 0xffffff8011910910
R8: 0x0000000000000000, R9: 0x00000000000001f0, R10: 0xffffff80118ddb78, R11: 0x0000000000000000
R12: 0x00000000ffffffff, R13: 0x0000000000000000, R14: 0xffffff80117a1f7b, R15: 0xffffff8031c77a40
RFL: 0x0000000000010206, RIP: 0xffffff80113cec30, CS: 0x0000000000000008, SS: 0x0000000000000000
Fault CR2: 0x0000000100000057, Error code: 0x0000000000000000, Fault CPU: 0x1, PL: 0
Backtrace (CPU 1), Frame : Return Address
0xffffff9236923520 : 0xffffff80110e5357
0xffffff92369235a0 : 0xffffff80111d6a9a
0xffffff9236923780 : 0xffffff80111f4093
0xffffff92369237a0 : 0xffffff80113cec30
0xffffff9236923900 : 0xffffff80113d0229
0xffffff9236923920 : 0xffffff801140edc7
0xffffff9236923a40 : 0xffffff801140c838
0xffffff9236923ad0 : 0xffffff801135bdac
0xffffff9236923ba0 : 0xffffff801135bfff
0xffffff9236923bd0 : 0xffffff801139912d
0xffffff9236923d80 : 0xffffff80113983fd
0xffffff9236923da0 : 0xffffff8011596f61
0xffffff9236923de0 : 0xffffff80115f0ebc
0xffffff9236923e30 : 0xffffff80115f13ea
0xffffff9236923e50 : 0xffffff801158a610
0xffffff9236923ec0 : 0xffffff8011586140
0xffffff9236923f20 : 0xffffff801158a2b4
0xffffff9236923f60 : 0xffffff801162ace1
0xffffff9236923fb0 : 0xffffff80111f4896
BSD process name corresponding to current thread: MyApp
Mac OS version:
15A284
Kernel version:
Darwin Kernel Version 15.0.0: Wed Aug 26 16:57:32 PDT 2015; root:xnu- 3247.1.106~1/RELEASE_X86_64
Kernel UUID: 37BC582F-8BF4-3F65-AFBB-ECF792060C68
Kernel slide: 0x0000000010e00000
Kernel text base: 0xffffff8011000000
__HIB text base: 0xffffff8010f00000
System model name: MacBookPro11,3 (Mac-2BD1B31983FE1663)
System uptime in nanoseconds: 48947258824078
last loaded kext at 46570627969383: com.apple.driver.AppleMikeyHIDDriver 124 (addr 0xffffff7f940e6000, size 20480)
last unloaded kext at 48111600364863: com.apple.driver.AppleMikeyHIDDriver 124 (addr 0xffffff7f940e6000, size 12288)
loaded kexts:
com.apple.nke.rvi 2.0.0
com.apple.filesystems.smbfs 3.0.0
com.apple.filesystems.afpfs 11.0
com.apple.nke.asp-tcp 8.0.0
...
I'm trying to set my CPU cache write policy to 'write back' so I need to set CR0.NW = 1.
I wrote a kernel module:
int
init_module (void)
{
printk (KERN_INFO "init_module\n\n\n");
uint64_t cr0;
asm volatile ("mov %%cr0,%%rax\n\t":"=a"(cr0));
printk(KERN_INFO"CR0 ===== %ld\n",cr0);
asm volatile("push %rax\n\t" "push %rbx\n\t");
asm volatile( //disable cache before changing cr0.nw
"mov $1,%rbx\n\t"
"shl $30,%rbx\n\t"
"mov %cr0,%rax\n\t"
"xor %rbx,%rax\n\t"
"mov %rax,%cr0\n\t"
"wbinvd\n\t" //flush
);
asm volatile( //invert bit
"mov $1,%rbx\n\t"
"shl $29,%rbx\n\t"
"mov %cr0,%rax\n\t"
"xor %rbx,%rax\n\t"
"mov %rax,%cr0\n\t"
);
asm volatile( //enable cache
"mov $1,%rbx\n\t"
"shl $30,%rbx\n\t"
"mov %cr0,%rax\n\t"
"xor %rbx,%rax\n\t" //xor : 1 => 0 , 0 => 1
"mov %rax,%cr0\n\t"
"wbinvd\n\t" //flush
);
asm volatile("pop %rbx\n\t" "pop %rax\n\t");
return 0;
}
but it doesn't work. dmesg give me :
[ 1190.301973] general protection fault: 0000 [#1] SMP
[ 1190.301975] Modules linked in: cache(POE+) ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables bnep rfcomm bluetooth 6lowpan_iphc binfmt_misc nls_iso8859_1 nvidia(POE) cp210x usbserial joydev snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap snd_hda_codec_realtek snd_hda_codec_generic intel_rapl snd_hda_intel x86_pkg_temp_thermal intel_powerclamp snd_hda_controller coretemp snd_hda_codec kvm_intel snd_hwdep snd_pcm kvm snd_seq_midi snd_seq_midi_event crct10dif_pclmul crc32_pclmul snd_rawmidi ghash_clmulni_intel aesni_intel snd_seq aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd nouveau serio_raw mxm_wmi ttm snd_seq_device snd_timer lpc_ich drm_kms_helper drm snd mei_me mei soundcore i2c_algo_bit shpchp wmi video mac_hid soc_button_array tpm_infineon parport_pc ppdev lp parport uas usb_storage hid_generic usbhid hid ahci r8169 psmouse libahci mii
[ 1190.302013] CPU: 5 PID: 5159 Comm: insmod Tainted: P OE 3.16.0-45-generic #60~14.04.1-Ubuntu
[ 1190.302014] Hardware name: ASUS All Series/Z87-A, BIOS 1007 05/17/2013
[ 1190.302015] task: ffff8807d95765e0 ti: ffff8807d95b0000 task.ti: ffff8807d95b0000
[ 1190.302016] RIP: 0010:[<ffffffffc0fd402a>] [<ffffffffc0fd402a>] init_module+0x2a/0x40 [cache]
[ 1190.302019] RSP: 0018:ffff8807d95b3d30 EFLAGS: 00010206
[ 1190.302019] RAX: 00000000a0050033 RBX: 0000000020000000 RCX: 0000000000000000
[ 1190.302020] RDX: ffff88081ed4ee40 RSI: ffff88081ed4d418 RDI: 0000000000000246
[ 1190.302021] RBP: ffff8807d95b3d40 R08: 0000000000000082 R09: 00000000000012e5
[ 1190.302022] R10: 0000000000000000 R11: ffff8807d95b3a5e R12: ffff8807ef817de0
[ 1190.302022] R13: 0000000000000000 R14: ffffffffc0fd4000 R15: ffffffffc0fd6000
[ 1190.302023] FS: 00007f405d04f740(0000) GS:ffff88081ed40000(0000) knlGS:0000000000000000
[ 1190.302024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1190.302025] CR2: 00007f405d26a248 CR3: 00000007e5d81000 CR4: 00000000001407e0
[ 1190.302026] Stack:
[ 1190.302026] ffffffff81c1a020 000000000000000d ffff8807d95b3db8 ffffffff81002144
[ 1190.302028] 0000000000000001 0000000000000001 0000000000000001 ffff8800dca9b440
[ 1190.302029] 0000000000000001 ffff8807d95b3da0 ffffffff8119d7d2 ffffffffc0fd6018
[ 1190.302030] Call Trace:
[ 1190.302035] [<ffffffff81002144>] do_one_initcall+0xd4/0x210
[ 1190.302037] [<ffffffff8119d7d2>] ? __vunmap+0xb2/0x100
[ 1190.302039] [<ffffffff810edd79>] load_module+0x13d9/0x1b90
[ 1190.302043] [<ffffffff810e9910>] ? store_uevent+0x40/0x40
[ 1190.302044] [<ffffffff810ee6a6>] SyS_finit_module+0x86/0xb0
[ 1190.302048] [<ffffffff8176e34d>] system_call_fastpath+0x1a/0x1f
[ 1190.302048] Code: <0f> 22 c0 5b 58 31 c0 5d c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00
[ 1190.302055] RIP [<ffffffffc0fd402a>] init_module+0x2a/0x40 [cache]
[ 1190.302056] RSP <ffff8807d95b3d30>
[ 1190.302057] ---[ end trace bf14887f4e905bad ]---
Do you know what is happening? doesn't it mean i can't change CR0.NW ?
My CPU : i7-4770K
According to the information provided by Intel's manual (See Table 11-5),
it is not allowed to set CR0.NW to 1 when CR0.CD is 0.
I assume you tried changing the write-policy (CR.NW=1) and using caches (CR.CD = 0) at the same time, which is invalid.
I found it when I was looking for the same thing as you did...
I'm referencing this answer for crash help in analyzing this bit of code which caused problems. The context for everyone, I'm working a character driver, which will act as a pass through from user space directly to the hardware, for the ahci driver. I'm modifying the ahci driver accordingly for this purpose.
I'm starting small. I want to peek at the port registers for the HBA port 0 of the AHCI HBA on my VM. My character driver ioctl code:
switch (cmd) {
case AHCIP_GPORT_REG:
pPciDev = pci_get_device(0x8086, 0x2829, NULL);
if (pPciDev) {
/* This will set ret to the value that it needs to be. This
* is true of __put_user() too */
if ((ret = __get_user(off, (u32*)obj))) {
printk(KERN_INFO "unable to read from user space\n");
goto ioctl_quick_out;
}
reg = get_port_reg(&pPciDev->dev, off);
if ((ret = __put_user(reg, (u32*)obj)))
{
printk(KERN_INFO "Unable to write to user space\n");
}
pci_dev_put(pPciDev);
}
// This break wasn't in the code when it crashed
break;
default:
// POSIX compliance with this one (REF of LDD3)
ret = -ENOTTY;
}
The code from my modified version of ahci.c which this character driver calls into:
u32 get_port_reg(struct device *dev, u32 off)
{
struct Scsi_Host *shost = class_to_shost(dev);
struct ata_port *ap = ata_shost_to_port(shost);
void __iomem *port_mmio = ahci_port_base(ap);
return ioread32(port_mmio + off);
}
EXPORT_SYMBOL(get_port_reg);
The kernel oops that this caused, happened here:
PID: 3357 TASK: ffff88011c9b7500 CPU: 0 COMMAND: "peek"
#0 [ffff8800abfc79f0] machine_kexec at ffffffff8103b5bb
#1 [ffff8800abfc7a50] crash_kexec at ffffffff810c9852
#2 [ffff8800abfc7b20] oops_end at ffffffff8152e0f0
#3 [ffff8800abfc7b50] no_context at ffffffff8104c80b
#4 [ffff8800abfc7ba0] __bad_area_nosemaphore at ffffffff8104ca95
#5 [ffff8800abfc7bf0] bad_area at ffffffff8104cbbe
#6 [ffff8800abfc7c20] __do_page_fault at ffffffff8104d36f
#7 [ffff8800abfc7d40] do_page_fault at ffffffff8153003e
#8 [ffff8800abfc7d70] page_fault at ffffffff8152d3f5
[exception RIP: get_port_reg+18]
RIP: ffffffffa03c4cd2 RSP: ffff8800abfc7e28 RFLAGS: 00010246
RAX: 0000000000020101 RBX: 00007fff17273960 RCX: ffffffff812b0710
RDX: ffff88011ddd5000 RSI: 0000000000000000 RDI: ffff88011ddd5090
RBP: ffff8800abfc7e28 R8: 0000000000000000 R9: 0000000000000000
R10: 00000000000007d5 R11: 0000000000000006 R12: ffff88011ddd5000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
As you can see, the instruction pointer was get_port_reg+18. Since this function is quite small, here's the full disassembly
crash> dis get_port_reg
0xffffffffa03c4cc0 <get_port_reg>: push %rbp
0xffffffffa03c4cc1 <get_port_reg+1>: mov %rsp,%rbp
0xffffffffa03c4cc4 <get_port_reg+4>: nopl 0x0(%rax,%rax,1)
0xffffffffa03c4cc9 <get_port_reg+9>: mov 0x240(%rdi),%rax
0xffffffffa03c4cd0 <get_port_reg+16>: mov %esi,%esi
0xffffffffa03c4cd2 <get_port_reg+18>: mov 0x2838(%rax),%rdx
0xffffffffa03c4cd9 <get_port_reg+25>: mov 0x28(%rax),%eax
0xffffffffa03c4cdc <get_port_reg+28>: mov 0x10(%rdx),%rdx
0xffffffffa03c4ce0 <get_port_reg+32>: shl $0x7,%eax
0xffffffffa03c4ce3 <get_port_reg+35>: mov %eax,%eax
0xffffffffa03c4ce5 <get_port_reg+37>: add 0x28(%rdx),%rax
0xffffffffa03c4ce9 <get_port_reg+41>: lea 0x100(%rax,%rsi,1),%rdi
0xffffffffa03c4cf1 <get_port_reg+49>: callq 0xffffffff8129dde0 <ioread32>
0xffffffffa03c4cf6 <get_port_reg+54>: leaveq
0xffffffffa03c4cf7 <get_port_reg+55>: retq
0xffffffffa03c4cf8 <get_port_reg+56>: nopl 0x0(%rax,%rax,1)
As you might have guessed, I'm something of an assembly neophyte. Which line of code would be get_port_reg+18? I'm puzzled because I'm calling functions on each line of that function but the only call I see is to ioread32().
For reference, I've modeled my function get_port_reg after ahci_show_port_cmd() within the same file. I could not think of any other means of getting the struct pci_dev structure necessary on which this is to operate. Am I making bad use of get_pci_device() and pci_dev_put()? Is this not the issue at all?
Thanks for any help
Andy
I am going to post my own answer. The two commentators of my question have put me onto the correct path for fixing this. As I mentioned, my approach was to do something which I'd seen done elsewhere in the ahci driver (ahci.c). Basically, the assumption was simple, this function in ahci.c required a struct device* and from that was able to get the ata_port information that was required. I'd seen, in ahci.c, that the author had done struct device* = &pdev->dev; occasionally. In other words, I figured that the dev member of struct pci_dev was getting me what I needed. I was apparently unaware of "class types" or something similar (see #myaut's first comment). #alexhoppus essentially draws the same/similar conclusion based on the code and disassembly which I posted.
The fix which I have employed, and which does work nicely, is as follows:
/* ioctl code in character driver */
switch (cmd) {
case AHCIP_GPORT_REG:
pPciDev = pci_get_device(0x8086, 0x2829, NULL);
if (pPciDev) {
struct ata_host *pHost = NULL;
struct ata_port *pPort = NULL;
printk(KERN_INFO "found the PCI device\n");
/* Get the devices driver data */
pHost = pci_get_drvdata(pPciDev);
if (!pHost) {
ret = -EFAULT;
goto ioctl_valid_pci_dev_out;
}
/* for this test, we'll use just port 0 */
pPort = pHost->ports[0];
if (!pPort) {
ret = -EFAULT;
goto ioctl_valid_pci_dev_out;
}
/* This will set ret to the value that it needs to be. This
* is true of __put_user() too */
if ((ret = __get_user(off, (u32*)obj))) {
printk(KERN_INFO "unable to read from user space\n");
goto ioctl_valid_pci_dev_out;
}
reg = get_port_reg(pPort, off);
if ((ret = __put_user(reg, (u32*)obj)))
{
printk(KERN_INFO "Unable to write to user space\n");
}
}
break;
default:
// POSIX compliance with this one (REF of LDD3)
ret = -ENOTTY;
}
The ahci driver was modified thusly as well
u32 get_port_reg(struct ata_port* pPort, u32 off)
{
void __iomem *port_mmio = ahci_port_base(pPort);
return ioread32(port_mmio + off);
}
EXPORT_SYMBOL(get_port_reg);
Though this has fixed the issue for me, I would really appreciate someone explaining to me what is placed in (struct pci_dev)device.dev.p->driver_data. I can use, and have, the Linux cross referencing tools to see the data types. What is supposed to be stored instruct device_private`? This is the structure which I'm now using to get the data I need. I'd truly appreciate someone commenting on this answer to explain that one.
Thanks to #myaut and #alexhoppus
I have a kernel module for splitting incoming rtp packets and merging rtp outgoing packets. The program crashes once in 2/3 days. If would be very convenient for me if its possible to find the exact line where the module crashes.
I have given the crash dump below. Is it possible to find the exact line in the code from crash dump?
PID: 1256 TASK: ffff88020fc71700 CPU: 0 COMMAND: "rtpproxy"
#0 [ffff880212faf2f0] machine_kexec at ffffffff8103bb7a
#1 [ffff880212faf360] crash_kexec at ffffffff810bb968
#2 [ffff880212faf430] oops_end at ffffffff8169fad8
#3 [ffff880212faf460] die at ffffffff81017808
#4 [ffff880212faf490] do_general_protection at ffffffff8169f5d2
#5 [ffff880212faf4c0] general_protection at ffffffff8169eef5
[exception RIP: pkt_queue+388]
RIP: ffffffffa00f3fa0 RSP: ffff880212faf578 RFLAGS: 00010292
RAX: ffff8802110ae400 RBX: ffff880213a53f38 RCX: 00015d910000a20f
RDX: 497d74565cede60c RSI: 000000006df1ed57 RDI: 00000000e46e0cfc
RBP: ffff880212faf728 R8: ffff880211a8b000 R9: ffff880212fafa60
R10: ffff880212fafbc8 R11: 0000000000000293 R12: 00000000134ab2b4
R13: 000000008386615c R14: 00000000000000e3 R15: 00000000000000e3
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#6 [ffff880212faf730] obsf_tg at ffffffffa00f34a0 [xt_OBSF]
#7 [ffff880212faf890] ipt_do_table at ffffffffa00e41a5 [ip_tables]
#8 [ffff880212faf970] ipt_mangle_out at ffffffffa00dd129 [iptable_mangle]
#9 [ffff880212faf9c0] iptable_mangle_hook at ffffffffa00dd1eb [iptable_mangle]
#10 [ffff880212faf9d0] nf_iterate at ffffffff815aded5
#11 [ffff880212fafa20] nf_hook_slow at ffffffff815adf85
#12 [ffff880212fafaa0] __ip_local_out at ffffffff815babb2
#13 [ffff880212fafac0] ip_local_out at ffffffff815babd6
#14 [ffff880212fafae0] ip_send_skb at ffffffff815bbefb
#15 [ffff880212fafb00] udp_send_skb at ffffffff815df1d1
#16 [ffff880212fafb50] udp_sendmsg at ffffffff815e0286
#17 [ffff880212fafc90] inet_sendmsg at ffffffff815eabc4
#18 [ffff880212fafcd0] sock_sendmsg at ffffffff8156a437
#19 [ffff880212fafe50] sys_sendto at ffffffff8156d91d
#20 [ffff880212faff80] system_call_fastpath at ffffffff816a7029
RIP: 00007f17363b83a3 RSP: 00007ffff2965f90 RFLAGS: 00010213
RAX: 000000000000002c RBX: ffffffff816a7029 RCX: 00007ffff29ff99b
RDX: 0000000000000020 RSI: 00007f1737da4378 RDI: 0000000000000006
RBP: 0000000000000001 R8: 00007f1737da67a0 R9: 0000000000000010
R10: 0000000000000000 R11: 0000000000000293 R12: 00007f1737da4378
R13: 0000000000000001 R14: 00007f1737da42a0 R15: 0000000000000000
ORIG_RAX: 000000000000002c CS: 0033 SS: 002b
[157707.736203] general protection fault: 0000 [#1] SMP
[157707.736955] CPU 0
[157707.736973] Modules linked in:
[157707.737654] arc4 xt_tcpudp xt_OBSF(O) iptable_mangle ip_tables x_tables ghash_clmulni_intel aesni_intel cryptd aes_x86_64 joydev hid_generic microcode ext2 usbhid psmouse hid serio_raw i2c_piix4 virtio_balloon lp parport mac_hid floppy
[157707.740018]
[157707.740102] Pid: 1256, comm: rtpproxy Tainted: G O 3.5.0-23-generic #35~precise1-Ubuntu Bochs Bochs
[157707.740102] RIP: 0010:[<ffffffffa00f3fa0>] [<ffffffffa00f3fa0>] pkt_queue+0x184/0x48a [xt_OBSF]
[157707.740102] RSP: 0018:ffff880212faf578 EFLAGS: 00010292
[157707.740102] RAX: ffff8802110ae400 RBX: ffff880213a53f38 RCX: 00015d910000a20f
[157707.740102] RDX: 497d74565cede60c RSI: 000000006df1ed57 RDI: 00000000e46e0cfc
[157707.740102] RBP: ffff880212faf728 R08: ffff880211a8b000 R09: ffff880212fafa60
[157707.740102] R10: ffff880212fafbc8 R11: 0000000000000293 R12: 00000000134ab2b4
[157707.740102] R13: 000000008386615c R14: 00000000000000e3 R15: 00000000000000e3
[157707.740102] FS: 00007f1736ad9700(0000) GS:ffff88021fc00000(0000) knlGS:0000000000000000
[157707.740102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[157707.740102] CR2: 00007fd8a39f8000 CR3: 0000000211ad7000 CR4: 00000000000407f0
[157707.740102] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[157707.740102] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[157707.740102] Process rtpproxy (pid: 1256, threadinfo ffff880212fae000, task ffff88020fc71700)
[157707.740102] Stack:
[157707.740102] ffff880212faf5a8 0000000000015d91 134ab2b400000008 000008f58386615c
[157707.740102] 00015d910000a20f a080527800000014 3a78560000d1fa00 564812de1a006045
[157707.740102] ffff880212faf618 ffffffff81872e20 0000000000000000 ffff880210ca9000
[157707.740102] Call Trace:
[157707.740102] [<ffffffff8169e7de>] ? _raw_spin_lock+0xe/0x20
[157707.740102] [<ffffffff815a0958>] ? sch_direct_xmit+0x88/0x1c0
[157707.740102] [<ffffffff81090833>] ? update_cpu_power+0x63/0x100
[157707.740102] [<ffffffff810909c3>] ? update_group_power+0xf3/0x100
[157707.740102] [<ffffffff81090db2>] ? update_sd_lb_stats+0x3e2/0x5f0
[157707.740102] [<ffffffffa00f34a0>] obsf_tg+0x9c0/0x133c [xt_OBSF]
[157707.740102] [<ffffffff81090ff9>] ? find_busiest_group+0x39/0x4a0
[157707.740102] [<ffffffff81091541>] ? load_balance+0xe1/0x4a0
[157707.740102] [<ffffffffa00e41a5>] ipt_do_table+0x315/0x450 [ip_tables]
[157707.740102] [<ffffffffa00dd129>] ipt_mangle_out+0x99/0x100 [iptable_mangle]
[157707.740102] [<ffffffffa00dd1eb>] iptable_mangle_hook+0x5b/0x60 [iptable_mangle]
[157707.740102] [<ffffffff815aded5>] nf_iterate+0x85/0xc0
[157707.740102] [<ffffffff815b8e50>] ? ip_forward_options+0x200/0x200
[157707.740102] [<ffffffff815adf85>] nf_hook_slow+0x75/0x150
[157707.740102] [<ffffffff815b8e50>] ? ip_forward_options+0x200/0x200
[157707.740102] [<ffffffff815babb2>] __ip_local_out+0xa2/0xb0
[157707.740102] [<ffffffff815babd6>] ip_local_out+0x16/0x30
[157707.740102] [<ffffffff815bbefb>] ip_send_skb+0x1b/0x50
[157707.740102] [<ffffffff815df1d1>] udp_send_skb+0x111/0x2a0
[157707.740102] [<ffffffff815b9070>] ? ip_setup_cork+0x150/0x150
[157707.740102] [<ffffffff815e0286>] udp_sendmsg+0x316/0x960
[157707.740102] [<ffffffff815eabc4>] inet_sendmsg+0x64/0xb0
[157707.740102] [<ffffffff812f31b7>] ? apparmor_socket_sendmsg+0x17/0x20
[157707.740102] [<ffffffff8156a437>] sock_sendmsg+0x117/0x130
[157707.740102] [<ffffffff8119a510>] ? __pollwait+0xf0/0xf0
[157707.740102] [<ffffffff8119a510>] ? __pollwait+0xf0/0xf0
[157707.740102] [<ffffffff8119a510>] ? __pollwait+0xf0/0xf0
[157707.740102] [<ffffffff8156b58d>] ? move_addr_to_user+0xbd/0xd0
[157707.740102] [<ffffffff8156ce7a>] ? move_addr_to_kernel+0x5a/0xa0
[157707.740102] [<ffffffff8156d91d>] sys_sendto+0x13d/0x190
[157707.740102] [<ffffffff8103fcc9>] ? kvm_clock_read+0x19/0x20
[157707.740102] [<ffffffff8103fcd9>] ? kvm_clock_get_cycles+0x9/0x10
[157707.740102] [<ffffffff810a3bd7>] ? getnstimeofday+0x57/0xe0
[157707.740102] [<ffffffff810a3cca>] ? do_gettimeofday+0x1a/0x50
[157707.740102] [<ffffffff816a7029>] system_call_fastpath+0x16/0x1b
[157707.740102] Code: f7 f1 48 8b 8d 70 fe ff ff 4c 63 f2 41 89 d7 49 69 c6 68 01 00 00 48 01 c3 48 8b 83 58 01 00 00 48 2d 58 01 00 00 48 89 c2 eb 20 <44> 39 62 04 0f 85 c0 02 00 00 44 39 6a 08 0f 85 b6 02 00 00 48
[157707.740102] RIP [<ffffffffa00f3fa0>] pkt_queue+0x184/0x48a [xt_OBSF]
[157707.740102] RSP <ffff880212faf578>
[157707.736203] general protection fault: 0000 [#1] SMP
Says that you are doing something horrible in memory (e.g dereferencing a null pointer)
[157707.740102] RIP: 0010:[<ffffffffa00f3fa0>] [<ffffffffa00f3fa0>] pkt_queue+0x184/0x48a
This line is reporting to you the instruction pointer value when your module crashed; it says that it died inside a function named "pkt_queue" after an offset of "0x184".
(btw, the same value appears in the first crash dump, 388 in decimal = 0x184)
Now, you can use objdump to dump the assembly + debug information about your code and you add the address of the function pkt_queue to 0x184 and you get to the offending instruction.
Let's say your pkt_queue function appears(unreasonably hypothetical) at address 0x01 in objdump, it means you should look at line: 0x184 + 0x01 = 0x185 in the assembly to see what's going on.
Objdump allows you view the source + the assembly and line numbers:
objdump -S your_object_file.o this will not only list the assembly but also the corresponding source code assuming the debug symbols are added when compiling.
Oh and for your future reference:https://opensourceforu.com/2011/01/understanding-a-kernel-oops/
You can also use:
eu-addr2line -f -e object_file.o pkt_queue+0x184
Where -f tells the command that the function name is used with line number and -e is the executable or object file containing the line number.
there is also the script scripts/decode_stacktrace.sh in the kernel source code.
You should enable CONFIG_DEBUG_INFO then run the script:
./scripts/decode_stacktrace.sh /path/to/vmlinux /path/to/kernel/tree /path/to/modules/dir < dmesg.log
for example starting in the kernel source code root:
make O=~/kbuild/x86/ -j9
cd ~/kbuild/x86/
make INSTALL_MOD_PATH=~/modpath modules_install
cd -
./scripts/decode_stacktrace.sh ~/kbuild/x86/vmlinux . ~/modpath < crash.log
see https://lwn.net/Articles/592724/