I make performance measurements on the Raspberry Pi and would like to have higher resolution then just clock_gettime().
Therefor I need to enable the CPU register PMUSERENR.
According to /proc/cpuinfo on my Raspberry Pi I have an "ARMv7 Processor rev 5 (v7l)". So I downloaded the architecture reference manual and found that I have to use:
CRn = c9, opc1 = 0, CRm = c14, opc2 = 0
asm ("MCR p15, 0, %0, C9, C14, 0\n\t" :: "r"(1));
But when using this code in a kernel module, I get following oops:
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.604846] Internal error: Oops: 5 [#1] SMP ARM
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.715199] Process insmod (pid: 2944, stack limit = 0xac80e210)
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.722399] Stack: (0xac80fe90 to 0xac810000)
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.727920] fe80: 7f1ba0f8 00007fff 80098128 ac80fea8
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.738472] fea0: 80578128 0000002d 00000000 7f1ba0f8 7f1ba0f8 7f1ba26c 7f1ba264 7f1ba134
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.749092] fec0: ac80e000 8057608c ac80feec ac80fed8 8057608c 00000000 00000000 00000000
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.759724] fee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
pi#raspberrypi ~/projects/timer $
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.770389] ff00: 00000000 00000000 00000000 00000000 76f5e948 00007390 00000000 76f7b390
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.781147] ff20: 76f5e948 b91a2390 ac80e000 00000000 ac80ffa4 ac80ff40 8009b6c4 80099714
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.792000] ff40: b919b000 00007390 b91a0598 b91a0457 b91a22c8 0000026c 000002ac 00000000
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.802930] ff60: 00000000 00000000 0000001f 00000020 0000000c 0000000a 00000009 00000000
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.813947] ff80: 7ea4371c 00000000 55afc028 00000080 8000fc28 ac80e000 00000000 ac80ffa8
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.825056] ffa0: 8000fa20 8009b5f0 7ea4371c 00000000 76f74000 00007390 76f5e948 76f74000
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.836170] ffc0: 7ea4371c 00000000 55afc028 00000080 55afbf78 00007390 76f5e948 00000000
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.847338] ffe0: 00000000 7ea436c4 76f55fb4 76ec0534 60000010 76f74000 00000000 00000000
Message from syslogd#raspberrypi at Apr 29 20:58:12 ...
kernel:[ 84.880715] Code: e51b3084 e1530005 e2455008 0a000009 (e5953014)
I am new to kernel modules, so I do not know which direction I should go. Is there a good way to debug such things?
Linux kernel comes with a PMU driver. You could just leverage that. I would write a tutorial for you, but why reinvent the wheel when this guy explains it so well:
http://www.carbondesignsystems.com/virtual-prototype-blog/using-the-arm-performance-monitor-unit-pmu-linux-driver
Related
I am trying to start an Oracle 11g database but it is failing with ORA-01092 and ORA-00600 errors:
Microsoft Windows [Version 6.0.6001]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.
C:\Users\Administrator>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Sep 11 15:21:30 2019
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup upgrade
ORACLE instance started.
Total System Global Area 430075904 bytes
Fixed Size 2176448 bytes
Variable Size 356518464 bytes
Database Buffers 67108864 bytes
Redo Buffers 4272128 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 5044
Session ID: 1 Serial number: 5
SQL> conn
Enter user-name: delhipilot
Enter password:
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Process ID: 0
Session ID: 0 Serial number: 0
SQL>
How can I start my database properly?
Here is an example of patching the system rollback segment header to avoid errors ORA-600 [4193] and ORA-600 [4194] during startup. Note that in this example the segment header is located in file 1 block 9 and the example in note 452620.1 is using file 1 block 2 as the segment header.
parnassusdata can also provide the recovery service.
It is a partial block dump for system rbs segment header file 1 block 9:
TRN CTL:: seq: 0x003a chd: 0x0017 ctl: 0x0052 inc: 0x00000000 nfb: 0x0001
mgc: 0x8002 xts: 0x0068 flg: 0x0001 opt: 2147483646 (0x7ffffffe)
uba: 0x00400197.003a.02 scn: 0x0000.004fbbf0
Version: 0x01
FREE BLOCK POOL::
uba: 0x00400197.003a.02 ext: 0x4 spc: 0x1dd2
uba: 0x00000000.0037.05 ext: 0x1 spc: 0x1d6c
uba: 0x00000000.0035.37 ext: 0x5 spc: 0x538
uba: 0x00000000.0000.00 ext: 0x0 spc: 0x0
1. Generate the bbed executable:
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk `pwd`/bbed
mv bbed $ORACLE_HOME/bin
2. Create file file.lis with the datafile where the system rollback segment header is stored:
file.lis has:
<relative file#> <datafile name> <size in bytes: v$datafile.bytes>
In our session file.lis contains:
1 /oradata/s102/system01.dbf 524288000
3. Create file bbed.par
bbed.par has:
MODE=EDIT
LISTFILE=<File name created in step2>
BLOCKSIZE=<db_block_size>
In our session bbed.par contains
MODE=EDIT
LISTFILE=file.lis
BLOCKSIZE=8192
4. Run bbed. Use password blockedit:
$ bbed parfile=bbed.par
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Thu Sep 27 10:06:25 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED>
5. Go to Block where the system rollback segment header is stored. In our example it is block 9:
BBED> set block 9
BLOCK# 9
6. Run map to see the C structures for the block and the DBA:
BBED> map
File: /oradata/s102/system01.dbf (1)
Block: 9 Dba:0x00400009
------------------------------------------------------------
Unlimited Undo Segment Header
struct kcbh, 20 bytes #0
struct ktech, 72 bytes #20
struct ktemh, 16 bytes #92
struct ktetb[6], 48 bytes #108
struct ktuxc, 104 bytes #4148
struct ktuxe[255], 10200 bytes #4252
ub4 tailchk #8188
Note that dba=0x00400009 is file 1 block 9, so we are positioned in the correct block.
7. Print the structure ktuxc:
BBED> print ktuxc
struct ktuxc, 104 bytes #4148
struct ktuxcscn, 8 bytes #4148
ub4 kscnbas #4148 0x004fbbf1
ub2 kscnwrp #4152 0x0000
struct ktuxcuba, 8 bytes #4156
ub4 kubadba #4156 0x00400197
ub2 kubaseq #4160 0x003a
ub1 kubarec #4162 0x03
sb2 ktuxcflg #4164 1 (KTUXCFSK)
ub2 ktuxcseq #4166 0x003a
sb2 ktuxcnfb #4168 1
ub4 ktuxcinc #4172 0x00000000
sb2 ktuxcchd #4176 6
sb2 ktuxcctl #4178 23
ub2 ktuxcmgc #4180 0x8002
ub4 ktuxcopt #4188 0x7ffffffe
struct ktuxcfbp[0], 12 bytes #4192
struct ktufbuba, 8 bytes #4192
ub4 kubadba #4192 0x00400197
ub2 kubaseq #4196 0x003a
ub1 kubarec #4198 0x0c
sb2 ktufbext #4200 4
sb2 ktufbspc #4202 5630
8. Modify ktuxc.ktuxcnfb to 0x0000
BBED> set offset ktuxc.ktuxcnfb
OFFSET 4168
BBED> print
ktuxc.ktuxcnfb
--------------
sb2 ktuxcnfb #4168 1
BBED> modify 0x0000
File: /oradata/s102/system01.dbf (1)
Block: 9 Offsets: 4168 to 4679 Dba:0x00400009
------------------------------------------------------------------------
00000000 00000000 06001700 02800100 68000000 feffff7f 97014000 3a000c00
0400fe15 00000000 37000500 01006c1d 00000000 35003700 05003805 00000000
00000000 00000000 00000000 00000000 00000000 30000000 93014000 191f5300
00000000 09005f00 00000000 00000000 00000000 01000000 00000000 31000000
96014000 a03e5b00 00000000 09005c00 00000000 00000000 00000000 01000000
00000000 31000000 96014000 9e3e5b00 00000000 09000e00 00000000 00000000
00000000 01000000 00000000 30000000 93014000 f4bb4f00 00000000 09001600
00000000 00000000 00000000 01000000 00000000 31000000 96014000 c13a5b00
00000000 09004800 00000000 00000000 00000000 01000000 00000000 31000000
96014000 983e5b00 00000000 09006000 00000000 00000000 00000000 01000000
00000000 30000000 93014000 f2bb4f00 00000000 09001400 00000000 00000000
00000000 01000000 00000000 31000000 96014000 933e5b00 00000000 09006100
00000000 00000000 00000000 01000000 00000000 31000000 96014000 8d3e5b00
00000000 09004700 00000000 00000000 00000000 01000000 00000000 30000000
94014000 87d15900 00000000 09002100 00000000 00000000 00000000 01000000
00000000 30000000 94014000 211f5300 00000000 09001d00 00000000 00000000
<32 bytes per line>
9. Modify ktuxc.ktuxcfbp[0].ktufbuba to 0x00000000
BBED> set offset ktuxc.ktuxcfbp[0].ktufbuba
OFFSET 4192
BBED> print
ktuxc.ktuxcfbp[0].ktufbuba.kubadba
----------------------------------
ub4 kubadba #4192 0x00400197
BBED> modify 0x00000000
File: /oradata/s102/system01.dbf (1)
Block: 9 Offsets: 4192 to 4703 Dba:0x00400009
------------------------------------------------------------------------
00000000 3a000c00 0400fe15 00000000 37000500 01006c1d 00000000 35003700
05003805 00000000 00000000 00000000 00000000 00000000 00000000 30000000
93014000 191f5300 00000000 09005f00 00000000 00000000 00000000 01000000
00000000 31000000 96014000 a03e5b00 00000000 09005c00 00000000 00000000
00000000 01000000 00000000 31000000 96014000 9e3e5b00 00000000 09000e00
00000000 00000000 00000000 01000000 00000000 30000000 93014000 f4bb4f00
00000000 09001600 00000000 00000000 00000000 01000000 00000000 31000000
96014000 c13a5b00 00000000 09004800 00000000 00000000 00000000 01000000
00000000 31000000 96014000 983e5b00 00000000 09006000 00000000 00000000
00000000 01000000 00000000 30000000 93014000 f2bb4f00 00000000 09001400
00000000 00000000 00000000 01000000 00000000 31000000 96014000 933e5b00
00000000 09006100 00000000 00000000 00000000 01000000 00000000 31000000
96014000 8d3e5b00 00000000 09004700 00000000 00000000 00000000 01000000
00000000 30000000 94014000 87d15900 00000000 09002100 00000000 00000000
00000000 01000000 00000000 30000000 94014000 211f5300 00000000 09001d00
00000000 00000000 00000000 01000000 00000000 30000000 93014000 0d1f5300
<32 bytes per line>
BBED>
10. Disable the block Checksum by changing the kcbh.flg_kcbh-4 and kcbh.chkval_kcbh to 0x0000:
BBED> map
File: /oradata/s102/system01.dbf (1)
Block: 9 Dba:0x00400009
------------------------------------------------------------
Unlimited Undo Segment Header
struct kcbh, 20 bytes #0
struct ktech, 72 bytes #20
struct ktemh, 16 bytes #92
struct ktetb[6], 48 bytes #108
struct ktuxc, 104 bytes #4148
struct ktuxe[255], 10200 bytes #4252
ub4 tailchk #8188
BBED> print kcbh
struct kcbh, 20 bytes #0
ub1 type_kcbh #0 0x0e
ub1 frmt_kcbh #1 0xa2
ub1 spare1_kcbh #2 0x00
ub1 spare2_kcbh #3 0x00
ub4 rdba_kcbh #4 0x00400009
ub4 bas_kcbh #8 0x005b3f76
ub2 wrp_kcbh #12 0x0000
ub1 seq_kcbh #14 0x01
ub1 flg_kcbh #15 0x04 (KCBHFCKV)
ub2 chkval_kcbh #16 0xe264
ub2 spare3_kcbh #18 0x0000
BBED> set offset kcbh.flg_kcbh
OFFSET 15
BBED> print
kcbh.flg_kcbh
-------------
ub1 flg_kcbh #15 0x04 (KCBHFCKV)
BBED> modify 0x00
File: /oradata/s102/system01.dbf (1)
Block: 9 Offsets: 15 to 526 Dba:0x00400009
------------------------------------------------------------------------
0064e200 00000000 00000000 00000000 00000000 00060000 002f0000 00201000
00040000 00060000 00080000 00970140 00000000 00040000 00000000 00000000
00000000 00000000 00000000 00060000 00000000 00000000 00000000 400a0040
00070000 00110040 00080000 00810140 00080000 00890140 00080000 00910140
00080000 00990140 00080000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<32 bytes per line>
BBED> set offset kcbh.chkval_kcbh
OFFSET 16
BBED> print
kcbh.chkval_kcbh
----------------
ub2 chkval_kcbh #16 0xe264
BBED> modify 0x0000
File: /oradata/s102/system01.dbf (1)
Block: 9 Offsets: 16 to 527 Dba:0x00400009
------------------------------------------------------------------------
00000000 00000000 00000000 00000000 00000000 06000000 2f000000 20100000
04000000 06000000 08000000 97014000 00000000 04000000 00000000 00000000
00000000 00000000 00000000 06000000 00000000 00000000 00000040 0a004000
07000000 11004000 08000000 81014000 08000000 89014000 08000000 91014000
08000000 99014000 08000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<32 bytes per line>
11. Verify the the block has no corruptions:
BBED> verify
DBVERIFY - Verification starting
FILE = /oradata/s102/system01.dbf
BLOCK = 9
DBVERIFY - Verification complete
Total Blocks Examined : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing (Index): 0
Total Blocks Empty : 0
Total Blocks Marked Corrupt : 0
Total Blocks Influx : 0
12. exit, open the database and shrink the system rollback segment:
BBED> exit
[oracle#arem example]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.3.0 - Production on Thu Sep 27 10:28:00 2007
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 167772160 bytes
Fixed Size 1260696 bytes
Variable Size 62915432 bytes
Database Buffers 100663296 bytes
Redo Buffers 2932736 bytes
Database mounted.
Database opened.
SQL> alter rollback segment system shrink;
Rollback segment altered.
SQL>
We have setup procdump as the (AeDebug) postmortem debugger to capture dumps of unhandled exceptions. The registry key is set to "c:\my\sysinternals\procdump.exe" -accepteula -ma -j "c:\dumps" %ld %ld %p
Currently I'm looking at a dump where the process that triggered a crash dump is still running hours after dumping the process has completed?!
It was my assumption that any process that triggers a crash dump is about to get terminated?
From WinDbg
Debug session time: Tue Dec 1 01:53:06.000 2015 (UTC + 1:00)
System Uptime: 18 days 18:09:24.556
Process Uptime: 1 days 0:09:31.000
0:000> ~
. 0 Id: 178c.eb4 Suspend: 0 Teb: fffdd000 Unfrozen
...
0:000> ? 178c
Evaluate expression: 6028 = 0000178c
From ProcExp
Looking at PID 6028 on Tue Dec 1 10:00:00, hours after the dump from 01:53:06.000
<3thParty>.exe:6028 Properties
Started: 1:43:35 30/11/2015
Edit
0:000> ~*kb1000
. 0 Id: 178c.eb4 Suspend: 0 Teb: fffdd000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0018de38 77cd8567 00000574 00000001 00000000 ntdll!ZwWaitForSingleObject+0x15
01 0018debc 77cd8695 0018e0c8 0018e118 00000001 ntdll!RtlReportExceptionEx+0x14b
02 0018df14 772f8dd8 0018e0c8 0018e118 00000001 ntdll!RtlReportException+0x86
03 0018df34 772f8edb 0018dfa4 0018dfa4 00000000 ole32!SilentlyReportExceptions+0x79
04 0018df48 772f94e5 0018dfa4 00000000 00000000 ole32!ServerExceptionFilter+0xbb
05 0018df60 77360827 0018dfa4 0148502c 01048240 ole32!AppInvokeExceptionFilterWithMethodAddress+0x11
06 0018df7c 77183f21 00000000 0018faf8 77278760 ole32!SyncStubInvoke+0x82
07 0018df90 77183eae 00000000 00000000 00000000 msvcrt!_EH4_CallFilterFunc+0x12
08 0018dfbc 77286771 7736662c 7726ea10 0018e0c8 msvcrt!_except_handler4_common+0x8e
09 0018dfdc 77c8b499 0018e0c8 0018fae8 0018e118 ole32!_except_handler4+0x20
0a 0018e000 77c8b46b 0018e0c8 0018fae8 0018e118 ntdll!ExecuteHandler2+0x26
0b 0018e024 77c8b40e 0018e0c8 0018fae8 0018e118 ntdll!ExecuteHandler+0x24
0c 0018e0b0 77c40133 0118e0c8 0018e118 0018e0c8 ntdll!RtlDispatchException+0x127
0d 0018e0b0 00404880 0118e0c8 0018e118 0018e0c8 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
0e 0018f61c 75a15932 01485018 01549420 00000000 <3thParty>+0x4880
0f 0018f640 75a905f1 00523859 0018f828 00000004 rpcrt4!Invoke+0x2a
10 0018fa44 7735aec1 010ac688 0101fb58 0104ea78 rpcrt4!NdrStubCall2+0x2ea
11 0018fa8c 76acffd3 010ac688 0104ea78 0101fb58 ole32!CStdStubBuffer_Invoke+0x3c
12 0018fab0 7735d876 010ac688 0104ea78 0101fb58 oleaut32!CUnivStubWrapper::Invoke+0xcb
13 0018faf8 7735ddd0 0104ea78 01048240 01095448 ole32!SyncStubInvoke+0x3c
14 0018fb44 77278a43 0104ea78 010be578 01097470 ole32!StubInvoke+0xb9
15 0018fc20 77278938 0101fb58 00000000 01097470 ole32!CCtxComChnl::ContextInvoke+0xfa
16 0018fc3c 7727950a 0104ea78 00000001 01097470 ole32!MTAInvoke+0x1a
17 0018fc68 7735dccd 0104ea78 00000001 01097470 ole32!STAInvoke+0x46
18 0018fc9c 7735db41 d0908070 0101fb58 01097470 ole32!AppInvoke+0xab
19 0018fd7c 7735e1fd 0104ea20 0101fef0 00000400 ole32!ComInvokeWithLockAndIPID+0x372
1a 0018fda4 77279367 0104ea20 00000400 00ff3548 ole32!ComInvoke+0xc5
1b 0018fdb8 77279326 0104ea20 00000000 77279286 ole32!ThreadDispatch+0x23
1c 0018fdfc 76eb62fa 00e2019e 00000400 0000babe ole32!ThreadWndProc+0x161
1d 0018fe28 76eb6d3a 77279286 00e2019e 00000400 user32!InternalCallWinProc+0x23
1e 0018fea0 76eb77c4 00000000 77279286 00e2019e user32!UserCallWinProcCheckWow+0x109
1f 0018ff00 76eb788a 77279286 00000000 00e2019e user32!DispatchMessageWorker+0x3bc
20 0018ff10 004aea6a 0018ff34 00010001 0018ff70 user32!DispatchMessageW+0xf
21 0018ff2c 004aeaaf 00e2019e 00000400 0000babe <3thParty>+0xaea6a
22 0018ff50 00a0c705 0018ff78 00a0c733 0018ff70 <3thParty>+0xaeaaf
23 0018ff70 00a3024b 0018ffc4 00404cac 0018ff88 <3thParty>!FrobWidget+0x4db279
24 0018ff88 7766338a fffde000 0018ffd4 77c69f72 <3thParty>!FrobWidget+0x4fedbf
25 0018ff94 77c69f72 fffde000 2ca36a95 00000000 kernel32!BaseThreadInitThunk+0xe
26 0018ffd4 77c69f45 00a30220 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
27 0018ffec 00000000 00a30220 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
1 Id: 178c.17a4 Suspend: 0 Teb: fffda000 Unfrozen
# ChildEBP RetAddr Args to Child
00 023cfed4 76b614ab 00000180 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
01 023cff40 77661194 00000180 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
02 023cff58 77661148 00000180 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
03 023cff6c 76bb511d 00000180 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
04 023cff88 7766338a 00234b88 023cffd4 77c69f72 ws2_32!SockAsyncThread+0x25
05 023cff94 77c69f72 00234b88 2e876a95 00000000 kernel32!BaseThreadInitThunk+0xe
06 023cffd4 77c69f45 76bb50f8 00234b88 ffffffff ntdll!__RtlUserThreadStart+0x70
07 023cffec 00000000 76bb50f8 00234b88 00000000 ntdll!_RtlUserThreadStart+0x1b
2 Id: 178c.1108 Suspend: 0 Teb: fffd7000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0273fdf4 77c82f91 00000003 010169d0 00000001 ntdll!NtWaitForMultipleObjects+0x15
01 0273ff88 7766338a 00000000 0273ffd4 77c69f72 ntdll!TppWaiterpThread+0x33d
02 0273ff94 77c69f72 010169a0 2ec86a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0273ffd4 77c69f45 77c82e65 010169a0 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0273ffec 00000000 77c82e65 010169a0 00000000 ntdll!_RtlUserThreadStart+0x1b
3 Id: 178c.1794 Suspend: 0 Teb: fffac000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0908fe28 77c83392 000001c0 0908fedc 25b36ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 0908ff88 7766338a 01015748 0908ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 0908ff94 77c69f72 01015748 25b36a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0908ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0908ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b
4 Id: 178c.5e4 Suspend: 0 Teb: fffaf000 Unfrozen
# ChildEBP RetAddr Args to Child
00 08b0fe28 77c83392 000001c4 08b0fedc 240b6ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 08b0ff88 7766338a 01015748 08b0ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 08b0ff94 77c69f72 01015748 240b6a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 08b0ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 08b0ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b
5 Id: 178c.c80 Suspend: 0 Teb: fffa9000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0b03fed8 76b63bd5 00000000 0b03ff1c 57962de4 ntdll!ZwDelayExecution+0x15
01 0b03ff40 76b644a5 0000ea60 00000000 0b03ff78 KERNELBASE!SleepEx+0x65
02 0b03ff50 7724d98d 0000ea60 010bd460 7724cd48 KERNELBASE!Sleep+0xf
03 0b03ff5c 7724cd48 00000000 00000000 010bd460 ole32!CROIDTable::WorkerThreadLoop+0x14
04 0b03ff78 7724d87a 00000000 00000000 0b03ff94 ole32!CRpcThread::WorkerLoop+0x26
05 0b03ff88 7766338a 010bd460 0b03ffd4 77c69f72 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x16
06 0b03ff94 77c69f72 010bd460 27b86a95 00000000 kernel32!BaseThreadInitThunk+0xe
07 0b03ffd4 77c69f45 7724d864 010bd460 ffffffff ntdll!__RtlUserThreadStart+0x70
08 0b03ffec 00000000 7724d864 010bd460 00000000 ntdll!_RtlUserThreadStart+0x1b
6 Id: 178c.14f4 Suspend: 0 Teb: fffa6000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0b13fe28 77c83392 000001bc 0b13fedc 27a86ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 0b13ff88 7766338a 01015748 0b13ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 0b13ff94 77c69f72 01015748 27a86a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0b13ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0b13ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b
7 Id: 178c.ee4 Suspend: 0 Teb: fffa3000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0b23fcbc 76b614ab 00000578 00000000 0b23fd04 ntdll!ZwWaitForSingleObject+0x15
01 0b23fd28 77661194 00000578 00002710 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
02 0b23fd40 77661148 00000578 00002710 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
03 0b23fd54 6e22b3d6 00000578 00002710 00000000 kernel32!WaitForSingleObject+0x12
04 0b23ff88 7766338a 01072188 0b23ffd4 77c69f72 comsvcs!PingThread+0x131
05 0b23ff94 77c69f72 01072188 27986a95 00000000 kernel32!BaseThreadInitThunk+0xe
06 0b23ffd4 77c69f45 6e22b320 01072188 ffffffff ntdll!__RtlUserThreadStart+0x70
07 0b23ffec 00000000 6e22b320 01072188 00000000 ntdll!_RtlUserThreadStart+0x1b
8 Id: 178c.914 Suspend: 0 Teb: fff9d000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0d26ff5c 72be635c 00000534 0d26ff90 0d26ff84 ntdll!ZwRemoveIoCompletion+0x15
01 0d26ff88 7766338a 72be64b3 0d26ffd4 77c69f72 mswsock!SockAsyncThread+0x83
02 0d26ff94 77c69f72 0101d3e8 219d6a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0d26ffd4 77c69f45 72be62ee 0101d3e8 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0d26ffec 00000000 72be62ee 0101d3e8 00000000 ntdll!_RtlUserThreadStart+0x1b
Edit 2
0:000> .exr -1
ExceptionAddress: 00404880 (<3thParty>+0x00004880)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000000
Attempt to read from address 00000000
0:000> ~* e s -d esp L0xffff 0x1003f
0018e118 0001003f 00000000 00000000 00000000 ?...............
0018e9a8 0001003f 00000000 00000000 00000000 ?...............
0018efa8 0001003f 00000000 00000000 00000000 ?...............
0:000> .cxr 0018e118
eax=00fc2c80 ebx=00000000 ecx=00000000 edx=00fc2c50 esi=00000000 edi=00000000
eip=00404880 esp=0018e400 ebp=0018f61c iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
<3thParty>+0x4880:
00404880 8b11 mov edx,dword ptr [ecx] ds:002b:00000000=????????
0:000> kbnf
*** Stack trace for last set context - .thread/.cxr resets it
# Memory ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0018f61c 75a15932 01485018 01549420 00000000 <3thParty>+0x4880
01 24 0018f640 75a905f1 00523859 0018f828 00000004 rpcrt4!Invoke+0x2a
02 404 0018fa44 7735aec1 010ac688 0101fb58 0104ea78 rpcrt4!NdrStubCall2+0x2ea
03 48 0018fa8c 76acffd3 010ac688 0104ea78 0101fb58 ole32!CStdStubBuffer_Invoke+0x3c
04 24 0018fab0 7735d876 010ac688 0104ea78 0101fb58 oleaut32!CUnivStubWrapper::Invoke+0xcb
05 48 0018faf8 7735ddd0 0104ea78 01048240 01095448 ole32!SyncStubInvoke+0x3c
06 4c 0018fb44 77278a43 0104ea78 010be578 01097470 ole32!StubInvoke+0xb9
07 dc 0018fc20 77278938 0101fb58 00000000 01097470 ole32!CCtxComChnl::ContextInvoke+0xfa
08 1c 0018fc3c 7727950a 0104ea78 00000001 01097470 ole32!MTAInvoke+0x1a
09 2c 0018fc68 7735dccd 0104ea78 00000001 01097470 ole32!STAInvoke+0x46
0a 34 0018fc9c 7735db41 d0908070 0101fb58 01097470 ole32!AppInvoke+0xab
0b e0 0018fd7c 7735e1fd 0104ea20 0101fef0 00000400 ole32!ComInvokeWithLockAndIPID+0x372
0c 28 0018fda4 77279367 0104ea20 00000400 00ff3548 ole32!ComInvoke+0xc5
0d 14 0018fdb8 77279326 0104ea20 00000000 77279286 ole32!ThreadDispatch+0x23
0e 44 0018fdfc 76eb62fa 00e2019e 00000400 0000babe ole32!ThreadWndProc+0x161
0f 2c 0018fe28 76eb6d3a 77279286 00e2019e 00000400 user32!InternalCallWinProc+0x23
10 78 0018fea0 76eb77c4 00000000 77279286 00e2019e user32!UserCallWinProcCheckWow+0x109
11 60 0018ff00 76eb788a 77279286 00000000 00e2019e user32!DispatchMessageWorker+0x3bc
12 10 0018ff10 004aea6a 0018ff34 00010001 0018ff70 user32!DispatchMessageW+0xf
13 1c 0018ff2c 004aeaaf 00e2019e 00000400 0000babe <3thParty>+0xaea6a
14 24 0018ff50 00a0c705 0018ff78 00a0c733 0018ff70 <3thParty>+0xaeaaf
15 20 0018ff70 00a3024b 0018ffc4 00404cac 0018ff88 <3thParty>!FrobWidget+0x4db279
16 18 0018ff88 7766338a fffde000 0018ffd4 77c69f72 <3thParty>!FrobWidget+0x4fedbf
17 c 0018ff94 77c69f72 fffde000 2ca36a95 00000000 kernel32!BaseThreadInitThunk+0xe
18 40 0018ffd4 77c69f45 00a30220 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
19 18 0018ffec 00000000 00a30220 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
Normally, an unhandled exception does cause the process to be terminated once the dump has been captured. What's happening here is that you are running afoul of a compatibility behavior: If a COM server crashes while handling an inbound request, COM historically "handles" the crash by abandoning the call, returning RPC_E_SERVERFAULT to the external caller, and then pretending that the crash never occurred.
That's why the process is still running after the crash dump is taken: The exception was handled. Mind you, it was "handled" by ignoring it, but technically it was handled.
I would recommend using IGlobalOptions to set the COMGLB_EXECPTION_HANDLING property to one of the COMGLB_EXCEPTION_DONOT_HANDLE... values.
I have a script that initializes a driver on startup, which worked beautifully before I enabled kernel tracing and recompiled the kernel to try and debug an issue with a piece of software. If I try to initialize the driver in any way (modprobe, insmod, etc) this output prints to the screen:
[ 26.263308] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 26.263322] IP: [<c108664d>] trace_module_notify+0x16b/0x20a
[ 26.263325] *pde = 00000000
[ 26.263329] Oops: 0000 [#1] PREEMPT SMP
[ 26.263335] Modules linked in: phddrv(O+)
[ 26.263343] Pid: 704, comm: insmod Tainted: G O 3.6.3-rt9 #21 Advanced Digital Logic, Inc CB4053/ADLS15PC
[ 26.263346] EIP: 0060:[<c108664d>] EFLAGS: 00010213 CPU: 0
[ 26.263350] EIP is at trace_module_notify+0x16b/0x20a
[ 26.263353] EAX: ee6e9274 EBX: f082550c ECX: ee6e920c EDX: f082550c
[ 26.263356] ESI: 00000000 EDI: ee6e92dc EBP: ee6ebf4c ESP: ee6ebf24
[ 26.263359] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 26.263362] CR0: 8005003b CR2: 00000000 CR3: 2f2ea000 CR4: 000007d0
[ 26.263365] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 26.263367] DR6: ffff0ff0 DR7: 00000400
[ 26.263371] Process insmod (pid: 704, ti=ee6ea000 task=ef218000 task.ti=ee6ea000)
[ 26.263372] Stack:
[ 26.263381] ee6e9274 ee6e9344 ee6e92dc ee6e920c ee6e9274 ee6e9344 c2086424 c15a5d58
[ 26.263388] 00000000 00000001 ee6ebf68 c1046d33 f082550c c15a51bc c15a3778 00000000
[ 26.263396] c15a3790 ee6ebf8c c1046fa9 fffffffd 00000000 f082550c 00000001 f082550c
[ 26.263397] Call Trace:
[ 26.263407] [<c1046d33>] notifier_call_chain+0x2b/0x4d
[ 26.263413] [<c1046fa9>] __blocking_notifier_call_chain+0x3c/0x51
[ 26.263419] [<c1046fcf>] blocking_notifier_call_chain+0x11/0x13
[ 26.263426] [<c10671b7>] sys_init_module+0x57/0x190
[ 26.263434] [<c13a3d10>] sysenter_do_call+0x12/0x26
[ 26.263489] Code: 00 c7 42 04 64 5d 5a c1 89 15 64 5d 5a c1 89 45 ec 8d 42 74 83 c2 0c 89 45 e8 89 55 e4 eb 19 57 8b 4d e4 89 da ff 75 ec ff 75 e8 <8b> 06 83 c6 04 e8 c2 fb ff ff 83 c4 0c 3b 75 f0 72 e2 eb 77 b8
[ 26.263495] EIP: [<c108664d>] trace_module_notify+0x16b/0x20a SS:ESP 0068:ee6ebf24
[ 26.263497] CR2: 0000000000000000
[ 26.267381] ---[ end trace 0000000000000002 ]---
Any hint as to what is going on would be greatly appreciated!
I got similar issue as yours (almost the same stack trace of panic).
The root cause on my side is that after I changed the kernel config (enable trace point) I only rebuilt the kernel bzImage but forgot to rebuilt the ko modules! That may cause some execution mismatch between the new kernel and old ko modules.
After rebuild and update both kernel image and ko modules, the issue is gone.
Somewhere in the driver there is a NULL pointer. A pointer variabile has value NULL and the driver is trying to use it.
myPtr->value; /* if myPtr is NULL, this will raise the kernel oops */
You have to debug the driver to find where and why there is a NULL pointer
I have already encountered some Oops in my developer's life and whereas I am familiar with some information that I can retrieve from these Oops, there are still pieces of information I can't understand and therefore, can't use to solve problems.
Below you will find an Oops example and I will describe what I can deduce from it. Then, I will ask what the remaining info can teach me about the problem.
[ 716.485951] BUG: unable to handle kernel paging request at fc132158
[ 716.485973] IP: [<fc1936e7>] ubi_change_vtbl_record+0x87/0x1c0 [ubi]
[ 716.485986] *pdpt = 00000000019e6001 *pde = 000000002c558067 *pte = 0000000000000000
[ 716.485997] Oops: 0002 [#1] SMP
[ 716.486004] Modules linked in: ubi(O) mtdchar nandsim nand mtd nand_ids nand_bch bch nand_ecc bnep rfcomm bluetooth parport_pc ppdev lp parport nfsd nfs_acl auth_rpcgss nfs fscache lockd sunrpc binfmt_misc dm_crypt snd_hda_codec_hdmi snd_hda_codec_analog kvm_intel snd_hda_intel snd_hda_codec snd_hwdep kvm snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event hid_generic snd_seq cdc_acm snd_timer snd_seq_device mei tpm_tis snd mac_hid serio_raw soundcore lpc_ich snd_page_alloc microcode coretemp usbhid hid nouveau usb_storage ttm drm_kms_helper drm floppy e1000e i2c_algo_bit mxm_wmi video wmi
[ 716.486128] Pid: 3994, comm: ubimkvol Tainted: G O 3.8.0-rc3+ #3 LENOVO 6239AS8/LENOVO
[ 716.486136] EIP: 0060:[<fc1936e7>] EFLAGS: 00010246 CPU: 0
[ 716.486144] EIP is at ubi_change_vtbl_record+0x87/0x1c0 [ubi]
[ 716.486151] EAX: 000000ac EBX: eb5ea000 ECX: 0000002b EDX: 00000000
[ 716.486157] ESI: eb4d1d74 EDI: fc132158 EBP: eb4d1d40 ESP: eb4d1d20
[ 716.486164] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 716.486170] CR0: 8005003b CR2: fc132158 CR3: 27542000 CR4: 000407f0
[ 716.486176] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 716.486183] DR6: ffff0ff0 DR7: 00000400
[ 716.486188] Process ubimkvol (pid: 3994, ti=eb4d0000 task=ec01d9b0 task.ti=eb4d0000)
[ 716.486195] Stack:
[ 716.486199] e755f000 eb4d1d2c c11cad11 eb4d1d34 eb543c00 eb5ea000 00000000 eb4d1e20
[ 716.486215] eb4d1e30 fc195412 e755f000 fc1adf01 eb5ea26c 00000002 0000009e eb5ea480
[ 716.486232] 00000002 e755f22c e755f2ac e755f000 eb4d1d74 2a000000 01000000 00000000
[ 716.486248] Call Trace:
[ 716.486257] [<c11cad11>] ? sysfs_create_file+0x21/0x30
[ 716.486266] [<fc195412>] ubi_create_volume+0x4b2/0x790 [ubi]
[ 716.486277] [<fc19967a>] ubi_cdev_ioctl+0x5da/0xac0 [ubi]
[ 716.486285] [<c117202a>] ? link_path_walk+0x5a/0x7d0
[ 716.486294] [<fc1990a0>] ? vol_cdev_ioctl+0x440/0x440 [ubi]
[ 716.486842] [<c1177e12>] do_vfs_ioctl+0x82/0x5b0
[ 716.487703] [<c1171ced>] ? final_putname+0x1d/0x40
[ 716.488564] [<c1171ced>] ? final_putname+0x1d/0x40
[ 716.489422] [<c1171ced>] ? final_putname+0x1d/0x40
[ 716.489891] [<c1171eb4>] ? putname+0x24/0x40
[ 716.489891] [<c1167239>] ? do_sys_open+0x169/0x1d0
[ 716.489891] [<c11783b0>] sys_ioctl+0x70/0x80
[ 716.489891] [<c16205cd>] sysenter_do_call+0x12/0x38
[ 716.489891] Code: ac 00 00 00 03 bb c8 04 00 00 f7 c7 01 00 00 00 0f 85 ee 00 00 00 f7 c7 02 00 00 00 0f 85 ca 00 00 00 89 c1 31 d2 c1 e9 02 a8 02 <f3> a5 74 0b 0f b7 16 66 89 17 ba 02 00 00 00 a8 01 74 07 0f b6
[ 716.489891] EIP: [<fc1936e7>] ubi_change_vtbl_record+0x87/0x1c0 [ubi] SS:ESP 0068:eb4d1d20
[ 716.489891] CR2: 00000000fc132158
[ 716.516453] ---[ end trace 473b15a7780e19ea ]---
It seems that the kernel wanted to access a wrong page. Now,
The Oops code 0002 tells me that it occurred while trying to read something in user-mode.
The Instruction Pointer is at ubi_change_vtbl_record, which means the offending instruction is located in this function.
I can deduce the path that lead to the faulting function from the
call trace (an ioctl launched from process ubimkvol)
From there, Is the "stack" a dump of the raw stack of the task ? I can see that some values mentioned are also function addresses found in the call trace. Then, I got fancy looking values like EAX, EBX ... DR7. I think they are CPU registers but still, I don't know what they really are.
Finally, the following line gets me lost :
[ 716.485986] *pdpt = 00000000019e6001 *pde = 000000002c558067 *pte = 0000000000000000
What are pdpt, pde and pte ? I feel they are information about the page fault but I could not retrieve further information after some googling around.
Yes, EAX, etc. are 32-bit x86 processor registers. pdpt (page directory pointer table), pde (page directory entry), and pte (page table entry) are all paging structures.
IP (also EIP for 32-bit or RIP for 64-bit processors) is the instruction pointer at the time of the Oops.
The stack is the raw stack for this processor. Each processor will have its own stack. Note that on this architecture the stack grows down (addresses start with 0xfxxxxxx).
Correct me if I am wrong but,
OOPS 0002 means no page found when writing in kernel mode:
bit 0 == 0 means no page found, 1 means a protection fault
bit 1 == 0 means read, 1 means write
bit 2 == 0 means kernel, 1 means user-mode
I have the ARM OMAP system which is generatin Oops during some tests.
The system is still alive, but the thread which has created the Oops has died.
I have pasted oops logs below. Please help me to debug this issue.
From log i see that:
Process appl (pid: 427, stack limit = 0x86faa2e8)
This proc 427 is one of 5 threads created within main process. I have the /proc/kallsyms file, and checked that the PC is showing the correct function (put_page+0x14) of crash.
LOGS:
[Wed Mar 27 19:34:51.796 2013] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[Wed Mar 27 19:34:51.796 2013] pgd = 86f94000
[Wed Mar 27 19:34:51.796 2013] [00000000] *pgd=87a9f031, *pte=00000000, *ppte=00000000
[Wed Mar 27 19:34:51.796 2013] Internal error: Oops: 17 [#1] PREEMPT
[Wed Mar 27 19:34:51.796 2013] last sysfs file: /sys/devices/virtual/spi/spi/dev
[Wed Mar 27 19:34:51.796 2013] Modules linked in: spi_mod hr_driver isp(P) spv(P) gp
[Wed Mar 27 19:34:51.796 2013] CPU: 0 Tainted: P (2.6.33_Visor #1)
[Wed Mar 27 19:34:51.796 2013] PC is at put_page+0x14/0x184
[Wed Mar 27 19:34:51.796 2013] LR is at __pskb_pull_tail+0x210/0x2b4
[Wed Mar 27 19:34:51.796 2013] pc : [<800b22c4>] lr : [<80230254>] psr: 60000013
[Wed Mar 27 19:34:51.812 2013] sp : 86fabb28 ip : 800b22c4 fp : 86fabb44
[Wed Mar 27 19:34:51.812 2013] r10: 00000054 r9 : 00000006 r8 : 00000007
[Wed Mar 27 19:34:51.812 2013] r7 : 00000007 r6 : 00000000 r5 : 00000000 r4 : 87340500
[Wed Mar 27 19:34:51.812 2013] r3 : 86fe1cc0 r2 : 86fe1d44 r1 : 86fe1d14 r0 : 00000000
[Wed Mar 27 19:34:51.812 2013] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[Wed Mar 27 19:34:51.812 2013] Control: 10c5387d Table: 86f94019 DAC: 00000015
[Wed Mar 27 19:34:51.812 2013] Process appl (pid: 427, stack limit = 0x86faa2e8)
[Wed Mar 27 19:34:51.812 2013] Stack: (0x86fabb28 to 0x86fac000)
[Wed Mar 27 19:34:51.812 2013] bb20: 86fe1cc0 87340500 00000000 00000000 86fabb6c 86fabb48
[Wed Mar 27 19:34:51.812 2013] bb40: 80230254 800b22bc 87340500 86d49140 87a2a800 00000000 86d49154 00000000
[Wed Mar 27 19:34:51.812 2013] bb60: 86fabb9c 86fabb70 802392a0 80230050 87340500 0000000e 00000000 87340500
[Wed Mar 27 19:34:51.812 2013] bb80: 86d49140 0000000e 00000000 86d49154 86fabbc4 86fabba0 8025a6a0 802391f4
[Wed Mar 27 19:34:51.812 2013] bba0: 87340500 86faa000 87a2a800 00000000 86fe1c8c 0000ffff 86fabbe4 86fabbc8
[Wed Mar 27 19:34:51.812 2013] bbc0: 8025a770 8025a434 8025a6d8 87340500 86fe1c78 87340500 86fabbfc 86fabbe8
[Wed Mar 27 19:34:51.812 2013] bbe0: 802583d8 8025a6e4 00000000 86d7fa80 86fabc94 86fabc00 80259fec 802583b0
[Wed Mar 27 19:34:51.812 2013] bc00: 802555a4 802555e4 00000001 86fabc38 00000102 00000000 802c69bc 80237f9c
[Wed Mar 27 19:34:51.812 2013] bc20: 87a2ab88 86fabc38 86fabc54 86fabc38 8002d4dc 8002bf80 8002bfa4 00000001
[Wed Mar 27 19:34:51.812 2013] bc40: 8026c77c 86faa000 86fabc7c 86fabc58 8002bfe0 8002d4c4 86fabcec 00000001
[Wed Mar 27 19:34:51.828 2013] bc60: 8026c77c 87ab3080 00000014 86d7fa80 87340500 86faa000 00000014 86fe1c8c
[Wed Mar 27 19:34:51.828 2013] bc80: 00000006 0000ffff 86fabcec 86fabc98 8026c7a0 80259d04 86fabcec 00000020
[Wed Mar 27 19:34:51.828 2013] bca0: 000000c0 87340500 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Mar 27 19:34:51.828 2013] bcc0: 87340500 86d7fa80 87340500 86d7fae4 000000fa 86faa000 00000006 86d7fd0c
[Wed Mar 27 19:34:51.828 2013] bce0: 86fabd04 86fabcf0 8026c960 8026c02c 86d7fa80 00000000 86fabd24 86fabd08
[Wed Mar 27 19:34:51.828 2013] bd00: 8026104c 8026c898 8022f70c 800d3964 86d7fa80 00000006 86fabd9c 86fabd28
[Wed Mar 27 19:34:51.828 2013] bd20: 802617bc 80260f04 8002d4dc 8002bf80 8002bfa4 00000001 00000000 86d7fae4
[Wed Mar 27 19:34:51.828 2013] bd40: 86d7fde8 86d7fb18 86d7fd28 00000000 86fabf3c 00000001 00000000 00000000
[Wed Mar 27 19:34:51.828 2013] bd60: 00000001 00000000 86fabd8c 86fabd78 80049998 86fabf3c 803c0e8c 00000040
[Wed Mar 27 19:34:51.828 2013] bd80: 00000000 00000040 86faa000 2e4128a2 86fabdd4 86fabda0 80229d34 80261068
[Wed Mar 27 19:34:51.828 2013] bda0: 00000040 00000000 86fabdb4 00000000 00000500 00000000 00000000 86fabde0
[Wed Mar 27 19:34:51.828 2013] bdc0: 00000100 86fabf3c 86fabeb4 86fabdd8 8022767c 80229cf4 00000040 86fabde8
[Wed Mar 27 19:34:51.828 2013] bde0: 802277c4 80261de0 00000000 00000001 ffffffff 00000000 00000000 00000000
[Wed Mar 27 19:34:51.828 2013] be00: 00000000 00000000 87a70740 86fabe18 00000000 00000000 8002bfa4 00000001
[Wed Mar 27 19:34:51.828 2013] be20: 86fabe58 86fabf60 86fabe5c 86fabe38 8002bfe0 8002d4c4 86fabe94 00000001
[Wed Mar 27 19:34:51.828 2013] be40: 800d9094 86fabf60 00000000 00000040 86fabe74 86fabe60 80049998 8009ec3c
[Wed Mar 27 19:34:51.828 2013] be60: 00000040 00000100 87541080 86fabe78 00000000 86fabf3c 86fabe78 fffffff7
[Wed Mar 27 19:34:51.843 2013] be80: 80226524 802278a8 86fabeb4 86fabe98 802278a8 8022651c 00000040 87541080
[Wed Mar 27 19:34:51.843 2013] bea0: 00000100 006766f0 86fabf8c 86fabeb8 8022804c 802275d8 8002bfa4 87803140
[Wed Mar 27 19:34:51.843 2013] bec0: 86f67800 a0000013 87800440 00000003 86fabefc 86fabee0 800d3944 8009ed38
[Wed Mar 27 19:34:51.843 2013] bee0: 00000000 00000508 87a25290 86faa000 86fabf14 86fabf00 80186658 800d3810
[Wed Mar 27 19:34:51.843 2013] bf00: 86f67800 00000508 86fabf7c 86fabf18 80186d90 80186648 86fabf3c 2e4128a0
[Wed Mar 27 19:34:51.843 2013] bf20: 86fabf8c 87a25290 87a25298 803b0d7c 86fabf84 00100100 00200200 86fabebc
[Wed Mar 27 19:34:51.843 2013] bf40: 00000080 86fabf58 00000001 00000000 00000000 2e41289c 006766f6 000000fa
[Wed Mar 27 19:34:51.843 2013] bf60: 00000001 fffffff7 86faa000 006766ec 0066b138 00000000 00000123 8002a328
[Wed Mar 27 19:34:51.843 2013] bf80: 86fabfa4 86fabf90 802280cc 80227fc4 00000000 00000000 00000000 86fabfa8
[Wed Mar 27 19:34:51.843 2013] bfa0: 8002a100 802280b0 006766ec 0066b138 0000000f 006766f0 00000100 00000040
[Wed Mar 27 19:34:51.843 2013] bfc0: 006766ec 0066b138 00000000 00000123 00000740 0066b138 2e4128a2 000005d8
[Wed Mar 27 19:34:51.843 2013] bfe0: 00000000 2e411a30 2ad8cfe4 2ad7fc24 80000010 0000000f 00000000 00000000
[Wed Mar 27 19:34:51.968 2013] Backtrace:
[Wed Mar 27 19:34:51.968 2013] [<800b22b0>] (put_page+0x0/0x184) from [<80230254>] (__pskb_pull_tail+0x210/0x2b4)
[Wed Mar 27 19:34:51.968 2013] r6:00000000 r5:00000000 r4:87340500 r3:86fe1cc0
[Wed Mar 27 19:34:51.984 2013] [<80230044>] (__pskb_pull_tail+0x0/0x2b4) from [<802392a0>] (dev_queue_xmit+0xb8/0x474)
[Wed Mar 27 19:34:51.984 2013] [<802391e8>] (dev_queue_xmit+0x0/0x474) from [<8025a6a0>] (ip_finish_output+0x278/0x2b0)
[Wed Mar 27 19:34:51.984 2013] r8:86d49154 r7:00000000 r6:0000000e r5:86d49140 r4:87340500
[Wed Mar 27 19:34:51.984 2013] [<8025a428>] (ip_finish_output+0x0/0x2b0) from [<8025a770>] (ip_output+0x98/0xa0)
[Wed Mar 27 19:34:51.984 2013] [<8025a6d8>] (ip_output+0x0/0xa0) from [<802583d8>] (ip_local_out+0x34/0x38)
[Wed Mar 27 19:34:51.984 2013] r6:87340500 r5:86fe1c78 r4:87340500 r3:8025a6d8
[Wed Mar 27 19:34:51.984 2013] [<802583a4>] (ip_local_out+0x0/0x38) from [<80259fec>] (ip_queue_xmit+0x2f4/0x364)
[Wed Mar 27 19:34:51.984 2013] r4:86d7fa80 r3:00000000
[Wed Mar 27 19:34:51.984 2013] [<80259cf8>] (ip_queue_xmit+0x0/0x364) from [<8026c7a0>] (tcp_transmit_skb+0x780/0x7dc)
[Wed Mar 27 19:34:51.984 2013] [<8026c020>] (tcp_transmit_skb+0x0/0x7dc) from [<8026c960>] (tcp_send_ack+0xd4/0xdc)
[Wed Mar 27 19:34:51.984 2013] [<8026c88c>] (tcp_send_ack+0x0/0xdc) from [<8026104c>] (tcp_cleanup_rbuf+0x154/0x164)
[Wed Mar 27 19:34:51.984 2013] r5:00000000 r4:86d7fa80
[Wed Mar 27 19:34:51.984 2013] [<80260ef8>] (tcp_cleanup_rbuf+0x0/0x164) from [<802617bc>] (tcp_recvmsg+0x760/0x87c)
[Wed Mar 27 19:34:51.984 2013] r5:00000006 r4:86d7fa80
[Wed Mar 27 19:34:51.984 2013] [<8026105c>] (tcp_recvmsg+0x0/0x87c) from [<80229d34>] (sock_common_recvmsg+0x4c/0x60)
[Wed Mar 27 19:34:51.984 2013] [<80229ce8>] (sock_common_recvmsg+0x0/0x60) from [<8022767c>] (sock_recvmsg+0xb0/0xcc)
[Wed Mar 27 19:34:51.984 2013] r6:86fabf3c r5:00000100 r4:86fabde0
[Wed Mar 27 19:34:51.984 2013] [<802275cc>] (sock_recvmsg+0x0/0xcc) from [<8022804c>] (sys_recvfrom+0x94/0xec)
[Wed Mar 27 19:34:51.984 2013] r6:006766f0 r5:00000100 r4:87541080
[Wed Mar 27 19:34:51.984 2013] [<80227fb8>] (sys_recvfrom+0x0/0xec) from [<802280cc>] (sys_recv+0x28/0x30)
[Wed Mar 27 19:34:51.984 2013] r8:8002a328 r7:00000123 r6:00000000 r5:0066b138 r4:006766ec
[Wed Mar 27 19:34:51.984 2013] [<802280a4>] (sys_recv+0x0/0x30) from [<8002a100>] (ret_fast_syscall+0x0/0x38)
[Wed Mar 27 19:34:51.984 2013] Code: e92dd878 e24cb004 e52de004 ebfddfbf (e5903000)
Guess some bug in TCP/IP stack, conflict with S/G IO features in your network interface card (NIC) driver.
you can check if your NIC driver has these feature flags NETIF_F_FRAGLIST and NETIF_F_SG set, if yes, try unset them, see if this problem still happens.