We have a java application that calls a native method from a dll. This works for us most of the time but very rarely on some pcs the JNI is calling wrong native method. As we have finally a machine that reproduces the issue regurally we would like to investigate what is the reason and hopefully fix it, but we are out of ideas how to investigate the issue.
The following is a java crash dump. It is visible there that we are trying to invoke method setVolumeNative; the method create is invoked however.
We would expect to see the following on stack:
Java_net_inspiredbroadcast_legion_jni_volume_MainVolume_setVolumeNative
But we see there this:
Java_net_inspiredbroadcast_legion_jni_process_ProcessImpl_create
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6294c518, pid=14900, tid=7008
#
# JRE version: 7.0
# Java VM: OpenJDK Client VM (22.0-b10 mixed mode windows-x86 )
# Problematic frame:
# C [common-win32-native-1.0.6-SNAPSHOT.dll+0xc518]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x11969c00): JavaThread "Bootstrap" [_thread_in_native, id=7008, stack(0x12890000,0x128e0000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x00000000
Registers:
EAX=0x00000000, EBX=0x0d3871c0, ECX=0x730a3130, EDX=0x128df744
ESP=0x128df730, EBP=0x128df754, ESI=0x0d3871c0, EDI=0x11969c00
EIP=0x6294c518, EFLAGS=0x00010247
Top of Stack: (sp=0x128df730)
0x128df730: 6299f488 00000001 00000000 128df744
0x128df740: 00000001 00000000 80070490 00000000
0x128df750: 00000000 128df788 6294bb31 11969d28
0x128df760: 128df7c4 00000000 00000006 00000008
0x128df770: 11969c00 0d3871c0 0d3871c0 128df7bc
0x128df780: 0245a4b7 00969c00 128df7bc 0245a5b7
0x128df790: 11969d28 128df7c4 00000000 128df79c
0x128df7a0: 0d3871c0 128df7cc 0d387920 00000000
Instructions: (pc=0x6294c518)
0x6294c4f8: ff d1 c7 45 fc 00 00 00 00 c7 45 f0 00 00 00 00
0x6294c508: 8d 55 f0 52 6a 00 6a 01 68 88 f4 99 62 8b 45 f8
0x6294c518: 8b 08 8b 55 f8 52 8b 41 0c ff d0 89 45 f4 83 7d
0x6294c528: f4 00 74 16 68 70 f1 99 62 68 90 f1 99 62 8b 4d
Register to memory mapping:
EAX=0x00000000 is an unknown value
EBX=0x0d3871c0 is an oop
{method}
- klass: {other class}
ECX=0x730a3130 is an unknown value
EDX=0x128df744 is pointing into the stack for thread: 0x11969c00
ESP=0x128df730 is pointing into the stack for thread: 0x11969c00
EBP=0x128df754 is pointing into the stack for thread: 0x11969c00
ESI=0x0d3871c0 is an oop
{method}
- klass: {other class}
EDI=0x11969c00 is a thread
Stack: [0x12890000,0x128e0000], sp=0x128df730, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [common-win32-native-1.0.6-SNAPSHOT.dll+0xc518] Java_net_inspiredbroadcast_legion_jni_process_ProcessImpl_create+0xa410
C [common-win32-native-1.0.6-SNAPSHOT.dll+0xbb31] Java_net_inspiredbroadcast_legion_jni_process_ProcessImpl_create+0x9a29
j net.inspiredbroadcast.legion.jni.volume.MainVolume.setVolumeNative(I)V+0
j net.inspiredbroadcast.legion.jni.volume.MainVolume.setVolume(I)V+49
j net.inspiredbroadcast.legion.peripheral.win32.MainSystemVolume.setVolume(I)V+100
j net.inspiredbroadcast.legion.peripheral.win32.MainSystemVolume.startUp()V+49
j net.inspiredbroadcast.legion.ikernel.service.impl.ServiceLoaderImpl.<init>()V+210
j net.inspiredbroadcast.legion.ikernel.service.impl.ServiceHandler.initialise()V+176
j net.inspiredbroadcast.legion.ikernel.core.BootstrapDelegate.start()Ljava/lang/Integer;+318
j net.inspiredbroadcast.legion.ikernel.core.Bootstrap.start([Ljava/lang/String;)Ljava/lang/Integer;+14
j org.tanukisoftware.wrapper.WrapperManager$11.run()V+75
v ~StubRoutines::call_stub
V [jvm.dll+0x1233ba]
V [jvm.dll+0x1c8f7e]
V [jvm.dll+0x1235a3]
V [jvm.dll+0x123607]
V [jvm.dll+0xcd1ef]
V [jvm.dll+0x1430af]
V [jvm.dll+0x1711e9]
C [msvcr100.dll+0x10fac] freefls+0x130
C [msvcr100.dll+0x110b1] __fls_getvalue+0x58
C [kernel32.dll+0x133aa] BaseThreadInitThunk+0x12
C [ntdll.dll+0x3bf32] RtlInitializeExceptionChain+0x63
C [ntdll.dll+0x3bf05] RtlInitializeExceptionChain+0x36
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j net.inspiredbroadcast.legion.jni.volume.MainVolume.setVolumeNative(I)V+0
j net.inspiredbroadcast.legion.jni.volume.MainVolume.setVolume(I)V+49
j net.inspiredbroadcast.legion.peripheral.win32.MainSystemVolume.setVolume(I)V+100
j net.inspiredbroadcast.legion.peripheral.win32.MainSystemVolume.startUp()V+49
j net.inspiredbroadcast.legion.ikernel.service.impl.ServiceLoaderImpl.<init>()V+210
j net.inspiredbroadcast.legion.ikernel.service.impl.ServiceHandler.initialise()V+176
j net.inspiredbroadcast.legion.ikernel.core.BootstrapDelegate.start()Ljava/lang/Integer;+318
j net.inspiredbroadcast.legion.ikernel.core.Bootstrap.start([Ljava/lang/String;)Ljava/lang/Integer;+14
j org.tanukisoftware.wrapper.WrapperManager$11.run()V+75
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x11e76c00 JavaThread "CashHandlingWorker" daemon [_thread_blocked, id=5732, stack(0x18e60000,0x18eb0000)]
0x11e76800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=15856, stack(0x13220000,0x13270000)]
0x11e76000 JavaThread "NetConn Monitor" daemon [_thread_blocked, id=15932, stack(0x18560000,0x185b0000)]
0x11c02800 JavaThread "AudioPlayerThread" [_thread_blocked, id=15000, stack(0x18b90000,0x18be0000)]
0x11c02000 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=14308, stack(0x185d0000,0x18620000)]
0x11c01c00 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=8760, stack(0x17860000,0x178b0000)]
0x11c01400 JavaThread "AWT-Windows" daemon [_thread_in_native, id=12852, stack(0x13280000,0x132d0000)]
0x11c01000 JavaThread "AWT-Shutdown" [_thread_blocked, id=7156, stack(0x12f70000,0x12fc0000)]
0x11c00800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=16016, stack(0x12840000,0x12890000)]
0x11c00000 JavaThread "TelnetServer" [_thread_in_native, id=6376, stack(0x12ed0000,0x12f20000)]
0x11bffc00 JavaThread "EventWorker" daemon [_thread_blocked, id=14384, stack(0x12280000,0x122d0000)]
0x11bff000 JavaThread "ScheduleHandler" daemon [_thread_blocked, id=15584, stack(0x13050000,0x130a0000)]
0x11a21400 JavaThread "MarimbaDownloadStatusMessenger" [_thread_blocked, id=10380, stack(0x113e0000,0x11430000)]
0x11a06c00 JavaThread "MetersWorkingThread" [_thread_blocked, id=7028, stack(0x10d00000,0x10d50000)]
0x119e4800 JavaThread "SuspensionEvaluatorDaemon" daemon [_thread_blocked, id=12132, stack(0x12690000,0x126e0000)]
0x119e3000 JavaThread "Announcer" daemon [_thread_blocked, id=12116, stack(0x129f0000,0x12a40000)]
0x119e2800 JavaThread "Asyncmessage sending thread" [_thread_blocked, id=16036, stack(0x125f0000,0x12640000)]
=>0x11969c00 JavaThread "Bootstrap" [_thread_in_native, id=7008, stack(0x12890000,0x128e0000)]
0x009adc00 JavaThread "DestroyJavaVM" [_thread_blocked, id=14360, stack(0x00b20000,0x00b70000)]
0x1192bc00 JavaThread "Wrapper-Connection" daemon [_thread_in_native, id=15560, stack(0x12640000,0x12690000)]
0x118f9c00 JavaThread "Wrapper-Startup-Runner" [_thread_blocked, id=9208, stack(0x12760000,0x127b0000)]
0x11918c00 JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_blocked, id=15772, stack(0x12700000,0x12750000)]
0x118c9400 JavaThread "Cerberus 1" daemon [_thread_blocked, id=10816, stack(0x12330000,0x12380000)]
0x11325000 JavaThread "ProcessHandler" daemon [_thread_blocked, id=14040, stack(0x11430000,0x11480000)]
0x10529800 JavaThread "Service Thread" daemon [_thread_blocked, id=11748, stack(0x10ad0000,0x10b20000)]
0x10525000 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=11232, stack(0x10bc0000,0x10c10000)]
0x10523c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=16228, stack(0x106f0000,0x10740000)]
0x10522c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10324, stack(0x10b20000,0x10b70000)]
0x104ca000 JavaThread "Finalizer" daemon [_thread_blocked, id=11864, stack(0x109d0000,0x10a20000)]
0x104c4000 JavaThread "Reference Handler" daemon [_thread_blocked, id=15388, stack(0x10a30000,0x10a80000)]
Other Threads:
0x00aff800 VMThread [stack: 0x10760000,0x107b0000] [id=15344]
0x1053d400 WatcherThread [stack: 0x10f10000,0x10f60000] [id=10968]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
def new generation total 4928K, used 718K [0x04450000, 0x049a0000, 0x06ef0000)
eden space 4416K, 9% used [0x04450000, 0x044bda98, 0x048a0000)
from space 512K, 54% used [0x04920000, 0x04966020, 0x049a0000)
to space 512K, 0% used [0x048a0000, 0x048a0000, 0x04920000)
tenured generation total 10944K, used 6794K [0x06ef0000, 0x079a0000, 0x0c450000)
the space 10944K, 62% used [0x06ef0000, 0x07592988, 0x07592a00, 0x079a0000)
compacting perm gen total 15616K, used 15582K [0x0c450000, 0x0d390000, 0x10450000)
the space 15616K, 99% used [0x0c450000, 0x0d387a40, 0x0d387c00, 0x0d390000)
No shared spaces configured.
Code Cache [0x02450000, 0x02638000, 0x04450000)
total_blobs=1078 nmethods=832 adapters=182 free_code_cache=30832Kb largest_free_block=31570240
Dynamic libraries:
0x00bf0000 - 0x00c1c000 C:\ikernel\dist\openjdk7\bin\java.exe
0x77900000 - 0x77a80000 C:\Windows\SysWOW64\ntdll.dll
0x767e0000 - 0x768f0000 C:\Windows\syswow64\kernel32.dll
0x76790000 - 0x767d7000 C:\Windows\syswow64\KERNELBASE.dll
0x766f0000 - 0x76790000 C:\Windows\syswow64\ADVAPI32.dll
0x75870000 - 0x7591c000 C:\Windows\syswow64\msvcrt.dll
0x774e0000 - 0x774f9000 C:\Windows\SysWOW64\sechost.dll
0x76600000 - 0x766f0000 C:\Windows\syswow64\RPCRT4.dll
0x75320000 - 0x75380000 C:\Windows\syswow64\SspiCli.dll
0x75310000 - 0x7531c000 C:\Windows\syswow64\CRYPTBASE.dll
0x76cc0000 - 0x76dc0000 C:\Windows\syswow64\USER32.dll
0x772a0000 - 0x77330000 C:\Windows\syswow64\GDI32.dll
0x76cb0000 - 0x76cba000 C:\Windows\syswow64\LPK.dll
0x769d0000 - 0x76a6d000 C:\Windows\syswow64\USP10.dll
0x71020000 - 0x711be000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\COMCTL32.dll
0x77460000 - 0x774b7000 C:\Windows\syswow64\SHLWAPI.dll
0x76570000 - 0x765d0000 C:\Windows\system32\IMM32.DLL
0x76fd0000 - 0x7709c000 C:\Windows\syswow64\MSCTF.dll
0x68000000 - 0x68036000 C:\Windows\system32\QIPCAP.DLL
0x76920000 - 0x769af000 C:\Windows\syswow64\OLEAUT32.dll
0x76b50000 - 0x76cac000 C:\Windows\syswow64\ole32.dll
0x622b0000 - 0x6236d000 C:\ikernel\dist\openjdk7\bin\msvcr100.dll
0x5a810000 - 0x5ab36000 C:\ikernel\dist\openjdk7\bin\client\jvm.dll
0x740a0000 - 0x740d2000 C:\Windows\system32\WINMM.dll
0x769b0000 - 0x769b5000 C:\Windows\syswow64\PSAPI.DLL
0x71620000 - 0x7162c000 C:\ikernel\dist\openjdk7\bin\verify.dll
0x645e0000 - 0x64600000 C:\ikernel\dist\openjdk7\bin\java.dll
0x64580000 - 0x64593000 C:\ikernel\dist\openjdk7\bin\zip.dll
0x64320000 - 0x64334000 C:\ikernel\dist\openjdk7\bin\net.dll
0x770a0000 - 0x770d5000 C:\Windows\syswow64\WS2_32.dll
0x77330000 - 0x77336000 C:\Windows\syswow64\NSI.dll
0x744b0000 - 0x744ec000 C:\Windows\system32\mswsock.dll
0x71ba0000 - 0x71ba6000 C:\Windows\System32\wship6.dll
0x718f0000 - 0x718ff000 C:\ikernel\dist\openjdk7\bin\nio.dll
0x74190000 - 0x741a6000 C:\Windows\system32\CRYPTSP.dll
0x74150000 - 0x7418b000 C:\Windows\system32\rsaenh.dll
0x72160000 - 0x72177000 C:\Windows\system32\USERENV.dll
0x73cc0000 - 0x73ccb000 C:\Windows\system32\profapi.dll
0x742f0000 - 0x74300000 C:\Windows\system32\NLAapi.dll
0x742e0000 - 0x742f0000 C:\Windows\system32\napinsp.dll
0x742c0000 - 0x742d2000 C:\Windows\system32\pnrpnsp.dll
0x74270000 - 0x742b4000 C:\Windows\system32\DNSAPI.dll
0x74260000 - 0x74268000 C:\Windows\System32\winrnr.dll
0x74250000 - 0x7425d000 C:\Windows\system32\wshbth.dll
0x645b0000 - 0x645d7000 C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live\WLIDNSP.DLL
0x74560000 - 0x7457c000 C:\Windows\system32\IPHLPAPI.DLL
0x74550000 - 0x74557000 C:\Windows\system32\WINNSI.DLL
0x74210000 - 0x74248000 C:\Windows\System32\fwpuclnt.dll
0x74200000 - 0x74206000 C:\Windows\system32\rasadhlp.dll
0x71610000 - 0x7161a000 C:\ikernel\dist\openjdk7\bin\management.dll
0x10c10000 - 0x10c5e000 C:\ikernel\dist\lib\wrapper.dll
0x744a0000 - 0x744a5000 C:\Windows\System32\wshtcpip.dll
0x62940000 - 0x629bd000 C:\ikernel\dist\lib\common-win32-native-1.0.6-SNAPSHOT.dll
0x73690000 - 0x73699000 C:\Windows\system32\HID.DLL
0x770e0000 - 0x7727d000 C:\Windows\syswow64\SETUPAPI.dll
0x76fa0000 - 0x76fc7000 C:\Windows\syswow64\CFGMGR32.dll
0x77280000 - 0x77292000 C:\Windows\syswow64\DEVOBJ.dll
0x75920000 - 0x7656b000 C:\Windows\syswow64\SHELL32.dll
0x64060000 - 0x64158000 C:\Windows\system32\CRYPTUI.dll
0x755a0000 - 0x756bd000 C:\Windows\syswow64\CRYPT32.dll
0x75730000 - 0x7573c000 C:\Windows\syswow64\MSASN1.dll
0x768f0000 - 0x7691d000 C:\Windows\syswow64\WINTRUST.dll
0x741e0000 - 0x741f2000 C:\Windows\system32\dhcpcsvc.DLL
0x73910000 - 0x7391d000 C:\Windows\system32\dhcpcsvc6.DLL
0x751b0000 - 0x751d1000 C:\Windows\system32\ntmarta.dll
0x76b00000 - 0x76b45000 C:\Windows\syswow64\WLDAP32.dll
0x731a0000 - 0x731b2000 C:\Windows\system32\SAMLIB.dll
0x72f50000 - 0x72f72000 C:\Windows\system32\logoncli.dll
0x74330000 - 0x74339000 C:\Windows\system32\netutils.dll
0x5e750000 - 0x5e891000 C:\ikernel\dist\openjdk7\bin\awt.dll
0x721a0000 - 0x72220000 C:\Windows\system32\uxtheme.dll
0x73010000 - 0x73068000 C:\Program Files (x86)\Common Files\microsoft shared\ink\tiptsf.dll
0x75240000 - 0x75249000 C:\Windows\system32\version.dll
0x73e30000 - 0x73e43000 C:\Windows\system32\dwmapi.dll
0x70600000 - 0x707c3000 C:\Windows\system32\d3d9.dll
0x72f40000 - 0x72f46000 C:\Windows\system32\d3d8thk.dll
0x69130000 - 0x69f98000 C:\Windows\system32\nvd3dum.dll
0x72a80000 - 0x72aa5000 C:\Windows\system32\powrprof.dll
0x628d0000 - 0x6293d000 C:\ikernel\dist\openjdk7\bin\freetype.dll
0x602c0000 - 0x60345000 C:\ikernel\dist\openjdk7\bin\fontmanager.dll
0x76a70000 - 0x76af3000 C:\Windows\syswow64\CLBCatQ.DLL
0x737a0000 - 0x737dc000 C:\Windows\system32\oleacc.dll
0x70ed0000 - 0x71000000 C:\Windows\system32\WindowsCodecs.dll
0x744f0000 - 0x7453c000 C:\Windows\system32\apphelp.dll
0x637b0000 - 0x637c2000 C:\Program Files (x86)\Common Files\TortoiseOverlays\TortoiseOverlays.dll
0x72ac0000 - 0x72ad4000 C:\Program Files\TortoiseSVN\bin\TortoiseStub32.dll
0x63740000 - 0x637b0000 C:\Program Files\TortoiseSVN\bin\TortoiseSVN32.dll
0x76dd0000 - 0x76f94000 C:\Windows\syswow64\WININET.dll
0x756c0000 - 0x756c4000 C:\Windows\syswow64\api-ms-win-downlevel-user32-l1-1-0.dll
0x774d0000 - 0x774d4000 C:\Windows\syswow64\api-ms-win-downlevel-shlwapi-l1-1-0.dll
0x77340000 - 0x77344000 C:\Windows\syswow64\api-ms-win-downlevel-version-l1-1-0.dll
0x76dc0000 - 0x76dc3000 C:\Windows\syswow64\api-ms-win-downlevel-normaliz-l1-1-0.dll
0x774c0000 - 0x774c3000 C:\Windows\syswow64\normaliz.DLL
0x75380000 - 0x75595000 C:\Windows\syswow64\iertutil.dll
0x778d0000 - 0x778d5000 C:\Windows\syswow64\api-ms-win-downlevel-advapi32-l1-1-0.dll
0x633d0000 - 0x63735000 C:\Program Files\TortoiseSVN\bin\libsvn_tsvn32.dll
0x633a0000 - 0x633c7000 C:\Program Files\TortoiseSVN\bin\libapr_tsvn32.dll
0x632c0000 - 0x63396000 C:\Windows\system32\MSVCR110.dll
0x63280000 - 0x632b8000 C:\Program Files\TortoiseSVN\bin\libaprutil_tsvn32.dll
0x63fe0000 - 0x63fed000 C:\Program Files\TortoiseSVN\bin\intl3_tsvn32.dll
0x63260000 - 0x63273000 C:\Program Files\TortoiseSVN\bin\libsasl32.dll
0x72190000 - 0x72198000 C:\Windows\system32\Secur32.dll
0x631d0000 - 0x63255000 C:\Windows\system32\MSVCP110.dll
0x63150000 - 0x63181000 C:\Windows\system32\EhStorShell.dll
0x71410000 - 0x71505000 C:\Windows\system32\PROPSYS.dll
0x71e00000 - 0x71e70000 C:\Windows\system32\ntshrui.dll
0x74310000 - 0x74329000 C:\Windows\system32\srvcli.dll
0x741d0000 - 0x741db000 C:\Windows\system32\cscapi.dll
0x72b60000 - 0x72b6a000 C:\Windows\system32\slc.dll
0x73990000 - 0x73a14000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.17514_none_ec83dffa859149af\comctl32.dll
0x6f950000 - 0x6fae8000 C:\Windows\system32\NetworkExplorer.dll
0x726a0000 - 0x726ce000 C:\Windows\System32\shdocvw.dll
0x741b0000 - 0x741c2000 C:\Windows\system32\MPR.dll
0x71980000 - 0x71988000 C:\Windows\System32\drprov.dll
0x73e70000 - 0x73e99000 C:\Windows\System32\WINSTA.dll
0x71920000 - 0x71934000 C:\Windows\System32\ntlanman.dll
0x73780000 - 0x73798000 C:\Windows\System32\davclnt.dll
0x73800000 - 0x73808000 C:\Windows\System32\DAVHLPR.dll
0x74300000 - 0x7430f000 C:\Windows\system32\wkscli.dll
0x5dc80000 - 0x5deb8000 C:\Windows\system32\wpdshext.dll
0x72ba0000 - 0x72d30000 C:\Windows\WinSxS\x86_microsoft.windows.gdiplus_6595b64144ccf1df_1.1.7601.17514_none_72d18a4386696c80\gdiplus.dll
0x60750000 - 0x607d9000 C:\Windows\system32\PortableDeviceApi.dll
0x64370000 - 0x6439b000 C:\Windows\system32\PortableDeviceTypes.dll
0x62270000 - 0x622af000 C:\Windows\system32\audiodev.dll
0x5d5b0000 - 0x5d817000 C:\Windows\system32\WMVCore.DLL
0x642e0000 - 0x6431d000 C:\Windows\system32\WMASF.DLL
0x64340000 - 0x64362000 C:\Windows\system32\EhStorAPI.dll
0x73070000 - 0x730a9000 C:\Windows\System32\MMDevApi.dll
VM Arguments:
jvm_args: -Djava.security.auth.login.config=jaas.config -Djava.net.preferIPv4Stack=true -XX:-UseSplitVerifier -Dfile.encoding=Cp1252 -Duser.language=en -Xmx128m -Djava.library.path=../lib/patch;../lib/level3;../lib/level2;../lib/level1;../lib;. -Dwrapper.key=dxgPdptyCLpDE9FBk2w3wf_1tGFfWs7N -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=14920 -Dwrapper.version=3.5.15-st -Dwrapper.native_library=wrapper -Dwrapper.cpu.timeout=180 -Dwrapper.jvmid=2 -Dwrapper.lang.domain=wrapper
java_command: net.inspiredbroadcast.legion.ikernel.core.Bootstrap
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=C:\ikernel\dist\lib;C:\ikernel\dist\lib;C:\ikernel\dist\bin\\..\lib
USERNAME=robert.dziwinski
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows 7 , 64 bit Build 7601 Service Pack 1
CPU:total 4 (2 cores per cpu, 2 threads per core) family 6 model 42 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht
Memory: 4k page, physical 4166716k(1004748k free), swap 8331576k(2764928k free)
vm_info: OpenJDK Client VM (22.0-b10) for windows-x86 JRE (1.7.0-INGG-administrator_2012_03_05_18_26-b00), built on Mar 5 2012 18:48:09 by "Administrator" with unknown MS VC++:1600
time: Tue Mar 17 14:39:08 2015
elapsed time: 3 seconds
How to investigate such issues?
Related
We have a NDIS LWF driver, and on a single machine we get a DPC_WATCHDOG_VIOLATION 133/1 bugcheck when they try to connect to their VPN to connect to the internet. This could be related to our NdisFIndicateReceiveNetBufferLists, as the IRQL is raised to DISPATCH before calling it (and obviously lowered to whatever it was afterward), and that does appear in the output of !dpcwatchdog shown below. This is done due to a workaround for another bug explained here:
IRQL_UNEXPECTED_VALUE BSOD after NdisFIndicateReceiveNetBufferLists?
Now this is the bugcheck:
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
DPC_WATCHDOG_VIOLATION (133)
The DPC watchdog detected a prolonged run time at an IRQL of DISPATCH_LEVEL
or above.
Arguments:
Arg1: 0000000000000001, The system cumulatively spent an extended period of time at
DISPATCH_LEVEL or above. The offending component can usually be
identified with a stack trace.
Arg2: 0000000000001e00, The watchdog period.
Arg3: fffff805422fb320, cast to nt!DPC_WATCHDOG_GLOBAL_TRIAGE_BLOCK, which contains
additional information regarding the cumulative timeout
Arg4: 0000000000000000
STACK_TEXT:
nt!KeBugCheckEx
nt!KeAccumulateTicks+0x1846b2
nt!KiUpdateRunTime+0x5d
nt!KiUpdateTime+0x4a1
nt!KeClockInterruptNotify+0x2e3
nt!HalpTimerClockInterrupt+0xe2
nt!KiCallInterruptServiceRoutine+0xa5
nt!KiInterruptSubDispatchNoLockNoEtw+0xfa
nt!KiInterruptDispatchNoLockNoEtw+0x37
nt!KxWaitForSpinLockAndAcquire+0x2c
nt!KeAcquireSpinLockAtDpcLevel+0x5c
wanarp!WanNdisReceivePackets+0x4bb
ndis!ndisMIndicateNetBufferListsToOpen+0x141
ndis!ndisMTopReceiveNetBufferLists+0x3f0e4
ndis!ndisCallReceiveHandler+0x61
ndis!ndisInvokeNextReceiveHandler+0x1df
ndis!NdisMIndicateReceiveNetBufferLists+0x104
ndiswan!IndicateRecvPacket+0x596
ndiswan!ApplyQoSAndIndicateRecvPacket+0x20b
ndiswan!ProcessPPPFrame+0x16f
ndiswan!ReceivePPP+0xb3
ndiswan!ProtoCoReceiveNetBufferListChain+0x442
ndis!ndisMCoIndicateReceiveNetBufferListsToNetBufferLists+0xf6
ndis!NdisMCoIndicateReceiveNetBufferLists+0x11
raspptp!CallIndicateReceived+0x210
raspptp!CallProcessRxNBLs+0x199
ndis!ndisDispatchIoWorkItem+0x12
nt!IopProcessWorkItem+0x135
nt!ExpWorkerThread+0x105
nt!PspSystemThreadStartup+0x55
nt!KiStartSystemThread+0x28
SYMBOL_NAME: wanarp!WanNdisReceivePackets+4bb
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: wanarp
IMAGE_NAME: wanarp.sys
And this following is the output of !dpcwatchdog, but I still can't find what is causing this bugcheck, and can't find which function is consuming too much time in DISPATCH level which is causing this bugcheck. Although I think this could be related to some spin locking done by wanarp? Could this be a bug with wanarp? Note that we don't use any spinlocking in our driver, and us raising the IRQL should not cause any issue as it is actually very common for indication in Ndis to be done at IRQL DISPATCH.
So How can I find the root cause of this bugcheck? There are no other third party LWF in the ndis stack.
3: kd> !dpcwatchdog
All durations are in seconds (1 System tick = 15.625000 milliseconds)
Circular Kernel Context Logger history: !logdump 0x2
DPC and ISR stats: !intstats /d
--------------------------------------------------
CPU#0
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
dpcs: no pending DPCs found
--------------------------------------------------
CPU#1
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
1: Normal : 0xfffff80542220e00 0xfffff805418dbf10 nt!PpmCheckPeriodicStart
1: Normal : 0xfffff80542231d40 0xfffff8054192c730 nt!KiBalanceSetManagerDeferredRoutine
1: Normal : 0xffffbd0146590868 0xfffff80541953200 nt!KiEntropyDpcRoutine
DPC Watchdog Captures Analysis for CPU #1.
DPC Watchdog capture size: 641 stacks.
Number of unique stacks: 1.
No common functions detected!
The captured stacks seem to indicate that only a single DPC or generic function is the culprit.
Try to analyse what other processors were doing at the time of the following reference capture:
CPU #1 DPC Watchdog Reference Stack (#0 of 641) - Time: 16 Min 17 Sec 984.38 mSec
# RetAddr Call Site
00 fffff805418d8991 nt!KiUpdateRunTime+0x5D
01 fffff805418d2803 nt!KiUpdateTime+0x4A1
02 fffff805418db1c2 nt!KeClockInterruptNotify+0x2E3
03 fffff80541808a45 nt!HalpTimerClockInterrupt+0xE2
04 fffff805419fab9a nt!KiCallInterruptServiceRoutine+0xA5
05 fffff805419fb107 nt!KiInterruptSubDispatchNoLockNoEtw+0xFA
06 fffff805418a9a9c nt!KiInterruptDispatchNoLockNoEtw+0x37
07 fffff805418da3cc nt!KxWaitForSpinLockAndAcquire+0x2C
08 fffff8054fa614cb nt!KeAcquireSpinLockAtDpcLevel+0x5C
09 fffff80546ba1eb1 wanarp!WanNdisReceivePackets+0x4BB
0a fffff80546be0b84 ndis!ndisMIndicateNetBufferListsToOpen+0x141
0b fffff80546ba7ef1 ndis!ndisMTopReceiveNetBufferLists+0x3F0E4
0c fffff80546bddfef ndis!ndisCallReceiveHandler+0x61
0d fffff80546ba4a94 ndis!ndisInvokeNextReceiveHandler+0x1DF
0e fffff8057c32d17e ndis!NdisMIndicateReceiveNetBufferLists+0x104
0f fffff8057c30d6c7 ndiswan!IndicateRecvPacket+0x596
10 fffff8057c32d56b ndiswan!ApplyQoSAndIndicateRecvPacket+0x20B
11 fffff8057c32d823 ndiswan!ProcessPPPFrame+0x16F
12 fffff8057c308e62 ndiswan!ReceivePPP+0xB3
13 fffff80546c5c006 ndiswan!ProtoCoReceiveNetBufferListChain+0x442
14 fffff80546c5c2d1 ndis!ndisMCoIndicateReceiveNetBufferListsToNetBufferLists+0xF6
15 fffff8057c2b0064 ndis!NdisMCoIndicateReceiveNetBufferLists+0x11
16 fffff8057c2b06a9 raspptp!CallIndicateReceived+0x210
17 fffff80546bd9dc2 raspptp!CallProcessRxNBLs+0x199
18 fffff80541899645 ndis!ndisDispatchIoWorkItem+0x12
19 fffff80541852b65 nt!IopProcessWorkItem+0x135
1a fffff80541871d25 nt!ExpWorkerThread+0x105
1b fffff80541a00778 nt!PspSystemThreadStartup+0x55
1c ---------------- nt!KiStartSystemThread+0x28
--------------------------------------------------
CPU#2
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
2: Normal : 0xffffbd01467f0868 0xfffff80541953200 nt!KiEntropyDpcRoutine
DPC Watchdog Captures Analysis for CPU #2.
DPC Watchdog capture size: 641 stacks.
Number of unique stacks: 1.
No common functions detected!
The captured stacks seem to indicate that only a single DPC or generic function is the culprit.
Try to analyse what other processors were doing at the time of the following reference capture:
CPU #2 DPC Watchdog Reference Stack (#0 of 641) - Time: 16 Min 17 Sec 984.38 mSec
# RetAddr Call Site
00 fffff805418d245a nt!KeClockInterruptNotify+0x453
01 fffff80541808a45 nt!HalpTimerClockIpiRoutine+0x1A
02 fffff805419fab9a nt!KiCallInterruptServiceRoutine+0xA5
03 fffff805419fb107 nt!KiInterruptSubDispatchNoLockNoEtw+0xFA
04 fffff805418a9a9c nt!KiInterruptDispatchNoLockNoEtw+0x37
05 fffff805418a9a68 nt!KxWaitForSpinLockAndAcquire+0x2C
06 fffff8054fa611cb nt!KeAcquireSpinLockRaiseToDpc+0x88
07 fffff80546ba1eb1 wanarp!WanNdisReceivePackets+0x1BB
08 fffff80546be0b84 ndis!ndisMIndicateNetBufferListsToOpen+0x141
09 fffff80546ba7ef1 ndis!ndisMTopReceiveNetBufferLists+0x3F0E4
0a fffff80546bddfef ndis!ndisCallReceiveHandler+0x61
0b fffff80546be3a81 ndis!ndisInvokeNextReceiveHandler+0x1DF
0c fffff80546ba804e ndis!ndisFilterIndicateReceiveNetBufferLists+0x3C611
0d fffff8054e384d77 ndis!NdisFIndicateReceiveNetBufferLists+0x6E
0e fffff8054e3811a9 ourdriver+0x4D70
0f fffff80546ba7d40 ourdriver+0x11A0
10 fffff8054182a6b5 ndis!ndisDummyIrpHandler+0x100
11 fffff80541c164c8 nt!IofCallDriver+0x55
12 fffff80541c162c7 nt!IopSynchronousServiceTail+0x1A8
13 fffff80541c15646 nt!IopXxxControlFile+0xC67
14 fffff80541a0aab5 nt!NtDeviceIoControlFile+0x56
15 ---------------- nt!KiSystemServiceCopyEnd+0x25
--------------------------------------------------
CPU#3
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
dpcs: no pending DPCs found
Target machine version: Windows 10 Kernel Version 19041 MP (4 procs)
Also note that we also pass the NDIS_RECEIVE_FLAGS_DISPATCH_LEVEL flag to the NdisFIndicateReceiveNetBufferLists, if the current IRQL is dispatch.
Edit1:
This is also the output of !locks and !qlocks and !ready, And the contention count on one of the resources is 49135, is this normal or too high? Could this be related to our issue? The threads that are waiting on it or own it are for normal processes such as chrome, csrss, etc.
3: kd> !kdexts.locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks.
Resource # nt!ExpTimeRefreshLock (0xfffff80542219440) Exclusively owned
Contention Count = 17
Threads: ffffcf8ce9dee640-01<*>
KD: Scanning for held locks.....
Resource # 0xffffcf8cde7f59f8 Shared 1 owning threads
Contention Count = 62
Threads: ffffcf8ce84ec080-01<*>
KD: Scanning for held locks...............................................................................................
Resource # 0xffffcf8ce08d0890 Exclusively owned
Contention Count = 49135
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 6
Threads: ffffcf8cf18e3080-01<*> ffffcf8ce3faf080-01
Threads Waiting On Exclusive Access:
ffffcf8ceb6ce080 ffffcf8ce1d20080 ffffcf8ce77f1080 ffffcf8ce92f4080
ffffcf8ce1d1f0c0 ffffcf8ced7c6080
KD: Scanning for held locks.
Resource # 0xffffcf8ce08d0990 Shared 1 owning threads
Threads: ffffcf8cf18e3080-01<*>
KD: Scanning for held locks.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Resource # 0xffffcf8ceff46350 Shared 1 owning threads
Threads: ffffcf8ce6de8080-01<*>
KD: Scanning for held locks......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Resource # 0xffffcf8cf0cade50 Exclusively owned
Contention Count = 3
Threads: ffffcf8ce84ec080-01<*>
KD: Scanning for held locks.........................
Resource # 0xffffcf8cf0f76180 Shared 1 owning threads
Threads: ffffcf8ce83dc080-02<*>
KD: Scanning for held locks.......................................................................................................................................................................................................................................................
Resource # 0xffffcf8cf1875cb0 Shared 1 owning threads
Contention Count = 3
Threads: ffffcf8ce89db040-02<*>
KD: Scanning for held locks.
Resource # 0xffffcf8cf18742d0 Shared 1 owning threads
Threads: ffffcf8cee5e1080-02<*>
KD: Scanning for held locks....................................................................................
Resource # 0xffffcf8cdceeece0 Shared 2 owning threads
Contention Count = 4
Threads: ffffcf8ce3a1c080-01<*> ffffcf8ce5625040-01<*>
Resource # 0xffffcf8cdceeed48 Shared 1 owning threads
Threads: ffffcf8ce5625043-02<*> *** Actual Thread ffffcf8ce5625040
KD: Scanning for held locks...
Resource # 0xffffcf8cf1d377d0 Exclusively owned
Threads: ffffcf8cf0ff3080-02<*>
KD: Scanning for held locks....
Resource # 0xffffcf8cf1807050 Exclusively owned
Threads: ffffcf8ce84ec080-01<*>
KD: Scanning for held locks......
245594 total locks, 13 locks currently held
3: kd> !qlocks
Key: O = Owner, 1-n = Wait order, blank = not owned/waiting, C = Corrupt
Processor Number
Lock Name 0 1 2 3
KE - Unused Spare
MM - Unused Spare
MM - Unused Spare
MM - Unused Spare
CC - Vacb
CC - Master
EX - NonPagedPool
IO - Cancel
CC - Unused Spare
IO - Vpb
IO - Database
IO - Completion
NTFS - Struct
AFD - WorkQueue
CC - Bcb
MM - NonPagedPool
3: kd> !ready
KSHARED_READY_QUEUE fffff8053f1ada00: (00) ****------------------------------------------------------------
SharedReadyQueue fffff8053f1ada00: No threads in READY state
Processor 0: No threads in READY state
Processor 1: Ready Threads at priority 15
THREAD ffffcf8ce9dee640 Cid 2054.2100 Teb: 000000fab7bca000 Win32Thread: 0000000000000000 READY on processor 1
Processor 2: No threads in READY state
Processor 3: No threads in READY state
3: kd> dt nt!_ERESOURCE 0xffffcf8ce08d0890
+0x000 SystemResourcesList : _LIST_ENTRY [ 0xffffcf8c`e08d0610 - 0xffffcf8c`e08cf710 ]
+0x010 OwnerTable : 0xffffcf8c`ee6e8210 _OWNER_ENTRY
+0x018 ActiveCount : 0n1
+0x01a Flag : 0xf86
+0x01a ReservedLowFlags : 0x86 ''
+0x01b WaiterPriority : 0xf ''
+0x020 SharedWaiters : 0xffffae09`adcae8e0 Void
+0x028 ExclusiveWaiters : 0xffffae09`a9aabea0 Void
+0x030 OwnerEntry : _OWNER_ENTRY
+0x040 ActiveEntries : 1
+0x044 ContentionCount : 0xbfef
+0x048 NumberOfSharedWaiters : 1
+0x04c NumberOfExclusiveWaiters : 6
+0x050 Reserved2 : (null)
+0x058 Address : (null)
+0x058 CreatorBackTraceIndex : 0
+0x060 SpinLock : 0
3: kd> dx -id 0,0,ffffcf8cdcc92040 -r1 (*((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8ce08d08c0))
(*((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8ce08d08c0)) [Type: _OWNER_ENTRY]
[+0x000] OwnerThread : 0xffffcf8cf18e3080 [Type: unsigned __int64]
[+0x008 ( 0: 0)] IoPriorityBoosted : 0x0 [Type: unsigned long]
[+0x008 ( 1: 1)] OwnerReferenced : 0x0 [Type: unsigned long]
[+0x008 ( 2: 2)] IoQoSPriorityBoosted : 0x1 [Type: unsigned long]
[+0x008 (31: 3)] OwnerCount : 0x1 [Type: unsigned long]
[+0x008] TableSize : 0xc [Type: unsigned long]
3: kd> dx -id 0,0,ffffcf8cdcc92040 -r1 ((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8cee6e8210)
((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8cee6e8210) : 0xffffcf8cee6e8210 [Type: _OWNER_ENTRY *]
[+0x000] OwnerThread : 0x0 [Type: unsigned __int64]
[+0x008 ( 0: 0)] IoPriorityBoosted : 0x1 [Type: unsigned long]
[+0x008 ( 1: 1)] OwnerReferenced : 0x1 [Type: unsigned long]
[+0x008 ( 2: 2)] IoQoSPriorityBoosted : 0x1 [Type: unsigned long]
[+0x008 (31: 3)] OwnerCount : 0x0 [Type: unsigned long]
[+0x008] TableSize : 0x7 [Type: unsigned long]
Thanks for reporting this. I've tracked this down to an OS bug: there's a deadlock in wanarp. This issue appears to affect every version of the OS going back to Windows Vista.
I've filed internal issue task.ms/42393356 to track this: if you have a Microsoft support contract, your rep can get you status updates on that issue.
Meanwhile, you can partially work around this issue by either:
Indicating 1 packet at a time (NumberOfNetBufferLists==1); or
Indicating on a single CPU at a time
The bug in wanarp is exposed when 2 or more CPUs collectively process 3 or more NBLs at the same time. So either workaround would avoid the trigger conditions.
Depending on how much bandwidth you're pushing through this network interface, those options could be rather bad for CPU/battery/throughput. So please try to avoid pessimizing batching unless it's really necessary. (For example, you could make this an option that's off-by-default, unless the customer specifically uses wanarp.)
Note that you cannot fully prevent the issue yourself. Other drivers in the stack, including NDIS itself, have the right to group packets together, which would have the side effect re-batching the packets that you carefully un-batched. However, I believe that you can make a statistically significant dent in the crashes if you just indicate 1 NBL at a time, or indicate multiple NBLs on 1 CPU at a time.
Sorry this is happening to you again! wanarp is... a very old codebase.
Following is error log file content, anybody help me？thanks
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006e7ea1be, pid=17964, tid=0x00000000000046d8
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x14a1be]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x000000001c619000): JavaThread "Disp-1" [_thread_in_vm, id=18136, stack(0x000000001df70000,0x000000001e070000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000000
Registers:
RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0x0000000000000900, RDX=0x0000000000000000
RSP=0x000000001e06e190, RBP=0x000000001e06f2a8, RSI=0x000000001c6191f8, RDI=0x000000001c619000
R8 =0x0000000000000000, R9 =0x000000001a2fafb8, R10=0x000000001d410550, R11=0x00000000d696d810
R12=0x0000000000000000, R13=0x000000001d7e55a8, R14=0x000000001e06f2d0, R15=0x000000001c619000
RIP=0x000000006e7ea1be, EFLAGS=0x0000000000010246
Top of Stack: (sp=0x000000001e06e190)
0x000000001e06e190: 000000001db87860 000000006e7fc623
0x000000001e06e1a0: 0000b1be7181fe0f 000000006e7e13cd
0x000000001e06e1b0: 000000001d7e55a8 000000001c6191f8
0x000000001e06e1c0: 000000001c619000 000000006e7ddd28
0x000000001e06e1d0: 000000001c619000 0000000000000000
0x000000001e06e1e0: 0000000000000000 000000001c619000
0x000000001e06e1f0: 0000000000000000 0000000002a3460c
0x000000001e06e200: 000000001d7e55b0 000000001e06f2a8
0x000000001e06e210: 000000001c619000 000000001e06f2a8
0x000000001e06e220: 000000001d410aa0 000000001c6191f8
0x000000001e06e230: 000000001dbac750 0000000000000034
0x000000001e06e240: 000000001d410aa0 000000001a2fafc8
0x000000001e06e250: 0000000002c9835d 000000001c619000
0x000000001e06e260: 000000001d7e55b0 0000000002a3446f
0x000000001e06e270: 000000001c6191f8 000000001db87860
0x000000001e06e280: 0000000000000000 000000001a2fafb8
Instructions: (pc=0x000000006e7ea1be)
0x000000006e7ea19e: 00 48 83 7f 08 00 48 89 7c 24 58 48 c7 44 24 60
0x000000006e7ea1ae: 00 00 00 00 74 0a 48 8d 4c 24 58 e8 02 18 13 00
0x000000006e7ea1be: 48 8b 0b e8 aa e3 f1 ff 4c 8b c7 33 d2 48 8b d8
0x000000006e7ea1ce: 48 8b 00 48 8b cb ff 50 78 48 83 7f 08 00 74 04
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=0x0000000000000000 is an unknown value
RCX=0x0000000000000900 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x000000001e06e190 is pointing into the stack for thread: 0x000000001c619000
RBP=0x000000001e06f2a8 is pointing into the stack for thread: 0x000000001c619000
RSI=0x000000001c6191f8 is an unknown value
RDI=0x000000001c619000 is a thread
R8 =0x0000000000000000 is an unknown value
R9 =0x000000001a2fafb8 is an unknown value
R10=0x000000001d410550 is an unknown value
R11=0x00000000d696d810 is an oop
[error occurred during error reporting (printing register info), id 0xc0000005]
Stack: [0x000000001df70000,0x000000001e070000], sp=0x000000001e06e190, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x14a1be]
C [tibrvj.dll+0x446f]
C [tibrvj.dll+0x46fe]
C [tibrvj.dll+0x6357]
C [tibrvcmq.dll+0xad17]
C [tibrv.dll+0x4ce28]
C [tibrv.dll+0x4d4a5]
C [tibrv.dll+0x4d633]
C [tibrvj.dll+0x16f6]
C 0x0000000002ca7f74
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.tibco.tibrv.TibrvImplQueueC.natDispatch(I)V+0
j com.tibco.tibrv.TibrvImplQueueC.dispatch()V+5
j com.tibco.tibrv.TibrvQueue.dispatch()V+14
j com.tibco.tibrv.TibrvDispatcher.run()V+112
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
=>0x000000001c619000 JavaThread "Disp-1" [_thread_in_vm, id=18136, stack(0x000000001df70000,0x000000001e070000)]
0x000000001778c000 JavaThread "Service Thread" daemon [_thread_blocked, id=416, stack(0x00000000193f0000,0x00000000194f0000)]
0x000000001774d000 JavaThread "C1 CompilerThread2" daemon [_thread_blocked, id=16164, stack(0x00000000192f0000,0x00000000193f0000)]
0x000000001774c000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=7268, stack(0x00000000191f0000,0x00000000192f0000)]
0x0000000017747000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=15800, stack(0x00000000190f0000,0x00000000191f0000)]
0x0000000017745800 JavaThread "Attach Listener" daemon [_thread_blocked, id=18400, stack(0x0000000018ff0000,0x00000000190f0000)]
0x0000000018b33800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=14912, stack(0x0000000018ef0000,0x0000000018ff0000)]
0x000000001771a800 JavaThread "Finalizer" daemon [_thread_blocked, id=9832, stack(0x0000000018990000,0x0000000018a90000)]
0x0000000002c7f000 JavaThread "Reference Handler" daemon [_thread_blocked, id=12364, stack(0x0000000018890000,0x0000000018990000)]
0x0000000002b90800 JavaThread "main" [_thread_blocked, id=13568, stack(0x0000000002a90000,0x0000000002b90000)]
Other Threads:
0x00000000176f8000 VMThread [stack: 0x0000000018790000,0x0000000018890000] [id=16944]
0x0000000018bb6000 WatcherThread [stack: 0x00000000194f0000,0x00000000195f0000] [id=1016]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap:
PSYoungGen total 70656K, used 12341K [0x00000000d6300000, 0x00000000dad00000, 0x0000000100000000)
eden space 65536K, 11% used [0x00000000d6300000,0x00000000d6a113e8,0x00000000da300000)
from space 5120K, 99% used [0x00000000da800000,0x00000000dacfc060,0x00000000dad00000)
to space 5120K, 0% used [0x00000000da300000,0x00000000da300000,0x00000000da800000)
ParOldGen total 86016K, used 1328K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 1% used [0x0000000082800000,0x000000008294c390,0x0000000087c00000)
Metaspace used 15572K, capacity 15896K, committed 16128K, reserved 1062912K
class space used 1932K, capacity 2012K, committed 2048K, reserved 1048576K
Card table byte_map: [0x0000000012050000,0x0000000012440000] byte_map_base: 0x0000000011c3c000
Marking Bits: (ParMarkBitMap*) 0x000000006eebb6d0
Begin Bits: [0x0000000012ae0000, 0x0000000014a40000)
End Bits: [0x0000000014a40000, 0x00000000169a0000)
Polling page: 0x0000000002670000
CodeCache: size=245760Kb used=7054Kb max_used=7082Kb free=238705Kb
bounds [0x0000000002c90000, 0x0000000003390000, 0x0000000011c90000]
total_blobs=2306 nmethods=1958 adapters=260
compilation: enabled
Compilation events (10 events):
Event: 3.200 Thread 0x000000001774c000 nmethod 1721 0x000000000335a690 code [0x000000000335a7c0, 0x000000000335a818]
Event: 3.200 Thread 0x000000001774c000 1954 4 java.util.AbstractMap::<init> (5 bytes)
Event: 3.200 Thread 0x000000001774c000 nmethod 1954 0x0000000003357a90 code [0x0000000003357bc0, 0x0000000003357c18]
Event: 3.200 Thread 0x000000001774c000 1790 4 java.util.LinkedList$ListItr::hasNext (20 bytes)
Event: 3.201 Thread 0x000000001774c000 nmethod 1790 0x0000000003357850 code [0x0000000003357980, 0x0000000003357a18]
Event: 3.201 Thread 0x000000001774c000 1654 4 java.util.HashMap$EntryIterator::next (5 bytes)
Event: 3.203 Thread 0x000000001774c000 nmethod 1654 0x000000000335b990 code [0x000000000335bae0, 0x000000000335bcb8]
Event: 3.203 Thread 0x000000001774c000 1657 4 org.springframework.core.type.classreading.AnnotationMetadataReadingVisitor::getAnnotationAttributes (50 bytes)
Event: 3.207 Thread 0x000000001774c000 nmethod 1657 0x000000000335d290 code [0x000000000335d400, 0x000000000335d698]
Event: 3.221 Thread 0x0000000017747000 nmethod 1656 0x00000000033729d0 code [0x0000000003372ee0, 0x00000000033791e0]
GC Heap History (8 events):
Event: 1.198 GC heap before
{Heap before GC invocations=1 (full 0):
PSYoungGen total 37888K, used 32768K [0x00000000d6300000, 0x00000000d8d00000, 0x0000000100000000)
eden space 32768K, 100% used [0x00000000d6300000,0x00000000d8300000,0x00000000d8300000)
from space 5120K, 0% used [0x00000000d8800000,0x00000000d8800000,0x00000000d8d00000)
to space 5120K, 0% used [0x00000000d8300000,0x00000000d8300000,0x00000000d8800000)
ParOldGen total 86016K, used 0K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082800000,0x0000000087c00000)
Metaspace used 8557K, capacity 8842K, committed 9088K, reserved 1056768K
class space used 1058K, capacity 1119K, committed 1152K, reserved 1048576K
Event: 1.206 GC heap after
Heap after GC invocations=1 (full 0):
PSYoungGen total 37888K, used 4074K [0x00000000d6300000, 0x00000000d8d00000, 0x0000000100000000)
eden space 32768K, 0% used [0x00000000d6300000,0x00000000d6300000,0x00000000d8300000)
from space 5120K, 79% used [0x00000000d8300000,0x00000000d86fa8d8,0x00000000d8800000)
to space 5120K, 0% used [0x00000000d8800000,0x00000000d8800000,0x00000000d8d00000)
ParOldGen total 86016K, used 80K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082814010,0x0000000087c00000)
Metaspace used 8557K, capacity 8842K, committed 9088K, reserved 1056768K
class space used 1058K, capacity 1119K, committed 1152K, reserved 1048576K
}
Event: 2.106 GC heap before
{Heap before GC invocations=2 (full 0):
PSYoungGen total 37888K, used 36842K [0x00000000d6300000, 0x00000000d8d00000, 0x0000000100000000)
eden space 32768K, 100% used [0x00000000d6300000,0x00000000d8300000,0x00000000d8300000)
from space 5120K, 79% used [0x00000000d8300000,0x00000000d86fa8d8,0x00000000d8800000)
to space 5120K, 0% used [0x00000000d8800000,0x00000000d8800000,0x00000000d8d00000)
ParOldGen total 86016K, used 80K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082814010,0x0000000087c00000)
Metaspace used 13165K, capacity 13474K, committed 13568K, reserved 1060864K
class space used 1653K, capacity 1747K, committed 1792K, reserved 1048576K
Event: 2.113 GC heap after
Heap after GC invocations=2 (full 0):
PSYoungGen total 37888K, used 5106K [0x00000000d6300000, 0x00000000d8d00000, 0x0000000100000000)
eden space 32768K, 0% used [0x00000000d6300000,0x00000000d6300000,0x00000000d8300000)
from space 5120K, 99% used [0x00000000d8800000,0x00000000d8cfc860,0x00000000d8d00000)
to space 5120K, 0% used [0x00000000d8300000,0x00000000d8300000,0x00000000d8800000)
ParOldGen total 86016K, used 473K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082876530,0x0000000087c00000)
Metaspace used 13165K, capacity 13474K, committed 13568K, reserved 1060864K
class space used 1653K, capacity 1747K, committed 1792K, reserved 1048576K
}
Event: 2.487 GC heap before
{Heap before GC invocations=3 (full 0):
PSYoungGen total 37888K, used 37874K [0x00000000d6300000, 0x00000000d8d00000, 0x0000000100000000)
eden space 32768K, 100% used [0x00000000d6300000,0x00000000d8300000,0x00000000d8300000)
from space 5120K, 99% used [0x00000000d8800000,0x00000000d8cfc860,0x00000000d8d00000)
to space 5120K, 0% used [0x00000000d8300000,0x00000000d8300000,0x00000000d8800000)
ParOldGen total 86016K, used 473K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082876530,0x0000000087c00000)
Metaspace used 13907K, capacity 14176K, committed 14336K, reserved 1062912K
class space used 1720K, capacity 1784K, committed 1792K, reserved 1048576K
Event: 2.496 GC heap after
Heap after GC invocations=3 (full 0):
PSYoungGen total 37888K, used 5032K [0x00000000d6300000, 0x00000000dad00000, 0x0000000100000000)
eden space 32768K, 0% used [0x00000000d6300000,0x00000000d6300000,0x00000000d8300000)
from space 5120K, 98% used [0x00000000d8300000,0x00000000d87ea1b0,0x00000000d8800000)
to space 5120K, 0% used [0x00000000da800000,0x00000000da800000,0x00000000dad00000)
ParOldGen total 86016K, used 481K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082878530,0x0000000087c00000)
Metaspace used 13907K, capacity 14176K, committed 14336K, reserved 1062912K
class space used 1720K, capacity 1784K, committed 1792K, reserved 1048576K
}
Event: 2.911 GC heap before
{Heap before GC invocations=4 (full 0):
PSYoungGen total 37888K, used 37800K [0x00000000d6300000, 0x00000000dad00000, 0x0000000100000000)
eden space 32768K, 100% used [0x00000000d6300000,0x00000000d8300000,0x00000000d8300000)
from space 5120K, 98% used [0x00000000d8300000,0x00000000d87ea1b0,0x00000000d8800000)
to space 5120K, 0% used [0x00000000da800000,0x00000000da800000,0x00000000dad00000)
ParOldGen total 86016K, used 481K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 0% used [0x0000000082800000,0x0000000082878530,0x0000000087c00000)
Metaspace used 15169K, capacity 15480K, committed 15616K, reserved 1062912K
class space used 1891K, capacity 1980K, committed 2048K, reserved 1048576K
Event: 2.922 GC heap after
Heap after GC invocations=4 (full 0):
PSYoungGen total 70656K, used 5104K [0x00000000d6300000, 0x00000000dad00000, 0x0000000100000000)
eden space 65536K, 0% used [0x00000000d6300000,0x00000000d6300000,0x00000000da300000)
from space 5120K, 99% used [0x00000000da800000,0x00000000dacfc060,0x00000000dad00000)
to space 5120K, 0% used [0x00000000da300000,0x00000000da300000,0x00000000da800000)
ParOldGen total 86016K, used 1328K [0x0000000082800000, 0x0000000087c00000, 0x00000000d6300000)
object space 86016K, 1% used [0x0000000082800000,0x000000008294c390,0x0000000087c00000)
Metaspace used 15169K, capacity 15480K, committed 15616K, reserved 1062912K
class space used 1891K, capacity 1980K, committed 2048K, reserved 1048576K
}
Deoptimization events (10 events):
Event: 2.145 Thread 0x0000000002b90800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00000000030441fc method=java.util.regex.Pattern$CharProperty.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z # 37
Event: 2.145 Thread 0x0000000002b90800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00000000030441fc method=java.util.regex.Pattern$CharProperty.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z # 37
Event: 2.145 Thread 0x0000000002b90800 Uncommon trap: reason=predicate action=maybe_recompile pc=0x0000000003026da8 method=java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z # 21
Event: 2.523 Thread 0x0000000002b90800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000003247108 method=java.util.AbstractCollection.isEmpty()Z # 4
Event: 2.524 Thread 0x0000000002b90800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000003261fd0 method=org.springframework.core.type.classreading.AnnotationReadingVisitorUtils.convertClassValues(Ljava/lang/Object;Ljava/lang/ClassLoader;Lorg/springframework/core/annotation/Ann
Event: 2.683 Thread 0x0000000002b90800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00000000030f7a68 method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; # 192
Event: 2.813 Thread 0x0000000002b90800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00000000030f7ae4 method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; # 206
Event: 2.989 Thread 0x0000000002b90800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00000000030ec124 method=java.util.Properties$LineReader.readLine()I # 180
Event: 3.017 Thread 0x0000000002b90800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000003179698 method=java.lang.String.startsWith(Ljava/lang/String;I)Z # 25
Event: 3.017 Thread 0x0000000002b90800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000002e2ca94 method=java.lang.String.startsWith(Ljava/lang/String;I)Z # 25
Internal exceptions (10 events):
Event: 2.974 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': org/springframework/beans/factory/SmartInitializingSingletonCustomizer> (0x00000000d66e33c8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDict
Event: 2.974 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': org/springframework/context/ApplicationContextAwareCustomizer> (0x00000000d66f6548) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDictionary.cp
Event: 2.975 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': java/lang/ObjectCustomizer> (0x00000000d6701db0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 2.976 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': org/springframework/context/event/DefaultEventListenerFactoryCustomizer> (0x00000000d6719af8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDic
Event: 2.976 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': org/springframework/context/event/EventListenerFactoryCustomizer> (0x00000000d6733d78) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDictionary
Event: 2.977 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': org/springframework/core/OrderedCustomizer> (0x00000000d6746268) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 2.977 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': java/lang/ObjectCustomizer> (0x00000000d6751a50) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 2.978 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': com/example/ControllerDemo/ControllerDemoApplicationCustomizer> (0x00000000d6767c20) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\share\vm\classfile\systemDictionary.c
Event: 2.978 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': com/example/ControllerDemo/ControllerDemoApplication$$EnhancerBySpringCGLIB$$b10776ccCustomizer> (0x00000000d6781448) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\shar
Event: 2.979 Thread 0x0000000002b90800 Exception <a 'java/lang/ClassNotFoundException': org/springframework/context/annotation/ConfigurationClassEnhancer$EnhancedConfigurationCustomizer> (0x00000000d679f4c8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u121\8372\hotspot\src\sh
Events (10 events):
Event: 3.025 loading class com/tibco/tibrv/TibrvImplCmTPortC done
Event: 3.031 loading class com/tibco/tibrv/TibrvCmListener
Event: 3.031 loading class com/tibco/tibrv/TibrvCmListener done
Event: 3.035 loading class com/tibco/tibrv/TibrvDispatcher
Event: 3.035 loading class com/tibco/tibrv/TibrvDispatcher done
Event: 3.037 loading class com/tibco/tibrv/TibrvQueueGroup
Event: 3.037 loading class com/tibco/tibrv/TibrvQueueGroup done
Event: 3.038 Thread 0x000000001c619000 Thread added: 0x000000001c619000
Event: 4.093 Executing VM operation: EnableBiasedLocking
Event: 4.093 Executing VM operation: EnableBiasedLocking done
Dynamic libraries:
0x00007ff6d9c80000 - 0x00007ff6d9cb7000 C:\ProgramData\Oracle\Java\javapath\java.exe
0x00007ffd2dde0000 - 0x00007ffd2dfbb000 C:\Windows\SYSTEM32\ntdll.dll
0x00007ffd2b780000 - 0x00007ffd2b82e000 C:\Windows\System32\KERNEL32.DLL
0x00007ffd2a650000 - 0x00007ffd2a899000 C:\Windows\System32\KERNELBASE.dll
0x00007ffd2b3b0000 - 0x00007ffd2b451000 C:\Windows\System32\ADVAPI32.dll
0x00007ffd2d5e0000 - 0x00007ffd2d67d000 C:\Windows\System32\msvcrt.dll
0x00007ffd2b6f0000 - 0x00007ffd2b749000 C:\Windows\System32\sechost.dll
0x00007ffd2cf70000 - 0x00007ffd2d095000 C:\Windows\System32\RPCRT4.dll
0x00007ffd2d490000 - 0x00007ffd2d5da000 C:\Windows\System32\USER32.dll
0x00007ffd2b270000 - 0x00007ffd2b28e000 C:\Windows\System32\win32u.dll
0x00007ffd2ddb0000 - 0x00007ffd2ddd7000 C:\Windows\System32\GDI32.dll
0x00007ffd2a410000 - 0x00007ffd2a597000 C:\Windows\System32\gdi32full.dll
0x00007ffd2a370000 - 0x00007ffd2a40a000 C:\Windows\System32\msvcp_win.dll
0x00007ffd2a8a0000 - 0x00007ffd2a996000 C:\Windows\System32\ucrtbase.dll
0x00007ffd080c0000 - 0x00007ffd08327000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.15063.483_none_26002d27e7c744a2\COMCTL32.dll
0x00007ffd2d110000 - 0x00007ffd2d409000 C:\Windows\System32\combase.dll
0x00007ffd2a300000 - 0x00007ffd2a36a000 C:\Windows\System32\bcryptPrimitives.dll
0x00007ffd2b8f0000 - 0x00007ffd2b91d000 C:\Windows\System32\IMM32.DLL
0x000000006fe40000 - 0x000000006ff12000 C:\Program Files\Java\jre1.8.0_121\bin\msvcr100.dll
0x000000006e6a0000 - 0x000000006ef3b000 C:\Program Files\Java\jre1.8.0_121\bin\server\jvm.dll
0x00007ffd2b3a0000 - 0x00007ffd2b3a8000 C:\Windows\System32\PSAPI.DLL
0x00007ffd1e1b0000 - 0x00007ffd1e1ba000 C:\Windows\SYSTEM32\VERSION.dll
0x00007ffd227c0000 - 0x00007ffd227e3000 C:\Windows\SYSTEM32\WINMM.dll
0x00007ffd12a10000 - 0x00007ffd12a19000 C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ffd2d0a0000 - 0x00007ffd2d10c000 C:\Windows\System32\WS2_32.dll
0x00007ffd22640000 - 0x00007ffd2266b000 C:\Windows\SYSTEM32\winmmbase.dll
0x00007ffd2a5a0000 - 0x00007ffd2a5e9000 C:\Windows\System32\cfgmgr32.dll
0x0000000070b20000 - 0x0000000070b2f000 C:\Program Files\Java\jre1.8.0_121\bin\verify.dll
0x000000006fff0000 - 0x0000000070019000 C:\Program Files\Java\jre1.8.0_121\bin\java.dll
0x000000006ffd0000 - 0x000000006ffe6000 C:\Program Files\Java\jre1.8.0_121\bin\zip.dll
0x00007ffd2ba80000 - 0x00007ffd2ceb7000 C:\Windows\System32\SHELL32.dll
0x00007ffd2cec0000 - 0x00007ffd2cf6a000 C:\Windows\System32\shcore.dll
0x00007ffd2ab70000 - 0x00007ffd2b261000 C:\Windows\System32\windows.storage.dll
0x00007ffd2b340000 - 0x00007ffd2b391000 C:\Windows\System32\shlwapi.dll
0x00007ffd2a250000 - 0x00007ffd2a261000 C:\Windows\System32\kernel.appcore.dll
0x00007ffd2a290000 - 0x00007ffd2a2dc000 C:\Windows\System32\powrprof.dll
0x00007ffd2a2e0000 - 0x00007ffd2a2f5000 C:\Windows\System32\profapi.dll
0x000000006e690000 - 0x000000006e69d000 C:\Program Files\Java\jre1.8.0_121\bin\management.dll
0x000000006e670000 - 0x000000006e68a000 C:\Program Files\Java\jre1.8.0_121\bin\net.dll
0x00007ffd29b00000 - 0x00007ffd29b5c000 C:\Windows\system32\mswsock.dll
0x00007ffd1d840000 - 0x00007ffd1d856000 C:\Windows\system32\napinsp.dll
0x00007ffd1d820000 - 0x00007ffd1d83a000 C:\Windows\system32\pnrpnsp.dll
0x00007ffd24140000 - 0x00007ffd24158000 C:\Windows\system32\NLAapi.dll
0x00007ffd298e0000 - 0x00007ffd29984000 C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ffd2b750000 - 0x00007ffd2b758000 C:\Windows\System32\NSI.dll
0x00007ffd298a0000 - 0x00007ffd298d7000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ffd1d810000 - 0x00007ffd1d81e000 C:\Windows\System32\winrnr.dll
0x00007ffd1d7f0000 - 0x00007ffd1d805000 C:\Windows\System32\wshbth.dll
0x00007ffd1d7e0000 - 0x00007ffd1d7ea000 C:\Windows\System32\rasadhlp.dll
0x00007ffd22940000 - 0x00007ffd229ab000 C:\Windows\System32\fwpuclnt.dll
0x00007ffd2a120000 - 0x00007ffd2a145000 C:\Windows\SYSTEM32\bcrypt.dll
0x0000000002a30000 - 0x0000000002a41000 C:\tibco\tibrv\8.4\bin\tibrvj.dll
0x0000000070a50000 - 0x0000000070b19000 C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.9307_none_88daf3492fb349de\MSVCR80.dll
0x0000000002a60000 - 0x0000000002a75000 C:\tibco\tibrv\8.4\bin\tibrvcmq.dll
0x000000001c040000 - 0x000000001c0ca000 C:\tibco\tibrv\8.4\bin\tibrv.dll
0x000000001c0d0000 - 0x000000001c0f2000 C:\tibco\tibrv\8.4\bin\tibrvcm.dll
0x000000001c120000 - 0x000000001c12d000 C:\tibco\tibrv\8.4\bin\tibrvft.dll
0x00007ffd2a150000 - 0x00007ffd2a180000 C:\Windows\SYSTEM32\SspiCli.dll
0x00007ffd244c0000 - 0x00007ffd24669000 C:\Windows\SYSTEM32\dbghelp.dll
VM Arguments:
java_command: ControllerDemo-0.0.1-SNAPSHOT.jar
java_class_path (initial): ControllerDemo-0.0.1-SNAPSHOT.jar
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_121
CLASSPATH=C:\tibco\tibrv\8.4\lib\tibrvnative.jar;.;C:\Users\YaQingHe\AppData\Local\Temp\LREC93D.tmp\lib\dt.jar;C:\Users\YaQingHe\AppData\Local\Temp\LREC93D.tmp\lib\tools.jar;
PATH=C:\app\YaQingHe\product\12.1.0\dbhome_1\bin;C:\tibco\tibrv\8.4\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;"C:\Users\YaQingHe\AppData\Local\Temp\LREC93D.tmp\bin;C:\Users\YaQingHe\AppData\Local\Temp\LREC93D.tmp\jre\bin;";C:\0\project\csot\apache-maven-3.5.0\bin;C:\0\project\csot\apache-activemq-5.15.0\bin;C:\Users\YaQingHe\AppData\Local\Microsoft\WindowsApps;
USERNAME=YaQingHe
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows 10.0 , 64 bit Build 15063 (10.0.15063.296)
CPU:total 4 (2 cores per cpu, 2 threads per core) family 6 model 142 stepping 9, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx
Memory: 4k page, physical 8223584k(2894684k free), swap 12024672k(1774324k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for windows-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 18:21:36 by "java_re" with MS VC++ 10.0 (VS2010)
time: Fri Mar 02 15:49:06 2018
elapsed time: 15 seconds (0d 0h 0m 15s)
Shot in the dark answer since there is not much info:
I had a similar problem, jvm crashed when receiving TIB-reply, after upgrading tib client stuff from 8.1.2 to 8.3+.
What helped me was to use tibrvnative.jar instead of tibrvj.jar.
Could you provide additional info about external dependencies that you are using? Try to use compatible versions of dependencies.
The application is basically a Servlet on top of Apaches Tomcat 8 and crashes every 3 to n days. The most interesting part is, that the last 3 crashes happened at 1.30 PM + 3 to 4 seconds. Every other crash happens at 15, 30, 45 or the full hour. The same application worked for months on Windows 7 with Java 7 something.
I found this report so I checked the available space at /tmp and I have plenty of space here. However, the next time I ran the VM with:
-XX:-UsePerfData
-XX:+PerfDisableSharedMem
and the hsperfdata file is no longer written but the problem persist. According to this report PerfData might be still used to some extend even if it is disabled.
It seems to be this piece of code:
void PerfLongVariant::sample() {
// JJJ - This should not happen. Maybe the first sample is taken
// while the _sample_helper is being null'ed out.
// assert(_sample_helper != NULL || _sampled != NULL, "unexpected state");
if (_sample_helper == NULL) return;
if (_sample_helper != NULL) {
*(jlong*)_valuep = _sample_helper->take_sample();
}
else if (_sampled != NULL) {
*(jlong*)_valuep = *_sampled;
}
}
I assume the problem occures here:
*(jlong*)_valuep = _sample_helper->take_sample();
but I am not sure which implementation is used at runtime:
statSampler seems unlikely as I don't see the connection to elapsed_counter except for the above mentioned phenomena with the crash time.
/*
* helper class to provide for sampling of the elapsed_counter value
* maintained in the OS class.
*/
class HighResTimeSampler : public PerfSampleHelper {
public:
jlong take_sample() { return os::elapsed_counter(); }
};
or gSpaceCounter
inline jlong take_sample() {
return _gen->used();
}
and spaceCounters and cSpaceCounter I have the impression that the size of one of the pools in the Java memory pool is returned before it is updated. Which ultimately lead to SIGBUS 7, bad memory access.
What are my options?
The last crash report (dynamic libraries removed):
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f463c87a82e, pid=10876, tid=0x00007f4624cfc700
#
# JRE version: OpenJDK Runtime Environment (8.0_101-b13) (build 1.8.0_101-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.101-b13 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.1.0
# Distribution: Custom build (Fri Jul 29 23:44:32 UTC 2016)
# Problematic frame:
# V [libjvm.so+0x7f582e] PerfLongVariant::sample()+0x1e
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
--------------- T H R E A D ---------------
Current thread (0x00007f46340ce800): WatcherThread [stack: 0x00007f4624bfc000,0x00007f4624cfd000] [id=10909]
siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007f463dc89420
Registers:
RAX=0x0000000007114e40, RBX=0x00007f463dc89420, RCX=0x0000000000000000, RDX=0x00007f463ce2d210
RSP=0x00007f4624cfbe00, RBP=0x00007f4624cfbe10, RSI=0x0000000000000032, RDI=0x00007f4634018fa0
R8 =0x00007f463ce69a48, R9 =0x0000000000000001, R10=0x00182e09a9af66e8, R11=0x0000000000000000
R12=0x0000000000000008, R13=0x00007f463401b1a0, R14=0x0000000000000002, R15=0x00007f463ce6f3c0
RIP=0x00007f463c87a82e, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000007
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f4624cfbe00)
0x00007f4624cfbe00: 00007f46340ce800 0000000000000001
0x00007f4624cfbe10: 00007f4624cfbe40 00007f463c8ecd44
0x00007f4624cfbe20: 00007f4624cfbe40 0000000000000001
0x00007f4624cfbe30: 0000000000000032 0000000000000032
0x00007f4624cfbe40: 00007f4624cfbe90 00007f463c92e380
0x00007f4624cfbe50: 00007f46340cfad0 00007f4634007630
0x00007f4624cfbe60: 0000000000000000 0000000000000032
0x00007f4624cfbe70: 00007f46340ce800 00007f46340ce800
0x00007f4624cfbe80: 0000000000000000 00007f4624cfc700
0x00007f4624cfbe90: 00007f4624cfbf10 00007f463c954bf7
0x00007f4624cfbea0: 0000000000000000 0000000000000000
0x00007f4624cfbeb0: 0000000000000001 00007f46340ce800
0x00007f4624cfbec0: 00007f4624cfbf10 00007f463c80b8cf
0x00007f4624cfbed0: 0000000000000000 00007f46340cfbc0
0x00007f4624cfbee0: 00007f46340cfad0 00007f46340cfbc0
0x00007f4624cfbef0: 00007f46340cfad0 00007f46340ce800
0x00007f4624cfbf00: 0000000000000000 00007f4624cfc700
0x00007f4624cfbf10: 00007f4624cfbf50 00007f463c83e6d2
0x00007f4624cfbf20: 0000000000000000 0000000000000000
0x00007f4624cfbf30: 0000000000000000 00007f463dc70b5f
0x00007f4624cfbf40: 0000000000000000 00007f4624cfc700
0x00007f4624cfbf50: 0000000000000000 00007f463ce92734
0x00007f4624cfbf60: 0000000000000000 00007f4624cfc700
0x00007f4624cfbf70: 00007f4624cfc700 c5fe0a2abaafece4
0x00007f4624cfbf80: 0000000000000000 00007f463dc70b5f
0x00007f4624cfbf90: 0000000000000000 00007f4624cfc700
0x00007f4624cfbfa0: 3b7243b5c46fece4 3b7273f8f755ece4
0x00007f4624cfbfb0: 0000000000000000 0000000000000000
0x00007f4624cfbfc0: 0000000000000000 0000000000000000
0x00007f4624cfbfd0: 0000000000000000 0000000000000000
0x00007f4624cfbfe0: 0000000000000000 0000000000000000
0x00007f4624cfbff0: 00007f4624cfc700 00007f463d5aad3d
Instructions: (pc=0x00007f463c87a82e)
0x00007f463c87a80e: a0 ff 48 8b 47 38 48 85 c0 74 1e 55 48 89 e5 53
0x00007f463c87a81e: 48 83 ec 08 48 8b 5f 28 48 8b 10 48 89 c7 ff 12
0x00007f463c87a82e: 48 89 03 48 83 c4 08 5b 5d f3 c3 90 66 0f 1f 44
0x00007f463c87a83e: 00 00 55 48 8d 05 08 02 5b 00 48 89 e5 41 57 41
Stack: [0x00007f4624bfc000,0x00007f4624cfd000], sp=0x00007f4624cfbe00, free space=1023k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x7f582e] PerfLongVariant::sample()+0x1e
V [libjvm.so+0x867d44] StatSampler::sample_data(PerfDataList*)+0x34
V [libjvm.so+0x8a9380] PeriodicTask::real_time_tick(int)+0x90
V [libjvm.so+0x8cfbf7] WatcherThread::run()+0x47
V [libjvm.so+0x7b96d2] java_start(Thread*)+0xe2
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f45f0011000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=15853, stack(0x00007f45e72f9000,0x00007f45e73fa000)]
0x00007f45f0010000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=15852, stack(0x00007f45f48ce000,0x00007f45f49cf000)]
0x00007f45f0018800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=15851, stack(0x00007f45e75fa000,0x00007f45e76fb000)]
0x00007f4600699000 JavaThread "BackendCache" daemon [_thread_blocked, id=15363, stack(0x00007f45e76fb000,0x00007f45e77fc000)]
0x00007f46352da800 JavaThread "Thread-9" daemon [_thread_blocked, id=15362, stack(0x00007f45e77fc000,0x00007f45e78fd000)]
0x00007f45ea07c800 JavaThread "Timer-0" daemon [_thread_blocked, id=15356, stack(0x00007f45f4063000,0x00007f45f4164000)]
0x00007f4634dd2000 JavaThread "Thread-4" daemon [_thread_blocked, id=15355, stack(0x00007f45f4564000,0x00007f45f4665000)]
0x00007f45ec04b000 JavaThread "http-apr-8080-exec-10" daemon [_thread_blocked, id=11253, stack(0x00007f45f49cf000,0x00007f45f4ad0000)]
0x00007f45f000b000 JavaThread "http-apr-8080-exec-9" daemon [_thread_blocked, id=11252, stack(0x00007f45f4ad0000,0x00007f45f4bd1000)]
0x00007f45f0009800 JavaThread "http-apr-8080-exec-8" daemon [_thread_blocked, id=11251, stack(0x00007f45f4bd1000,0x00007f45f4cd2000)]
0x00007f45f0007800 JavaThread "http-apr-8080-exec-7" daemon [_thread_blocked, id=11250, stack(0x00007f45f4cd2000,0x00007f45f4dd3000)]
0x00007f45f0006000 JavaThread "http-apr-8080-exec-6" daemon [_thread_blocked, id=11249, stack(0x00007f45f4dd3000,0x00007f45f4ed4000)]
0x00007f45ec03f800 JavaThread "http-apr-8080-exec-5" daemon [_thread_blocked, id=11248, stack(0x00007f45f4ed4000,0x00007f45f4fd5000)]
0x00007f45f0004000 JavaThread "http-apr-8080-exec-4" daemon [_thread_blocked, id=11247, stack(0x00007f45f4fd5000,0x00007f45f50d6000)]
0x00007f45f0002800 JavaThread "http-apr-8080-exec-3" daemon [_thread_blocked, id=10969, stack(0x00007f45f50d6000,0x00007f45f51d7000)]
0x00007f45ec079000 JavaThread "http-apr-8080-exec-2" daemon [_thread_blocked, id=10968, stack(0x00007f45f60c4000,0x00007f45f61c5000)]
0x00007f45f0001800 JavaThread "http-apr-8080-exec-1" daemon [_thread_blocked, id=10967, stack(0x00007f45f5fc3000,0x00007f45f60c4000)]
0x00007f463459b800 JavaThread "ajp-apr-8009-AsyncTimeout" daemon [_thread_blocked, id=10930, stack(0x00007f45f53d7000,0x00007f45f54d8000)]
0x00007f4634599800 JavaThread "ajp-apr-8009-Acceptor-0" daemon [_thread_in_native, id=10929, stack(0x00007f45f54d8000,0x00007f45f55d9000)]
0x00007f4634597800 JavaThread "ajp-apr-8009-Poller" daemon [_thread_blocked, id=10928, stack(0x00007f45f55d9000,0x00007f45f56da000)]
0x00007f4634595800 JavaThread "http-apr-8080-AsyncTimeout" daemon [_thread_blocked, id=10927, stack(0x00007f45f56da000,0x00007f45f57db000)]
0x00007f4634594800 JavaThread "http-apr-8080-Acceptor-0" daemon [_thread_in_native, id=10926, stack(0x00007f45f59db000,0x00007f45f5adc000)]
0x00007f4634592800 JavaThread "http-apr-8080-Sendfile" daemon [_thread_blocked, id=10925, stack(0x00007f45f5adc000,0x00007f45f5bdd000)]
0x00007f4634591000 JavaThread "http-apr-8080-Poller" daemon [_thread_blocked, id=10924, stack(0x00007f45f5c94000,0x00007f45f5d95000)]
0x00007f463458d800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=10923, stack(0x00007f45f5ec2000,0x00007f45f5fc3000)]
0x00007f4634515800 JavaThread "GC Daemon" daemon [_thread_blocked, id=10911, stack(0x00007f45f7434000,0x00007f45f7535000)]
0x00007f4634145800 JavaThread "AsyncFileHandlerWriter-1878246837" daemon [_thread_blocked, id=10910, stack(0x00007f4624afb000,0x00007f4624bfc000)]
0x00007f46340cc000 JavaThread "Service Thread" daemon [_thread_blocked, id=10908, stack(0x00007f4624cfd000,0x00007f4624dfe000)]
0x00007f46340c7800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=10907, stack(0x00007f4624dfe000,0x00007f4624eff000)]
0x00007f46340ba000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=10906, stack(0x00007f4624eff000,0x00007f4625000000)]
0x00007f46340b8000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10905, stack(0x00007f4638052000,0x00007f4638153000)]
0x00007f463408d800 JavaThread "Finalizer" daemon [_thread_blocked, id=10904, stack(0x00007f4638197000,0x00007f4638298000)]
0x00007f4634089000 JavaThread "Reference Handler" daemon [_thread_blocked, id=10903, stack(0x00007f4638298000,0x00007f4638399000)]
0x00007f4634008000 JavaThread "main" [_thread_in_native, id=10899, stack(0x00007f463db71000,0x00007f463dc72000)]
Other Threads:
0x00007f463407f800 VMThread [stack: 0x00007f4638399000,0x00007f463849a000] [id=10902]
=>0x00007f46340ce800 WatcherThread [stack: 0x00007f4624bfc000,0x00007f4624cfd000] [id=10909]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007f4634007630] PeriodicTask_lock - owner thread: 0x00007f46340ce800
Heap:
PSYoungGen total 161792K, used 115891K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 71% used [0x00000000f6180000,0x00000000fd294e40,0x00000000fff00000)
from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
to space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Card table byte_map: [0x00007f463aaea000,0x00007f463abd9000] byte_map_base: 0x00007f463a3d8000
Marking Bits: (ParMarkBitMap*) 0x00007f463ce6d740
Begin Bits: [0x00007f4639a08000, 0x00007f463a178000)
End Bits: [0x00007f463a178000, 0x00007f463a8e8000)
Polling page: 0x00007f463dc91000
CodeCache: size=245760Kb used=13583Kb max_used=14910Kb free=232176Kb
bounds [0x00007f4625000000, 0x00007f4625ec0000, 0x00007f4634000000]
total_blobs=4397 nmethods=3905 adapters=407
compilation: enabled
GC Heap History (10 events):
Event: 241117.477 GC heap before
{Heap before GC invocations=451 (full 2):
PSYoungGen total 161792K, used 161456K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
from space 512K, 34% used [0x00000000fff80000,0x00000000fffac030,0x0000000100000000)
to space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31406K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 241117.480 GC heap after
Heap after GC invocations=451 (full 2):
PSYoungGen total 161792K, used 128K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
from space 512K, 25% used [0x00000000fff00000,0x00000000fff20020,0x00000000fff80000)
to space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31406K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}
Event: 241985.401 GC heap before
{Heap before GC invocations=453 (full 2):
PSYoungGen total 161792K, used 161408K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
from space 512K, 25% used [0x00000000fff80000,0x00000000fffa0020,0x0000000100000000)
to space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 241985.404 GC heap after
Heap after GC invocations=453 (full 2):
PSYoungGen total 161792K, used 96K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
to space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}
Event: 242421.299 GC heap before
{Heap before GC invocations=454 (full 2):
PSYoungGen total 161792K, used 161376K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
to space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 242421.301 GC heap after
Heap after GC invocations=454 (full 2):
PSYoungGen total 161792K, used 112K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
from space 512K, 21% used [0x00000000fff80000,0x00000000fff9c010,0x0000000100000000)
to space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}
Event: 242856.234 GC heap before
{Heap before GC invocations=455 (full 2):
PSYoungGen total 161792K, used 161392K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
from space 512K, 21% used [0x00000000fff80000,0x00000000fff9c010,0x0000000100000000)
to space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 242856.236 GC heap after
Heap after GC invocations=455 (full 2):
PSYoungGen total 161792K, used 96K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
to space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
ParOldGen total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
Metaspace used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
class space used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}
Internal exceptions (10 events):
Event: 57534.673 Thread 0x00007f4634dd2000 Exception <a 'java/lang/ClassNotFoundException': org/hibernate/util/CloneableCustomizer> (0x00000000faf05e08) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 57534.674 Thread 0x00007f4634dd2000 Exception <a 'java/lang/ClassNotFoundException': org/hibernate/event/EventListenersCustomizer> (0x00000000faf3abc8) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 57540.284 Thread 0x00007f4634bee000 Exception <a 'java/net/SocketTimeoutException': connect timed out> (0x00000000fcb2fac8) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 57943.337 Thread 0x00007f46352da800 Implicit null exception at 0x00007f4625d42a9f to 0x00007f4625d4931d
Event: 57943.340 Thread 0x00007f4600699000 Implicit null exception at 0x00007f4625d42a9f to 0x00007f4625d4931d
Event: 58820.887 Thread 0x00007f45ea078800 Implicit null exception at 0x00007f462514eab4 to 0x00007f462514f049
Event: 61135.304 Thread 0x00007f4634dd2000 Implicit null exception at 0x00007f46258c9192 to 0x00007f46258ca9a4
Event: 61135.314 Thread 0x00007f4634dd2000 Implicit null exception at 0x00007f46256f414e to 0x00007f46256f4371
Event: 61135.315 Thread 0x00007f4634dd2000 Implicit null exception at 0x00007f4625902079 to 0x00007f4625903285
Event: 63765.574 Thread 0x00007f45f0012800 Exception <a 'java/lang/InterruptedException'> (0x00000000fb423328) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/runtime/objectMonitor.cpp, line 1495]
VM Arguments:
jvm_args: -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/lib/jvm/jre
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/sh
--------------- S Y S T E M ---------------
OS:SUSE Linux Enterprise Server 12 (x86_64)
VERSION = 12
PATCHLEVEL = 2
uname:Linux 4.4.49-92.11-default #1 SMP Fri Feb 17 08:29:30 UTC 2017 (8f9478a) x86_64
libc:glibc 2.22 NPTL 2.22
rlimit: STACK 8192k, CORE 0k, NPROC 7522, NOFILE 4096, AS infinity
load average:0.00 0.00 0.00
/proc/meminfo:
MemTotal: 1943796 kB
MemFree: 117384 kB
MemAvailable: 1056832 kB
Buffers: 8 kB
Cached: 1116816 kB
SwapCached: 3152 kB
Active: 1430656 kB
Inactive: 284260 kB
Active(anon): 438228 kB
Inactive(anon): 233656 kB
Active(file): 992428 kB
Inactive(file): 50604 kB
Unevictable: 80 kB
Mlocked: 80 kB
SwapTotal: 4187132 kB
SwapFree: 4181900 kB
Dirty: 156 kB
Writeback: 0 kB
AnonPages: 595020 kB
Mapped: 68856 kB
Shmem: 73792 kB
Slab: 75320 kB
SReclaimable: 55520 kB
SUnreclaim: 19800 kB
KernelStack: 3664 kB
PageTables: 5648 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 5159028 kB
Committed_AS: 1247032 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 471040 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 366528 kB
DirectMap2M: 1730560 kB
CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 37 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, aes, clmul, tsc, tscinvbit
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 37
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 1
microcode : 0x13
cpu MHz : 2400.085
cache size : 12288 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm ida arat epb dtherm
bugs :
bogomips : 4800.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 37
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 1
microcode : 0x13
cpu MHz : 2400.085
cache size : 12288 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm ida arat epb dtherm
bugs :
bogomips : 4800.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 1943796k(117384k free), swap 4187132k(4181900k free)
vm_info: OpenJDK 64-Bit Server VM (25.101-b13) for linux-amd64 JRE (1.8.0_101-b13), built on Jul 30 2016 00:00:27 by "abuild" with gcc 4.8.5
time: Sat Nov 11 13:30:04 2017
elapsed time: 243167 seconds (2d 19h 32m 47s)
Register to memory mapping:
RAX=0x0000000007114e40 is an unknown value
RBX=0x00007f463dc89420 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x00007f463ce2d210: <offset 0xda8210> in /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server/libjvm.so at 0x00007f463c085000
RSP=0x00007f4624cfbe00 is an unknown value
RBP=0x00007f4624cfbe10 is an unknown value
RSI=0x0000000000000032 is an unknown value
RDI=0x00007f4634018fa0 is an unknown value
R8 =0x00007f463ce69a48: <offset 0xde4a48> in /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server/libjvm.so at 0x00007f463c085000
R9 =0x0000000000000001 is an unknown value
R10=0x00182e09a9af66e8 is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x0000000000000008 is an unknown value
R13=0x00007f463401b1a0 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x00007f463ce6f3c0: <offset 0xdea3c0> in /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server/libjvm.so at 0x00007f463c085000
When I try to use the help command in cqlsh I get a segfault:
Connected to Test Cluster at localhost:9160.
[cqlsh 4.0.1 | Cassandra 2.0.1 | CQL spec 3.1.1 | Thrift protocol 19.37.0]
Use HELP for help.
cqlsh> help
Segmentation fault: 11
Cassandra was installed using homebrew. Java version:
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
I tried to change per-thread stack size in cassandra-env.sh.
Values I tried: 256k, 300k, 512k, 1024k.
I verified that cassandra uses these values by executing command:
cassandra -f
output for stack size of 300k :
xss = -ea -javaagent:/usr/local/Cellar/cassandra/2.0.1/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4G -Xmx4G -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss300k
I'm on OS X Mavericks and I have 16 gigs of RAM if that's relevant.
edit:
I don't have similar problems with cassandra-cli.
Per Nikolai Grigoriev's suggestion I took a look at the system log:
cat system.log | tail -n 20
relevant output:
Oct 29 23:10:12 Andrews-MacBook-Pro.local ReportCrash[5690]: Saved crash report for python[5685] version 0 to /Users/andrew/Library/Logs/DiagnosticReports/python_2013-10-29-231012_Andrews-MacBook-Pro.crash
output of:
cat /Users/andrew/Library/Logs/DiagnosticReports/python_2013-10-29-231012_Andrews-MacBook-Pro.crash
is here:
Process: python [5685]
Path: /Users/USER/*/python
Identifier: python
Version: 0
Code Type: X86-64 (Native)
Parent Process: bash [1330]
Responsible: iTerm [1313]
User ID: 501
Date/Time: 2013-10-29 23:10:10.697 +0100
OS Version: Mac OS X 10.9 (13A603)
Report Version: 11
Anonymous UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sleep/Wake UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
VM Regions Near 0:
-->
__TEXT 0000000109ab3000-0000000109ab4000 [ 4K] r-x/rwx SM=COW /Users/USER/*
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 readline.so 0x000000010a229f17 call_readline + 599 (readline.c:1087)
1 libpython2.7.dylib 0x0000000109abb354 PyOS_Readline + 244 (myreadline.c:211)
2 libpython2.7.dylib 0x0000000109b50fdd builtin_raw_input + 365 (bltinmodule.c:2046)
3 libpython2.7.dylib 0x0000000109b57e2b PyEval_EvalFrameEx + 18011 (ceval.c:4021)
4 libpython2.7.dylib 0x0000000109b5b0e7 PyEval_EvalCodeEx + 2103 (ceval.c:3253)
5 libpython2.7.dylib 0x0000000109b5b298 fast_function + 296 (ceval.c:4117)
6 libpython2.7.dylib 0x0000000109b57edd PyEval_EvalFrameEx + 18189 (ceval.c:4042)
7 libpython2.7.dylib 0x0000000109b5b0e7 PyEval_EvalCodeEx + 2103 (ceval.c:3253)
8 libpython2.7.dylib 0x0000000109b5b298 fast_function + 296 (ceval.c:4117)
9 libpython2.7.dylib 0x0000000109b57edd PyEval_EvalFrameEx + 18189 (ceval.c:4042)
10 libpython2.7.dylib 0x0000000109b5b0e7 PyEval_EvalCodeEx + 2103 (ceval.c:3253)
11 libpython2.7.dylib 0x0000000109ae83cb function_call + 347 (funcobject.c:526)
12 libpython2.7.dylib 0x0000000109abfa21 PyObject_Call + 97 (abstract.c:2529)
13 libpython2.7.dylib 0x0000000109b5879a PyEval_EvalFrameEx + 20426 (ceval.c:4334)
14 libpython2.7.dylib 0x0000000109b5b0e7 PyEval_EvalCodeEx + 2103 (ceval.c:3253)
15 libpython2.7.dylib 0x0000000109b5b166 PyEval_EvalCode + 54 (ceval.c:665)
16 libpython2.7.dylib 0x0000000109b79934 PyRun_FileExFlags + 164 (pythonrun.c:1354)
17 libpython2.7.dylib 0x0000000109b7abe8 PyRun_SimpleFileExFlags + 392 (pythonrun.c:943)
18 libpython2.7.dylib 0x0000000109b8e3da Py_Main + 2874 (main.c:639)
19 libdyld.dylib 0x00007fff90eee5fd start + 1
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x0000000000000004 rcx: 0x00007fa1f8d00000 rdx: 0x0000000000000000
rdi: 0x0000000000000000 rsi: 0x00007fa1f8dbbcf0 rbp: 0x00007fff5614b970 rsp: 0x00007fff5614b8a0
r8: 0x00007fa1f8d00000 r9: 0x0000000000000bbc r10: 0x000000000020000b r11: 0x0000000000000001
r12: 0x00007fff5614b8c0 r13: 0x00007fa1f8d0dc00 r14: 0x0000000000000001 r15: 0x000000010a252960
rip: 0x000000010a229f17 rfl: 0x0000000000010206 cr2: 0x0000000000000000
Logical CPU: 0
Error Code: 0x00000004
Trap Number: 14
Binary Images:
0x109ab3000 - 0x109ab3ff7 +python (0) <7A897AB3-9853-3A76-93BE-F864DB4ECC8D> /Users/USER/*/python
0x109ab6000 - 0x109bf2fff +libpython2.7.dylib (0) <A6E60E14-9E72-34C5-874C-600772A7F298> /Users/USER/*/libpython2.7.dylib
0x10a0e2000 - 0x10a0e8ff7 +itertools.so (0) <045A60E6-26D8-394A-B37E-F25D54307FED> /Users/USER/*/itertools.so
0x10a0f3000 - 0x10a0f4fff +_functools.so (0) <765061F4-C740-386C-B0EF-73D9E03AECB2> /Users/USER/*/_functools.so
0x10a0f7000 - 0x10a10bff7 +_ctypes.so (0) <5406D7E2-2D30-313B-B3B6-46EB46F31F8B> /Users/USER/*/_ctypes.so
0x10a11c000 - 0x10a120ff7 +_struct.so (0) <374ECB70-B58E-3D8A-9E1D-65A5F4F62D7C> /Users/USER/*/_struct.so
0x10a128000 - 0x10a12bff7 +strop.so (0) <14A6EEF5-4C55-32D1-B8CF-5EDE2E43B7B2> /Users/USER/*/strop.so
0x10a130000 - 0x10a132fff +time.so (0) <DA27EF10-D49C-3536-879D-237E5A73EC45> /Users/USER/*/time.so
0x10a138000 - 0x10a13cfff +operator.so (0) <FECCF5F1-1BE1-316D-BFC0-C3F841DC9DE8> /Users/USER/*/operator.so
0x10a143000 - 0x10a144ff7 +_locale.so (0) <EE16349A-9907-3DEA-B419-7E39D4C6DCF2> /Users/USER/*/_locale.so
0x10a148000 - 0x10a14bff7 +_collections.so (0) <F2E4C6A7-C3E8-3AC5-9153-16A76BC0B857> /Users/USER/*/_collections.so
0x10a151000 - 0x10a151ff7 +_bisect.so (0) <6E1E38C3-D597-356F-9CCF-D077E2030B7B> /Users/USER/*/_bisect.so
0x10a154000 - 0x10a155ff7 +_heapq.so (0) <B5765696-25FC-36F3-9CD8-B5F02E69DE5F> /Users/USER/*/_heapq.so
0x10a219000 - 0x10a21cfff +_csv.so (0) <417DF404-51EB-3D9B-9F13-96C971352277> /Users/USER/*/_csv.so
0x10a222000 - 0x10a223ff7 +cStringIO.so (0) <9ABC813E-BDA3-3129-A31C-564144C1821C> /Users/USER/*/cStringIO.so
0x10a228000 - 0x10a22afff +readline.so (0) <1DD349FC-7CCC-3D87-98C1-39EE6AFE5226> /Users/USER/*/readline.so
0x10a231000 - 0x10a24fffb libedit.3.dylib (39) <1B0596DB-F336-32E7-BB9F-51BF70DB5305> /usr/lib/libedit.3.dylib
0x10a260000 - 0x10a266fff +math.so (0) <B8D32AA6-1136-3D9B-8F3D-063E373CE751> /Users/USER/*/math.so
0x10a26d000 - 0x10a274ff7 +_socket.so (0) <9B1A31D6-C635-34F7-A605-82CD7FAAB760> /Users/USER/*/_socket.so
0x10a27e000 - 0x10a282fff +_ssl.so (0) <C1F761A1-803F-34C0-99D6-61E90020294E> /Users/USER/*/_ssl.so
0x10a288000 - 0x10a295fff +datetime.so (0) <C0564CBB-516A-385B-8B96-6767B73BEDC3> /Users/USER/*/datetime.so
0x10a2a1000 - 0x10a2a6fff +_json.so (0) <293437E3-C642-3B8C-85A1-9AB3304B3F05> /Users/USER/*/_json.so
0x10a2ab000 - 0x10a2adff7 +binascii.so (0) <46D54602-4338-36A3-85BC-4866FAEA6036> /Users/USER/*/binascii.so
0x10a2b1000 - 0x10a2b4fff +select.so (0) <C84B6BEF-4DE8-3BD1-8B5D-BD8E22295D6E> /Users/USER/*/select.so
0x10a2ba000 - 0x10a2bbff7 +fcntl.so (0) <B687409B-BF1D-3104-82A5-BF0F642185B6> /Users/USER/*/fcntl.so
0x10a3bf000 - 0x10a3c2ff7 +zlib.so (0) <E3BB63E5-B4ED-35D2-80CB-C3614CCDAA22> /Users/USER/*/zlib.so
0x10a447000 - 0x10a449ff7 +fastbinary.so (0) <AAF034FA-51DB-38A0-BF1C-01F5C19B1D46> /Users/USER/*/fastbinary.so
0x7fff6d77e000 - 0x7fff6d7b1817 dyld (239.3) <D1DFCF3F-0B0C-332A-BCC0-87A851B570FF> /usr/lib/dyld
0x7fff882c8000 - 0x7fff882caff3 libsystem_configuration.dylib (596.12) <C4F633D9-94C8-35D9-BB2D-84C5122533C7> /usr/lib/system/libsystem_configuration.dylib
0x7fff88365000 - 0x7fff883eeff7 libsystem_c.dylib (997.1.1) <61833FAA-7281-3FF9-937F-686B6F20427C> /usr/lib/system/libsystem_c.dylib
0x7fff88488000 - 0x7fff884caff7 libauto.dylib (185.5) <F45C36E8-B606-3886-B5B1-B6745E757CA8> /usr/lib/libauto.dylib
0x7fff88c80000 - 0x7fff88c87fff libcompiler_rt.dylib (35) <4CD916B2-1B17-362A-B403-EF24A1DAC141> /usr/lib/system/libcompiler_rt.dylib
0x7fff88ff2000 - 0x7fff890d1fff libcrypto.0.9.8.dylib (50) <B95B9DBA-39D3-3EEF-AF43-44608B28894E> /usr/lib/libcrypto.0.9.8.dylib
0x7fff897c8000 - 0x7fff897c9fff libunc.dylib (28) <62682455-1862-36FE-8A04-7A6B91256438> /usr/lib/system/libunc.dylib
0x7fff897e3000 - 0x7fff897e5ff7 libquarantine.dylib (71) <7A1A2BCB-C03D-3A25-BFA4-3E569B2D2C38> /usr/lib/system/libquarantine.dylib
0x7fff89830000 - 0x7fff8984afff libdispatch.dylib (339.1.9) <46878A5B-4248-3057-962C-6D4A235EEF31> /usr/lib/system/libdispatch.dylib
0x7fff89f8d000 - 0x7fff89f8efff com.apple.TrustEvaluationAgent (2.0 - 25) <334A82F4-4AE4-3719-A511-86D0B0723E2B> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
0x7fff8a3e9000 - 0x7fff8a3f3fff libcommonCrypto.dylib (60049) <8C4F0CA0-389C-3EDC-B155-E62DD2187E1D> /usr/lib/system/libcommonCrypto.dylib
0x7fff8a46d000 - 0x7fff8a46effb libremovefile.dylib (33) <3543F917-928E-3DB2-A2F4-7AB73B4970EF> /usr/lib/system/libremovefile.dylib
0x7fff8a91f000 - 0x7fff8a925ff7 libsystem_platform.dylib (24.1.4) <331BA4A5-55CE-3B95-99EB-44E0C89D7FB8> /usr/lib/system/libsystem_platform.dylib
0x7fff8ad08000 - 0x7fff8ad19ff7 libz.1.dylib (53) <42E0C8C6-CA38-3CA4-8619-D24ED5DD492E> /usr/lib/libz.1.dylib
0x7fff8ad8a000 - 0x7fff8adbefff libssl.0.9.8.dylib (50) <B15F967C-B002-36C2-9621-3456D8509F50> /usr/lib/libssl.0.9.8.dylib
0x7fff8aee2000 - 0x7fff8aefeff7 libsystem_kernel.dylib (2422.1.72) <D14913DB-47F1-3591-8DAF-D4B4EF5F8818> /usr/lib/system/libsystem_kernel.dylib
0x7fff8b98e000 - 0x7fff8b992fff libsystem_stats.dylib (93.1.26) <B9E26A9E-FBBC-3938-B8B7-6CF7CA8C99AD> /usr/lib/system/libsystem_stats.dylib
0x7fff8bc70000 - 0x7fff8bc71ff7 libsystem_blocks.dylib (63) <FB856CD1-2AEA-3907-8E9B-1E54B6827F82> /usr/lib/system/libsystem_blocks.dylib
0x7fff8cb46000 - 0x7fff8cb4bff7 libunwind.dylib (35.3) <78DCC358-2FC1-302E-B395-0155B47CB547> /usr/lib/system/libunwind.dylib
0x7fff8ccec000 - 0x7fff8ce99f27 libobjc.A.dylib (551.1) <AD7FD984-271E-30F4-A361-6B20319EC73B> /usr/lib/libobjc.A.dylib
0x7fff8d889000 - 0x7fff8d88aff7 libDiagnosticMessagesClient.dylib (100) <4CDB0F7B-C0AF-3424-BC39-495696F0DB1E> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff8dd93000 - 0x7fff8dd93ff7 libkeymgr.dylib (28) <3AA8D85D-CF00-3BD3-A5A0-E28E1A32A6D8> /usr/lib/system/libkeymgr.dylib
0x7fff8e0cd000 - 0x7fff8e0d2fff libmacho.dylib (845) <1D2910DF-C036-3A82-A3FD-44FF73B5FF9B> /usr/lib/system/libmacho.dylib
0x7fff8e511000 - 0x7fff8e518ff7 liblaunch.dylib (842.1.4) <FCBF0A02-0B06-3F97-9248-5062A9DEB32C> /usr/lib/system/liblaunch.dylib
0x7fff8e5c8000 - 0x7fff8e5efffb libsystem_info.dylib (449.1.3) <7D41A156-D285-3849-A2C3-C04ADE797D98> /usr/lib/system/libsystem_info.dylib
0x7fff8e8af000 - 0x7fff8e8b0ff7 libSystem.B.dylib (1197.1.1) <BFC0DC97-46C6-3BE0-9983-54A98734897A> /usr/lib/libSystem.B.dylib
0x7fff8ee3b000 - 0x7fff8eff3ff3 libicucore.A.dylib (511.25) <3ED7B656-416E-3071-AEC8-E85C90232F78> /usr/lib/libicucore.A.dylib
0x7fff90c52000 - 0x7fff90c59ff7 libsystem_pthread.dylib (53.1.4) <AB498556-B555-310E-9041-F67EC9E00E2C> /usr/lib/system/libsystem_pthread.dylib
0x7fff90eeb000 - 0x7fff90eeeff7 libdyld.dylib (239.3) <62F4D752-4089-31A8-8B73-B95A68893B3C> /usr/lib/system/libdyld.dylib
0x7fff9155e000 - 0x7fff91587ff7 libc++abi.dylib (48) <8C16158F-CBF8-3BD7-BEF4-022704B2A326> /usr/lib/libc++abi.dylib
0x7fff91634000 - 0x7fff91635ff7 libsystem_sandbox.dylib (278.10) <A47E7E11-3C76-318E-B67D-98972B86F094> /usr/lib/system/libsystem_sandbox.dylib
0x7fff91777000 - 0x7fff9177ffff libsystem_dnssd.dylib (522.1.11) <270DCF6C-502D-389A-AA9F-DE4624A36FF7> /usr/lib/system/libsystem_dnssd.dylib
0x7fff91bec000 - 0x7fff91c1cfff libncurses.5.4.dylib (42) <BF763D62-9149-37CB-B1D2-F66A2510E6DD> /usr/lib/libncurses.5.4.dylib
0x7fff91fb6000 - 0x7fff91fc7ff7 libsystem_asl.dylib (217.1.4) <655FB343-52CF-3E2F-B14D-BEBF5AAEF94D> /usr/lib/system/libsystem_asl.dylib
0x7fff92396000 - 0x7fff923b1ff7 libsystem_malloc.dylib (23.1.10) <FFE5C472-B23A-318A-85BF-77CDE61900D1> /usr/lib/system/libsystem_malloc.dylib
0x7fff932f1000 - 0x7fff9333ffff libcorecrypto.dylib (161.1) <F3973C28-14B6-3006-BB2B-00DD7F09ABC7> /usr/lib/system/libcorecrypto.dylib
0x7fff93530000 - 0x7fff93582fff libc++.1.dylib (120) <4F68DFC5-2077-39A8-A449-CAC5FDEE7BDE> /usr/lib/libc++.1.dylib
0x7fff93b66000 - 0x7fff93b6fff3 libsystem_notify.dylib (121) <52571EC3-6894-37E4-946E-064B021ED44E> /usr/lib/system/libsystem_notify.dylib
0x7fff93b8a000 - 0x7fff93b8eff7 libcache.dylib (62) <BDC1E65B-72A1-3DA3-A57C-B23159CAAD0B> /usr/lib/system/libcache.dylib
0x7fff941a8000 - 0x7fff941cfff7 libsystem_network.dylib (241.3) <8B1E1F1D-A5CC-3BAE-8B1E-ABC84337A364> /usr/lib/system/libsystem_network.dylib
0x7fff94515000 - 0x7fff94539fff libxpc.dylib (300.1.17) <4554927A-9467-365C-91F1-5A116989DD7F> /usr/lib/system/libxpc.dylib
0x7fff94afa000 - 0x7fff94b01ff3 libcopyfile.dylib (103) <5A881779-D0D6-3029-B371-E3021C2DDA5E> /usr/lib/system/libcopyfile.dylib
0x7fff94c7e000 - 0x7fff94cadfd2 libsystem_m.dylib (3047.16) <B7F0E2E4-2777-33FC-A787-D6430B630D54> /usr/lib/system/libsystem_m.dylib
0x7fff95794000 - 0x7fff95979ff7 com.apple.CoreFoundation (6.9 - 855.11) <E22C6A1F-8996-349C-905E-96C3BBE07C2F> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 29978
thread_create: 2
thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=78.0M resident=65.2M(84%) swapped_out_or_unallocated=12.8M(16%)
Writable regions: Total=52.1M written=7924K(15%) resident=9.9M(19%) swapped_out=0K(0%) unallocated=42.2M(81%)
REGION TYPE VIRTUAL
=========== =======
Kernel Alloc Once 4K
MALLOC 43.6M
MALLOC (admin) 16K
STACK GUARD 56.0M
Stack 8192K
VM_ALLOCATE 16K
__DATA 1784K
__LINKEDIT 66.2M
__TEXT 11.8M
__UNICODE 544K
shared memory 4K
=========== =======
TOTAL 188.0M
I grabbed the distribution of Cassandra from the source. The problem persists. Could it be really a JVM bug?
How to fix that?
SOLVED
It's not a JVM bug. I use pythonbrew for managing python interpreters (forgot to mention). When I installed cql on global interpreter everything went fine. Apparently there are some problems with python path. I'll use global python interpreter when using Cassandra then.
I narrowed the problem down to pythonbrew. No wonder it's deprecated. There's no segfault when I use pyenv.
I had similar problem with Python crashing on OS X 10.9.5 when running cqlsh.
The solution was to install https://www.python.org/ftp/python/2.7.9/python-2.7.9-macosx10.6.pkg
Do you have the similar problem when running other tools (cassandra-cli etc)? Is there anything in the kernel log that might give a hint? It is most likely a bug in the JVM itself. Cassandra does not include by default any native libs. I am wondering if homebrew installs also something like JNA library or JEMalloc with Cassandra.
I would suggest to grab the distribution of Cassandra from the source (it would be pure Java). If it crashes the same way - it would be a JVM bug. And one of thousands of issues with Maverick ;( If not, then search carefully for additional native libraries you have got with homebrew package of Cassandra.
I'm trying to publish one of my games on the Mac AppStore, but I got the following report from the App Review team :
Here is how we found this crash:
The app crashes on second launch of the app.
and here the crash log :
Process: Tennis Elbow Manager [19919]
Path: /Applications/Tennis Elbow Manager.app/Contents/MacOS/Tennis Elbow Manager
Identifier: com.managames.Tennis-Elbow-Manager
Version: 1.9.38 (38)
App Item ID: 0
App External ID: 0
Code Type: X86 (Native)
Parent Process: launchd [160]
User ID: 201
Date/Time: 2013-03-06 16:22:44.968 -0800
OS Version: Mac OS X 10.8.2 (12C54)
Report Version: 10
Crashed Thread: 1 Dispatch queue: com.apple.libdispatch-manager
Exception Type: EXC_CRASH (SIGBUS)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Thread 0:: Dispatch queue: com.apple.main-thread
0 com.managames.Tennis-Elbow-Manager 0x000e3214 0x6b000 + 492052
1 ??? 0x492e5245 0 + 1227772485
Thread 1 Crashed:: Dispatch queue: com.apple.libdispatch-manager
0 libsystem_kernel.dylib 0x9a6979ae kevent + 10
1 libdispatch.dylib 0x9775dc71 _dispatch_mgr_invoke + 993
2 libdispatch.dylib 0x9775d7a9 _dispatch_mgr_thread + 53
Thread 2:
0 libsystem_kernel.dylib 0x9a6970ee __workq_kernreturn + 10
1 libsystem_c.dylib 0x9105d04c _pthread_workq_return + 45
2 libsystem_c.dylib 0x9105ce19 _pthread_wqthread + 448
3 libsystem_c.dylib 0x91044cca start_wqthread + 30
Thread 1 crashed with X86 Thread State (32-bit):
eax: 0x00000004 ebx: 0xb0102700 ecx: 0xb01025ac edx: 0x9a6979ae
edi: 0x00000000 esi: 0x9775d8a1 ebp: 0xb0102718 esp: 0xb01025ac
ss: 0x00000023 efl: 0x00000247 eip: 0x9a6979ae cs: 0x0000000b
ds: 0x00000023 es: 0x00000023 fs: 0x00000023 gs: 0x0000000f
cr2: 0x0269e000
Logical CPU: 0
Binary Images:
0x6b000 - 0x192ff3 +com.managames.Tennis-Elbow-Manager (1.9.38 - 38) <B6D93CA3-ADB3-3C94-AB00-25C63C74250A> /Applications/Tennis Elbow Manager.app/Contents/MacOS/Tennis Elbow Manager
0x38b000 - 0x3d6fc0 +SDL (1.2.14 - 1.2.14) <84ABD881-2313-3168-889A-EDEA4EBAA928> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL.framework/Versions/A/SDL
0x3e9000 - 0x400ff7 +org.libsdl.SDL-mixer (1.2.12 - 1.2.12) <879D26B7-6FC0-32FD-BA92-484F89677158> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_mixer.framework/Versions/A/SDL_mixer
0x433000 - 0x437ffb +org.libsdl.SDL-image (1.2.12 - 1.2.12) <C8A71F96-C7E3-3290-A886-ECA7994E847D> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_image.framework/Versions/A/SDL_image
0x441000 - 0x46fff7 +mikmod (3.1.10r2 - 3.1.12) <484ED43A-1C22-DF87-240A-594F8F37B9BE> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_mixer.framework/Versions/A/Frameworks/mikmod.framework/Versions/A/mikmod
0x484000 - 0x4bbffa +smpeg (0.0.1d1) <900C8FCF-8E29-3941-AE32-14F3208DBAE8> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_mixer.framework/Versions/A/Frameworks/smpeg.framework/Versions/A/smpeg
0x4e5000 - 0x4e8ff4 +org.xiph.ogg (1.1.4 - 1.1.4) <B4E7F924-42D6-3E2A-9EF0-E55CD71BF308> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_mixer.framework/Versions/A/Frameworks/Ogg.framework/Versions/A/Ogg
0x4ef000 - 0x672ff2 +org.xiph.vorbis (1.2.3 - 1.2.3) <F9FCDD01-A9B1-35AD-B1DB-DE0C8F347869> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_mixer.framework/Versions/A/Frameworks/Vorbis.framework/Versions/A/Vorbis
0x6a6000 - 0x6eeff4 +FLAC (1.2.1 - 1.2.1) <5F0B6D50-0DFC-357C-89EA-CA56EDEC997D> /Applications/Tennis Elbow Manager.app/Contents/Frameworks/SDL_mixer.framework/Versions/A/Frameworks/FLAC.framework/Versions/A/FLAC
[...]
Full log is here => http://a1.phobos.apple.com/us/r30/Purple/v4/da/b2/03/dab203db-b866-3d55-8ea9-88df1de4c285/temp..cqdsammn.crash
(gdb) info line *0x000e3214
gives :
No line number information available for address 0xe3214 <_ZN10CSysCursorD1Ev+2958>
The game is working well on my OSX 10.8.2, both with my account & a guest account, I can launch it many times without issue.
The crash seems to happen right on start-up before it gets into any of my functions (ie: it's still probably in the main() or even the start).
I looked for identical crash "Thread 1 Crashed:: Dispatch queue: com.apple.libdispatch-manager" on Google & I could find a few almost identical reports, mainly about a Skype version, but that didn't teach me much...
So now I'm kinda stuck, and my only idea is to resubmit hoping it was a rare random crash, but I doubt it's the best move...
Anyone would have a better idea ? Thanks in advance for any answer ! :)
I'm using : SDL 1.2.15, SDL_Image 1.2.12, SDL_Mixer 1.2.12, roddi's ValidateStoreReceipt from https://github.com/roddi/ValidateStoreReceipt .