WebSphere spring-websocket clientOutboundChannelExecutor deadlock - websphere-liberty

We are using spring-websocket with WebSphere 16.0.0.3
And Time after time we are facing issue when all clientOutboundChannelExecutor are stuck.
3XMTHREADINFO "clientOutboundChannel-1" J9VMThread:0x0000000001834900, j9thread_t:0x00007F2EDEFD1030, java/lang/Thread:0x00000000E2E30A28, state:B, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x44, isDaemon:false)
3XMTHREADINFO1 (native thread ID:0xDF6A, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x01000201)
3XMTHREADINFO2 (native stack address range from:0x00007F2EDE11A000, to:0x00007F2EDE15B000, size:0x41000)
3XMCPUTIME CPU usage total: 49.720264588 secs, current category="Application"
3XMTHREADBLOCK Blocked on: java/lang/Object#0x00000000E30B30F8 Owned by: "clientOutboundChannel-4" (J9VMThread:0x00000000019CEE00, java/lang/Thread:0x00000000E2E30C98)
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.resetRequest(AbstractHttpSockJsSession.java:334(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.disconnect(AbstractHttpSockJsSession.java:330(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.close(AbstractSockJsSession.java:216(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/WebSocketSessionDecorator.close(WebSocketSessionDecorator.java:158(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:192(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:457(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))
3XMTHREADINFO "clientOutboundChannel-2" J9VMThread:0x00000000019A5A00, j9thread_t:0x00007F2EDEFD15B0, java/lang/Thread:0x00000000E2E30AF8, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x45, isDaemon:false)
3XMTHREADINFO1 (native thread ID:0xDF6B, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x01000101)
3XMTHREADINFO2 (native stack address range from:0x00007F2EDDE5F000, to:0x00007F2EDDEA0000, size:0x41000)
3XMCPUTIME CPU usage total: 0.281253777 secs, current category="Application"
3XMTHREADBLOCK Waiting on: com/ibm/ws/tcpchannel/internal/SimpleSync#0x00000000E3155FC8 Owned by: <unowned>
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:172)
4XESTACKTRACE at com/ibm/ws/tcpchannel/internal/SimpleSync.simpleWait(SimpleSync.java:35)
5XESTACKTRACE (entered lock: com/ibm/ws/tcpchannel/internal/SimpleSync#0x00000000E3155FC8, entry count: 1)
4XESTACKTRACE at com/ibm/ws/tcpchannel/internal/NioTCPWriteRequestContextImpl.processSyncWriteRequest(NioTCPWriteRequestContextImpl.java:77(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcpchannel/internal/TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:119(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.synchWrite(HttpServiceContextImpl.java:2438(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2174(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/inbound/HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:753(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flushBuffers(HttpOutputStreamImpl.java:481(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flush(HttpOutputStreamImpl.java:579(Compiled Code))
4XESTACKTRACE at com/ibm/wsspi/http/ee7/HttpOutputStreamEE7.flush(HttpOutputStreamEE7.java:293(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/osgi/response/WCOutputStream.flush(WCOutputStream.java:234(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer31/osgi/response/WCOutputStream31.flush(WCOutputStream31.java:599(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:705(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:670)
4XESTACKTRACE at org/springframework/http/server/ServletServerHttpResponse.flush(ServletServerHttpResponse.java:96(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.writeFrameInternal(AbstractHttpSockJsSession.java:365)
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.writeFrame(AbstractSockJsSession.java:322(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/StreamingSockJsSession.flushCache(StreamingSockJsSession.java:86(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.sendMessageInternal(AbstractHttpSockJsSession.java:302(Compiled Code))
5XESTACKTRACE (entered lock: java/lang/Object#0x00000000E33995A8, entry count: 1)
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.sendMessage(AbstractSockJsSession.java:166(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.tryFlushMessageBuffer(ConcurrentWebSocketSessionDecorator.java:132(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.sendMessage(ConcurrentWebSocketSessionDecorator.java:104(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:440(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))
3XMTHREADINFO "clientOutboundChannel-3" J9VMThread:0x00000000019C9D00, j9thread_t:0x00007F2EDEFDAE00, java/lang/Thread:0x00000000E2E30BC8, state:B, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x46, isDaemon:false)
3XMTHREADINFO1 (native thread ID:0xDF6C, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x01000201)
3XMTHREADINFO2 (native stack address range from:0x00007F2EDDE1E000, to:0x00007F2EDDE5F000, size:0x41000)
3XMCPUTIME CPU usage total: 42.712892035 secs, current category="Application"
3XMTHREADBLOCK Blocked on: java/lang/Object#0x00000000E33995A8 Owned by: "clientOutboundChannel-2" (J9VMThread:0x00000000019A5A00, java/lang/Thread:0x00000000E2E30AF8)
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.resetRequest(AbstractHttpSockJsSession.java:334(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.disconnect(AbstractHttpSockJsSession.java:330)
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.close(AbstractSockJsSession.java:216(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/WebSocketSessionDecorator.close(WebSocketSessionDecorator.java:158)
4XESTACKTRACE at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:192)
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:457(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))
3XMTHREADINFO "clientOutboundChannel-4" J9VMThread:0x00000000019CEE00, j9thread_t:0x00007F2EDEFDB380, java/lang/Thread:0x00000000E2E30C98, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x47, isDaemon:false)
3XMTHREADINFO1 (native thread ID:0xDF6D, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x01000101)
3XMTHREADINFO2 (native stack address range from:0x00007F2EDDDDD000, to:0x00007F2EDDE1E000, size:0x41000)
3XMCPUTIME CPU usage total: 7.811379586 secs, current category="Application"
3XMTHREADBLOCK Waiting on: com/ibm/ws/tcpchannel/internal/SimpleSync#0x00000000E30B31E8 Owned by: <unowned>
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:172(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcpchannel/internal/SimpleSync.simpleWait(SimpleSync.java:35(Compiled Code))
5XESTACKTRACE (entered lock: com/ibm/ws/tcpchannel/internal/SimpleSync#0x00000000E30B31E8, entry count: 1)
4XESTACKTRACE at com/ibm/ws/tcpchannel/internal/NioTCPWriteRequestContextImpl.processSyncWriteRequest(NioTCPWriteRequestContextImpl.java:77(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcpchannel/internal/TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:119(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.synchWrite(HttpServiceContextImpl.java:2438(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2174(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/inbound/HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:753(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flushBuffers(HttpOutputStreamImpl.java:481(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flush(HttpOutputStreamImpl.java:579(Compiled Code))
4XESTACKTRACE at com/ibm/wsspi/http/ee7/HttpOutputStreamEE7.flush(HttpOutputStreamEE7.java:293(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/osgi/response/WCOutputStream.flush(WCOutputStream.java:234(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer31/osgi/response/WCOutputStream31.flush(WCOutputStream31.java:599(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:705(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:670(Compiled Code))
4XESTACKTRACE at org/springframework/http/server/ServletServerHttpResponse.flush(ServletServerHttpResponse.java:96(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.writeFrameInternal(AbstractHttpSockJsSession.java:365(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.writeFrame(AbstractSockJsSession.java:322(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/StreamingSockJsSession.flushCache(StreamingSockJsSession.java:86(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.sendMessageInternal(AbstractHttpSockJsSession.java:302(Compiled Code))
5XESTACKTRACE (entered lock: java/lang/Object#0x00000000E30B30F8, entry count: 1)
4XESTACKTRACE at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.sendMessage(AbstractSockJsSession.java:166(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.tryFlushMessageBuffer(ConcurrentWebSocketSessionDecorator.java:132(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.sendMessage(ConcurrentWebSocketSessionDecorator.java:104(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:440(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))
WebSphere Application Server 16.0.0.3 (1.0.14.cl160320160831-1555) on IBM J9 VM, version pxa6480sr3fp11-20160817_02 (SR3 FP11) (en_US)
Spring-websocket 4.3.3.RELEASE
It looks like SRTServletResponse.flushBuffer implementation can lock threads infinitely,
SimpleSync.simpleWait never relese the thread. I tried to set tcpOptions inactivityTimeout, but it did't help.
WebSphere bug ?

We are having the same issue.
No workaround or fix yet.
Details:
Java 8.0.65
Windows Server 2012 R2 x64
Spring Boot 1.3.6.RELEASE
Spring Websocket 4.2.6.RELEASE
Undertow 1.3.22.Final
XNIO 3.3.6.Final

Related

Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response

I' trying to connect from raspberry 3 B to a Modbus device (sensor temperature) using a serial connection using a USB RS485 converter .This is my code:
from pymodbus.constants import Endian
from pymodbus.payload import BinaryPayloadDecoder
from pymodbus.client.sync import ModbusSerialClient as ModbusClient
from pymodbus.constants import Defaults
Defaults.RetryOnEmpty = True
import time
import logging
FORMAT = ('%(asctime)-15s %(threadName)-15s '
'%(levelname)-8s %(module)-15s:%(lineno)-8s %(message)s')
logging.basicConfig(format=FORMAT)
log = logging.getLogger()
log.setLevel(logging.DEBUG)
def run_sync_client():
while True:
#try:
client = ModbusClient(method='rtu', port='/dev/ttyUSB0', timeout=9, baudrate=9600, parity='N', stopbits=1, bytesize=8,reset_socket=False) # Implementation of the SHT31 as a modbus serial client
if client.connect():
print("connected")
print ("begin reading")
#for i in range(0 ,100):
request = client.read_holding_registers(address=0 , count=2, unit=4)
if request.isError():
print(request)
else:
print("Read OK! " + str(request.registers[1]))
time.sleep(1)
print(request)
result = request.registers
print(result)
temp= BinaryPayloadDecoder.fromRegisters(result, Endian.little, wordorder=Endian.little )
temperature= temp.decode_16bit_int()
print( temperature)
time.sleep(4)
# except:
#print("An exception occurred")
client.close()
while True:
if __name__ == "__main__":
run_sync_client()
The output in the console is:
connected
begin reading
2022-02-24 14:56:37,901 MainThread DEBUG transaction :139 Current transaction state - IDLE
2022-02-24 14:56:37,902 MainThread DEBUG transaction :144 Running transaction 1
2022-02-24 14:56:37,904 MainThread DEBUG transaction :273 SEND: 0x4 0x3 0x0 0x1 0x0 0x2 0x95 0x9e
2022-02-24 14:56:37,905 MainThread DEBUG sync :76 New Transaction state 'SENDING'
2022-02-24 14:56:37,906 MainThread DEBUG transaction :287 Changing transaction state from 'SENDING' to 'WAITING FOR REPLY'
2022-02-24 14:56:38,068 MainThread DEBUG transaction :375 Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
2022-02-24 14:56:38,069 MainThread DEBUG transaction :297 RECV: 0x4 0x3 0x2 0x17 0x58 0x2c 0x2e 0x3f 0x88
2022-02-24 14:56:38,071 MainThread DEBUG rtu_framer :237 Frame check failed, ignoring!!
2022-02-24 14:56:38,072 MainThread DEBUG rtu_framer :119 Resetting frame - Current Frame in buffer - 0x4 0x3 0x2 0x17 0x58 0x2c 0x2e 0x3f 0x88
2022-02-24 14:56:38,073 MainThread DEBUG transaction :465 Getting transaction 4
2022-02-24 14:56:38,074 MainThread DEBUG transaction :224 Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'
Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response
I would love to have a little help understanding the error I'm getting. Thanks

How to fix Weblogic 12.1.3 start up with Java 8?

I'm using Weblogic 12.1.3. It works fine with Java 7 but when I decided to use Java 8, Weblogic server doesn't reach running state. It hangs at the following line:
<Security> <WL-090082> <Security initializing using security realm myrealm.
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" #19 daemon prio=5 os_prio=0 tid=0x000000005b77d000 nid=0xdf8 runnable [0x0000000058f3b000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.isReachable0(Native Method)
at java.net.Inet6AddressImpl.isReachable(Inet6AddressImpl.java:77)
at java.net.InetAddress.isReachable(InetAddress.java:503)
at java.net.InetAddress.isReachable(InetAddress.java:462)
at weblogic.utils.net.AddressUtils$AddressMaker.getAllAddresses(AddressUtils.java:109)
at weblogic.utils.net.AddressUtils$AddressMaker.<clinit>(AddressUtils.java:48)
at weblogic.utils.net.AddressUtils.getLocalHost(AddressUtils.java:192)
at weblogic.cluster.ClusterHelper.getMachineName(ClusterHelper.java:44)
at weblogic.cluster.ClusterService.startService(ClusterService.java:116)
at weblogic.server.ServiceActivator.start(ServiceActivator.java:118)
at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d9fd32a0> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d9fd2b60> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d9fd2420> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d9f11020> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d9f108e0> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x0000000080a77fa0> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:753)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
-----------------------------------------------------------------------
"[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" #20 daemon prio=5 os_prio=0 tid=0x000000005bdb2800 nid=0x1884 in Object.wait() [0x000000005c72e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:186)
- locked <0x000000008000b848> (a org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000dab5ac08> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x0000000080a78360> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
-----------------------------------------------------------------------
"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" #24 daemon prio=5 os_prio=0 tid=0x000000005bdb5000 nid=0xc58 in Object.wait() [0x000000005ac1c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:186)
- locked <0x000000008000b848> (a org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d68cd6f8> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d68cca58> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d68cc318> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x00000000d68cb8d8> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
- locked <0x0000000080a795f8> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
Added stack trace for some of the threads running which could point to the problem.
It doesn't show any error but Weblogic server never goes to running state. I would appreciate your help on how to resolve this. Thanks.

Spark streaming receiver Out of Memory (OOM)

I have a problem where the receiver restarts again and again.
I am using Spark 1.6.1.
I use Spark Streaming to receive from a streaming, then use map to deserialize pb data.
My testing contains two cases:
Just receive the data and print directly: the app is stable
Receive and deserialize: this produces problems. The occurrence time is not regular.
There is 500Mb/min. I have set executor memory at 8GB. The problem is just like something allocate memory extremely. But, I don't know why.
My code:
val conf = new SparkConf().setAppName(args(8))
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.streaming.stopGracefullyOnShutdown", "true")
conf.set("spark.streaming.backpressure.enabled","true")
conf.set("spark.speculation","true")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc, Seconds(args(7).toInt))
val bigPipeStreams = (1 to args(3).toInt).map{
i => ssc.networkStream(
new MyBigpipeLogagentReceiver(args(0),args(1),args(2),i,args(4),args(5),args(6).toInt)
)
}
val lines = ssc.union(bigPipeStreams)
def deserializePbData(value: String) : String = {
if (null == value || value.isEmpty) {
return ""
}
var cuid = ""
var os = ""
var channel = ""
var sv = ""
var resid = ""
var appid = ""
var prod = ""
try { //if exception,useless data,just drop it
val timeStrIndex = value.indexOf(",\"time_str\"")
var strAfterTruncation = ""
if (-1 != timeStrIndex) {
strAfterTruncation = value.substring(0,timeStrIndex) + "}"
} else {
strAfterTruncation = value
}
val jsonData = JSONObject.fromObject(strAfterTruncation)
//val jsonData = value.getAsJsonArray()
val binBody = jsonData.getString("bin_body")
val pbData = binBody.substring(1,binBody.length()-1).split(",").foldLeft(ArrayBuffer.empty[Byte])((b,a) => b +java.lang.Byte.parseByte(a)).drop(8).toArray
Lighttpd.lighttpd_log.parseFrom(pbData).getRequest().getUrl().getUrlFields().getAutokvList().asScala.foreach(a =>
a.getKey() match {
case "cuid" => cuid += a.getValue()
case "os" => os += a.getValue()
case "channel" => channel += a.getValue()
case "sv" => sv += a.getValue()
case "resid" => resid += a.getValue()
case "appid" => appid += a.getValue()
case "prod" => prod += a.getValue()
case _ => null
}
)
val decodeCuid = URLDecoder.decode(cuid, "UTF-8")
os = os.toLowerCase()
if (os.matches("android(.*)")) {
os = "android"
} else if (os.matches("iphone(.*)")) {
os = "iphone"
} else if (os.matches("ipad(.*)")) {
os = "ipad"
} else if (os.matches("s60(.*)")) {
os = "symbian"
} else if (os.matches("wp7(.*)")) {
os = "wp7"
} else if (os.matches("wp(.*)")) {
os = "wp"
} else if (os.matches("tizen(.*)")) {
os = "tizen"
val ifHasLogid = Lighttpd.lighttpd_log.parseFrom(pbData).hasLogid()
val time = Lighttpd.lighttpd_log.parseFrom(pbData).getTime()
if (ifHasLogid) {
val logid = Lighttpd.lighttpd_log.parseFrom(pbData).getLogid()
if (logid.isEmpty || logid.toString().equals("-") || !resid.toString().equals("01") || channel.isEmpty |!appid.isEmpty || !prod.isEmpty) {
""
} else {
decodeCuid + "\001" + os + "\001" + channel + "\001" + sv + "\001" + "1" + "\001" + "1" + "\001" + time + "\n"
}
} else {
""
}
} catch {
case _:Throwable => ""
}
}
lines.map(parseData).print()
The error text:
016-07-12T12:00:01.546+0800: 5096.643: [GC (Allocation Failure)
Desired survivor size 442499072 bytes, new threshold 1 (max 15)
[PSYoungGen: 0K->0K(2356736K)] 5059009K->5059009K(7949312K), 0.0103342 secs] [Times: user=0.21 sys=0.00, real=0.01 secs]
2016-07-12T12:00:01.556+0800: 5096.654: [Full GC (Allocation Failure) [PSYoungGen: 0K->0K(2356736K)] [ParOldGen: 5059009K->5057376K(5592576K)] 5059009K->5057376K(7949312K), [Metaspace: 44836K->44490K(1089536K)], 0.8769617 secs] [Times: user=17.88 sys=0.04, real=0.88 secs]
2016-07-12T12:00:02.434+0800: 5097.531: Total time for which application threads were stopped: 1.2951974 seconds, Stopping threads took: 0.0000662 seconds
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid24310.hprof ...
2016-07-12T12:00:30.960+0800: 5126.057: Total time for which application threads were stopped: 28.5260812 seconds, Stopping threads took: 0.0000995 seconds
Heap dump file created [5211252802 bytes in 28.526 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill %p"
# Executing /bin/sh -c "kill 24310"...
2016-07-12T12:00:31.589+0800: 5126.686: Total time for which application threads were stopped: 0.6289627 seconds, Stopping threads took: 0.0001258 seconds
2016-07-12T12:00:31.595+0800: 5126.692: Total time for which application threads were stopped: 0.0004822 seconds, Stopping threads took: 0.0001493 seconds
2016-07-12 12:00:31.597 [Thread-5] ERROR [Logging.scala:95] - Uncaught exception in thread Thread[Thread-5,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_51]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_51]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_51]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_51]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_51]
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) ~[na:1.8.0_51]
at com.esotericsoftware.kryo.io.Output.flush(Output.java:155) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at com.esotericsoftware.kryo.io.Output.require(Output.java:135) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:153) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:146) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:194) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1196) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:858) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:77) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:157) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushArrayBuffer(ReceiverSupervisorImpl.scala:128) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl$$anon$3.onPushBlock(ReceiverSupervisorImpl.scala:109) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.BlockGenerator.pushBlock(BlockGenerator.scala:296) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks( BlockGenerator.scala:268) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:109) ~[ spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
2016-07-12 12:00:31.600 [SIGTERM handler] ERROR [SignalLogger.scala:57] - RECEIVED SIGNAL 15: SIGTERM
2016-07-12T12:00:31.611+0800: 5126.708: Total time for which application threads were stopped: 0.0005602 seconds, Stopping threads took: 0.0001765 seconds
2016-07-12T12:00:31.617+0800: 5126.714: Total time for which application threads were stopped: 0.0004800 seconds, Stopping threads took: 0.0001412 seconds
2016-07-12 12:00:32.483 [Bigpipe Receiver-SendThread(cq01-bigpipe-proxy01.cq01.baidu.com:2181)] WARN [ClientCnxnSocket.java:139] - Connected to an old server; r-o mode will be unavailable
2016-07-12T12:00:32.507+0800: 5127.604: Total time for which application threads were stopped: 0.0004604 seconds, Stopping threads took: 0.0001198 seconds
2016-07-12T12:00:32.509+0800: 5127.606: Total time for which application threads were stopped: 0.0002919 seconds, Stopping threads took: 0.0001800 seconds
2016-07-12T12:00:32.509+0800: 5127.607: Total time for which application threads were stopped: 0.0002692 seconds, Stopping threads took: 0.0001612 seconds
2016-07-12 12:00:32.549 [Bigpipe Receiver-SendThread(tc-bigpipe-proxy03.tc.baidu.com:2181)] WARN [ClientCnxnSocket.java:139] - Connected to an old server; r-o mode will be unavailable
2016-07-12T12:00:34.220+0800: 5129.317: [GC (Allocation Failure)
Desired survivor size 424148992 bytes, new threshold 2 (max 15)
[PSYoungGen: 1931776K->188775K(2363904K)] 6989152K->5246152K(7956480K), 0.2569385 secs] [Times: user=0.00 sys=5.19, real=0.26 secs]
2016-07-12T12:00:34.477+0800: 5129.575: Total time for which application threads were stopped: 0.2575019 seconds, Stopping threads took: 0.0000384 seconds
2016-07-12T12:00:35.478+0800: 5130.575: Total time for which application threads were stopped: 0.0002786 seconds, Stopping threads took: 0.0000424 seconds
2016-07-12T12:00:37.600+0800: 5132.697: [GC (Allocation Failure)
Desired survivor size 482344960 bytes, new threshold 3 (max 15)
[PSYoungGen: 2120551K->387013K(2268160K)] 7177928K->5444389K(7860736K), 0.5153031 secs] [Times: user=0.00 sys=9.89, real=0.52 secs]
2016-07-12T12:00:38.116+0800: 5133.213: Total time for which application threads were stopped: 0.5157529 seconds, Stopping threads took: 0.0000427 seconds
2016-07-12T12:00:40.116+0800: 5135.213: Total time for which application threads were stopped: 0.0003171 seconds, Stopping threads took: 0.0001000 seconds
2016-07-12T12:00:40.419+0800: 5135.516: [GC (Allocation Failure)
Desired survivor size 599785472 bytes, new threshold 2 (max 15)
[PSYoungGen: 2240965K->471033K(2324992K)] 7298341K->5633517K(7917568K), 0.3621433 secs] [Times: user=0.12 sys=7.11, real=0.36 secs]
2016-07-12T12:00:40.781+0800: 5135.878: Total time for which application threads were stopped: 0.3626080 seconds, Stopping threads took: 0.0000429 seconds
2016-07-12T12:00:41.781+0800: 5136.879: Total time for which application threads were stopped: 0.0003301 seconds, Stopping threads took: 0.0000947 seconds
2016-07-12T12:00:43.108+0800: 5138.205: [GC (Allocation Failure)
Desired survivor size 620756992 bytes, new threshold 3 (max 15)
[PSYoungGen: 2324985K->378481K(2054656K)] 7487469K->5831048K(7647232K), 0.2593685 secs] [Times: user=0.66 sys=4.96, real=0.26 secs]
2016-07-12T12:00:43.368+0800: 5138.465: [Full GC (Ergonomics) [PSYoungGen: 378481K->0K(2054656K)] [ParOldGen: 5452566K->4713601K(5592576K)] 5831048K->4713601K(7647232K), [Metaspace: 44635K->44635K(1089536K)], 4.3137405 secs] [Times: user=9.78 sys=74.53, real=4.31 secs]
2016-07-12T12:00:47.682+0800: 5142.779: Total time for which application threads were stopped: 4.5736603 seconds, Stopping threads took: 0.0000449 seconds
2016-07-12T12:00:47.682+0800: 5142.779: Total time for which application threads were stopped: 0.0002430 seconds, Stopping threads took: 0.0000856 seconds
2016-07-12T12:00:49.954+0800: 5145.052: [GC (Allocation Failure)
Desired survivor size 597688320 bytes, new threshold 4 (max 15)
[PSYoungGen: 1583616K->161266K(2189824K)] 6297217K->4874867K(7782400K), 0.0388138 secs] [Times: user=0.00 sys=0.84, real=0.04 secs]
2016-07-12T12:00:49.993+0800: 5145.091: Total time for which application threads were stopped: 0.0392926 seconds, Stopping threads took: 0.0000449 seconds
2016-07-12T12:00:51.903+0800: 5147.000: [GC (Allocation Failure)
Desired survivor size 596115456 bytes, new threshold 5 (max 15)
[PSYoungGen: 1744882K->324587K(2213888K)] 6458483K->5038189K(7806464K), 0.0334029 secs] [Times: user=0.69 sys=0.03, real=0.04 secs]
2016-07-12T12:00:51.936+0800: 5147.034: Total time for which application threads were stopped: 0.0338707 seconds, Stopping threads took: 0.0000404 seconds
2016-07-12T12:00:53.942+0800: 5149.039: [GC (Allocation Failure)
Desired survivor size 654835712 bytes, new threshold 6 (max 15)
[PSYoungGen: 1954795K->490438K(2120704K)] 6668397K->5204039K(7713280K), 0.0441762 secs] [Times: user=0.95 sys=0.02, real=0.05 secs]
2016-07-12T12:00:53.986+0800: 5149.083: Total time for which application threads were stopped: 0.0446174 seconds, Stopping threads took: 0.0000456 seconds
2016-07-12T12:00:56.102+0800: 5151.199: [GC (Allocation Failure)
Desired survivor size 763887616 bytes, new threshold 5 (max 15)
[PSYoungGen: 2120646K->639467K(1943552K)] 6834247K->5370280K(7536128K), 0.1124828 secs] [Times: user=1.07 sys=1.30, real=0.11 secs]
2016-07-12T12:00:56.214+0800: 5151.312: Total time for which application threads were stopped: 0.1129348 seconds, Stopping threads took: 0.0000396 seconds
2016-07-12T12:00:57.784+0800: 5152.881: [GC (Allocation Failure)
Desired survivor size 895483904 bytes, new threshold 4 (max 15)
[PSYoungGen: 1943531K->745977K(2050048K)] 6674344K->5504073K(7642624K), 0.0971717 secs] [Times: user=1.20 sys=0.67, real=0.10 secs]
2016-07-12T12:00:57.881+0800: 5152.979: Total time for which application threads were stopped: 0.0977363 seconds, Stopping threads took: 0.0000941 seconds
2016-07-12T12:00:59.406+0800: 5154.504: [GC (Allocation Failure)
Desired survivor size 935329792 bytes, new threshold 5 (max 15)
[PSYoungGen: 2050041K->599188K(1715200K)] 6808137K->5647517K(7307776K), 0.3651465 secs] [Times: user=0.98 sys=5.88, real=0.37 secs]
2016-07-12T12:00:59.772+0800: 5154.869: Total time for which application threads were stopped: 0.3656089 seconds, Stopping threads took: 0.0000479 seconds
2016-07-12T12:01:00.968+0800: 5156.066: [GC (Allocation Failure)
Desired survivor size 954204160 bytes, new threshold 4 (max 15)
[PSYoungGen: 1568404K->697830K(1667072K)] 6616733K->5746159K(7259648K), 0.0978955 secs] [Times: user=1.91 sys=0.04, real=0.09 secs]
2016-07-12T12:01:01.066+0800: 5156.164: Total time for which application threads were stopped: 0.0983759 seconds, Stopping threads took: 0.0000482 seconds
2016-07-12T12:01:02.189+0800: 5157.287: [GC (Allocation Failure)
Desired survivor size 954204160 bytes, new threshold 3 (max 15)
[PSYoungGen: 1667046K->465454K(1864192K)] 6715375K->5855655K(7456768K), 0.1261993 secs] [Times: user=2.41 sys=0.29, real=0.12 secs]
2016-07-12T12:01:02.316+0800: 5157.413: [Full GC (Ergonomics) [PSYoungGen: 465454K->65236K(1864192K)] [ParOldGen: 5390200K->5592328K(5592576K)] 5855655K->5657564K(7456768K), [Metaspace: 44635K->44635K(1089536K)], 3.2729437 secs] [Times: user=12.34 sys=57.11, real=3.28 secs]
2016-07-12T12:01:05.589+0800: 5160.686: Total time for which application threads were stopped: 3.3998619 seconds, Stopping threads took: 0.0000521 seconds
2016-07-12T12:01:05.589+0800: 5160.686: Total time for which application threads were stopped: 0.0002330 seconds, Stopping threads took: 0.0000949 seconds
2016-07-12T12:01:05.688+0800: 5160.785: Total time for which application threads were stopped: 0.0002935 seconds, Stopping threads took: 0.0000514 seconds
Heap
PSYoungGen total 1864192K, used 146620K [0x0000000715580000, 0x00000007c0000000, 0x00000007c0000000)
eden space 932352K, 8% used [0x0000000715580000,0x000000071a4fa138,0x000000074e400000)
from space 931840K, 7% used [0x0000000787200000,0x000000078b1b5290,0x00000007c0000000)
to space 931840K, 0% used [0x000000074e400000,0x000000074e400000,0x0000000787200000)
ParOldGen total 5592576K, used 5592328K [0x00000005c0000000, 0x0000000715580000, 0x0000000715580000)
object space 5592576K, 99% used [0x00000005c0000000,0x00000007155420a8,0x0000000715580000)
Metaspace used 44654K, capacity 44990K, committed 45864K, reserved 1089536K
class space used 6212K, capacity 6324K, committed 6440K, reserved 1048576K
New Error: I think it's the upload error which call oom problem.I'd like to know how to fix this upload error?
2016-07-15 11:41:47.307 [shuffle-client-0] ERROR [TransportChannelHandler.java:128] - Connection to nmg01-taihang-d10207.nmg01.baidu.com/10.76.48.22:30456 has been quiet for 120000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong.
2016-07-15 11:41:47.309 [shuffle-client-0] ERROR [TransportResponseHandler.java:122] - Still have 1 requests outstanding when connection from nmg01-taihang-d10207.nmg01.baidu.com/10.76.48.22:30456 is closed
2016-07-15 11:41:47.314 [shuffle-client-0] ERROR [Logging.scala:95] - Error while uploading block input-0-1468553896200
java.io.IOException: Connection from nmg01-taihang-d10207.nmg01.baidu.com/10.76.48.22:30456 closed
at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2016-07-15T11:41:47.316+0800: 2176.487: Total time for which application threads were stopped: 0.0002632 seconds, Stopping threads took: 0.0000521 seconds
2016-07-15 11:41:47.316 [Thread-5] WARN [Logging.scala:91] - Failed to replicate input-0-1468553896200 to BlockManagerId(2, nmg01-taihang-d10207.nmg01.baidu.com, 30456), failure #0
java.io.IOException: Connection from nmg01-taihang-d10207.nmg01.baidu.com/10.76.48.22:30456 closed
at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ~[spark-assembly-1.6.1.0-baidu-SNAPSHOT-hadoop2.5.1.3-baidu-SNAPSHOT.jar:1.6.1.0-baidu-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
2016-07-15T11:41:48.316+0800: 2177.487: Total time for which application threads were stopped: 0.0003391 seconds, Stopping threads took: 0.0000979 seconds
2016-07-15T11:41:51.312+0800: 2180.483: [GC (Allocation Failure) --[PSYoungGen: 2894863K->2894863K(3007488K)] 8299519K->9550273K(9998336K), 0.7462118 secs] [Times: user=9.78 sys=0.02, real=0.74 secs]
2016-07-15T11:41:52.059+0800: 2181.230: [Full GC (Ergonomics) [PSYoungGen: 2894863K->0K(3007488K)] [ParOldGen: 6655410K->6895736K(6990848K)] 9550273K->6895736K(9998336K), [Metaspace: 44409K->44409K(1087488K)], 0.4061892 secs] [Times: user=7.50 sys=0.01, real=0.41 secs]
Your code appears to have an error in structure. In the process of looking at your code (to re-indenting it to reflect the structure as posted), I found that your last else if statement:
} else if (os.matches("tizen(.*)")) {
os = "tizen"
opens a block, but does not close the block where it "should". Instead, the block is actually terminated with:
} catch {
The full code as it appears was intended (and re-indented) is:
val conf = new SparkConf().setAppName(args(8))
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.streaming.stopGracefullyOnShutdown", "true")
conf.set("spark.streaming.backpressure.enabled","true")
conf.set("spark.speculation","true")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc, Seconds(args(7).toInt))
val bigPipeStreams = (1 to args(3).toInt).map{
i => ssc.networkStream(
new MyBigpipeLogagentReceiver(args(0),args(1),args(2),i,args(4),args(5),args(6).toInt)
)
}
val lines = ssc.union(bigPipeStreams)
def deserializePbData(value: String) : String = {
if (null == value || value.isEmpty) {
return ""
}
var cuid = ""
var os = ""
var channel = ""
var sv = ""
var resid = ""
var appid = ""
var prod = ""
try { //if exception,useless data,just drop it
val timeStrIndex = value.indexOf(",\"time_str\"")
var strAfterTruncation = ""
if (-1 != timeStrIndex) {
strAfterTruncation = value.substring(0,timeStrIndex) + "}"
} else {
strAfterTruncation = value
}
val jsonData = JSONObject.fromObject(strAfterTruncation)
//val jsonData = value.getAsJsonArray()
val binBody = jsonData.getString("bin_body")
val pbData = binBody.substring(1,binBody.length()-1).split(",").foldLeft(ArrayBuffer.empty[Byte])((b,a) => b +java.lang.Byte.parseByte(a)).drop(8).toArray
Lighttpd.lighttpd_log.parseFrom(pbData).getRequest().getUrl().getUrlFields().getAutokvList().asScala.foreach(a =>
a.getKey() match {
case "cuid" => cuid += a.getValue()
case "os" => os += a.getValue()
case "channel" => channel += a.getValue()
case "sv" => sv += a.getValue()
case "resid" => resid += a.getValue()
case "appid" => appid += a.getValue()
case "prod" => prod += a.getValue()
case _ => null
}
)
val decodeCuid = URLDecoder.decode(cuid, "UTF-8")
os = os.toLowerCase()
if (os.matches("android(.*)")) {
os = "android"
} else if (os.matches("iphone(.*)")) {
os = "iphone"
} else if (os.matches("ipad(.*)")) {
os = "ipad"
} else if (os.matches("s60(.*)")) {
os = "symbian"
} else if (os.matches("wp7(.*)")) {
os = "wp7"
} else if (os.matches("wp(.*)")) {
os = "wp"
} else if (os.matches("tizen(.*)")) {
os = "tizen"
}
val ifHasLogid = Lighttpd.lighttpd_log.parseFrom(pbData).hasLogid()
val time = Lighttpd.lighttpd_log.parseFrom(pbData).getTime()
if (ifHasLogid) {
val logid = Lighttpd.lighttpd_log.parseFrom(pbData).getLogid()
if (logid.isEmpty || logid.toString().equals("-") || !resid.toString().equals("01") || channel.isEmpty |!appid.isEmpty || !prod.isEmpty) {
""
} else {
decodeCuid + "\001" + os + "\001" + channel + "\001" + sv + "\001" + "1" + "\001" + "1" + "\001" + time + "\n"
}
} else {
""
}
} catch {
case _:Throwable => ""
}
}
lines.map(parseData).print()
I have not checked your code for functionality. This is just a syntax/structure issue that stood out when very briefly looking at your the code you posted.

add_timer causes kernel stack dump for multiple PCI boards

We are using FPGA cards with PCI express drivers to move data around with DMA engines. This all works fine for a single card in a machine, however with two cards it fails. As an initial investigation, I have narrowed an error down to the add_timer function that is used to set up the polling mechanism. When insmod adds the driver modules, a stack trace is produced as the poll_timer routine is the same for both instances. The code has been reduced to
static int dat_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
{
struct timer_list * timer = &poll_timer;
int i;
/* Start polling routine */
log_normal(KERN_INFO "DEBUG ADD TIMER: Starting poll routine with %x\n", pdev);
init_timer(timer);
// random number added so that expires value is different for both instances of timer
get_random_bytes(&i, 1);
timer->expires=jiffies+HZ+i;
timer->data=(unsigned long) pdev;
timer->function = poll_routine;
log_verbose("DEBUG ADD TIMER: Timer expires %x\n", timer->expires);
log_verbose("DEBUG ADD TIMER: Timer data %x\n", timer->data);
log_verbose("DEBUG ADD TIMER: Timer function %x\n", timer->function);
// ***** THIS IS WHERE STACK TRACE OCCURS (WHEN CALLED FOR SECOND TIME)
add_timer(timer);
log_verbose("DEBUG ADD TIMER: Value of HZ is %d\n", HZ);
log_verbose("DEBUG ADD TIMER: End of probe\n");
return 0;
}
the stack trace produces
list_add corruption. prev->next should be next (ffffffff81f76228), but was (null). (prev=ffffffffa050a3c0).
and
list_add double add: new=ffffffffa050a3c0, prev=ffffffffa050a3c0, next=ffffffff81f76228.
Looking at the printk statements, it is clear that the add_timer is trying to add the same routine to the linked list. Is this correct?
DEBUG ADD TIMER: Timer expires fffd9cd3
DEBUG ADD TIMER: Timer data 6c0ac000
DEBUG ADD TIMER: Timer function **a0508150**
DEBUG ADD TIMER: Value of HZ is 1000
DEBUG ADD TIMER: End of probe
DEBUG ADD TIMER: Starting poll routine with 6c0ad000
DEBUG ADD TIMER: Timer expires fffd9c7d
DEBUG ADD TIMER: Timer data 6c0ad000
DEBUG ADD TIMER: Timer function **a0508150**
So my question(s) is(are), how should I configure the timer for multiple instantations of the same driver? (Assuming that is what is happening when multiple boards are inserted into the machine).
full stack trace
DEBUG ADD TIMER: Inserting driver into kernel.
DEBUG ADD TIMER: Starting poll routine with 6c0ac000
DEBUG ADD TIMER: Timer expires fffd9cd3
DEBUG ADD TIMER: Timer data 6c0ac000
DEBUG ADD TIMER: Timer function a0508150
DEBUG ADD TIMER: Value of HZ is 1000
DEBUG ADD TIMER: End of probe
DEBUG ADD TIMER: Starting poll routine with 6c0ad000
DEBUG ADD TIMER: Timer expires fffd9c7d
DEBUG ADD TIMER: Timer data 6c0ad000
DEBUG ADD TIMER: Timer function a0508150
------------[ cut here ]------------
WARNING: CPU: 0 PID: 2201 at lib/list_debug.c:33 __list_add+0xa0/0xd0()
list_add corruption. prev->next should be next (ffffffff81f76228), but was (null). (prev=ffffffffa050a3c0).
Modules linked in: xdma_v7(POE+) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller crc32c_intel eeepc_wmi ghash_clmulni_intel asus_wmi ftdi_sio iTCO_wdt snd_hda_codec sparse_keymap raid0 iTCO_vendor_support
snd_hda_core rfkill sb_edac ipmi_ssif video mxm_wmi edac_core snd_hwdep mei_me snd_seq snd_seq_device ipmi_msghandler snd_pcm mei acpi_pad tpm_infineon lpc_ich mfd_core snd_timer tpm_tis shpchp tpm snd soundcore i2c_i801 wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ast drm_kms_helper ttm drm igb serio_raw ptp pps_core dca i2c_algo_bit
CPU: 0 PID: 2201 Comm: insmod Tainted: P OE 4.1.8-100.fc21.x86_64 #1
Hardware name: ASUSTeK COMPUTER INC. Z10PE-D8 WS/Z10PE-D8 WS, BIOS 1001 03/17/2015
0000000000000000 00000000ec73155d ffff880457123928 ffffffff81792065
0000000000000000 ffff880457123980 ffff880457123968 ffffffff810a163a
0000000000000246 ffffffffa050a3c0 ffffffff81f76228 ffffffffa050a3c0
Call Trace:
[<ffffffff81792065>] dump_stack+0x45/0x57
[<ffffffff810a163a>] warn_slowpath_common+0x8a/0xc0
[<ffffffff810a16c5>] warn_slowpath_fmt+0x55/0x70
[<ffffffff810f8250>] ? vprintk_emit+0x3b0/0x560
[<ffffffff813c7c30>] __list_add+0xa0/0xd0
[<ffffffff81108412>] __internal_add_timer+0xb2/0x130
[<ffffffff811084bf>] internal_add_timer+0x2f/0xb0
[<ffffffff8110a1ca>] mod_timer+0x12a/0x210
[<ffffffff8110a2c8>] add_timer+0x18/0x30
[<ffffffffa050810f>] dat_probe+0xbf/0x100 [xdma_v7]
[<ffffffff813f6da5>] local_pci_probe+0x45/0xa0
[<ffffffff812a8da2>] ? sysfs_do_create_link_sd.isra.2+0x72/0xc0
[<ffffffff813f8109>] pci_device_probe+0xf9/0x150
[<ffffffff814e7e59>] driver_probe_device+0x209/0x4b0
[<ffffffff814e81db>] __driver_attach+0x9b/0xa0
[<ffffffff814e8140>] ? __device_attach+0x40/0x40
[<ffffffff814e5973>] bus_for_each_dev+0x73/0xc0
[<ffffffff814e772e>] driver_attach+0x1e/0x20
[<ffffffff814e72e0>] bus_add_driver+0x180/0x250
[<ffffffffa000a000>] ? 0xffffffffa000a000
[<ffffffff814e89d4>] driver_register+0x64/0xf0
[<ffffffff813f662c>] __pci_register_driver+0x4c/0x50
[<ffffffffa000a02c>] dat_init+0x2c/0x1000 [xdma_v7]
[<ffffffff81002148>] do_one_initcall+0xd8/0x210
[<ffffffff812094f9>] ? kmem_cache_alloc_trace+0x1a9/0x230
[<ffffffff817911bc>] ? do_init_module+0x28/0x1cc
[<ffffffff817911f5>] do_init_module+0x61/0x1cc
[<ffffffff811270bb>] load_module+0x20db/0x2550
[<ffffffff81122990>] ? store_uevent+0x70/0x70
[<ffffffff8122e860>] ? kernel_read+0x50/0x80
[<ffffffff81127766>] SyS_finit_module+0xa6/0xe0
[<ffffffff8179892e>] system_call_fastpath+0x12/0x71
---[ end trace 340e5d7ba2d89081 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 2201 at lib/list_debug.c:36 __list_add+0xcb/0xd0()
list_add double add: new=ffffffffa050a3c0, prev=ffffffffa050a3c0, next=ffffffff81f76228.
Modules linked in: xdma_v7(POE+) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller crc32c_intel eeepc_wmi ghash_clmulni_intel asus_wmi ftdi_sio iTCO_wdt snd_hda_codec sparse_keymap raid0 iTCO_vendor_support
snd_hda_core rfkill sb_edac ipmi_ssif video mxm_wmi edac_core snd_hwdep mei_me snd_seq snd_seq_device ipmi_msghandler snd_pcm mei acpi_pad tpm_infineon lpc_ich mfd_core snd_timer tpm_tis shpchp tpm snd soundcore i2c_i801 wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ast drm_kms_helper ttm drm igb serio_raw ptp pps_core dca i2c_algo_bit
CPU: 0 PID: 2201 Comm: insmod Tainted: P W OE 4.1.8-100.fc21.x86_64 #1
Hardware name: ASUSTeK COMPUTER INC. Z10PE-D8 WS/Z10PE-D8 WS, BIOS 1001 03/17/2015
0000000000000000 00000000ec73155d ffff880457123928 ffffffff81792065
0000000000000000 ffff880457123980 ffff880457123968 ffffffff810a163a
0000000000000246 ffffffffa050a3c0 ffffffff81f76228 ffffffffa050a3c0
Call Trace:
[<ffffffff81792065>] dump_stack+0x45/0x57
[<ffffffff810a163a>] warn_slowpath_common+0x8a/0xc0
[<ffffffff810a16c5>] warn_slowpath_fmt+0x55/0x70
[<ffffffff810f8250>] ? vprintk_emit+0x3b0/0x560
[<ffffffff813c7c5b>] __list_add+0xcb/0xd0
[<ffffffff81108412>] __internal_add_timer+0xb2/0x130
[<ffffffff811084bf>] internal_add_timer+0x2f/0xb0
[<ffffffff8110a1ca>] mod_timer+0x12a/0x210
[<ffffffff8110a2c8>] add_timer+0x18/0x30
[<ffffffffa050810f>] dat_probe+0xbf/0x100 [xdma_v7]
[<ffffffff813f6da5>] local_pci_probe+0x45/0xa0
[<ffffffff812a8da2>] ? sysfs_do_create_link_sd.isra.2+0x72/0xc0
[<ffffffff813f8109>] pci_device_probe+0xf9/0x150
[<ffffffff814e7e59>] driver_probe_device+0x209/0x4b0
[<ffffffff814e81db>] __driver_attach+0x9b/0xa0
[<ffffffff814e8140>] ? __device_attach+0x40/0x40
[<ffffffff814e5973>] bus_for_each_dev+0x73/0xc0
[<ffffffff814e772e>] driver_attach+0x1e/0x20
[<ffffffff814e72e0>] bus_add_driver+0x180/0x250
[<ffffffffa000a000>] ? 0xffffffffa000a000
[<ffffffff814e89d4>] driver_register+0x64/0xf0
[<ffffffff813f662c>] __pci_register_driver+0x4c/0x50
[<ffffffffa000a02c>] dat_init+0x2c/0x1000 [xdma_v7]
[<ffffffff81002148>] do_one_initcall+0xd8/0x210
[<ffffffff812094f9>] ? kmem_cache_alloc_trace+0x1a9/0x230
[<ffffffff817911bc>] ? do_init_module+0x28/0x1cc
[<ffffffff817911f5>] do_init_module+0x61/0x1cc
[<ffffffff811270bb>] load_module+0x20db/0x2550
[<ffffffff81122990>] ? store_uevent+0x70/0x70
[<ffffffff8122e860>] ? kernel_read+0x50/0x80
[<ffffffff81127766>] SyS_finit_module+0xa6/0xe0
[<ffffffff8179892e>] system_call_fastpath+0x12/0x71
---[ end trace 340e5d7ba2d89082 ]---
DEBUG ADD TIMER: Value of HZ is 1000
DEBUG ADD TIMER: End of probe
The problem is that the second call to dat_probe is clobbering the poll_timer variable that was initialized and queued by the first call to dat_probe. You are clobbering the pointers in the kernel's timer list.
You need to get rid of the poll_timer variable and give each device its own dynamically allocated private data structure containing its own struct timer_list member. Call pci_set_drvdata to set the private data pointer for the PCI device. The other PCI driver functions can call pci_get_drvdata to retrieve that pointer.

Devise Not Woking on Heroku Server

I am developing a site with devise gem. Locally it's work but on heroku it is not working. Devise also send confirmation email locally but show nothing on heroku.
http://incomon.herokuapp.com/
Heroku Logs
2014-02-25T06:05:47.922900+00:00 app[web.1]: [deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
2014-02-25T06:05:47.934244+00:00 heroku[router]: at=info method=POST path=/users/sign_up host=incomon.herokuapp.com request_id=140cdcbc-c6be-4dbb-bcaf-4ab77f7a76e0 fwd="182.185.250.158" dyno=web.1 connect=2ms service=76ms status=200 bytes=3573
2014-02-25T06:24:01.826741+00:00 heroku[router]: at=info method=GET path=/ host=incomon.herokuapp.com request_id=8c3a9cc2-26c2-441f-b079-601d53025787 fwd="182.185.250.158" dyno=web.1 connect=88ms service=61ms status=304 bytes=1022
2014-02-25T06:24:03.441070+00:00 heroku[router]: at=info method=POST path=/users/sign_up host=incomon.herokuapp.com request_id=52e81d7a-cd33-4bf8-9a7c-c85ef57f9ca6 fwd="182.185.250.158" dyno=web.1 connect=1ms service=15ms status=200 bytes=3573
2014-02-25T06:24:15.053833+00:00 app[web.1]: FATAL: terminating connection due to administrator command
2014-02-25T06:27:07.154449+00:00 heroku[router]: at=info method=GET path=/ host=incomon.herokuapp.com request_id=377101eb-2804-46d3-a41d-d0f65fd07726 fwd="182.185.250.158" dyno=web.1 connect=0ms service=67ms status=200 bytes=1636
2014-02-25T06:27:10.767345+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=incomon.herokuapp.com request_id=8681650f-b5ea-438f-b16a-ccf863e49eab fwd="182.185.250.158" dyno=web.1 connect=1ms service=6ms status=200 bytes=228
2014-02-25T06:27:26.227841+00:00 heroku[router]: at=info method=POST path=/users host=incomon.herokuapp.com request_id=18c1662e-070f-4cc4-b800-347d2ba8663a fwd="182.185.250.158" dyno=web.1 connect=1ms service=136ms status=500 bytes=1543
2014-02-25T06:27:26.692345+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=incomon.herokuapp.com request_id=a5491fe3-e684-4b3e-b094-7ca5f096866f fwd="182.185.250.158" dyno=web.1 connect=2ms service=7ms status=304 bytes=133
2014-02-25T06:27:35.903201+00:00 heroku[router]: at=info method=GET path=/ host=incomon.herokuapp.com request_id=b826446e-15d7-487f-a93f-86a667a034bb fwd="182.185.250.158" dyno=web.1 connect=1ms service=10ms status=304 bytes=848
2014-02-25T06:27:36.553396+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=incomon.herokuapp.com request_id=03a94dbd-2bd0-417a-8333-67a4996605f6 fwd="182.185.250.158" dyno=web.1 connect=0ms service=8ms status=304 bytes=133
2014-02-25T06:27:39.361486+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=incomon.herokuapp.com request_id=d0eae806-b61b-4f84-af8e-d0fc73b30c64 fwd="182.185.250.158" dyno=web.1 connect=0ms service=7ms status=304 bytes=133
2014-02-25T06:27:40.517027+00:00 heroku[router]: at=info method=POST path=/users/sign_in host=incomon.herokuapp.com request_id=45769d05-2579-4ada-ae77-330e68f14ea7 fwd="182.185.250.158" dyno=web.1 connect=1ms service=23ms status=200 bytes=2710
2014-02-25T06:27:40.907652+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=incomon.herokuapp.com request_id=70b3f49a-fa9c-4281-8515-9399636277d3 fwd="182.185.250.158" dyno=web.1 connect=2ms service=11ms status=304 bytes=133
2014-02-25T06:27:46.970682+00:00 app[web.1]: I, [2014-02-25T06:05:47.869467 #2] INFO -- : Processing by User::RegistrationsController#new as HTML
2014-02-25T06:27:46.970682+00:00
app[web.1]: => Ctrl-C to shutdown server
2014-02-25T06:27:46.970682+00:00 app[web.1]: I, [2014-02-25T06:04:46.336464 #2] INFO -- : Started GET "/" for 182.185.250.158 at 2014-02-25 06:04:46 +0000
2014-02-25T06:27:46.970682+00:00 app[web.1]: I, [2014-02-25T06:04:46.444481 #2] INFO -- : Processing by HomeController#index as HTML
2014-02-25T06:27:46.970682+00:00 app[web.1]: I, [2014-02-25T06:04:46.506698 #2] INFO -- : Rendered home/index.html.haml within layouts/application (23.4ms)
2014-02-25T06:27:46.970682+00:00 app[web.1]: I, [2014-02-25T06:04:46.508911 #2] INFO -- : Completed 200 OK in 64ms (Views: 30.2ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:58.328060 #2] INFO -- : Processing by User::RegistrationsController#create as HTML
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:59.225468 #2] INFO -- : Completed 500 Internal Server Error in 897ms
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:47.932458 #2] INFO -- : Completed 200 OK in 63ms (Views: 44.8ms | ActiveRecord: 12.0ms)
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:59.223051 #2] INFO -- : Rendered vendor/bundle/ruby/2.0.0/gems/devise-3.2.2/app/views/devise/mailer/confirmation_instructions.html.erb (48.4ms)
2014-02-25T06:27:46.970962+00:00 app[web.1]: F, [2014-02-25T06:05:59.228397 #2] FATAL -- :
2014-02-25T06:27:46.970962+00:00 app[web.1]: ActionView::Template::Error (Missing host to link to! Please provide the :host parameter, set default_url_options[:host], or set :only_path to true):
2014-02-25T06:27:46.971156+00:00 app[web.1]: 5: <p><%= link_to 'Confirm my account', confirmation_url(#resource, :confirmation_token => #token) %></p>
2014-02-25T06:27:46.971156+00:00 app[web.1]: 2:
2014-02-25T06:27:46.971156+00:00 app[web.1]: 4:
2014-02-25T06:27:46.971156+00:00 app[web.1]:
2014-02-25T06:27:46.971156+00:00 app[web.1]: 3: <p>You can confirm your account email through the link below:</p>
2014-02-25T06:27:46.971156+00:00 app[web.1]: app/controllers/user/registrations_controller.rb:16:in `create'
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:03.428963 #2] INFO -- : Started GET "/users/sign_up" for 182.185.250.158 at 2014-02-25 06:24:03 +0000
2014-02-25T06:27:46.971156+00:00 app[web.1]: I, [2014-02-25T06:24:01.789030 #2] INFO -- : Rendered home/index.html.haml within layouts/application (1.0ms)
2014-02-25T06:27:46.971156+00:00 app[web.1]: I, [2014-02-25T06:24:01.784470 #2] INFO -- : Started GET "/" for 182.185.250.158 at 2014-02-25 06:24:01 +0000
2014-02-25T06:27:46.971156+00:00 app[web.1]:
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:03.431179 #2] INFO -- : Processing by User::RegistrationsController#new as HTML
2014-02-25T06:27:46.971156+00:00 app[web.1]: I, [2014-02-25T06:24:01.786775 #2] INFO -- : Processing by HomeController#index as HTML
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:14.949567 #2] INFO -- : Processing by User::RegistrationsController#create as HTML
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:03.437029 #2] INFO -- : Rendered user/registrations/new.html.haml within layouts/application (3.3ms)
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:01.790129 #2] INFO -- : Completed 200 OK in 3ms (Views: 2.5ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:03.431343 #2] INFO -- : Parameters: {"authenticity_token"=>"AkkP/F7SZjNfBGPERz80uQLTrnw3gi8P8/gfa6OIOL0="}
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:14.949696 #2] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"AkkP/F7SZjNfBGPERz80uQLTrnw3gi8P8/gfa6OIOL0=", "user"=>{"firstName"=>"Tayyab", "lastName"=>"Zahid", "email"=>"tayyabzahid96#gmail.com", "password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "userName"=>"tayyab"}, "commit"=>"Sign Up"}
2014-02-25T06:27:46.972148+00:00 app[web.1]: I, [2014-02-25T06:24:15.054836 #2] INFO -- : Completed 500 Internal Server Error in 105ms
2014-02-25T06:27:46.972148+00:00 app[web.1]: : BEGIN
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:03.437836 #2] INFO -- : Completed 200 OK in 6ms (Views: 4.5ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.971915+00:00 app[web.1]: I, [2014-02-25T06:24:14.947905 #2] INFO -- : Started POST "/users" for 182.185.250.158 at 2014-02-25 06:24:14 +0000
2014-02-25T06:27:46.971915+00:00 app[web.1]: E, [2014-02-25T06:24:15.054427 #2] ERROR -- : PG::ConnectionBad: PQconsumeInput() SSL connection has been closed unexpectedly
2014-02-25T06:27:46.972148+00:00 app[web.1]: F, [2014-02-25T06:24:15.056230 #2] FATAL -- :
2014-02-25T06:27:46.972148+00:00 app[web.1]: I, [2014-02-25T06:27:07.092850 #2] INFO -- : Started GET "/" for 182.185.250.158 at 2014-02-25 06:27:07 +0000
2014-02-25T06:27:46.972148+00:00 app[web.1]: app/controllers/user/registrations_controller.rb:16:in `create'
2014-02-25T06:27:46.972148+00:00 app[web.1]: I, [2014-02-25T06:27:07.137126 #2] INFO -- : Processing by HomeController#index as HTML
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:07.141871 #2] INFO -- : Rendered home/index.html.haml within layouts/application (1.5ms)
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:07.153884 #2] INFO -- : Completed 200 OK in 16ms (Views: 14.2ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:12.070759 #2] INFO -- : Started GET "/users/sign_up" for 182.185.250.158 at 2014-02-25 06:27:12 +0000
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:12.080240 #2] INFO -- : Rendered user/registrations/new.html.haml within layouts/application (4.0ms)
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:12.073526 #2] INFO -- : Processing by User::RegistrationsController#new as HTML
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:12.073625 #2] INFO -- : Parameters: {"authenticity_token"=>"w+7mt9jtpi52dlmiEQXc996D+gYN9r/amyAP3RahSuM="}
2014-02-25T06:27:46.972811+00:00 app[web.1]: E, [2014-02-25T06:27:26.203898 #2] ERROR -- : PG::UndefinedTable: ERROR: relation "users" does not exist
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:12.081285 #2] INFO -- : Completed 200 OK in 8ms (Views: 5.5ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:26.097156 #2] INFO -- : Started POST "/users" for 182.185.250.158 at 2014-02-25 06:27:26 +0000
2014-02-25T06:27:46.972811+00:00 app[web.1]: ^
2014-02-25T06:27:46.972811+00:00 app[web.1]: LINE 1: SELECT 1 AS one FROM "users" WHERE "users"."email" = 'tayy...
2014-02-25T06:27:46.972811+00:00 app[web.1]: LINE 1: SELECT 1 AS one FROM "users" WHERE "users"."email" = 'tayy...
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:26.098986 #2] INFO -- : Processing by User::RegistrationsController#create as HTML
2014-02-25T06:27:46.972620+00:00 app[web.1]: I, [2014-02-25T06:27:26.099114 #2] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"w+7mt9jtpi52dlmiEQXc996D+gYN9r/amyAP3RahSuM=", "user"=>{"firstName"=>"Tayyab", "lastName"=>"Zahid", "email"=>"tayyabz#stis.pk", "password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "userName"=>"tayyab"}, "commit"=>"Sign Up"}
2014-02-25T06:27:46.972811+00:00 app[web.1]: I, [2014-02-25T06:27:26.225627 #2] INFO -- : Completed 500 Internal Server Error in 126ms
2014-02-25T06:27:46.973311+00:00 app[web.1]:
2014-02-25T06:27:46.973311+00:00 app[web.1]: app/controllers/user/registrations_controller.rb:16:in `create'
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:35.897992 #2] INFO -- : Processing by HomeController#index as HTML
2014-02-25T06:27:46.972811+00:00 app[web.1]: : SELECT 1 AS one FROM "users" WHERE "users"."email" = 'tayyabz#stis.pk' LIMIT 1):
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:35.896428 #2] INFO -- : Started GET "/" for 182.185.250.158 at 2014-02-25 06:27:35 +0000
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:35.901762 #2] INFO -- : Completed 200 OK in 4ms (Views: 2.7ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.972811+00:00 app[web.1]: : SELECT 1 AS one FROM "users" WHERE "users"."email" = 'tayyabz#stis.pk' LIMIT 1
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:40.499174 #2] INFO -- : Started GET "/users/sign_in" for 182.185.250.158 at 2014-02-25 06:27:40 +0000
2014-02-25T06:27:46.973493+00:00 app[web.1]: I, [2014-02-25T06:27:40.516752 #2] INFO -- : Completed 200 OK in 14ms (Views: 11.5ms | ActiveRecord: 0.0ms)
2014-02-25T06:27:46.972811+00:00 app[web.1]: F, [2014-02-25T06:27:26.227535 #2] FATAL -- :
2014-02-25T06:27:46.972811+00:00 app[web.1]: ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "users" does not exist
2014-02-25T06:27:46.973493+00:00 app[web.1]: I, [2014-02-25T06:27:46.964988 #2] INFO -- : Started POST "/users/sign_in" for 182.185.250.158 at 2014-02-25 06:27:46 +0000
2014-02-25T06:27:46.973311+00:00 app[web.1]:
2014-02-25T06:27:46.973493+00:00 app[web.1]: I, [2014-02-25T06:27:46.969713 #2] INFO -- : Processing by User::SessionsController#create as HTML
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:47.869564 #2] INFO -- : Parameters: {"authenticity_token"=>"AkkP/F7SZjNfBGPERz80uQLTrnw3gi8P8/gfa6OIOL0="}
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:58.324570 #2] INFO -- : Started POST "/users" for 182.185.250.158 at 2014-02-25 06:05:58 +0000
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:58.328309 #2] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"AkkP/F7SZjNfBGPERz80uQLTrnw3gi8P8/gfa6OIOL0=", "user"=>{"firstName"=>"Tayyab", "lastName"=>"Zahid", "email"=>"tayyabzahid96#gmail.com", "password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "userName"=>"tayyab"}, "commit"=>"Sign Up"}
2014-02-25T06:27:46.972148+00:00 app[web.1]: : BEGIN):
2014-02-25T06:27:46.972148+00:00 app[web.1]:
2014-02-25T06:27:46.972148+00:00 app[web.1]:
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:35.900793 #2] INFO -- : Rendered home/index.html.haml within layouts/application (1.2ms)
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:40.502401 #2] INFO -- : Processing by User::SessionsController#new as HTML
2014-02-25T06:27:46.973493+00:00 app[web.1]: I, [2014-02-25T06:27:46.969849 #2] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"w+7mt9jtpi52dlmiEQXc996D+gYN9r/amyAP3RahSuM=", "user"=>{"email"=>"tayyabz#stis.pk", "password"=>"[FILTERED]", "remember_me"=>"0"}, "commit"=>"Sign in"}
2014-02-25T06:27:46.973311+00:00 app[web.1]: I, [2014-02-25T06:27:40.502524 #2] INFO -- : Parameters: {"authenticity_token"=>"w+7mt9jtpi52dlmiEQXc996D+gYN9r/amyAP3RahSuM="}
2014-02-25T06:27:46.970682+00:00 app[web.1]: => Booting WEBrick
2014-02-25T06:27:46.970682+00:00 app[web.1]: => Rails 4.0.2 application starting in production on http://0.0.0.0:22113
2014-02-25T06:27:46.970682+00:00 app[web.1]: => Run `rails server -h` for more startup options
2014-02-25T06:27:46.970682+00:00 app[web.1]: I, [2014-02-25T06:05:47.866396 #2] INFO -- : Started GET "/users/sign_up" for 182.185.250.158 at 2014-02-25 06:05:47 +0000
2014-02-25T06:27:46.970962+00:00 app[web.1]: I, [2014-02-25T06:05:47.931496 #2] INFO -- : Rendered user/registrations/new.html.haml within layouts/application (40.6ms)
2014-02-25T06:27:46.972148+00:00 app[web.1]: ActiveRecord::StatementInvalid (PG::ConnectionBad: PQconsumeInput() SSL connection has been closed unexpectedly
2014-02-25T06:27:46.972811+00:00 app[web.1]: ^
2014-02-25T06:27:46.973493+00:00 app[web.1]: I, [2014-02-25T06:27:40.515825 #2] INFO -- : Rendered user/sessions/new.html.haml within layouts/application (8.2ms)
2014-02-25T06:27:46.981461+00:00 heroku[router]: at=info method=POST path=/users/sign_in host=incomon.herokuapp.com request_id=870f7f2b-252a-4932-acee-b155b11ff492 fwd="182.185.250.158" dyno=web.1 connect=0ms service=24ms status=500 bytes=1543
2014-02-25T06:27:47.372924+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=incomon.herokuapp.com request_id=527d7c85-5306-42c5-8880-093b22f2e39f fwd="182.185.250.158" dyno=web.1 connect=6ms service=6ms status=304 bytes=133
Error on Webpage
We're sorry, but something went wrong.
If you are the application owner check the logs for more information.
Why this will be happening? Any Help?
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "users" does not exist
It looks like maybe you haven't run your migrations yet?

Resources