currently i am using kafka 0.8.0 and high level consumer group to consume messages. topic has been created with 3 replica and 100 partition so that max 100 threads can consume messages simultaneously, but i am seeing that mostly threads are in waiting state and lag is getting increased.
Below is output of kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 91 119585 142834 23249 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-64
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 92 84963 144019 59056 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-65
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 93 723228 776541 53313 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-66
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 94 93068 143567 50499 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-67
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 95 84419 143149 58730 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-68
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 96 727430 777824 50394 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-69
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 97 104414 143725 39311 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-7
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 98 84337 143264 58927 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-8
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 99 721316 777124 55808 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-9
Below is out put of consumer thread group's thread_dump
KafkaConsumer-24" prio=10 tid=0x00007f6da5726800 nid=0x3c8a waiting on condition [0x00007f6d4f8f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c11ce780> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:63)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.safeRun(KafkaEventListenerContainer.java:141)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.run(KafkaEventListenerContainer.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"KafkaConsumer-23" prio=10 tid=0x00007f6da5724000 nid=0x3c89 waiting on condition [0x00007f6d4f9f8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c11cde90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:63)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.safeRun(KafkaEventListenerContainer.java:141)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.run(KafkaEventListenerContainer.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
I am not sure why my consumer thread's are waiting on this condition if messages are available in kafka partition.
Related
I'm having some very strange behavior in Tomcat 7. While startup Tomcat get a stuck for 2 minutes in internal.QueryTranslatorFactoryInitiator (QueryTranslatorFactoryInitiator.java:47) - HHH000397: Using ASTQueryTranslatorFactory
The server log looks like:
2017-04-30 16:51:13,850 INFO [RMI TCP Connection(3)-127.0.0.1] dialect.Dialect (Dialect.java:157) - HHH000400: Using dialect: org.hibernate.dialect.HSQLDialect
2017-04-30 16:51:14,013 INFO [RMI TCP Connection(3)-127.0.0.1] jpa.AbstractEntityManagerFactoryBean (AbstractEntityManagerFactoryBean.java:379) - Initialized JPA EntityManagerFactory for persistence unit 'cg.data.internal'
2017-04-30 16:51:14,038 INFO [RMI TCP Connection(3)-127.0.0.1] version.VersionManager (VersionManager.java:32) - version initialized: 18.0.0.1I0
2017-04-30 16:51:26,743 INFO [RMI TCP Connection(3)-127.0.0.1] internal.QueryTranslatorFactoryInitiator (QueryTranslatorFactoryInitiator.java:47) - HHH000397: Using ASTQueryTranslatorFactory
2017-04-30 16:53:30,394 INFO [RMI TCP Connection(3)-127.0.0.1] framework.CglibAopProxy (CglibAopProxy.java:265) - Unable to proxy method [public final javax.sql.DataSource org.springframework.jdbc.core.support.JdbcDaoSupport.getDataSource()] because it is final: All calls to this method via a proxy will NOT be routed to the target instance.
2017-04-30 16:53:30,395 INFO [RMI TCP Connection(3)-127.0.0.1] framework.CglibAopProxy (CglibAopProxy.java:265) - Unable to proxy method [public final void org.springframework.jdbc.core.support.JdbcDaoSupport.setDataSource(javax.sql.DataSource)] because it is final: All calls to this method via a proxy will NOT be routed to the target instance.
2017-04-30 16:53:30,395 INFO [RMI TCP Connection(3)-127.0.0.1] framework.CglibAopProxy (CglibAopProxy.java:265) - Unable to proxy method [protected final java.sql.Connection org.springframework.jdbc.core.support.JdbcDaoSupport.getConnection() throws org.springframework.jdbc.CannotGetJdbcConnectionException] because it is final: All calls to this method via a proxy will NOT be routed to the target instance.
2017-04-30 16:53:30,395 INFO [RMI TCP Connection(3)-127.0.0.1] framework.CglibAopProxy (CglibAopProxy.java:265) - Unable to proxy method [protected final org.springframework.jdbc.support.SQLExceptionTranslator org.springframework.jdbc.core.support.JdbcDaoSupport.getExceptionTranslator()] because it is final: All calls to this method via a proxy will NOT be routed to the target instance.
...
As you can see the startup stuck between 16:51 to 16:53.
At the end Catalina logs show that application started successfully.
I took java dump at this moment, but it looks like good as well.
Java dump:
2017-04-30 16:29:40
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.66-b18 mixed mode):
"HSQLDB Timer #2e5e4dd0" #44 daemon prio=5 os_prio=0 tid=0x000000005c472800 nid=0x3518 in Object.wait() [0x000000006a40f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)
- locked <0x00000000fba02cb8> (a org.hsqldb.lib.HsqlTimer$TaskQueue)
at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
- locked <0x00000000fba02cb8> (a org.hsqldb.lib.HsqlTimer$TaskQueue)
at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
"Hikari housekeeper (pool HikariPool-0)" #43 daemon prio=5 os_prio=0 tid=0x000000005c474800 nid=0x2eec waiting on condition [0x0000000069d6e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fbc94da0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Hikari housekeeper (pool CG_Proficiency)" #41 daemon prio=5 os_prio=0 tid=0x000000005c471800 nid=0x2b8c waiting on condition [0x000000006166f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fbe515a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Connection(idle)" #39 daemon prio=5 os_prio=0 tid=0x000000005c473000 nid=0x2f90 waiting on condition [0x00000000653cf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000008061b440> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"JMX server connection timeout 37" #37 daemon prio=5 os_prio=0 tid=0x000000005c470000 nid=0x2468 in Object.wait() [0x000000005ee1f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000008064ab70> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
- locked <0x000000008064ab70> (a [I)
at java.lang.Thread.run(Thread.java:745)
"ajp-apr-8009-AsyncTimeout" #33 daemon prio=5 os_prio=0 tid=0x000000005c46f800 nid=0x3650 waiting on condition [0x000000005e93f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$AsyncTimeout.run(AprEndpoint.java:1141)
at java.lang.Thread.run(Thread.java:745)
"ajp-apr-8009-Acceptor-0" #32 daemon prio=5 os_prio=0 tid=0x000000005c46e800 nid=0x277c runnable [0x000000005ecae000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1067)
at java.lang.Thread.run(Thread.java:745)
"ajp-apr-8009-Poller" #31 daemon prio=5 os_prio=0 tid=0x000000005c46e000 nid=0x3598 in Object.wait() [0x000000005ea9f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1756)
- locked <0x000000008064d310> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:745)
"http-apr-80-AsyncTimeout" #30 daemon prio=5 os_prio=0 tid=0x000000005c46d000 nid=0x378c waiting on condition [0x000000005e7ff000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$AsyncTimeout.run(AprEndpoint.java:1141)
at java.lang.Thread.run(Thread.java:745)
"http-apr-80-Acceptor-0" #29 daemon prio=5 os_prio=0 tid=0x000000005c46c800 nid=0x1bb8 runnable [0x000000005e67e000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1067)
at java.lang.Thread.run(Thread.java:745)
"http-apr-80-Sendfile" #28 daemon prio=5 os_prio=0 tid=0x000000005c46b800 nid=0x2454 in Object.wait() [0x000000005e47f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000806e46b0> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Object.wait(Object.java:502)
at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2264)
- locked <0x00000000806e46b0> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Thread.run(Thread.java:745)
"http-apr-80-Poller" #27 daemon prio=5 os_prio=0 tid=0x000000005c46b000 nid=0x3768 in Object.wait() [0x000000005e2cf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1756)
- locked <0x00000000806e4520> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:745)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" #26 daemon prio=5 os_prio=0 tid=0x000000005c46a000 nid=0x3674 waiting on condition [0x000000005df4f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1510)
at java.lang.Thread.run(Thread.java:745)
"GC Daemon" #24 daemon prio=2 os_prio=-2 tid=0x000000005c468800 nid=0x3048 in Object.wait() [0x000000005d8bf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000808d1a68> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x00000000808d1a68> (a sun.misc.GC$LatencyLock)
"RMI Scheduler(0)" #21 daemon prio=5 os_prio=0 tid=0x000000005bcd2800 nid=0x332c waiting on condition [0x000000005de0e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080681600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Connection(idle)" #20 daemon prio=5 os_prio=0 tid=0x000000005cb44800 nid=0x3310 waiting on condition [0x000000005d4ff000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000008061b440> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Connection(3)-127.0.0.1" #19 daemon prio=5 os_prio=0 tid=0x000000005bf99800 nid=0x31d4 runnable [0x000000005d3c9000]
java.lang.Thread.State: RUNNABLE
at org.aspectj.weaver.ReferenceType.findDerivativeType(ReferenceType.java:1264)
- locked <0x00000000fbc117a0> (a java.util.ArrayList)
at org.aspectj.weaver.TypeFactory.createParameterizedType(TypeFactory.java:44)
at org.aspectj.weaver.reflect.JavaLangTypeToResolvedTypeConverter.fromType(JavaLangTypeToResolvedTypeConverter.java:79)
at org.aspectj.weaver.reflect.JavaLangTypeToResolvedTypeConverter.fromTypes(JavaLangTypeToResolvedTypeConverter.java:129)
at org.aspectj.weaver.reflect.Java15GenericSignatureInformationProvider.getGenericParameterTypes(Java15GenericSignatureInformationProvider.java:48)
at org.aspectj.weaver.reflect.ReflectionBasedResolvedMemberImpl.getGenericParameterTypes(ReflectionBasedResolvedMemberImpl.java:112)
at org.aspectj.weaver.ResolvedMemberImpl.matches(ResolvedMemberImpl.java:1003)
at org.aspectj.weaver.ResolvedType.lookupResolvedMember(ResolvedType.java:627)
at org.aspectj.weaver.JoinPointSignatureIterator.findSignaturesFromSupertypes(JoinPointSignatureIterator.java:192)
at org.aspectj.weaver.JoinPointSignatureIterator.hasNext(JoinPointSignatureIterator.java:68)
at org.aspectj.weaver.patterns.SignaturePattern.matches(SignaturePattern.java:317)
at org.aspectj.weaver.patterns.KindedPointcut.matchInternal(KindedPointcut.java:197)
at org.aspectj.weaver.patterns.Pointcut.match(Pointcut.java:137)
at org.aspectj.weaver.internal.tools.PointcutExpressionImpl.getShadowMatch(PointcutExpressionImpl.java:319)
at org.aspectj.weaver.internal.tools.PointcutExpressionImpl.matchesExecution(PointcutExpressionImpl.java:129)
at org.aspectj.weaver.internal.tools.PointcutExpressionImpl.matchesMethodExecution(PointcutExpressionImpl.java:110)
at org.springframework.security.config.method.ProtectPointcutPostProcessor.attemptMatch(ProtectPointcutPostProcessor.java:142)
at org.springframework.security.config.method.ProtectPointcutPostProcessor.postProcessBeforeInitialization(ProtectPointcutPostProcessor.java:125)
- locked <0x00000000fbc93450> (a java.util.HashSet)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:129)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x0000000082142178> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:753)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
- locked <0x00000000822606c0> (a java.lang.Object)
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:666)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:632)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:680)
at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:551)
at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:492)
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:136)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1282)
- locked <0x0000000081122540> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
- locked <0x0000000081122540> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
- locked <0x0000000081122540> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
- locked <0x0000000080789b40> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
- locked <0x0000000080789b40> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.manageApp(HostConfig.java:1863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:301)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at org.apache.catalina.mbeans.MBeanFactory.createStandardContext(MBeanFactory.java:618)
at org.apache.catalina.mbeans.MBeanFactory.createStandardContext(MBeanFactory.java:565)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:301)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1471)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:832)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$256(TCPTransport.java:683)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1846791746.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-0" #18 daemon prio=5 os_prio=0 tid=0x0000000058db4800 nid=0x3464 runnable [0x000000005b63f000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000000806819a8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-1099" #17 daemon prio=5 os_prio=0 tid=0x0000000058dee000 nid=0x32c8 runnable [0x000000005b51e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x0000000080681e50> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-0" #16 daemon prio=5 os_prio=0 tid=0x0000000058de4800 nid=0x3494 runnable [0x000000005b26e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x000000008064b170> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #13 daemon prio=9 os_prio=0 tid=0x0000000058b14000 nid=0x2cc0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #12 daemon prio=9 os_prio=2 tid=0x0000000058b2c800 nid=0x3328 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #11 daemon prio=9 os_prio=2 tid=0x0000000058b1a000 nid=0x36f4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #10 daemon prio=9 os_prio=2 tid=0x0000000058b12000 nid=0x30f0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #9 daemon prio=9 os_prio=2 tid=0x0000000058b1d000 nid=0x2e04 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Command Reader" #8 daemon prio=10 os_prio=0 tid=0x0000000058b07000 nid=0x3440 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" #7 daemon prio=10 os_prio=0 tid=0x0000000058b01800 nid=0x31b8 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" #6 daemon prio=10 os_prio=0 tid=0x0000000058afc800 nid=0x2b00 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000058aec800 nid=0x2ee4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000058aeb000 nid=0xd38 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000057b48000 nid=0x5c4 in Object.wait() [0x000000005a09f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000080553528> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000057b41000 nid=0x3724 in Object.wait() [0x0000000059eaf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000000805507c8> (a java.lang.ref.Reference$Lock)
"main" #1 prio=5 os_prio=0 tid=0x00000000021bd800 nid=0x35b4 runnable [0x000000000248e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000000807a60c8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:470)
at org.apache.catalina.startup.Catalina.await(Catalina.java:781)
at org.apache.catalina.startup.Catalina.start(Catalina.java:727)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
Why this happens? May it be some issue related to connection pool configuration?
I use Tomcat 7.0.69, Spring 4, Hibernate 4.
I am trying to run simple hive queries like
select * from table_name;
But getting error message as mentioned below.
Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:266)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:213)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:333)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:719)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:252)
... 11 more
Caused by: java.io.IOException: Too many bytes before newline: 2147483648
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:249)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:91)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:136)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)
... 16 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 748 Reduce: 856 Cumulative CPU: 2938.2 sec HDFS Read: 173243267086 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 48 minutes 58 seconds 200 msec
I could not find a way around this problem. I also looked into this question
But could not find a way around Integer.MAX_VALUE. I would really appreciate any suggestion.
My scenario: read data from elasticsearch then do some calculations, the final result of the calculation is stored elasticsearch.
I use a small amount of data to test is successful,But switch to big amount of data always occur this error. I am really confused
spark version : 1.6.1
elasticsearch version : 2.3.1
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 37, 10.10.150.231): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: null
c2NhbjsxOzMxMzY0OlpFSWVjWnh5Ukxtd1diMUdoVXJINVE7MTt0b3RhbF9oaXRzOjQ2NzIwOw==
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:478)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:436)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:426)
at org.elasticsearch.hadoop.rest.RestClient.scroll(RestClient.java:496)
at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:454)
at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:86)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:43)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:67)
at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:52)
at org.elasticsearch.spark.package$SparkRDDFunctions.saveToEs(package.scala:37)
at BothwayForPU$.main(BothwayForPU.scala:82)
at BothwayForPU.main(BothwayForPU.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: null
c2NhbjsxOzMxMzY0OlpFSWVjWnh5Ukxtd1diMUdoVXJINVE7MTt0b3RhbF9oaXRzOjQ2NzIwOw==
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:478)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:436)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:426)
at org.elasticsearch.hadoop.rest.RestClient.scroll(RestClient.java:496)
at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:454)
at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:86)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:43)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I use spring boot(1.3.1.RELEASE) to define an interval task(fixedDelay = 500) which call restTemplate.getForObject, but suddenly a few days later the task is no longer scheduled by spring, I use jconsole，I found the method has been waiting for restTemplate.getForObject return for a day, but it don't
who knows the reason?
in the thread stack, you can see
"
- locked <0x00000000e272c850> (a java.io.BufferedInputStream)
at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
"
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000e272c850> (a java.io.BufferedInputStream)
at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
- locked <0x00000000e272c878> (a sun.net.www.http.ChunkedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3322)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore(UTF8StreamJsonParser.java:180)
at com.fasterxml.jackson.core.base.ParserBase.loadMoreGuaranteed(ParserBase.java:459)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2434)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2414)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:285)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:514)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:653)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:496)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2810)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:221)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:208)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:95)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:599)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:264)
at com.dd.ddfd.clairvoyant.collector.ScheduledSummaryCollector.scheduledCollector(ScheduledSummaryCollector.java:85)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I am occasionally running into deadlock issue when starting my osgi application. My project has following external dependencies (does NOT contain my project specific bundles):
id State Bundle
0 ACTIVE org.eclipse.osgi_3.8.2.v20130124-134944
1 ACTIVE com.google.gson_2.1.0.v201203072145
2 ACTIVE org.eclipse.equinox.console_1.0.100.v20121001-124408
3 ACTIVE org.apache.felix.gogo.shell_0.10.0.v201211091412
4 ACTIVE org.apache.felix.gogo.runtime_0.10.0.v201209301036
5 ACTIVE org.apache.felix.gogo.command_0.10.0.v201209301215
6 ACTIVE org.apache.felix.configadmin_1.6.0
7 ACTIVE org.springframework.osgi.log4j.osgi_1.2.15.SNAPSHOT
Fragments=23
8 ACTIVE org.apache.felix.eventadmin_1.3.2
9 ACTIVE org.apache.felix.fileinstall_3.2.6
12 ACTIVE com.springsource.edu.emory.mathcs.backport_3.1.0
13 ACTIVE org.eclipse.gemini.blueprint.core_1.0.2.RELEASE
14 ACTIVE org.springframework.beans_3.2.2.RELEASE
16 ACTIVE org.apache.felix.http.jetty_2.2.0
17 ACTIVE org.eclipse.gemini.blueprint.extender_1.0.2.RELEASE
19 ACTIVE org.eclipse.gemini.blueprint.io_1.0.2.RELEASE
20 ACTIVE org.springframework.context_3.2.2.RELEASE
24 ACTIVE org.springframework.expression_3.2.2.RELEASE
26 ACTIVE org.springframework.core_3.2.2.RELEASE
28 ACTIVE org.apache.felix.http.whiteboard_2.2.0
29 ACTIVE org.eclipse.orbit.mongodb_2.7.3.v20120213-1927
30 ACTIVE org.springframework.context.support_3.2.2.RELEASE
31 ACTIVE com.springsource.org.apache.commons.logging_1.1.1
32 ACTIVE org.springframework.aop_3.2.2.RELEASE
34 ACTIVE com.springsource.org.aopalliance_1.0.0
The thread dump of my application is as follows :
Found one Java-level deadlock:
"Gogo shell":
waiting to lock monitor 7fe00d8b58d8 (object 7ad401900, a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry),
which is held by "EclipseGeminiBlueprintExtenderThread-24"
"EclipseGeminiBlueprintExtenderThread-24":
waiting to lock monitor 7fe00c078660 (object 7ad35bae8, a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader),
which is held by "EclipseGeminiBlueprintExtenderThread-12"
"EclipseGeminiBlueprintExtenderThread-12":
waiting to lock monitor 7fe00d8b58d8 (object 7ad401900, a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry),
which is held by "EclipseGeminiBlueprintExtenderThread-24"
Java stack information for the threads listed above:
"Gogo shell":
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.lookupServiceRegistrations(ServiceRegistry.java:953)
- waiting to lock <7ad401900> (a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.notifyHooksPrivileged(ServiceRegistry.java:1217)
at org.eclipse.osgi.internal.baseadaptor.weaving.WovenClassImpl.callHooks(WovenClassImpl.java:182)
at org.eclipse.osgi.internal.baseadaptor.weaving.WeavingHookConfigurator.processClass(WeavingHookConfigurator.java:68)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.defineClass(ClasspathManager.java:600)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(ClasspathManager.java:568)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(ClasspathManager.java:492)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:465)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:464)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:88)
at org.apache.felix.gogo.shell.Activator.run(Activator.java:75)
at java.lang.Thread.run(Thread.java:680)
"EclipseGeminiBlueprintExtenderThread-24":
at org.eclipse.gemini.blueprint.util.internal.MapBasedDictionary.keys(MapBasedDictionary.java:183)
at org.eclipse.osgi.internal.serviceregistry.ServiceProperties.<init>(ServiceProperties.java:37)
- locked <7a922b678> (a org.eclipse.gemini.blueprint.util.internal.MapBasedDictionary)
at org.eclipse.osgi.internal.serviceregistry.ServiceProperties.<init>(ServiceProperties.java:57)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.createProperties(ServiceRegistrationImpl.java:294)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:121)
- locked <7a922b928> (a java.lang.Object)
- locked <7ad401900> (a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.gemini.blueprint.context.support.AbstractOsgiBundleApplicationContext.publishContextAsOsgiServiceIfNecessary(AbstractOsgiBundleApplicationContext.java:361)
at org.eclipse.gemini.blueprint.context.support.AbstractOsgiBundleApplicationContext.finishRefresh(AbstractOsgiBundleApplicationContext.java:223)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:328)
- locked <7ad7ef180> (a java.lang.Object)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.lang.Thread.run(Thread.java:680)
"EclipseGeminiBlueprintExtenderThread-12":
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.lookupServiceRegistrations(ServiceRegistry.java:953)
- waiting to lock <7ad401900> (a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.notifyHooksPrivileged(ServiceRegistry.java:1217)
at org.eclipse.osgi.internal.baseadaptor.weaving.WovenClassImpl.callHooks(WovenClassImpl.java:182)
at org.eclipse.osgi.internal.baseadaptor.weaving.WeavingHookConfigurator.processClass(WeavingHookConfigurator.java:68)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.defineClass(ClasspathManager.java:600)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(ClasspathManager.java:568)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(ClasspathManager.java:492)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:465)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:464)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2475)
at java.lang.Class.getMethod0(Class.java:2718)
at java.lang.Class.getMethod(Class.java:1630)
at java.lang.Class.getEnumConstantsShared(Class.java:2982)
at java.lang.Class.enumConstantDirectory(Class.java:3011)
at java.lang.Enum.valueOf(Enum.java:191)
at org.eclipse.gemini.blueprint.config.internal.ServiceBeanDefinitionParser$ServiceAttributeCallback.process(ServiceBeanDefinitionParser.java:75)
at org.eclipse.gemini.blueprint.config.internal.util.ParserUtils.parseAttributes(ParserUtils.java:65)
at org.eclipse.gemini.blueprint.config.internal.util.ParserUtils.parseCustomAttributes(ParserUtils.java:93)
at org.eclipse.gemini.blueprint.config.internal.ServiceBeanDefinitionParser.parseAttributes(ServiceBeanDefinitionParser.java:200)
at org.eclipse.gemini.blueprint.config.internal.ServiceBeanDefinitionParser.doParse(ServiceBeanDefinitionParser.java:132)
at org.springframework.beans.factory.xml.AbstractSingleBeanDefinitionParser.parseInternal(AbstractSingleBeanDefinitionParser.java:85)
at org.springframework.beans.factory.xml.AbstractBeanDefinitionParser.parse(AbstractBeanDefinitionParser.java:59)
at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:73)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1438)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1428)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:185)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:139)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:108)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:493)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:334)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:302)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:174)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:209)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:180)
at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:170)
at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:140)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:130)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:537)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242)
- locked <7ae07c8b8> (a java.lang.Object)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157)
at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:211)
at java.lang.Thread.run(Thread.java:680)
Found 1 deadlock.
The threads EclipseGeminiBlueprintExtenderThread-24 and EclipseGeminiBlueprintExtenderThread-12 are deadlocked and Gogo shell is involved somehow.
My question is what is causing this deadlock? - is it something in my code or it's a bug in gemini? How do I fix this issue? Any pointers will he highly appreciated.