I am occasionally running into deadlock issue when starting my osgi application. My project has following external dependencies (does NOT contain my project specific bundles):
id State Bundle
0 ACTIVE org.eclipse.osgi_3.8.2.v20130124-134944
1 ACTIVE com.google.gson_2.1.0.v201203072145
2 ACTIVE org.eclipse.equinox.console_1.0.100.v20121001-124408
3 ACTIVE org.apache.felix.gogo.shell_0.10.0.v201211091412
4 ACTIVE org.apache.felix.gogo.runtime_0.10.0.v201209301036
5 ACTIVE org.apache.felix.gogo.command_0.10.0.v201209301215
6 ACTIVE org.apache.felix.configadmin_1.6.0
7 ACTIVE org.springframework.osgi.log4j.osgi_1.2.15.SNAPSHOT
Fragments=23
8 ACTIVE org.apache.felix.eventadmin_1.3.2
9 ACTIVE org.apache.felix.fileinstall_3.2.6
12 ACTIVE com.springsource.edu.emory.mathcs.backport_3.1.0
13 ACTIVE org.eclipse.gemini.blueprint.core_1.0.2.RELEASE
14 ACTIVE org.springframework.beans_3.2.2.RELEASE
16 ACTIVE org.apache.felix.http.jetty_2.2.0
17 ACTIVE org.eclipse.gemini.blueprint.extender_1.0.2.RELEASE
19 ACTIVE org.eclipse.gemini.blueprint.io_1.0.2.RELEASE
20 ACTIVE org.springframework.context_3.2.2.RELEASE
24 ACTIVE org.springframework.expression_3.2.2.RELEASE
26 ACTIVE org.springframework.core_3.2.2.RELEASE
28 ACTIVE org.apache.felix.http.whiteboard_2.2.0
29 ACTIVE org.eclipse.orbit.mongodb_2.7.3.v20120213-1927
30 ACTIVE org.springframework.context.support_3.2.2.RELEASE
31 ACTIVE com.springsource.org.apache.commons.logging_1.1.1
32 ACTIVE org.springframework.aop_3.2.2.RELEASE
34 ACTIVE com.springsource.org.aopalliance_1.0.0
The thread dump of my application is as follows :
Found one Java-level deadlock:
"Gogo shell":
waiting to lock monitor 7fe00d8b58d8 (object 7ad401900, a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry),
which is held by "EclipseGeminiBlueprintExtenderThread-24"
"EclipseGeminiBlueprintExtenderThread-24":
waiting to lock monitor 7fe00c078660 (object 7ad35bae8, a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader),
which is held by "EclipseGeminiBlueprintExtenderThread-12"
"EclipseGeminiBlueprintExtenderThread-12":
waiting to lock monitor 7fe00d8b58d8 (object 7ad401900, a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry),
which is held by "EclipseGeminiBlueprintExtenderThread-24"
Java stack information for the threads listed above:
"Gogo shell":
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.lookupServiceRegistrations(ServiceRegistry.java:953)
- waiting to lock <7ad401900> (a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.notifyHooksPrivileged(ServiceRegistry.java:1217)
at org.eclipse.osgi.internal.baseadaptor.weaving.WovenClassImpl.callHooks(WovenClassImpl.java:182)
at org.eclipse.osgi.internal.baseadaptor.weaving.WeavingHookConfigurator.processClass(WeavingHookConfigurator.java:68)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.defineClass(ClasspathManager.java:600)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(ClasspathManager.java:568)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(ClasspathManager.java:492)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:465)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:464)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:88)
at org.apache.felix.gogo.shell.Activator.run(Activator.java:75)
at java.lang.Thread.run(Thread.java:680)
"EclipseGeminiBlueprintExtenderThread-24":
at org.eclipse.gemini.blueprint.util.internal.MapBasedDictionary.keys(MapBasedDictionary.java:183)
at org.eclipse.osgi.internal.serviceregistry.ServiceProperties.<init>(ServiceProperties.java:37)
- locked <7a922b678> (a org.eclipse.gemini.blueprint.util.internal.MapBasedDictionary)
at org.eclipse.osgi.internal.serviceregistry.ServiceProperties.<init>(ServiceProperties.java:57)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.createProperties(ServiceRegistrationImpl.java:294)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:121)
- locked <7a922b928> (a java.lang.Object)
- locked <7ad401900> (a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.gemini.blueprint.context.support.AbstractOsgiBundleApplicationContext.publishContextAsOsgiServiceIfNecessary(AbstractOsgiBundleApplicationContext.java:361)
at org.eclipse.gemini.blueprint.context.support.AbstractOsgiBundleApplicationContext.finishRefresh(AbstractOsgiBundleApplicationContext.java:223)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:328)
- locked <7ad7ef180> (a java.lang.Object)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.lang.Thread.run(Thread.java:680)
"EclipseGeminiBlueprintExtenderThread-12":
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.lookupServiceRegistrations(ServiceRegistry.java:953)
- waiting to lock <7ad401900> (a org.eclipse.osgi.internal.serviceregistry.ServiceRegistry)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.notifyHooksPrivileged(ServiceRegistry.java:1217)
at org.eclipse.osgi.internal.baseadaptor.weaving.WovenClassImpl.callHooks(WovenClassImpl.java:182)
at org.eclipse.osgi.internal.baseadaptor.weaving.WeavingHookConfigurator.processClass(WeavingHookConfigurator.java:68)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.defineClass(ClasspathManager.java:600)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(ClasspathManager.java:568)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(ClasspathManager.java:492)
at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:465)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:464)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2475)
at java.lang.Class.getMethod0(Class.java:2718)
at java.lang.Class.getMethod(Class.java:1630)
at java.lang.Class.getEnumConstantsShared(Class.java:2982)
at java.lang.Class.enumConstantDirectory(Class.java:3011)
at java.lang.Enum.valueOf(Enum.java:191)
at org.eclipse.gemini.blueprint.config.internal.ServiceBeanDefinitionParser$ServiceAttributeCallback.process(ServiceBeanDefinitionParser.java:75)
at org.eclipse.gemini.blueprint.config.internal.util.ParserUtils.parseAttributes(ParserUtils.java:65)
at org.eclipse.gemini.blueprint.config.internal.util.ParserUtils.parseCustomAttributes(ParserUtils.java:93)
at org.eclipse.gemini.blueprint.config.internal.ServiceBeanDefinitionParser.parseAttributes(ServiceBeanDefinitionParser.java:200)
at org.eclipse.gemini.blueprint.config.internal.ServiceBeanDefinitionParser.doParse(ServiceBeanDefinitionParser.java:132)
at org.springframework.beans.factory.xml.AbstractSingleBeanDefinitionParser.parseInternal(AbstractSingleBeanDefinitionParser.java:85)
at org.springframework.beans.factory.xml.AbstractBeanDefinitionParser.parse(AbstractBeanDefinitionParser.java:59)
at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:73)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1438)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1428)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:185)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:139)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:108)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:493)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:334)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:302)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:174)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:209)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:180)
at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:170)
at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:140)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:130)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:537)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242)
- locked <7ae07c8b8> (a java.lang.Object)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157)
at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:211)
at java.lang.Thread.run(Thread.java:680)
Found 1 deadlock.
The threads EclipseGeminiBlueprintExtenderThread-24 and EclipseGeminiBlueprintExtenderThread-12 are deadlocked and Gogo shell is involved somehow.
My question is what is causing this deadlock? - is it something in my code or it's a bug in gemini? How do I fix this issue? Any pointers will he highly appreciated.
Related
I have just Installed the WebSphere Application Server, The status on localHost shows its started but when I try to start it from Eclipse I am getting following :
[00000000 WsServerImpl E WSVR0009E: Error occurred during startup
com.ibm.ws.exception.RuntimeError: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4 vmcid: 0x49421000 minor code: 56 completed: No
at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:427)
at com.ibm.ws.runtime.component.ContainerHelper.startComponents(ContainerHelper.java:515)
at com.ibm.ws.runtime.component.ContainerImpl.startComponents(ContainerImpl.java:631)
at com.ibm.ws.runtime.component.ContainerImpl.start(ContainerImpl.java:621)
at com.ibm.ws.runtime.component.ServerImpl.start(ServerImpl.java:520)
at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:298)
at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:214)
at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:666)
at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:213)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:93)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:74)
at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:78)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:68)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:400)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
at org.eclipse.core.launcher.Main.run(Main.java:981)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:330)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:108)
Caused by: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4 vmcid: 0x49421000 minor code: 56 completed: No
at com.ibm.ws.orbimpl.transport.WSTransport.createListener(WSTransport.java:866)
at com.ibm.ws.orbimpl.transport.WSTransport.initTransports(WSTransport.java:604)
at com.ibm.rmi.iiop.TransportManager.initTransports(TransportManager.java:151)
at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1231)
at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1681)
at org.omg.CORBA.ORB.init(ORB.java:364)
at com.ibm.ws.orb.GlobalORBFactory.init(GlobalORBFactory.java:92)
at com.ibm.ejs.oa.EJSORBImpl.initializeORB(EJSORBImpl.java:179)
at com.ibm.ejs.oa.EJSServerORBImpl.<init>(EJSServerORBImpl.java:102)
at com.ibm.ejs.oa.EJSORB.init(EJSORB.java:55)
at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:420)
... 29 more
What may be the reason behind this.
I use spring boot(1.3.1.RELEASE) to define an interval task(fixedDelay = 500) which call restTemplate.getForObject, but suddenly a few days later the task is no longer scheduled by spring, I use jconsole，I found the method has been waiting for restTemplate.getForObject return for a day, but it don't
who knows the reason?
in the thread stack, you can see
"
- locked <0x00000000e272c850> (a java.io.BufferedInputStream)
at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
"
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000e272c850> (a java.io.BufferedInputStream)
at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
- locked <0x00000000e272c878> (a sun.net.www.http.ChunkedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3322)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore(UTF8StreamJsonParser.java:180)
at com.fasterxml.jackson.core.base.ParserBase.loadMoreGuaranteed(ParserBase.java:459)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2434)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2414)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:285)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:514)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:653)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:496)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2810)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:221)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:208)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:95)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:599)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:264)
at com.dd.ddfd.clairvoyant.collector.ScheduledSummaryCollector.scheduledCollector(ScheduledSummaryCollector.java:85)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I am trying to investigate what happens when an exception is throw in the writer. As per theory I was expecting a roll-back to occur and then retry the chunk setting commit size as 1. but this is not happening and i am receiving the following error.
I am using Spring Batch -JSR 352 job design.
In the writer I have 3 JDBC calls. If i throw an skippable exception after 2 JDBC calls have been successfully executed , then I could find the Spring batch is not able to rollback the transaction for the chunk and retry the chunk by placing the commit size as one. I am receiving the following execption after which the entire chunk is getting skipped and we are having an chunk commit. As a result the data already written to 2 JDBC tables are getting persisted missing the 3rd table data.
20 Apr 2015 18:38:28,811 ERROR [gov.state.nextgen.in.batch.sdx.chunk.writer.SdxRcvDlyWriter.writeItems(SdxRcvDlyWriter.java:74)] - Error while writing sdxRcvRecord :Exception occured during processing
at gov.state.nextgen.in.batch.sdx.bo.impl.SdxRcvDlyBoImpl.write(SdxRcvDlyBoImpl.java:125)
at gov.state.nextgen.in.batch.sdx.chunk.writer.SdxRcvDlyWriter.writeItems(SdxRcvDlyWriter.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy18.writeItems(Unknown Source)
at org.springframework.batch.jsr.item.ItemWriterAdapter.write(ItemWriterAdapter.java:55)
at org.springframework.batch.core.jsr.step.item.JsrChunkProcessor.doPersist(JsrChunkProcessor.java:243)
at org.springframework.batch.core.jsr.step.item.JsrFaultTolerantChunkProcessor$5.doWithRetry(JsrFaultTolerantChunkProcessor.java:298)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:263)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:168)
at org.springframework.batch.core.step.item.BatchRetryTemplate.execute(BatchRetryTemplate.java:222)
at org.springframework.batch.core.jsr.step.item.JsrFaultTolerantChunkProcessor.persist(JsrFaultTolerantChunkProcessor.java:348)
at org.springframework.batch.core.jsr.step.item.JsrChunkProcessor.process(JsrChunkProcessor.java:114)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
at org.springframework.batch.core.jsr.job.flow.JsrFlowJob.doExecute(JsrFlowJob.java:82)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
at org.springframework.batch.core.jsr.launch.JsrJobOperator$2.run(JsrJobOperator.java:675)
at java.lang.Thread.run(Thread.java:795)
20 Apr 2015 18:38:50,157 DEBUG [org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:293)] - Checking for rethrow: count=1
20 Apr 2015 18:38:52,584 DEBUG [org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:313)] - Retry failed last attempt: count=1
20 Apr 2015 18:38:53,403 DEBUG [org.springframework.batch.core.jsr.step.item.JsrFaultTolerantChunkProcessor$6.recover(JsrFaultTolerantChunkProcessor.java:333)] - Skipping after failed write
org.springframework.batch.core.step.item.ForceRollbackForWriteSkipException: Force rollback on skippable exception so that skipped item can be located.
at org.springframework.batch.core.jsr.step.item.JsrFaultTolerantChunkProcessor$5.doWithRetry(JsrFaultTolerantChunkProcessor.java:317)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:263)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:168)
at org.springframework.batch.core.step.item.BatchRetryTemplate.execute(BatchRetryTemplate.java:222)
at org.springframework.batch.core.jsr.step.item.JsrFaultTolerantChunkProcessor.persist(JsrFaultTolerantChunkProcessor.java:348)
at org.springframework.batch.core.jsr.step.item.JsrChunkProcessor.process(JsrChunkProcessor.java:114)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
at org.springframework.batch.core.jsr.job.flow.JsrFlowJob.doExecute(JsrFlowJob.java:82)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
at org.springframework.batch.core.jsr.launch.JsrJobOperator$2.run(JsrJobOperator.java:675)
at java.lang.Thread.run(Thread.java:795)
Caused by: gov.state.nextgen.framework.batch.exception.NGBatchSkipRecordException: gov.state.nextgen.in.batch.exception.INBatchException: Exception occured during processing
at gov.state.nextgen.in.batch.sdx.chunk.writer.SdxRcvDlyWriter.writeItems(SdxRcvDlyWriter.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy18.writeItems(Unknown Source)
at org.springframework.batch.jsr.item.ItemWriterAdapter.write(ItemWriterAdapter.java:55)
at org.springframework.batch.core.jsr.step.item.JsrChunkProcessor.doPersist(JsrChunkProcessor.java:243)
at org.springframework.batch.core.jsr.step.item.JsrFaultTolerantChunkProcessor$5.doWithRetry(JsrFaultTolerantChunkProcessor.java:298)
... 26 more
Caused by: gov.state.nextgen.in.batch.exception.INBatchException: Exception occured during processing
at gov.state.nextgen.in.batch.sdx.bo.impl.SdxRcvDlyBoImpl.write(SdxRcvDlyBoImpl.java:125)
at gov.state.nextgen.in.batch.sdx.chunk.writer.SdxRcvDlyWriter.writeItems(SdxRcvDlyWriter.java:65)
... 41 more20 Apr 2015 18:39:14,808 DEBUG [org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:88)] - Inputs not busy, ended: false
20 Apr 2015 18:39:17,985 DEBUG [org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:437)] - Applying contribution: [StepContribution: read=2, written=0, filtered=0, readSkips=0, writeSkips=1, processSkips=0, exitStatus=EXECUTING]
20 Apr 2015 18:39:18,984 INFO [gov.state.nextgen.in.batch.sdx.chunk.writer.SdxRcvDlyWriter.checkpointInfo(SdxRcvDlyWriter.java:87)] - ParisRcvFedWriter - checkpointInfo(): step name: process, step execution id: 14939
20 Apr 2015 18:39:19,314 TRACE [org.springframework.transaction.support.TransactionSynchronizationManager.getResource(TransactionSynchronizationManager.java:140)] - Retrieved value [org.springframework.jdbc.datasource.ConnectionHolder#5c4a884f] for key [org.apache.commons.dbcp.BasicDataSource#d9bdc230] bound to thread [SimpleAsyncTaskExecutor-1]
20 Apr 2015 18:39:19,314 DEBUG [org.springframework.transaction.support.AbstractPlatformTransactionManager.handleExistingTransaction(AbstractPlatformTransactionManager.java:472)] - Participating in existing transaction
20 Apr 2015 18:39:19,314 TRACE [org.springframework.transaction.interceptor.TransactionAspectSupport.prepareTransactionInfo(TransactionAspectSupport.java:447)] - Getting transaction for [org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext]
20 Apr 2015 18:39:19,315 DEBUG [org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:908)] - Executing prepared SQL update
20 Apr 2015 18:39:19,315 DEBUG [org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:627)] - Executing prepared SQL statement [UPDATE BATCH_STEP_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE STEP_EXECUTION_ID = ?]
I am attaching my configuration files for reference. By debugging I could find my insert statements are getting executed under transaction managed by spring and I am sure there is no issue with the transaction manager configuration
My app deployed on weblogic 10.3, use a constraint for control some isert, but, from a few days ago, this is what happend when the constraint is thrown:
why oracle is disconnecting my weblogic pool user from database?
2014-04-01 14:07:20,047 - ERROR org.springframework.orm.hibernate3.HibernateTransactionManager - Commit exception overridden by rollback exception
org.springframework.dao.DataAccessResourceFailureException: Could not execute JDBC batch update; nested exception is org.hibernate.exception.JDBCConnectionException: Could not execute JDBC batch update
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:609)
at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:717)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:586)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy97.save(Unknown Source)
at -core.operaciones.OperacionObtenerFacturasProformaPdtesLeer.registrarFacturaProforma(OperacionObtenerFacturasProformaPdtesLeer.java:100)
at -core.operaciones.OperacionObtenerFacturasProformaPdtesLeer.execute(OperacionObtenerFacturasProformaPdtesLeer.java:53)
at -core.operaciones.AbstractGenericOperacion.execute(AbstractGenericOperacion.java:111)
at -core.procesos.AbstractGenericProceso.execute(AbstractGenericProceso.java:63)
at -util.LanzaProcesos.doPost(LanzaProcesos.java:76)
at -util.LanzaProcesos.doGet(LanzaProcesos.java:188)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:180)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3684)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: org.hibernate.exception.JDBCConnectionException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:253)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:237)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:578)
... 32 more
Caused by: java.sql.BatchUpdateException: Io exception: Size Data Unit (SDU) mismatch
at oracle.jdbc.driver.DatabaseError.throwBatchUpdateException(DatabaseError.java:629)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9467)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:211)
at weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:246)
... 40 more
<Apr 1, 2014 2:07:20 PM CEST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "OracleDS" failed with exception: "java.sql.SQLException: Closed Connection".>
Thanks
currently i am using kafka 0.8.0 and high level consumer group to consume messages. topic has been created with 3 replica and 100 partition so that max 100 threads can consume messages simultaneously, but i am seeing that mostly threads are in waiting state and lag is getting increased.
Below is output of kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 91 119585 142834 23249 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-64
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 92 84963 144019 59056 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-65
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 93 723228 776541 53313 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-66
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 94 93068 143567 50499 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-67
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 95 84419 143149 58730 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-68
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 96 727430 777824 50394 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-69
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 97 104414 143725 39311 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-7
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 98 84337 143264 58927 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-8
kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com kafka_topic_coms_esb_prod_coms 99 721316 777124 55808 kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1396988657075-27d01963-9
Below is out put of consumer thread group's thread_dump
KafkaConsumer-24" prio=10 tid=0x00007f6da5726800 nid=0x3c8a waiting on condition [0x00007f6d4f8f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c11ce780> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:63)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.safeRun(KafkaEventListenerContainer.java:141)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.run(KafkaEventListenerContainer.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"KafkaConsumer-23" prio=10 tid=0x00007f6da5724000 nid=0x3c89 waiting on condition [0x00007f6d4f9f8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c11cde90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:63)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.safeRun(KafkaEventListenerContainer.java:141)
at com.snapdeal.coms.kafka.KafkaEventListenerContainer$KafkaConsumer.run(KafkaEventListenerContainer.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
I am not sure why my consumer thread's are waiting on this condition if messages are available in kafka partition.