When I connect to the url og my app on google app engine I see this stack trace on google Logging section
F 16:17:52.664 Uncaught exception from servlet
java.lang.ExceptionInInitializerError
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:121)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:332)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:282)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:206)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:179)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:136)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:469)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:256)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:235)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createSecurityManager")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:382)
at java.security.AccessController.checkPermission(AccessController.java:572)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at com.google.apphosting.runtime.security.CustomSecurityManager.checkPermission(CustomSecurityManager.java:55)
at java.lang.SecurityManager.<init>(SecurityManager.java:299)
at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
at org.slf4j.helpers.Util.<clinit>(Util.java:49)
... 27 more
I 16:17:52.666 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
What does it means?
My application is a Maven application, with Spring, Jpa, Hibernate and some jsp pages
Related
I am working on a spring batch application, which is using a comboPooledDataSource to connect to the database. And the application is running on tomcat9.
When I stop the pod(container) to deploy another version of application, the logs shows that the application has been stoped, but before destroying the container it throws some warnings.
19-Aug-2020 09:58:16.235 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
19-Aug-2020 09:58:16.235 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [C3P0PooledConnectionPoolManager[]-DeferredStatementDestroyerThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
19-Aug-2020 09:58:16.236 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
19-Aug-2020 09:58:16.236 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [Resource Destroyer in BasicResourcePool.close()] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
oracle.net.ns.Packet.receive(Packet.java:311)
oracle.net.ns.DataPacket.receive(DataPacket.java:105)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:59)
oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:844)
oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2498)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:642)
com.mchange.v2.c3p0.impl.NewPooledConnection.closeMaybeCheckedOut(NewPooledConnection.java:255)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:622)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:1076)
com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1101)
com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1062)
com.mchange.v2.resourcepool.BasicResourcePool.access$100(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$5.run(BasicResourcePool.java:1316)
19-Aug-2020 09:58:16.239 SEVERE [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#19931e16]) and a value of type [org.apache.logging.log4j.core.LoggerContext] (value [org.apache.logging.log4j.core.LoggerContext#205f573e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
I Have checked other related questions where it says
**
These are just warnings, tomcat has removed or stopped it forcibly.
**
But I want to make changes in my application. So that, no other warnings are to be shown from application design point of view.
Could anyone have any idea, what to check in the application and how to solve it.
We use SpringBoot 1.4.7 web to deploy ActiveMq 5.13.5 broker. The build artifact is a jar file which is launched by java command. Most of the time, it runs smoothly without any problem. However, without restarting the app, it randomly throws the following exception once or twice a day:
java.io.IOException: Unexpected error occurred: java.lang.NoClassDefFoundError: org/apache/activemq/command/ActiveMQTextMessage
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:222)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/activemq/command/ActiveMQTextMessage
at org.apache.activemq.openwire.v11.ActiveMQTextMessageMarshaller.createObject(ActiveMQTextMessageMarshaller.java:55)
at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:360)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:277)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
... 1 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.activemq.command.ActiveMQTextMessage
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 common frames omitted
I ensured the class ActiveMQTextMessage had been loaded before the exception time by outputting the class loading debug. I also ensured the system has plenty of free capacity for the cpu, memory(heap and meta), disk, file handles and sockets.
I have a few questions here:
Anyone saw similar problem and know the cause and fix?
Does SpringBoot unload class at runtime and reload them back?
Any hint to further debug this type of problem is highly appreciated.
Edit:
The dependency tree for compile and runtime is posted here: https://gist.github.com/liuzy163/8b8f8c7119624b5bdbe30e133d5f5910
i am trying to connect db2 from my java code in web sphere application server.
i am getting following exception.please suggest any one on this.
Caused by: com.ibm.websphere.naming.CannotInstantiateObjectException: Exception occurred while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is javax.xml.stream.FactoryConfigurationError: Provider javax.xml.stream.XMLInputFactory could not be instantiated: java.util.ServiceConfigurationError: javax.xml.stream.XMLInputFactory: Provider com.sun.xml.internal.stream.XMLInputFactoryImpl not a subtype] at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookupExt(Helpers.java:1232) at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookup(Helpers.java:925) at com.ibm.ws.naming.jndicos.CNContextImpl.processBoundObjectForLookup(CNContextImpl.java:2877) at com.ibm.ws.naming.jndicos.CNContextImpl.processResolveResults(CNContextImpl.java:3974) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1876) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1777) at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1434) at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:616) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:165) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179) at org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:161) at javax.naming.InitialContext.lookup(InitialContext.java:428) at com.deere.u90.iaf.jdbc.connection.ConnectionManager.initializeEnvironment(ConnectionManager.java:276) ... 38 more Caused by: javax.xml.stream.FactoryConfigurationError: Provider javax.xml.stream.XMLInputFactory could not be instantiated: java.util.ServiceConfigurationError: javax.xml.stream.XMLInputFactory: Provider com.sun.xml.internal.stream.XMLInputFactoryImpl not a subtype at javax.xml.stream.XMLInputFactory.newFactory(Unknown Source) at javax.xml.stream.XMLInputFactory.newInstance(Unknown Source) at com.ibm.websphere.product.metadata.im.IMMetadata.setHistoryEventsFromHistoryXml(IMMetadata.java:745) at com.ibm.websphere.product.metadata.im.IMMetadata.parseHistoryXmlFile(IMMetadata.java:587) at com.ibm.websphere.product.metadata.im.IMMetadata.parseInstallRegistryFiles(IMMetadata.java:399) at com.ibm.websphere.product.metadata.im.IMMetadata.(IMMetadata.java:269) at com.ibm.websphere.product.metadata.im.IMMetadata.getIMMetadataInstance(IMMetadata.java:133) at com.ibm.websphere.product.metadata.WASMetadata.parseMetadataFiles(WASMetadata.java:939) at com.ibm.websphere.product.metadata.WASMetadata.(WASMetadata.java:784) at com.ibm.websphere.product.metadata.WASMetadata.getWASMetadataInstance(WASMetadata.java:215) at com.ibm.websphere.product.WASDirectory.initMetadataInstance(WASDirectory.java:1415) at com.ibm.websphere.product.WASDirectory.getIMLogLocation(WASDirectory.java:435) at com.ibm.websphere.product.VersionInfo.printSource(VersionInfo.java:1534) at com.ibm.websphere.product.VersionInfo.printReport(VersionInfo.java:1322) at com.ibm.websphere.product.VersionInfo.runReport(VersionInfo.java:1064) at com.ibm.websphere.product.VersionInfo.runReport(VersionInfo.java:1025) at com.ibm.ws.rsadapter.spi.ServerFunction$7.run(ServerFunction.java:596) at com.ibm.ws.rsadapter.spi.ServerFunction$7.run(ServerFunction.java:590) at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118) at com.ibm.ws.rsadapter.spi.ServerFunction.getServerVersion(ServerFunction.java:588) at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.(WSManagedConnectionFactoryImpl.java:748) at java.lang.J9VMInternals.newInstanceImpl(Native Method) at java.lang.Class.newInstance(Class.java:1899) at com.ibm.ejs.j2c.J2CUtilityClass.createMCFEntry(J2CUtilityClass.java:468) at com.ibm.ejs.j2c.ConnectionFactoryBuilderServerImpl.createMCFandPM(ConnectionFactoryBuilderServerImpl.java:592) at com.ibm.ejs.j2c.ConnectionFactoryBuilderServerImpl.processObjectInstance(ConnectionFactoryBuilderServerImpl.java:1185) at com.ibm.ejs.j2c.ServerFunction.processObjectInstance(ServerFunction.java:2009) at com.ibm.ejs.j2c.ConnectionFactoryBuilderImpl.getObjectInstance(ConnectionFactoryBuilderImpl.java:662) at org.apache.aries.jndi.ObjectFactoryHelper.getObjectInstanceUsingObjectFactoryBuilders(ObjectFactoryHelper.java:349) at org.apache.aries.jndi.ObjectFactoryHelper.getObjectInstance(ObjectFactoryHelper.java:89) at org.apache.aries.jndi.OSGiObjectFactoryBuilder.getObjectInstance(OSGiObjectFactoryBuilder.java:62) at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:311) at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookupExt(Helpers.java:1122) ... 50 more
I was facing the same issue and here was my resolution
I had wrongly spelled my jta data source in IBM Admin Console and was using a different name in my persistence xml so Websphere couldn't connect.
Please check the Resources Tab in the Admin Console and validate your jta data source name as declared in persistence xml
I am trying to configure spring data with cassandra.
But I am getting bellow error , when my app is deploying in tomcat.
When I check the connection, it is available to the given port. (127.0.0.1:9042). I have include stack trace and spring configuration bellow.
Does anyone having idea on this error?
Full stack trace :
2015-12-06 17:46:25 ERROR web.context.ContextLoader:331 - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession': Invocation of init method failed; nested exception is com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:759)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:434)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4994)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5492)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1245)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1895)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230)
at com.datastax.driver.core.Cluster.init(Cluster.java:157)
at com.datastax.driver.core.Cluster.connect(Cluster.java:245)
at com.datastax.driver.core.Cluster.connect(Cluster.java:278)
at org.springframework.cassandra.config.CassandraCqlSessionFactoryBean.afterPropertiesSet(CassandraCqlSessionFactoryBean.java:82)
at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.afterPropertiesSet(CassandraSessionFactoryBean.java:43)
===================================================================
Spring Configuration :
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans ...>
<cassandra:cluster id="cassandraCluster"
contact-points="127.0.0.1" port="9042" />
<cassandra:converter />
<cassandra:session id="cassandraSession" cluster-ref="cassandraCluster"
keyspace-name="blood" />
<cassandra:template id="cqlTemplate" />
<cassandra:repositories base-package="com.blood.dao.nosql" />
<cassandra:mapping entity-base-packages="com.blood.domain.nosql" />
</beans:beans>
The problem is that Spring Data Cassandra (as of December 2015 when I write this) does not provide support for Cassandra 3.x. Here's an excerpt from a conversation with one of the developers in the #spring channel on freenode:
[13:49] <_amicable> Hi all, does anybody know if spring data cassandra supports cassandra 3.x? All dependencies & datastax drivers seem to be 2.x
[13:49] <#_ollie> amicable: Not in the near future.
[13:49] <_amicable> _ollie: thanks.
[13:50] <_amicable> I'll go and look at the relative merits of 2.x vs 3.x then
;)
[13:51] <#_ollie> SD Cassandra is a community project (so far) and its progress highly depends on how much time the developers can actually spend on it.
[13:51] <#_ollie> We will have someone joining the team in February 2016 to get the project more closely aligned to the core Spring Data projects.
It looks like you are using an older version of the driver with Cassandra 3.0. Cassandra 3.0 changed its internal schema metadata representation, and only the latest drivers can parse this metadata.
Use Java Cassandra driver 3.0.0-alpha5 to connect to Cassandra 3.0.
Yesterday we had a power outage and were able to get all of our machines back online with the exception of one box.
When firing up our application we see the log
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.levelsbeyond.search.elasticsearch.ElasticSearchTransportClientProvider]: Constructor threw exception; nested exception is org.elasticsearch.client.transport.NoNodeAvailableException: No node available (org.mule.api.lifecycle.InitialisationException)
at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:52)
at org.mule.config.builders.AbstractResourceConfigurationBuilder.configure(AbstractResourceConfigurationBuilder.java:78)
at org.mule.context.DefaultMuleContextFactory.createMuleContext(DefaultMuleContextFactory.java:97)
at org.mule.config.builders.MuleXmlBuilderContextListener.createMuleContext(MuleXmlBuilderContextListener.java:169)
at org.mule.config.builders.MuleXmlBuilderContextListener.initialize(MuleXmlBuilderContextListener.java:98)
at org.mule.config.builders.MuleXmlBuilderContextListener.contextInitialized(MuleXmlBuilderContextListener.java:74)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:983)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1660)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
and then consistently repeated we see
2014-09-19 11:35:19,200 WARN [org.elasticsearch.transport.netty] (elasticsearch[Dominic Fortune][transport_client_worker][T#5]{New I/O worker #5}) - <[Dominic Fortune] Message not fully read (response) for [12] handler future(org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler$1#625badaa), error [true], resetting>
Everything was working perfectly fine until the power outage. This is a single node in the cluster and it is running on the same machine as the java application (centos 6.5) so I know this isn't the same issue you keep finding on SO and on google that states this issue is caused by different version of elasticsearch and/or Java.
Does anyone no how to recover from this and get back up and running?
Thanks.
Turns out that when the power went out, a restart triggered an auto update of elasticsearch and the upgraded version didn't support the transport drivers in use.