elastic search shutting down throws TransportNodesShutdownAction$1$2 not found - elasticsearch

I'm using elastic search 1.1.0.
While trying to shutdown the elastic search I'm getting NoClassDefFoundError. Code used to shutdown and stack trace is given below.
Maven:
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.1.0</version>
</dependency>
Code: node.client().admin().cluster().prepareNodesShutdown().execute().get();
2015-08-05 21:45:11,113 [Thread-15] INFO org.elasticsearch.action.admin.cluster.node.shutdown internalInfo - [Captain Ultra] [cluster_shutdown]: done shutting down all nodes except master, proceeding to master
INFO: Illegal access: this web application instance has been stopped already. Could not load org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1$2. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1.run(TransportNodesShutdownAction.java:153)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Thread-15" java.lang.NoClassDefFoundError: org/elasticsearch/action/admin/cluster/node/shutdown/TransportNodesShutdownAction$1$2
at org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1.run(TransportNodesShutdownAction.java:153)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1$2
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
... 2 more
This issue is not allowing elastic search to shutdown gracefully. Tomcat is waiting for certain period(timeout) and then killing all those threads forcefully.
How to fix this issue?
Same issue is coming with 1.0.0 also.

Finally I got the solution. Elastic search is trying to shutdown in separate thread. By the time elastic search tries to access TransportNodesShutdownAction, tomcat will unload elastic jar from its class loader. So adding some sleep statement or adding elastic and its dependent jar to tomcat/lib will solve the problem.

Related

Sonarqube analysis failing for huge code base - None of the configured nodes were available

Sonarqube analysis completes, but the report processing fails for project with huge code base. Issue is showing up for project with 220K lines of code.
Sonarqube version - 7.9.1 - Running on Kubernetes
I tried deleting the data/es folder for fresh indices. Still no luck.
Error Log
Caused by: java.lang.IllegalStateException: Fail to execute ES refresh request on indices 'components'
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:44)
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:32)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:120)
at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:165)
at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:107)
at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:71)
at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:76)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.access$000(PurgeDatastoresStep.java:38)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep$1.visitProject(PurgeDatastoresStep.java:63)
at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51)
at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
... 18 common frames omitted
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [{sonarqube}{}{}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}]
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:294)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:533)
at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:48)
at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:60)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:253)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1262)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:42)
... 31 common frames omitted
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [sonarqube][127.0.0.1:9001] Node not connected
at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151)
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:557)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:529)
I got over this issue by increasing the memory limits for the SonarQube Kubernetes pod. Thanks to #Przemek Nowak answer on this question.

Getting "Caused by: java.lang.IllegalStateException: HTTP Server cannot be loaded: No implementation of HttpServerFacade found on classpath." error

I am trying out the distributed random forest implementation of H2O using sparkling-water. But I am facing the following error when I run the spark-submit command.
Exception in thread "H2O Launcher thread" java.lang.ExceptionInInitializerError
at water.init.NetworkInit.initializeNetworkSockets(NetworkInit.java:77)
at water.H2O.startLocalNode(H2O.java:1621)
at water.H2O.main(H2O.java:2081)
at water.H2OStarter.start(H2OStarter.java:22)
at water.H2OStarter.start(H2OStarter.java:47)
at org.apache.spark.h2o.backends.internal.InternalBackendUtils$$anonfun$6$$anon$1.run(InternalBackendUtils.scala:173)
Caused by: java.lang.IllegalStateException: HTTP Server cannot be loaded: No implementation of HttpServerFacade found on classpath. Please refer to https://0xdata.atlassian.net/browse/TN-13 for details.
at water.webserver.iface.HttpServerLoader.<clinit>(HttpServerLoader.java:16)
... 6 more
I have tried out the solution mentioned at the location https://0xdata.atlassian.net/browse/TN-13
but for some reason it still isn't able to find the ai.h2o:h2o-jetty-8 on the classpath.
I resolved the issue by adding the following maven coordinates to the packages option in spark-submit
--packages ai.h2o:h2o-jetty-8:3.24.0.3

Unable to start neo4j afer copying elasticsearch jar files to plugin folder

I am getting below error in neo4j console.log after copying jar files from dependency folder of elasticsearch to plugin folder of neo4j
2015-12-11 08:08:21.697+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attach
ed cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attached cause exceptio
n.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.extension.KernelExtensions#28d2afd8' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: org.neo4j.kernel.impl.util.UnsatisfiedDependencyException: java.lang.IllegalArgumentException: No dependency satisfies type class org.neo4j.kernel.impl.util.StringLogger
at org.neo4j.kernel.impl.util.DependenciesProxy$ProxyHandler.invoke(DependenciesProxy.java:79)
at com.sun.proxy.$Proxy6.getStringLogger(Unknown Source)
at org.neo4j.elasticsearch.ElasticSearchKernelExtensionFactory.newKernelExtension(ElasticSearchKernelExtensionFactory.java:39)
at org.neo4j.elasticsearch.ElasticSearchKernelExtensionFactory.newKernelExtension(ElasticSearchKernelExtensionFactory.java:20)
Your problem could be because Neo4j and ElasticSearch using different version of Lucene.
In JVM you can't have one jar with two different versions.

RuntimeMBeanCallable.call() exception

When a .wlapp file is deployed in MobileFirst Console, we get the following error in the logs. This happened after we upgraded MobileFirst Server from 6.3 to 7, but don't know if this is related to the version of MobileFirst.
000001c6 BaseTransacti E RuntimeMBeanCallable.call() exception
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy166.deployApplication(Unknown Source)
at com.ibm.worklight.admin.actions.ApplicationDeploymentTransaction.prepareMBean(ApplicationDeploymentTransaction.java:919)
at com.ibm.worklight.admin.actions.util.RuntimeMBeanWorkerThreadCaller$RuntimeMBeanCallable.call(RuntimeMBeanWorkerThreadCaller.java:76)
at com.ibm.worklight.admin.actions.util.RuntimeMBeanWorkerThreadCaller.callSynchronously(RuntimeMBeanWorkerThreadCaller.java:183)
at com.ibm.worklight.admin.actions.util.RuntimeMBeanPoolCaller.callRuntimeMBeans(RuntimeMBeanPoolCaller.java:91)
at com.ibm.worklight.admin.actions.BaseTransaction.prepare(BaseTransaction.java:450)
at com.ibm.worklight.admin.actions.BaseTransaction.internalRun(BaseTransaction.java:348)
at com.ibm.worklight.admin.actions.BaseTransaction$1.run(BaseTransaction.java:235)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)
Caused by: com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:894)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:689)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:679)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:665)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:487)
at com.sun.proxy.$Proxy121.invoke(Unknown Source)
at com.ibm.ws.management.AdminClientImpl.invoke(AdminClientImpl.java:224)
at com.worklight.common.util.jmx.WASRuntimeMBeanHandler$AdminClientMBeanServerConnection.invoke(WASRuntimeMBeanHandler.java:521)
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:146)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:160)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:246)
... 11 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out; targetException=java.net.SocketTimeoutException: Read timed out]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:479)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:510)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$8.run(SOAPConnectorClient.java:852)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:845)
... 21 more
MobileFirst version 7
WAS server version 8.5.5.5
JRE version 1.6
Re-deploying the MobileFirst artifacts to the application server should normally not be required as inspecting the logs and configuration usually will lead to the root cause of the problem. However in some cases starting over to get things right will work.
As mentioned in the comments, re-deployment of the server artifacts nullified the issue that was experienced.

Tomcat webpage not coming up

My tomcat default web page does not come up, please could someone tell me the reason following i have pasted the last 100 lines for catalina log
Exception in thread "Timer-0" java.lang.NoClassDefFoundError: org/apache/commons/pool/impl/CursorableLinkedList$Cursor
at org.apache.commons.pool.impl.CursorableLinkedList.cursor(CursorableLinkedList.java:305)
at org.apache.commons.pool.impl.GenericObjectPool.evict(GenericObjectPool.java:1488)
at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1700)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.pool.impl.CursorableLinkedList$Cursor
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
... 5 more
The stack trace indicates that Tomcat cannot find a class from the Apache Commons Pool library. Try adding the commons-pool jar under the WEB-INF/lib directory of your webapp.

Resources