Sonarqube analysis failing for huge code base - None of the configured nodes were available - elasticsearch

Sonarqube analysis completes, but the report processing fails for project with huge code base. Issue is showing up for project with 220K lines of code.
Sonarqube version - 7.9.1 - Running on Kubernetes
I tried deleting the data/es folder for fresh indices. Still no luck.
Error Log
Caused by: java.lang.IllegalStateException: Fail to execute ES refresh request on indices 'components'
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:44)
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:32)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:120)
at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:165)
at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:107)
at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:71)
at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:76)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.access$000(PurgeDatastoresStep.java:38)
at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep$1.visitProject(PurgeDatastoresStep.java:63)
at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51)
at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
... 18 common frames omitted
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [{sonarqube}{}{}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}]
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:294)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:533)
at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:48)
at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:60)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:253)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1262)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:42)
... 31 common frames omitted
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [sonarqube][127.0.0.1:9001] Node not connected
at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151)
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:557)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:529)

I got over this issue by increasing the memory limits for the SonarQube Kubernetes pod. Thanks to #Przemek Nowak answer on this question.

Related

Fail to process request http://localhost:9000/api/ce/submit?projectKey

I am facing the issue with SQ scan of few of my projects.
2017.11.20 20:26:36 ERROR web[AV/cUNmR+tiFEjO/AAAg][o.s.s.w.WebServiceEngine] Fail to process request http://localhost:9000/api/ce/submit?projectKey=com.org.app:app&projectName=ReactorProject
java.lang.IllegalStateException: Can't read file part at org.sonar.server.ws.ServletRequest.readPart(ServletRequest.java:102) at org.sonar.server.ws.ServletRequest.readInputStreamParam(ServletRequest.java:85) at org.sonar.api.server.ws.internal.ValidatingRequest.paramAsInputStream(ValidatingRequest.java:83)
at org.sonar.server.ce.ws.SubmitAction.handle(SubmitAction.java:10
.......................
Caused by: java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: Unexpected EOF read on the socket
at org.apache.catalina.connector.Request.parseParts(Request.java:2919)
at org.apache.catalina.connector.Request.parseParameters(Request.java:3215)
at org.apache.catalina.connector.Request.getParameter(Request.java:1145)
Caused by: org.apache.tomcat.util.http.fileupload.FileUploadException: Unexpected EOF read on the socket
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:308)
at org.apache.catalina.connector.Request.parseParts(Request.java:2871)
... 52 common frames omitted
Caused by: java.io.EOFException: Unexpected EOF read on the socket
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:718)
at org.apache.coyote.http11.Http11InputBuffer.access$300(Http11InputBuffer.java:40)
at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1063)
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:140)
at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:257)
at org.apache.coyote.Request.doRead(Request.java:574)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326)
at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:349)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:183)
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:881)
at java.io.InputStream.read(Unknown Source)
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:98)
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:68)
at org.apache.tomcat.util.http.fileupload.MultipartStream.readBodyData(MultipartStream.java:571)
at org.apache.tomcat.util.http.fileupload.MultipartStream.discardBodyData(MultipartStream.java:595)
at org.apache.tomcat.util.http.fileupload.MultipartStream.skipPreamble(MultipartStream.java:613)
at org.apache.tomcat.util.http.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:874)
at org.apache.tomcat.util.http.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:854)
at org.apache.tomcat.util.http.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:256)
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:280)
... 53 common frames omitted
I am using the embedded H2 database.
Sonar version is 6.5 and installed on Windows 7.
The file upload completes successfully when the compressed content is < 50 KB in size (I am guessing, based on how it has worked out for me).
This always fails for me:
[INFO] Analysis report generated in 1022ms, dir size=384 KB
[INFO] Analysis reports compressed in 3019ms, zip size=190 KB
Anything beyond that fails with the above exception.
Are you aware of any setting that limits the size of the file being uploaded to SQ server?
Any help will be high appreciated.

Sonarqube Background Task (too many open files)

We have a large project (~35,000 java files) that currently has about 10,000 issues. About 3 weeks ago, our nightly scan started failing on version sonarqube 6.5. I upgraded to 6.6 and found the same problem. First, it was failing due to heap space. I have upgraded this machine to give it more memory and that seems to have allowed the analysis to finish. Now the scan fails every night during the background task to post the analysis due to "Too many open files". We have upped the limits for open files for the sonar user but that doesn't seem to have any effect. We have other smaller projects and they all finish easily. It's just this very large project that constantly fails. Has anyone seen this before?
This morning I installed sonarqube 6.7 to see if that fixes it. I'm currently running an analysis but it takes about 3 hours to finish and fail.
We increased the number of open files allowed for the sonar user.
-sh-4.1$ whoami
sonar
-sh-4.1$ ulimit -Hs
unlimited
-sh-4.1$ ulimit -Hn
1048576
Here is the error we are seeing
org.sonar.server.computation.task.projectanalysis.component.VisitException: Visit of Component {key=applications:sonar:src/main/java/com/MyFileName.java,type=FILE} failed
at org.sonar.server.computation.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:74)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
at org.sonar.server.computation.task.projectanalysis.step.ExecuteVisitorsStep.execute(ExecuteVisitorsStep.java:51)
at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:92)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:59)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Fail to process issues of component 'applications:sonar:src/main/java/com/MyFileName.java'
at org.sonar.server.computation.task.projectanalysis.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:83)
at org.sonar.server.computation.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:63)
at org.sonar.server.computation.task.projectanalysis.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:82)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:117)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:100)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
... 21 more
Caused by: java.lang.IllegalStateException: Fail to traverse file: /opt/sonarqube-6.5/temp/ce/6622607282221286408/1421069522563365386/source-11991.txt
at org.sonar.server.computation.task.projectanalysis.batch.BatchReportReaderImpl.readFileSource(BatchReportReaderImpl.java:153)
at org.sonar.server.computation.task.projectanalysis.source.SourceLinesRepositoryImpl.readLines(SourceLinesRepositoryImpl.java:45)
at org.sonar.server.computation.task.projectanalysis.issue.TrackerRawInputFactory$RawLazyInput.loadLineHashSequence(TrackerRawInputFactory.java:80)
at org.sonar.core.issue.tracking.LazyInput.getLineHashSequence(LazyInput.java:34)
at org.sonar.server.computation.task.projectanalysis.issue.TrackerRawInputFactory$RawLazyInput.loadIssues(TrackerRawInputFactory.java:105)
at org.sonar.core.issue.tracking.LazyInput.getIssues(LazyInput.java:50)
at org.sonar.core.issue.tracking.Tracking.<init>(Tracking.java:46)
at org.sonar.core.issue.tracking.Tracker.track(Tracker.java:37)
at org.sonar.server.computation.task.projectanalysis.issue.TrackerExecution.track(TrackerExecution.java:41)
at org.sonar.server.computation.task.projectanalysis.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:76)
... 26 more
Caused by: java.io.FileNotFoundException: /opt/sonarqube-6.5/temp/ce/6622607282221286408/1421069522563365386/source-11991.txt (Too many open files)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:301)
at org.sonar.server.computation.task.projectanalysis.batch.BatchReportReaderImpl.readFileSource(BatchReportReaderImpl.java:151)
... 35 more
Upgrading to SonarQube 6.7 fixed this error with too many open files.

Unable to start neo4j afer copying elasticsearch jar files to plugin folder

I am getting below error in neo4j console.log after copying jar files from dependency folder of elasticsearch to plugin folder of neo4j
2015-12-11 08:08:21.697+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attach
ed cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attached cause exceptio
n.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#623dc0b8' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.extension.KernelExtensions#28d2afd8' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: org.neo4j.kernel.impl.util.UnsatisfiedDependencyException: java.lang.IllegalArgumentException: No dependency satisfies type class org.neo4j.kernel.impl.util.StringLogger
at org.neo4j.kernel.impl.util.DependenciesProxy$ProxyHandler.invoke(DependenciesProxy.java:79)
at com.sun.proxy.$Proxy6.getStringLogger(Unknown Source)
at org.neo4j.elasticsearch.ElasticSearchKernelExtensionFactory.newKernelExtension(ElasticSearchKernelExtensionFactory.java:39)
at org.neo4j.elasticsearch.ElasticSearchKernelExtensionFactory.newKernelExtension(ElasticSearchKernelExtensionFactory.java:20)
Your problem could be because Neo4j and ElasticSearch using different version of Lucene.
In JVM you can't have one jar with two different versions.

elastic search shutting down throws TransportNodesShutdownAction$1$2 not found

I'm using elastic search 1.1.0.
While trying to shutdown the elastic search I'm getting NoClassDefFoundError. Code used to shutdown and stack trace is given below.
Maven:
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.1.0</version>
</dependency>
Code: node.client().admin().cluster().prepareNodesShutdown().execute().get();
2015-08-05 21:45:11,113 [Thread-15] INFO org.elasticsearch.action.admin.cluster.node.shutdown internalInfo - [Captain Ultra] [cluster_shutdown]: done shutting down all nodes except master, proceeding to master
INFO: Illegal access: this web application instance has been stopped already. Could not load org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1$2. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1.run(TransportNodesShutdownAction.java:153)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Thread-15" java.lang.NoClassDefFoundError: org/elasticsearch/action/admin/cluster/node/shutdown/TransportNodesShutdownAction$1$2
at org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1.run(TransportNodesShutdownAction.java:153)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$1$2
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
... 2 more
This issue is not allowing elastic search to shutdown gracefully. Tomcat is waiting for certain period(timeout) and then killing all those threads forcefully.
How to fix this issue?
Same issue is coming with 1.0.0 also.
Finally I got the solution. Elastic search is trying to shutdown in separate thread. By the time elastic search tries to access TransportNodesShutdownAction, tomcat will unload elastic jar from its class loader. So adding some sleep statement or adding elastic and its dependent jar to tomcat/lib will solve the problem.

RuntimeMBeanCallable.call() exception

When a .wlapp file is deployed in MobileFirst Console, we get the following error in the logs. This happened after we upgraded MobileFirst Server from 6.3 to 7, but don't know if this is related to the version of MobileFirst.
000001c6 BaseTransacti E RuntimeMBeanCallable.call() exception
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy166.deployApplication(Unknown Source)
at com.ibm.worklight.admin.actions.ApplicationDeploymentTransaction.prepareMBean(ApplicationDeploymentTransaction.java:919)
at com.ibm.worklight.admin.actions.util.RuntimeMBeanWorkerThreadCaller$RuntimeMBeanCallable.call(RuntimeMBeanWorkerThreadCaller.java:76)
at com.ibm.worklight.admin.actions.util.RuntimeMBeanWorkerThreadCaller.callSynchronously(RuntimeMBeanWorkerThreadCaller.java:183)
at com.ibm.worklight.admin.actions.util.RuntimeMBeanPoolCaller.callRuntimeMBeans(RuntimeMBeanPoolCaller.java:91)
at com.ibm.worklight.admin.actions.BaseTransaction.prepare(BaseTransaction.java:450)
at com.ibm.worklight.admin.actions.BaseTransaction.internalRun(BaseTransaction.java:348)
at com.ibm.worklight.admin.actions.BaseTransaction$1.run(BaseTransaction.java:235)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)
Caused by: com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:894)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:689)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:679)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:665)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:487)
at com.sun.proxy.$Proxy121.invoke(Unknown Source)
at com.ibm.ws.management.AdminClientImpl.invoke(AdminClientImpl.java:224)
at com.worklight.common.util.jmx.WASRuntimeMBeanHandler$AdminClientMBeanServerConnection.invoke(WASRuntimeMBeanHandler.java:521)
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:146)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:160)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:246)
... 11 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out; targetException=java.net.SocketTimeoutException: Read timed out]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:479)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:510)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$8.run(SOAPConnectorClient.java:852)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:845)
... 21 more
MobileFirst version 7
WAS server version 8.5.5.5
JRE version 1.6
Re-deploying the MobileFirst artifacts to the application server should normally not be required as inspecting the logs and configuration usually will lead to the root cause of the problem. However in some cases starting over to get things right will work.
As mentioned in the comments, re-deployment of the server artifacts nullified the issue that was experienced.

Resources