SonarQube consistent build errors when using SonarQube Scanner for MSBuild - sonarqube

Question

I recently set up a TFS 2017 build definition to scan and analyse a few projects. One is a msvc solution with 2 configurations, the other is a single solution/configuration.
This gets ran at around 2:30am, as the analysis takes a long time.
Recently I have been getting the following error a lot from the begin command:
2017-11-28T03:24:19.3357159Z 03:24:19.293 Downloading from http://XX.XX.XX.XX:1234/api/rules/search?f=repo,name,severity,lang,internalKey,templateKey,params,actives&ps=500&activation=true&qprofile=cs-sonar-way-89107&p=1...
2017-11-28T03:24:20.4458969Z ##[error]03:24:20.406 Failed to request and parse 'http://XX.XX.XX.XX:1234/api/rules/search?f=repo,name,severity,lang,internalKey,templateKey,params,actives&ps=500&activation=true&qprofile=cs-sonar-way-89107&p=1': The remote server returned an error: (500) Internal Server Error.
2017-11-28T03:24:20.4468972Z ##[error]
2017-11-28T03:24:20.7509185Z [error]Unhandled Exception: System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
2017-11-28T03:24:20.7519228Z ##[error]at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
2017-11-28T03:24:20.7529181Z ##[error]at System.Net.WebClient.DownloadString(Uri address)
2017-11-28T03:24:20.7539201Z ##[error]at System.Net.WebClient.DownloadString(String address)
2017-11-28T03:24:20.7539201Z ##[error]at SonarQube.TeamBuild.PreProcessor.WebClientDownloader.Download(String url)
2017-11-28T03:24:20.7549145Z ##[error]at SonarQube.TeamBuild.PreProcessor.SonarWebService.<>c__DisplayClass8_0.b__0()
2017-11-28T03:24:20.7559193Z ##[error]at SonarQube.TeamBuild.PreProcessor.SonarWebService.DoLogExceptions[T](Func1 op, String url)
2017-11-28T03:24:20.7569201Z ##[error]at SonarQube.TeamBuild.PreProcessor.SonarWebService.GetActiveRules(String qprofile)
2017-11-28T03:24:20.7579169Z ##[error]at SonarQube.TeamBuild.PreProcessor.TeamBuildPreProcessor.FetchArgumentsAndRulesets(ISonarQubeServer server, ProcessedArgs args, TeamBuildSettings settings, IDictionary2& serverSettings, List`1& analyzersSettings)
2017-11-28T03:24:20.7589220Z ##[error]at SonarQube.TeamBuild.PreProcessor.TeamBuildPreProcessor.DoExecute(ProcessedArgs localSettings)
2017-11-28T03:24:20.7589220Z ##[error]at SonarQube.TeamBuild.PreProcessor.TeamBuildPreProcessor.Execute(String[] args)
2017-11-28T03:24:20.7599212Z ##[error]at SonarQube.Bootstrapper.BootstrapperClass.PreProcess()
2017-11-28T03:24:20.7609185Z ##[error]at SonarQube.Bootstrapper.BootstrapperClass.Execute()
2017-11-28T03:24:20.7619216Z ##[error]at SonarQube.Bootstrapper.Program.Execute(String[] args, ILogger logger)
2017-11-28T03:24:20.7619216Z ##[error]at SonarQube.Bootstrapper.Program.Main(String[] args)
2017-11-28T03:24:20.7629220Z ##[error]at SonarQube.Old.Bootstrapper.Program.Main(String[] args)
2017-11-28T03:24:21.6009841Z ##[error]System.Exception: Unexpected exit code received from batch file: 255
2017-11-28T03:24:21.6019955Z [error] at Microsoft.TeamFoundation.DistributedTask.Task.Internal.PowerShell.InvokeBatchScriptCmdlet.ProcessRecord()
2017-11-28T03:24:21.6029912Z ##[error] at System.Management.Automation.CommandProcessor.ProcessRecord()
This seems fairly consistent, although last night I received an error on the server analysis instead, which then failed to complete:
java.lang.IllegalStateException: Fail to execute ES refresh request on indices 'issues'
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:44)
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:32)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:100)
at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:123)
at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:113)
at org.sonar.server.issue.index.IssueIndexer.indexProject(IssueIndexer.java:86)
at org.sonar.server.computation.task.projectanalysis.step.IndexAnalysisStep.execute(IndexAnalysisStep.java:45)
at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:92)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:59)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{127.0.0.1:9001}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:326)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:223)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:295)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:42)
... 20 more
I'm not sure if these are related, but hoping someone can provide some context to either or both of these issues.
Server info:
SonarQube scanner for MSBuild version - 3.0.2
SonarQube version - 6.4.0.25310
TFS Build Definition
Has anyone seen these issue(s) before? If there is any more information that can help, I'd be happy to provide it.
Edit:
Unfortunately not solved this yet, as it happens so sporadically. I found this error in my web logs on the server:
2017.12.12 06:42:05 INFO web[][o.e.client.transport] [Invisible Girl] failed to get node info for {#transport#-1}{127.0.0.1}{127.0.0.1:9001}, disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][127.0.0.1:9001][cluster:monitor/nodes/liveness] request_id [220282] timed out after [5009ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017.12.12 06:42:11 ERROR web[AWAIdx6r18FZDcBpACYf][o.s.s.w.WebServiceEngine] Fail to process request http://10.0.170.20:9000/api/rules/search?f=repo,name,severity,lang,internalKey,templateKey,params,actives&ps=500&activation=true&qprofile=cs-sonar-way-89107&p=1
java.lang.IllegalStateException: Fail to execute ES search request '{"from":0,"size":500,"query":{"bool":{"must":{"match_all":{}},"filter":{"bool":{"must":[{"terms":{"lang":["cs"]}},{"has_child":{"query":{"bool":{"must":[{"term":{"profile":"cs-sonar-way-89107"}},{"term":{"organizationUuid":"AVyM3QhnpAs1dsLppIwH"}}]}},"child_type":"activeRule"}},{"bool":{"must_not":{"term":{"status":"REMOVED"}}}}]}}}},"sort":[{"updatedAt":{"order":"desc"}},{"key.sortable_analyzer":{"order":"asc"}}]}' on indices '[rules]' on types '[rule]'
at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:48)
at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:36)
at org.sonar.server.rule.index.RuleIndex.search(RuleIndex.java:151)
at org.sonar.server.rule.ws.SearchAction.doSearch(SearchAction.java:344)
at org.sonar.server.rule.ws.SearchAction.handle(SearchAction.java:161)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:103)
at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:86)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:95)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:76)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:72)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.sonar.server.platform.web.requestid.RequestIdFilter.doFilter(RequestIdFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{127.0.0.1:9001}]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:326)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:223)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:295)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:46)
... 46 common frames omitted

Answer 1

Your build is failing because the SonarQube server is returning 500's
The remote server returned an error: (500) Internal Server Error
The error you found on the server side may or may not be related. You'll want to check all the files in the $SONARQUBE_HOME/logs directory (other than access.log) and clear up whatever errors you find there.