Sonarqube Background Task (too many open files) - sonarqube

Question

We have a large project (~35,000 java files) that currently has about 10,000 issues. About 3 weeks ago, our nightly scan started failing on version sonarqube 6.5. I upgraded to 6.6 and found the same problem. First, it was failing due to heap space. I have upgraded this machine to give it more memory and that seems to have allowed the analysis to finish. Now the scan fails every night during the background task to post the analysis due to "Too many open files". We have upped the limits for open files for the sonar user but that doesn't seem to have any effect. We have other smaller projects and they all finish easily. It's just this very large project that constantly fails. Has anyone seen this before?
This morning I installed sonarqube 6.7 to see if that fixes it. I'm currently running an analysis but it takes about 3 hours to finish and fail.
We increased the number of open files allowed for the sonar user.
-sh-4.1$ whoami
sonar
-sh-4.1$ ulimit -Hs
unlimited
-sh-4.1$ ulimit -Hn
1048576
Here is the error we are seeing
org.sonar.server.computation.task.projectanalysis.component.VisitException: Visit of Component {key=applications:sonar:src/main/java/com/MyFileName.java,type=FILE} failed
at org.sonar.server.computation.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:74)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:110)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:97)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
at org.sonar.server.computation.task.projectanalysis.step.ExecuteVisitorsStep.execute(ExecuteVisitorsStep.java:51)
at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:92)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:59)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Fail to process issues of component 'applications:sonar:src/main/java/com/MyFileName.java'
at org.sonar.server.computation.task.projectanalysis.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:83)
at org.sonar.server.computation.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:63)
at org.sonar.server.computation.task.projectanalysis.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:82)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:117)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:100)
at org.sonar.server.computation.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
... 21 more
Caused by: java.lang.IllegalStateException: Fail to traverse file: /opt/sonarqube-6.5/temp/ce/6622607282221286408/1421069522563365386/source-11991.txt
at org.sonar.server.computation.task.projectanalysis.batch.BatchReportReaderImpl.readFileSource(BatchReportReaderImpl.java:153)
at org.sonar.server.computation.task.projectanalysis.source.SourceLinesRepositoryImpl.readLines(SourceLinesRepositoryImpl.java:45)
at org.sonar.server.computation.task.projectanalysis.issue.TrackerRawInputFactory$RawLazyInput.loadLineHashSequence(TrackerRawInputFactory.java:80)
at org.sonar.core.issue.tracking.LazyInput.getLineHashSequence(LazyInput.java:34)
at org.sonar.server.computation.task.projectanalysis.issue.TrackerRawInputFactory$RawLazyInput.loadIssues(TrackerRawInputFactory.java:105)
at org.sonar.core.issue.tracking.LazyInput.getIssues(LazyInput.java:50)
at org.sonar.core.issue.tracking.Tracking.<init>(Tracking.java:46)
at org.sonar.core.issue.tracking.Tracker.track(Tracker.java:37)
at org.sonar.server.computation.task.projectanalysis.issue.TrackerExecution.track(TrackerExecution.java:41)
at org.sonar.server.computation.task.projectanalysis.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:76)
... 26 more
Caused by: java.io.FileNotFoundException: /opt/sonarqube-6.5/temp/ce/6622607282221286408/1421069522563365386/source-11991.txt (Too many open files)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:301)
at org.sonar.server.computation.task.projectanalysis.batch.BatchReportReaderImpl.readFileSource(BatchReportReaderImpl.java:151)
... 35 more

Answer 1

Upgrading to SonarQube 6.7 fixed this error with too many open files.