I'm pretty new to Sonar, I'm using surefire for the metrics, and I've been just wondering how can I modify the Sonar overall coverage to increase or reduce this value to make it fail my Jenkins job.
Can someone point me in the right direction? I cannot find any documentation so far.
I'm not up to date what with the jenkins integration of sonar. Earlies you could install the build breaker plugin. Depending on your Sonarqube version this will no longer work.
Usually the thresholds for your project are not configured as parameters in the project but within sonarqube. Prior to 4.3 this was done through alerts. Nowadays it's called Quality Gates.
Actually a quite good question. I could not find anything for the maven plugin itself to set up some thresholds.
It seems to be an ongoing discussion.
Related
Sonar analysis is a nice way to check code compliance against a centrally defined policy. This is why I would like to use a profile with Blocker a Critical rules to signify a broken build.
I am using Jenkins do builds and invoke the SonarRunnerBuilder to get standard analysis on nightly builds of projects.
To achieve the features I outlined it would be best if I could run the analysis, check the results for violations (preferrably not sending them to Sonar server) and fail the build if there were any violations. So far I have not found way to do this using Google and looking through SonarRunner source code.
What I have considered is a workaround. I would implement a Decorator collecting violations and when the decoration reaches the project resource I would throw a RuntimeException to break the analysis. This would in turn fail the build.
would this work? Is there any better way to achieve this?
What you are looking for seems to be what the build breaker plugin offers. It relies on the Quality gate configuration to detect when violations (now called issues) in the current analysis require to break the whole build.
Please consider that this plugin won't be supported in SonarQube 5.2. Technically, deep changes in SonarQube's architecture make it impossible to provide the same feature. Philosophically, this plugin does not match the experience SonarQube wants to offer.
Still, another solution which covers the same use case is very likely to be offered in future version of SonarQube but this is yet to be defined.
We've recently set our Continuous Integration environment to do an incremental code analysis before a Merge Request on Git can be accepted. We already use the HTML issues report and the Build Breaker plugin. Everything works beautifully. The thing is, we're having some issues because the build is broken on < 85% code coverage. I didn't find any plugin to let the user know where the coverage dropped. Is there any way to show that comparison between the analisys that is recorded on my server and the one Jenkins is incrementally building ?
Thanks a lot!
Im using sonar4.5.4 and we use the Coverage on new code metric in our quality gate, in that metric we compare with the previous analysis
Does Sonar offer any way to raise alerts and fail a build when the trend for certain metrics is bad?
Background: In our legacy project using a static threshold for example for code coverage ("red alert when coverage is below 80%") does not make much sense. But we would like to make sure that the coverage does not go down any further.
Please do not give any advice on lowering the bar by using a less restrictive rule set. This is no option in our case.
There is a build breaker plug-in that will fail the build if you breach a Warning or Error threshold setup in the quality profile.
Plug-in details are here:
http://docs.sonarqube.org/display/PLUG/Build+Breaker+Plugin
Not aware of any functionality that enables you to a metric trend.
We use Sonar as the second last step in our release process. The build breaker ensures that releases do not breach predetermined quality criteria.
We tried exactly the same, using the build breaker plugin. After a while, it showed to be too unflexible (and configuring Sonar is a mess), so we moved from sonar to Jenkins/Hudson plugins like Cobertura (for code coverage) or PMD for code style:
https://wiki.jenkins-ci.org/display/JENKINS/PMD+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
With these plugins, very fine-granular settings are possible, to set for example the build to yellow at <70% code coverage or to red by <50%; even the weather-symbol for each build is setable.
In the meanwhile we scripted our own buildbreaker that gets excecuted within our build. We use Groovy to query the REST API of Sonar to retrieve a certain set of metrics (including their historical values). The retrieval of metrics is provided by a build plugin that is provided for our whole division.
Each team can parameterize their build with a set of rules regarding those metrics that have to be verified for their project. Of course, the rules are also provides as Groovy snippets :-)
Typical are:
Number of (major|critical|blocker) violations is less or equal than in previous build
No new duplicates
Coverage not lower than in previous build
Bad findings can then be used for breaking the build or just for reporting.
I've seen a number of posts on making a Maven-backed Jenkins build fail for a given project if a coverage threshold isn't met i.e. coverage must be at least 80% or the build fails.
I'm wondering if there is a way to configure Jenkins to fail a build if the coverage is lower than the last build i.e. if the coverage for build N is 20%, and N+1 is 19%, then the build fails. I don't want to put in an explicit threshold, but I want the coverage to stay steady or get higher over time.
I haven't tried it, but assuming you are using maven cobertura plugin, I believe it can be configured to fail as documented here. Would jenkins not honour the failure?
I also see an open feature request for this.
Save last coverage percentage on the side. Use Groovy script (run via Groovy Plugin) that taps into Jenkins Java API to compare current coverage level with previous one and fail the build if it drops. If you seriously decide to go this way I can provide more details and code samples.
Since the 1.6 version of Cobertura plugin, the issue has been fixed, and a new option appeared that allow the plugin to make the build failed if coverage is lower than previous build.
This should be configurable in the configuration section of the plugin, that calculates the coverage. E.g. for the PMD plugin you can configure limits for unstable or failed build on the Job configuration page in the section "Build settings".
In NCover I specify a certain percent through a config file. You could create a python script (or similar to alter that value if you wanted). Other than that I do not think there are any config variables
We are working on a web project from scratch and are looking at the following static code analysis tools.
Conventions (Checkstyle)
Bad practices (PMD)
Potential bugs (FindBugs)
The project is built on Maven. Instead of using multiple tools for the purpose, I was looking at a single flexible solution and came across SonarQube.
Is it true that we can achieve the results from Checkstyle, PMD and Findbugs with SonarQube?
Sonar will run CheckStyle, FindBugs and PMD, as well as a few other "plugins" such as Cobertura (code coverage) by default for Java projects. The main added value, however, is that it stores the history in a database. You can then see the trend. Are you improving the code base or are you doing the opposite? Only a tool with memory can tell you that.
You should run Sonar in your CI system so that even things that take some time to execute (such as CPD – copy paste detector) can run. And you'll have your history. Whereas with an Eclipse plugin, for example, you'll detect violations sooner – which is great – but you will be tempted to run it less often if it starts taking too long, or run less "quality plugins" (such as skipping CPD or skipping code coverage analysis). And you won't have history.
Also, Sonar generates visual reports, "Dashboard" style. Which makes it very easy to grasp. With Sonar in Jenkins, you'll be able to show developers and your management the effects of the work that was performed on the quality of the code base over the last few weeks and months.
Sonar uses these 3 tools as plugins and aggregates the data from all three giving addition value by showing graphs and such from these tools. So they are complementary to sonar.
Yes and no. In addition to the other answers.
SonarQube is currently on the way to deprecate PMD, Checkstyle and Findbugs and use their own technology to analyze Java code (called SonarJava). They do it, because they don't want to spend their time fixing, upgrading (or waiting on it) those libraries (e.g. for Java 8), which for example uses outdated libraries.
They also got a new set of plugins for your personal IDE called SonarLint.
Sonar is great, but if you want to use the mentioned tools separately and still have nice graphs, you can use the Analysis Collector Plugin as part of your Jenkins CI build. A slight advantage of this is that you can check in your PMD/Findbugs/Checkstyle configuration into your SCM and have it integrated into your Maven build, rather than relying on a separate Sonar server.
... a few years later: no, it is not! SonarQube supposes to be able to cover all the rules with its own analyzer, but there are still rules from PMD or CheckStyle not covered by SonarQube. See for example: PMD ReturnFromFinallyBlock.
Sonar is much more than these tools alone.
The greatest benefits is the gui, which lets you configure anything easily.
The statistics it offers are very detailed (lines of code etc).
And it even offers great support for test coverage etc :)
Here you can take a good look:
http://nemo.sonarsource.org/
I would still use these tools in addition to sonar because they can fail the maven build when someone violates a rule. Where as sonar is more retrospective.
Well at least since SonarQube 6.3+ it seems to be that Findbugs is (at the moment) no longer supported as a plugin. Sonarsource is working on replacements of Findbugs-rules with its own Java-plugin.
They even had a list for the replacement status of each rule here, but it got removed by now.
See https://community.sonarsource.com/t/where-is-dist-sonarsource-com-content/5353 for more details.