Does Sonar offer any way to raise alerts and fail a build when the trend for certain metrics is bad?
Background: In our legacy project using a static threshold for example for code coverage ("red alert when coverage is below 80%") does not make much sense. But we would like to make sure that the coverage does not go down any further.
Please do not give any advice on lowering the bar by using a less restrictive rule set. This is no option in our case.
There is a build breaker plug-in that will fail the build if you breach a Warning or Error threshold setup in the quality profile.
Plug-in details are here:
http://docs.sonarqube.org/display/PLUG/Build+Breaker+Plugin
Not aware of any functionality that enables you to a metric trend.
We use Sonar as the second last step in our release process. The build breaker ensures that releases do not breach predetermined quality criteria.
We tried exactly the same, using the build breaker plugin. After a while, it showed to be too unflexible (and configuring Sonar is a mess), so we moved from sonar to Jenkins/Hudson plugins like Cobertura (for code coverage) or PMD for code style:
https://wiki.jenkins-ci.org/display/JENKINS/PMD+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
With these plugins, very fine-granular settings are possible, to set for example the build to yellow at <70% code coverage or to red by <50%; even the weather-symbol for each build is setable.
In the meanwhile we scripted our own buildbreaker that gets excecuted within our build. We use Groovy to query the REST API of Sonar to retrieve a certain set of metrics (including their historical values). The retrieval of metrics is provided by a build plugin that is provided for our whole division.
Each team can parameterize their build with a set of rules regarding those metrics that have to be verified for their project. Of course, the rules are also provides as Groovy snippets :-)
Typical are:
Number of (major|critical|blocker) violations is less or equal than in previous build
No new duplicates
Coverage not lower than in previous build
Bad findings can then be used for breaking the build or just for reporting.
Related
It has been cleared here that from Sonarqube version 6.2 that coverage reports are merged and there won't be separate unit and integration coverage report anymore.
We still interested to have these two coverage reports separately. So, We have three sonar projects: unit-tests, integration-tests, whole-project(which is responsible to create overall coverage report)
Problem: All source files are analysed in all three projects. Since the number of files are too many, it takes several minutes to perform the analysis.
Question: Is it possible to turn off sonar issue analyser somehow in a project? It is desired to report only test coverage in the the first two projects(unit-tests & integration-tests) without analysing all files, and then run the issue analyzer only on the last project(whole-project). It could help us to analyse all files once instead of three times.
Additional info: We use sonar gradle plugin version 2.6.2 and sonarqube version 7.4
SonarQube/SonarCloud main responsibility is informing users about issues. Displaying code coverage is just an additional feature. It means there is no flag/parameter which allows you to do it.
Luckily, there is a workaround. You can create empty quality profiles, and use them to scan those two projects (unit-tests & integration-tests). You will get 0 issues because there are zero rules enabled.
The following feature request should be interesting for you: Making test coverage measures mode useful. Feel free to vote on it.
Sonar analysis is a nice way to check code compliance against a centrally defined policy. This is why I would like to use a profile with Blocker a Critical rules to signify a broken build.
I am using Jenkins do builds and invoke the SonarRunnerBuilder to get standard analysis on nightly builds of projects.
To achieve the features I outlined it would be best if I could run the analysis, check the results for violations (preferrably not sending them to Sonar server) and fail the build if there were any violations. So far I have not found way to do this using Google and looking through SonarRunner source code.
What I have considered is a workaround. I would implement a Decorator collecting violations and when the decoration reaches the project resource I would throw a RuntimeException to break the analysis. This would in turn fail the build.
would this work? Is there any better way to achieve this?
What you are looking for seems to be what the build breaker plugin offers. It relies on the Quality gate configuration to detect when violations (now called issues) in the current analysis require to break the whole build.
Please consider that this plugin won't be supported in SonarQube 5.2. Technically, deep changes in SonarQube's architecture make it impossible to provide the same feature. Philosophically, this plugin does not match the experience SonarQube wants to offer.
Still, another solution which covers the same use case is very likely to be offered in future version of SonarQube but this is yet to be defined.
I'm trying to run a preview analysis for a (Java) project of ours with SonarQube 5.1.1. I am able to get a local report generated, however I get no coverage data, and I also get the message [INFO] [XX:YY:ZZ.ZZZ] Build Breaker plugin is no more supported in preview/incremental mode.
If I check here, the page says that Starting with SonarQube 5.1, the Build Breaker plugin does not work any longer in the preview & incremental modes..
I'm confused - I thought that for continuous inspection one needs the build breaker plugin. Is that no longer so? Has the concept in SonarQube changed?
Why am I not getting coverage data when running a preview analysis?
I don't know where you've read this, but continuous inspection is not specifically related to the preview/incremental mode nor to the build breaker plugin - it's not even related to SonarQube (even though it has been pushed by SonarSource from the very beginning).
Here are the key points:
Continuous inspection is about analyzing your code as often as you can in order to monitor (and eventually improve) the quality of your code. Whatever the tool.
On SonarQube, this means running analyses that will push information on the server so that you can monitor what's going on and take the required actions for your application portfolio.
Obviously, when you are a developer, you'd like to manage those issues early, before they even get pushed to the source code repository. But experience tells us that preventing any code push because of issues is a bad pattern - because some issues might be false-positive or not relevant in the context (and still you want - and have the right, to push your code). This is why we feel that the build breaker plugin is not aligned with all this, and it will be replaced in upcoming versions of SQ by native features that match better these concepts:
Very efficient code analysis to display issues in the IDE at the speed of light - but without computing metrics
Preview mode that will compute everything and make it possible to check quality gate before pushing code to the source code repository - without impacting the results on the server
and in this case, using some specific information found in the logs, it will be possible for a CI to fail a build
I've seen a number of posts on making a Maven-backed Jenkins build fail for a given project if a coverage threshold isn't met i.e. coverage must be at least 80% or the build fails.
I'm wondering if there is a way to configure Jenkins to fail a build if the coverage is lower than the last build i.e. if the coverage for build N is 20%, and N+1 is 19%, then the build fails. I don't want to put in an explicit threshold, but I want the coverage to stay steady or get higher over time.
I haven't tried it, but assuming you are using maven cobertura plugin, I believe it can be configured to fail as documented here. Would jenkins not honour the failure?
I also see an open feature request for this.
Save last coverage percentage on the side. Use Groovy script (run via Groovy Plugin) that taps into Jenkins Java API to compare current coverage level with previous one and fail the build if it drops. If you seriously decide to go this way I can provide more details and code samples.
Since the 1.6 version of Cobertura plugin, the issue has been fixed, and a new option appeared that allow the plugin to make the build failed if coverage is lower than previous build.
This should be configurable in the configuration section of the plugin, that calculates the coverage. E.g. for the PMD plugin you can configure limits for unstable or failed build on the Job configuration page in the section "Build settings".
In NCover I specify a certain percent through a config file. You could create a python script (or similar to alter that value if you wanted). Other than that I do not think there are any config variables
We are working on a web project from scratch and are looking at the following static code analysis tools.
Conventions (Checkstyle)
Bad practices (PMD)
Potential bugs (FindBugs)
The project is built on Maven. Instead of using multiple tools for the purpose, I was looking at a single flexible solution and came across SonarQube.
Is it true that we can achieve the results from Checkstyle, PMD and Findbugs with SonarQube?
Sonar will run CheckStyle, FindBugs and PMD, as well as a few other "plugins" such as Cobertura (code coverage) by default for Java projects. The main added value, however, is that it stores the history in a database. You can then see the trend. Are you improving the code base or are you doing the opposite? Only a tool with memory can tell you that.
You should run Sonar in your CI system so that even things that take some time to execute (such as CPD – copy paste detector) can run. And you'll have your history. Whereas with an Eclipse plugin, for example, you'll detect violations sooner – which is great – but you will be tempted to run it less often if it starts taking too long, or run less "quality plugins" (such as skipping CPD or skipping code coverage analysis). And you won't have history.
Also, Sonar generates visual reports, "Dashboard" style. Which makes it very easy to grasp. With Sonar in Jenkins, you'll be able to show developers and your management the effects of the work that was performed on the quality of the code base over the last few weeks and months.
Sonar uses these 3 tools as plugins and aggregates the data from all three giving addition value by showing graphs and such from these tools. So they are complementary to sonar.
Yes and no. In addition to the other answers.
SonarQube is currently on the way to deprecate PMD, Checkstyle and Findbugs and use their own technology to analyze Java code (called SonarJava). They do it, because they don't want to spend their time fixing, upgrading (or waiting on it) those libraries (e.g. for Java 8), which for example uses outdated libraries.
They also got a new set of plugins for your personal IDE called SonarLint.
Sonar is great, but if you want to use the mentioned tools separately and still have nice graphs, you can use the Analysis Collector Plugin as part of your Jenkins CI build. A slight advantage of this is that you can check in your PMD/Findbugs/Checkstyle configuration into your SCM and have it integrated into your Maven build, rather than relying on a separate Sonar server.
... a few years later: no, it is not! SonarQube supposes to be able to cover all the rules with its own analyzer, but there are still rules from PMD or CheckStyle not covered by SonarQube. See for example: PMD ReturnFromFinallyBlock.
Sonar is much more than these tools alone.
The greatest benefits is the gui, which lets you configure anything easily.
The statistics it offers are very detailed (lines of code etc).
And it even offers great support for test coverage etc :)
Here you can take a good look:
http://nemo.sonarsource.org/
I would still use these tools in addition to sonar because they can fail the maven build when someone violates a rule. Where as sonar is more retrospective.
Well at least since SonarQube 6.3+ it seems to be that Findbugs is (at the moment) no longer supported as a plugin. Sonarsource is working on replacements of Findbugs-rules with its own Java-plugin.
They even had a list for the replacement status of each rule here, but it got removed by now.
See https://community.sonarsource.com/t/where-is-dist-sonarsource-com-content/5353 for more details.