SonarQube - Sonar way Coverage over leak period fails even when no modifications occur on codebase - sonarqube

Maybe this is my ignorance in understanding the Quality Gate, but I have a failing quality gate due to the default 40% Coverage over leak period when using the sonarway code quality gate via VSTS build. The issue is that there has been no modifications to the code between the initial analysis and the latest analysis, so to reference the metaphor in the docs.. there is no additional water in the kitchen.. hence I am not seeing a reason for this criterion to be failing.
Has anyone else experienced this and/or can anyone explain the logic if this is indeed the expected behavior? IMO, I would expect the Leak Period Code Coverage check to not apply when no modifications have occurred on the code base during the leak period.
My SQ analysis is being executed via VSTS and the version of SQ is 6.7.3.
Summary of Analysis with failing QG due to coverage leak (also the percentage coverage is still the same)
The issue also occurs when there are code modifications and those specific modifications have 100% code coverage
The sonarway quality gate configuration is as follows (default configuration):
As requested, I have also created a simple demo project which also demonstrates the behavior (running builds with SQ analysis and the second build fails due to 0.0% coverage over leak period although there is no new code). Sample project can be found here
Appreciate if someone can explain this behavior to me as it seems contrary to the documentation.

In case anyone stumbles upon this question, it turned out to be a bit of a red herring!
Following some good discussions with colleagues we were all in agreement that a coverage gate condition with Over Leak Period set to always was an incorrect gate configuration.
I was under the false assumption that the SonarQube way was the built-in default and was not editable. It turns out however, that the gate is in fact editable. Read-only mode was only enforced from v7.0 onwards as per release notes. So the root cause of our misconfiguration is still under investigation but we can be sure that it's not the default and recommended Gate as per SonarQube documentation
In the example posted above, the gate complains that the additional 0.7% coverage is below the minimum 40% error level. The condition itself is applying as configured but it really should be impossible to enable leak period for coverage. Instead the Coverage on New Code should be used.
So the simple one liner solution:
Make sure Over Leak Period is set to never (and/or not selected) if using the Coverage condition

Related

Suppressing bugs in next SonarQube analysis

We have started using SonarQube analysis for C#, JavaScript. Our application is old one. So when we did analysis for the first time (for first release) it showed bugs in thousands.Now what we want is to set benchmark for bugs. Now when I go for next scan for the same project I should not get same thousand defects again, instead it should give only new bugs related to current release(second release). Do we have something in SonarQube to configure which sets benchmark.
What you want is fixing the leak. You can configure your quality gates to rely on issues introduced during the leak period (instead of the absolute value)

Does continuous inspection still work with Sonar 5.1.X?

I'm trying to run a preview analysis for a (Java) project of ours with SonarQube 5.1.1. I am able to get a local report generated, however I get no coverage data, and I also get the message [INFO] [XX:YY:ZZ.ZZZ] Build Breaker plugin is no more supported in preview/incremental mode.
If I check here, the page says that Starting with SonarQube 5.1, the Build Breaker plugin does not work any longer in the preview & incremental modes..
I'm confused - I thought that for continuous inspection one needs the build breaker plugin. Is that no longer so? Has the concept in SonarQube changed?
Why am I not getting coverage data when running a preview analysis?
I don't know where you've read this, but continuous inspection is not specifically related to the preview/incremental mode nor to the build breaker plugin - it's not even related to SonarQube (even though it has been pushed by SonarSource from the very beginning).
Here are the key points:
Continuous inspection is about analyzing your code as often as you can in order to monitor (and eventually improve) the quality of your code. Whatever the tool.
On SonarQube, this means running analyses that will push information on the server so that you can monitor what's going on and take the required actions for your application portfolio.
Obviously, when you are a developer, you'd like to manage those issues early, before they even get pushed to the source code repository. But experience tells us that preventing any code push because of issues is a bad pattern - because some issues might be false-positive or not relevant in the context (and still you want - and have the right, to push your code). This is why we feel that the build breaker plugin is not aligned with all this, and it will be replaced in upcoming versions of SQ by native features that match better these concepts:
Very efficient code analysis to display issues in the IDE at the speed of light - but without computing metrics
Preview mode that will compute everything and make it possible to check quality gate before pushing code to the source code repository - without impacting the results on the server
and in this case, using some specific information found in the logs, it will be possible for a CI to fail a build

Coverage Reports on SonarQube Preview/Incremental analysis

We've recently set our Continuous Integration environment to do an incremental code analysis before a Merge Request on Git can be accepted. We already use the HTML issues report and the Build Breaker plugin. Everything works beautifully. The thing is, we're having some issues because the build is broken on < 85% code coverage. I didn't find any plugin to let the user know where the coverage dropped. Is there any way to show that comparison between the analisys that is recorded on my server and the one Jenkins is incrementally building ?
Thanks a lot!
Im using sonar4.5.4 and we use the Coverage on new code metric in our quality gate, in that metric we compare with the previous analysis

How should sonarqube quality gates be used with automated builds?

Our organization is currently using sonarqube to fail a SCM triggered automated builds. Right now, our quality gate is firing when there are any open or reopened issues (with the understanding that someone needs to fix or "accept" the issue before the build will continue). Unfortunately, there are concerns that we are delaying code propagation for code formatting issues and other very minor offenses. Also, this is requiring quite a bit more manual intervention than we originally expected.
We want certain classes of issues to trigger a build failure, but the "new issues" filter only fire once. on the subsequent automated build they are no longer new so they no longer fire the gate.
Is there anyway to set up the equivalent to "Break the build if there are more than 0 open/reopened breaking/critical issues."?
EDIT: Sorry, I forgot to mention that we have about 15k "confirmed" issues that is serving as our backlog. so the non-new issue filters wont work either. Also I am open to minor changes to our workflow.
From what I understand, the following configuration should do the job:

Fail build when trend in Sonar is bad

Does Sonar offer any way to raise alerts and fail a build when the trend for certain metrics is bad?
Background: In our legacy project using a static threshold for example for code coverage ("red alert when coverage is below 80%") does not make much sense. But we would like to make sure that the coverage does not go down any further.
Please do not give any advice on lowering the bar by using a less restrictive rule set. This is no option in our case.
There is a build breaker plug-in that will fail the build if you breach a Warning or Error threshold setup in the quality profile.
Plug-in details are here:
http://docs.sonarqube.org/display/PLUG/Build+Breaker+Plugin
Not aware of any functionality that enables you to a metric trend.
We use Sonar as the second last step in our release process. The build breaker ensures that releases do not breach predetermined quality criteria.
We tried exactly the same, using the build breaker plugin. After a while, it showed to be too unflexible (and configuring Sonar is a mess), so we moved from sonar to Jenkins/Hudson plugins like Cobertura (for code coverage) or PMD for code style:
https://wiki.jenkins-ci.org/display/JENKINS/PMD+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
With these plugins, very fine-granular settings are possible, to set for example the build to yellow at <70% code coverage or to red by <50%; even the weather-symbol for each build is setable.
In the meanwhile we scripted our own buildbreaker that gets excecuted within our build. We use Groovy to query the REST API of Sonar to retrieve a certain set of metrics (including their historical values). The retrieval of metrics is provided by a build plugin that is provided for our whole division.
Each team can parameterize their build with a set of rules regarding those metrics that have to be verified for their project. Of course, the rules are also provides as Groovy snippets :-)
Typical are:
Number of (major|critical|blocker) violations is less or equal than in previous build
No new duplicates
Coverage not lower than in previous build
Bad findings can then be used for breaking the build or just for reporting.

Resources