I've seen a number of posts on making a Maven-backed Jenkins build fail for a given project if a coverage threshold isn't met i.e. coverage must be at least 80% or the build fails.
I'm wondering if there is a way to configure Jenkins to fail a build if the coverage is lower than the last build i.e. if the coverage for build N is 20%, and N+1 is 19%, then the build fails. I don't want to put in an explicit threshold, but I want the coverage to stay steady or get higher over time.
I haven't tried it, but assuming you are using maven cobertura plugin, I believe it can be configured to fail as documented here. Would jenkins not honour the failure?
I also see an open feature request for this.
Save last coverage percentage on the side. Use Groovy script (run via Groovy Plugin) that taps into Jenkins Java API to compare current coverage level with previous one and fail the build if it drops. If you seriously decide to go this way I can provide more details and code samples.
Since the 1.6 version of Cobertura plugin, the issue has been fixed, and a new option appeared that allow the plugin to make the build failed if coverage is lower than previous build.
This should be configurable in the configuration section of the plugin, that calculates the coverage. E.g. for the PMD plugin you can configure limits for unstable or failed build on the Job configuration page in the section "Build settings".
In NCover I specify a certain percent through a config file. You could create a python script (or similar to alter that value if you wanted). Other than that I do not think there are any config variables
Related
I have set up SonarQube, Maven and Cobertura to upload unit test coverage at every build run. I want to use the differential views to get coverage for the delta (new code only).
The coverage itself is successfully uploaded, and I can see the delta in lines of code. However, I can't get the "Coverage on new Code" to work.
I tried different values for the leak period, including previous_analysis and previous_version (changing the version from 1.0 to 1.1 in the new analysis). I also used -Dsonar.projectDate to simulate a past date of analysis but still no luck.
Any thoughts?
The key requirement to obtain new_code -related information/metrics is to leverage SonarQube SCM integration. And to benefit from that you have to install a compatible SCM Plugin applicable to your project (e.g. Git Plugin, SVN Plugin etc.).
"Coverage on New Code" is only displayed if you are using/activating SCM support (SVN, Git, ...).
sonar.scm.disabled=false
Adding this sonar.java.binaries=classes directory, most likely target/classes
helped me in fixing this issue. No JaCoCo analysis of project coverage can be done since there is no class files
I'm pretty new to Sonar, I'm using surefire for the metrics, and I've been just wondering how can I modify the Sonar overall coverage to increase or reduce this value to make it fail my Jenkins job.
Can someone point me in the right direction? I cannot find any documentation so far.
I'm not up to date what with the jenkins integration of sonar. Earlies you could install the build breaker plugin. Depending on your Sonarqube version this will no longer work.
Usually the thresholds for your project are not configured as parameters in the project but within sonarqube. Prior to 4.3 this was done through alerts. Nowadays it's called Quality Gates.
Actually a quite good question. I could not find anything for the maven plugin itself to set up some thresholds.
It seems to be an ongoing discussion.
I would like TeamCity to run my mSpec tests and report on the code covered by the tests.
I would also like TeamCity to report that a build has failed if code coverage in certain namespaces doesn't meet a threshold (e.g. MyProduct.ImportantStuff must be 100%, but MyProduct.LegacyStuff must be [23% or whatever it currently is to ensure we don't add new stuff without covering tests].
I initially looked at dotCover as it's integrated into TeamCity. I have since been looking at OpenCover as I couldn't get TC to fail the build on low coverage.
I got OpenCover working but I would still like to know (as I'm sure a lot of people would) how to get TC to fail a build if code coverage is too low.
Are you using the latest TeamCity, ie version 7?
When setting up a build configuration you can specify this:
There are lots more options in the dropdown related to code coverage. You can also force your build to fail if you're using some other code coverage tool.
For example you can echo a line to the console that will then be picked up by teamcity :
##teamcity[buildStatus status='FAILURE' text='something failed']
see official docs on this here
Does Sonar offer any way to raise alerts and fail a build when the trend for certain metrics is bad?
Background: In our legacy project using a static threshold for example for code coverage ("red alert when coverage is below 80%") does not make much sense. But we would like to make sure that the coverage does not go down any further.
Please do not give any advice on lowering the bar by using a less restrictive rule set. This is no option in our case.
There is a build breaker plug-in that will fail the build if you breach a Warning or Error threshold setup in the quality profile.
Plug-in details are here:
http://docs.sonarqube.org/display/PLUG/Build+Breaker+Plugin
Not aware of any functionality that enables you to a metric trend.
We use Sonar as the second last step in our release process. The build breaker ensures that releases do not breach predetermined quality criteria.
We tried exactly the same, using the build breaker plugin. After a while, it showed to be too unflexible (and configuring Sonar is a mess), so we moved from sonar to Jenkins/Hudson plugins like Cobertura (for code coverage) or PMD for code style:
https://wiki.jenkins-ci.org/display/JENKINS/PMD+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
With these plugins, very fine-granular settings are possible, to set for example the build to yellow at <70% code coverage or to red by <50%; even the weather-symbol for each build is setable.
In the meanwhile we scripted our own buildbreaker that gets excecuted within our build. We use Groovy to query the REST API of Sonar to retrieve a certain set of metrics (including their historical values). The retrieval of metrics is provided by a build plugin that is provided for our whole division.
Each team can parameterize their build with a set of rules regarding those metrics that have to be verified for their project. Of course, the rules are also provides as Groovy snippets :-)
Typical are:
Number of (major|critical|blocker) violations is less or equal than in previous build
No new duplicates
Coverage not lower than in previous build
Bad findings can then be used for breaking the build or just for reporting.
We have a huge project with many submodules. A full build takes currently over 30mins.
I wonder how this time distributes over different plugins/goals, e.g. tests, static analysis (findbugs, pmd, checkstyle, etc ...)
Would it be possible to time the build to see where (in both dimensions: modules and goals) most time is spent?
The maven-buildtime-extension is a maven plugin that can be used to see the times of each goal:
https://github.com/timgifford/maven-buildtime-extension
If you run the build in a CI server like TeamCity or Jenkins (formerly Hudson), it will give you timestamps for every step in the build process and you should be able to use these values to determine which goals/projects are taking the most time.
I don't think there is any way built in to maven to do this. In fact, in the related question artbristol posted, there is a link to a Maven feature request for this functionality. Unfortunately, this issue is unresolved and I don't know if it will ever be added.
The other potential solution is to write your own plugin which would provide this build metadata for you.
I don't think there is a way to determine the timing of particular goals. What you can do is run the particular goals separately to see how long they take. So instead of doing a "mvn install" which runs all of your tests, checkstyle, etc.. just do "mvn checkstyle:checkstyle" to see how long that takes for a particular module.
Having everything done every time is nice when its done by an automated server (continuum/jenkins/hudson) but when you are building locally, sometimes its better to be able to just compile. Some of the things you can do are have the static analysis goals ONLY run when you pass in a certain parameter or profile. Another option is to only have them ran when maven.test.skip=false.
If you are using a continuous build, try having the static analysis only done every 4 hours, or daily.