Using TeamCity running an MsBuild task for an MVC2 C# application, we successfully run 1561 XUNit tests in both the UAT and the Live builds but the NCover coverage falls from 51% on the UAT build to 35% on the live build. The soulution has identical configuration manager settings.
As our minimum covergae is less than 50% our build subsequently fails with the following error:
"NCover.Reporting.exe" exited with code 3.
A bit lost as to why the coverage drops when it is the same source from svn and identical test run being performed.
Has anyone else experienced this?
My recommendation would be to drop us an email at support#ncover.com, ideally with the two coverage files attached. It's not unusual for us to see small coverage differences between Debug and Release builds running against the same tests (because the build types generate slightly different code), but never on the same build type running against the same test.
Related
I am using SonarQube 4.3 version to show Unit tests and other code metrics analysis results. Now I am looking for incremental preview or mode for my analysis results as described here.
I am looking to see whether the last code checked-in has a 100% unit test coverage, no security violations and has other quality gates passed.
Currently sonar gives overall combined report for all the files within project. Can we have one for the changed files in the last build as well?
I was working on team internal KPI's lately and I thought that code coverage/number of unit tests will be a great measure.
In our CI build pipeline we're using NUnit to run test and JetBrains dotCover to generate coverage report - everything works great and we have that fancy "statistics" page in Team City control panel:
What I would like to do is to export this coverage data so I can generate HTML output for the current coverage status and keep it under control not necessarily in the Team City console?
Thanks in advance.
For any reports that teamcity shows, the convention is that it needs a zip file with an index.html page at the root. So for coverage report it would be coverage.zip and so on.
If you go the workout directory for your builds, you can find the artefacts that you want (coverage, test results etc) in the artefacts folder.
I am writing some unit tests for one of my DLL libraries.
The 'Code Coverage Results' pane shows a breakdown of the assemblies covered and tested.
For some strange reason - my test project itself appears in the coverage results ! (at approx. 90% covered).
This seems stupid... what's the deal with this ?
The reason the percentage is so high is that projects for code coverage are instrumented to keep track of which lines are hit by a test run, since you are running the tests from this project, almost all lines of code in the project will be run.
You can choose which projects/DLLs to collect Coverage statistics on in the Test Settings.
So if you don't need to capture stats on the test project (which you shouldn't really), you can simply remove this project from the settings you're using for coverage.
See http://msdn.microsoft.com/en-us/library/ms182534.aspx (steps 5 - 7 in particular) for more details.
I would like TeamCity to run my mSpec tests and report on the code covered by the tests.
I would also like TeamCity to report that a build has failed if code coverage in certain namespaces doesn't meet a threshold (e.g. MyProduct.ImportantStuff must be 100%, but MyProduct.LegacyStuff must be [23% or whatever it currently is to ensure we don't add new stuff without covering tests].
I initially looked at dotCover as it's integrated into TeamCity. I have since been looking at OpenCover as I couldn't get TC to fail the build on low coverage.
I got OpenCover working but I would still like to know (as I'm sure a lot of people would) how to get TC to fail a build if code coverage is too low.
Are you using the latest TeamCity, ie version 7?
When setting up a build configuration you can specify this:
There are lots more options in the dropdown related to code coverage. You can also force your build to fail if you're using some other code coverage tool.
For example you can echo a line to the console that will then be picked up by teamcity :
##teamcity[buildStatus status='FAILURE' text='something failed']
see official docs on this here
We're using Jenkins and attempting to make our project's CI build as modular as possible, i.e. independent jobs for checkout/build/test/analysis/deploy which can then be chained together as and when needed.
The problem is that I can't figure out how to get Sonar to only run tests or only run analysis. Regarding the former, I'm completely lost; for the latter, I understand I can set sonar.dynamicAnalysis = reuseReport. But our tests are in MSTest format and we use Gallio to run them, and since Sonar only seems to support surefire/cobertura/clover report files - not Gallio/MSTest - I'm not sure how to get Sonar to read the test output.
So how do I get Sonar to split its test and analysis phases?
IMO, you could let Sonar handle tests and analysis at the same time - this will make your configurations far simpler (=> using "reuseReport" can lead to lors of troubles if assemblies have been moved between the build/test and the Sonar analysis).
So basically, what I'm saying is that in your "analysis" job, Sonar would do test + static analysis. That's what we do at SonarSource, we have:
a continuous integration process that does "checkout/build/test/deploy" on every commit
a continuous inspection process that does ""checkout/build/sonar" every night