I am writing some unit tests for one of my DLL libraries.
The 'Code Coverage Results' pane shows a breakdown of the assemblies covered and tested.
For some strange reason - my test project itself appears in the coverage results ! (at approx. 90% covered).
This seems stupid... what's the deal with this ?
The reason the percentage is so high is that projects for code coverage are instrumented to keep track of which lines are hit by a test run, since you are running the tests from this project, almost all lines of code in the project will be run.
You can choose which projects/DLLs to collect Coverage statistics on in the Test Settings.
So if you don't need to capture stats on the test project (which you shouldn't really), you can simply remove this project from the settings you're using for coverage.
See http://msdn.microsoft.com/en-us/library/ms182534.aspx (steps 5 - 7 in particular) for more details.
Related
Here's a screenshot so you better understand what I'm talking about:
Shouldn't all directories have coverage?
I have to say that those directories (without any number) are not covered by any tests but doesn't this mean that the coverage is 0%?
Yes, they should. But they don't because when no coverage engine reports any coverage on a file, is that because the file is executable but there are no tests on it, or because the file is not executable?
SonarQube v6.2 will begin to address this by automatically forcing to 0 the coverage metrics on files not covered in the unit test reports. However, this behavior will only be fully enabled when each of the language plugins reports the "executable lines" for each file.
So in practice the new behavior enabled in 6.2 (not released at this writing, but "soon") won't be truly available until the language plugins start supporting it. Probably over the next few months.
I have a TFS build in VS2010. Following the build unit tests are executed. In the Build summary it tells me that "1 test run(s) completed - 100% average pass rate" but below this it states "No tests were impacted". I guess Impacted Tests relate to functionality providing the ability to only run tests that were impacted by code checked in? Is this correct and since I'm happy to run all tests following each build, how can I turn off the "No tests were impacted" message?
Thanks,
Rob
You're right about test impact analysis - it looks at the changesets since the last build and which tests intersect with that code. To disable test impact analysis for a build:
Right-click your build definition in Team Explorer and choose Edit...
Go to the Process category and expand the Advanced section
Set "Analyze Test Impact" to False
You can also set this value on a per-build basis - queue a manual build and the setting is available via the Parameters tab.
I would like TeamCity to run my mSpec tests and report on the code covered by the tests.
I would also like TeamCity to report that a build has failed if code coverage in certain namespaces doesn't meet a threshold (e.g. MyProduct.ImportantStuff must be 100%, but MyProduct.LegacyStuff must be [23% or whatever it currently is to ensure we don't add new stuff without covering tests].
I initially looked at dotCover as it's integrated into TeamCity. I have since been looking at OpenCover as I couldn't get TC to fail the build on low coverage.
I got OpenCover working but I would still like to know (as I'm sure a lot of people would) how to get TC to fail a build if code coverage is too low.
Are you using the latest TeamCity, ie version 7?
When setting up a build configuration you can specify this:
There are lots more options in the dropdown related to code coverage. You can also force your build to fail if you're using some other code coverage tool.
For example you can echo a line to the console that will then be picked up by teamcity :
##teamcity[buildStatus status='FAILURE' text='something failed']
see official docs on this here
I have been tasked with introducing automated unit test coverage reporting to our CI build environment (we use Hudson and MSBuild). I have used dotCover to produce the coverage data across several projects within a build job, merge that data, produce a single HTML report and integrated that with the Hudson dashboard for that job using the HTML Publisher Hudson add-in.
So far so good. But, what we ultimately want is a graphical display of unit test coverage metrics across the entire system code base (half a dozen build jobs, circa 50 projects) up on one of our team displays; a highly visible 'at a glance' test coverage status. I can build this myself with an app that will merge the dotCover coverage reports for all builds, report to xml and build my own UI around that, but is there a product out there that does this already? We are not married to dotCover, that can change, and it's doesn't have to be free or open source either.
We are using NCover to get the test coverage and a bunch of other code metrics(symbol coverage, branch coverage and cyclomatic complexity).
It is pretty good for highlighting the coverage, you can browse through the source code as part of the reports and see which lines are covered and which are not.
You can also exclude test categories or specific namespaces from your reports.
As far as I know it integrates with Hudson.
OpenCover and PartCover are both open source code coverage tools that can be integrated into build automation systems.
With both you can use ReportGenerator to display results; though PartCover does come with it's own viewer I prefer to use the ReportGenerator one myself as the HTML can be integrated quite nicely into a build status report.
You may also build your own reports using XSLT or such like as both tools have an XML output.
Our C# Test Coverage Tool tool can combine test coverage vectors for separate C# artifacts into a single overall view, out of the box.
In fact, our family of test coverage tools will can combine results from multiple different languages (e.g., C#, VB.net, C++, Java, PHP, COBOL, ...) in the same way.
The tool(s) provide both visual view of the covered code, and reports of the coverage data including in an XML form.
I like the treemap functionality from SonarQube a lot, but this only works for one 'project' in SonarQube, nevertheless, a great tool to look into.
Google for SonarQube and treemap for examples.
Using TeamCity running an MsBuild task for an MVC2 C# application, we successfully run 1561 XUNit tests in both the UAT and the Live builds but the NCover coverage falls from 51% on the UAT build to 35% on the live build. The soulution has identical configuration manager settings.
As our minimum covergae is less than 50% our build subsequently fails with the following error:
"NCover.Reporting.exe" exited with code 3.
A bit lost as to why the coverage drops when it is the same source from svn and identical test run being performed.
Has anyone else experienced this?
My recommendation would be to drop us an email at support#ncover.com, ideally with the two coverage files attached. It's not unusual for us to see small coverage differences between Debug and Release builds running against the same tests (because the build types generate slightly different code), but never on the same build type running against the same test.