I have a Visual Studio solution consisting of several projects. I set a build process for it in TeamCity, and I have added a an NUnit runner build step, with these settings:
nunit runner 2.6.3
run tests from: %teamcity.build.workingDir%\Sources\**\bin\*.Web.UnitTests.dll
.net coverage tool: JetBrains dotCover
Filters: --empty--
In the Code Coverage tab of the build, under Coverage Breakdown, I can only see:
MyProject.Domain
MyProject.Web.UnitTests
MyProject.Web, the object of the tests, and a few other assemblies in the solution are not displayed there, and are ignored by the dotCover code coverage calculations. Why? How can I tell it which assemblies to use?
I've unsuccessfully tried various +: / -: combinations in the Filters field. But with Filters empty, I am expecting MyProject.Web to be included, since MyProject.Domain is - why isn't it included?
Are you building in Release or Debug mode? I had the same problem with Release mode and had to enable PDB output which is what dotCover requires to determine the code coverage. PDB is not enabled by default for Release builds.
To enable it go to Project Settings -> Build -> Advanced (button at the bottom) -> Debug Info -> Choose "pdb-only" or "full".
Additionally, dotCover provides this statement when no code coverage can be determined. It may be helpful to check all of the reasons listed:
No executable code was detected.
The issue could be caused by one of the following:
Include / exclude patterns are incorrect
Assemblies are compiled without debugging information
.pdb files are not available
Visual Studio code coverage is enabled for MSTest
.testrunconfig is used for MSTest and Visual Studio code coverage is not disabled (CodeCoverage section with enable="true" is present)
I'm afraid without log files and your snapshot it will be difficult to provide the right answer. Please submit a request and attach the files: http://dotnettools-support.jetbrains.com/anonymous_requests/new
If logs are not enabled yet, add the corresponding parameter to your build configuration to activate them: https://confluence.jetbrains.com/display/TCD9/Reporting+Issues#ReportingIssues-dotCoverIssues
Related
I installed the SonarLint extension for Visual Studio and connected successfully to our SonarQube server and successfully ran Code Analysis to display sonar issues in VS. So it was working OK but for some reason I am now no longer getting any sonar Sxxx warnings and instead now see the following 2 warnings:
> Warning CA0507 Post-build Code Analysis (FxCopCmd.exe) has been
> deprecated in favor of FxCop analyzers, which run during build. Refer
> to https://aka.ms/fxcopanalyzers to migrate to FxCop analyzers.
>
> Warning CA0064 : No analysis was performed because the
> specified rule set could not be loaded or did not contain any managed
> code analysis rules.
But I am not using FxCop and I am not aware of having done anything to turn it on.
If I check one rule inside "Managed Binary Analysis" the CA0064 warning will go away, but CA0507 remains and still no sonar Sxxx warnings.
The "Run Code Analysis" menu items only apply to the legacy FxCop rules. You don't need to use those menu items for Roslyn-based analyzers (like the C# and VB.NET rules in SonarLint) - Visual Studio will automatically trigger the analysis in the background. See the Microsoft docs for more info.
If you are not seeing Sxxx issues being reported there are a couple of things to look at:
Check whether you have the Enable full solution analysis option checked. If not, VS will only report issues in open files.
You've connected the solution to your SonarQube server so SonarLint will have generated a ruleset based on the Quality Profile. The ruleset will be in the solution-level .sonarlint folder. Check whether the rules you are expecting to report issues are enabled in the ruleset file. If not, update the Quality Profile on the server then close and re-open the solution. SonarLint will detect that the Quality Profile has changed and prompt you to update it.
Try to Compile again the project. With first compilation this errors will appear but with second compilation probably not.
No idea why, but it worked for me.
What is the best way to run and report Visual Studio 2012 code analysis in TeamCity?
Since FxCop or Code Analysis integrates directly into MsBuild, it's probably as simple as adding /p:RunCodeAnalysis=Always or /p:RunCodeAnalysis=True to the MsBuild commandline when building the project. Since this will generate standard compiler warnings, TeamCity should pick them up automatically.
Visual Studio 2010+ Code Analysis is (based on) FxCop 10.0. You can simply add a 'FxCop' build runner type build step.
When the 'Report XSLT file' setting is configured the build runner will output a HTML report. FxCop 10.0 includes a default XSLT file under '%system.FxCopRoot%/Xml/FxCopReport.xsl' to generate a default HTML report.
I'm trying to find a way to migrate from mstest to xunit and still benefit from the IDE integration which made me choose mstest in the first place. Gallio seems to accomplish this noble goal, plus it's free (don't want Reshaper or TestDriven.net). But I can't get code coverage to work.
My solution contains of two projects: project SUT (the assembly I need to test) and project SUT.Tests which is a VS test project (this allows Gallio to display xunit tests in VS's Test View). I have code coverage enabled in Local.testSettings for SUT.dll and instrumentation is in place. After the test run completes there is no code coverage. Code Coverage Results window reports: Empty results generated: none of the instrumented binary was used. Look at test run details for any instrumentation problems. Unfortunately the test run details do not contain any "instrumentation problems". I tried unchecking the instrument assemblies in place checkbox and re-running the unit test; same result.
Any idea what's wrong?
My setup:
- Windows 7 x64
- VS 2010 Premium (SP1)
- xUnit 1.8
- Gallio 3.3.1 x64 (installer, not zip)
Apologies for answering my own question:
Turns out Gallio loads SUT.dll from SUT.Tests\Bin\Debug rather than SUT\Bin\Debug. I added this path to code coverage details, selected it instead of the default SUT.dll path and now it works! This is better described at here, towards the end of the post.
I'm building VS2010 solutions on a TFS2008 build agent with VS2010 Ultimate installed; code compiles fine, and the tests all run fine, but when it comes to code coverage, nothing is being reported. I've checked that the .testsettings file has Code Coverage enabled and that I have selected some assemblies to instrument, and if I run the tests locally on my developer PC, code coverage is analysed and reported successfully.
However when built on the build server, code coverage is not reported. I can see a data.coverage file sitting in the build's TestResults\guid\In\MachineName folder, so it's obviously done something, but the build logs contain no mention of anything to do with code coverage, and no warnings/errors.
Update: checking on the TFS server itself, the application log contains the following error:
The Code Coverage Analysis Service could not analyze the code coverage data for build Daily Build>20100812.2 (Any CPU/DEBUG). This is frequently caused by publishing different versions of the same binary to a particular build.
Processing of code coverage data from run svcTFSService#W2411193 2010-08-12 12:52:34_Any CPU_Debug caused the following exception to be thrown:
Error when creating coverage info: Error loading symbol file. Symbol and binary files should be in the same folder as the coverage file or on the symbol path: D:\Source\CommonPlatform\Common Platform Development Daily Build\Binaries\Debug\Core.dll.
Yet when I look on the build agent in the specified folder, both the assembly and its debug symbol file are in the expected location.
Update 2: On a whim, I tried opening the data.coverage file that I mentioned above in VS 2010 on the build agent, and I get the following error:
Code Coverage Analysis engine threw exception(s): Image file "D:\Source\CommonPlatform\Common Platform Development Daily Build\Binaries\Debug\Core.dll" could not be found.
Yet I can plainly see the file sitting in the filesystem, as well as its PDB.
You cannot have code coverage using TFS 2008 and .Net 4 code. You have to upgrade to tfs 2010.
http://social.msdn.microsoft.com/Forums/en-US/tfsbuild/thread/39aaaf43-0e3a-41c7-8b72-c1cabc9438e1/
Check to see if the file timestamps are essentially the same on both the symbol file and the assembly. They shouldn't be off by more than a second (or two at the most). If they are, it might be that you are building Core.dll in more than one place, and only one is creating the symbol file.
It might be a long shot, but it's probably a good place to start.
I'm beginning to get the grips of BDD and MSpec, but I'm still really bugged by the fact that I'm unable to debug my tests/specs, and that I have to leave the IDE to go to a html report file to see the results.
Currently, I have a post build event configured to run mspec.exe $(TargetFileName) --html “$(ProjectDir)Report.html”, but there must be some better way to do this.
Does anyone know any good add-ins, test runners or whatever that will let me
debug tests, instead of just running them
show the test results in a nice way inside Visual Studio
(Footnote: I'm running VS2010 Professional on Windows 7, if that matters.)
There are basically three options you have:
You can set up a custom tool in
Tools | External Tools to start
mspec.exe with the current project's
assembly to run the contexts and
generate the HTML report.
Install TestDriven.Net and
ensure that
Machine.Specifications.dll.tdnet
and
Machine.Specifications.TDNetRunner.dll
are in your project's copy of MSpec.
You can then run and debug your
contexts from the context menu: "Run
Tests", "Run With | Debugger" without further installation.
There's an example of what the MSpec folder
looks like for all of my projects.
If you use ReSharper 4.1, 4.5, 5.0
or the latest 5.1 EAP (== beta)
there are runners for each of these
versions.
The ZIP download
contains batch files that install
the runner for each respective
version of ReSharper. ReSharper's
unit test support is pretty
extensive in terms of UI
widgets/shortcut support, the reporting
tree view and debugging.
If you're
a dotTrace user you can also profile
right from within Visual Studio.
dotCover (another JetBrains product)
allows you to calcualate code
coverage results from your MSpec
runs.
On top of that, you get
all the nice coding and navigation features that ReSharper provides.
Be aware that only the first option will generate the HTML report as both the TestDriven.Net and ReSharper runners do not support HTML report generation. From my point of view this isn't an issue since the TD.Net and ReSharper runners offer fairly complete reporting mechanisms through the Visual Studio UI.
Another option that might work (I haven't used it myself) is to leverage the Gallio support that MSpec has. Gallio is a runner/framework for several testing frameworks; it might as well support debug runs with MSpec. Contact #smaclell if you have questions about Gallio support.