I'm trying to find a way to migrate from mstest to xunit and still benefit from the IDE integration which made me choose mstest in the first place. Gallio seems to accomplish this noble goal, plus it's free (don't want Reshaper or TestDriven.net). But I can't get code coverage to work.
My solution contains of two projects: project SUT (the assembly I need to test) and project SUT.Tests which is a VS test project (this allows Gallio to display xunit tests in VS's Test View). I have code coverage enabled in Local.testSettings for SUT.dll and instrumentation is in place. After the test run completes there is no code coverage. Code Coverage Results window reports: Empty results generated: none of the instrumented binary was used. Look at test run details for any instrumentation problems. Unfortunately the test run details do not contain any "instrumentation problems". I tried unchecking the instrument assemblies in place checkbox and re-running the unit test; same result.
Any idea what's wrong?
My setup:
- Windows 7 x64
- VS 2010 Premium (SP1)
- xUnit 1.8
- Gallio 3.3.1 x64 (installer, not zip)
Apologies for answering my own question:
Turns out Gallio loads SUT.dll from SUT.Tests\Bin\Debug rather than SUT\Bin\Debug. I added this path to code coverage details, selected it instead of the default SUT.dll path and now it works! This is better described at here, towards the end of the post.
Related
I am making changes to an existing code base and ran Visual Studio Code Coverage tool to obtain code coverage against all tests in the test project. I see that areas that have code coverage are highlighted. However, is there an easy way to find which test provided coverage for a portion of code?
Visual Studio CodeLens provides a feature called Associated Unit tests which shows all unit tests associated with the method. This was what I was looking for.
I recently installed Visual Studio 2013 and CodeLens is amazing! The problem that I find is that whenever I open a class file that has methods in it, it doesn't seem to find the Unit Tests associated to the method if it is not written in MSTest. Is there anything that I have to do so that it can find other Unit tests like MSpec?
Is it because Machine Specifications has a different approach when creating unit tests vs MSTest or other testing framework out there?
The tested by and test status indicators are powered by the test explorer.
So if your MSpec tests show up in the test explorer and are written in C# or VB, they should also show up in CodeLens.
(same answer as Visual Studio 2013 feature Code Lens with NUnit)
I just did it with NUnit on a clean machine by installing Nunit and the Nunit extension in extension manager. Here Main shows one reference and one passing test, and the test itself shows the checkmark after successfully running.
I did the same thing that John Gardner showed in his answer, except I split it into 2 projects (to mimic the set up that I was working with in my real solution). At first it didn't work even though I could see the tests in Test Explorer. After a bit of searching, I stumbled onto the right answer for my situation.
It turns out that you need to:
(1) Create a new Unit Test Project (not a class library)
(2) Add NUnit reference to the Unit Test project (I used NuGet) and have the NUnit Test Adapter installed in VS2013
(3) Move your tests to this new project
(4) Once you save and build, now you can go back to your production code and see the "x/y passing" message and see the unit tests in the Test Explorer window.
Most of us that have been using NUnit for a long time are used to creating class libraries for our code instead of Unit Test Projects. It would be nice if the CodeLens documentation actually directly covered this (the documentation states "Test status indicators appear automatically in test projects" which was my clue).
I'm trying to run MSpec tests in Visual Studio's test system using Gallio, but it appears to be ignoring the MSpec plugin from the IDE. What I did:
Installed Gallio
Added the MSpec NuGet package to my test project
Added the Gallio NuGet package to my test project
Copied the three files required for the MSpec Gallio integration to an "MSpec" folder in Gallio's installation directory
Wrote an MSpec test
Ran "All Tests in Solution" in the IDE
This yielded the following error
"No tests were run because no tests are loaded or the selected tests are disabled"
To check Gallio itself, I added a reference to MbUnit and created a simple MbUnit test. I also ran Gallio's other tools to check if the plugin was being picked up at all. I even restarted VS for good measure.
Results
Running the tests using Gallio.Echo or Gallio.Icarus successfully executes both MbUnit and MSpec tests, so the plugin is being picked up. Running the tests from Visual Studio runs the MbUnit test but not the MSpec test, so Gallio integration appears to be working (just not the MSpec plugin). Any thoughts?
Before anyone mentions it: yes, I'd rather be using ReSharper but licenses are not an option for my situation at the moment.
Update: The is now a project to integrate MSpec into VS2012's test system. It's available on the Visual Studio Gallery and the source is on GitHub.
After starting to debug the problem with both project's sources, it appears there are two underlying issues causing the incompatibility between MSpec and Gallio's VS runner. However, both issues stem from the fact that the VS runner uses Cecil to load type information from assemblies (presumably to avoid loading the test assembly into the main VS AppDomain).
I'll update this answer as I uncover more (hopefully resulting in one or more patch to fix the issues).
Issue 1: Nested types are ignored
I had been using nested types for my tests (SubjectSpec+when_something), which are ignored by the Cecil-based reflection used by Gallio's VS runner to avoid loading the test assembly into the app domain.
This ended up being a relatively simple fix, which I've submitted as a patch to the Gallio developers. I'll update this answer if/when I get confirmation of which release it will be part of.
Issue 2: MSpec causes "An exception occurred while invoking a test driver"
If the container type is removed to avoid issue 1, MSpec throws this error. The reason for this is that MSpec is trying to create an instance of the test object, but Gallio is only supplying it a wrapper Type object that can't actually be created.
I'm beginning to get the grips of BDD and MSpec, but I'm still really bugged by the fact that I'm unable to debug my tests/specs, and that I have to leave the IDE to go to a html report file to see the results.
Currently, I have a post build event configured to run mspec.exe $(TargetFileName) --html “$(ProjectDir)Report.html”, but there must be some better way to do this.
Does anyone know any good add-ins, test runners or whatever that will let me
debug tests, instead of just running them
show the test results in a nice way inside Visual Studio
(Footnote: I'm running VS2010 Professional on Windows 7, if that matters.)
There are basically three options you have:
You can set up a custom tool in
Tools | External Tools to start
mspec.exe with the current project's
assembly to run the contexts and
generate the HTML report.
Install TestDriven.Net and
ensure that
Machine.Specifications.dll.tdnet
and
Machine.Specifications.TDNetRunner.dll
are in your project's copy of MSpec.
You can then run and debug your
contexts from the context menu: "Run
Tests", "Run With | Debugger" without further installation.
There's an example of what the MSpec folder
looks like for all of my projects.
If you use ReSharper 4.1, 4.5, 5.0
or the latest 5.1 EAP (== beta)
there are runners for each of these
versions.
The ZIP download
contains batch files that install
the runner for each respective
version of ReSharper. ReSharper's
unit test support is pretty
extensive in terms of UI
widgets/shortcut support, the reporting
tree view and debugging.
If you're
a dotTrace user you can also profile
right from within Visual Studio.
dotCover (another JetBrains product)
allows you to calcualate code
coverage results from your MSpec
runs.
On top of that, you get
all the nice coding and navigation features that ReSharper provides.
Be aware that only the first option will generate the HTML report as both the TestDriven.Net and ReSharper runners do not support HTML report generation. From my point of view this isn't an issue since the TD.Net and ReSharper runners offer fairly complete reporting mechanisms through the Visual Studio UI.
Another option that might work (I haven't used it myself) is to leverage the Gallio support that MSpec has. Gallio is a runner/framework for several testing frameworks; it might as well support debug runs with MSpec. Contact #smaclell if you have questions about Gallio support.
If I install Gallio 3.x will it also install a test runner plugin for Visual Studio?
Or must I use an additional plug-in like TestDriven.NET or Visual Nunit to run MbUnit test classes from within VS?
Install Gallio 3.1 on the dev machine. Then in VS2008, you'll have the option to create a "MbUnit v3 Test Project". This doesn't just include all of the Gallio dlls for you, it has a magic line in the project which identifies it to VS as a Test project.
You can now just use the in-built VS2008 Test runner.
If you have any existing projects with unit tests in, rather than making new projets, edit your existing project file and add the following line on line 9 (underneath the <ProjectGuid> on line 8):
<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
(If you have a VB project, it has a different second GUID: <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids> You can find the correct values by creating a new MbUnit test project from the templates installed with Gallio and then looking at the project file (.csproj or .vbproj) in a text editor.)
Now when you reload the project, VS2008 will recognise it as a test project.
A distinct advantage that I found over using Icarus was that debugging is now far more straight forward with break points being hit as expected.
Good Luck, Lee
TestDriven.Net works really well. Gallio also supports the ReSharper unit test runner and Visual Studio test tools. We will be shipping a new release of Gallio this week with support for R# 5.0 and VS 2010.
These are instruction for running MBUnit tests in Visual Studio 2012 and above using a neat NUnit trick.
Firstly, install the NUnit Test Adapter extension (yes, NUnit)
Tools > Extension and Updates > Online > search for NUnit > install
NUnit Test Adapter.
You may need to restart the Visual Studio IDE.
Then, you simply need to add a new NUnit test attribute to your test methods. See example code here (notice the using statements at the top) ...
//C# example
using MbUnit.Framework;
using NuTest = NUnit.Framework.TestAttribute;
namespace MyTests
{
[TestFixture]
public class UnitTest1
{
[Test, NuTest]
public void myTest()
{
//this will pass
}
}
}
You can run and debug the test in visual studio as NUnit and Gallio Icarus GUI Test Runner will run them as MBUnit (enabling parallel runs for example). You will need to stop Gallio from running the NUnit tests by deleting the NUnit folder in the gallio install location i.e. C:\Program Files\Gallio\bin\NUnit
Hope this helps, this is a simple working method so please vote up, many thanks.