Introduction
After a long struggle i finally found out how to run my specflow scenario's through the MSTest Runner. However i could not find a convenient way of showing the test results in the orderly overview that many BDD books and courses promote.
What i've tried so far
The way i currently generate my reports is with the following CMD commands, executed in the Visual Studio Dev CMD.
generic
"navigate\to\specific\visual\studio\folder\mstest.exe" /testcontainer:"navigate\to\test\dll\relative\to\mstest" /resultsfile:\folder\in\my\project\result.html
"navigate\to\specific\specflow\folder\in\project\specflow.exe mstestexecutionreport "absolute\path\to\my\csproj\file\under\test" /testResult:relative\directory\in\project\testResults.trx /out:directory\relative\in\project\TestResults.html
my specific unreadable (obfuscated) case
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\mstest.exe" /testcontainer:"C:\Workspace\Branch\Sub1\Sub2a\Project\bin\Debug\Project.dll" /resultsfile:\Sub2a\TestResults\TestResults.trx
"C:\Workspace\Branch\Sub1\Sub2b\SpecFlow.2.0.0\tools\specflow.exe" mstestexecutionreport "C:\Workspace\Branch\Sub1\sub2a\Project\Project.csproj" /testResult:Sub1\TestResults\TestResults.trx /out:Sub1\TestResults\TestResults.html
I probably don't have to mention how tighlty coupled this script is to my specific machine, workspace, project, visual studio version and tests.
There must be some way to make these tools work better with a more generic script or some kind of plugin for the mstest report without going through so much cmd line statements with huge parameters.
Thanks in advance.
Related
I use Xorax IncrediBuild to build Visual Studio 2013 (or later) solutions and projects, they're mostly .vcxproj with a bunch of .csproj ones.
It took me a little bit of digging, but I've learned that:
When IncrediBuild is used with the regular Visual Studio, it uses the Devenv.exe by default.
Since version 5.0, BuildConsole.exe supports a new switch: /UseMSBuild to specifically instruct IncrediBuild to use MSBuild's build engine instead of the default Visual Studio's DevEnv.
So, in case of Visual Studio projects, there are two modes available:
BuildConsole.exe MyProj.vcxproj which uses DevEnv.exe
BuildConsole.exe MyProj.vcxproj /usemsbuild which uses MSBuild.exe
I'd like to learn if there are any differences between using the two engines.
I have made some tests and observed that:
IncrediBuild "Initializing..." phase takes slightly longer in case of DevEnv.exe.
BuildConsole.exe generates different output, obviously.
No (or insignificant) difference in build performance.
In case of building individual C/C++ native projects (.vcxproj) as well as whole solutions (.sln), what are advantages and disadvantages of using DevEnv.exe versus MSBuild.exe?
** disclaimer: I work at IncrediBuild **
We've determined together with Microsoft that in order to get builds that behave in the same manner as building from within Visual Studio (without IncrediBuild), DevEnv should be used. MSBuild executes builds in a slightly different manner than VS both in the build output it produces and in the way it behaves when executing custom steps and some other minor things. If a user wants IncrediBuild builds to behave in the same manner as he is used to when building from Visual Studio, the default way should be used (IncrediBuild executing DevEnv). If a user is used to execute his builds using MSBuild, whether from the command line or through TFS, the UseMSBuild switch should be used. We wanted to allow users to choose the way they would like IncrediBuild to work depending on the way they are used to do that, same as Microsoft supporting both DevEnv and MSBuild.
Additional comments:
The initialization stage is indeed longer when using Devenv, since devenv loads and parses the solution and .vcxproj files in a different manner than msbuild. The more Projects a solution has – the longer this phase should take. The increase of time it takes this phase to complete usually greatly offset by the speed increase in the actual build time – when building several projects at the same time.
Devenv is highly recommended for our Predicted Execution feature which can provide up to 20% of additional build acceleration due to the above way of work which would be not possible using MSBuild.
Microsoft is pretty explicit about it:
For build-related tasks, it is now recommended that you use MSBuild instead of devenv. For more information, see MSBuild Command Line Reference.
A note in the Devenv documentation since VS2010, the first version of VS that started supporting building C++ projects with MSBuild and also changed the default project file extension from .vcproj to vcxproj.
The wisdom of this kind of advice can only be inferred when they are not explicit about it. One you ready saw, Devenv.exe is a pretty heavy process with many DLL dependencies, it takes a while to get going. Another you could fret about with the way you are doing it now, original guidance was to use Devenv.com instead of Devenv.exe. Those dependencies are also troublemakers, the kind that tend to get in the way or just plain fall over when Devenv.exe runs in an unusual runtime environment like a service. Sample story-from-hell is this Q+A, three answers and none look correct. There are others.
Plain advice is to use the recommended way.
I recently installed Visual Studio 2013 and CodeLens is amazing! The problem that I find is that whenever I open a class file that has methods in it, it doesn't seem to find the Unit Tests associated to the method if it is not written in MSTest. Is there anything that I have to do so that it can find other Unit tests like MSpec?
Is it because Machine Specifications has a different approach when creating unit tests vs MSTest or other testing framework out there?
The tested by and test status indicators are powered by the test explorer.
So if your MSpec tests show up in the test explorer and are written in C# or VB, they should also show up in CodeLens.
(same answer as Visual Studio 2013 feature Code Lens with NUnit)
I just did it with NUnit on a clean machine by installing Nunit and the Nunit extension in extension manager. Here Main shows one reference and one passing test, and the test itself shows the checkmark after successfully running.
I did the same thing that John Gardner showed in his answer, except I split it into 2 projects (to mimic the set up that I was working with in my real solution). At first it didn't work even though I could see the tests in Test Explorer. After a bit of searching, I stumbled onto the right answer for my situation.
It turns out that you need to:
(1) Create a new Unit Test Project (not a class library)
(2) Add NUnit reference to the Unit Test project (I used NuGet) and have the NUnit Test Adapter installed in VS2013
(3) Move your tests to this new project
(4) Once you save and build, now you can go back to your production code and see the "x/y passing" message and see the unit tests in the Test Explorer window.
Most of us that have been using NUnit for a long time are used to creating class libraries for our code instead of Unit Test Projects. It would be nice if the CodeLens documentation actually directly covered this (the documentation states "Test status indicators appear automatically in test projects" which was my clue).
There is option of Create Unit Tests in VS2010.
It creates a separate test project and uses MS test frame work. Is it possible for configure VS to use Nunit framework to create unit test cases?
From what i know Visual Studio generates only MSTests. But there are many commercial tools (You can always evaluate :)) which are quite helpful.
For example:
Other topic
Parasoft dotTEST
Probably by your own You will find few more.
Though I ask it to, Visual Studio 2010 does not appear to generate code coverage statistics for my F# modules. The unit tests are Ms Tests written in C#, so are run natively by Visual Studio, and execute cleanly. But there does not appear to be any coverage information created for the F# library being tested. The C# libraries involved are analyzed just fine.
Am I missing something?
This link describes using the VS testing system with F#. You do it pretty much the same way as with C#.
The downside is that apparently the VS IDE won't automatically pick it up -- I believe you need to run from the command line using mstest.exe.
Edit: Oh, another cool thing with F#, is FsCheck, an automatic testing tool for F#, is basically a port of Haskell's QuickCheck. It allows you to provide a specification of the program, in the form of properties that the functions or methods should satisfy, and FsCheck tests that the properties hold in a large number of randomly generated cases.
also fsunit and fstest
I'm beginning to get the grips of BDD and MSpec, but I'm still really bugged by the fact that I'm unable to debug my tests/specs, and that I have to leave the IDE to go to a html report file to see the results.
Currently, I have a post build event configured to run mspec.exe $(TargetFileName) --html “$(ProjectDir)Report.html”, but there must be some better way to do this.
Does anyone know any good add-ins, test runners or whatever that will let me
debug tests, instead of just running them
show the test results in a nice way inside Visual Studio
(Footnote: I'm running VS2010 Professional on Windows 7, if that matters.)
There are basically three options you have:
You can set up a custom tool in
Tools | External Tools to start
mspec.exe with the current project's
assembly to run the contexts and
generate the HTML report.
Install TestDriven.Net and
ensure that
Machine.Specifications.dll.tdnet
and
Machine.Specifications.TDNetRunner.dll
are in your project's copy of MSpec.
You can then run and debug your
contexts from the context menu: "Run
Tests", "Run With | Debugger" without further installation.
There's an example of what the MSpec folder
looks like for all of my projects.
If you use ReSharper 4.1, 4.5, 5.0
or the latest 5.1 EAP (== beta)
there are runners for each of these
versions.
The ZIP download
contains batch files that install
the runner for each respective
version of ReSharper. ReSharper's
unit test support is pretty
extensive in terms of UI
widgets/shortcut support, the reporting
tree view and debugging.
If you're
a dotTrace user you can also profile
right from within Visual Studio.
dotCover (another JetBrains product)
allows you to calcualate code
coverage results from your MSpec
runs.
On top of that, you get
all the nice coding and navigation features that ReSharper provides.
Be aware that only the first option will generate the HTML report as both the TestDriven.Net and ReSharper runners do not support HTML report generation. From my point of view this isn't an issue since the TD.Net and ReSharper runners offer fairly complete reporting mechanisms through the Visual Studio UI.
Another option that might work (I haven't used it myself) is to leverage the Gallio support that MSpec has. Gallio is a runner/framework for several testing frameworks; it might as well support debug runs with MSpec. Contact #smaclell if you have questions about Gallio support.