My company is upgrading our Visual Studio 2012 to 2013 Premium. In the midst of this, we are also looking to start automating our tests using Visual Studio Team Services
I have read a couple of MSTest vs nUnit posts and articles in the past but most of it compares the older version of MSTest. Also, nUnit had a lot of favourable reviews as compared to MSTest.
My question is, considering Microsoft's commitment towards the ALM, Agile practices and all the new stuff they've added into VS2013 Premium and Visual Studio Team Services to facilitate and encourage automated testing, how does MSTest compare to nUnit?
What other considerations should I take before making a decision on a testing framework to use?
MSTest hasn't changed much since it was originally introduced, so those old comparison posts are still valid. Microsoft's focus on ALM is mostly targeted at further tooling and server products, but the fundamental testing framework hasn't changed much.
It's also worth noticing that MSTest and their entire ALM strategy is targeted at many different styles of automated testing, including Integration Testing, System Testing, Performance Testing, etc., so while it attempts to be a one-size-fits-all, it's quite ill-suited for unit testing, because it's too heavy to work with.
While NUnit is better, I wouldn't recommend it either. It, too, hasn't changed much for years, and the extensibility model is frankly a mess.
Instead, I'd recommend xUnit.net. Although it's not perfect either, it's currently the best mainstream alternative on .NET. There are many reasons why xUnit.net is better than MSTest.
MSTest Vs NUnit:
MSTest is integrated with VS so it'll be easy to use. NUnit will require third-party tools (some are free, some are paid).
VS will give you Code Coverage in MSTest. NUnit requires DotCover (which is a paid tool).
MSTest has an option to execute your tests in parallel if they don't depend on each other. This isn't a feature that NUnit provides.
NUNit has TestCaseSourceAttribute which helps you to achieve parametrized test cases but in MSTest you'll need DataSourceAttribute which would be in XML file and will be difficult to manage when you have complex logic in the method.
NUnit is faster as compared to MSTest.
Overall both frameworks are good to use, but I'd suggest going for NUnit.
Related
What is the standard way to do BDD in VS2010?
SpecFlow or Cuke4Nuke.
Here's a link to the advantages and disadvantages of both projects: Cuke4Nuke or SpecFlow?
SpecFlow is probably what you're asking for. It's advantages (I lifted this from the aforementioned article):
A pure .NET solution (so no
installation of Ruby is necessary and
Ruby is not involved at runtime)
There is a basic integration with
VisualStudio (and there are plans to
evolve this)
Scenarios are basically UnitTests and
can be run with your existing
infrastructure (NUnit.Runners,
ReSharper, VisualStudio MSTest
Integration ...)
Scenarios and steps are easily
debuggable out of VisualStudio (just
set a breakpoint)
Integration in your continuous build
should be a breeze, since the
infrastructure to run unit-tests is
most certainly there already
Almighty Steve Sanderson has a blog post about using SpecFlow for BDD in Asp.Net MVC: http://blog.stevensanderson.com/2010/03/03/behavior-driven-development-bdd-with-specflow-and-aspnet-mvc/
Here's the SpecFlow homepage:
http://specflow.org/
Good luck!
I'm trying to find good comparison about performance of mstest(VS 2008) and nunit(newest).
I have found only articles about features, not examples about times of execution :/
I would be grateful for help
I have experience with using MSTest and NUnit. From my experience, MSTest is much slower than NUnit. The actual execution of tests may be almost the same, however, the setup and teardown time for MSTest is significantly more.
Note that I have run the MSTest unit tests through the integrated VS GUI, which is probably why it's so slow (compared to the light-weight NUnit GUI). Apparently you can run MSTest through the command line, which may improve the setup and teardown time.
There are other things to consider though. MSTest is missing a bunch of features that NUnit has and NUnit has been around longer is more mature. Also, bug fixes will arrive much faster with NUnit than MSTest (how often are service packs released?).
Also NUnit works without visual studio, which means you can run these tests on a server without needing another VS license. I have read ways to get MSTest running without VS, but it involves copying files, which may or may not violate the licensing terms.
The test runner actually does quite a lot - it has to find all your test classes, and then identify all the test methods before calling them (probably by means of reflection, but possibly not). It also has to assemble and report the results. All of these can be efficient or otherwise, so there could be a noticeable difference in performance especially with a large number of tests in the suite.
I am interested in improving my testing methodologies but am not sure if I prefer NUnit or VS 2008 test. I also have a book that uses NUnit in it's examples. Do the two frameworks coexist well with each other?
yes, check out this links NUnit/MSTest Dual Testing
and Strengthening Visual Studio Unit Tests
You would normally choose one or the other test framework rather than working with two. Although in fact, as Sergey mentions, NUnit and MSTest frameworks are similar enough that you can, with care, write common tests that can work with both frameworks. There are some subtle differences, however, so watch out and stick to the lowest common denominator!
If you want to run a suite of tests that are written for several frameworks, take a look at Gallio which supports this scenario.
If you use different frameworks in different classes, there is absolutely no problem, the can coexist without trouble (the project then references both VisualStudio test libraries and NUnit.Framework). You can also use hacks like the one Sergey posted to make them coexist on a same class but be aware that this has some limitation and you won't be able to use advanced functionalities of both frameworks and map them in this way (e.g. TestCase attribute for NUnit 2.5 etc.).
A colleague and I are writing an application that we intend to go Open Source with and are trying to decide on a testing framework. Without integration of NUnit (one of us has resharper while the other does not) we are leaning toward MSTest. However, if the majority of people that work on opensource .net applications only have the express edition of Visual Studio, we may have to change our mind on that.
What version of VS do you use and would MSTest turn you off to an Open Source Project?
Personally, I'm using VS:TS, so no, test units in MSTest wouldn't turn anything off for me ;), on the other hand, most probably, you will receive different answers from the majority of people, that are not professional programmers as well (working for money doesn't mean you can not work with / on OS projects) (the question is, how many you will find here).
However, if I understand you correctly, you don't want to use nUnit, because you it's not integrated with VS08, you might then check out these projects:
1. http://sourceforge.net/projects/visualnunit/
2. http://sourceforge.net/projects/nunitaddin/
Integrating NUnit with VS08 doesn't seem to the biggest problem, on the other hand, getting VS with built in testing framework might be a bit harder, at least for some people.
If I truly felt strongly enough to participate in the Open Source project, MSTest alone wouldn't be enough to turn me away. I'd be willing to adapt to whatever the development tools/environment was.
...and for what it's worth, I use the full version of Visual Studio with Resharper.
Frankly I never heared of anyone using the Express editions. everybody I know or I talked with either use the fully fledged Visual Studio environment, or they use the compiler/libraries from the free SDK and some editor on top of it (CodeWarrior, CodeInsight etc) and rely on some variant of make to build.
As for the test framework, is a double edged sword. I'd recommend NUnit, but then... remember what happened with NDoc?
I use MonoDevelop and NUnit.
I use VS 2k8 pro & MSTest as much as possible.
Even Microsoft's open source code (Prism/Composite Wpf) doesn't use MSTest. They use NUnit for that project, I believe.
You may be able to use visual nunit to add an nunit test runner in VS. link I've never used it but it may be worth a shot.
If your problem is integration with Visual Studio, try Gallio. It includes MbUnit, but its runner can run tests from other frameworks, like NUnit, xUnit, etc.
It would turn me off. See MSBuild, NAnt, NUnit, MSTest, and frustration (not written by me!) - I suspect a number of people feel that way. MSBuild seems like a poorly-done copy of NAnt (we use both), and MSTest likewise looks like Microsoft's usual "not invented here" answer to open source. While I realize that there are good business reasons for Microsoft to develop their own tools, those reasons don't apply to their customers.
There are plug-ins to use NUnit inside Visual Studio without ReSharper, or you can always configure it to run from the Tools menu.
If you don't like NUnit, please consider one of the other open source testing alternatives. See NUnit vs. MbUnit vs. MSTest vs. xUnit.net
I think that people who don't use windows will be knocked out of the project by the choice to use MSTest. Most other c# testing libraries are available in mono, which is what mac and linux users would be using.
I don't know how many cross-platform developers you are trying to attract, but if it's an issue then you need to consider Linux/Mac devs.
It seems like you should be able to configure your preferred unit testing framework as a plugin to Visual Studio/Team system like you can with SourceControl providers. Is anyone aware of an effort within microsoft to allow for swapping out MStest for nUnit, mbUnit or others. Obviosly nothing prevents me from using these other frameworks but they aren't integrated into the IDE.
If you have a look at TestDriven.NET and Gallio, they both have test runners that integrate to the IDE and allow the use of other testing frameworks.
If you are adverse to spending a little this is just on of the many benefits of ReSharper. (r#)
Not sure MS would want to encourage us to use another one unfortunately.
Kindness,
Dan
Visual Studio does provide the concept of custom test adapters that would allow custom test frameworks to execute under the MSTest platform. That is, tests for those frameworks would appear in the Test List Editor, etc.
The developers of the custom frameworks would have to develop a VSPackage that would find and provide tests to MSTest. Unfortunately, there's a lot of work involved.
TestDriven.NET, Resharper (and several others) provide their own VSPackage to provide their own Tool Window, but do not integrate directly with MSTest.
Looks like this is coming to Visual Studio 11