I'm trying to find good comparison about performance of mstest(VS 2008) and nunit(newest).
I have found only articles about features, not examples about times of execution :/
I would be grateful for help
I have experience with using MSTest and NUnit. From my experience, MSTest is much slower than NUnit. The actual execution of tests may be almost the same, however, the setup and teardown time for MSTest is significantly more.
Note that I have run the MSTest unit tests through the integrated VS GUI, which is probably why it's so slow (compared to the light-weight NUnit GUI). Apparently you can run MSTest through the command line, which may improve the setup and teardown time.
There are other things to consider though. MSTest is missing a bunch of features that NUnit has and NUnit has been around longer is more mature. Also, bug fixes will arrive much faster with NUnit than MSTest (how often are service packs released?).
Also NUnit works without visual studio, which means you can run these tests on a server without needing another VS license. I have read ways to get MSTest running without VS, but it involves copying files, which may or may not violate the licensing terms.
The test runner actually does quite a lot - it has to find all your test classes, and then identify all the test methods before calling them (probably by means of reflection, but possibly not). It also has to assemble and report the results. All of these can be efficient or otherwise, so there could be a noticeable difference in performance especially with a large number of tests in the suite.
Related
My company is upgrading our Visual Studio 2012 to 2013 Premium. In the midst of this, we are also looking to start automating our tests using Visual Studio Team Services
I have read a couple of MSTest vs nUnit posts and articles in the past but most of it compares the older version of MSTest. Also, nUnit had a lot of favourable reviews as compared to MSTest.
My question is, considering Microsoft's commitment towards the ALM, Agile practices and all the new stuff they've added into VS2013 Premium and Visual Studio Team Services to facilitate and encourage automated testing, how does MSTest compare to nUnit?
What other considerations should I take before making a decision on a testing framework to use?
MSTest hasn't changed much since it was originally introduced, so those old comparison posts are still valid. Microsoft's focus on ALM is mostly targeted at further tooling and server products, but the fundamental testing framework hasn't changed much.
It's also worth noticing that MSTest and their entire ALM strategy is targeted at many different styles of automated testing, including Integration Testing, System Testing, Performance Testing, etc., so while it attempts to be a one-size-fits-all, it's quite ill-suited for unit testing, because it's too heavy to work with.
While NUnit is better, I wouldn't recommend it either. It, too, hasn't changed much for years, and the extensibility model is frankly a mess.
Instead, I'd recommend xUnit.net. Although it's not perfect either, it's currently the best mainstream alternative on .NET. There are many reasons why xUnit.net is better than MSTest.
MSTest Vs NUnit:
MSTest is integrated with VS so it'll be easy to use. NUnit will require third-party tools (some are free, some are paid).
VS will give you Code Coverage in MSTest. NUnit requires DotCover (which is a paid tool).
MSTest has an option to execute your tests in parallel if they don't depend on each other. This isn't a feature that NUnit provides.
NUNit has TestCaseSourceAttribute which helps you to achieve parametrized test cases but in MSTest you'll need DataSourceAttribute which would be in XML file and will be difficult to manage when you have complex logic in the method.
NUnit is faster as compared to MSTest.
Overall both frameworks are good to use, but I'd suggest going for NUnit.
There is option of Create Unit Tests in VS2010.
It creates a separate test project and uses MS test frame work. Is it possible for configure VS to use Nunit framework to create unit test cases?
From what i know Visual Studio generates only MSTests. But there are many commercial tools (You can always evaluate :)) which are quite helpful.
For example:
Other topic
Parasoft dotTEST
Probably by your own You will find few more.
Though I ask it to, Visual Studio 2010 does not appear to generate code coverage statistics for my F# modules. The unit tests are Ms Tests written in C#, so are run natively by Visual Studio, and execute cleanly. But there does not appear to be any coverage information created for the F# library being tested. The C# libraries involved are analyzed just fine.
Am I missing something?
This link describes using the VS testing system with F#. You do it pretty much the same way as with C#.
The downside is that apparently the VS IDE won't automatically pick it up -- I believe you need to run from the command line using mstest.exe.
Edit: Oh, another cool thing with F#, is FsCheck, an automatic testing tool for F#, is basically a port of Haskell's QuickCheck. It allows you to provide a specification of the program, in the form of properties that the functions or methods should satisfy, and FsCheck tests that the properties hold in a large number of randomly generated cases.
also fsunit and fstest
I am interested in improving my testing methodologies but am not sure if I prefer NUnit or VS 2008 test. I also have a book that uses NUnit in it's examples. Do the two frameworks coexist well with each other?
yes, check out this links NUnit/MSTest Dual Testing
and Strengthening Visual Studio Unit Tests
You would normally choose one or the other test framework rather than working with two. Although in fact, as Sergey mentions, NUnit and MSTest frameworks are similar enough that you can, with care, write common tests that can work with both frameworks. There are some subtle differences, however, so watch out and stick to the lowest common denominator!
If you want to run a suite of tests that are written for several frameworks, take a look at Gallio which supports this scenario.
If you use different frameworks in different classes, there is absolutely no problem, the can coexist without trouble (the project then references both VisualStudio test libraries and NUnit.Framework). You can also use hacks like the one Sergey posted to make them coexist on a same class but be aware that this has some limitation and you won't be able to use advanced functionalities of both frameworks and map them in this way (e.g. TestCase attribute for NUnit 2.5 etc.).
A colleague and I are writing an application that we intend to go Open Source with and are trying to decide on a testing framework. Without integration of NUnit (one of us has resharper while the other does not) we are leaning toward MSTest. However, if the majority of people that work on opensource .net applications only have the express edition of Visual Studio, we may have to change our mind on that.
What version of VS do you use and would MSTest turn you off to an Open Source Project?
Personally, I'm using VS:TS, so no, test units in MSTest wouldn't turn anything off for me ;), on the other hand, most probably, you will receive different answers from the majority of people, that are not professional programmers as well (working for money doesn't mean you can not work with / on OS projects) (the question is, how many you will find here).
However, if I understand you correctly, you don't want to use nUnit, because you it's not integrated with VS08, you might then check out these projects:
1. http://sourceforge.net/projects/visualnunit/
2. http://sourceforge.net/projects/nunitaddin/
Integrating NUnit with VS08 doesn't seem to the biggest problem, on the other hand, getting VS with built in testing framework might be a bit harder, at least for some people.
If I truly felt strongly enough to participate in the Open Source project, MSTest alone wouldn't be enough to turn me away. I'd be willing to adapt to whatever the development tools/environment was.
...and for what it's worth, I use the full version of Visual Studio with Resharper.
Frankly I never heared of anyone using the Express editions. everybody I know or I talked with either use the fully fledged Visual Studio environment, or they use the compiler/libraries from the free SDK and some editor on top of it (CodeWarrior, CodeInsight etc) and rely on some variant of make to build.
As for the test framework, is a double edged sword. I'd recommend NUnit, but then... remember what happened with NDoc?
I use MonoDevelop and NUnit.
I use VS 2k8 pro & MSTest as much as possible.
Even Microsoft's open source code (Prism/Composite Wpf) doesn't use MSTest. They use NUnit for that project, I believe.
You may be able to use visual nunit to add an nunit test runner in VS. link I've never used it but it may be worth a shot.
If your problem is integration with Visual Studio, try Gallio. It includes MbUnit, but its runner can run tests from other frameworks, like NUnit, xUnit, etc.
It would turn me off. See MSBuild, NAnt, NUnit, MSTest, and frustration (not written by me!) - I suspect a number of people feel that way. MSBuild seems like a poorly-done copy of NAnt (we use both), and MSTest likewise looks like Microsoft's usual "not invented here" answer to open source. While I realize that there are good business reasons for Microsoft to develop their own tools, those reasons don't apply to their customers.
There are plug-ins to use NUnit inside Visual Studio without ReSharper, or you can always configure it to run from the Tools menu.
If you don't like NUnit, please consider one of the other open source testing alternatives. See NUnit vs. MbUnit vs. MSTest vs. xUnit.net
I think that people who don't use windows will be knocked out of the project by the choice to use MSTest. Most other c# testing libraries are available in mono, which is what mac and linux users would be using.
I don't know how many cross-platform developers you are trying to attract, but if it's an issue then you need to consider Linux/Mac devs.