I was researching unit testing frameworks and the Wikipedia list has a column which lists whether a framework is considered the "xUnit type" or "compatible". Mocha was listed as not being of the "xUnit type" – why? What are the core features of the xUnit family?
XUnit frameworks share the following concepts:
Test runner - the program that runs the tests
Test case - Where all tests inherit from
Test fixture - the state needed to run the tests
Test suites - tests that share the same fixture
Assertion - the function that verifies the state of the test
Test formatter - shows the results un one or more formats. This is a bit tricky, since the formats are not always the same. For example, Inria's page specifies the xml tags as test-case and test-suite. JUnit, on the other hand, uses testcase and testsuite.
You can think of XUnit frameworks as *Unit... where the * is replaced by the language (e.g., JUnit for Java).
What's very tricky is that XUnit.net is different from XUnit. XUnit.net is a framework on itself that also incorporates these aforementioned concepts. The XML output format uses different tags though... such as assembly, class, etc. It can get very confusing when googling for issues.
Related
I am trying to apply sonarqube analysis to our system. The system is roughly laid out in multiple modules. We have some small and medium tests in each module - and hope to create large tests in the future. We are trying to use "Google" test naming.
Sonarqube seems to refer to unit and integration (roughly equivalent to small and medium in our environment). I'm wondering if anyone knows a simple way to modify the labeling to better match what we are trying to setup.
This is not possible to change the labels in SonarQube. Unit and integration tests are two very common types of tests, IMO you should stick to this convention.
Just to share some information: at SonarSource, we have unit tests, medium tests and integration tests, and when we analyse our code on our internal SonarQube instance, the medium tests end up in the "unit test" category (they are executed at the same time BTW).
I was able to switch the labels Unit -> Small and Integration -> Medium by creating a language pack plugin. I started from the French language pack, and modified the existing core.properties file. The solution gives me a "localized" version of site using our naming convention.
I was working on setting up Selenium in a project today, and the thought came to mind: "Should I be using the NUnit framework in correlation with my Selenium tests?"
Here's my concern with using the NUnit framework: From the NUnit website, it states that: "NUnit is a unit-testing framework for all .Net languages". The purpose of the framework is to build unit tests, not integration tests.
Selenium tests are typically (I don't know of any instance when they aren't) integration tests. So, going back to my question, is it good practice to use a unit testing framework to do integration tests? Are there integration test frameworks that are robust enough to compete with NUnit, of which would be more appropriate?
The two main options for .NET development are, for the most part, NUnit, and MSTest. IMHO, neither are very well optimized for automated browser tests.
You can force each to do what you want, but features such as TestNG's #DataProvider are a pain to implement, especially if you are dynamically changing your data provider for each test (say, loading your browsers to test through a properties file) -- this is trivial with TestNG, but NUnit and MSTest take a significant amount of "hacking" to make it work.
TL;DR version: Someone really needs to port TestNG over to .NET :)
I have all the tests for my web application (written with the Visual Studio test framework -- Microsoft.Quality DLLs) divided into several (currently two) ordered tests. Is there an easy way to find all the tests that are not in any list?
(The reason I need to use ordered tests is because the initial tests test that installation/setup/configuration of my application worked, and subsequent tests would fail without that.)
There's on easy way to do this. The best thing to do is switch to a framework that doesn't require every test to be on a list -- I recommend MbUnit. It has a great DependsOn attribute to easily configure dependencies between tests.
We use Visual Studio 2010 Ultimate with tests written in MSTest. Both our unit tests and integration tests* are written in MSTest.
**By our definition, an integration test is an MSTest TestMethod that takes a while to run and/or calls out to external components such as a database or web services.*
I'm looking for a way of easily filtering out the integration tests so that only unit tests run without all the integration tests running too.
My ideas so far:
Mark integration tests with the [Ignore] attribute. This works but is a real pain when you do want to run the integration tests.
Assign a [TestCategory] attribute to the different test types. This allows them to be run separately but only through the Test View panel. You can't use CTRL+R, A (Run All Tests in Solution) or other similar shortcuts/hotkeys.
The integration tests are in a separate project, is there something that could be done to stop them running at the project level? As long as it's easy to toggle.
Write the integration tests in a different test framework, e.g. NUnit. This would keep them totally separate from a tooling point of view.
Does anyone have any other suggestions? Are there any plug-ins that can help with this?
I recommend having a different project (or projects) for integration tests because depending on your runner the only real way to run or not run tests across all runners is to include or not include a test class library.
But, I also recommend, if you're using MSTest, to use the TestCategoryAttribute to tag non-unit tests. You can then "filter" tests to be run in Test View with MSTest.
Runners like Resharper and apparently TestDriven.net (http://bit.ly/tmtYC2) allow you to then filter-out those tests from general unit-test executions.
If your unit test project is in a separate namespace, you could use the keyboard shortcut CTRL+R, T to run all tests in the current context (i.e. namespace MyApp.Tests.Unit). To do this place the cursor just after the opening curly brace in the namespace clause of any unit test class.
I have a suggestion but you won't like it.
Abandon MSTest entirely, while other unit test frameworks have been evolving MSTest as almost stopped in time. Yes, it has a major benefit of integrating directly with VS, but if I'm not mistaken that will change in VS 2011 which will provide native support for custom unit test runners integration.
(Note: The stopped in time part may be not true because I confess not paying to much attention to MSTest since I used it sparingly with VS 2008)
I use NUnit and separate my unit tests from the integration tests by using a different class library project. Then I automate the running of the tests using Gallio command line runner allowing me to configure separate scripts for running unit and integration tests.
Finally, personal opinions aside, I'm not sure but the TestDriven.net plugin may have support for running tests with a specific category only, so you could check that.
I just upgraded to XCode 4 and I was wondering if I need to 'include unit tests' when setting up an application? Also, what does that mean exactly?
You do not need to include unit tests.
What does "unit testing" mean? (from the unit-testing FAQ)
Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure. Unit tests are created by programmers or occasionally by white box testers.
Ideally, each test case is independent from the others: substitutes like method stubs, mock objects, fakes and test harnesses can be used to assist testing a module in isolation. Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended.Wikipedia
Unit testing is closely related to Test Driven Development.
#ToddH points out:
It's easier to include [unit tests] when you setup the project. If you do it later there are quite a few steps involved in doing it correctly: http://twobitlabs.com/2011/06/...
Thanks for the protip, Todd!