I am new to testing so please bear with the obvious unfamiliarity.
I have a solution with multiple projects, configurations and app settings. There is already a compiler directive to warn on non-Release builds but this is often ignored. I'm looking for a more visible warning during testing. Something that acts as a checklist for testers to do before they can proceed.
I have a unit test project in the solution. There are three questions here:
Can a unit test be set up to run as a post-build event?
Is there a way to prevent publishing the solution in non-Release mode?
Is using a unit test trigger standard practice?
If there are other alternatives, what are they?
Related
I have experience unit testing but I have never unit tested a Xamarian solution.
My question is should I add two unit test projects to my solution? One would be a MSTest or NUnit test that tests the logic and another project Xamarian.UITest that only tests the UI? From my browsing on the internet it seems the Xamarian.UITest is only able to test the UI? Is this true or can you test the logic as well?
Thanks.
Don't try to make Xamarin.UITest test the entire application. Use Xamarin.UITest to automate the testing of the UI and, providing you have a decent separation of concerns, use xUnit, NUnit, etc, to test the logic part(s) of your app.
I created a build definition that runs automated tests using MTM build environments and test suites. I recently created a Visual Studio Load Test, which can be added to a test suite just like any test method marked with the [TestMethod] attribute. However, when I run the build, I get no errors and it appears the aggregate tests don't run. Is there a way to make this work?
I found this article: https://blogs.msdn.microsoft.com/testingspot/2013/01/22/how-to-automatically-run-a-load-test-as-part-of-a-build/ which describes a way to do it, but I can't find a build template that matches what he describes, and it appears this only allows you to run a single load test.
Also, when you configure a test controller, there is an option to configure it for load testing, but to do this, you must unregister it from the Team Project Collection. If this is done, it appears the controller can no longer be used in an environments to run project automated tests. This defeats the purpose of what I want to do and makes it seem that Load Tests and Team Projects are mutually exclusive. Is this the case? If so, this is a big oversight. Load tests are the kind of thing you would like to run automatically. Thanks for the help.
You are unfortunately right. A test controller used for load testing cannot be used for other automated test execution 'at the same time'. In your scenario I would recommend that you setup a different test controller and agent for load testing and you would be able to queue it as a part of your build to achieve what you are looking for.
There is no special build process template for this case.
I am aware of multiple methods one can use to keep certain types of test out of a test playlist: filtering by TestCategory or class name etc.
I am also aware that one can instruct TFS builds to only run certain categories or classes of test.
However, I find it quick and convenient to be able to check my unit test runs just by going into Test Explorer and clicking "run all". It's good practice to do this regularly and prior to check-in to ensure the build is likely to pass.
Is there a straightforward way I can configure my tests to ensure that "run all" just picks up the quick unit tests, and leaves the slower system and regression tests alone? Ideally, I'd like a method that can easily be applied to everyone else working on the code at the same time.
It's a little incomprehensible. Since the MSDN explains very clear.Run all just mean run all tests. And there are also filters or other settings can achieve just run unit tests.
And also as your have mentioned, you can run unit tests with your builds directly. Why you need this function. If you just want to make sure the check-in will not cause build fail. You can use Gated Check-in directly. This also applied to everyone else.
We use Visual Studio 2010 Ultimate with tests written in MSTest. Both our unit tests and integration tests* are written in MSTest.
**By our definition, an integration test is an MSTest TestMethod that takes a while to run and/or calls out to external components such as a database or web services.*
I'm looking for a way of easily filtering out the integration tests so that only unit tests run without all the integration tests running too.
My ideas so far:
Mark integration tests with the [Ignore] attribute. This works but is a real pain when you do want to run the integration tests.
Assign a [TestCategory] attribute to the different test types. This allows them to be run separately but only through the Test View panel. You can't use CTRL+R, A (Run All Tests in Solution) or other similar shortcuts/hotkeys.
The integration tests are in a separate project, is there something that could be done to stop them running at the project level? As long as it's easy to toggle.
Write the integration tests in a different test framework, e.g. NUnit. This would keep them totally separate from a tooling point of view.
Does anyone have any other suggestions? Are there any plug-ins that can help with this?
I recommend having a different project (or projects) for integration tests because depending on your runner the only real way to run or not run tests across all runners is to include or not include a test class library.
But, I also recommend, if you're using MSTest, to use the TestCategoryAttribute to tag non-unit tests. You can then "filter" tests to be run in Test View with MSTest.
Runners like Resharper and apparently TestDriven.net (http://bit.ly/tmtYC2) allow you to then filter-out those tests from general unit-test executions.
If your unit test project is in a separate namespace, you could use the keyboard shortcut CTRL+R, T to run all tests in the current context (i.e. namespace MyApp.Tests.Unit). To do this place the cursor just after the opening curly brace in the namespace clause of any unit test class.
I have a suggestion but you won't like it.
Abandon MSTest entirely, while other unit test frameworks have been evolving MSTest as almost stopped in time. Yes, it has a major benefit of integrating directly with VS, but if I'm not mistaken that will change in VS 2011 which will provide native support for custom unit test runners integration.
(Note: The stopped in time part may be not true because I confess not paying to much attention to MSTest since I used it sparingly with VS 2008)
I use NUnit and separate my unit tests from the integration tests by using a different class library project. Then I automate the running of the tests using Gallio command line runner allowing me to configure separate scripts for running unit and integration tests.
Finally, personal opinions aside, I'm not sure but the TestDriven.net plugin may have support for running tests with a specific category only, so you could check that.
Is it possible to enable Resharper to automatically run all unit tests within a solution automatically when performing a local build using Visual Studio? I believe NUnit has an option "re-run last tests run" but I don't see such an option through Resharper or VS.
Thanks in advance
JP
[Edit] This may or may not matter, but I should point out that I am a one-man-team writing a medium sized application.
Posting this one for anyone that comes across it in the future:
If you're running Visual Studio 2012, someone wrote a plugin that will do exactly what you are looking for:
http://visualstudiogallery.msdn.microsoft.com/5dca9c5c-29cf-4fd7-b3ff-573e5776f0bd
I don't think ReSharper has that option, but you could always turn to something like NCrunch or Continuous Testing one of the other continuous testing frameworks for .NET. Just be aware that they are still fairly new and a little rough around the edges.
I suggest creating a build script (MSBuild, nAnt, rake, whatever) - use this script to build your solution then run tests on it.
Separating your build process from your IDE will allow you to add more things to the build (other types of tests, static checks, style checks etc).
See this blog entry on the subject.
You need dotCover, which is technically part of ReSharper Ultimate. dotCover is JetBrains' testing coverage extension. Once you have this extension there is a "Continuous Testing" feature that you can turn on and configure it to run tests on build, save, etc in addition to code coverage.
I think you're on the right track though, test early and as often as possible while you are writing the code.